Next: , Previous: Executing Auto-pilot, Up: Top


4 Benchmarking Scripts

Auto-pilot includes benchmarking scripts to run Postmark and compile benchmarks on several file systems. We also include hooks that add functionality to the basic scripts, including measuring network utilization, I/O operations, benchmarking JFS and XFS, benchmarking stackable file systems and more. The scripts run as the same user as Auto-pilot, so many benchmarks can be executed without root privileges. However, our file system scripts often need to format, mount, and unmount partitions: which requires root access on most Unix systems.

All of the included scripts first require commonsettings, which is located in /usr/local/share/auto-pilot by default. commonsettings first verifies that various Auto-pilot variables are set. Auto-pilot automatically sets the following variables:

APEPOCH
An integer value with the iteration of the benchmark
APTHREAD
An integer value that indicates our thread number (each thread in a multi-threaded benchmark gets a unique number from 1 to THREADS).
APMODE
A string containing the mode that Auto-pilot is executing in. This matches the directive used to execute the script. Possible values are SETUP, EXEC, CLEANUP, PRESETUP, and POSTCLEANUP.
APLOG
The log file this thread should write results to. Each thread's log is concatenated into the final results file.
THREADS
An integer value that indicates how many total threads there are.

You must set the following variable manually:

TESTROOT
What directory the test file system should be mounted on. There are also other variables that commonsettings sets to reasonable default values:

commonsettings automatically assigns sensible defaults to these variables:

TESTDIR
What directory the test should be executed in, by default TESTROOT.
BLOCKSIZE
The block size of the file system.
APTIMER
The program that is executed to time your workload. By default it is aptime $APLOG.

commonsettings includes commonfunctions, which has various shell functions. For extensibility, commonsettings searches each path in the APLIB environment variable for a directory named commonsettings.d, each file in that directory is sourced by the shell. Additionally, commonsettings.`uname -n` is loaded. uname -n expands to the machine's host name, so different machines can have slightly different configurations, but share scripts. The functions defined by commonfunctions are as follows:

ap_action
Execute another program or function. Print [ OK ] or [ FAILED ] (in color) based on the exit code.


ap_hook
ap_requirehook
These functions are used to execute hooks, which are described in Hooks. These functions take one or more arguments. The first argument is the type of hook that should be executed, and the remaining arguments are hook specific. For a hook of type TYPE, the value of the environment variable AP_TYPE_HOOK is executed. If no hook is defined, then ap_hook returns success, but ap_requirehook returns failure.


ap_log
ap_logexec
These functions append to the current thread's results log (each thread has a separate results log that Auto-pilot combines into a single log after each test). ap_log uses echo to write its arguments to the file, whereas ap_logexec executes a program and saves its stdout stream to the log file.


ap_measure
This function actually takes a measurement for a process and creates a measurement block within the results file. A measurement block begins with [measurement], and each line has a name-value pair separated by an equal sign. An example of a measurement block is: 
          [measurement]
          thread = 2
          epoch = 2
          command = postmark /tmp/postmark_config-9868
          user = 0.200000
          sys = 1.170000
          elapsed = 5.470682
          status = 0

The [measurement], thread, and epoch lines are produced by ap_measure. Then a measurement hook is called. The first argument to a measurement hook is either premeasure, start, end, or final. The remaining arguments are the program being executed. The measure hook is called with premeasure first. A premeasure hook can be used to execute relatively long running initial measurements that should not be included by other hooks, or perform any other operation that prepares for measurement. Next, the measurement hook is called with start, which should save any information that is needed for a measurement (e.g., the initial value of a measured quantity). At this point the actual benchmark is executed through $APTIMER. By default, the aptime program is used. aptime produces the command, user, sys, elapsed, and status portions of the measurement block. Finally, the measurement hook is called two more times. First, it is called with an argument of "end". At this point measurements should complete, and more quantities can be added to the measurement block. New blocks should not be created yet, because other measurement hooks may want to add quantities to the main measurement block. After all measurement hooks have been executed with the "end" argument, they are called one last time with "final." At this point new blocks can be created.


ap_snap
ap_snap adds several blocks to the file that describe the machine being executed on. The first is a uname block that prints the vital statistics from uname. A users block contains the output of w. cpuinfo, meminfo, and mounts contain the contents of their respective /proc files. df contains the output of df -k, and if TESTDEV is set to an Ext2 file system, dumpe2fs contains the output of dumpe2fs -h. This snapshot of the machine state can be used to reproduce results more accurately, or help explain why results are different.


ap_unmount
ap_unmount attempts to unmount the file system mounted on a directory or device. If the file system is not mounted, then it returns success. If the umount command fails, or file system is not unmounted after an unmount is attempted, then it returns failure. Before executing the umount command, an unmount hook is called with an argument of the file system to be unmounted.
commonsettings concludes by running ap_snap on the setup phase of the first epoch.

See File System Scripts, for information on how Auto-pilot formats, mounts, and unmounts file systems. See Postmark, for information about how Auto-pilot runs the Postmark benchmark. See Compile Benchmarks, for information on executing compile benchmarks. See Included Script Plugins, for information about the various plugins that are included with Auto-pilot's benchmarking scripts.