2.1 Directive Reference

INCLUDE FILE

OPTINCLUDE FILE

INCLUDE and OPTINCLUDE insert FILE into the current Auto-pilot configuration. If the file does not exist, then INCLUDE causes the script to fail, whereas OPTINCLUDE does not raise any errors. OPTINCLUDE is useful to allow users to define optional local settings.

Auto-pilot first reads the entire configuration, then processes it. This means that you can not use variable substitution within INCLUDE or OPTINCLUDE.

INCLUDEPATH PATH

Insert PATH into the list of directories to search for INCLUDE files.

Auto-pilot first reads the entire configuration, then processes it. This means that you can not use variable substitution within INCLUDEPATH.

ECHO STRING

Prints STRING on standard out, leading and trailing spaces are stripped.

VAR VAR=VAL

VAREX VAR=VAL

Sets the value of the internal Auto-pilot variable VAR to VAL. VAREX is the same, but VAL is first passed to Perl's eval function.

ENV VAR=VAL

ENVEX VAR=VAL

Sets the value of the environment variable VAR to VAL. This also updates the corresponding internal variable with the same name. ENVEX is the same, but VAL is first passed to Perl's eval function.

FOREACH VAR VAL1 [VAL2 ... VALN]

Set VAR to VAL1 and then execute all directives until a corresponding DONE is reached. Not ending the loop with a DONE statement results in an error. VAR is then set to VAL2, and the directives are executed again. This process repeats until VALN is reached. Each value is separated by one or more whitespace characters.

FOR VAR=BEGIN TO END [FACTOR M|ADD I]

Set VAR to BEGIN, then execute all directives until a corresponding DONE is reached. Not ending the loop with a DONE statement results in an error. VAR is then updated and the directives are executed until VAR reaches END. If FACTOR is specified, then VAR is multiplied by M after each iteration. If ADD is specified then VAR is incremented by I after each iteration. If neither FACTOR nor ADD is specified, then VAR is incremented by one.

IF VAR OP VAL

VAR is the name of an Auto-pilot variable. VAL is a value for comparison. OP can "=", ">", "<", "<=", or ">=". A "!" before the operator negates the result of the comparison. Only "=" and "!=" can be used with strings. For more complicated conditions you can use VAREX, which sets a variable to an evaluated Perl expression. For example, if you wanted to test a complex condition such as a regular expression, you could use the following code:

          VAREX SCSI=if ("%TESTDEV%" =~ /sd.[0-9]$/) \
              { return 1; } else { return 0; }
          IF SCSI=1
                  ECHO %TESTDEV% is a SCSI device.
          ELSE
                  ECHO %TESTDEV% is not a SCSI device.
          FI
     

If the condition evaluates to true, then Auto-pilot executes all lines until an ELSE or a FI. If there is an optional ELSE block, then the code between ELSE and FI is only executed if the condition was false. Auto-pilot also supports an arbitrary number of ELSE IF blocks, with the expected semantics. For example:

          IF FOO = 0
                  ECHO Foo is zero.
          ELSE IF FOO < 0
                  ECHO Foo is negative.
          ELSE
                  ECHO Foo is positive.
          FOO
     

WHILE VAR OP VAL

WHILE repeatedly executes all directives until the corresponding DONE, as long as the condition remains true. Not ending the loop with a DONE statement results in an error. See IF for a description of valid conditions.

RESULTS=PATH

Sets the path for the files that contain the Auto-pilot results. PATH must be a directory that already exists. The file names within this directory are determined by the NAME in the TEST directive.

LOGS=PATH

Sets the path for benchmark log files (these files record the stdout and stderr streams). As with RESULTS, PATH must be a directory that already exists, and the file names within this directory are determined by the NAME in the TEST directive.

THREADS=N

By default, Auto-pilot runs a single benchmark process at a time. The THREADS directive instructs Auto-pilot to create N processes when running a benchmark. Each process executes the same command, but they each have a different value for the environment variable APTHREAD (ranging from 1 to N). By using the value of APTHREAD, each process can perform a slightly different action (e.g., use separate test directories or thread 1 may execute the server while thread 2 executes a client).

Semaphores are used to synchronize the start of multi-threaded benchmarks. Before executing a benchmark, a semaphore is set to the number of threads that will be executed. The semaphore's key is stored in the environment variable APIPCKEY. Using the semdec utility, this semaphore can be decremented by each of the threads. semdec then blocks until the semaphore reaches zero.

CHECKPOINT FILE

Write all internal Auto-pilot state to FILE. If Auto-pilot is invoked with FILE as its argument instead of a script, then it reads the state and resumes execution where it left off (starting with the line after checkpoint).

After checkpoint executes, the value of the Auto-pilot variable RESTORE is "0", but after resuming from the checkpoint the value of RESTORE "1". This can be thought of like the return value of Unix's fork system call. For example, if RESTORE is 0, then your configuration could reboot the system, but it should not reboot the system after restoration (because it should continue running benchmarks).

TEST NAME EPOCHS [INCREMENT PREDICATE]

The TEST directive begins a benchmark description, which is ended by a corresponding DONE directive. Not ending the loop with a DONE statement results in an error. Each benchmark can have several additional directives between the TEST and DONE, including control structures (e.g., IF) and benchmark scripts. TEST directives can not be nested.

Log and results files are saved to a file, which has a name derived from the benchmark NAME. If NAME is /dev/null, then both the results and log file are discarded. If NAME is any other name that begins with a slash, then the absolute path is used, but the results are stored in NAME.res and the stdout/stderr streams are stored in NAME.log. Finally, if the name does not begin with a slash, then the directories previously specified by RESULTS and LOGS are prepended to the results and log file name, respectively. For example, if the results directory is /home/cwright/results and the NAME is "ext2:1", the results are stored in /home/cwright/results/ext2:1.res.

EPOCHS specifies the minimum number of iterations to run the benchmark. If no additional arguments are specified, then the benchmark runs exactly EPOCHS times. INCREMENT and PREDICATE must be specified together, and control how many additional times the benchmark should be run. After running the benchmark EPOCH times, Auto-pilot runs the script PREDICATE with an argument of its results file each additional INCREMENT iterations. If predicate returns true (an exit-code of zero), then the benchmark is finished. Otherwise, the benchmark is run INCREMENT more times, and the predicate is checked again. See Getstats, for information on using Getstats to evaluate predicates based on the results.

SETUP SCRIPT

CLEANUP SCRIPT

PRESETUP SCRIPT

POSTCLEANUP SCRIPT

EXEC SCRIPT

These directives run a benchmark script. Each TEST directive can include any number of EXEC, SETUP, CLEANUP, PRESETUP, and POSTCLEANUP scripts. EXEC is the simplest one, and the script it specifies is run for each iteration of the test. EXEC should be used for the actual processes that are being measured. If THREADS has been specified, EXEC spawns multiple threads.

SETUP is also run for each iteration of the test, but should be used for scripts that prepare the test environment (e.g, mounting file systems and clearing the cache). CLEANUP is the complement of SETUP. It is also run for each iteration of the test, but should be used to restore the test environment to a "clean" state (e.g., unmounting test file systems). Only one instance of SETUP and CLEANUP scripts are run at a time, even if THREADS has been set.

PRESETUP and POSTCLEANUP are similar to SETUP and CLEANUP, but they are only run once per TEST directive. PRESETUP only runs on the first epoch, and POSTCLEANUP runs only after the last epoch. These directives can be used for more global setup and cleanup procedures that need not be repeated for each iteration of a benchmark (e.g., creating a data set that is used by each iteration of a read-only benchmark). Only one instance of PRESETUP and POSTCLEANUP scripts are run at a time, even if THREADS has been set.

QUIET

QUIET (true|false)

By default, Auto-pilot sends a benchmark's output to stdout and a log file. Setting QUIET to true prevents benchmarks from sending output to stdout, but still sends the output to a log file. If neither true or false is specified after QUIET, then it is sent to true. This option can prevent "noisy" benchmarks from being artificially slowed due to lots of information being sent to the monitor (or network).

FASTFAIL

FASTFAIL SCRIPT

FASTFAIL causes Auto-pilot to immediately abort if a single benchmark run fails. By default, Auto-pilot continues running the next iteration of the benchmark (or if all iterations are complete, moves on to the next directive). The optional SCRIPT parameter tells Auto-pilot to additionally run a script (e.g., to send you an email or page informing you of the failure).

EVAL PERL

Execute the Perl snippet PERL.

STOP

Terminates execution of Auto-pilot immediately.

SYSTEM PROGRAM

Execute the program PROGRAM. The return code is stored in the Auto-pilot variable RETURN.