Project specific configuration and plugins

The config directory

vvtest can be given a directory in which it looks for a control plugin, named vvtest_user_plugin.py. This file can contain functions which vvtest will call at various points in the testing process (described below). Also, this directory is added to the Python path so tests can easily import and use utilities specialized for the project.

The config directory can be given to vvtest on the command line, with the option "--config <path>".

If not on the command line, vvtest will look for the environment variable called VVTEST_CONFIGDIR.

If neither of those, it will look for a subdirectory called config in the directory containing the script being executed (even if a soft link).

The plugin file

The Python plugin file is imported during vvtest startup and can contain pretty much anything, but only certain function names are called.

• prologue( argv ): This function is called once during vvtest startup. The 'argv' argument is a Python list of the vvtest command line arguments (actually just sys.argv).
• results_directory( platform_name, options ): This function is called to determine the directory containing test results files (which are written with the --save-results option), and is used to read previous test runtimes.
• validate_test( testinfo ): Called prior to execution of each test. If this function returns a non-empty string, the test is skipped and the string will be the reason reported to the user.
  • The 'testinfo' argument is a Python dictionary containing information on the test, such as "name", "keywords", "parameters", and "timeout" as well as the current "platform" and "options"
• test_runtime( testinfo ): Called to determine or override a test runtime value (the time it takes to run the whole test).
  • If the return value is not None, it is treated as the runtime in seconds. If None, the runtime is unchanged
  • The argument 'testinfo' is the same as for validate_test()
• test_timeout( testinfo ): Called to determine or override a test timeout value.
  • If the return value is not None, it is treated as a timeout in seconds. If None, the timeout is unchanged
  • The argument 'testinfo' is the same as for validate_test(), which may contain the current "timeout"
• test_preload( testinfo ): Called just prior to the execution of a test.
  • The argument 'testinfo' is the same as validate_test(), except that an additional dictionary entry may be present with key "preload". Its value is taken from the "#VVT: preload = value" header directive, if present
  • If the return value of test_preload() is a string, it will be used to execute the test script
  • The original use case was for executing the test script using a specific Python version, but any script or executable may be specified
  • If the returned path has the extension .py, then Python is used to execute the script
• epilogue( testdict ): This function is called once during vvtest shutdown.
  • The 'testdict' is a dictionary mapping test names to 'testinfo', where 'testinfo' is the same as that given to validate_test(), but with additional entries
  • The dict will contain "result", whose value will be one of pass, fail, diff, timeout, notdone, notrun, or skip
  • 'runtime' will contain the seconds of runtime for the test
  • 'rundir' will contain the test execution directory

The bootstrap file

If a file named vvtest_bootstrap.py sits next to vvtest or a soft link to vvtest, then that file is imported during startup. The primary use case is for this file to set the VVTEST_CONFIGDIR environment variable to the config directory.