filtering

Filtering and selecting tests

The tests that are selected to run in a given vvtest invocation is called "filtering the tests", which is important when managing a large set of tests. Described next are the three main ways to filter tests via the command line: by keyword expression, by parameter expression, and by platform expression. Test filtering based on command line options is described in a different section.

Filtering by keyword

Using -k on the command line, filters (selects) tests based on the keywords defined in each test. For example, consider two test files, "test1.vvt":

#VVT: keywords : 3D mhd circuit
print ( 'running test1' )

and "test2.vvt":

#VVT: keywords : 3D mhd conduction
print ( 'running test2' )

Using the command vvtest -k 3D would cause both tests to run, because they both have the keyword "3D". Using the command vvtest -k circuit would only run "test2", because only that test has the "circuit" keyword defined.

Multiple -k options are AND'ed together. So vvtest -k 3D -k conduction means run tests that satisfy the statement "3D is present in the keywords AND conduction is present".

The option -K has the same syntax as -k but means exclude tests that have the keyword. So vvtest -K conduction means run tests such that "NOT conduction is present" or equivalently, "the keyword conduction is NOT present".

Both these options allow complex expressions, such as the following:

expression	meaning
-k foo -K bar	foo and not bar ("foo" is present but "bar" is a keyword)
-k 'foo or bar'	foo or bar ("foo" or "bar" is a keyword)
-k foo -K 'foo and bar'	foo and not (foo and bar)
-k 'foo and not (foo and bar)'	foo and not (foo and bar)
-K 'foo or bar'	not (foo or bar) equivalent to not foo and not bar
-k 'foo*'	run any test that has a keyword starting with "foo"
-K 'foo*'	exclude any test that has a keyword starting with "foo"
-k 'foo* and not foo-*	a keyword starts with "foo" but does not match "foo-*"
-k foo/bar	foo or bar
-k foo/bar/baz	foo or bar or baz
-K foo/bar	not foo or not bar (yes, the or here is correct)

Pattern matching is simple shell style (not regular expression).

Note that the "/" operator is a convenience shorthand and cannot be used in more complex expressions.

Filtering by parameters

When tests define parameters using the "parameterize" directive, then the resulting parameter names and values can be used to select tests. Consider the test file "p1.vvt"

#VVT: parameterize : np = 1 4
print ( 'running test p1' )

and the test file "p2.vvt"

#VVT: parameterize : MODEL = elastic elasticplastic
print ( 'running test p2' )

The command vvtest -p np would only run test "p1", because the "np" parameter is only defined in that test file and not in "p2". (More accurately, it would run both the "p1.np=1" and "p1.np=4" tests.) In general, specifying a parameter name means include the test if the parameter is defined by the test.

The value of a parameter can be specified as well. For example, the command vvtest -p MODEL=elastic would only run the "p2.MODEL=elastic" test and no others. In general, "-p name=value" means run any test that defines the parameter "name" and which has the value "value".

More comparison operators in addition to "=" can be used, such as ">", "<", ">=", "!=", etc. For example, the command vvtest -p 'np>1' would run the "p1.np=4" test and no others.

Similar to keyword expressions, the -P option excludes tests that satisfy the parameter expression. For example, the command vvtest -P np would run the two "p2" tests because neither of them define the parameter "np".

More complex expressions are supported:

expression	meaning
-p np=4	np and np=4 (same as "np is defined and np equals 4")
-p 'np>=2' -p 'np<=8'	(np and np>=2) and (np and np<=8)
-p 'not np or not ndevice'	not np or not ndevice (same as "np is not defined or ndevice is not defined")
-p '!np and ndevice!=1'	not np and ndevice!=1
-p np -P ndevice	np and not ndevice
-P 'ndevice>4'	not (ndevice and ndevice>4) equivalent to not ndevice or ndevice<=4
-p 'foo and not (foo and bar)'	foo and not (foo and bar)
-p foo/bar	foo or bar
-P foo/bar	not foo or not bar (yes, the or is correct)

Note that the "/" operator is a convenience shorthand and cannot be used in more complex expressions.

Also note that ">" and "<" are special shell characters and will have to be quoted (such as single quotes). The exclamation point is also special and usually has to be escaped with a backward slash.

Lastly, If the value of the parameter appears to be an integer or floating point number, then it is interpreted that way. Otherwise, less than and greater than are alphabetical.

Filtering by platform

A test can use the "enable" test directive to limit the platforms that will run the test. For example, the test "atest.vvt"

#VVT: enable (platforms=Darwin)
pass

will only run if the platform name is "Darwin". Expressions are allowed as the 'platform' attribute value, such as "platforms=Darwin or Linux", or "platforms=not Darwin".

The platform filtering options -x and -X can be used to modify this behavior. Suppose the current platform is "Darwin", then the command vvtest -x Linux would not run the test "atest" (because it only runs on "Darwin"). In general, "-x platname" means run tests that would run on the platform name "platname". Conversely, "-X platname" means run tests that would not run on platform name "platname".

More complex expressions can be specified on the command line, such as the following:

expression	meaning
-x Linux -x Darwin	tests that run on "Linux" and run on "Darwin"
-x 'Linux and not Darwin'	tests that run on "Linux" but not on "Darwin"
-x Linux -X Darwin	tests that run on "Linux" and not on "Darwin"
-x 'Linux and not (Darwin or Windows)'	tests that run on "Linux" but not on "Darwin" and not on "Windows"
-x Linux/Darwin	tests that run on "Linux" or on "Darwin"
-X Linux/Darwin	tests that do not run on "Linux" or do not run on "Darwin"

Note that the "/" operator is a convenience shorthand and cannot be used in more complex expressions.

Finally, note that the platform expressions in tests are case insensitive. So this test

#VVT: enable (platforms=darwin or lin*)
pass

would run run on platform names "Darwin" and on "Linux".