Command Line Usage

Output from -h:

    -f, --fail-fast                  Stop testing on first failure
    -b, --fail-blank                 Fail if there are no examples
    -d, --dry-run                    Don't run any tests, output what would have run
    -r, --rand                       Randomize the execution order of tests
    --seed INTEGER                   Set the seed for the random number generator (implies -r)
    --order ORDER                    Set the test execution order. ORDER should be one of: defined, rand, or rand:SEED
    -e, --example STRING             Run examples whose full nested names include STRING
    -l, --line LINE                  Run examples whose line matches LINE
    --location FILE:LINE             Run the example at line 'LINE' in the file 'FILE', multiple allowed
    --tag TAG                        Run examples with the specified TAG, or exclude examples by adding ~ before the TAG.
    -v, --verbose                    Verbose output using document formatter
    -h, --help                       Show this help
    -p, --profile                    Display the 10 slowest specs
    --json                           Generate JSON output
    --tap                            Generate TAP output (Test Anything Protocol)
    --junit_output OUTPUT_DIR        Generate JUnit XML output
    --html_output OUTPUT_DIR         Generate HTML output
    --no-color                       Disable colored output

Run Modes

You can change how Spectator runs tests by using the following options.

Fail-fast

Enable fail-fast with the -f or --fail-fast option. This will abort as soon as failing test is encountered.

Fail-blank

Enable fail-blank with the -b or --fail-blank option. This will consider the test run to be a failure if no tests ran. You may want to enable this if you have shady developers on your team. 😒

Dry-run

Enable dry-run with the -d or --dry-run option. This will skip all tests, but output them as passed.

Random Test Execution

Tests can be run in the order they were defined, or a random order. Sometimes running tests in a random order shake out bugs that might not normally appear. There are multiple ways to control randomness in Spectator.

The simplest way to enable random test execution is by specifying the -r or --rand option. This will randomize the order that tests are run in.

A seed can be supplied to the internal random number generator. Using the same seed across runs will generate the same random numbers and test order. To supply a seed, use --seed INTEGER. Using --seed also implies -r.

Lastly, to mimic RSpec, there is a --order option. One of three things can be provided to --order. The first, defined - --order defined will run tests in the order they are written/defined. The test execution order will not be randomized. Second, rand - --order rand will run tests in a randomized order. This has the same effect as -r. Lastly, rand:SEED - --order rand:SEED where SEED is an integer to seed the random number generator with. This is the same as --seed.

Test Filtering

Tests can be filtered out so that they don't run. This is different than skipping tests. Tests that are filtered out do not appear as pending.

There are multiple ways to filter tests. The simplest is to just pass the files you want to test to crystal spec. For instance:

crystal spec spec/core/*_spec.cr

Tags are the next easiest way to filter tests. Use the --tag TAG[:VALUE] option to specify which tags to include. Place a ~ before the tag to exclude it, i.e. --tag ~slow.

Additionally, you can specify options to filter the tests in included files.

First is -e STRING or --example STRING, where STRING is the full name of the example. Second is by line number, -l LINE or --line LINE. This will run tests only on the specified LINE. The test's line number is the line the it keyword is on. Lastly is by file and line number. Use --location FILE:LINE to specify the file and line number of the test to run.

Output

By default, Spectator will use a "dots" formatter. This produces a dot for every passed test, and another character for tests that failed or were skipped. A summary of the test run will follow, with details of any failures that occurred.

To list the names of examples and contexts/groups as they run, use the -v or --verbose option. If you don't want colorized output, then specify the --no-color option. To generate profiling information and display the slowest tests, use the -p or --profile option.

Formatters

Spectator can output in other formats for different testing frameworks. The ones available at the time of this writing are: JSON, TAP, and JUnit. See the documentation on output formats for more information.

To output in JSON format, specify the --json option. The output will not be formatted, so you'll want to use a tool like jq to view it. The JSON output has the most detailed output of all the formatters. It includes expectation details and test values.

To output in TAP format, specify the --tap option.

To produce an HTML report, use the --html_output OUTPUT_DIR option. The OUTPUT_DIR is a directory to place the generated report in. This can simply be . for the current directory. The generated HTML file will be called output.html.

To produce a JUnit report, use the --junit_output OUTPUT_DIR option. The OUTPUT_DIR is a directory to place the generated report in. This can simply be . for the current directory. The generated XML file will be called output.xml.