Testing

Before making a Pull Request to the ESCMI/CIME repository on GitHub, test your changes.

Testing changes

CIME splits it's testing into two categories unit and sys.

The unit category covers both doctests and unit tests. While the sys category covers regression tests. The tests are named accordingly e.g. unit tests are found as CIME/tests/test_unit*.

There are two possible methods to run these tests.

pytest

CIME also supports running tests using pytest. By using pytest coverage reports are automatically generated.

To get started install pytest and pytest-cov.

pip install pytest pytest-cov
# or
# conda install -c conda-forge pytest pytest-cov

Examples

Run all tests

pytest

Running tests by category

pytest CIME/tests/test_unit*
# or
pytest CIME/tests/test_sys*

Running a specific test file

pytest CIME/tests/test_unit_case.py

Running a specific test case

pytest CIME/tests/test_unit_case.py::TestCaseSubmit::test_check_case

scripts_regression_tests.py

The scripts_regression_tests.py script is located under CIME/tests.

You can pass either the module name or the file path of a test.

Examples

Running all tests

python CIME/tests/scripts_regression_tests.py

Running tests by category

python CIME/tests/scripts_regression_tests.py CIME/tests/test_unit*
# or
python CIME/tests/scripts_regression_tests.py CIME/tests/test_sys*

Running a specific test file

python CIME/tests/scripts_regression_tests.py CIME.tests.test_unit_case

Running a specific test case

python CIME/tests/scripts_regression_tests.py CIME.tests.test_unit_case.TestCaseSubmit.test_check_case

Recommended

./scripts_regression_tests.py B_CheckCode will run pylint on all python files with a few exceptions.

The above command invokes "code_checker" that lives in scripts/Tools. It supports checking the full code base in parallel and checking individual files. It also encapsulated the pylint settings that the team has agreed upon.

For more information on using scripts_regression_tests, see the CIME manual

scripts/Tools/advanced-py-prof Will profile your python code.

Useful

python doctest (https://docs.python.org/2/library/doctest.html) is used in many of the modules and the doctests are run as part of the A_RunUnitTests portion of scripts_regression_tests.py It's also possible to run doctest on an individual file. To do this you would run PYTHONPATH=$CIMEROOT/scripts/lib python -mdoctest testthis.py where testthis.py is the module you are interested in testing.

Continuous Intgeration Testing

CIME uses Jenkins to run scripts_regression_tests nightly. Latest results are on http://my.cdash.org/index.php?project=CIME.

What if something fails?

scripts_regression_tests.py outputs one line for each test class (some classes include more than one test). If the test passes that line will look like:

test_a_createnewcase (__main__.J_TestCreateNewcase) ... ok

if however the test fails that line will look like:

test_build_time_base_flags (__main__.H_TestMakeMacros) Test selection of base flags based on build-time attributes. ... FAIL

After all tests have completed, more detailed output will be printed for failed tests:

FAIL: test_build_time_base_flags_same_parent (__main__.I_TestCMakeMacros)

Test selection of base flags in the same parent element.

----------------------------------------------------------------------

Traceback (most recent call last):

File "./scripts_regression_tests.py", line 1800, in test_build_time_base_flags_same_parent

`tester.assert_variable_equals("FFLAGS", "-O2")`

File "./scripts_regression_tests.py", line 1599, in assert_variable_equals

`self.parent.assertEqual(self.query_var(var_name, env, var), value)`

AssertionError: '-O3' != '-O2'

If you want to rerun just the failed test you can do it by specifying the test name on the command line:

scripts_regression_tests.py H_TestMakeMacros

Changes to Fortran or to the test scripts in SystemTests

If there are changes to Fortran code in CIME or to the scripts in /utils/python/CIME/SystemTests, you must run the cime_developer tests.

Run the X and A system tests with ::

./create_test cime_developer

Note: The list of cime_developer tests is defined in cime/scripts/lib/update_acme_tests.py

As noted under "Changes to python", the cime_developer system tests are run automatically if you run scripts_regression_tests.py without the --fast argument - so if you have already run that, you do not have to separately run the above command.

If any changes have been made to Fortran code in share/ or driver_cpl/, then run cime unit tests: From the top-level directory in cime, follow the instructions in README.unit_testing (currently only works out-of-the-box on yellowstone).

In addition to CIME standalone testing there is testing associated with each of the CIME models - ACME and CESM. The workflows for each modeling group are a little different and are described below: