Testing

Testing

Before making a Pull Request to the ESCMI/CIME github, test your changes.

---

Changes to python

CIME uses doctest and unittest for unit testing, and unittest for regression tests. All of these tests can be run at once by running scripts_regression_tests.py.

Required

Run the regression test suite.

cd to utils/python/tests ::

./scripts_regression_tests.py

This will run X and A system tests (the same as what you get from running ./create_test cime_developer) in addition to the python-specific tests. If you want to skip the system tests, you can add --fast, but running the system tests is required prior to making a pull request.

** On Yellowstone **

1. Set the CIME_MODEL to cesm in the environment
2. Make sure you have modules git/2.3.0, python/2.7.7 and pylint/1.6.4 loaded
3. Run using execca to get a caldera node, tests will not run on login nodes

Recommended

scripts_regression_tests B_CheckCode will run pylint on all python files with a few exceptions.

The above command invokes "code_checker" that lives in scripts/Tools. It supports checking the full code base in parallel and checking individual files. It also encapsulated the pylint settings that the team has agreed upon.

scripts/Tools/advanced-py-prof Will profile your python code.

Useful

python doctest (https://docs.python.org/2/library/doctest.html) is used in many of the modules and the doctests are run as part of the A_RunUnitTests portion of scripts_regression_tests.py It's also possible to run doctest on an individual file. To do this you would run PYTHONPATH=$CIMEROOT/scripts/lib python -mdoctest testthis.py where testthis.py is the module you are interested in testing.

Continuous Intgeration Testing

CIME uses Jenkins to run scripts_regression_tests nightly. Latest results are on http://my.cdash.org/index.php?project=CIME.

What if something fails?

scripts_regression_tests.py outputs one line for each test class (some classes include more than one test). If the test passes that line will look like:

test_a_createnewcase (__main__.J_TestCreateNewcase) ... ok

if however the test fails that line will look like:

test_build_time_base_flags (__main__.H_TestMakeMacros) Test selection of base flags based on build-time attributes. ... FAIL

After all tests have completed, more detailed output will be printed for failed tests:

FAIL: test_build_time_base_flags_same_parent (__main__.I_TestCMakeMacros)

Test selection of base flags in the same parent element.

----------------------------------------------------------------------

Traceback (most recent call last):

File "./scripts_regression_tests.py", line 1800, in test_build_time_base_flags_same_parent

`tester.assert_variable_equals("FFLAGS", "-O2")`

File "./scripts_regression_tests.py", line 1599, in assert_variable_equals

`self.parent.assertEqual(self.query_var(var_name, env, var), value)`

AssertionError: '-O3' != '-O2'

If you want to rerun just the failed test you can do it by specifying the test name on the command line:

scripts_regression_tests.py H_TestMakeMacros

---

Changes to Fortran or to the test scripts in SystemTests

If there are changes to Fortran code in CIME or to the scripts in /utils/python/CIME/SystemTests, you must run the cime_developer tests.

1. Run the X and A system tests with ::

   ./create_test cime_developer

Note: The list of cime_developer tests is defined in cime/utils/python/update_acme_tests.py

As noted under "Changes to python", the cime_developer system tests are run automatically if you run scripts_regression_tests.py without the --fast argument - so if you have already run that, you do not have to separately run the above command.

2. If any changes have been made to Fortran code in share/ or driver_cpl/, then run cime unit tests: From the top-level directory in cime, follow the instructions in README.unit_testing (currently only works out-of-the-box on yellowstone).

In addition to CIME standalone testing there is testing associated with each of the CIME models - ACME and CESM. The workflows for each modeling group are a little different and are described below:

• CESM System testing workflow