by Juan Altmayer Pizzorno and Emery Berger at UMass Amherst's PLASMA lab.
CoverUp adds to your module's test suite so that more of its code gets tested (i.e., increases its code coverage). It can also create a suite from scratch if you don't yet have one. The new tests are based on your code, making them useful for regression testing.
CoverUp is designed to work closely with the pytest test framework. To generate tests, it first measures your suite's coverage using SlipCover and selects portions of the code that need more testing. It then engages in a conversation with an LLM, prompting for tests, checking the results and re-prompting for adjustments as necessary. Finally, CoverUp checks that the new tests work well together, attempting to resolve any issues it finds.
CoverUp is available on PyPI, so you can install simply with
python3 -m pip install coverupCoverUp currently requires an OpenAI account to run.
Your account will also need to have a positive balance.
Create an API key and store its "secret key" (usually a
string starting with sk-) in an environment variable named OPENAI_API_KEY:
export OPENAI_API_KEY=<...your-api-key...>If your module's source code is in src and your tests in tests, you can run CoverUp as
coverup --source-dir src --tests-dir testsCoverUp then creates tests named test_coverup_N.py, where N is a number, under the tests directory.
Here we have CoverUp create additional tests for the popular package Flask:
$ coverup --source-dir src/flask --tests-dir tests
Measuring test suite coverage... starting coverage: 90.2%
Prompting gpt-4-1106-preview for tests to increase coverage...
100%|███████████████████████████████████████████████████| 95/95 [02:49<00:00, 1.79s/it, usage=~$3.30, G=51, F=141, U=22, R=0]
Checking test suite... tests/test_coverup_2.py is failing, looking for culprit(s)...
Disabling tests/test_coverup_19.py
Checking test suite... tests ok!
Measuring test suite coverage... end coverage: 94.2%
In just short of 3 minutes, CoverUp increased Flask's test coverage from 90.2% to 94.2%.
It then detected that one of the new tests, test_coverup_19, was causing another test
to fail and disabled it.
That test remains as disabled_test_coverup_19.py, where it can be reviewed for the cause
and possibly re-added to the suite.
To evaluate the tests generated by the LLM, CoverUp must execute them. To minimize security concerns, including the risk of damage to your system, we recommend that you run CoverUp using Docker.
Please enjoy this early release of CoverUp, and pardon any disruptions we may cause while we actively develop it.
This material is based upon work supported by the National Science Foundation under Grant No. 1955610. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.