diff --git a/README.md b/README.md index cdc5910..365eef3 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ CoverUp is available on PyPI, so you can install simply with python3 -m pip install coverup ``` -CoverUp currently requires an [OpenAI account](https://platform.openai.com/signup) to run. +While work is underway to support local LLMs, CoverUp currently requires an [OpenAI account](https://platform.openai.com/signup) to run. Your account will also need to have a [positive balance](https://platform.openai.com/account/usage). Create an [API key](https://platform.openai.com/api-keys) and store its "secret key" (usually a string starting with `sk-`) in an environment variable named `OPENAI_API_KEY`: @@ -57,11 +57,21 @@ to fail and disabled it. That test remains as `disabled_test_coverup_19.py`, where it can be reviewed for the cause and possibly re-added to the suite. -### Best Used With Docker +### Better With Docker To evaluate the tests generated by the LLM, CoverUp must execute them. To minimize security concerns, including the risk of damage to your system, we recommend that you run CoverUp using [Docker](https://www.docker.com/). +## Evaluation + + +The graph shows CoverUp in comparison to the state-of-the-art [CodaMosa](https://www.carolemieux.com/codamosa_icse23.pdf), +which improves upon the [Pynguin](https://github.com/se2p/pynguin) test generator by incorporating LLM queries. +The bars show the difference in coverage percentage between CoverUp and CodaMosa for various Python modules; +green bars, above 0, indicate that CoverUp achieved a higher coverage. +As the graph shows, CoverUp achieves better coverage in almost every case. +Both CoverUp and CodaMosa created tests "from scratch", that is, ignoring any existing test suite. + ## Work In Progress Please enjoy this early release of CoverUp, and pardon any disruptions while we actively develop it.