TESTING.md

• Testing
  • Code Coverage Collecting
    • OpenSearch plugin
      • Gradle project
      • Maven project
      • Limitation
    • OpenSearch Dashboards plugin
  • Code Coverage Reporting
    • Code coverage report upload through CI workflow
    • A status badge in README file to show the code coverage
    • Codecov integration in Pull Requests

Testing

Code Coverage Collecting

Code coverage is a measurement of how many lines of code are executed under the test suites.

OpenSearch plugin

It is recommended to use JaCoCo library to generate the code coverage report. The following guide may help you use JaCoCo in OpenSearch plugins with ease.

Gradle project

1. Add JaCoCo Gradle plugin to build.gradle in your repository. example
2. Set Jacoco to export coverage report in xml format. example
3. Call gradle task "jacocoTestReport" to generate report (./gradlew jacocoTestReport) after running your tests. (Or add a dependsOn to make "jacocoTestReport" task depending on your test tasks. example)

Maven project

Since OpenSearch uses Gradle as the build tool, we don't recommend using Maven. Code coverage for a Maven project can be generated by JaCoCo Maven plugin, and here is an example.

Limitation

1. JaCoCo Gradle plugin can not collect code coverage properly for integration tests (tests that extend OpenSearchIntegTestCase or OpenSearchRestTestCase, see here to learn more). However, there is a workaround to collect coverage when the integration test is running on a local single node cluster. For your reference, this file is a custom Gradle plugin to help achieve, and it can be applied conditionally like here.
2. Additional work is needed to get coverage report by JaCoCo for the test classes which use PowerMock, because those classes must be instrumented before the test executes. Please see the PowerMock wiki for detail, and there is an example for using JaCoCo Maven plugin. But for JaCoCo Gradle plugin, there is no native support, see the issue in gradle repository for detail.
3. JaCoCo Gradle plugin can not merge code coverage reports from different sub-projects out-of-box, when the code repository contains a Gradle multi-project build. If you need the aggrectiated code coverage report from multiple sub-projects, please check the example of a custom Gradle plugin in OpenSearch Alerting plugin, or the sample from Gradle.

OpenSearch Dashboards plugin

OpenSearch Dashboards mainly uses Jest for unit testing, and plugins are recommended to utilize Jest as the unit testing framework.

To generate code coverage report under Jest framework, adding the flag --coverage in the command to run unit tests. See here for an example in the CI workflow.

Code Coverage Reporting

OpenSearch/Dashboards plugins are using Codecov for code coverage reporting and analysis. The dashboard can be seen here.

All the OpenSearch plugins are required to have the following items:

Code coverage report upload through CI workflow

1. Generate code coverage report in xml format in a CI workflow of Github Actions
2. Upload the report to Codecov by adding a step of codecov-action in the workflow.
3. For private repositories, an upload token is needed, visit the repository page of Codecov to get your token: https://codecov.io/gh/opensearch-project/REPO-NAME-HERE. Please follow Github docs to use Github Secrets to store the token and not expose to the public.
4. Set the workflow to be triggered by push and pull request on main and release branches.
5. To see your code coverage results from Codecov, visiting https://codecov.io/gh/opensearch-project/REPO-NAME-HERE after the CI workflow is triggered.

See CI Workflows for example.

A status badge in README file to show the code coverage

See here for an example of the status badge.

Add the following line at the header of the README markdown file of the repository:

[![codecov](https://codecov.io/gh/opensearch-project/REPO-NAME-HERE/branch/main/graph/badge.svg)](https://codecov.io/gh/opensearch-project/REPO-NAME-HERE)

Link of the badge can also be accessed from the repository's settings page of Codecov: https://app.codecov.io/gh/opensearch-project/REPO-NAME-HERE/settings/badge

Codecov integration in Pull Requests

1. Add a codecov.yml file into the repository (example)
2. Set the target code coverage properly in case it blocks your PR in Github checks.

See Codecov docs to learn more about codecov yaml file and common configurations.