Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Disable PR mutants actions #5806

Open
kantai opened this issue Feb 5, 2025 · 4 comments
Open

CI: Disable PR mutants actions #5806

kantai opened this issue Feb 5, 2025 · 4 comments
Labels
Milestone

Comments

@kantai
Copy link
Member

kantai commented Feb 5, 2025

The PR mutants actions aren't really providing the review process or PR authors with much value at the moment. Lots of the mutants are covered by integration tests. The integration tests don't get run in the mutant suite, and it'll be a non-trivial challenge to make that happen (the total runtime of the integration tests is like 400-500 minutes, which is pretty untenable for running about mutants), which means we get mutant coverage reports which are overly pessimistic. These reports are, in virtually all cases, ignored.

Now, I think mutant reports could be very useful to us, but the current way we're using them just isn't yet. Getting to a place where they are more useful will require non-trivial effort beyond devops and CI: I suspect we need to create a unit testing framework which can run much more quickly and cover portions of our codebase that are currently only covered via integration tests. This would be a great thing to do, but we aren't there yet, and it isn't a quick and simple project.

Until that time, I think it's simply a waste to expend the amount of time we currently do on the mutant CI actions: looking at action usage, mutant jobs are 9 of the top 13 jobs by usage, and 16% of the total action usage of the last month (note, they don't run on PRs that touch only stack-signer, and over the last month, I frequently just manually canceled these jobs).

@github-project-automation github-project-automation bot moved this to Status: 🆕 New in Stacks Core Eng Feb 5, 2025
@kantai kantai added the CI label Feb 5, 2025
@kantai kantai added this to the 3.1.0.0.6 milestone Feb 5, 2025
@wileyj
Copy link
Collaborator

wileyj commented Feb 5, 2025

tagging @ASuciuX - maybe there's some other way we can still get the mutant data outside of ci

@ASuciuX
Copy link
Contributor

ASuciuX commented Feb 5, 2025

Thank you for the tag!

In theory, we could run them periodically (weekly/monthly) on a VM only on the code changes that happened on the develop branch. But in practice, the integration tests take too long to be feasible.

For example, mutating 40 functions would take 40 × 400 minutes = 1 week and 4 days to complete.

If we could categorize Stacks-core subprojects into:
1. Those requiring the integration test suite
2. Those that don’t

We could limit the tests to only the relevant subprojects. However, this would still leave gaps in coverage for the ones that need integration tests.

@aldur aldur moved this from Status: 🆕 New to Status: 📋 Backlog in Stacks Core Eng Feb 6, 2025
@kantai
Copy link
Member Author

kantai commented Feb 6, 2025

For example, mutating 40 functions would take 40 × 400 minutes = 1 week and 4 days to complete.

Yep -- that's one of the reasons I'd advocate for more unit testing of the codebase. A lot of things that are currently only covered by integration tests could be covered by unit tests, which wouldn't have all the overhead of having to spin up a chain, etc.

@ASuciuX
Copy link
Contributor

ASuciuX commented Feb 10, 2025

Are there any subprojects within stacks-core that contain their own test suites without relying on tests from other subprojects? If so, I could keep only those and remove the test runs from all the others.

Otherwise, I can remove them entirely, either by preventing them from being triggered or by deleting them altogether.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Status: 📋 Backlog
Development

No branches or pull requests

3 participants