-
Notifications
You must be signed in to change notification settings - Fork 679
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CI: Disable PR mutants actions #5806
Comments
tagging @ASuciuX - maybe there's some other way we can still get the mutant data outside of ci |
Thank you for the tag! In theory, we could run them periodically (weekly/monthly) on a VM only on the code changes that happened on the For example, mutating 40 functions would take 40 × 400 minutes = 1 week and 4 days to complete. If we could categorize Stacks-core subprojects into: We could limit the tests to only the relevant subprojects. However, this would still leave gaps in coverage for the ones that need integration tests. |
Yep -- that's one of the reasons I'd advocate for more unit testing of the codebase. A lot of things that are currently only covered by integration tests could be covered by unit tests, which wouldn't have all the overhead of having to spin up a chain, etc. |
Are there any subprojects within stacks-core that contain their own test suites without relying on tests from other subprojects? If so, I could keep only those and remove the test runs from all the others. Otherwise, I can remove them entirely, either by preventing them from being triggered or by deleting them altogether. |
The PR mutants actions aren't really providing the review process or PR authors with much value at the moment. Lots of the mutants are covered by integration tests. The integration tests don't get run in the mutant suite, and it'll be a non-trivial challenge to make that happen (the total runtime of the integration tests is like 400-500 minutes, which is pretty untenable for running about mutants), which means we get mutant coverage reports which are overly pessimistic. These reports are, in virtually all cases, ignored.
Now, I think mutant reports could be very useful to us, but the current way we're using them just isn't yet. Getting to a place where they are more useful will require non-trivial effort beyond devops and CI: I suspect we need to create a unit testing framework which can run much more quickly and cover portions of our codebase that are currently only covered via integration tests. This would be a great thing to do, but we aren't there yet, and it isn't a quick and simple project.
Until that time, I think it's simply a waste to expend the amount of time we currently do on the mutant CI actions: looking at action usage, mutant jobs are 9 of the top 13 jobs by usage, and 16% of the total action usage of the last month (note, they don't run on PRs that touch only stack-signer, and over the last month, I frequently just manually canceled these jobs).
The text was updated successfully, but these errors were encountered: