Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve accounting of tests marked as driver bugs #2177

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

LingMan
Copy link

@LingMan LingMan commented Oct 29, 2024

  1. Count tests marked as driver bugs as executed, since they actually do get run.
  2. Currently all tests marked as driver bugs get counted as "bug" no matter if they succeed or fail. Count them separately to make it clear how many of them actually fail.

Before:

d3d12: 3872234 tests executed (0 failures, 412 successful todo, 88 skipped, 332 todo, 2738 bugs).

After:

d3d12: 3874972 tests executed (0 failures, 412 successful todo, 88 skipped, 332 todo, 1 bugs, 2737 successful bugs).

@doitsujin
Copy link
Collaborator

"successful bugs" reads a bit weird, maybe "fixed" would be better?

Just a nit though, I do agree that we should only count bugs as actual bugs if the corresponding test fails.

@LingMan
Copy link
Author

LingMan commented Oct 29, 2024

Right, I actually used "fixed" at first. Then I thought that a passing test doesn't say all that much about bugs being fixed. With flaking tests, tests that only fail on some hardware, etc. That made me mirror the todo/successful todo wording instead.

Happy to use whatever terminology you'd prefer though.

@Blisto91
Copy link
Contributor

Blisto91 commented Oct 31, 2024

Can't it just be explicitly named? Something like "not our bugs", "driver bugs" or "invalid bugs" etc. Or does that sound weird.

Currently all tests marked as driver bugs get counted as "bug" no matter if
they succeed or fail. This makes it clearer how many of them actually fail.
Calling them "driver bugs" is clearer than just "bugs", especially since
another category is called "failures".
@LingMan
Copy link
Author

LingMan commented Nov 1, 2024

  • Rephrased from "successful" to "fixed". That terminology is already used in the individual lines that get printed for each test marked as bug. Almost more importantly, it's kind of the natural term, even if maybe not 100% accurate. Everyone who's going to run these tests will know how to interpret "fixed" in this context.

  • Added a third commit rephrasing from "bug" to "driver bug". Not as short, but I do remember asking about the difference between "bug" and "failure" in the past. Makes it clearer that it's (probably) not a bug in vkd3d-proton.

@R1chterScale
Copy link

Forgive me if I'm misunderstanding, but this is bugs that are from native DX12 yes? If so, perhaps "Bugs replicated" might be a preferred term?

@LingMan
Copy link
Author

LingMan commented Dec 14, 2024

No, these are bugs in the underlying Vulkan driver and should ideally be fixed there. Modulo misattributions of course.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants