better heuristics for baseline matching

[KotlinIsland]

Currently it's just +/- 50 lines, what if the following information was used for matching:

• The ast/src for the line of code of the error
• The name of the enclosing function
• offset from class / function
• version control information

Other Ideas:

• if real error 2 matches a baseline error, but 20 lines later, apply that offset to subsequent errors.

[Zeckie]

Could it also use version control history where available?

eg. to determine which lines have changed since the baseline was generated, and which lines have just moved due to other changes

[Zeckie]

And as a workaround if this is going to take a while to fix, maybe a switch to ignore line numbers (but still check that there aren't new errors)

[DetachHead]

if basedmypy was git aware it could just filter out any errors that aren't on new code without the need for a baseline file. instead there could be a file that maps commit hashes to new flags (ie. from this commit onwards, these are the rules)

[Zeckie]

The change causing the error could also be to configuration / external dependencies

[Zeckie]

It could store the commit hash of lines in the baseline json, and use it as an additional hint to use when matching lines to errors.

If run on files with no local changes, it could store the commit hash when each source file was last modified. When checking a later version, it could use a git diff to determine exactly where / how many lines were inserted or deleted. This could get complicated if commits are rebased (commits referenced in the baseline file might not be accessible)

[akaihola]

changes to one line cause an error on another line

I've been pondering about how to solve this in Darker. See this discussion comment.

(Darker is a tool which, among other things, can run mypy and filter the output to just those lines modified since a given commits)

[Zeckie]

Another, related idea - use git diffs to adjust line numbers stored in the baseline.

When writing the baseline, store the current commit hash (should be ok to store one one hash for the baseline, as long as not using submodules). Line numbers should be stored as at that commit (so would need to be adjusted if there are local changes in the file that add or remove lines above the baselines error). When checking a later commit (or even the same commit with different local changes) perform the same line number changes.

For example, if there was an error at line 10 as at the commit referenced in the baseline, and there had since been 2 lines added after line 5 (and no other changes to the file), that error should now be at line 12.

This would also mean that the baseline wouldn't need to be rewritten in cases where there are no changes to baselined errors.

[akaihola]

That sounds like a plan @Zeckie. And it utilizes very similar techniques to what Darker does when deciding which Black reformattings to apply.

Basically Darker compares a file before and after Black reformatting and generates a sequence of diff chunks, and ends up with the following kind of information:

• file path relative to repository root: path/foo.py
• list of line numbers in <commit2> for those lines with a leading + in git diff <commit1>..<commit2>
• contents of each line in path/foo.py at <commit2> before reformatting
• contents of each line in path/foo.py after reformatting
• diff chunks generated by comparing contents before and after reformatting, for example:
  • old lines 1–5 equal to new lines 1–5
  • old lines 6–7 delete
  • old lines 8–10 equal to new lines 6–8
  • old line 11 replace with new lines 9–18
  • old line 12 equal to new line 19
  • insert new lines 20–25
  • old lines 13–15 equal to new lines 26–28

Based on that information, it can iterate through the diff chunks, the "<commit2> before reformatting" contents and the "after reformatting" contents in parallel, re-assemble the contents chunk by chunk, and decide at each replace chunk which version to take it from – depending on whether any of the line numbers of + lines in git diff overlap with the chunk in "<commit2> before reformatting".

For example, if git diff shows that I've changed a word on line 11 of the old file, then lines 9–18 of the reformatted file should be picked. But if line 11 was unchanged, we pick the chunk from the "<commit2> before reformatting" version instead. As a result, no Black reformattings are applied unless they fall on areas which were modified between e.g. main and the head of the feature branch.

As you can see, the diff chunks give us a mapping between equal line numbers between two versions of the file. In the baseline case, the diff chunks would of course be generated from git diff output instead.

If we were to implement baseline support for darker --lint mypy, we'd probably not store the baseline anywhere, but simply run mypy twice – once for each of the commits we want to compare – so it would be possible to run it on any two commits at any time without any preparation.

Update: Landing in upcoming Darker 1.7.0, see akaihola/darker#393.

[Zeckie]

What about an option to display partial matches, such as:

• offset has changed > 50 lines (or maybe a lower threshold such as 20)
• target, code or message has changed

I sometimes find that after making changes, basedmypy tells me there are errors in code I haven't touched, and I run with --write-baseline and look at the diff of the baseline to determine whether there are new errors hidden by old ones (false negatives), or just changes to line numbers (false positives).

The false negatives usually occur due to newly introduced errors matching the code of an old error, where the line number offset of the new error is within 50 of the baselined error's offset, and the new error is before the old one in the file.

The false positives occur either because of the baselined errors being incorrectly matched with new ones, or because the offset changes by more than 50 lines.

[Zeckie]

Would be even better if it could show these and any new errors in PyCharm

[KotlinIsland]

From #183 by @Zeckie

Locations of errors stored in baseline file are currently using filename and line number within that file.

Sometimes, making a change such as adding a new function to a file moves the other lines within the file far enough that they are not recognised (If I remember correctly, they can move up to 50 lines and still be recognised).

Would it be possible to also record the class / function(s), and an offset relative to the function def? That way, if a one function is added / modified, the baselined errors in a different function will still reliably be recognised.

[akaihola]

Darker tracks moved code by using git diff between two commits (or a commit and the working tree).

But it currently only goes halfway and filters mypy errors to lines that were actually changed, so related errors elsewhere in the file or in other files are not yet shown. See the discussion for adding this feature in Darker.

[akaihola]

Baseline diffing for running linters landed in Darker 1.7.0, see akaihola/darker#393.

[DetachHead]

now that the src code is stored in the baseline file, i think when a function is moved from one file to another it should match as long as the source code is the same

[KotlinIsland]

Mostly addressed in #415, we now use the source line as a metric, and everything seems based. Not to say this problem is resolved, I don't think it is possible, but we are now in a much better place. I'm resolving this, any follow up should be done in a new discussion/issue.