scanner: compute token line/column lazily on errors

[bluetech]

The scanner functions are hot, and the line/column location tracking is quite expensive. We only use it for errors, which don't need to be fast, because we bail if there are too many; and for warnings, which are usually not shown by default. So only keep the token start pos, and compute the line/column lazily from that. This will also allow some further improvements ahead.

bench/rulescomp

before: compiled 1000 keymaps in 1.669028s
after:  compiled 1000 keymaps in 1.550411s

bench/compose:

before: compiled 1000 compose tables in 2.145217s
after:  compiled 1000 compose tables in 2.016044s

[wismill]

Really nice! With faster position computation, we could also use it more frequently to get better logs.

This reminds me this article about Megaparsec, a quite fast Haskell parser with really good error messages. One big improvement was computing the exact position (line, column) only on demand. But it also caches it, so the computation does not start from scratch each time. I do not think this is the case here, but it could maybe improve the perf further?

[bluetech]

Nice article, talks exactly about this. I like the caching idea, it should be easy and work well for us, will try it a bit later.

[bluetech]

I added the caching.

[wismill]

Really nice! Using bench-compile-keymap --layout de --variant neo --stdev 0.3, I observed a 1.075x speedup without the cache and a 1.097x speedup with the cache. That’s a lot just for token position computation!

I would love to see some automated tests based on the stderr output. That would be test/log.c with a dummy keymap. It can be a follow-up.

[wismill]

Changing the milestone, I am sure this speedup is very much welcome!

[wismill]

I am going to write some tests in an independent MR.

[wismill]

(did not mean to close the PR, sorry for the noise)

See #630 for the tests.