Support attaching testbed related metadata to a report

[jschwe]

It would be nice to have the option to attach additional metadata to a report that relate to the testbed.

In particular, I'm thinking about the OS Version and the kernel version, but I guess there might also be other options that are expected to change over time as the testbed is updated (e.g. monthly security updates on an android phone, updated Linux kernel ...), and which might influence the performance, but usually do not.

So when seeing a regression in a metric it would be nice if we could compare the bencher reports and directly see if there was an environment change on the testbed.

[epompeii]

Hey @jschwe thank you for using Bencher!

I definitely agree that this is something that Bencher should support/make much easier.
There are currently two different approaches I'm considering for this:

1. Custom key/values for Reports: Support arbitrary key/value pairs as report metadata #127
2. Custom labels for all dimensions: Create custom tags for all dimensions #240

Currently, I'm thinking Option 2 would not be tracked over time. For example, if you updated a label for a dimension like adding an os:linux label to a Testbed then that would show up on the Testbed as a whole not just a specific commit/version. This means you wouldn't be able to track and compare the changes to labels over time.

So it seems like Option 1 would probably be more inline with your use case then?

Maybe having an option for bencher run like --metadata "kernel:6.1". Then when you were looking at Metrics, it would show you all of the metadata for that Report. Would that be sufficient? Or would that metadata need to be tied specifically to the Testbed for some reason?

[jschwe]

Maybe having an option for bencher run like --metadata "kernel:6.1". Then when you were looking at Metrics, it would show you all of the metadata for that Report. Would that be sufficient? Or would that metadata need to be tied specifically to the Testbed for some reason?

That sounds perfectly fine. I don't think there would be a reason to tie the metadata to the testbed.
I guess I'll be closing this issue then, and subscribing to 127.