Statistical stopping criteria

[patricoferris]

Currently, we have 100 pairs of matching pixels which we take forward to calculate additionality and leakage. This is an arbitrary number, with the idea being it is sufficiently large enough to allow the matches to "converge". Instead, we should use some form of statistical stopping criterion to decide when to stop.

The image shows the additionality for the 100 matching counterfactuals, quite a lot of variation. Was 100 enough? This is why we need a stopping criteria.

Unfortunately there might be quite a lot of code churn with this change as we made a simplifying early assumption to have one script that would generate the whole lot in one go, but now we don't know ahead of time how many pairs we want. A lot of the code will remain unchanged but how we call it will be different. Presumably we'll also want some upper bound that kicks in saying that things aren't stabilising.

[robinmessage]

Tom has written up start of this (not confirmed yet I don't think) here: https://www.notion.so/forests/Decorrelated-counterfactuals-placebos-and-stopping-criterion-74e042e448a7491383afb20a14b83fdb