quantify consistency improvement

[JohnShiuMK]

close #76

[tonyshumlh]

@JohnShiuMK I add my 2 cents here.
Although the result shall be the same after the adjustment, below are some suggested adjustments

1. In the example, I notice that var_curr is indeed smaller than var_prev, meaning the F = var_prev / var_curr, and thus p_value should be = 1 - scipy.stats.f.cdf(F, df_prev, df_curr). Therefore, you might want to make the code flexible.

F = var_curr / var_prev if var_prev < var_curr else var_prev / var_curr
...
p_value = 1 - scipy.stats.f.cdf(F, df_curr, df_prev)

2. I recommend doing two-tailed test rather than one-tailed test. In the example, H_0 is var_prev <= var_curr and H_A is var_prev > var_curr, and the p_value is 0.33 which failed to reject H_A.
   However, our point of interest should be whether var_curr < var_prev. Therefore, I recommend H_0: var_prev = var_curr and H_A: var_prev != var_curr. If sample var_curr < sample var_prev and the result is significant, that implies population var_curr < population var_prev

[JohnShiuMK]

@JohnShiuMK I add my 2 cents here. Although the result shall be the same after the adjustment, below are some suggested adjustments

1. In the example, I notice that var_curr is indeed smaller than var_prev, meaning the F = var_prev / var_curr, and thus p_value should be = 1 - scipy.stats.f.cdf(F, df_prev, df_curr). Therefore, you might want to make the code flexible.

F = var_curr / var_prev if var_prev < var_curr else var_prev / var_curr
...
p_value = 1 - scipy.stats.f.cdf(F, df_curr, df_prev)

2. I recommend doing two-tailed test rather than one-tailed test. In the example, H_0 is var_prev <= var_curr and H_A is var_prev > var_curr, and the p_value is 0.33 which failed to reject H_A.
   However, our point of interest should be whether var_curr < var_prev. Therefore, I recommend H_0: var_prev = var_curr and H_A: var_prev != var_curr. If sample var_curr < sample var_prev and the result is significant, that implies population var_curr < population var_prev

wait, i thought the F is already defined in the way you specified?

[JohnShiuMK]

According to #72 and #99
I will update the consistency F-test from one-tail to two-tail (keep it in jupyter notebook)

• (optional) to look at stat package if have time

[JohnShiuMK]

@tonyshumlh I have updated the demo to 2-tail test, ready to review and merge, thanks

[JohnShiuMK]

@tonyshumlh I have updated the demo to 2-tail test, ready to review and merge, thanks

hold on, the file / code structure are actually ugly. Let me tidy it up a little bit first, sorry.

[JohnShiuMK]

Hi @SoloSynth1 @tonyshumlh

In this demo of F-score comparison, I'm comparing the code of week 3 (before refactoring, i.e. old code base) vs. week 4 (after refactoring). Therefore, I have to keep the old code base (archive/analyze.py) and adjust the ConsistencyEvaluator (archive/llm_eval/consistency_eval.py) so that it also works for the old code.

We may delete them in the future when we are having a comparison between newer versions. But, for now, I guess it's better to keep a record of the above comparison, in case someone asks for it.

In order not to disturb the latest code base, I put those related to the demo and the old code base under archive/.

What do you think? Do you have any better ways to proceed whenever we encounter a situation like this?

[JohnShiuMK]

I have added a note here, to avoid confusion of the archive/

I think we can merge for now.
And if we need to clean up later, we could create another PR, what do you think?

[tonyshumlh]

The 2-tailed f-test p-value calculation looks good to me