Metrics limited to a single Expected Calibration Error (ECE) formula #277
Labels
Backlog
This is in the MAPIE team development backlog, yet to be prioritised.
Contributors welcome 👋🏻
Especially relevant issue/PR for contributors to work on.
Other or internal
If no other grey tag is relevant or if issue from the MAPIE team
Describe the bug
MAPIE offers only one way to calculate the ECE, but there are different estimators of the ECE in the literature (reference: https://arxiv.org/abs/2109.03480).
To Reproduce
When using the
expected_calibration_error(y_true: ArrayLike, y_scores: ArrayLike)
function, the computed estimator of ECE is the class-specific ECE associated to class c (see https://arxiv.org/abs/2109.03480 in page 4).To use this function correctly, the array
y_true
must be the boolean array indicating whether the example at indexi
is associated with the class considered (namedc
). So for any example at the indexi
,y_true[i] = 1
if this example is related to the considered classc
.Expected behavior
It will be interesting to implement other ECE estimators, such as the confidence ECE presented in https://arxiv.org/abs/1706.04599.
Additional context
Links for references:
The text was updated successfully, but these errors were encountered: