Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics limited to a single Expected Calibration Error (ECE) formula #277

Open
thibaultcordier opened this issue Mar 2, 2023 · 0 comments
Labels
Backlog This is in the MAPIE team development backlog, yet to be prioritised. Contributors welcome 👋🏻 Especially relevant issue/PR for contributors to work on. Other or internal If no other grey tag is relevant or if issue from the MAPIE team

Comments

@thibaultcordier
Copy link
Collaborator

Describe the bug
MAPIE offers only one way to calculate the ECE, but there are different estimators of the ECE in the literature (reference: https://arxiv.org/abs/2109.03480).

To Reproduce
When using the expected_calibration_error(y_true: ArrayLike, y_scores: ArrayLike) function, the computed estimator of ECE is the class-specific ECE associated to class c (see https://arxiv.org/abs/2109.03480 in page 4).

To use this function correctly, the array y_true must be the boolean array indicating whether the example at index i is associated with the class considered (named c). So for any example at the index i, y_true[i] = 1 if this example is related to the considered class c.

Expected behavior
It will be interesting to implement other ECE estimators, such as the confidence ECE presented in https://arxiv.org/abs/1706.04599.

Additional context
Links for references:

  • Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017, July). On calibration of modern neural networks. In International conference on machine learning (pp. 1321-1330). PMLR. (https://arxiv.org/abs/1706.04599)
  • Posocco, N., & Bonnefoy, A. (2021). Estimating expected calibration errors. In Artificial Neural Networks and Machine Learning–ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part IV 30 (pp. 139-150). Springer International Publishing. (https://arxiv.org/abs/2109.03480)
@thibaultcordier thibaultcordier added the Good first issue Easy issue to start to contribute to MAPIE label Mar 16, 2023
@thibaultcordier thibaultcordier added this to the Backlog H2 milestone Aug 3, 2023
@LacombeLouis LacombeLouis removed this from the Backlog H2 2023 milestone Aug 3, 2023
@jawadhussein462 jawadhussein462 added Discussion in progress Discussion ongoing between the Mapie team and the author. Other or internal If no other grey tag is relevant or if issue from the MAPIE team labels Nov 18, 2024
@Valentin-Laurent Valentin-Laurent added Needs decision The MAPIE team is deciding what to do next. Contributors welcome 👋🏻 Especially relevant issue/PR for contributors to work on. Backlog This is in the MAPIE team development backlog, yet to be prioritised. and removed Good first issue Easy issue to start to contribute to MAPIE Discussion in progress Discussion ongoing between the Mapie team and the author. Needs decision The MAPIE team is deciding what to do next. Documentation labels Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Backlog This is in the MAPIE team development backlog, yet to be prioritised. Contributors welcome 👋🏻 Especially relevant issue/PR for contributors to work on. Other or internal If no other grey tag is relevant or if issue from the MAPIE team
Projects
None yet
Development

No branches or pull requests

4 participants