Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calculate goodness of fit (r2_score) based on posterior expectation rather than posterior prediction #429

Merged
merged 3 commits into from
Nov 21, 2024

Conversation

drbenvincent
Copy link
Collaborator

@drbenvincent drbenvincent commented Nov 19, 2024

  • Previously, the r2_score was calculated by comparing y_true and y_pred, where y_pred was the likelihood-influenced predictions i.e. with the observation noise added. Now we set y_pred to be based on the model's expected outcomes.
  • Closes Fix failing doctest #428

📚 Documentation preview 📚: https://causalpy--429.org.readthedocs.build/en/429/

@drbenvincent drbenvincent added the bug Something isn't working label Nov 19, 2024
@drbenvincent
Copy link
Collaborator Author

Oh, I think this is because of rng being used multiple times (in the docstring there was a call to rng.normal before the model.score) which changed its state. Simple fix will be to rearrange order of calculations in the docstring. Though I'm not totally on top of why doctests were passing locally and not remotely.

@drbenvincent
Copy link
Collaborator Author

Why are these f*king doctests passing locally but not remotely. Does rng have different behaviour on different systems or numpy versions?

@drbenvincent
Copy link
Collaborator Author

These doctests have been a constant source of frustration. So far I've kept them around in the hope that they reveal errors or misunderstandings. But if it's nonsense like different behaviours of rng on different OS's or numpy versions, then we'll at the least get rid of output checking in the doctests.

Copy link

codecov bot commented Nov 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.40%. Comparing base (142694f) to head (cdf2ee5).
Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #429   +/-   ##
=======================================
  Coverage   94.40%   94.40%           
=======================================
  Files          31       31           
  Lines        1985     1985           
=======================================
  Hits         1874     1874           
  Misses        111      111           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

@drbenvincent drbenvincent merged commit e61bfd0 into main Nov 21, 2024
8 checks passed
@drbenvincent drbenvincent deleted the fix-failing-doctest branch November 21, 2024 21:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix failing doctest
3 participants