Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify inter-annotator agreement computation #181

Merged
merged 6 commits into from
Jan 20, 2025
Merged

Modify inter-annotator agreement computation #181

merged 6 commits into from
Jan 20, 2025

Conversation

kasnerz
Copy link
Collaborator

@kasnerz kasnerz commented Jan 20, 2025

Before

Until now, factgenie provided a way to compute Pearson r correlation on dataset- and example-level. The results were displayed in a pop-up with some basic comments.

❌ This was not great for multiple reasons.

First, it impovered the users: it gave them less control over the computed results (unless they wanted to look into and/or modify factgenie code).

Second, it turned out to be inappropriate for long-running computations (such as computing the γ score), which would require displaying a separate page with asynchronous updates.

After

We decided to keep it a simple. On the IAA page, we only provide the users with CSV data files on which they can run data analysis themselves 🔎

We also provide a 📕 Jupyter notebook using the above files. The notebook goes through all the nooks and crannies of IAA computation with detailed comments.

This pull request resolves #99.

@kasnerz kasnerz merged commit 742f421 into main Jan 20, 2025
3 checks passed
@kasnerz kasnerz deleted the add-gamma-iaa branch January 20, 2025 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add token-level inter-annotator agreement
1 participant