Modify inter-annotator agreement computation #181
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before
Until now, factgenie provided a way to compute Pearson r correlation on dataset- and example-level. The results were displayed in a pop-up with some basic comments.
❌ This was not great for multiple reasons.
First, it impovered the users: it gave them less control over the computed results (unless they wanted to look into and/or modify factgenie code).
Second, it turned out to be inappropriate for long-running computations (such as computing the γ score), which would require displaying a separate page with asynchronous updates.
After
We decided to keep it a simple. On the IAA page, we only provide the users with CSV data files on which they can run data analysis themselves 🔎
We also provide a 📕 Jupyter notebook using the above files. The notebook goes through all the nooks and crannies of IAA computation with detailed comments.
This pull request resolves #99.