Skip to content

Latest commit

 

History

History
53 lines (43 loc) · 3.81 KB

README.md

File metadata and controls

53 lines (43 loc) · 3.81 KB

Hack RussianSuperGlue

Welcome to this repository!

Here you can find the results of our study where we attempted to solve natural language understanding tasks with the help of rule-based heuristics.

What you can find in this repository:

  • Python notebooks with detailed descriptions of each RussianSuperGlue dataset with various graphics and schemes
  • Baseline for how to solve each dataset if we:
    • Picked a majority class
    • Picked random option
    • Picked random option, were classes balanced
  • Suspicious cases in annotation, where you either cannot give a straight answer to or the answer is wrong
  • Heuristics specified for every dataset separately to solve it and achieve plausible or even comparable performance with models listed in the leaderboard

What you should not expect to find in this repository:

  • Data from RussianSuperGlue, since it is loaded to the workspace in the first cell of each notebook.

What to do with all of these:

In order to gain access to our findings (and maybe contribute to our study) we recommend following this algorithm:

Below you can find links to all the notebooks in Colab

  1. Choose the task you liked below:
  2. Explore away
  3. If you find anything interesting, feel free to create new code/text cell at the beginning of the notebook and tell us about it. If you want to, you can change the code in our cells as well, yet if you want to show it to us, please do so and create the text cell at the beginning of the file describing your updates. It will help us find your contribution faster!

BEWARE: next step may not work

  1. When you are finished, click File in the Colab menu panel. Choose Save a copy in Github, add minor changes to the filename, so we would be able to discern your notebooks from ours and select the repository to submit your updates.

https://github.com/tatiana-iazykova/2020_HACK_RUSSIANSUPERGLUE

  1. If step 4 doesn’t work, share the link to the notebook via email: 
@misc{iazykova2021unreasonable,
      title={Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks}, 
      author={Tatyana Iazykova and Denis Kapelyushnik and Olga Bystrova and Andrey Kutuzov},
      year={2021},
      eprint={2105.01192},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}