RSG_RUSSE.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "RSG_RUSSE.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "RQCtN9CknDMH",
        "yUz8ovXonGUV",
        "9Wb3fB0LorSc",
        "hROuboI4O47_",
        "IDr9yTK6YHl-",
        "Lr75O558GsoL"
      ],
      "toc_visible": true,
      "authorship_tag": "ABX9TyNtscO6moIZ/N6b9DXvMgq0",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tatiana-iazykova/2020_HACK_RUSSIANSUPERGLUE/blob/main/RSG_RUSSE.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xGZSqd8Rb6_D"
      },
      "source": [
        "**Результаты so far:**\r\n",
        "\r\n",
        "*   Баланс классов в train/valid почти не отличается (разница меньше 1%)\r\n",
        "*   SOTA Accuracy на лидерборде = 0.99 (mBART)\r\n",
        "*   Воспроизведен бейзлай RSG (Accuracy 0.66 на валидации)\r\n",
        "*   Тюнинг параметров Tf-idf (Accuracy 0.64 на валидации) \r\n",
        "*   Эвристика same_form (Accuracy 0.56 на валидации)\r\n",
        "*   Вопросы по разметке вынесены в конец ноутбука\r\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RQCtN9CknDMH"
      },
      "source": [
        "# Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FpR1j_wefIYD"
      },
      "source": [
        "%%capture\n",
        "%%bash\n",
        "# change url if you want to work with a different RSG dataset\n",
        "wget -q --show-progress \"https://russiansuperglue.com/tasks/download/RUSSE\" -O temp.zip\n",
        "unzip temp.zip -d data\n",
        "\n",
        "# remove unnecessary directories and files\n",
        "rm temp.zip\n",
        "rm -r data/__MACOSX\n",
        "rm -r sample_data/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6B35hR1UfhM-"
      },
      "source": [
        "import pandas as pd\n",
        "pd.set_option('display.max_colwidth', 1000)\n",
        "pd.set_option('display.max_rows', 1000)\n",
        "\n",
        "class JSONL_handler():\n",
        "    \"\"\" opens a jsonl file and turns it into a necessary data structure \"\"\"\n",
        "    \n",
        "    def __init__(self, path):\n",
        "        self.path = path # path to jsonl file\n",
        "\n",
        "    def to_pandas(self):\n",
        "        \"\"\" get jsonl file content as a pandas DataFrame\"\"\"\n",
        "        return pd.read_json(path_or_buf=self.path, lines=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IvQYkbBoFb8y"
      },
      "source": [
        "train = JSONL_handler('data/RUSSE/train.jsonl')\r\n",
        "train = train.to_pandas()\r\n",
        "\r\n",
        "valid = JSONL_handler('data/RUSSE/val.jsonl')\r\n",
        "valid = valid.to_pandas()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 245
        },
        "id": "sHG40xUs8-e_",
        "outputId": "b948422e-201d-4c61-bab8-3a12f3b4d1e1"
      },
      "source": [
        "train.head(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>idx</th>\n",
              "      <th>word</th>\n",
              "      <th>sentence1</th>\n",
              "      <th>sentence2</th>\n",
              "      <th>start1</th>\n",
              "      <th>end1</th>\n",
              "      <th>start2</th>\n",
              "      <th>end2</th>\n",
              "      <th>label</th>\n",
              "      <th>gold_sense1</th>\n",
              "      <th>gold_sense2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>двор</td>\n",
              "      <td>В нашей деревне осталось от силы двадцать дворов</td>\n",
              "      <td>Солнце стояло уже высоко, когда справа от дороги я увидел деревеньку дворов в пятнадцать</td>\n",
              "      <td>42</td>\n",
              "      <td>49</td>\n",
              "      <td>69</td>\n",
              "      <td>76</td>\n",
              "      <td>True</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>доклад</td>\n",
              "      <td>Табличка на дверях: «Без доклада не входить»</td>\n",
              "      <td>Вчера отбыл в столицу первый секретарь обкома […], вызванный то ли «на ковёр», то ли на сезонный доклад об урожае</td>\n",
              "      <td>25</td>\n",
              "      <td>33</td>\n",
              "      <td>97</td>\n",
              "      <td>104</td>\n",
              "      <td>False</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>засада</td>\n",
              "      <td>У нас вообще […] засада с героями, способными дотягивать в жизни до собственного творчества</td>\n",
              "      <td>Там в воскресенье все магазины закрыты – вот ведь засада!</td>\n",
              "      <td>17</td>\n",
              "      <td>24</td>\n",
              "      <td>50</td>\n",
              "      <td>57</td>\n",
              "      <td>True</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   idx    word  ... gold_sense1 gold_sense2\n",
              "0    0    двор  ...           1           1\n",
              "1    1  доклад  ...           4           2\n",
              "2    2  засада  ...           2           2\n",
              "\n",
              "[3 rows x 11 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yUz8ovXonGUV"
      },
      "source": [
        "# Data Exploration"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dk3eArpSFGvK"
      },
      "source": [
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 232
        },
        "id": "7GfXs-cBFFlk",
        "outputId": "4677b69c-e925-4d4a-eb32-d6ba9cd263ce"
      },
      "source": [
        "# Train/Valid Label Distribution \r\n",
        "\r\n",
        "plt.subplot(1,2,1)\r\n",
        "plt.title('Label distribution\\n in Train dataset\\n', fontsize=14)\r\n",
        "plt.pie(train['label'].value_counts(), \r\n",
        "        autopct=\"%.1f%%\", \r\n",
        "        labels=train['label'].value_counts().keys(), pctdistance=0.5, \r\n",
        "        textprops = dict(fontsize = 12))\r\n",
        "\r\n",
        "plt.subplot(1,2,2)\r\n",
        "plt.title('Label distribution\\n in Validation dataset\\n', fontsize=14)\r\n",
        "plt.pie(valid['label'].value_counts(), \r\n",
        "        autopct=\"%.1f%%\", \r\n",
        "        labels=valid['label'].value_counts().keys(), pctdistance=0.5, \r\n",
        "        textprops = dict(fontsize = 12))\r\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADXCAYAAAC51IK9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2dd5hU1fnHP+/M7ixLW1CaWBgQFBuxoLFFsSRGxxYTjVFji0ZNjGJljDGO/iwTu9FgNBbsLbGPBVRAVNSIBEWxgUOTDjuwy/Y5vz/OXRyG2T73ninn8zz32Z17zz3ne2feeefU94hSCovFYrF4g8+0AIvFYikmrNO1WCwWD7FO12KxWDzEOl2LxWLxEOt0LRaLxUOs07VYLBYPyWmnKyKni0hVFvKZIiJ3d/CeX4mISnmdFS0tlLWRPhGJi8ilLpXl2nNY2o+1bVfKygvbdtXpisgEEXnFzTI85GlgWHsSikhQRJSIjG5n3scBV3RaWcs6lIj8Ku10u5/D0jLWtq1td5YS0wLyBaVUDVCTzTxFJKCUqldKrc5mvq3hxnNY8htr295itHtBRC4WkU9FpFpEFovI/SLSJ0O6o0TkaxGpFZHJIjIsw/UZzvXvROR6EQl0UMupIjJfRNY7NZiBadc3arqIyNYi8qKIrHbu+VJETnQuf+f8/a/zizzFuWeCiLwiIuNEZBGwyDmfqYnYU0QeE5EqEVma3iTL9Euf2nQTkbhz+lknbTzTczjnzhGRb0Wk3vl7doayfi8izzqf1TwROaXNN7WIsbZtbbslTPfpJoGxwE7AScBewF1pacqAq4EzgH0AP/CciAiAiBwGPA7c7eRzJvAr4Ib2ihCRHwMTgPuAXYGXgWvbuG080B04yCl3LFDpXNvL+ftzYAt0E6uZA4FRzrVDWsn/YmAOsDv6+W8QkeNaSZ/Ons7fsx0Ne2ZKJCK/QL93dwA7A3cC40XkqLSkfwVeBH6EbsY9KCLbdEBPsWFtu2WK27aVUq4d6A/7lQ6k/zlQB/ic16cDCtgvJc0QoAk41Hn9DnBVWj7HAlWAOK+nAHe3Uu4TwKS0c/frt2fD69OBqpTXnwJXt5Bf0NE9OsP7sQIoSzu/kT4g3oKed1NeK+BXaWniwKVtpEl/jveABzPoTC/rxpTXJcB64BQ37SeXD2vb1rY7e5juXjhYRCaJyCIRWQc8BwSAQSnJksBHzS+UUvOB74EdnVN7AFc6TZUqp3nxBNAjLZ/W2AGYnnYu/XU6dwJ/EZHpInKdiOzRzrJmK6Xq2pEuk54dMyXsIjugjTOVdzOU9WnzP0qpRvQXbIALegoCa9utUtS2bczpisgQIIZuZhyPNrAzncvpfVathULzAdegm07NxyhgBPrNcwWl1APAUOAhYDvgfRGJtOPW6mxJACTtXGmW8m7OP5WGDNdNd0/lJNa2uy6BArZtk1+a0WgDvEgpNV0p9TUwOEM6Hz/0I+H0tQxGGzTAJ8BIpdS3GY7GdmqZA+yddi799SYopRYppe5TSp2A7hf6vXOp3vnrb2f5mcikZ07K6xXo/iwARGRg6muHhnZomAPsl3Zuf+CLdiu1pGNtu3WK2ra9mDLWW0R2TTtXCXyDNrqxIvIc+o0fm+H+RuAOEbkQPR3kduBz4E3n+rXAKyIyH3jGSb8zsJdS6vJ2avw7+tf8CuDfwBjgF63dICJ3Aq8BXwO90X12zR/mckfrYc7Iaq1SKtFOLc3snabnVODklOtvA38UkffR/YA3ALVpecSBQ0RkKlCnlFqToZyb0aPAM4CJznOczMYDJJbMWNu2tt1x3OosTum0VhmOfzvXLwAWoz/Et4ATnOvB1I5x4Bi0IdcBU4HhaeX8DJiG7gBfC3wMnN9SZ34LWs8AFjhaXgPOp/XBhrscTbXoX+angC1Trp/l5NcETEl5PzYZfEnXhzaoCPCk8/zLgHFp9wx2dFYBc4Ffsulgw1GOxgYgnuk5nHPnAt866b4Fzk673ubARrEd1ratbXf2aB4BtVgsFosH2IEQi8Vi8RDrdC0Wi8VDrNO1WCwWD7FO12KxWDzEuNOVjoeK84wWgnVkI99LU4J2WHKIXLJHSQsfmf66hXteEZEJ2S7bS0RkdjsXY+Qlxp0usBA98fl/nbk55UvS2hHppDZXYoF2hkyRlzwq17Wg0zlKl+wRQHR0sQdauHa481lu14msLwSyGgFLRMY4evq5XZZbZIou5lG5Lb13rWI8nq5SqglY2oUsmr8kzZyHXnKZGnkoPdxbqVIqfelfJm2exQK15AZZsEeAB4DrROQCpVT60tjfAdOUXqXWUW0dXYTQabwsq9gwXtNNb86l/HocIiIfio7n+bGI7J7pfqVUk1JqafMBrAOaUl6PBNaJyBEi8pGI1KNX02wrOmboUtFxND8RkSPTtGXaauQvInKviKx1gplc1o5nvNwpp0pEHgF6pl3fU0QmishKJ993RWSf1HKdf9Pjh7bnGY5zal41ouOjTnWWVTZfbzFeq+hYqUOAm5tbDW09a77TVXt0eBQdK+CEtLz7A0cD94vI5iLypGNDNSLyuYic0Ya29O6G7s65KhFZJiJ/znDPKSLyXxFZJyLLRceN3bL5WYHJTtIVznNOaKGsMhG5wymnVkQ+EJH9U6535n1CRAY4NlwjOubvmRnStBibWETGoGNE9JC0lm1rz+5cLxWRv4vI9yJSJyILRSSacj0gIn9zPqP1Tl6HtfXetUkOrOwJkhIqDr0sUKGjLx2EdppvoNdRSzvyuxRnhUpafp+hV/cMA/qjY2eeC+wCDAeuRK8rH9nGappV6BU9w4E/OXnv04qeE5x8z0EHD7kSvbIoVePBwG/RUZFGomOArgE2d673d8o5Cx1dqr9zvtVncNLWA5c47/POTh4DneuHOVrOALZ13u+vgFuc65uhWxLXOHkNMm0v+WKP6FVc09LOXYxeJtwd2BK4DB3EZhg6tkE9cEhK+gmkrPLK8Ho8etXbYc5n+6zzeU5ISXMmcIRTxl5oR/GOc82P7kJT6Mhbg4CKFsq6E1gChBw7/Re6BblFF9+nV9FLn/cDdkN/56qASEqasejvSBAds/dT4FHnWgDdFVLdbKNAz7ae3bl+iWPfBwDbAPsCZ6Rcfxz4wLk+DP29r0d/71p879q0sRw28sNS0uznnNuqHfm15HR/2Y57PwD+kvJ6Cps63SfT7vkm9Z4Meb4P/Cvt3JupGjPcI46Bn5JybpPlim09AzpItAKGtJC2PfFa4xTRct9s2SNwqJNmu5Rzs4F7WrnnKeD+lNcTaMHpoltLdcDJKdd7op36hFbKGJmqPeX5+qWlSy2rB9rZnJpy3Y9eontdZ98ndCVEkTmmcKSVZ8gUm7iqpfStPPvf0Uu0N/lRQFdCksA2aedfAMa39t61dRjvXmiFT1P+/97525UYlx+nvhCRHiJyk4h8ISJrRHfEj0b/4rVXV7O21nS1Gc/UaWLdK3rblgS6i2RAW1ra8Qyz0A5+toj8R0TOc5q4zWQjXmux0FF7fAu9tc2ZsGEHh53Q/b2IiF9ErnSazauc9/442ra/ZrZF1/I22JJSqgrdotuAiOzuNN/ni47r2/w96MjOCNuiu0s2xKZVuu87UxzcjrxPO9ByTOHUZ2hPbOJNaMezT0C3NL4WkX+ISEhEmn3i7ujKzxdp348Q+v3oNMYH0lohdaCruS+xKz8S6QMat6B/MS9F11bXA4+wabzT1nQ1a+vqj9fD6H2rLkLXLOvQX9q2tLT6DEqpJhH5GTrK1c/Qgzg3isiBSqlZ/BCv9dkMebsWrzVP6ZA9KqWUiDwEnCciV6Lf+1lKqeYv/qXo5u2FaEdZhY6mlbXg2SLSA93EfxPdfbUc6IcOoNOhfdZaIb2fvzPf2xbHCuSH2MT/QoeYXIV2iE/SyjO059mVUp84fbOHobcXehiYJSI/dTQr9IB8+ne+S5tf5rLTdZv9gUeUUv8BEJFu6F+wDo8qt0FzPNMHU86lxxPdH7hAKRVztLQ3fmibz6B0O2g6MF1ErkX3n/0aXQveEK+1Ff31Gcq1tI+H0BG1jgdOBFIHuvYHXlZKPQogIoJublfSPuaibWJvYJ6TRw903+5cJ81ItKP5s1LqOydNeljD9sTHneuk2685bxHxo/d1e6KdejPxJT/EFH7fybc5pnAzqbGJm5w0R6blk8lG2/PsKKXWoUNM/tsZCPsAPT4yE13THaSUmpx+X0q5ZCi7VXK5e8FtvgZ+4TRBdgEeA7q5UM6dwGkicraIjBAdR/THGbScIiI7isie6L69+rQ0cXT80EEi0rc9zyAie4uebbGnY8xHA1vzQ2zUa4GTRORaEdlZREaKyK9E5Ka0cn8iIltKB+cjFjtKqUXo2tZ4dPP88ZTLX6M/z/1FpHnwdGgH8q5Cd1X8TUR+KiI7oX/YUx3AAnSr6XwRGSYiIeD/0rKaj67RhUSkv4j0TLuO0tPe7nHKOkJEdnBeD3SerVMopb4CXgfuFZF9RMcmnsDGNcnU2MRDReQ3bBqbOA50c96HfiLSvT3PLnpWxG9EZAcRGY7eQHQtsEjpKX2PAxOc78QwERktemFTs/Nu873LRDE73YvRTY5p6NidHzj/ZxWl1NPo2s716F/PXYDb0pKdiR4EmYF2uA+iDSmVS9CjwgudfKDtZ0igayevoI33VuD/lFKPOdreQPdRHYTuV/sICKMNtpm/oh31XGyXQ2e4H+gLPKc2DrR9Hfr9fg09oFnNxk65PVyKHpF/3vk728kLAKXUCuA09ODoF+iddy9OzUAptdg5fz06tm1LKzDHoXfKfQi9cGQU8HOl1JIOak7ndHTf99vonYqfIMX2lVKfortgLnae4Sz0c6c+w/vAP9FdDiuAy9vz7Oixk8vQn8Mn6P7dw5VS653rZzjPexO6Vv4KeibDfKfc9r53G2Hj6VosFouHFHNN12KxWDzHOl2LxWLxEOt0LRaLxUOs07VYLBYPsU7XYrFYPMQ6XYvFYvEQ63QtFovFQ6zTtVgsFg+xTtdisVg8xDpdi8Vi8RDrdC0Wi8VDrNO1WCwWD7FO12KxWDzEOl2LxWLxEOt0LRaLxUOs07VYLBYPsU7XYrFYPMQ6XYvFYvEQ63QtFovFQ6zTtVgsFg+xTtdisVg8xDpdi8Vi8RDrdC0Wi8VDrNO1WCwWD7FO12KxWDzEOl2LxWLxEOt0LRaLxUOs07VYLBYPsU7XYrFYPMQ6XYvFYvGQnHW6IjJFRM4yrcNicQNr38WLZ05XROIiUiMiVSnHYK/Kt1jcxNq3pb2UeFzeUUqpNz0u02LxCmvfljYx1r0gIn1F5BURWSEia5z/t2oh7XARmSoiCRFZKSJPp1wbKSKTRGS1iHwlIid49xQWS2asfVtawmSfrg94CBgCbAPUAHe3kPb/gIlAX2Ar4C4AEekBTAKeAAYAJwLjRWRHV5VbLG1j7duSEa+7F14QkUbn/ylKqWObL4jI9cDkFu5rQBvvYKXUIuBd5/yRQFwp9ZDzeqaI/Ac4Hrgm6+ozEAzHAkCFc/QE6oF1QBWwLh4NNbZyu6WwKBj7DoZjApQD3Z2jB/qHJAFUxqOhKjfLL2S8drrHNvd5iUh3EbkX+Dn6Fx6gl4j4lVJNafddjq4NfCQia4BblVIPog31xyJSmZK2BHg0W4KD4dgQ4EcpRxDtYHs7f8tauV0Fw7E1wDJgOTAfmAl8AsyMR0PrsqXTkhPkjX0Hw7FyYCSwE7Cjc4wENkc72G6AtHJ/I7ACWOwci4DPgI+Az2xlo2W8drqpXAJsD/xYKbVURHZFO6RNPmil1FLgbAAR2R94U0TeARYCU5VSP82GoGA4tjlwBDAa2BUYBfTpQpYCbOYcOzjnTnX+qmA49g3aAX8CfAy8F4+G6rtQniV3yCn7DoZjw9A150OAndGVh650L5YAWzjH6LRrNcFwbCbaAX8EfBiPhuZ1oayCwqTT7YXu56oUkc2Aq1tKKCLHA9OdptcaQAFJ4BUgKiK/BZ5yku8KVCml5rRHRDAc2xo4FvgFcADg79zjdBgBtnOOE51zlcFw7GXgWWBiPBqq80iLJfsYte9gOFYC7Id2tCF++NH3gnJgX+do1hMHngGejEdD//NQS85h0unegR4gWAl8D9yKdn6Z2BO4Q0Qq0E31C5VS8wBE5GfAbc7hA2YBF7dWcDAc2x74JdrRpv9Km6QP8FvnWJvigN+IR0O1RpVZOorn9h0Mx3zA4cAp6G6NrrTSsk0Q3Y1yeTAc+xJ4Gu2AvzKqygCilDKtwRMcgzwaGAscaFhOR1kHPAzcHI+GFpgWY8ktguFYf+B3wDlo55ZP/A94AHggHg3VmBbjBQXvdIPhWDfgLHTtYKhhOV2lAXgciBZjDcGyMcFwbDhwKXAaeuArn1kG3ALcE4+Gqk2LcZOCdbrBcKwHcB56QGOQYTnZJgk8D9wQj4Y+MS3G4i3BcGw74HrgOHI4fkonWYnuSrm7UGf3FKTTDYZjp6D70AaY1uIBrwOXxaOh2aaFWNwlGI71BK5Cd5EFDMtxmzXAncBtheZ8C8rpBsOxEcA96GkxxUQjcDtwTaE3zYqVYDj2G+BmYEvTWjxmEXBBPBp63rSQbFEQTtdZFTYO+DP537fVFRYC58WjoZhpIZbsEAzHdkEvC863wd9s8yJwfjwaWmRaSFfJe6cbDMcOAO5Fr6axaB4Gxsajoco2U1pykmA45geuQw+UmZzamUtUomu9WVtxaoK8dbrOFLAo2ihbXK5YxCwGTo1HQ2+bFmLpGMFwbCB6Hmux125b4nngnHg0tMK0kM6Ql043GI71Qa/QOcy0lhynEV3j/YdpIZb2EQzH9kev3NrCtJYcZwFwZDwa+sy0kI6Sd043GI7tgO7fGWFaSx4xHrjQBiHJbYLh2EXATdjuhPayDjgxHg29alpIR8grpxsMx45ELw7obVpLHvImcEI8GlpjWohlY5ypYA+iQzZaOkYTcFE8GrrLtJD2kjdONxiO/Rkd/q7QJoN7yTfoJtnXpoVYNMFwrAIdwHwv01rynH+gW3PpYTNzjrxwusFw7Bb0yjJL16kEQvFo6H3TQoodZ2xiIjrgjaXrvA4cn+sB1nPe6QbDsWuAv5rWUWBUAmPi0dAs00KKFcfhTiK3otwVAm+hKxU5GxY1p5vqwXDscqzDdYM+wBvOCj6LxwTDsb7oPnbrcLPPIcDjzpTSnCRnhQXDsT8CfzOto4AZCEwKhmMZd6i1uEOKw93DtJYC5pfAP02LaImcdLrBcOwMnB1RLa4yBO14+5kWUgw4YUZfB3Y3raUIODsYjt1gWkQmcs7pBsOxY4H7savMvGIk8HowHLPT8NznAewsBS+5wpn7nFPklNN14oQ+TI7pKgL2ACaYFlHIBMOxccBJpnUUIbcGw7FfmhaRSs7MXnCCjn+I3hLaYoZz49HQvaZFFBrBcOxQ4A1sZcIUa4BRuRKhLJeM4B6swzXN7cFwbEfTIgqJYDi2BXoVZS5914qNvsCjuTKjISdEBMOxk9A74FrMUg48GQzHykwLKQSc8IxPUBw7mOQ6Y9C7ERvHuNMNhmND0AFZLLnBKPQOBZaucwn6y27JDa4NhmPG50Ybd7rogbMK0yIsG/EnJ7iQpZM485/twp7cohS9cKKHSRFGQ8gFw7ETsYGac5UHguHYiHg0tNa0kDzlNsC1L3f1F1OpfO9JmtatwN+jL5sfMRZfeW9WxW6jcc0SAAKDhtP30HMI9NsmYx5rZ7xM9ey3qF8Rp8cOB9Iv9MPsqsa1K1jxQpTGNYvpscuhbHbwWRuuLXvmavr85BTKtsjLBY3boVtyfzAlwNjsBWei+JfoCfpZJxtG2ZhYxqqJ46lf/CWUlNJj+/3oe8jvEZ+fZF01K16IUrfka8q3HU2/0MWIzw/AqtfvonzoHnTffl83Hs1Lbo1HQ5eaFpFvOLMVJrmVf813M1n1+t/pf/Q4AoO3o6lqNQC+0m4k66rx9x4AKsm6T2JUfTqRwWfenTGf9V+9DyLUfPcJqrF+I6e7auJ4Av2D9NjxQJZMuJB+R4+jbIsRVM95h9qFs9n8Z8Z8VjZIArubij1isnthLC453JrvZrJm6gT6HTGWrS96loEnRSnpM4iSnpvR/9gr2OrCp9jqgicoH/5jVr50U4v5rJo4Hn/3Pmx1/qMMPv0uahfMZt0nes/Hdf97jcDAbdn6/MdoSixn/dfTAahbPIemqtWF4HABLrDxGTpGMBwrxeXVlIl3H6di399QtuVIRHyU9OpHSa9++Lr1pKRiICJ6XZH4fBsqGJnovv2+dN9uH3zlm66LaaxcRrdtRuEr60Fg0AgaK5eSrFtP4oN/0/eAU117No/woVsixgr3nGA4NgC4wq38s2WUjZXL6DFyf6QkgL9nX8qH7UHDygUbrnXbZhekpJSyrXaiMbEUlWxi9Vv30/fQc9x6NK8pBW4xLSLPuAgXN0lVySbqln5LsibB4nvPZtE/TmP1pHtINvwQVGvBHb9mwS2/YPWke+m9T+fiogf6D6EmPpNkbRX1y76ltN82VE57lN6jj8HXrWe2HsckBwfDsaNNFGyqpnstLu3+kE2j7L3nMVTPeYdkQy2N61ZSM+9jyofpZfOB/kOonT+LZEMddYs+p7TfENbNeJnyYXtQ2meQG49miqOD4diPTYvIB5w5uVe5WUZTdSUkG1n/1XsMPPlvbHHG36lfNo/E9Kc3pNlm7NNsPfYZNvvpuQQGbtupcnrvfTx1i75g6RNheu0WgmQj9SvilA/fixUv3czSx8exdsbL2XosU1xvYu6u5wUGw7GdgLPaTNhJsmmU3bbamYaVC1h4+wksHn86gUEjKB+xDwA9R/2MZF01Sx+9hLKtdiLQfyjVn0+m957HsOqNu1n6+DjWvJPXO0Wncr1pAXnCWMDVaqCU6inUvXY/ipKem+HvXkGvPY+lZu7HG6XzBbrRc7fDWfXKbfo70UH85b3of8w4Bp95N71GH83qSfey2aHnsPaDZwn0H8LAE6+j6n+v0bByYVaeyxA7A7/2ulATNd1rAL9bmWfLKJVKsuzZv9J9u33Z5uL/sNUFT5CsraJyykO6nJIAm//8Tww+8276jjmdNW//iz4HnEr151NAKQaeFKX++6+omTfDrUf1kkOC4djBpkXkMs40pN+7XY6/W0/8vfptFA5KWooNpRSqsY6mqlVdKrPqf69TNnh7Av2D1K+YT2DQcMRfSmn/IPUr4l3KOweIOItYPMNTpxsMxwYDx7hZRraMMlmzjqa1K+i1x5FISSn+8t703OVQauZ9vEnamnkzQCnKh+1B/Yo4gUEjEBECWwynfsV32Xo00+RctKYc43R0cHjX6bnLoayb8QpN1ZU01Vax9uMX6L7tntR8N5P6ZXNRySaSdetZ8/b9+Lr1pHTzrTPmo5JNqMZ6UE2gkqjGelRy4y3GmqorWfdJjIr9dayekj4DqV3wKcn6GuqXfkNJ/nelbQcc52WBXtd0z8KDucHZMEp/9wpKKgaybuarOn1tFVWz36K0/9CN0qnGetZMnUDfQ3QlRxvlZ6imBuoWzaGkIu+NspnDg+HYlqZF5CLBcEyAC70qr2LfEwlsMYLF/zqH7+/XXWQV+/5aT2N86WYW3vFrFt97Fo1rljDg+GuQkgAAienPsOyZqzfkk3j/KRbcehxrP/g31Z9PZsGtx5F4/6mNyloz+QEq9vsNvkC5Lnvv46md/ymLx59O+bZ75etc3XRc6+7MhGfzdJ0qfBxwfacC1dTI6rfuo/qLqUhJKT1G/oS+Y85g/bcfUTntMZrWrURKApRtsR19DjyNwADtSBPTn6F24ecMPOEaAOqXzWP1W/fRsPw78Pnpts0oNvvpOfh79N1QVuW0x5BAORU/1tHjNszf/f4ryrfdk35H/jB/twC4Kh4NXWdaRK4RDMeOAl4yrcPSaZLA0Hg0tMCLwrx0uscAL3hSmMUt4sCweDSUG/FAc4RgOPY2cJBpHZYucU08Gop4UZCX3QvneliWxR2CwE9Ni8glguHYzliHWwic4dX0MU8KCYZjQ4HDvCjL4jpnmxaQY+TUrgSWTrMN8DMvCvKqpvtb7J5nhcLRwXCsv2kROcRRpgVYssbvvCjEK6fryS+IxRMCgA37yIYpkHZn38Lh6GA4Vu52Ia47XWeXWbuMtLCwCyU0R2JbcIVEAHA9UpUXNd0xGI7ba8k6duBIYyRgisVVXLdtL5yuHe0uPLYMhmPbmRZhkmA41h04xLQOS9YpCKd7qAdlWLyn2Gu7hwDdTIuwZJ093d7Ox1Wn6+wT5VpsUYtRir1ftyCi1Fs2oRTYz80C3K7p2uZX4TLGtADD7GZagMU1XG3Fue10R7mcv8UcA4Lh2HDTIgxinW7hsr+bmbvtdHdwOX+LWYaZFmCCYDg2CBhgWofFNTq33UY7sU7X0hWCpgUYwo5TFDaDguFYmVuZu+Z0HdGZ9za3FApD205SkGxvWoDFVQSXdioHd2u6W7ucv8U8QdMCDFHUc5SLhKBbGbvpFG0tt/AJmhZgCGvbhU/QrYyt07V0hWLtXnB1x19LThB0K2M3nW4/F/O25AYDvYjKlINYp1v4BN3K2E2na4PcFAcVpgUYwNVlopacwLWdnd10ugWzG6OlVUpNCzCAdbqFj2t2bZ2upasETAswgHW6hY9rdu1mF0DROd2DfDNn9aGqwbQOL6mkpw9CpmV4TVH16famKnGIb+Y3pnV4SQP+RW7ZtXW6WeKikmenXVjy/E9M6zDDX0wL8JrupgV4RXdqq98tu3Bhb6kZbVqLx9TA9a5kbLsXssBhvo9mXuB/fm/TOgxRZ1qAAdabFuAFpTTWTy276MveUrOzaS0GqHcrYzedblGsRhspC+b9s/SOYSJFOaAELhpnDrPCtAC3EZLJSYHLZvSXxB6mtRgiL53uahfzzgk2I7Hq5cCVJSJFOW2qmWJ0uitNC3CbFwNXvRf0LdvHtA6DuNaCc9PpLnQxb+MEaKibXHbJ4lJpKuaVdw1AwrQIAxS0032o9G9TRvm+K9LxiQ2scStj63Q7yauBK2ZUyPpiD9L+HZFEk2kRBihYp3tTyb1TD/LPGmNaRw7g2mwN63Q7wfjSOzvP/ccAABHUSURBVKYO931v98hy0TBznILs07205OlpJ5RMPdC0jhzha7cydtPpLgKSLuZvhHP9L713hP8ja5iab00LMETB1XRP878+/Y/+F13dkDHPyL+abjwaagSWupW/CQ7yzZw1ruSpPU3ryCGKtaZbUK24I33TZ0RKHtlDpDhmHLUDRT46XYeCMc5h8v38B0pv2VqkKJe9tkSxOt0ZpgVki/19n312V+ldI61db8RiIokatzJ32+kWRPOzgqrK1wLhpE/UZqa15BjF6nS/oQBmbYySud88Unrj1iI2lkQartq12073PZfzd50SGhsml10SL5PGYg3Y3RL1wALTIkwQj4YU8LFpHV1hqHy/4PnAX3v7xL0QhnmMa4No4L7Tnepy/q7zUuAvH24m63Y1rSMHmVuk08WayVunO4jVyyYGxuEXNdC0lhwlf2u68WjoC/J4es1tpeOn7uhbsL9pHTnKm6YFGOa/pgV0hgqqKqeUXbS2yBf1tMU7bmbuxWjlux6UkXVO878+/Re+dw8wrSOHiZkWYJi8c7rdqa2eVjZ2YTdpGGFaSw6zBJdbMV443bzrYtjXN/vzSMkju4kgprXkKNXAFNMiTBKPhhaQR1MiS2msn1J20Ze9Zf0uprXkOK8SSSg3C/DC6bpaVc82Q2TposdKbxwgQjfTWnKYt4gkijGkYzqvmhbQHoRkcmLgshkDijdiWEd42e0CvHC6s4BKD8rpMr2oTrwRGFfrE9XftJYcp9i7Fpp5zrSA9vBC4K/vDS3uiGHtpRYPxipcd7rxaCgJvOB2OV3FT1PjW2WXfttNGoab1pIH5EUNzwPeBNaZFtEaD5beNOVHvnnFHjGsvUwmkqh2uxCvlv097lE5nebZwDXv2+ZXu5hFJLHItIhcIB4N1eFBc7Sz/K3k3ikH+/83xrSOPOIVLwrxyum+DXzvUVkd5rqSB6bu7vvWzlRoH7ZrYWNyskJxSckz035dMnWMaR15RuE4XaeLISeN80T/2x+e7H/LNr/aRxKYYFpEjjERWG5aRCqn+t+Yfr7/BRsxrGNMJ5LwZIWll1GF/oWO3pMzjJav5txYcv/ONrpSu3mRSKJY4y1kxImm94RpHc2EfB/MuKbkYRsxrOPc7FVBnn0w8WjoG2CyV+W1xZasWPJ04Nq+XQn2ccpzNWxx6zp637iW7e6q4v5PftgubH2D4g+xGvrdtI6K6FoOeChz/3xdo+J3L9Yw5I519LpxLbv+s4rXvmnYcH1hIsne91ez2d/WcskbtRvde/jj1Xz8vacrcW/xsrA84i7A+JLo/XyzZ99d+vcuRwzLhl0DzFnRxMEPV1MRXcvwv6/j+Tk5a9dfAS96VViJVwU53AMc7HGZm9CDmqpJZZev9Yvaviv5XLF/gAeO7kZZifDlyibGTFjPboP87DHYz+9frqUxqZjzxx5sVi78b2nmeO6NSdi6Qph6eg+2qRBe/aaRE/5dw2fn+Qn28XHju3Wc9qNSTtqlO7vfV8Vvdill9GA/T89uYGgfH6MHe7bT/XQiife9KiyfiEdD84Lh2JPAKaY07CLzvnm09IatshExLDt2rTjmqRrOHV3KpN92Z+r8Jo56cj0zB/jYbnN/Ltk1wC1EEp5tuOB1E+R54EuPy9wIH8mmSWWXfdFd6rrkcAF2GuCnrEQvWhNABOauSfLlyiZe+qqB+44qp38PH36fsEcLRtQjIETGdCPYx4dPhCO3K2VoHx8znF/67yqTHDzUT0U3Yc/BfuatSbK2ThF9r44bDvF0/Yat5bbODRjaKSUoSxa+ELgqaxHDsmHXX65M8v26JBftHcDvEw4eWsJ+W/t5dJau7eaQXS8BHvWyQE+dbjwaagKu9LLMdJ4IXP/eYFm9V7by+0Oshu7Xr2XkP6rZoqdwxIgSPlrcxJA+Pq6eXEe/m9axyz1V/OeLhrYzA5ZVJfl6VZKdBuiPZuf+fibNa6KyVjFjSZKd+vu46u06xv44QJ9unq1S/oY8mGttkng0NAcDiyUGsXrZpMDlKtsRw7Jt16AHdGav0L9LOWLXAHd6vbrS8872eDT0HPCR1+UC/KXksXf29s3J6tSw8aFy1l3Ri2lndOe4HUop88OitYrZy5NUdIPvL+nJ3Yd347QXapizovV+qoYmxcnP1XDaj0oZ2U/XIK74SRnTFjRy4IRq/jC6lPom+HR5E0dtX8pJ/1nPAQ9Vc/dH9a3mmwVu97L5lcdc52VhFVRVTi672JWIYV216+039zGgh3Dz+/U0NCkmzm1karyJ9Q16LD1H7Hot8E+3C0lHlPJ+QkEwHDsIPXfXM471vfvx7aXjdxPBtc6ic1+pYcf+PpqSMO7NOtZf2YsSn/7VPurJ9Rw61M+Fe5dlvDepFCf9p4a1dfDiieWU+jf9tU8qxQEPreefR3bjkVkN9OkmXLpvgN3vrebZ48vZob8rj7YCGOLm9iWFRDAcexk40u1yyqlb/2HZH+d6EcCms3b96bIm/vRaLbOXJxk92Ef/7j7K/PDAMeUbpTNk1wA3EUmMcyvzljAyrSQeDU0GJnlV3iiZ+83tpeO3d9Phgh4Um7taMWrgpsW01mBSSvG7l2pZVq34zwmZHS7AfTMa2HsrPzsP8PPZ8iZGD/YT8Au7DPTx2XLXKqJXWofbIa7B5amRpTTWTy276AuvIoZ11q5HDfQz9fQerLq8F2+c0oN5a5LsteWmeRiy67XAbW5l3hom5/KF8WDe7kBWL38ucHV3EXplM9/l1Umemt1AVb2iKal449tGnpzdwCHD/BwwxM82FcKN0+ppTCreW9DI5Hgjhw3PPFnkvFgtc1Ykefk33SkvzWzGy6uT/OO/9UTG6BrF0D4+Jn/XSFW94uPvkwzr68pH+QFwvxsZFyrxaOhj9CwdVxCSyTcCl88YIJWj3cg/m3b96bImahsV6xsUt7xfx5KqJKfvWrpJeQbsGuDPRBLL3Mq8NYx0LzQTDMceB05yK/9y6tb/t+y8eE+p3THbea+oTvKrZ2uYtbSJpIIhfXxcsFeAs/fQUyQ/X97EWS/X8umyJoZU+Lj+4DJ+sYM2uBum1TFtQSOvndyD+ZVJgndWUeaHkhT7uvfIck4e9YOBnvp8DUdtV8LxO+lzCxNJfvXser5eleSMXQPcdljWR3ybgD2JJGZmO+NCJxiO9QI+B7bOdt4vBK6atqtvrmsrKLNl1wCXTazl/pn1NDTBT4b4uevwcoZvtrETNWDXoAPQ721qnMK0090c+AzYIvu5K/VO4KIPt/Et3zv7eRcFdxJJjDUtIl8JhmNHkOU4FQ+U3jzlEP/MMdnMswgxXpkwulQwHg2tAk7HhW6Gh0pvmmodbqf5FvizaRH5TDwaepUsxhuJltxnHW52uNl06834+ux4NDQRvYwya1xS8sy0g/yzxmQzzyIiCZxBJLHetJAC4EKysDHrJSXPTDuxZMqYrsspemYBV5sWYdzpOowDZmcjo5/7PvzkfP8Ltobbee4gksjLzURzDacl96eu5PFb/8QPzve/sG+WJBUzdcApRBKuT/5ti5xwuvFoqBY4Gf3GdJodZP7ce0rv3FaE0rZTWzIwHcMrBguNeDT0NJ2czXCE78NPri2ZsLvbUx2LhCuJJLJSsesqOeF0AeLR0Kd0oR9xcxIrXwr8pVSEiizKKibmAccQSdS2mdLSUS6gg3tv7eebPfsfpXdu39WIYRYA7ieSuNW0iGZyxukCxKOh2+jEvNAADXVvl12yxI3lkEXCauAIIoku9z9aNsWJuXs87Qz2tJN89+2jpTdsmY2IYRZeBc4zLSKVnHK6DufSoeAqSr0WCM+o8Gh1TgFSD/yCSOIr00IKmXg0VIleHryqtXRBWbLwpcBfevqEvt4oK2g+Bk4gkmg0LSSVnHO6TiSy3wDvtCf9PaV3vLOtb4kdaOg8ZxJJtOu9tnSNeDQ0F/glkDE010BWL58UuDzpFzXIW2UFyVwg5MXuvh0l55wubBhYOxo9xaNF/uB/8b3D/f890BtVBcnVRBI5uXddoRKPhqYC55A2N703VYkpZRcnSqVpiBllBcVK4HAiiZzau66ZnHS6APFoKAH8HPgu0/WDfDNnXVby9J7eqiooHiSSuNa0iGIkHg09RIrjLadu/bSysQvKpX6EWWUFQQ1wVC7v5ZezThcgHg0tBX4GbLRL57ayeP4DpbdsbUd2O00UOMu0iGImHg39Czi9hMbaqWUXfWHHJLLCOuBYIokPTAtpDaOxF9pLMBzbEj0KOaoP69Z8VPbHyoA0DjWtKw9pAv5AJHGfaSEWzftX7XP0vv4vngEyB1q2tJc4uoabE3NxWyMvnC5AMBzrHaDhmQ/L/jiwr1TtalpPHlKFHsl9zbQQSxqRip+gd6O1MxY6x3R0DTcn+3DTyRunC0CkohS4G/i9aSl5xhL0SK4N05irRCp2AF4D7EBax3gSPQMnbxb15JfTbSZS8QfgDrDLfdvB5+iFDwvaTGkxS6RiEPAwehzD0jYRIolrTIvoKPnpdAEiFQcAzwBZ3QW1wHgaOIdIImFaiKWdRCoE+ANwE9DdsJpcpRYdCe8p00I6Q/46XYBIRV/gZuBMWt+uqdhYjB4we8m0EEsniVSMAB4BbMS8jZkCnEck0a4l1blIfjvdZiIVBwL3AtublmIYhY5dcZmt3RYAkQo/ei/Bq7FdaSuBS4kkHjYtpKsUhtMFiFSUocMSjoOinL87FzibSGKyaSGWLBOp2A1d693ZtBQDKOAh4HIiiVbjVuQLheN0m9GjwPcB+5uW4hFNwO3AX+1W6QWMrlRcCFwK9DesxivmAOcWWmyQwnO60DwYcRq61jvSsBq3SKIHEq8jkvjctBiLR0QquqNDFV5G4Q4iV6LHam7JhZ0esk1hOt1mtPM9ArgEOMiwmmzRBDwBXG/DMRYxkYpydPyGy3FlN20jLEa32u4jklhnWoxbFLbTTUX3i10C/BooMaymM6xBD5L9g0hivmkxlhwhUtENHUcjDGxpWE1nmYle9PRYIdZs0ykep9tMpGIr9PYpZwN9DKtpCwXMQDvbR9N36BWRqpSX3dF7zDU5r89RStmwjcWC7vMNoeP1Hgn0NiuoTWrQ88jvIZL4yLQYLyk+p9uMXlK8H7r74XByZ2R4PXo/rZeBGJHEkvbcJCJx4Cyl1CZ7cYlIiVIqp6LnW1xEO+BD0Q74GGAzs4I28D0wCZgIvEYksSZTokKvTBSv000nUrE12vkejjbYnh6WvgCIoR3t5M6sI091uiIyBngMuAu4CG3obznX90+5RwEjlFLfikgZcD1wAjri1fPARUopOyMin4lUlABj0A74KLztglgPTKXZ0XZiwLcQKxP52LfpDpHEQvRUs/uIVATQteDdgB2AHZ2/XY0CtRr4Ou34nEjiiy7mm4lB6BrOEHTc5F+3kT4KbAvsit5O5gngr8AVLmizeIXeH+xN5zjPie+wO9q2m+17W7oWWrIJWATMd4556O223iOSqOtCvhuRqTIhInlXmbBONxO6M3+yc6Scr+gLbA1s4/xtni+p0FO4Mv1dQbOD9XZydxK4WilVByDS8ipp0Rd/D4xSSq12zt2AdrzW6RYSkcRSdGzqV384V+FD2/MI9EyIQBtHJTp+bbOTXUQk0YQ35H1lwjrdjqD7oNYAn5qW0g5WKKXa203RH913NiPFOQvgd0OYJceIJJL84EBznbyvTFinW7ikd9ZXkxK1SkRSd5xdiR5N3kkptdgDbRZLZ8n7ykRO75FmySqzgJ1EZFcR6QZEmi8opZLAv4DbRWQAgIhsKSKHGVFqsbRMZysTfZyjQinl5SD5JlinWyQopb4GrkUPqHwDvJuWZBzwLfCBiKx10hV71DZL7pN3lQk7ZcxiseQsmaZCKqW2SktzJXo2Qw26r/ZRfpi90A09cHYi0A+91PgepdTfvXuKjbFO12KxWDzEdi9YLBaLh1ina7FYLB5ina7FYrF4iHW6FovF4iHW6VosFouHWKdrsVgsHmKdrsVisXiIdboWi8XiIdbpWiwWi4dYp2uxWCweYp2uxWKxeIh1uhaLxeIh1ulaLBaLh1ina7FYLB5ina7FYrF4iHW6FovF4iH/D4pzGWpMVEh/AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HtgOkvybG-jo",
        "outputId": "af19b811-b6d1-4cba-adba-2be8bb7740e3"
      },
      "source": [
        "print(f\"Train datasen contains {len(train)} samples.\")\r\n",
        "print(f\"Valid dataset contains {len(valid)} samples.\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train datasen contains 19845 samples.\n",
            "Valid dataset contains 8505 samples.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wP3XKmPoQzZO",
        "outputId": "e949e182-f233-4173-8b82-251d2627e908"
      },
      "source": [
        "train.word.value_counts()[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "дело        717\n",
              "дух         665\n",
              "единица     563\n",
              "защита      505\n",
              "знак        441\n",
              "диск        439\n",
              "действие    382\n",
              "закон       365\n",
              "задержка    353\n",
              "день        333\n",
              "Name: word, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xuivwr68WRh1",
        "outputId": "7303f4e6-8c14-4760-b4bb-92b78684e9ba"
      },
      "source": [
        "train.word.value_counts(normalize=True)[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "дело        0.036130\n",
              "дух         0.033510\n",
              "единица     0.028370\n",
              "защита      0.025447\n",
              "знак        0.022222\n",
              "диск        0.022121\n",
              "действие    0.019249\n",
              "закон       0.018393\n",
              "задержка    0.017788\n",
              "день        0.016780\n",
              "Name: word, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2X3ThEhLJ7H5"
      },
      "source": [
        "# Base Models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hROuboI4O47_"
      },
      "source": [
        "## RSG Baseline"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eXVLSTWVO--P"
      },
      "source": [
        "%%capture\r\n",
        "%%bash\r\n",
        "wget -q --show-progress \"https://russiansuperglue.com/tasks/tf_idf\" -O temp.zip\r\n",
        "unzip temp.zip -d data"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y5_cYVlgO4O-"
      },
      "source": [
        "import joblib\r\n",
        "import codecs\r\n",
        "import json\r\n",
        "\r\n",
        "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\r\n",
        "from sklearn.linear_model import LogisticRegression"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q-bg2zwGPAU1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0bcd54db-4123-4246-dd94-9c9b121a5025"
      },
      "source": [
        "vect = joblib.load(\"data/tfidf.pkl\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator TfidfTransformer from version 0.21.3 when using version 0.22.2.post1. This might lead to breaking code or invalid results. Use at your own risk.\n",
            "  UserWarning)\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator TfidfVectorizer from version 0.21.3 when using version 0.22.2.post1. This might lead to breaking code or invalid results. Use at your own risk.\n",
            "  UserWarning)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O0GHM2aoWPb5",
        "outputId": "23c0ea21-8716-47e8-e562-6914fa58db82"
      },
      "source": [
        "# taken from https://github.com/RussianNLP/RussianSuperGLUE/blob/master/tfidf_baseline/RUSSE.py\r\n",
        "import codecs\r\n",
        "import json\r\n",
        "from sklearn.linear_model import LogisticRegression\r\n",
        "from sklearn.metrics import classification_report\r\n",
        "\r\n",
        "def build_feature_RUSSE(row):\r\n",
        "    sentence1 = row[\"sentence1\"].strip()\r\n",
        "    sentence2 = row[\"sentence2\"].strip()\r\n",
        "    word = row[\"word\"].strip()\r\n",
        "    label = row.get(\"label\")\r\n",
        "    res = f\"{sentence1} {sentence2} {word}\"\r\n",
        "    return res, label\r\n",
        "\r\n",
        "\r\n",
        "def build_features_RUSSE(path, vect):\r\n",
        "    with codecs.open(path, encoding='utf-8-sig') as reader:\r\n",
        "        lines = reader.read().split(\"\\n\")\r\n",
        "        lines = list(map(json.loads, filter(None, lines)))\r\n",
        "    res = list(map(build_feature_RUSSE, lines))\r\n",
        "    texts = list(map(lambda x: x[0], res))\r\n",
        "    labels = list(map(lambda x: x[1], res))\r\n",
        "    ids = [x[\"idx\"] for x in lines]\r\n",
        "    return (vect.transform(texts), labels), ids\r\n",
        "\r\n",
        "\r\n",
        "def fit_RUSSE(train, labels):\r\n",
        "    clf = LogisticRegression()\r\n",
        "    return clf.fit(train, labels)\r\n",
        "\r\n",
        "\r\n",
        "def eval_RUSSE(train_path, val_path, test_path, vect):\r\n",
        "    train, _ = build_features_RUSSE(train_path, vect)\r\n",
        "    val, _ = build_features_RUSSE(val_path, vect)\r\n",
        "    test, ids = build_features_RUSSE(test_path, vect)\r\n",
        "    clf = fit_RUSSE(*train)\r\n",
        "    try:\r\n",
        "        test_score = clf.score(*test)\r\n",
        "    except ValueError:\r\n",
        "        test_score = None\r\n",
        "    val_pred = clf.predict(val[0])\r\n",
        "    print(classification_report(val[1], val_pred))\r\n",
        "    return clf, {\r\n",
        "        \"train\": clf.score(*train),\r\n",
        "        \"val\": clf.score(*val),\r\n",
        "        \"test\": test_score,\r\n",
        "    }\r\n",
        "\r\n",
        "\r\n",
        "train_path = \"data/RUSSE/train.jsonl\"\r\n",
        "val_path = \"data/RUSSE/val.jsonl\"\r\n",
        "test_path = \"data/RUSSE/test.jsonl\"\r\n",
        "\r\n",
        "_, RUSSE_scores = eval_RUSSE(train_path, val_path, test_path, vect)\r\n",
        "train_scr = RUSSE_scores['train']\r\n",
        "valid_scr = RUSSE_scores['val']\r\n",
        "\r\n",
        "\r\n",
        "print(f\"Accuracy Score on Train: {train_scr}\") # should be around 0.71\r\n",
        "\r\n",
        "print(f\"Accuracy Score on Valid: {valid_scr}\") # should be around 0.66"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.68      0.90      0.77      5366\n",
            "        True       0.61      0.26      0.36      3139\n",
            "\n",
            "    accuracy                           0.67      8505\n",
            "   macro avg       0.64      0.58      0.57      8505\n",
            "weighted avg       0.65      0.67      0.62      8505\n",
            "\n",
            "Accuracy Score on Train: 0.7103552532123961\n",
            "Accuracy Score on Valid: 0.6654908877131099\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IDr9yTK6YHl-"
      },
      "source": [
        "## Modified"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3pQ9rlJ9bQRw"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\r\n",
        "from sklearn.linear_model import LogisticRegression\r\n",
        "\r\n",
        "import numpy as np\r\n",
        "\r\n",
        "from sklearn.model_selection import train_test_split\r\n",
        "from sklearn.metrics import classification_report, accuracy_score, f1_score\r\n",
        "\r\n",
        "from sklearn.metrics.pairwise import cosine_distances, cosine_similarity\r\n",
        "from sklearn.model_selection import GridSearchCV\r\n",
        "from sklearn.model_selection import cross_val_score"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bTCo22HdYSKK"
      },
      "source": [
        "# concatenate columns \r\n",
        "\r\n",
        "def build_feature_RUSSE(row):\r\n",
        "    sentence1 = row[\"sentence1\"].strip()\r\n",
        "    sentence2 = row[\"sentence2\"].strip()\r\n",
        "    word = row[\"word\"].strip()\r\n",
        "    res = f\"{sentence1} {sentence2} {word}\"\r\n",
        "    return res\r\n",
        "\r\n",
        "train_concat = []\r\n",
        "for i, row in train.iterrows():\r\n",
        "    train_concat.append(build_feature_RUSSE(row))\r\n",
        "train['concatenated'] = train_concat\r\n",
        "\r\n",
        "valid_concat = []\r\n",
        "for i, row in valid.iterrows():\r\n",
        "    valid_concat.append(build_feature_RUSSE(row))\r\n",
        "valid['concatenated'] =  valid_concat"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 650
        },
        "id": "ckvqC7xPaA6z",
        "outputId": "71576634-2f26-4c81-f165-1ccdd3aabf74"
      },
      "source": [
        "train.head(2)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>idx</th>\n",
              "      <th>word</th>\n",
              "      <th>sentence1</th>\n",
              "      <th>sentence2</th>\n",
              "      <th>start1</th>\n",
              "      <th>end1</th>\n",
              "      <th>start2</th>\n",
              "      <th>end2</th>\n",
              "      <th>label</th>\n",
              "      <th>gold_sense1</th>\n",
              "      <th>gold_sense2</th>\n",
              "      <th>concatenated</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>двор</td>\n",
              "      <td>В нашей деревне осталось от силы двадцать дворов</td>\n",
              "      <td>Солнце стояло уже высоко, когда справа от дороги я увидел деревеньку дворов в пятнадцать</td>\n",
              "      <td>42</td>\n",
              "      <td>49</td>\n",
              "      <td>69</td>\n",
              "      <td>76</td>\n",
              "      <td>True</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>В нашей деревне осталось от силы двадцать дворов Солнце стояло уже высоко, когда справа от дороги я увидел деревеньку дворов в пятнадцать двор</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>доклад</td>\n",
              "      <td>Табличка на дверях: «Без доклада не входить»</td>\n",
              "      <td>Вчера отбыл в столицу первый секретарь обкома […], вызванный то ли «на ковёр», то ли на сезонный доклад об урожае</td>\n",
              "      <td>25</td>\n",
              "      <td>33</td>\n",
              "      <td>97</td>\n",
              "      <td>104</td>\n",
              "      <td>False</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>Табличка на дверях: «Без доклада не входить» Вчера отбыл в столицу первый секретарь обкома […], вызванный то ли «на ковёр», то ли на сезонный доклад об урожае доклад</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   idx  ...                                                                                                                                                           concatenated\n",
              "0    0  ...                         В нашей деревне осталось от силы двадцать дворов Солнце стояло уже высоко, когда справа от дороги я увидел деревеньку дворов в пятнадцать двор\n",
              "1    1  ...  Табличка на дверях: «Без доклада не входить» Вчера отбыл в столицу первый секретарь обкома […], вызванный то ли «на ковёр», то ли на сезонный доклад об урожае доклад\n",
              "\n",
              "[2 rows x 12 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FRopr_RkKApQ",
        "outputId": "95d3aa0a-a136-4408-ac8b-51c42bf35008"
      },
      "source": [
        "from sklearn.pipeline import Pipeline\r\n",
        "from sklearn.preprocessing import StandardScaler\r\n",
        "from sklearn.preprocessing import FunctionTransformer\r\n",
        "\r\n",
        "steps = [('tfidf', TfidfVectorizer()),\r\n",
        "         ('logreg', LogisticRegression(class_weight='balanced'))]\r\n",
        "\r\n",
        "parameters = {'logreg__C': np.arange(0.01, 1.9),\r\n",
        "              'tfidf__min_df': np.arange(0.001, 0.1, 0.03),\r\n",
        "              'tfidf__max_df': np.arange(0.6 , 1.0, 0.1),\r\n",
        "              'tfidf__ngram_range': [(1,2), (1,3)],\r\n",
        "              'tfidf__analyzer': ['char_wb', 'word']\r\n",
        "              }\r\n",
        "\r\n",
        "X_train = train.concatenated\r\n",
        "X_test = valid.concatenated\r\n",
        "\r\n",
        "y_train = train.label\r\n",
        "y_test = valid.label\r\n",
        "\r\n",
        "pipeline = Pipeline(steps)\r\n",
        "\r\n",
        "search = GridSearchCV(pipeline, cv=3, param_grid = parameters)\r\n",
        "\r\n",
        "search.fit(X_train, y_train)\r\n",
        "y_pred = search.predict(X_test)\r\n",
        "\r\n",
        "print(\"Accuracy: {}\".format(search.score(X_test, y_test)))\r\n",
        "print(classification_report(y_test, y_pred))\r\n",
        "print(\"Tuned Model Parameters: {}\".format(search.best_params_))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
            "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
            "\n",
            "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
            "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
            "Please also refer to the documentation for alternative solver options:\n",
            "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
            "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Accuracy: 0.6398589065255732\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.74      0.66      0.70      5366\n",
            "        True       0.51      0.61      0.56      3139\n",
            "\n",
            "    accuracy                           0.64      8505\n",
            "   macro avg       0.63      0.63      0.63      8505\n",
            "weighted avg       0.66      0.64      0.64      8505\n",
            "\n",
            "Tuned Model Parameters: {'logreg__C': 1.01, 'tfidf__analyzer': 'word', 'tfidf__max_df': 0.6, 'tfidf__min_df': 0.001, 'tfidf__ngram_range': (1, 2)}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "efXcXgyzJ-cy"
      },
      "source": [
        "# Heruistics"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yFpCAr9q0Yd8",
        "outputId": "ea68a211-bfca-420c-9f41-e33202910b31"
      },
      "source": [
        "%%bash \r\n",
        "wget -q --show-progress \"https://raw.githubusercontent.com/tatiana-iazykova/2020_HACK_RUSSIANSUPERGLUE/main/base.py\" -O base.py"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "     0K ...                                                   100% 8.86M=0s"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sCIQ2EeqO92O"
      },
      "source": [
        "from base import BaseSolver\r\n",
        "from scipy import stats\r\n",
        "import numpy as np\r\n",
        "\r\n",
        "class RusseSolver(BaseSolver):\r\n",
        "   \r\n",
        "    def __init__(self, path: str, path_valid=None):\r\n",
        "        super(RusseSolver, self).__init__(path, path_valid)\r\n",
        "\r\n",
        "    \r\n",
        "    def preprocess(self, columns):\r\n",
        "\r\n",
        "        for column in columns:\r\n",
        "            self.train[f\"{column}_lemmas\"] = self.train[column].apply(self.lemmatize)\r\n",
        "            self.valid[f\"{column}_lemmas\"] = self.valid[column].apply(self.lemmatize)\r\n",
        "\r\n",
        "    \r\n",
        "    def heruistic_same_form(self, final_decision=None):\r\n",
        "        y_true = self.valid.label\r\n",
        "        y_pred = []\r\n",
        "\r\n",
        "        c = 0\r\n",
        "        c_true = 0\r\n",
        "        for i, row in self.valid.iterrows():\r\n",
        "            word1 = row.sentence1[row.start1:row.end1].lower()\r\n",
        "            word2 = row.sentence2[row.start2:row.end2].lower()\r\n",
        "            if word1 == word2:\r\n",
        "                c += 1\r\n",
        "                if row.label is False:\r\n",
        "                    c_true += 1\r\n",
        "                y_pred.append(False)\r\n",
        "            else:\r\n",
        "                options = np.array([final_desicion(test_size=1)[0] for i in range(0,3)])\r\n",
        "                y_pred.append(stats.mode(options)[0][0])\r\n",
        "        print(f'Heruistic happend for {c} samples, {c_true} of them correct')\r\n",
        "        self.show_report(y_true, y_pred)\r\n",
        "\r\n",
        "    def heruistic_vocab_intersection(self, final_decision=None):\r\n",
        "        y_true = self.valid.label\r\n",
        "        y_pred = []\r\n",
        "\r\n",
        "        c = 0\r\n",
        "        c_true = 0\r\n",
        "        for i, row in self.valid.iterrows():\r\n",
        "            tokens1 = set(row.sentence1.split())\r\n",
        "            tokens2 = set(row.sentence2.split())\r\n",
        "\r\n",
        "            if len(tokens1 & tokens2) / len(tokens1 | tokens2) > 0.10:\r\n",
        "                c += 1\r\n",
        "                if row.label is True:\r\n",
        "                    c_true += 1\r\n",
        "                y_pred.append(True)\r\n",
        "            else:\r\n",
        "                options = np.array([final_decision(test_size=1)[0] for i in range(0,3)])\r\n",
        "                y_pred.append(stats.mode(options)[0][0])\r\n",
        "        print(f'Heruistic happend for {c} samples, {c_true} of them correct')\r\n",
        "        self.show_report(y_true, y_pred)\r\n",
        "\r\n",
        "    def heruistic_sentence_length(self, final_decision=None):\r\n",
        "        y_true = self.valid.label\r\n",
        "        y_pred = []\r\n",
        "\r\n",
        "        c = 0\r\n",
        "        c_true = 0\r\n",
        "        for i, row in self.valid.iterrows():\r\n",
        "            tokens1 = len(row.sentence1.split())\r\n",
        "            tokens2 = len(row.sentence2.split())\r\n",
        "\r\n",
        "            if max([tokens1, tokens2]) - min([tokens1, tokens2]) > 6:\r\n",
        "                c += 1\r\n",
        "                if row.label is False:\r\n",
        "                    c_true += 1\r\n",
        "                y_pred.append(False)\r\n",
        "            else:\r\n",
        "                options = np.array([final_decision(test_size=1)[0] for i in range(0,3)])\r\n",
        "                y_pred.append(stats.mode(options)[0][0])\r\n",
        "        print(f'Heruistic happend for {c} samples, {c_true} of them correct')\r\n",
        "        self.show_report(y_true, y_pred)\r\n",
        "\r\n",
        "    def heruistic_combined(self, final_decision=None):\r\n",
        "        y_true = self.valid.label\r\n",
        "        y_pred = []\r\n",
        "\r\n",
        "        c = 0\r\n",
        "        c_true = 0\r\n",
        "        for i, row in self.valid.iterrows():\r\n",
        "            tokens1 = len(row.sentence1.split())\r\n",
        "            tokens2 = len(row.sentence2.split())\r\n",
        "            tokens1_set = set(row.sentence1.split())\r\n",
        "            tokens2_set = set(row.sentence2.split())\r\n",
        "            word1 = row.sentence1[row.start1:row.end1].lower()\r\n",
        "            word2 = row.sentence2[row.start2:row.end2].lower()\r\n",
        "\r\n",
        "            if max([tokens1, tokens2]) - min([tokens1, tokens2]) > 7:\r\n",
        "                y_pred.append(False)\r\n",
        "            else:\r\n",
        "                if len(tokens1_set & tokens2_set) / len(tokens1_set | tokens2_set) > 0.10:\r\n",
        "                    y_pred.append(True)\r\n",
        "                else:\r\n",
        "                    if word1 == word2:\r\n",
        "                        y_pred.append(False)\r\n",
        "                    else:\r\n",
        "                        options = np.array([final_decision(test_size=1)[0] for i in range(0,3)])\r\n",
        "                        y_pred.append(stats.mode(options)[0][0])\r\n",
        "        self.show_report(y_true, y_pred)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZqdlztbAYJrV"
      },
      "source": [
        "## Same form"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C1atRBs9_82_",
        "outputId": "a14ddb22-a493-4511-df5e-f84a9b6a7662"
      },
      "source": [
        "# Совпадение словоформы (same_form) + Majority Vote\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_same_form(final_desicion=lambda test_size: [1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Heruistic happend for 1265 samples, 747 of them correct\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.59      0.14      0.23      5366\n",
            "        True       0.36      0.83      0.51      3139\n",
            "\n",
            "    accuracy                           0.40      8505\n",
            "   macro avg       0.48      0.49      0.37      8505\n",
            "weighted avg       0.51      0.40      0.33      8505\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LcvvUyEuKBEt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "96100046-29e6-44f5-c8a5-90b8d71f8555"
      },
      "source": [
        "# Совпадение словоформы (same_form) + Majority Vote\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_same_form(final_desicion=solver.majority_class)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Heruistic happend for 1265 samples, 747 of them correct\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.63      1.00      0.77      5366\n",
            "        True       0.00      0.00      0.00      3139\n",
            "\n",
            "    accuracy                           0.63      8505\n",
            "   macro avg       0.32      0.50      0.39      8505\n",
            "weighted avg       0.40      0.63      0.49      8505\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OIrNSMzUQHC5",
        "outputId": "2d0db262-ab3d-449f-a215-ff8bbb6873ca"
      },
      "source": [
        "# Совпадение словоформы (same_form) + Random Choice\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_same_form(final_desicion=solver.random_choice)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Heruistic happend for 1265 samples, 747 of them correct\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.62      0.57      0.60      5366\n",
            "        True       0.36      0.41      0.38      3139\n",
            "\n",
            "    accuracy                           0.51      8505\n",
            "   macro avg       0.49      0.49      0.49      8505\n",
            "weighted avg       0.53      0.51      0.52      8505\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hhapQZFWQSC2",
        "outputId": "b937c947-887e-4e8d-9a2d-5dae9d3a3e64"
      },
      "source": [
        "# Совпадение словоформы (same_form) + Rancom balanced Choice\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_same_form(final_desicion=solver.random_balanced_choice)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Heruistic happend for 1265 samples, 747 of them correct\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.63      0.75      0.68      5366\n",
            "        True       0.36      0.24      0.29      3139\n",
            "\n",
            "    accuracy                           0.56      8505\n",
            "   macro avg       0.49      0.49      0.48      8505\n",
            "weighted avg       0.53      0.56      0.54      8505\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "myl3x93n2Pgh"
      },
      "source": [
        "## Vocab Intersection"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HzebUDBX2UOZ",
        "outputId": "8c33dc92-7b32-4c58-baeb-f6253991317d"
      },
      "source": [
        "# Пересечение токенов  + Majority Vote\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_vocab_intersection(final_decision=solver.majority_class)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Heruistic happend for 328 samples, 150 of them correct\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.63      0.97      0.77      5366\n",
            "        True       0.46      0.05      0.09      3139\n",
            "\n",
            "    accuracy                           0.63      8505\n",
            "   macro avg       0.55      0.51      0.43      8505\n",
            "weighted avg       0.57      0.63      0.52      8505\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ThOtryAU2xaK",
        "outputId": "cc3237ee-0064-47f7-fac6-3e77fa648d9e"
      },
      "source": [
        "# Пересечение токенов  + Random Choice\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_vocab_intersection(final_decision=solver.random_choice)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Heruistic happend for 328 samples, 150 of them correct\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.64      0.49      0.55      5366\n",
            "        True       0.37      0.53      0.44      3139\n",
            "\n",
            "    accuracy                           0.50      8505\n",
            "   macro avg       0.51      0.51      0.49      8505\n",
            "weighted avg       0.54      0.50      0.51      8505\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E-5RDrDp3HR5",
        "outputId": "be351894-6ec7-4f34-e83d-4e360a628905"
      },
      "source": [
        "# Пересечение токенов  + Random Balanced\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_vocab_intersection(final_decision=solver.random_balanced_choice)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Heruistic happend for 328 samples, 150 of them correct\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.63      0.68      0.65      5366\n",
            "        True       0.37      0.32      0.35      3139\n",
            "\n",
            "    accuracy                           0.55      8505\n",
            "   macro avg       0.50      0.50      0.50      8505\n",
            "weighted avg       0.54      0.55      0.54      8505\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aLjAc_2f3sRH"
      },
      "source": [
        "## Sentence Length"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3xdBSRkD3ufU",
        "outputId": "2e862ec6-2737-4c37-932e-d61121198362"
      },
      "source": [
        "# Разница в длинне предложений  + Majority Vote\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_sentence_length(final_decision=solver.majority_class)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Heruistic happend for 4105 samples, 2626 of them correct\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.63      1.00      0.77      5366\n",
            "        True       0.00      0.00      0.00      3139\n",
            "\n",
            "    accuracy                           0.63      8505\n",
            "   macro avg       0.32      0.50      0.39      8505\n",
            "weighted avg       0.40      0.63      0.49      8505\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r9Rgriaw32-s",
        "outputId": "dd05dace-3790-40d3-9bd1-134b3244c937"
      },
      "source": [
        "# Разница в длинне предложений  + Random Choice\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_sentence_length(final_decision=solver.random_choice)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Heruistic happend for 4105 samples, 2626 of them correct\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.63      0.74      0.68      5366\n",
            "        True       0.37      0.27      0.31      3139\n",
            "\n",
            "    accuracy                           0.56      8505\n",
            "   macro avg       0.50      0.50      0.50      8505\n",
            "weighted avg       0.54      0.56      0.54      8505\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qsWNa2AD34FZ",
        "outputId": "559f54e8-f893-4ea9-8ec6-728c2e369db7"
      },
      "source": [
        "# Разница в длинне предложений  + Random Balanced Choice\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_sentence_length(final_decision=solver.random_balanced_choice)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Heruistic happend for 4105 samples, 2626 of them correct\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.63      0.85      0.73      5366\n",
            "        True       0.39      0.16      0.23      3139\n",
            "\n",
            "    accuracy                           0.60      8505\n",
            "   macro avg       0.51      0.51      0.48      8505\n",
            "weighted avg       0.54      0.60      0.54      8505\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wcT7r9mP45nd"
      },
      "source": [
        "## Heruistics Combined"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8V4TOmmm4-kz",
        "outputId": "4c56557d-5e48-4b1b-c460-69f55f68a905"
      },
      "source": [
        "# Совпадение словоформы (same_form) + Majority Vote\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_combined(final_decision=solver.majority_class)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.63      0.97      0.77      5366\n",
            "        True       0.41      0.04      0.07      3139\n",
            "\n",
            "    accuracy                           0.63      8505\n",
            "   macro avg       0.52      0.50      0.42      8505\n",
            "weighted avg       0.55      0.63      0.51      8505\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y1_TkL0r5ROb",
        "outputId": "732fead7-b8c7-42c7-cbe1-67968db61b72"
      },
      "source": [
        "# Совпадение словоформы (same_form) + Random Choice\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_combined(final_decision=solver.random_choice)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.63      0.73      0.68      5366\n",
            "        True       0.37      0.28      0.32      3139\n",
            "\n",
            "    accuracy                           0.56      8505\n",
            "   macro avg       0.50      0.50      0.50      8505\n",
            "weighted avg       0.54      0.56      0.54      8505\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZDNUeenK5VPh",
        "outputId": "57267a99-1ca2-438e-d88b-16f6d9f9208d"
      },
      "source": [
        "# Совпадение словоформы (same_form) + Random Balanced Choice\r\n",
        "\r\n",
        "solver = RusseSolver(path='data/RUSSE/train.jsonl',\r\n",
        "                     path_valid='data/RUSSE/val.jsonl')\r\n",
        "solver.heruistic_combined(final_decision=solver.random_balanced_choice)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "       False       0.63      0.82      0.71      5366\n",
            "        True       0.37      0.17      0.24      3139\n",
            "\n",
            "    accuracy                           0.58      8505\n",
            "   macro avg       0.50      0.50      0.48      8505\n",
            "weighted avg       0.53      0.58      0.54      8505\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1CmKdOV66Gbt"
      },
      "source": [
        "# Samples with questionable markup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G9_WiR5J8oRz"
      },
      "source": [
        "# 17 vs 22\r\n",
        "# 32 vs 21"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 227
        },
        "id": "JCice7dOEqnu",
        "outputId": "1e1100db-3b31-4fe5-cb7f-5ba1d61963dc"
      },
      "source": [
        "train.iloc[[32, 21, 94]][['word', 'sentence1', 'sentence2', 'label']]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>word</th>\n",
              "      <th>sentence1</th>\n",
              "      <th>sentence2</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>должник</td>\n",
              "      <td>Должники или попросту прятались, выжидая, или исподволь скупали обесценивающиеся банковские векселя и всучивали их вместо живых денег</td>\n",
              "      <td>Наконец, если вы считаете, что я ваш должник, я готов расплатиться</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>должник</td>\n",
              "      <td>От ее слов у меня теплело на душе, и так хотелось ей в чем-то помочь, я до сих пор ощущал себя ее должником</td>\n",
              "      <td>Список должников</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>94</th>\n",
              "      <td>должник</td>\n",
              "      <td>Спасибо, считай, что теперь я твой должник</td>\n",
              "      <td>Без залога любой должник может воспользоваться временем, отведенным для апелляции, и просто удрать</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       word  ...  label\n",
              "32  должник  ...   True\n",
              "21  должник  ...  False\n",
              "94  должник  ...  False\n",
              "\n",
              "[3 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 179
        },
        "id": "KLX-CFzpEZIj",
        "outputId": "5588b90b-1080-4ccf-e531-62a3d337d485"
      },
      "source": [
        "train.iloc[[17, 22]][['word', 'sentence1', 'sentence2', 'label']]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>word</th>\n",
              "      <th>sentence1</th>\n",
              "      <th>sentence2</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>закрытие</td>\n",
              "      <td>Закрытие лагерей вылилось в настоящую бойню (иорданская армия уничтожила тысячи палестинцев) и вошло в историю палестинского народа под названием «черный сентябрь»</td>\n",
              "      <td>Закрытие файла</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>запуск</td>\n",
              "      <td>Строительство и запуск отеля занимает до двух лет и больше</td>\n",
              "      <td>Считается, что автомобиль после запуска двигателя должен сразу стартовать</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        word  ...  label\n",
              "17  закрытие  ...  False\n",
              "22    запуск  ...   True\n",
              "\n",
              "[2 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 881
        },
        "id": "HfUucRknEJ6T",
        "outputId": "f16c85ae-92c0-48cb-82a7-9f1ea0cece8f"
      },
      "source": [
        "train.iloc[[12, 15, 18, 29, 35, 56, 62, 66, 75, 93, 94, 98, 118]][['word', 'sentence1', 'sentence2', 'label']]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>word</th>\n",
              "      <th>sentence1</th>\n",
              "      <th>sentence2</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>деление</td>\n",
              "      <td>Районирование – это деление территории на районы</td>\n",
              "      <td>Современная наука и техника дают возможность превращать энергию деления в тепло и электричество</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>земля</td>\n",
              "      <td>Теперь до земли можно добраться и пешком, если льды окончательно преградят нам плавание</td>\n",
              "      <td>Гений – это как земля, а таланты – ее использующие пахари</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>задание</td>\n",
              "      <td>Муродали был предельно собран, как будто отправлял бойца на задание</td>\n",
              "      <td>Первым заданием по черчению, которого он, конечно, не сдал, был кран</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>душ</td>\n",
              "      <td>Гарднер был в новом, очень хорошем штатском костюме, выбритый, надушенный, и поэтому чувствовал невероятную легкость во всем, точно после хорошего душа</td>\n",
              "      <td>Пойти в душ</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>зуд</td>\n",
              "      <td>У чиновников снова реформаторский зуд</td>\n",
              "      <td>Нестерпимый зуд любопытства</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>запятая</td>\n",
              "      <td>Не знаю отчего, но чуть не с семилетнего возраста я писал без единой ошибки самые дремучие фразы. В запятых не ошибался никогда</td>\n",
              "      <td>Снова запятая!</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>62</th>\n",
              "      <td>дом</td>\n",
              "      <td>Дом – гнездо семьи, средоточие традиций и вместе с тем мое владение. Владение в принципе не отчуждаемое – как жизнь, как душа</td>\n",
              "      <td>Третий дом показывает, как человек учится, общается, каким образом добывает информацию об окружающем мире</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>зарядка</td>\n",
              "      <td>Не горящая при включении зажигания лампочка зарядки аккумулятора говорит о проблемах с генератором</td>\n",
              "      <td>Переносная зарядка</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75</th>\n",
              "      <td>закрытие</td>\n",
              "      <td>У «Пежо» на водительской двери большую ручку сменила узкая выемка в подлокотнике, что сделало закрытие двери не самым привычным</td>\n",
              "      <td>Компьютер при следующем входе в программу выдавал табличку: «Программа закрыта с ошибкой. Повторите ввод». Он также фиксировал время закрытия программы</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>дорожка</td>\n",
              "      <td>А совсем у воды тянется еще одна пешеходная дорожка</td>\n",
              "      <td>Но внутри все было как в дорогом отеле: светлые лестницы, красные дорожки на них, распахнутые окна, холлы и даже пальмы</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>94</th>\n",
              "      <td>должник</td>\n",
              "      <td>Спасибо, считай, что теперь я твой должник</td>\n",
              "      <td>Без залога любой должник может воспользоваться временем, отведенным для апелляции, и просто удрать</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>здоровье</td>\n",
              "      <td>Не даст господь здоровья, так никакие доктора тебе не помогут</td>\n",
              "      <td>О здоровье человека можно судить по состоянию его кожи</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>118</th>\n",
              "      <td>ерунда</td>\n",
              "      <td>Людмила […] смотрела на меня со смешанным выражением непонимания и возмущения: словно требуя, чтобы я объяснил ей, что же, черт возьми, происходит. «Видишь, какая ерунда», – сказал я, чтобы не молчать</td>\n",
              "      <td>Переживать из-за всякой ерунды</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         word  ...  label\n",
              "12    деление  ...  False\n",
              "15      земля  ...   True\n",
              "18    задание  ...  False\n",
              "29        душ  ...  False\n",
              "35        зуд  ...   True\n",
              "56    запятая  ...  False\n",
              "62        дом  ...  False\n",
              "66    зарядка  ...   True\n",
              "75   закрытие  ...  False\n",
              "93    дорожка  ...   True\n",
              "94    должник  ...  False\n",
              "98   здоровье  ...  False\n",
              "118    ерунда  ...  False\n",
              "\n",
              "[13 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    }
  ]
}