public_PLDI22AE_Poe_make_benchmark.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "public-PLDI22AE-Poe-make-benchmark",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/chyanju/Poe/blob/main/public_PLDI22AE_Poe_make_benchmark.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AZGoIr5tvdIx"
      },
      "source": [
        "# Poe: Customize your own benchmark and run the tool.\n",
        "This is a colab notebook that guides you to run through the full pipeline with customized benchmark.\n",
        "\n",
        "This colab is modified from the TaPas public notebooks. See [https://github.com/google-research/tapas/tree/master/notebooks](https://github.com/google-research/tapas/tree/master/notebooks).\n",
        "\n",
        "Table of Contents:\n",
        "1. Set up the environment\n",
        "2. Customize your own benchmark here\n",
        "3. Prepare the machine learning model\n",
        "4. Use the machine learning model to make initial predictions of your customized benchmark\n",
        "5. Construct new dataset for Poe by using the results generated so far\n",
        "6. Run Poe here on your customized benchmark"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y-m_JoVCFCV0"
      },
      "source": [
        "# 1. Set up the environment.\n",
        "(Note) All commands in this section should only be run once during the same runtime.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "First we install dependencies. It's OK to see an error that says: \"ERROR: pip's dependency resolver does not currently take into account all the packages that are installed.\". Just move on."
      ],
      "metadata": {
        "id": "V3ewFNEIGWil"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "uI6zyIM20Kw4",
        "outputId": "e58755e6-c76d-4d5d-8d0d-d28dd7ffd48b"
      },
      "source": [
        "! pip install -U pip setuptools wheel\n",
        "! pip install simplejson==3.17.5\n",
        "! pip install xxhash==2.0.2\n",
        "! pip install nltk==3.6.1\n",
        "! pip install sexpdata==0.0.3\n",
        "! pip install tabulate==0.8.9\n",
        "! pip install pandas==1.4.0\n",
        "! pip install spacy==3.1.2\n",
        "! python -m spacy download en_core_web_sm\n",
        "! pip install tapas-table-parsing==0.0.1.dev0"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: pip in /usr/local/lib/python3.7/dist-packages (21.1.3)\n",
            "Collecting pip\n",
            "  Downloading pip-22.0.4-py3-none-any.whl (2.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.1 MB 5.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (57.4.0)\n",
            "Collecting setuptools\n",
            "  Downloading setuptools-62.0.0-py3-none-any.whl (1.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1 MB 44.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: wheel in /usr/local/lib/python3.7/dist-packages (0.37.1)\n",
            "Installing collected packages: setuptools, pip\n",
            "  Attempting uninstall: setuptools\n",
            "    Found existing installation: setuptools 57.4.0\n",
            "    Uninstalling setuptools-57.4.0:\n",
            "      Successfully uninstalled setuptools-57.4.0\n",
            "  Attempting uninstall: pip\n",
            "    Found existing installation: pip 21.1.3\n",
            "    Uninstalling pip-21.1.3:\n",
            "      Successfully uninstalled pip-21.1.3\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "tensorflow 2.8.0 requires tf-estimator-nightly==2.8.0.dev2021122109, which is not installed.\n",
            "datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "Successfully installed pip-22.0.4 setuptools-62.0.0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "pkg_resources"
                ]
              }
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting simplejson==3.17.5\n",
            "  Downloading simplejson-3.17.5-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (129 kB)\n",
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/129.9 KB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.9/129.9 KB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: simplejson\n",
            "Successfully installed simplejson-3.17.5\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0mCollecting xxhash==2.0.2\n",
            "  Downloading xxhash-2.0.2-cp37-cp37m-manylinux2010_x86_64.whl (243 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m243.9/243.9 KB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: xxhash\n",
            "Successfully installed xxhash-2.0.2\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0mCollecting nltk==3.6.1\n",
            "  Downloading nltk-3.6.1-py3-none-any.whl (1.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m19.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from nltk==3.6.1) (4.63.0)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from nltk==3.6.1) (1.1.0)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from nltk==3.6.1) (2019.12.20)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from nltk==3.6.1) (7.1.2)\n",
            "Installing collected packages: nltk\n",
            "  Attempting uninstall: nltk\n",
            "    Found existing installation: nltk 3.2.5\n",
            "    Uninstalling nltk-3.2.5:\n",
            "      Successfully uninstalled nltk-3.2.5\n",
            "Successfully installed nltk-3.6.1\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0mCollecting sexpdata==0.0.3\n",
            "  Downloading sexpdata-0.0.3.tar.gz (6.3 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Building wheels for collected packages: sexpdata\n",
            "  Building wheel for sexpdata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sexpdata: filename=sexpdata-0.0.3-py3-none-any.whl size=7206 sha256=a11e4dbdf58607c0f50003925121d1d859a268d5e9eed0ec48843df9b02a6f71\n",
            "  Stored in directory: /root/.cache/pip/wheels/56/e1/83/f748d65656a82892415c360a9edcb5ef64f4c64750240174f6\n",
            "Successfully built sexpdata\n",
            "Installing collected packages: sexpdata\n",
            "Successfully installed sexpdata-0.0.3\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0mRequirement already satisfied: tabulate==0.8.9 in /usr/local/lib/python3.7/dist-packages (0.8.9)\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m\u001b[31mERROR: Could not find a version that satisfies the requirement pandas==1.4.0 (from versions: 0.1, 0.2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.21.1, 0.22.0, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0, 0.24.1, 0.24.2, 0.25.0, 0.25.1, 0.25.2, 0.25.3, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5)\u001b[0m\u001b[31m\n",
            "\u001b[0m\u001b[31mERROR: No matching distribution found for pandas==1.4.0\u001b[0m\u001b[31m\n",
            "\u001b[0mCollecting spacy==3.1.2\n",
            "  Downloading spacy-3.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.8/5.8 MB\u001b[0m \u001b[31m43.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting catalogue<2.1.0,>=2.0.4\n",
            "  Downloading catalogue-2.0.7-py3-none-any.whl (17 kB)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.1.2) (1.21.5)\n",
            "Collecting thinc<8.1.0,>=8.0.8\n",
            "  Downloading thinc-8.0.15-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (653 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m653.3/653.3 KB\u001b[0m \u001b[31m42.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy==3.1.2) (2.0.6)\n",
            "Collecting srsly<3.0.0,>=2.4.1\n",
            "  Downloading srsly-2.4.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (451 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m452.0/452.0 KB\u001b[0m \u001b[31m33.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: wasabi<1.1.0,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from spacy==3.1.2) (0.9.0)\n",
            "Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy==3.1.2) (3.10.0.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy==3.1.2) (62.0.0)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.1.2) (1.0.6)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.1.2) (0.4.1)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.1.2) (21.3)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.1.2) (2.23.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy==3.1.2) (4.63.0)\n",
            "Collecting pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4\n",
            "  Downloading pydantic-1.8.2-cp37-cp37m-manylinux2014_x86_64.whl (10.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.1/10.1 MB\u001b[0m \u001b[31m74.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy==3.1.2) (3.0.6)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from spacy==3.1.2) (2.11.3)\n",
            "Collecting typer<0.4.0,>=0.3.0\n",
            "  Downloading typer-0.3.2-py3-none-any.whl (21 kB)\n",
            "Collecting pathy>=0.3.5\n",
            "  Downloading pathy-0.6.1-py3-none-any.whl (42 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.8/42.8 KB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting spacy-legacy<3.1.0,>=3.0.7\n",
            "  Downloading spacy_legacy-3.0.9-py2.py3-none-any.whl (20 kB)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from catalogue<2.1.0,>=2.0.4->spacy==3.1.2) (3.7.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->spacy==3.1.2) (3.0.7)\n",
            "Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /usr/local/lib/python3.7/dist-packages (from pathy>=0.3.5->spacy==3.1.2) (5.2.1)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==3.1.2) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==3.1.2) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==3.1.2) (2021.10.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy==3.1.2) (2.10)\n",
            "Requirement already satisfied: click<7.2.0,>=7.1.1 in /usr/local/lib/python3.7/dist-packages (from typer<0.4.0,>=0.3.0->spacy==3.1.2) (7.1.2)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->spacy==3.1.2) (2.0.1)\n",
            "Installing collected packages: typer, spacy-legacy, pydantic, catalogue, srsly, pathy, thinc, spacy\n",
            "  Attempting uninstall: catalogue\n",
            "    Found existing installation: catalogue 1.0.0\n",
            "    Uninstalling catalogue-1.0.0:\n",
            "      Successfully uninstalled catalogue-1.0.0\n",
            "  Attempting uninstall: srsly\n",
            "    Found existing installation: srsly 1.0.5\n",
            "    Uninstalling srsly-1.0.5:\n",
            "      Successfully uninstalled srsly-1.0.5\n",
            "  Attempting uninstall: thinc\n",
            "    Found existing installation: thinc 7.4.0\n",
            "    Uninstalling thinc-7.4.0:\n",
            "      Successfully uninstalled thinc-7.4.0\n",
            "  Attempting uninstall: spacy\n",
            "    Found existing installation: spacy 2.2.4\n",
            "    Uninstalling spacy-2.2.4:\n",
            "      Successfully uninstalled spacy-2.2.4\n",
            "Successfully installed catalogue-2.0.7 pathy-0.6.1 pydantic-1.8.2 spacy-3.1.2 spacy-legacy-3.0.9 srsly-2.4.2 thinc-8.0.15 typer-0.3.2\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0mCollecting en-core-web-sm==3.1.0\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.1.0/en_core_web_sm-3.1.0-py3-none-any.whl (13.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.6/13.6 MB\u001b[0m \u001b[31m44.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: spacy<3.2.0,>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from en-core-web-sm==3.1.0) (3.1.2)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.7 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.0.9)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.11.3)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.23.0)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.21.5)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.0.6)\n",
            "Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.10.0.2)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.0.6)\n",
            "Requirement already satisfied: typer<0.4.0,>=0.3.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (0.3.2)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.4.2)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (0.6.1)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.8.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (21.3)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (0.4.1)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (62.0.0)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.0.6)\n",
            "Requirement already satisfied: thinc<8.1.0,>=8.0.8 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (8.0.15)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (0.9.0)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.4 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.0.7)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (4.63.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from catalogue<2.1.0,>=2.0.4->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.7.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.0.7)\n",
            "Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /usr/local/lib/python3.7/dist-packages (from pathy>=0.3.5->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (5.2.1)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2021.10.8)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.24.3)\n",
            "Requirement already satisfied: click<7.2.0,>=7.1.1 in /usr/local/lib/python3.7/dist-packages (from typer<0.4.0,>=0.3.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (7.1.2)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.0.1)\n",
            "Installing collected packages: en-core-web-sm\n",
            "  Attempting uninstall: en-core-web-sm\n",
            "    Found existing installation: en-core-web-sm 2.2.5\n",
            "    Uninstalling en-core-web-sm-2.2.5:\n",
            "      Successfully uninstalled en-core-web-sm-2.2.5\n",
            "Successfully installed en-core-web-sm-3.1.0\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('en_core_web_sm')\n",
            "Collecting tapas-table-parsing==0.0.1.dev0\n",
            "  Downloading tapas_table_parsing-0.0.1.dev0-py3-none-any.whl (195 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m195.6/195.6 KB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tensorflow-probability==0.10.1\n",
            "  Downloading tensorflow_probability-0.10.1-py2.py3-none-any.whl (3.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.5/3.5 MB\u001b[0m \u001b[31m47.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting apache-beam[gcp]==2.20.0\n",
            "  Downloading apache_beam-2.20.0-cp37-cp37m-manylinux1_x86_64.whl (3.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.5/3.5 MB\u001b[0m \u001b[31m61.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tf-slim~=1.1.0\n",
            "  Downloading tf_slim-1.1.0-py2.py3-none-any.whl (352 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m352.1/352.1 KB\u001b[0m \u001b[31m27.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pandas~=1.0.0\n",
            "  Downloading pandas-1.0.5-cp37-cp37m-manylinux1_x86_64.whl (10.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.1/10.1 MB\u001b[0m \u001b[31m74.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting scikit-learn~=0.22.1\n",
            "  Downloading scikit_learn-0.22.2.post1-cp37-cp37m-manylinux1_x86_64.whl (7.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m81.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tensorflow~=2.2.0\n",
            "  Downloading tensorflow-2.2.3-cp37-cp37m-manylinux2010_x86_64.whl (516.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m516.4/516.4 MB\u001b[0m \u001b[31m2.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tf-models-official~=2.2.0\n",
            "  Downloading tf_models_official-2.2.2-py2.py3-none-any.whl (711 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m711.5/711.5 KB\u001b[0m \u001b[31m34.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: nltk~=3.5 in /usr/local/lib/python3.7/dist-packages (from tapas-table-parsing==0.0.1.dev0) (3.6.1)\n",
            "Collecting kaggle<1.5.8\n",
            "  Downloading kaggle-1.5.6.tar.gz (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.2/58.2 KB\u001b[0m \u001b[31m5.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting frozendict==1.2\n",
            "  Downloading frozendict-1.2.tar.gz (2.6 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting avro-python3!=1.9.2,<1.10.0,>=1.8.1\n",
            "  Downloading avro-python3-1.9.2.1.tar.gz (37 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: grpcio<2,>=1.12.1 in /usr/local/lib/python3.7/dist-packages (from apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (1.44.0)\n",
            "Requirement already satisfied: crcmod<2.0,>=1.7 in /usr/local/lib/python3.7/dist-packages (from apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (1.7)\n",
            "Collecting httplib2<=0.12.0,>=0.8\n",
            "  Downloading httplib2-0.12.0.tar.gz (218 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m218.4/218.4 KB\u001b[0m \u001b[31m20.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting pymongo<4.0.0,>=3.8.0\n",
            "  Downloading pymongo-3.12.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (508 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m508.1/508.1 KB\u001b[0m \u001b[31m20.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pydot<2,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (1.3.0)\n",
            "Requirement already satisfied: pytz>=2018.3 in /usr/local/lib/python3.7/dist-packages (from apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (2018.9)\n",
            "Collecting mock<3.0.0,>=1.0.1\n",
            "  Downloading mock-2.0.0-py2.py3-none-any.whl (56 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.8/56.8 KB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: protobuf<4,>=3.5.0.post1 in /usr/local/lib/python3.7/dist-packages (from apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (3.17.3)\n",
            "Collecting typing-extensions<3.8.0,>=3.7.0\n",
            "  Downloading typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)\n",
            "Collecting oauth2client<4,>=2.0.1\n",
            "  Downloading oauth2client-3.0.0.tar.gz (77 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.2/77.2 KB\u001b[0m \u001b[31m8.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: future<1.0.0,>=0.16.0 in /usr/local/lib/python3.7/dist-packages (from apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (0.16.0)\n",
            "Collecting hdfs<3.0.0,>=2.1.0\n",
            "  Downloading hdfs-2.7.0-py3-none-any.whl (34 kB)\n",
            "Requirement already satisfied: python-dateutil<3,>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (2.8.2)\n",
            "Requirement already satisfied: numpy<2,>=1.14.3 in /usr/local/lib/python3.7/dist-packages (from apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (1.21.5)\n",
            "Collecting dill<0.3.2,>=0.3.1.1\n",
            "  Downloading dill-0.3.1.1.tar.gz (151 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.0/152.0 KB\u001b[0m \u001b[31m16.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting fastavro<0.22,>=0.21.4\n",
            "  Downloading fastavro-0.21.24-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m33.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pyarrow<0.17.0,>=0.15.1\n",
            "  Downloading pyarrow-0.16.0-cp37-cp37m-manylinux2014_x86_64.whl (63.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m63.1/63.1 MB\u001b[0m \u001b[31m9.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-cloud-dlp<=0.13.0,>=0.12.0\n",
            "  Downloading google_cloud_dlp-0.13.0-py2.py3-none-any.whl (151 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m151.8/151.8 KB\u001b[0m \u001b[31m13.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-cloud-bigtable<1.1.0,>=0.31.1\n",
            "  Downloading google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m232.1/232.1 KB\u001b[0m \u001b[31m15.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting cachetools<4,>=3.1.0\n",
            "  Downloading cachetools-3.1.1-py2.py3-none-any.whl (11 kB)\n",
            "Collecting google-cloud-datastore<1.8.0,>=1.7.1\n",
            "  Downloading google_cloud_datastore-1.7.4-py2.py3-none-any.whl (82 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m82.7/82.7 KB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-apitools<0.5.29,>=0.5.28\n",
            "  Downloading google-apitools-0.5.28.tar.gz (172 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m172.3/172.3 KB\u001b[0m \u001b[31m16.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting google-cloud-videointelligence<1.14.0,>=1.8.0\n",
            "  Downloading google_cloud_videointelligence-1.13.0-py2.py3-none-any.whl (177 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m177.7/177.7 KB\u001b[0m \u001b[31m17.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-cloud-spanner<1.14.0,>=1.13.0\n",
            "  Downloading google_cloud_spanner-1.13.0-py2.py3-none-any.whl (212 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m213.0/213.0 KB\u001b[0m \u001b[31m12.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-cloud-language<2,>=1.3.0\n",
            "  Downloading google_cloud_language-1.3.1-py2.py3-none-any.whl (83 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m83.6/83.6 KB\u001b[0m \u001b[31m9.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-cloud-pubsub<1.1.0,>=0.39.0\n",
            "  Downloading google_cloud_pubsub-1.0.2-py2.py3-none-any.whl (118 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m118.7/118.7 KB\u001b[0m \u001b[31m12.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting grpcio-gcp<1,>=0.2.2\n",
            "  Downloading grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)\n",
            "Requirement already satisfied: google-cloud-bigquery<=1.24.0,>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (1.21.0)\n",
            "Collecting google-cloud-vision<0.43.0,>=0.38.0\n",
            "  Downloading google_cloud_vision-0.42.0-py2.py3-none-any.whl (435 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m435.0/435.0 KB\u001b[0m \u001b[31m24.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: google-cloud-core<2,>=0.28.1 in /usr/local/lib/python3.7/dist-packages (from apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (1.0.3)\n",
            "Requirement already satisfied: gast>=0.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability==0.10.1->tapas-table-parsing==0.0.1.dev0) (0.5.3)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability==0.10.1->tapas-table-parsing==0.0.1.dev0) (1.15.0)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability==0.10.1->tapas-table-parsing==0.0.1.dev0) (4.4.2)\n",
            "Requirement already satisfied: cloudpickle==1.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability==0.10.1->tapas-table-parsing==0.0.1.dev0) (1.3.0)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from kaggle<1.5.8->tapas-table-parsing==0.0.1.dev0) (1.24.3)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from kaggle<1.5.8->tapas-table-parsing==0.0.1.dev0) (2021.10.8)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from kaggle<1.5.8->tapas-table-parsing==0.0.1.dev0) (2.23.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from kaggle<1.5.8->tapas-table-parsing==0.0.1.dev0) (4.63.0)\n",
            "Requirement already satisfied: python-slugify in /usr/local/lib/python3.7/dist-packages (from kaggle<1.5.8->tapas-table-parsing==0.0.1.dev0) (6.1.1)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from nltk~=3.5->tapas-table-parsing==0.0.1.dev0) (1.1.0)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from nltk~=3.5->tapas-table-parsing==0.0.1.dev0) (2019.12.20)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from nltk~=3.5->tapas-table-parsing==0.0.1.dev0) (7.1.2)\n",
            "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn~=0.22.1->tapas-table-parsing==0.0.1.dev0) (1.4.1)\n",
            "Collecting h5py<2.11.0,>=2.10.0\n",
            "  Downloading h5py-2.10.0-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.9/2.9 MB\u001b[0m \u001b[31m62.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.0.0)\n",
            "Collecting tensorboard<2.3.0,>=2.2.0\n",
            "  Downloading tensorboard-2.2.2-py3-none-any.whl (3.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m64.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.14.0)\n",
            "Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.6.3)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.1.2)\n",
            "Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (0.2.0)\n",
            "Collecting numpy<2,>=1.14.3\n",
            "  Downloading numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl (20.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.1/20.1 MB\u001b[0m \u001b[31m46.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (3.3.0)\n",
            "Collecting tensorflow-estimator<2.3.0,>=2.2.0\n",
            "  Downloading tensorflow_estimator-2.2.0-py2.py3-none-any.whl (454 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m454.6/454.6 KB\u001b[0m \u001b[31m20.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (0.37.1)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.1.0)\n",
            "Collecting gast>=0.3.2\n",
            "  Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)\n",
            "Collecting mlperf-compliance==0.0.10\n",
            "  Downloading mlperf_compliance-0.0.10-py3-none-any.whl (24 kB)\n",
            "Collecting opencv-python-headless\n",
            "  Downloading opencv_python_headless-4.5.5.64-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (47.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m47.8/47.8 MB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting typing==3.7.4.1\n",
            "  Downloading typing-3.7.4.1-py3-none-any.whl (25 kB)\n",
            "Requirement already satisfied: Pillow in /usr/local/lib/python3.7/dist-packages (from tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (7.1.2)\n",
            "Requirement already satisfied: google-api-python-client>=1.6.7 in /usr/local/lib/python3.7/dist-packages (from tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.12.11)\n",
            "Collecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.96-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m40.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tensorflow-model-optimization>=0.2.1\n",
            "  Downloading tensorflow_model_optimization-0.7.2-py2.py3-none-any.whl (237 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m237.6/237.6 KB\u001b[0m \u001b[31m15.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tensorflow-datasets in /usr/local/lib/python3.7/dist-packages (from tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (4.0.1)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (3.13)\n",
            "Collecting tensorflow-addons\n",
            "  Downloading tensorflow_addons-0.16.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m30.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: Cython in /usr/local/lib/python3.7/dist-packages (from tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (0.29.28)\n",
            "Collecting py-cpuinfo>=3.3.0\n",
            "  Downloading py-cpuinfo-8.0.0.tar.gz (99 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.8/99.8 KB\u001b[0m \u001b[31m10.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: tensorflow-hub>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (0.12.0)\n",
            "Requirement already satisfied: gin-config in /usr/local/lib/python3.7/dist-packages (from tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (0.5.0)\n",
            "Requirement already satisfied: psutil>=5.4.3 in /usr/local/lib/python3.7/dist-packages (from tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (5.4.8)\n",
            "Collecting dataclasses\n",
            "  Downloading dataclasses-0.6-py3-none-any.whl (14 kB)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (3.2.2)\n",
            "Requirement already satisfied: google-api-core<3dev,>=1.21.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.26.3)\n",
            "Collecting google-api-python-client>=1.6.7\n",
            "  Downloading google_api_python_client-2.43.0-py2.py3-none-any.whl (8.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.3/8.3 MB\u001b[0m \u001b[31m71.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-auth-httplib2>=0.1.0\n",
            "  Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)\n",
            "Collecting google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5\n",
            "  Downloading google_api_core-2.7.1-py3-none-any.whl (114 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.7/114.7 KB\u001b[0m \u001b[31m10.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-api-python-client>=1.6.7\n",
            "  Downloading google_api_python_client-2.42.0-py2.py3-none-any.whl (8.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.3/8.3 MB\u001b[0m \u001b[31m75.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.41.0-py2.py3-none-any.whl (8.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.3/8.3 MB\u001b[0m \u001b[31m69.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.40.0-py2.py3-none-any.whl (8.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.2/8.2 MB\u001b[0m \u001b[31m70.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.39.0-py2.py3-none-any.whl (8.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.2/8.2 MB\u001b[0m \u001b[31m69.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.38.0-py2.py3-none-any.whl (8.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.2/8.2 MB\u001b[0m \u001b[31m62.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.37.0-py2.py3-none-any.whl (8.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.1/8.1 MB\u001b[0m \u001b[31m70.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.36.0-py2.py3-none-any.whl (8.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.0/8.0 MB\u001b[0m \u001b[31m67.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.35.0-py2.py3-none-any.whl (8.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.0/8.0 MB\u001b[0m \u001b[31m66.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.34.0-py2.py3-none-any.whl (7.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.9/7.9 MB\u001b[0m \u001b[31m69.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.33.0-py2.py3-none-any.whl (7.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.9/7.9 MB\u001b[0m \u001b[31m65.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.32.0-py2.py3-none-any.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m18.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.31.0-py2.py3-none-any.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m70.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.30.0-py2.py3-none-any.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m67.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.29.0-py2.py3-none-any.whl (7.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.7/7.7 MB\u001b[0m \u001b[31m68.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.28.0-py2.py3-none-any.whl (7.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.7/7.7 MB\u001b[0m \u001b[31m66.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.27.0-py2.py3-none-any.whl (7.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.7/7.7 MB\u001b[0m \u001b[31m70.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.26.1-py2.py3-none-any.whl (7.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.6/7.6 MB\u001b[0m \u001b[31m70.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.26.0-py2.py3-none-any.whl (7.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.6/7.6 MB\u001b[0m \u001b[31m73.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.25.0-py2.py3-none-any.whl (7.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.5/7.5 MB\u001b[0m \u001b[31m65.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.24.0-py2.py3-none-any.whl (7.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.5/7.5 MB\u001b[0m \u001b[31m66.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.23.0-py2.py3-none-any.whl (7.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.5/7.5 MB\u001b[0m \u001b[31m72.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.22.0-py2.py3-none-any.whl (7.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.5/7.5 MB\u001b[0m \u001b[31m71.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.21.0-py2.py3-none-any.whl (7.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.5/7.5 MB\u001b[0m \u001b[31m66.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.20.0-py2.py3-none-any.whl (7.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m16.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.19.1-py2.py3-none-any.whl (7.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m63.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.19.0-py2.py3-none-any.whl (7.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m64.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.18.0-py2.py3-none-any.whl (7.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m72.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.17.0-py2.py3-none-any.whl (7.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.3/7.3 MB\u001b[0m \u001b[31m70.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.16.0-py2.py3-none-any.whl (7.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.3/7.3 MB\u001b[0m \u001b[31m73.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.15.0-py2.py3-none-any.whl (7.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.2/7.2 MB\u001b[0m \u001b[31m69.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.14.1-py2.py3-none-any.whl (7.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m72.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.14.0-py2.py3-none-any.whl (7.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m63.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.13.0-py2.py3-none-any.whl (7.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m76.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.12.0-py2.py3-none-any.whl (7.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m72.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.11.0-py2.py3-none-any.whl (7.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.0/7.0 MB\u001b[0m \u001b[31m64.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.10.0-py2.py3-none-any.whl (7.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.0/7.0 MB\u001b[0m \u001b[31m70.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.9.0-py2.py3-none-any.whl (7.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.0/7.0 MB\u001b[0m \u001b[31m72.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.8.0-py2.py3-none-any.whl (7.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.0/7.0 MB\u001b[0m \u001b[31m12.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.7.0-py2.py3-none-any.whl (7.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.3/7.3 MB\u001b[0m \u001b[31m62.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.6.0-py2.py3-none-any.whl (7.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.2/7.2 MB\u001b[0m \u001b[31m75.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.5.0-py2.py3-none-any.whl (7.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m67.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.4.0-py2.py3-none-any.whl (7.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m75.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.3.0-py2.py3-none-any.whl (7.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m75.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.2.0-py2.py3-none-any.whl (7.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.0/7.0 MB\u001b[0m \u001b[31m71.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.1.0-py2.py3-none-any.whl (6.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.6/6.6 MB\u001b[0m \u001b[31m71.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-2.0.2-py2.py3-none-any.whl (6.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.5/6.5 MB\u001b[0m \u001b[31m68.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-1.12.10-py2.py3-none-any.whl (61 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.5/61.5 KB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-1.12.8-py2.py3-none-any.whl (61 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.5/61.5 KB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: google-auth>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.35.0)\n",
            "  Downloading google_api_python_client-1.12.7-py2.py3-none-any.whl (61 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.5/61.5 KB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-1.12.6-py2.py3-none-any.whl (61 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.5/61.5 KB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-1.12.5-py2.py3-none-any.whl (61 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.4/61.4 KB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-1.12.4-py2.py3-none-any.whl (61 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.3/61.3 KB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-1.12.3-py2.py3-none-any.whl (61 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.3/61.3 KB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading google_api_python_client-1.12.2-py2.py3-none-any.whl (61 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.3/61.3 KB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: google-auth-httplib2>=0.0.3 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (0.0.4)\n",
            "Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (3.0.1)\n",
            "Collecting fasteners>=0.14\n",
            "  Downloading fasteners-0.17.3-py3-none-any.whl (18 kB)\n",
            "Requirement already satisfied: google-resumable-media!=0.4.0,<0.5.0dev,>=0.3.1 in /usr/local/lib/python3.7/dist-packages (from google-cloud-bigquery<=1.24.0,>=1.6.0->apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (0.4.1)\n",
            "Collecting grpc-google-iam-v1<0.13dev,>=0.12.3\n",
            "  Downloading grpc-google-iam-v1-0.12.3.tar.gz (13 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0\n",
            "  Downloading google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.3/93.3 KB\u001b[0m \u001b[31m10.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: docopt in /usr/local/lib/python3.7/dist-packages (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (0.6.2)\n",
            "Collecting pbr>=0.11\n",
            "  Downloading pbr-5.8.1-py2.py3-none-any.whl (113 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m113.0/113.0 KB\u001b[0m \u001b[31m12.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python3.7/dist-packages (from oauth2client<4,>=2.0.1->apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (0.4.8)\n",
            "Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/lib/python3.7/dist-packages (from oauth2client<4,>=2.0.1->apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (0.2.8)\n",
            "Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from oauth2client<4,>=2.0.1->apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (4.8)\n",
            "Requirement already satisfied: pyparsing>=2.1.4 in /usr/local/lib/python3.7/dist-packages (from pydot<2,>=1.2.0->apache-beam[gcp]==2.20.0->tapas-table-parsing==0.0.1.dev0) (3.0.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->kaggle<1.5.8->tapas-table-parsing==0.0.1.dev0) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->kaggle<1.5.8->tapas-table-parsing==0.0.1.dev0) (3.0.4)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (0.4.6)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.8.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.0.1)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (62.0.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (3.3.6)\n",
            "Requirement already satisfied: dm-tree~=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow-model-optimization>=0.2.1->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (0.1.6)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.4.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (0.11.0)\n",
            "Requirement already satisfied: text-unidecode>=1.3 in /usr/local/lib/python3.7/dist-packages (from python-slugify->kaggle<1.5.8->tapas-table-parsing==0.0.1.dev0) (1.3)\n",
            "Requirement already satisfied: typeguard>=2.7 in /usr/local/lib/python3.7/dist-packages (from tensorflow-addons->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (2.7.1)\n",
            "Requirement already satisfied: attrs>=18.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (21.4.0)\n",
            "Requirement already satisfied: tensorflow-metadata in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.7.0)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (5.4.0)\n",
            "Requirement already satisfied: promise in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (2.3)\n",
            "Requirement already satisfied: packaging>=14.3 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (21.3)\n",
            "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.56.0)\n",
            "INFO: pip is looking at multiple versions of google-api-core[grpc,grpcgcp] to determine which version is compatible with other requirements. This could take a while.\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (1.3.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (4.11.3)\n",
            "Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from importlib-resources->tensorflow-datasets->tf-models-official~=2.2.0->tapas-table-parsing==0.0.1.dev0) (3.7.0)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow~=2.2.0->tapas-table-parsing==0.0.1.dev0) (3.2.0)\n",
            "Building wheels for collected packages: frozendict, kaggle, avro-python3, dill, google-apitools, httplib2, oauth2client, py-cpuinfo, grpc-google-iam-v1\n",
            "  Building wheel for frozendict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for frozendict: filename=frozendict-1.2-py3-none-any.whl size=3166 sha256=41c4f2bcbfac41a9f5c06df856e39877bbe0ff8e26dabbddca40e66d2add9cfd\n",
            "  Stored in directory: /root/.cache/pip/wheels/68/17/69/ac196dd181e620bba5fae5488e4fd6366a7316dce13cf88776\n",
            "  Building wheel for kaggle (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for kaggle: filename=kaggle-1.5.6-py3-none-any.whl size=72854 sha256=fdaaf29732c4b3e8a3bde9390fbea39db600c703d5fb3007bb24e806e8f65e6f\n",
            "  Stored in directory: /root/.cache/pip/wheels/aa/e7/e7/eb3c3d514c33294d77ddd5a856bdd58dc9c1fabbed59a02a2b\n",
            "  Building wheel for avro-python3 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for avro-python3: filename=avro_python3-1.9.2.1-py3-none-any.whl size=43513 sha256=71a13329c2d7787ffcb11b88410f6630317c21cdf6e41512dc77a1e21f27a2b5\n",
            "  Stored in directory: /root/.cache/pip/wheels/bc/49/5f/fdb5b9d85055c478213e0158ac122b596816149a02d82e0ab1\n",
            "  Building wheel for dill (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for dill: filename=dill-0.3.1.1-py3-none-any.whl size=78544 sha256=9d97d6331fb68cee207b53d17f47f721fd24ec83b54edc04cf7e07770c2cf593\n",
            "  Stored in directory: /root/.cache/pip/wheels/a4/61/fd/c57e374e580aa78a45ed78d5859b3a44436af17e22ca53284f\n",
            "  Building wheel for google-apitools (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for google-apitools: filename=google_apitools-0.5.28-py3-none-any.whl size=130106 sha256=4412ba1facf96d96136045debb1a51b9feda847d36928713cf03f589e6097ad7\n",
            "  Stored in directory: /root/.cache/pip/wheels/34/3b/69/ecd8e6ae89d9d71102a58962c29faa7a9467ba45f99f205920\n",
            "  Building wheel for httplib2 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for httplib2: filename=httplib2-0.12.0-py3-none-any.whl size=93465 sha256=8beddc6cae5aa18cb376a68b1b6e53d3f08890f202a58b8df0fb3d6d7af3c540\n",
            "  Stored in directory: /root/.cache/pip/wheels/0d/e7/b6/0dd30343ceca921cfbd91f355041bd9c69e0f40b49f25b7b8a\n",
            "  Building wheel for oauth2client (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for oauth2client: filename=oauth2client-3.0.0-py3-none-any.whl size=106375 sha256=92263c9e3ccd405c189c2d5150e6797a43641a839252ea5ac2f7929b3eefbce7\n",
            "  Stored in directory: /root/.cache/pip/wheels/86/73/7a/3b3f76a2142176605ff38fbca574327962c71e25a43197a4c1\n",
            "  Building wheel for py-cpuinfo (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for py-cpuinfo: filename=py_cpuinfo-8.0.0-py3-none-any.whl size=22258 sha256=75f463b3880e7ea5425749fff10b8cab13da6ee0e52c97e3a9c1940180b0d067\n",
            "  Stored in directory: /root/.cache/pip/wheels/d2/f1/1f/041add21dc9c4220157f1bd2bd6afe1f1a49524c3396b94401\n",
            "  Building wheel for grpc-google-iam-v1 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for grpc-google-iam-v1: filename=grpc_google_iam_v1-0.12.3-py3-none-any.whl size=18515 sha256=64e2be02e4f8752612ce98e5e4640af72c6108704cc184010865d4d1df9bd979\n",
            "  Stored in directory: /root/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380\n",
            "Successfully built frozendict kaggle avro-python3 dill google-apitools httplib2 oauth2client py-cpuinfo grpc-google-iam-v1\n",
            "Installing collected packages: typing-extensions, typing, tensorflow-estimator, sentencepiece, py-cpuinfo, mlperf-compliance, httplib2, frozendict, fastavro, dataclasses, cachetools, tensorflow-addons, pymongo, pbr, numpy, gast, fasteners, dill, avro-python3, tf-slim, tensorflow-probability, tensorflow-model-optimization, pyarrow, pandas, opencv-python-headless, oauth2client, mock, kaggle, hdfs, h5py, grpcio-gcp, scikit-learn, google-apitools, google-api-core, apache-beam, tensorboard, grpc-google-iam-v1, google-api-python-client, tensorflow, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, tf-models-official, tapas-table-parsing\n",
            "  Attempting uninstall: typing-extensions\n",
            "    Found existing installation: typing-extensions 3.10.0.2\n",
            "    Uninstalling typing-extensions-3.10.0.2:\n",
            "      Successfully uninstalled typing-extensions-3.10.0.2\n",
            "  Attempting uninstall: tensorflow-estimator\n",
            "    Found existing installation: tensorflow-estimator 2.8.0\n",
            "    Uninstalling tensorflow-estimator-2.8.0:\n",
            "      Successfully uninstalled tensorflow-estimator-2.8.0\n",
            "  Attempting uninstall: httplib2\n",
            "    Found existing installation: httplib2 0.17.4\n",
            "    Uninstalling httplib2-0.17.4:\n",
            "      Successfully uninstalled httplib2-0.17.4\n",
            "  Attempting uninstall: cachetools\n",
            "    Found existing installation: cachetools 4.2.4\n",
            "    Uninstalling cachetools-4.2.4:\n",
            "      Successfully uninstalled cachetools-4.2.4\n",
            "  Attempting uninstall: pymongo\n",
            "    Found existing installation: pymongo 4.0.2\n",
            "    Uninstalling pymongo-4.0.2:\n",
            "      Successfully uninstalled pymongo-4.0.2\n",
            "  Attempting uninstall: numpy\n",
            "    Found existing installation: numpy 1.21.5\n",
            "    Uninstalling numpy-1.21.5:\n",
            "      Successfully uninstalled numpy-1.21.5\n",
            "  Attempting uninstall: gast\n",
            "    Found existing installation: gast 0.5.3\n",
            "    Uninstalling gast-0.5.3:\n",
            "      Successfully uninstalled gast-0.5.3\n",
            "  Attempting uninstall: dill\n",
            "    Found existing installation: dill 0.3.4\n",
            "    Uninstalling dill-0.3.4:\n",
            "      Successfully uninstalled dill-0.3.4\n",
            "  Attempting uninstall: tensorflow-probability\n",
            "    Found existing installation: tensorflow-probability 0.16.0\n",
            "    Uninstalling tensorflow-probability-0.16.0:\n",
            "      Successfully uninstalled tensorflow-probability-0.16.0\n",
            "  Attempting uninstall: pyarrow\n",
            "    Found existing installation: pyarrow 6.0.1\n",
            "    Uninstalling pyarrow-6.0.1:\n",
            "      Successfully uninstalled pyarrow-6.0.1\n",
            "  Attempting uninstall: pandas\n",
            "    Found existing installation: pandas 1.3.5\n",
            "    Uninstalling pandas-1.3.5:\n",
            "      Successfully uninstalled pandas-1.3.5\n",
            "  Attempting uninstall: oauth2client\n",
            "    Found existing installation: oauth2client 4.1.3\n",
            "    Uninstalling oauth2client-4.1.3:\n",
            "      Successfully uninstalled oauth2client-4.1.3\n",
            "  Attempting uninstall: kaggle\n",
            "    Found existing installation: kaggle 1.5.12\n",
            "    Uninstalling kaggle-1.5.12:\n",
            "      Successfully uninstalled kaggle-1.5.12\n",
            "  Attempting uninstall: h5py\n",
            "    Found existing installation: h5py 3.1.0\n",
            "    Uninstalling h5py-3.1.0:\n",
            "      Successfully uninstalled h5py-3.1.0\n",
            "  Attempting uninstall: scikit-learn\n",
            "    Found existing installation: scikit-learn 1.0.2\n",
            "    Uninstalling scikit-learn-1.0.2:\n",
            "      Successfully uninstalled scikit-learn-1.0.2\n",
            "  Attempting uninstall: google-api-core\n",
            "    Found existing installation: google-api-core 1.26.3\n",
            "    Uninstalling google-api-core-1.26.3:\n",
            "      Successfully uninstalled google-api-core-1.26.3\n",
            "  Attempting uninstall: tensorboard\n",
            "    Found existing installation: tensorboard 2.8.0\n",
            "    Uninstalling tensorboard-2.8.0:\n",
            "      Successfully uninstalled tensorboard-2.8.0\n",
            "  Attempting uninstall: google-api-python-client\n",
            "    Found existing installation: google-api-python-client 1.12.11\n",
            "    Uninstalling google-api-python-client-1.12.11:\n",
            "      Successfully uninstalled google-api-python-client-1.12.11\n",
            "  Attempting uninstall: tensorflow\n",
            "    Found existing installation: tensorflow 2.8.0\n",
            "    Uninstalling tensorflow-2.8.0:\n",
            "      Successfully uninstalled tensorflow-2.8.0\n",
            "  Attempting uninstall: google-cloud-language\n",
            "    Found existing installation: google-cloud-language 1.2.0\n",
            "    Uninstalling google-cloud-language-1.2.0:\n",
            "      Successfully uninstalled google-cloud-language-1.2.0\n",
            "  Attempting uninstall: google-cloud-datastore\n",
            "    Found existing installation: google-cloud-datastore 1.8.0\n",
            "    Uninstalling google-cloud-datastore-1.8.0:\n",
            "      Successfully uninstalled google-cloud-datastore-1.8.0\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "yellowbrick 1.4 requires scikit-learn>=1.0.0, but you have scikit-learn 0.22.2.post1 which is incompatible.\n",
            "tables 3.7.0 requires numpy>=1.19.0, but you have numpy 1.18.5 which is incompatible.\n",
            "pymc3 3.11.4 requires cachetools>=4.2.1, but you have cachetools 3.1.1 which is incompatible.\n",
            "pydrive 1.3.1 requires oauth2client>=4.0.0, but you have oauth2client 3.0.0 which is incompatible.\n",
            "multiprocess 0.70.12.2 requires dill>=0.3.4, but you have dill 0.3.1.1 which is incompatible.\n",
            "jaxlib 0.3.2+cuda11.cudnn805 requires numpy>=1.19, but you have numpy 1.18.5 which is incompatible.\n",
            "jax 0.3.4 requires numpy>=1.19, but you have numpy 1.18.5 which is incompatible.\n",
            "imbalanced-learn 0.8.1 requires scikit-learn>=0.24, but you have scikit-learn 0.22.2.post1 which is incompatible.\n",
            "google-colab 1.0.0 requires pandas>=1.1.0; python_version >= \"3.0\", but you have pandas 1.0.5 which is incompatible.\n",
            "datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.\n",
            "albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed apache-beam-2.20.0 avro-python3-1.9.2.1 cachetools-3.1.1 dataclasses-0.6 dill-0.3.1.1 fastavro-0.21.24 fasteners-0.17.3 frozendict-1.2 gast-0.3.3 google-api-core-1.31.5 google-api-python-client-1.12.2 google-apitools-0.5.28 google-cloud-bigtable-1.0.0 google-cloud-datastore-1.7.4 google-cloud-dlp-0.13.0 google-cloud-language-1.3.1 google-cloud-pubsub-1.0.2 google-cloud-spanner-1.13.0 google-cloud-videointelligence-1.13.0 google-cloud-vision-0.42.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 h5py-2.10.0 hdfs-2.7.0 httplib2-0.12.0 kaggle-1.5.6 mlperf-compliance-0.0.10 mock-2.0.0 numpy-1.18.5 oauth2client-3.0.0 opencv-python-headless-4.5.5.64 pandas-1.0.5 pbr-5.8.1 py-cpuinfo-8.0.0 pyarrow-0.16.0 pymongo-3.12.3 scikit-learn-0.22.2.post1 sentencepiece-0.1.96 tapas-table-parsing-0.0.1.dev0 tensorboard-2.2.2 tensorflow-2.2.3 tensorflow-addons-0.16.1 tensorflow-estimator-2.2.0 tensorflow-model-optimization-0.7.2 tensorflow-probability-0.10.1 tf-models-official-2.2.2 tf-slim-1.1.0 typing-3.7.4.1 typing-extensions-3.7.4.3\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1rHkOZhQuVii"
      },
      "source": [
        "## **You need to restart the runtime and resume from here. Click \"Runtime\" -> \"Restart runtime\".**"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following commands clones the Poe public repo and perform necessary set-ups for incorporation of the machine learning model."
      ],
      "metadata": {
        "id": "icCzFsZ9sgIr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! git clone https://github.com/chyanju/Poe.git\n",
        "! cp /content/Poe/benchmarks/VisQA/shared/run_task_main.py /usr/local/lib/python3.7/dist-packages/tapas/"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tYEnF1i-v4jE",
        "outputId": "b79b9dd1-1cc4-4b2e-932d-35493cee1b86"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'Poe'...\n",
            "remote: Enumerating objects: 2804, done.\u001b[K\n",
            "remote: Counting objects: 100% (2804/2804), done.\u001b[K\n",
            "remote: Compressing objects: 100% (850/850), done.\u001b[K\n",
            "remote: Total 2804 (delta 1951), reused 2801 (delta 1951), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (2804/2804), 9.20 MiB | 14.79 MiB/s, done.\n",
            "Resolving deltas: 100% (1951/1951), done.\n",
            "Checking out files: 100% (2751/2751), done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following commands set up nltk library."
      ],
      "metadata": {
        "id": "5GoOzNIuHAOT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('wordnet')\n",
        "nltk.download('omw-1.4')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lQ0q-9Hg-n6Q",
        "outputId": "7bd40413-d022-4aa9-aea2-21249ea6610a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/wordnet.zip.\n",
            "[nltk_data] Downloading package omw-1.4 to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/omw-1.4.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 2. Customize your own benchmark here\n",
        "You only need to modify the \"#----customizable zone----#\""
      ],
      "metadata": {
        "id": "5P8PO9dn4Et4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "import pandas as pd\n",
        "import spacy\n",
        "nlp = spacy.load(\"en_core_web_sm\")\n",
        "data_path = \"/content/\"\n",
        "\n",
        "def format_table(arg_df):\n",
        "  return arg_df.to_markdown(showindex=False,tablefmt=\"jira\",numalign=\"left\").replace(\"||\",\"|\").replace(\"|\\n|\",\"\\n\").strip(\"|\")"
      ],
      "metadata": {
        "id": "dwBTxm0E3hFm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following is the customizable zone.\n",
        "\n",
        "- Construct your own table in Pandas DataFrame and store it in the `customized_table` variable\n",
        "- Provide your own id in `customized_id`\n",
        "- Provide a natural language query in `customized_query`\n",
        "\n",
        "The currently block has some examples filled in for your reference."
      ],
      "metadata": {
        "id": "wkd1jV06HcaD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#----customizable zone----#\n",
        "table_data = [\n",
        "  {'name': 'Jay', 'math': 98, 'physics': 99},\n",
        "  {'name': 'Brian', 'math': 93, 'physics': 100},\n",
        "  {'name': 'Zoe', 'math': 97, 'physics': 95},\n",
        "]\n",
        "customized_table = pd.DataFrame.from_records(table_data)\n",
        "customized_id = \"customized-benchmark-00\"\n",
        "customized_query = \"Who's got the highest score?\""
      ],
      "metadata": {
        "id": "lXWZboNK3hHV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# preview table\n",
        "customized_table"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "IExo3dVd3hJ9",
        "outputId": "b6ed79ee-47a4-48a7-c25a-c99a1f2b6994"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    name  math  physics\n",
              "0    Jay    98       99\n",
              "1  Brian    93      100\n",
              "2    Zoe    97       95"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f590e913-ca33-4c2a-b55f-3eb3ea197c20\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>name</th>\n",
              "      <th>math</th>\n",
              "      <th>physics</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Jay</td>\n",
              "      <td>98</td>\n",
              "      <td>99</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Brian</td>\n",
              "      <td>93</td>\n",
              "      <td>100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Zoe</td>\n",
              "      <td>97</td>\n",
              "      <td>95</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f590e913-ca33-4c2a-b55f-3eb3ea197c20')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-f590e913-ca33-4c2a-b55f-3eb3ea197c20 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-f590e913-ca33-4c2a-b55f-3eb3ea197c20');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# preview query\n",
        "processed_query = \" \".join([str(p).lower() for p in nlp(customized_query)])\n",
        "processed_query"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "D19JXRoN3hMo",
        "outputId": "95adb2aa-5d68-4ae9-dc3e-7dd661bcbf8f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"who 's got the highest score ?\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# construct inputs for every benchmark\n",
        "tapas_inputs = []\n",
        "str_table = format_table(customized_table)\n",
        "tapas_inputs.append((customized_id, processed_query, str_table))\n",
        "visqa_dt = [{\n",
        "    \"id\": customized_id,\n",
        "    \"short_id\": customized_id,\n",
        "    \"query\": processed_query,\n",
        "    \"repr_answer\": None,\n",
        "    \"rendered_table\": customized_table,\n",
        "}]"
      ],
      "metadata": {
        "id": "vPi0Udcw3hPa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with open(\"{}/tapas_on_visqa_inputs.pkl\".format(data_path), \"wb\") as f:\n",
        "    pickle.dump(tapas_inputs, f)"
      ],
      "metadata": {
        "id": "TAZzpV5-3hR3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with open(\"{}/visqa_dataset.pkl\".format(data_path), \"wb\") as f:\n",
        "  pickle.dump(visqa_dt, f)"
      ],
      "metadata": {
        "id": "rG6bhtV43hUm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7We9ofHuFMuk"
      },
      "source": [
        "# 3. Prepare the machine learning model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JfGBVnJroVgF",
        "outputId": "58dd06e6-b1ed-446d-afd5-81e1313fd7aa"
      },
      "source": [
        "import tapas\n",
        "print(tapas.__file__)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/tapas/__init__.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "(Note) The following block only needs to be run once during the same runtime."
      ],
      "metadata": {
        "id": "K6qJpqAFyEH1"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B10C0Yz6gQyD",
        "outputId": "4e2b0794-ffb6-4b46-f520-0a799abdf58d"
      },
      "source": [
        "! gsutil cp \"gs://tapas_models/2020_08_05/tapas_wtq_wikisql_sqa_masklm_medium_reset.zip\" \"tapas_model.zip\" && unzip tapas_model.zip\n",
        "! mv tapas_wtq_wikisql_sqa_masklm_medium_reset tapas_model"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Copying gs://tapas_models/2020_08_05/tapas_wtq_wikisql_sqa_masklm_medium_reset.zip...\n",
            "| [1 files][388.7 MiB/388.7 MiB]                                                \n",
            "Operation completed over 1 objects/388.7 MiB.                                    \n",
            "Archive:  tapas_model.zip\n",
            "   creating: tapas_wtq_wikisql_sqa_masklm_medium_reset/\n",
            "  inflating: tapas_wtq_wikisql_sqa_masklm_medium_reset/bert_config.json  \n",
            "  inflating: tapas_wtq_wikisql_sqa_masklm_medium_reset/README.txt  \n",
            "  inflating: tapas_wtq_wikisql_sqa_masklm_medium_reset/model.ckpt.index  \n",
            "  inflating: tapas_wtq_wikisql_sqa_masklm_medium_reset/model.ckpt.data-00000-of-00001  \n",
            "  inflating: tapas_wtq_wikisql_sqa_masklm_medium_reset/vocab.txt  \n",
            "  inflating: tapas_wtq_wikisql_sqa_masklm_medium_reset/model.ckpt.meta  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E3107bGlGm7d"
      },
      "source": [
        "The following two blocks import the model and necessary libraries."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pnUjDlLqDd3m"
      },
      "source": [
        "import tensorflow.compat.v1 as tf\n",
        "import os \n",
        "import shutil\n",
        "import csv\n",
        "import pandas as pd\n",
        "import IPython\n",
        "\n",
        "tf.get_logger().setLevel('ERROR')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aml6oLFl1dSt"
      },
      "source": [
        "from tapas.utils import tf_example_utils\n",
        "from tapas.protos import interaction_pb2\n",
        "from tapas.utils import number_annotation_utils\n",
        "from tapas.scripts import prediction_utils"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IO0d_wFMy82O"
      },
      "source": [
        "The following two blocks load the checkpoint and prediction functionalities."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UKfxspnVFPsc"
      },
      "source": [
        "os.makedirs('results/wtq/tf_examples', exist_ok=True)\n",
        "os.makedirs('results/wtq/model', exist_ok=True)\n",
        "with open('results/wtq/model/checkpoint', 'w') as f:\n",
        "  f.write('model_checkpoint_path: \"model.ckpt-0\"')\n",
        "for suffix in ['.data-00000-of-00001', '.index', '.meta']:\n",
        "  shutil.copyfile(f'tapas_model/model.ckpt{suffix}', f'results/wtq/model/model.ckpt-0{suffix}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9RlvgDAmCNtP"
      },
      "source": [
        "max_seq_length = 512\n",
        "vocab_file = \"tapas_model/vocab.txt\"\n",
        "config = tf_example_utils.ClassifierConversionConfig(\n",
        "    vocab_file=vocab_file,\n",
        "    max_seq_length=max_seq_length,\n",
        "    max_column_id=max_seq_length,\n",
        "    max_row_id=max_seq_length,\n",
        "    strip_column_names=False,\n",
        "    add_aggregation_candidates=False,\n",
        ")\n",
        "converter = tf_example_utils.ToClassifierTensorflowExample(config)\n",
        "\n",
        "def convert_interactions_to_examples(tables_and_queries):\n",
        "  \"\"\"Calls Tapas converter to convert interaction to example.\"\"\"\n",
        "  for idx, (table, queries) in enumerate(tables_and_queries):\n",
        "    interaction = interaction_pb2.Interaction()\n",
        "    for position, query in enumerate(queries):\n",
        "      question = interaction.questions.add()\n",
        "      question.original_text = query\n",
        "      question.id = f\"{idx}-0_{position}\"\n",
        "    for header in table[0]:\n",
        "      interaction.table.columns.add().text = header\n",
        "    for line in table[1:]:\n",
        "      row = interaction.table.rows.add()\n",
        "      for cell in line:\n",
        "        row.cells.add().text = cell\n",
        "    number_annotation_utils.add_numeric_values(interaction)\n",
        "    for i in range(len(interaction.questions)):\n",
        "      try:\n",
        "        yield converter.convert(interaction, i)\n",
        "      except ValueError as e:\n",
        "        print(f\"Can't convert interaction: {interaction.id} error: {e}\")\n",
        "        \n",
        "def write_tf_example(filename, examples):\n",
        "  with tf.io.TFRecordWriter(filename) as writer:\n",
        "    for example in examples:\n",
        "      writer.write(example.SerializeToString())\n",
        "\n",
        "def aggregation_to_string(index):\n",
        "  if index == 0:\n",
        "    return \"NONE\"\n",
        "  if index == 1:\n",
        "    return \"SUM\"\n",
        "  if index == 2:\n",
        "    return \"AVERAGE\"\n",
        "  if index == 3:\n",
        "    return \"COUNT\"\n",
        "  raise ValueError(f\"Unknown index: {index}\")\n",
        "\n",
        "def predict(table_data, queries):\n",
        "  table = [list(map(lambda s: s.strip(), row.split(\"|\"))) \n",
        "           for row in table_data.split(\"\\n\") if row.strip()]\n",
        "  examples = convert_interactions_to_examples([(table, queries)])\n",
        "  write_tf_example(\"results/wtq/tf_examples/test.tfrecord\", examples)\n",
        "  write_tf_example(\"results/wtq/tf_examples/random-split-1-dev.tfrecord\", [])\n",
        "  \n",
        "  ! python -m tapas.run_task_main \\\n",
        "    --task=\"WTQ\" \\\n",
        "    --output_dir=\"results\" \\\n",
        "    --noloop_predict \\\n",
        "    --test_batch_size={len(queries)} \\\n",
        "    --tapas_verbosity=\"ERROR\" \\\n",
        "    --compression_type= \\\n",
        "    --reset_position_index_per_cell \\\n",
        "    --init_checkpoint=\"tapas_model/model.ckpt\" \\\n",
        "    --bert_config_file=\"tapas_model/bert_config.json\" \\\n",
        "    --mode=\"predict\" 2> error\n",
        "\n",
        "\n",
        "  results_path = \"results/wtq/model/test.tsv\"\n",
        "  all_coordinates = []\n",
        "  df = pd.DataFrame(table[1:], columns=table[0])\n",
        "  # display(IPython.display.HTML(df.to_html(index=False)))\n",
        "  # print()\n",
        "  with open(results_path) as csvfile:\n",
        "    reader = csv.DictReader(csvfile, delimiter='\\t')\n",
        "    for row in reader:\n",
        "      coordinates = sorted(prediction_utils.parse_coordinates(row[\"answer_coordinates\"]))\n",
        "      all_coordinates.append(coordinates)\n",
        "      answers = ', '.join([table[row + 1][col] for row, col in coordinates])\n",
        "      position = int(row['position'])\n",
        "      aggregation = aggregation_to_string(int(row[\"pred_aggr\"]))\n",
        "      print(\">\", queries[position])\n",
        "      answer_text = str(answers)\n",
        "      if aggregation != \"NONE\":\n",
        "        answer_text = f\"{aggregation} of {answer_text}\"\n",
        "      print(answer_text)\n",
        "  return all_coordinates"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gqu-I-M9QaoA"
      },
      "source": [
        "# 4. Use the machine learning model to make initial predictions of your customized benchmark"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Start collection. This may run for several hours if you have too many benchmarks.\n",
        "\n",
        "(Note) If you see an error like \"cap is not defined\", please run the following block again until the error is gone."
      ],
      "metadata": {
        "id": "EuKMRAijyVKo"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "coJrvTk75CnG"
      },
      "source": [
        "%%capture cap --no-stderr\n",
        "\n",
        "import pickle\n",
        "with open(\"/content/tapas_on_visqa_outputs.pkl\", \"wb\") as f:\n",
        "  pickle.dump([],f)\n",
        "\n",
        "import pickle\n",
        "with open(\"/content/tapas_on_visqa_inputs.pkl\", \"rb\") as f:\n",
        "  dt = pickle.load(f)\n",
        "\n",
        "for i in range(len(dt)):\n",
        "  print(\"# processing benchmark id={}/{}, query={}\".format(i, dt[i][0], dt[i][1]))\n",
        "  predict(dt[i][2], [dt[i][1]])\n",
        "\n",
        "with open(\"/content/tapas_on_visqa_outputs.log\", \"w\") as f:\n",
        "  f.write(cap.stdout)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Store the output as logs for future use."
      ],
      "metadata": {
        "id": "0kFaUVO0IsNS"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SIE7bTJMVuSh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "88195ada-e62d-4a2a-d0e0-e69d06ed70ba"
      },
      "source": [
        "print(cap.stdout)\n",
        "print(\"# done\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "# processing benchmark id=0/customized-benchmark-00, query=who 's got the highest score ?\n",
            "is_built_with_cuda: True\n",
            "is_gpu_available: False\n",
            "GPUs: []\n",
            "Training or predicting ...\n",
            "Evaluation finished after training step 0.\n",
            "> who 's got the highest score ?\n",
            "Jay\n",
            "\n",
            "# done\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 5. Construct new dataset for Poe by using the results generated so far"
      ],
      "metadata": {
        "id": "frXJh3Us6jyQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from Poe.trinity.utils.visqa import normalize_table, parse_value\n",
        "from Poe.trinity.utils.visqa_strategy import strategy_TaPas_A, strategy_TaPas_B, strategy_TaPas_C\n",
        "\n",
        "data_path = \"/content/\""
      ],
      "metadata": {
        "id": "lWXY67iW6jKY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def interpret_answer(arg_line):\n",
        "    if arg_line.startswith(\"COUNT of \"):\n",
        "        tmp_operands = [parse_value(p) for p in arg_line[len(\"COUNT of \"):].split(\", \")]\n",
        "        return [len(tmp_operands)]\n",
        "    elif arg_line.startswith(\"SUM of \"):\n",
        "        tmp_operands = [parse_value(p) for p in arg_line[len(\"SUM of \"):].split(\", \")]\n",
        "        return [sum(tmp_operands)]\n",
        "    elif arg_line.startswith(\"AVERAGE of \"):\n",
        "        tmp_operands = [parse_value(p) for p in arg_line[len(\"AVERAGE of \"):].split(\", \")]\n",
        "        return [sum(tmp_operands)/len(tmp_operands)]\n",
        "    else:\n",
        "        # no ops\n",
        "        tmp_operands = [parse_value(p) for p in arg_line.split(\", \")]\n",
        "        if len(tmp_operands)==0:\n",
        "            return [\"<no answer>\"]\n",
        "        elif len(tmp_operands)==1:\n",
        "            if isinstance(tmp_operands[0], str) and tmp_operands[0].strip()==\"\":\n",
        "                return [\"<no answer>\"]\n",
        "            else:\n",
        "                return [tmp_operands[0]]\n",
        "        else:\n",
        "            # len>1\n",
        "            return sorted([p for p in tmp_operands], key=lambda x:str(x))\n",
        "        \n",
        "def extract_answers_from_logs(arg_logs):\n",
        "    tmp_answers = []\n",
        "    for i in range(len(arg_logs)):\n",
        "        if arg_logs[i].startswith(\"Evaluation finished\"):\n",
        "            if arg_logs[i+1].startswith(\">\"):\n",
        "                try:\n",
        "                    tmp_answers.append(interpret_answer(arg_logs[i+2]))\n",
        "                except TypeError:\n",
        "                    tmp_answers.append([\"<type error>\"])\n",
        "            else:\n",
        "                # TaPas exception/error\n",
        "                tmp_answers.append([\"<tapas exception>\"])\n",
        "    return tmp_answers"
      ],
      "metadata": {
        "id": "T5X3dDxe6jNY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with open(\"{}/visqa_dataset.pkl\".format(data_path), \"rb\") as f:\n",
        "    dt = pickle.load(f)\n",
        "with open(\"{}/tapas_on_visqa_outputs.log\".format(data_path), \"r\") as f:\n",
        "    tapas_logs = f.readlines()\n",
        "tapas_logs = extract_answers_from_logs(tapas_logs)\n",
        "with open(\"{}/tapas_on_visqa_outputs.pkl\".format(data_path), \"rb\") as f:\n",
        "    tapas_outputs = pickle.load(f)\n",
        "tapas_outputs = [tapas_outputs[i] for i in range(len(tapas_outputs)) if i%2!=0]"
      ],
      "metadata": {
        "id": "TbRjRbbE6jPY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "assert len(tapas_outputs)==len(dt)\n",
        "assert len(tapas_logs)==len(dt)\n",
        "len(tapas_outputs)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RwL24Oe0BVqH",
        "outputId": "c535858e-cd7f-4579-832b-23d0a640b0ef"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# first extract all the cell pointers with probs\n",
        "tapas_parsed_outputs = []\n",
        "for i in range(len(tapas_outputs)):\n",
        "    # print(\"# i={}\".format(i))\n",
        "    if len(tapas_outputs[i])>0:\n",
        "        p = tapas_outputs[i][0] # always at 0 since we pass 1 benchmark to TaPas at a time\n",
        "        dop = p[\"pred_aggr\"] # predicted operator\n",
        "        qlist = p[\"probabilities\"]>0 # find all cells with prob>0\n",
        "        cpps = []\n",
        "        for j in range(len(qlist)):\n",
        "            if qlist[j]:\n",
        "                drow = p[\"row_ids\"][j]-1\n",
        "                dcol = p[\"column_ids\"][j]-1\n",
        "                dprob= p[\"probabilities\"][j]\n",
        "                cpps.append((drow,dcol,dprob))\n",
        "        cpps = sorted(cpps, key=lambda x:x[2], reverse=True)\n",
        "        tapas_parsed_outputs.append((dop,cpps)) # (aggr, cpps)\n",
        "    else:\n",
        "        # no outputs, could be something wrong?\n",
        "        print(\"# warning: no output for i={}\".format(i))\n",
        "        tapas_parsed_outputs.append((0,[])) # (aggr, cpps)"
      ],
      "metadata": {
        "id": "Z1sdYITM9q7Y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# then build \"expected_output\" table and \"candidate_outputs\" table\n",
        "for i in range(len(dt)):\n",
        "    print(\"\\r# processing {}/{}\".format(i, len(dt)), end=\"\")\n",
        "    p = dt[i]\n",
        "\n",
        "    # placeholder\n",
        "    dt[i][\"expected_output\"] = pd.DataFrame()\n",
        "\n",
        "    tmp_outputs_original = tapas_logs[i]\n",
        "    tmp_outputs_TaPas_A = strategy_TaPas_A(tapas_parsed_outputs[i], p[\"rendered_table\"])\n",
        "    tmp_outputs_TaPas_B = strategy_TaPas_B(tapas_parsed_outputs[i], p[\"rendered_table\"])\n",
        "    tmp_probs_TaPas_C, tmp_outputs_TaPas_C = strategy_TaPas_C(tapas_parsed_outputs[i], p[\"rendered_table\"])\n",
        "    dt[i][\"candidate_outputs\"] = {\n",
        "        \"TaPas_original\": tmp_outputs_original,\n",
        "        \"TaPas_A\": tmp_outputs_TaPas_A,\n",
        "        \"TaPas_B\": tmp_outputs_TaPas_B,\n",
        "        \"TaPas_C\": tmp_outputs_TaPas_C,\n",
        "        \"TaPas_probs_C\": tmp_probs_TaPas_C,\n",
        "    }\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kRbxs5-6_3Ug",
        "outputId": "332ff8bd-c431-4209-be6f-0ba08ecc2c29"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\r# processing 0/1"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "with open(\"{}/tapas_on_visqa_dataset.pkl\".format(data_path), \"wb\") as f:\n",
        "    pickle.dump(dt, f)"
      ],
      "metadata": {
        "id": "mGuKan6-CMR0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 6. Run Poe here on your customized benchmark\n",
        "- Since we are running customized benchmarks, we assume that there's no known groundtruth answer. So the command should not include `--expected-only` argument since that's for testing the groundtruth answer.\n",
        "\n",
        "Note that due to Python compatibality issue (TaPas model works on Python 3.7, but Poe may experience errors in Python 3.7), if you encounter errors executing the following command, please follow the stpes below to run the customized dataset on the other machine that is ready for running Poe only:\n",
        "1. Copy the file `/content/tapas_on_visqa_dataset.pkl` from colab to your local machine (open the \"Files\" view in colab so that you can browse the files and download them).\n",
        "2. Specify `-i <your tapas_on_visqa_dataset.pkl>` path when executing the command in the next block."
      ],
      "metadata": {
        "id": "6r_Te78QI9Ff"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! cd Poe/ && python ./test_TaPas_on_VisQA_benchmark.py --benchmark customized-benchmark-00 --dsl meta_visqa --skeletons visqa_simple --strategy TaPas_C  -i ../tapas_on_visqa_dataset.pkl"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1IqtF9EEFx7s",
        "outputId": "2e311423-479b-41c6-80f3-4677b91ccc3b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "# parsed arguments: Namespace(benchmark='customized-benchmark-00', dataset='../tapas_on_visqa_dataset.pkl', dsl='meta_visqa', expected_only=False, fallback='none', mode='full', skeletons='visqa_simple', strategy='TaPas_C', timeout=0)\n",
            "# loading benchmark...\n",
            "# table keywords: {'100', 'zoe', '95', 'name', '98', 'jay', '93', '99', '97', 'physics', 'brian', 'math'}\n",
            "# input type: [dtype('O'), dtype('int64'), dtype('int64')]\n",
            "# input is:\n",
            "    name  ...  physics\n",
            "0    Jay  ...       99\n",
            "1  Brian  ...      100\n",
            "2    Zoe  ...       95\n",
            "\n",
            "[3 rows x 3 columns]\n",
            "# query is: who 's got the highest score ?\n",
            "# expected output type:[]\n",
            "# expected output is:\n",
            "Empty DataFrame\n",
            "Columns: []\n",
            "Index: []\n",
            "# inferred DSL terminals:\n",
            "  # ConstVal: ['<NULL>']\n",
            "     # cmap: []\n",
            "  # AggrFunc: ['max', '<NULL>']\n",
            "     # amap: [('highest', 'max')]\n",
            "  # NumFunc: ['<NULL>']\n",
            "     # nmap: []\n",
            "  # BoolFunc: ['==', '<NULL>']\n",
            "     # bmap: [(None, '==')]\n",
            "  # IndFunc: ['eqmax', '<NULL>']\n",
            "     # imap: [('highest', 'eqmax')]\n",
            "# loading skeleton list...\n",
            "\n",
            "# ========== candidate program report ========== #\n",
            "# (t=0.00) i=0, candidate=[{'ANSWER': 'Jay'}]\n",
            "  # found 12 program(s)\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0'], max, 1), eqmax, 1), ['0'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name'], 'max', 'math'], 'eqmax', 'COL0'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0'], max, 1), eqmax, 1), ['-1'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name'], 'max', 'math'], 'eqmax', 'COL0'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 1), eqmax, 1), ['0'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'math'], 'max', 'math'], 'eqmax', 'math'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 1), eqmax, 1), ['-1', '-2'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'math'], 'max', 'math'], 'eqmax', 'math'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 1), eqmax, 2), ['0'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'math'], 'max', 'math'], 'eqmax', 'COL0'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 1), eqmax, 2), ['-1', '-2'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'math'], 'max', 'math'], 'eqmax', 'COL0'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 2), eqmax, 1), ['0'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'math'], 'max', 'physics'], 'eqmax', 'math'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 2), eqmax, 1), ['-1', '-2'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'math'], 'max', 'physics'], 'eqmax', 'math'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 1), eqmax, 2), ['0'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'physics'], 'max', 'math'], 'eqmax', 'COL0'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 1), eqmax, 2), ['-1', '-2'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'physics'], 'max', 'math'], 'eqmax', 'COL0'], ['name']]\n",
            "    # SelectCol(SelectRow0(@param0, eqmax, 1), ['0'])\n",
            "      --> ['SelectCol', ['SelectRow0', 'input@0', 'eqmax', 'math'], ['name']]\n",
            "    # SelectCol(SelectRow0(@param0, eqmax, 1), ['-1', '-2'])\n",
            "      --> ['SelectCol', ['SelectRow0', 'input@0', 'eqmax', 'math'], ['name']]\n",
            "# (t=1.36) i=1, candidate=[{'ANSWER': 'Brian'}]\n",
            "  # found 12 program(s)\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0'], max, 2), eqmax, 1), ['0'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name'], 'max', 'physics'], 'eqmax', 'COL0'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0'], max, 2), eqmax, 1), ['-1'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name'], 'max', 'physics'], 'eqmax', 'COL0'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 2), eqmax, 2), ['0'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'math'], 'max', 'physics'], 'eqmax', 'COL0'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 2), eqmax, 2), ['-1', '-2'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'math'], 'max', 'physics'], 'eqmax', 'COL0'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 1), eqmax, 1), ['0'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'physics'], 'max', 'math'], 'eqmax', 'physics'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 1), eqmax, 1), ['-1', '-2'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'physics'], 'max', 'math'], 'eqmax', 'physics'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 2), eqmax, 1), ['0'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'physics'], 'max', 'physics'], 'eqmax', 'physics'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 2), eqmax, 1), ['-1', '-2'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'physics'], 'max', 'physics'], 'eqmax', 'physics'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 2), eqmax, 2), ['0'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'physics'], 'max', 'physics'], 'eqmax', 'COL0'], ['name']]\n",
            "    # SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 2), eqmax, 2), ['-1', '-2'])\n",
            "      --> ['SelectCol', ['SelectRow0', ['GroupSum', 'input@0', ['name', 'physics'], 'max', 'physics'], 'eqmax', 'COL0'], ['name']]\n",
            "    # SelectCol(SelectRow0(@param0, eqmax, 2), ['0'])\n",
            "      --> ['SelectCol', ['SelectRow0', 'input@0', 'eqmax', 'physics'], ['name']]\n",
            "    # SelectCol(SelectRow0(@param0, eqmax, 2), ['-1', '-2'])\n",
            "      --> ['SelectCol', ['SelectRow0', 'input@0', 'eqmax', 'physics'], ['name']]\n",
            "# (t=2.74) i=2, candidate=[{'ANSWER': 'Zoe'}]\n",
            "  # found 0 program(s)\n",
            "\n",
            "# ========== review report ========== #\n",
            "# top-1, score: 1.33, answer: ['Jay']\n",
            "  # tprog: SelectCol(SelectRow0(@param0, eqmax, 1), ['0'])\n",
            "# top-2, score: 1.33, answer: ['Jay']\n",
            "  # tprog: SelectCol(SelectRow0(@param0, eqmax, 1), ['-1', '-2'])\n",
            "# top-3, score: 1.33, answer: ['Brian']\n",
            "  # tprog: SelectCol(SelectRow0(@param0, eqmax, 2), ['0'])\n",
            "# top-4, score: 1.33, answer: ['Brian']\n",
            "  # tprog: SelectCol(SelectRow0(@param0, eqmax, 2), ['-1', '-2'])\n",
            "# top-5, score: 1.27, answer: ['Jay']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0'], max, 1), eqmax, 1), ['0'])\n",
            "# top-6, score: 1.27, answer: ['Jay']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0'], max, 1), eqmax, 1), ['-1'])\n",
            "# top-7, score: 1.27, answer: ['Brian']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0'], max, 2), eqmax, 1), ['0'])\n",
            "# top-8, score: 1.27, answer: ['Brian']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0'], max, 2), eqmax, 1), ['-1'])\n",
            "# top-9, score: 1.26, answer: ['Jay']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 1), eqmax, 1), ['0'])\n",
            "# top-10, score: 1.26, answer: ['Jay']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 1), eqmax, 1), ['-1', '-2'])\n",
            "# top-11, score: 1.26, answer: ['Jay']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 1), eqmax, 2), ['0'])\n",
            "# top-12, score: 1.26, answer: ['Jay']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 1), eqmax, 2), ['-1', '-2'])\n",
            "# top-13, score: 1.26, answer: ['Jay']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 2), eqmax, 1), ['0'])\n",
            "# top-14, score: 1.26, answer: ['Jay']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 2), eqmax, 1), ['-1', '-2'])\n",
            "# top-15, score: 1.26, answer: ['Jay']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 1), eqmax, 2), ['0'])\n",
            "# top-16, score: 1.26, answer: ['Jay']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 1), eqmax, 2), ['-1', '-2'])\n",
            "# top-17, score: 1.26, answer: ['Brian']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 2), eqmax, 2), ['0'])\n",
            "# top-18, score: 1.26, answer: ['Brian']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '1'], max, 2), eqmax, 2), ['-1', '-2'])\n",
            "# top-19, score: 1.26, answer: ['Brian']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 1), eqmax, 1), ['0'])\n",
            "# top-20, score: 1.26, answer: ['Brian']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 1), eqmax, 1), ['-1', '-2'])\n",
            "# top-21, score: 1.26, answer: ['Brian']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 2), eqmax, 1), ['0'])\n",
            "# top-22, score: 1.26, answer: ['Brian']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 2), eqmax, 1), ['-1', '-2'])\n",
            "# top-23, score: 1.26, answer: ['Brian']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 2), eqmax, 2), ['0'])\n",
            "# top-24, score: 1.26, answer: ['Brian']\n",
            "  # tprog: SelectCol(SelectRow0(GroupSum(@param0, ['0', '2'], max, 2), eqmax, 2), ['-1', '-2'])\n"
          ]
        }
      ]
    }
  ]
}