notebooks/examples/run_on_google_colab.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "YjGcvdA1tgzm"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Andrei-Aksionov/nanoGPTplus/blob/main/notebooks/examples/run_on_google_colab.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KPzljRMpsKgr"
      },
      "source": [
        "<h1><center>RUNNING TRAINING AND SAMPLING</center></h1>\n",
        "<h5><center>in Google Colaboratory</center></h5>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6RsFy6wqsq05"
      },
      "source": [
        "If you don't have a GPU in your possession or don't want to install this project on your local machine, you can run this notebook in Google Colab. This service provides an instance with CPU, GPU and TPU (the latter we will not use).\n",
        "\n",
        "In order to do this you can:\n",
        "1. Click on `Open in Colab` badge.\n",
        "2. Copy this notebook to [Google Colab](https://colab.research.google.com/) manually and run it there."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F2fMRCTftl_X"
      },
      "source": [
        "# 1. Preparations"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "CZm3BT5Dtpy2"
      },
      "source": [
        "First we need to verify that the instance is ready, then clone the repository, create virtual environment and install all dependencies with the project itself."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zZvKha4SuBB0"
      },
      "source": [
        "## 1.1. Runtime type"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "HNmuWm3UAzUX"
      },
      "source": [
        "Google Colab provides CPU, GPU and TPU instance.\n",
        "\n",
        "The code will work on CPU and GPU, but I recommend to use GPU instance just for the sake of speed.\n",
        "\n",
        "Here is a [link](https://www.tutorialspoint.com/google_colab/google_colab_using_free_gpu.htm) on how to change runtime type."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cJpY3Jemuqyf"
      },
      "source": [
        "If GPU is selected and available, the code below will output info about available GPU and it's current status."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yOMt2BC0AWC5",
        "outputId": "45464b9f-8ae1-4368-a3e8-0e26f6d570d6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Sat Mar 18 14:12:12 2023       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   58C    P0    23W /  70W |      0MiB / 15360MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ],
      "source": [
        "!nvidia-smi"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BoA5TRPWvAzG"
      },
      "source": [
        "## 1.2. Clone repository"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W-kqQF--QD5Y",
        "outputId": "a5c94433-f65f-405a-b7a3-acd3f8d9b540"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Cloning into 'nanoGPTplus'...\n",
            "remote: Enumerating objects: 295, done.\u001b[K\n",
            "remote: Counting objects: 100% (177/177), done.\u001b[K\n",
            "remote: Compressing objects: 100% (118/118), done.\u001b[K\n",
            "remote: Total 295 (delta 90), reused 76 (delta 52), pack-reused 118\u001b[K\n",
            "Receiving objects: 100% (295/295), 672.00 KiB | 18.16 MiB/s, done.\n",
            "Resolving deltas: 100% (124/124), done.\n"
          ]
        }
      ],
      "source": [
        "!git clone https://github.com/Andrei-Aksionov/nanoGPTplus.git"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fLcK09smvX3a"
      },
      "source": [
        "Colab allows to `cd` into a directory. That means that from now on all the commands will be executed from this directory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tTTDQDUe1buZ",
        "outputId": "cfa72916-0023-4e15-ff0e-2a06bd714f76"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/content/nanoGPTplus\n",
            "LICENSE    poetry.lock\t   README.md   src\n",
            "notebooks  pyproject.toml  references  tests\n"
          ]
        }
      ],
      "source": [
        "%cd nanoGPTplus/\n",
        "!ls"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dGDbvBREvmS1"
      },
      "source": [
        "## 1.3. Prepare virtual environment"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ipBnvU05vq-E"
      },
      "source": [
        "Each instance of Google Colab comes with plethora of preinstalled packages. But for reproducibility in the future, since I don't control versions of all preinstalled packages, I'd rather create a new empty virtual environment and install project's dependencies into it. "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "XdmB78obwL9Q"
      },
      "source": [
        "**Important Note**: I can deal with packages, but I definatelly cannot control version of python interpreter. For now it's 3.9. If you have any issues with running cells first check that the output of the cell bellow is `Python 3.9.*`. \n",
        "\n",
        "The project should work on python 3.8 and up, but it was not tested in Colab, only 3.9 is tested."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WvaEJYEewluP",
        "outputId": "bad29f8c-e379-49df-ddf6-9573f4f68c24"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Python 3.9.16\n"
          ]
        }
      ],
      "source": [
        "!python --version"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V-_adTWI188f",
        "outputId": "96ce8bef-564a-4228-9795-6b0f2b5275e9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "created virtual environment CPython3.9.16.final.0-64 in 254ms\n",
            "  creator CPython3Posix(dest=/content/nanoGPTplus/venv, clear=False, no_vcs_ignore=False, global=False)\n",
            "  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)\n",
            "    added seed packages: pip==23.0.1, setuptools==67.4.0, wheel==0.38.4\n",
            "  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator\n"
          ]
        }
      ],
      "source": [
        "# install package that allows create virtual environments and create `venv` inside project's folder\n",
        "%pip install --quiet virtualenv\n",
        "!virtualenv venv"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "cKy5fH6gxK8h"
      },
      "source": [
        "One trick to activate virtual environment permanently, so all the commands are executed within this environment, is to change `$PATH` environment variable, so the venv is first in line."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t7_cBPZwLHag",
        "outputId": "8d51b82f-33f3-4222-dc2a-b9e7697b6d24"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/content/nanoGPTplus/venv/bin:/opt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "\n",
        "# there are some difficulties with standard exporting of environment variable,\n",
        "# so I use python's builtin `os` module\n",
        "os.environ[\"PATH\"] = f\"{os.getcwd()}/venv/bin:{os.environ['PATH']}\"\n",
        "!echo $PATH"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C-WYvliox0sr"
      },
      "source": [
        "Now we can check the the venv is activated: the command below shows that it's indeed an empty virtual environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rINXOd4lAMLo",
        "outputId": "15c1582e-dc2c-470d-8955-50fdd68d6b3b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Package    Version\n",
            "---------- -------\n",
            "pip        23.0.1\n",
            "setuptools 67.4.0\n",
            "wheel      0.38.4\n"
          ]
        }
      ],
      "source": [
        "!pip list"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n-0nOE62x__m"
      },
      "source": [
        "So now we can install dependencies that are specified in `pyproject.toml` into our venv."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EssNhOKdBgLS",
        "outputId": "cb7dab1d-61ea-491d-dd0b-2ec12fca5fe5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Checking if build backend supports build_editable ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build editable ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing editable metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building editable for nanogptplus (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "%pip install --quiet -e ."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_De9EFEly1QO"
      },
      "source": [
        "# 2. Running models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GoKKb7lXy4BU"
      },
      "source": [
        "Now we are all set.\n",
        "\n",
        "We can train Bigram model, sample from it new tokens. Do the same for GPT and even load weight from pretrained GPT2 model from Huggingface and use it for new token sampling."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AGTlj-CwzL1T"
      },
      "source": [
        "## 2.1. Download dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fuBMMZ_jzR-B"
      },
      "source": [
        "For simplicity this project uses tiny shakespeare dataset. You can definatelly use your own. You can check README on what needs to be done."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dgFz_bvGCzrs",
        "outputId": "bc431d0e-bb1c-43c1-8178-7c12c70f6746"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[32m2023-03-18 14:15:59.049\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.data.downloader\u001b[0m:\u001b[36mdownload\u001b[0m:\u001b[36m34\u001b[0m - \u001b[34m\u001b[1mDownloading https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt into /content/nanoGPTplus/data/raw/tiny_shakespeare\u001b[0m\n",
            "\u001b[32m2023-03-18 14:15:59.420\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.data.downloader\u001b[0m:\u001b[36mdownload\u001b[0m:\u001b[36m44\u001b[0m - \u001b[34m\u001b[1mDownloading is finished\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "!python src/data/scripts/download_tiny_shakespeare.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7BnECvLPzos-"
      },
      "source": [
        "## 2.2. Bigram language model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RRYixAgrztMv"
      },
      "source": [
        "First we start with something fairly simple: bigram language model. This model just learns what token is the most frequent after the current one and uses this statistics during sampling. More about it in `src/model/bigram_language_model/README.md`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PqPjJzzCC9VZ",
        "outputId": "ad5de1fc-6839-4130-8b42-3dfa459985ad"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[32m2023-03-18 14:16:44.527\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m60\u001b[0m - \u001b[34m\u001b[1mRandom seed is fixed for training.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:16:44.528\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m66\u001b[0m - \u001b[1mLoading the data...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:16:44.533\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m70\u001b[0m - \u001b[1mData is loaded.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:16:44.533\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m73\u001b[0m - \u001b[1mStarting tokenizing...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:16:44.662\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m76\u001b[0m - \u001b[1mTokenizing is done.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:16:44.663\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m79\u001b[0m - \u001b[1mSaving tokenizer...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:16:44.665\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m81\u001b[0m - \u001b[1mTokenizer is saved.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:16:44.666\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m84\u001b[0m - \u001b[1mPreparing data loaders...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:16:44.666\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m102\u001b[0m - \u001b[1mData loaders are prepared.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:16:44.667\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m106\u001b[0m - \u001b[1mStaring training...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:16:44.669\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m131\u001b[0m - \u001b[34m\u001b[1mLR warmup iters: 0\u001b[0m\n",
            "\u001b[32m2023-03-18 14:16:44.670\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m137\u001b[0m - \u001b[34m\u001b[1mLR decay iters: 31371\u001b[0m\n",
            "\u001b[32m2023-03-18 14:16:46.915\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.trainer\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m122\u001b[0m - \u001b[34m\u001b[1mTraining on 'cuda' device\u001b[0m\n",
            "=============== Epoch: 0 ===============\n",
            "train: 100% 31371/31371 [01:34<00:00, 332.72it/s, loss=2.47]\n",
            "eval: 100% 3486/3486 [00:08<00:00, 406.43it/s, loss=2.5]\n",
            "Eval averaged loss: 2.5011\n",
            "\u001b[32m2023-03-18 14:18:29.782\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36msrc.model.trainer\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m151\u001b[0m - \u001b[1mCurrent eval loss is `2.5011` which is smaller than current best loss of `inf`; saving the model...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:18:29.783\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36msrc.model.trainer\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m157\u001b[0m - \u001b[1mBest model is saved.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:18:29.784\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m156\u001b[0m - \u001b[1mTraining is finished\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "# for bigram is only large size is available\n",
        "!python src/model/train.py bigram --size large"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YU1clRmy0qx7"
      },
      "source": [
        "And now we can sample from trained model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lCYLOa8VDmeR",
        "outputId": "2339f065-5267-460d-ebda-04cfe7ab4c5f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[32m2023-03-18 14:18:36.245\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m69\u001b[0m - \u001b[34m\u001b[1mRandom seed is fixed for token generation.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:18:38.415\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m104\u001b[0m - \u001b[34m\u001b[1mGenerating tokens on 'cuda' device\u001b[0m\n",
            "\u001b[32m2023-03-18 14:18:38.443\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m120\u001b[0m - \u001b[1mNew generated tokens:  d\n",
            "O: as; nte tis, te othut mod thand he, preckn,\n",
            "\n",
            "Henthif o--wishelapinisers we s, orean,\n",
            "TAUCluprt,\u001b[0m\n",
            "\u001b[32m2023-03-18 14:18:38.444\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m121\u001b[0m - \u001b[34m\u001b[1mToken generation took: 0.0290 seconds\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "!python src/model/generate.py bigram --size large --max-new-tokens 100 --fix-seed"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "vOIzaYme1DJk"
      },
      "source": [
        "Yes, the model is fast, can't deny it. \n",
        "The output is somewhat similar to real words which is kinda ok for such a simple model.\n",
        "\n",
        "But still, this is not war do we want, don't we?\n",
        "\n",
        "Let's check what GPT can offer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HIIMZ9Jo1j59"
      },
      "source": [
        "## 2.3. GPT"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "PrVi1EFu1tjq"
      },
      "source": [
        "GPT accepts three sizes: `small`, `medium` and `large`.\n",
        "\n",
        "Small is good for debugging, while for more or less good result of course the bigger model the better. Also you can play with `--dataset-fraction` argument, which specifies what portion/fraction of dataset to use for training.\n",
        "\n",
        "Since the tokenizer is fairly simple and the training might take a while this time let's take only 10% of the dataset. Though you can try to use the full dataset if you have a more powerfull GPU in your posession (for example on Google Colab Pro/Pro+)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "AQZ6HhaH7XAf"
      },
      "outputs": [],
      "source": [
        "os.environ[\"gpt_size\"] = \"medium\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZT4Q1reoEW8r",
        "outputId": "42658fcb-ad13-48ca-b971-ccd40b7c40b3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[32m2023-03-18 14:29:47.277\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m60\u001b[0m - \u001b[34m\u001b[1mRandom seed is fixed for training.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:47.277\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m66\u001b[0m - \u001b[1mLoading the data...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:47.283\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m70\u001b[0m - \u001b[1mData is loaded.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:47.283\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m73\u001b[0m - \u001b[1mStarting tokenizing...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:47.412\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m76\u001b[0m - \u001b[1mTokenizing is done.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:47.413\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m79\u001b[0m - \u001b[1mSaving tokenizer...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:47.415\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m81\u001b[0m - \u001b[1mTokenizer is saved.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:47.415\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m84\u001b[0m - \u001b[1mPreparing data loaders...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:47.416\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m102\u001b[0m - \u001b[1mData loaders are prepared.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:47.416\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m106\u001b[0m - \u001b[1mStaring training...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:47.629\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36m__init__\u001b[0m:\u001b[36m114\u001b[0m - \u001b[34m\u001b[1mGPT language model is created with number of parameters: 10.65 million\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:47.630\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m131\u001b[0m - \u001b[34m\u001b[1mLR warmup iters: 156\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:47.630\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m137\u001b[0m - \u001b[34m\u001b[1mLR decay iters: 1490\u001b[0m\n",
            "\u001b[32m2023-03-18 14:29:49.723\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.trainer\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m122\u001b[0m - \u001b[34m\u001b[1mTraining on 'cuda' device\u001b[0m\n",
            "=============== Epoch: 0 ===============\n",
            "train: 100% 1569/1569 [10:51<00:00,  2.41it/s, loss=1.23]\n",
            "eval: 100% 174/174 [00:24<00:00,  7.17it/s, loss=3.4]\n",
            "Eval averaged loss: 3.4022\n",
            "\u001b[32m2023-03-18 14:41:05.071\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36msrc.model.trainer\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m151\u001b[0m - \u001b[1mCurrent eval loss is `3.4022` which is smaller than current best loss of `inf`; saving the model...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:41:05.157\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36msrc.model.trainer\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m157\u001b[0m - \u001b[1mBest model is saved.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:41:05.157\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m156\u001b[0m - \u001b[1mTraining is finished\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "!python src/model/train.py gpt --size $gpt_size --dataset-fraction 0.1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nzd0xSId2Hnp"
      },
      "source": [
        "And new tokens are:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dGW67vbyMamT",
        "outputId": "6975d28f-c8a5-4a3b-9849-c0d72492989d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[32m2023-03-18 14:46:14.743\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m69\u001b[0m - \u001b[34m\u001b[1mRandom seed is fixed for token generation.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:46:15.673\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36m__init__\u001b[0m:\u001b[36m114\u001b[0m - \u001b[34m\u001b[1mGPT language model is created with number of parameters: 10.65 million\u001b[0m\n",
            "\u001b[32m2023-03-18 14:46:17.848\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m104\u001b[0m - \u001b[34m\u001b[1mGenerating tokens on 'cuda' device\u001b[0m\n",
            "100% 1000/1000 [00:07<00:00, 137.21it/s]\n",
            "\u001b[32m2023-03-18 14:46:25.140\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m120\u001b[0m - \u001b[1mNew generated tokens:  d\n",
            "Our sufferance. There's ne'er arm in the war,\n",
            "Our still on in sufferity, or be some of them our truth: and deliver him\n",
            "Which our distinction; and it our shall answer\n",
            "The treasure of our strange.\n",
            "\n",
            "MENENIUS:\n",
            "Now, be gone, beseech you.\n",
            "\n",
            "CORIOLANUS:\n",
            "That I am sworn! this content:\n",
            "To bring unsul to be so, I do beseech you,\n",
            "Let me deserve consul, and sake me be consul.\n",
            "\n",
            "CORIOLANUS:\n",
            "Is thou own did fairst this should deeds,\n",
            "When I am so honour'd rather was to them; I think\n",
            "And such a soldier: if the lies if the world\n",
            "be from our Cominius.\n",
            "\n",
            "MENENIUS:\n",
            "And, let's well; sir,'tis a below heard.\n",
            "\n",
            "COMINIUS:\n",
            "Hear me speak:\n",
            "I have been so, and from their state,\n",
            "By Jove and charges me from hencests, whence\n",
            "In he did service of the nature force; mark I\n",
            "speak in the place of his country: seek meal me speak me, and the blood\n",
            "To know throw their hurs of Marcius sound not done\n",
            "How like a charge to both rest, which with his provater\n",
            "In proceed him should not be bollood, wherein he was\n",
            "And lack'd upon my co\u001b[0m\n",
            "\u001b[32m2023-03-18 14:46:25.140\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m121\u001b[0m - \u001b[34m\u001b[1mToken generation took: 7.2919 seconds\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "!python src/model/generate.py gpt --size $gpt_size --max-new-tokens 1000 --fix-seed"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "UARmUd1i2OM_"
      },
      "source": [
        "Ok, that looks much better than what Bigram LM did. Don't forget that the dataset is fairly small and tokenizer is a basic one. So the power of GPT isn't utilized fully.\n",
        "\n",
        "Also you can achieve better results with bigger model and training on the full dataset, but it will take a while on Nvidia T4."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bl-igDB8_EOP"
      },
      "source": [
        "## 2.4. GPT with pretrained weights"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eX3cFhUl2oeA"
      },
      "source": [
        "This GPT implementation supports loading pretrained weights for GPT2 model from Huggingface (weights are provided by OpenAI). That model was trained on large corpus of data and uses much more sophisticated [byte-pair tokenizer](https://huggingface.co/course/chapter6/5?fw=pt).\n",
        "\n",
        "**Note**: the weights are pretrained not on shakespeare dataset, so the output will be different to what we saw before."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uKEbWvr73rdl"
      },
      "source": [
        "GPT2 has 4 configs:\n",
        "1. gpt2 (124M parameters) \n",
        "2. gpt2-medium (350M)\n",
        "3. gpt2-large (774M)\n",
        "4. gpt2-xl (1.5B)\n",
        "\n",
        "*Google Colab with Nvidia T4 can handle up to gpt2-large.\n",
        "Though it possible to use even the largest one it will require change of how the model is loaded.*\n",
        "\n",
        "The large the model is the better the sampling, but it means that the memory consumption will be also increased."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "X2u06Mat_eUC"
      },
      "outputs": [],
      "source": [
        "os.environ[\"gpt2_config\"] = \"gpt2-medium\"\n",
        "os.environ[\"max_new_tokens\"] = \"1000\"\n",
        "os.environ[\"continue_tokens\"] = \"My name is Giovanni Giorgio but everybody calls me \""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "biUnerKq_Bb2",
        "outputId": "b10c3d1c-142b-4bb3-9ad5-95fc9c3f68d8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[32m2023-03-18 14:51:48.867\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m69\u001b[0m - \u001b[34m\u001b[1mRandom seed is fixed for token generation.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:51:51.251\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36mfrom_pretrained\u001b[0m:\u001b[36m340\u001b[0m - \u001b[34m\u001b[1mCreating GPT model with parameters: {'vocab_size': 50257, 'embeddings_size': 1024, 'context_size': 1024, 'num_layers': 24, 'num_heads': 16, 'head_size': None, 'feed_forward_scaling': 4, 'bias': True, 'dropout': 0.1}\u001b[0m\n",
            "\u001b[32m2023-03-18 14:52:00.173\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36m__init__\u001b[0m:\u001b[36m114\u001b[0m - \u001b[34m\u001b[1mGPT language model is created with number of parameters: 353.77 million\u001b[0m\n",
            "\u001b[32m2023-03-18 14:52:00.175\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36mfrom_pretrained\u001b[0m:\u001b[36m348\u001b[0m - \u001b[34m\u001b[1mLoading pretrained Huggingface model of size 'gpt2-medium' ...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:52:06.299\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36mfrom_pretrained\u001b[0m:\u001b[36m350\u001b[0m - \u001b[34m\u001b[1mHuggingface model is loaded.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:52:06.301\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36mfrom_pretrained\u001b[0m:\u001b[36m387\u001b[0m - \u001b[34m\u001b[1mStarting copying weights from pretrained Huggingface model into our implementation ...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:52:07.223\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36mfrom_pretrained\u001b[0m:\u001b[36m401\u001b[0m - \u001b[34m\u001b[1mWeights are copied.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:52:09.637\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m104\u001b[0m - \u001b[34m\u001b[1mGenerating tokens on 'cuda' device\u001b[0m\n",
            "100% 1000/1000 [02:07<00:00,  7.83it/s]\n",
            "\u001b[32m2023-03-18 14:54:17.288\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m120\u001b[0m - \u001b[1mNew generated tokens: My name is Giovanni Giorgio but everybody calls me ilexandro or protagonist or hero, and I have stopped doing this.\"But she managed to continue with the video on the AuctiCloud YouTube Channel. The video was also shared on various mobile phone stores.Now Endurance County is secured and Martin is beginning to regain her health. In the video, she talked about Tygerley Kimville and the story at that location:  Nice in Canada And this is released on their right to privacy where they are showing how charity can help with a dangerous disease and how skin, sealing electrons, can help contain this outbreak.\"We have suffered enough and we're here to say to everybody that we are dying - you don't breathe in what's shut on your mouth, your brain and your skin, the ultimate combination of love and pain can ease and remove viruses and infections.\"Australian Ken Hemul showed you, suffering in the US battling Ebola and taken by @mithenfurious - here is the vid of you being treated in another specialist in Nashville. Remember, burners live in Hot Cairo\"WHAT'S LIVE LIFE!?\": Also showing a video of Grandma Christine in New Zealand and asks if you did somebody 2008, this ONEENIGHT: Coverage working R-medic operating room including prompt mention of the looming solution pro Rom/EMA where I check in that important area of his genitalia\" take it home with you i look forward to direct pic you received; pic will of course make you want to carry and survive this 2002 inflammatory ad Ronald in Italy from the UK This column began : Tolerant fear kills and this goes over well\"Who says that news has to get out everyday? In book, with stories 2013 -current. Other tidbits of information : Ramadan Is Being Made Laws of World - deference to holy Prophet (Islamic Announcement) .\"Inflammation as a professional solution is off the table\"; \"http://files.greens.com.au/e-public/2016/10/Content_final.png\" Food systems become safer killed outside\n",
            "Listen to new Sounds Peace? on why vaccination is a contortion . Let Puffs Away,\n",
            "A user on the provenance family blog.\n",
            "Whinney wrote :\n",
            "I felt very helpless with how much he had all mapped out with cartoons, and bastard lives. He and his wife got emotional (on/off camera) at times of ending things to induce crying worse than tears.\n",
            "Does this sound familiar?! From NPR:\n",
            "Poor Calvin Puckett asked doctors whether switching vaccines could possibly impact his far deeper, more insidious disease in this brain hospital room lightly topless. A spokeswoman for St. Jude Medical Center told The Associated Press and WDAY that initial care supports medications plus a range of preventive measures, with dependent on how homesick parents respond. Under the age of 20, the aim is to reach frequent vaccination, if possible. With Prescription Drug Monitoring Program and CDPHP, one in 66 Kent face elimination within five days. More from Vanishing: New Zealand Post: Surprising, unnatural effects of yeast infection Despite the fields of yeast, it is \"much more comparable to peanut allergies,\" a study published September 11 in the journal PLOS One found.\n",
            "Quemmler says he's given no charge to the state of Oregon for his insurance because, indeed (after the morning chemotherapy treatment with twourized waste solution which captured only a mere echo of his debut Tlemofrog jaw), his opportunity to voice his less than conscientious, raw eyes is highly commercialized.  Conclusion :  \"virgin friends and families are begging for those who already have blinded artfully that they're things, monsters to stop letting die — that a mind as intense as mine could live long enough to matter.\" Citadel has taken record of more than 800 points in 2011/12, its 58 highest attending the Sacramento range over the past 12 months . Here are some stats on \"yet other events\n",
            "XXXX THE KEEPER Over to your friend Uni,\n",
            "About that nightly pounding over college speech '94 I imagine it's 140. The Expos announced, by the way, that Anthony Bonds and Alex Rodriguez will be honored with the Hall of Famer's baton, cybracologist statistician Fred Norton reported   \"Geniuses presented alternatively trotting reco Downing courts and tough guys standing on hot dogs. Bursts of automated commentary measured 108 to 178 to 82 pressure point. Payback? Maybe.\" Nice.\n",
            "With Toews backing away from Dirk Nowitzki  Toews, who turns 38 this month, speaks passionately at his baseball grave after the ax was removed from his neck and eventually ushers the puck to the Cy Young in Oakland. The perfect finish to a surprising afternoon's work . Funny thing is eleven dope stories just as a free Julep service ended around the same time. And in anticipation of your 3:00 pm funk.<|endoftext|>+ Split Screen with Rides!\n",
            "\u001b[0m\n",
            "\u001b[32m2023-03-18 14:54:17.288\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m121\u001b[0m - \u001b[34m\u001b[1mToken generation took: 127.6519 seconds\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "!python src/model/generate.py gpt --gpt2-config $gpt2_config --max-new-tokens \"$((max_new_tokens))\" --fix-seed --continue-tokens \"$continue_tokens\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LBXMcQPv4aEM"
      },
      "source": [
        "### 2.4.1. Key-Value cache"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o8jyMTAi4eaV"
      },
      "source": [
        "For GPT it's possible to use kv-cache in order to speed up new token generation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SNmvA2sS-PaP",
        "outputId": "5ab9e503-e36a-4f7a-ccaa-63d3b2241369"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[32m2023-03-18 14:54:23.120\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m69\u001b[0m - \u001b[34m\u001b[1mRandom seed is fixed for token generation.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:54:25.108\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36mfrom_pretrained\u001b[0m:\u001b[36m340\u001b[0m - \u001b[34m\u001b[1mCreating GPT model with parameters: {'vocab_size': 50257, 'embeddings_size': 1024, 'context_size': 1024, 'num_layers': 24, 'num_heads': 16, 'head_size': None, 'feed_forward_scaling': 4, 'bias': True, 'dropout': 0.1}\u001b[0m\n",
            "\u001b[32m2023-03-18 14:54:35.310\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36m__init__\u001b[0m:\u001b[36m114\u001b[0m - \u001b[34m\u001b[1mGPT language model is created with number of parameters: 353.77 million\u001b[0m\n",
            "\u001b[32m2023-03-18 14:54:35.312\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36mfrom_pretrained\u001b[0m:\u001b[36m348\u001b[0m - \u001b[34m\u001b[1mLoading pretrained Huggingface model of size 'gpt2-medium' ...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:54:41.583\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36mfrom_pretrained\u001b[0m:\u001b[36m350\u001b[0m - \u001b[34m\u001b[1mHuggingface model is loaded.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:54:41.586\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36mfrom_pretrained\u001b[0m:\u001b[36m387\u001b[0m - \u001b[34m\u001b[1mStarting copying weights from pretrained Huggingface model into our implementation ...\u001b[0m\n",
            "\u001b[32m2023-03-18 14:54:42.519\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36msrc.model.gpt_language_model.gpt\u001b[0m:\u001b[36mfrom_pretrained\u001b[0m:\u001b[36m401\u001b[0m - \u001b[34m\u001b[1mWeights are copied.\u001b[0m\n",
            "\u001b[32m2023-03-18 14:54:44.535\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m104\u001b[0m - \u001b[34m\u001b[1mGenerating tokens on 'cuda' device\u001b[0m\n",
            "100% 1000/1000 [00:18<00:00, 54.07it/s]\n",
            "\u001b[32m2023-03-18 14:55:03.032\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m120\u001b[0m - \u001b[1mNew generated tokens: My name is Giovanni Giorgio but everybody calls me ilexandro or protagonist or hero, and I have stopped doing this.\"But she managed to continue with the video on the AuctiCloud YouTube Channel. The video was also shared on various mobile phone stores.Now Endurance County is secured and Martin is beginning to regain her health. In the video, she talked about Tygerley Kimville and the story at that location:  Nice in Canada And this is released on their right to privacy where they are showing how charity can help with a dangerous disease and how skin, sealing electrons, can help contain this outbreak.\"We have suffered enough and we're here to say to everybody that we are dying - you don't breathe in what's shut on your mouth, your brain and your skin, the ultimate combination of love and pain can ease and remove viruses and infections.\"Australian Ken Hemul showed you, suffering in the US battling Ebola and taken by @mithenfurious - here is the vid of you being treated in another specialist in Nashville. Remember, burners live in Hot Cairo\"WHAT'S LIVE LIFE!?\": Also showing a video of Grandma Christine in New Zealand and asks if you did somebody 2008, this ONEENIGHT: Coverage working R-medic operating room including prompt mention of the looming solution pro Rom/EMA where I check in that important area of his genitalia\" take it home with you i look forward to direct pic you received; pic will of course make you want to carry and survive this 2002 inflammatory ad Ronald in Italy from the UK This column began : Tolerant fear kills and this goes over well\"Who says that news has to get out everyday? In book, with stories 2013 -current. Other tidbits of information : Ramadan Is Being Made Laws of World - deference to holy Prophet (Islamic Announcement) .\"Inflammation as a professional solution is off the table\"; \"http://files.greens.com.au/e-public/2016/10/Content_final.png\" Food systems become safer killed outside\n",
            "Listen to new Sounds Peace? on why vaccination is a contortion . Let Puffs Away,\n",
            "A user on the provenance family blog.\n",
            "Whinney wrote :\n",
            "I felt very helpless with how much he had all mapped out with cartoons, and bastard lives. He and his wife got emotional (on/off camera) at times of ending things to induce crying worse than tears.\n",
            "Does this sound familiar?! From NPR:\n",
            "Poor Calvin Puckett asked doctors whether switching vaccines could possibly impact his far deeper, more insidious disease in this brain hospital room lightly topless. A spokeswoman for St. Jude Medical Center told The Associated Press and WDAY that initial care supports medications plus a range of preventive measures, with dependent on how homesick parents respond. Under the age of 20, the aim is to reach frequent vaccination, if possible. With Prescription Drug Monitoring Program and CDPHP, one in 66 Kent face elimination within five days. More from Vanishing: New Zealand Post: Surprising, unnatural effects of yeast infection Despite the fields of yeast, it is \"much more comparable to peanut allergies,\" a study published September 11 in the journal PLOS One found.\n",
            "Quemmler says he's given no charge to the state of Oregon for his insurance because, indeed (after the morning chemotherapy treatment with twourized waste solution which captured only a mere echo of his debut Tlemofrog jaw), his opportunity to voice his less than conscientious, raw eyes is highly commercialized.  Conclusion :  \"virgin friends and families are begging for those who already have blinded artfully that they're things, monsters to stop letting die — that a mind as intense as mine could live long enough to matter.\" Citadel has taken record of more than 800 points in 2011/12, its 58 highest attending the Sacramento range over the past 12 months . Here are some stats on \"yet other events\n",
            "XXXX THE KEEPER Over to your friend Uni,\n",
            "About that nightly pounding over college speech '94 I imagine it's 140. The Expos announced, by the way, that Anthony Bonds and Alex Rodriguez will be honored with the Hall of Famer's baton, cybracologist statistician Fred Norton reported   \"Geniuses presented alternatively trotting reco Downing courts and tough guys standing on hot dogs. Bursts of automated commentary measured 108 to 178 to 82 pressure point. Payback? Maybe.\" Nice.\n",
            "With Toews backing away from Dirk Nowitzki  Toews, who turns 38 this month, speaks passionately at his baseball grave after the ax was removed from his neck and eventually ushers the puck to the Cy Young in Oakland. The perfect finish to a surprising afternoon's work . Funny thing is eleven dope stories just as a free Julep service ended around the same time. And in anticipation of your 3:00 pm funk.<|endoftext|>+ Split Screen with Rides!\n",
            "\u001b[0m\n",
            "\u001b[32m2023-03-18 14:55:03.032\u001b[0m | \u001b[34m\u001b[1mDEBUG   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36mgenerate_new_tokens\u001b[0m:\u001b[36m121\u001b[0m - \u001b[34m\u001b[1mToken generation took: 18.4969 seconds\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "!python src/model/generate.py gpt --gpt2-config $gpt2_config --max-new-tokens \"$((max_new_tokens))\" --fix-seed --continue-tokens \"$continue_tokens\" --use-kv-cache"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dbx9voRD4ut1"
      },
      "source": [
        "Look at the difference: 120-150 seconds without caching, 18-19 second - with. That's why kv-caching is widely adopted."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}