GPT2 for writing Python code.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ee58e025",
   "metadata": {},
   "source": [
    "In this Notebook we are going to explore how to finetune the GPT2 and create a Python Question answering mdoel like chatgpt. \n",
    "First we tokenize our model and preprocess our text to be formatted the way we want. then we feed it to the model and save the model. next we try using RL model to improve it and finally create a pipeline which is able to give you python code. \n",
    "Technologies that has been used:\n",
    "1. Stackoverflow dataset\n",
    "2. PyTorch\n",
    "3. GPT2 Tokenizer, LMHeadModel\n",
    "4. BERTScore\n",
    "5. Pipeline()\n",
    "6. PPOTrainer</p>\n",
    "\n",
    "\n",
    "You can download the raw data from here: [here](https://www.kaggle.com/datasets/stackoverflow/pythonquestions) \n",
    "I've included a small portion of the processed dataset in the files. So you won't need to run Preprocessing NoteBook. But if you want to use the whole dataset, you should download it and then use preprocessing and remove the limitation line. then the output is ready to rerun this model and get good results.\n",
    "P.S: You need to run all the code since running this code generates a model which is 1.0 GB and I couldnt upload because of the limitation in size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "70ad78eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/ssl/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import time\n",
    "import json\n",
    "import nltk\n",
    "import torch\n",
    "import random\n",
    "import datetime\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from transformers import AdamW, get_linear_schedule_with_warmup\n",
    "from transformers import GPT2LMHeadModel,  GPT2Tokenizer, GPT2Config, GPT2LMHeadModel\n",
    "from torch.utils.data import Dataset, DataLoader, random_split, RandomSampler, SequentialSampler\n",
    "torch.manual_seed(42)\n",
    "\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7ec41a44",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>question</th>\n",
       "      <th>context</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>For reference—future Python possibilities:\\nSt...</td>\n",
       "      <td>How do you express binary literals in Python?</td>\n",
       "      <td>How do you express an integer as a binary numb...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>This will do what you want:\\n\\nsignum = status...</td>\n",
       "      <td>How do I treat an integer as an array of bytes...</td>\n",
       "      <td>I'm trying to decode the result of the Python ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\n  what is a \"Unicode string\" in Python? Does...</td>\n",
       "      <td>Unicode vs UTF-8 confusion in Python / Django?</td>\n",
       "      <td>I stumbled over this passage in the Django tut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Use the python multiprocessing module which wi...</td>\n",
       "      <td>What's the best way to duplicate fork() in win...</td>\n",
       "      <td>How do I implement some logic that will allow ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I think Cog does what you want.\\n</td>\n",
       "      <td>Python code generator for Visual Studio?</td>\n",
       "      <td>I had an idea, if I add a python .py file to m...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              answer  \\\n",
       "0  For reference—future Python possibilities:\\nSt...   \n",
       "1  This will do what you want:\\n\\nsignum = status...   \n",
       "2  \\n  what is a \"Unicode string\" in Python? Does...   \n",
       "3  Use the python multiprocessing module which wi...   \n",
       "4                  I think Cog does what you want.\\n   \n",
       "\n",
       "                                            question  \\\n",
       "0      How do you express binary literals in Python?   \n",
       "1  How do I treat an integer as an array of bytes...   \n",
       "2     Unicode vs UTF-8 confusion in Python / Django?   \n",
       "3  What's the best way to duplicate fork() in win...   \n",
       "4           Python code generator for Visual Studio?   \n",
       "\n",
       "                                             context  \n",
       "0  How do you express an integer as a binary numb...  \n",
       "1  I'm trying to decode the result of the Python ...  \n",
       "2  I stumbled over this passage in the Django tut...  \n",
       "3  How do I implement some logic that will allow ...  \n",
       "4  I had an idea, if I add a python .py file to m...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=pd.read_csv('data.csv')\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b17c00a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>question</th>\n",
       "      <th>context</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>For reference—future Python possibilities:\\nSt...</td>\n",
       "      <td>How do you express binary literals in Python?</td>\n",
       "      <td>How do you express an integer as a binary numb...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>This will do what you want:\\n\\nsignum = status...</td>\n",
       "      <td>How do I treat an integer as an array of bytes...</td>\n",
       "      <td>I'm trying to decode the result of the Python ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\n  what is a \"Unicode string\" in Python? Does...</td>\n",
       "      <td>Unicode vs UTF-8 confusion in Python / Django?</td>\n",
       "      <td>I stumbled over this passage in the Django tut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Use the python multiprocessing module which wi...</td>\n",
       "      <td>What's the best way to duplicate fork() in win...</td>\n",
       "      <td>How do I implement some logic that will allow ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I think Cog does what you want.\\n</td>\n",
       "      <td>Python code generator for Visual Studio?</td>\n",
       "      <td>I had an idea, if I add a python .py file to m...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              answer  \\\n",
       "0  For reference—future Python possibilities:\\nSt...   \n",
       "1  This will do what you want:\\n\\nsignum = status...   \n",
       "2  \\n  what is a \"Unicode string\" in Python? Does...   \n",
       "3  Use the python multiprocessing module which wi...   \n",
       "4                  I think Cog does what you want.\\n   \n",
       "\n",
       "                                            question  \\\n",
       "0      How do you express binary literals in Python?   \n",
       "1  How do I treat an integer as an array of bytes...   \n",
       "2     Unicode vs UTF-8 confusion in Python / Django?   \n",
       "3  What's the best way to duplicate fork() in win...   \n",
       "4           Python code generator for Visual Studio?   \n",
       "\n",
       "                                             context  \n",
       "0  How do you express an integer as a binary numb...  \n",
       "1  I'm trying to decode the result of the Python ...  \n",
       "2  I stumbled over this passage in the Django tut...  \n",
       "3  How do I implement some logic that will allow ...  \n",
       "4  I had an idea, if I add a python .py file to m...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop NA's\n",
    "df = df.dropna()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "eb6cd714",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer</th>\n",
       "      <th>question</th>\n",
       "      <th>context</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n[RESPONSE]For reference—future Python possib...</td>\n",
       "      <td>[WP]How do you express binary literals in Python?</td>\n",
       "      <td>How do you express an integer as a binary numb...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\n[RESPONSE]This will do what you want:\\n\\nsig...</td>\n",
       "      <td>[WP]How do I treat an integer as an array of b...</td>\n",
       "      <td>I'm trying to decode the result of the Python ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\n[RESPONSE]\\n  what is a \"Unicode string\" in ...</td>\n",
       "      <td>[WP]Unicode vs UTF-8 confusion in Python / Dja...</td>\n",
       "      <td>I stumbled over this passage in the Django tut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\n[RESPONSE]Use the python multiprocessing mod...</td>\n",
       "      <td>[WP]What's the best way to duplicate fork() in...</td>\n",
       "      <td>How do I implement some logic that will allow ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\n[RESPONSE]I think Cog does what you want.\\n</td>\n",
       "      <td>[WP]Python code generator for Visual Studio?</td>\n",
       "      <td>I had an idea, if I add a python .py file to m...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              answer  \\\n",
       "0  \\n[RESPONSE]For reference—future Python possib...   \n",
       "1  \\n[RESPONSE]This will do what you want:\\n\\nsig...   \n",
       "2  \\n[RESPONSE]\\n  what is a \"Unicode string\" in ...   \n",
       "3  \\n[RESPONSE]Use the python multiprocessing mod...   \n",
       "4      \\n[RESPONSE]I think Cog does what you want.\\n   \n",
       "\n",
       "                                            question  \\\n",
       "0  [WP]How do you express binary literals in Python?   \n",
       "1  [WP]How do I treat an integer as an array of b...   \n",
       "2  [WP]Unicode vs UTF-8 confusion in Python / Dja...   \n",
       "3  [WP]What's the best way to duplicate fork() in...   \n",
       "4       [WP]Python code generator for Visual Studio?   \n",
       "\n",
       "                                             context  \n",
       "0  How do you express an integer as a binary numb...  \n",
       "1  I'm trying to decode the result of the Python ...  \n",
       "2  I stumbled over this passage in the Django tut...  \n",
       "3  How do I implement some logic that will allow ...  \n",
       "4  I had an idea, if I add a python .py file to m...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add marks for original promp and repsonse marks\n",
    "df['question'] = '[WP]' + df['question'] # + '?'\n",
    "df['answer'] = '\\n[RESPONSE]' + df['answer']\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "54dc9482",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    [WP]How do you express binary literals in Pyth...\n",
       "1    [WP]How do I treat an integer as an array of b...\n",
       "2    [WP]Unicode vs UTF-8 confusion in Python / Dja...\n",
       "3    [WP]What's the best way to duplicate fork() in...\n",
       "4    [WP]Python code generator for Visual Studio?\\n...\n",
       "dtype: object"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# merge just question and answer into one\n",
    "howtos = df[\"question\"] + df[\"answer\"] \n",
    "howtos.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f2f6abc3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       [WP]How do you express binary literals in Pyth...\n",
       "1       [WP]How do I treat an integer as an array of b...\n",
       "2       [WP]Unicode vs UTF-8 confusion in Python / Dja...\n",
       "3       [WP]What's the best way to duplicate fork() in...\n",
       "4       [WP]Python code generator for Visual Studio?\\n...\n",
       "                              ...                        \n",
       "5454    [WP]How can i solve this regular expression, P...\n",
       "5455    [WP]is_max = s == s.max() | How should I read ...\n",
       "5456    [WP]How two recursion function in program work...\n",
       "5457    [WP]Pandas: How to conditionally assign multip...\n",
       "5458    [WP]How to use a dict to subset a DataFrame?\\n...\n",
       "Length: 5459, dtype: object"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# clean up by removing certain characters or patterns that may cause issues downstream\n",
    "howtos.str.replace('\"', '').str.replace('\\n,\\n', '').str.replace('``', '').str.replace(',,', ',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "97d70ceb",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ssl/miniconda3/envs/cuda_tf2/lib/python3.7/site-packages/ipykernel_launcher.py:13: UserWarning: \n",
      "\n",
      "`distplot` is a deprecated function and will be removed in seaborn v0.14.0.\n",
      "\n",
      "Please adapt your code to use either `displot` (a figure-level function with\n",
      "similar flexibility) or `histplot` (an axes-level function for histograms).\n",
      "\n",
      "For a guide to updating your code to use the new functions, please see\n",
      "https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751\n",
      "\n",
      "  del sys.path[0]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:ylabel='Density'>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAGdCAYAAADpBYyuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABJz0lEQVR4nO3de3xU1b028GfPTGYm1wkQmEkwgQhRUC4RkBhE0Zq3QWklan0RPYK8FNSqhUZA4UColzZWCwdRjtRzqpRTEUtrsfVgKidoPZYY5Kp4odyDwCRASCaZXOa23j8meycTkpDLJHtmz/P9fOaTMLNmsmYTnMe1fmstSQghQEREREQKndodICIiIgo1DEhERERErTAgEREREbXCgERERETUCgMSERERUSsMSEREREStMCARERERtcKARERERNSKQe0OhCufz4czZ84gPj4ekiSp3R0iIiLqBCEEampqkJKSAp2u/XEiBqRuOnPmDFJTU9XuBhEREXXDqVOncMUVV7T7OANSN8XHxwPwX+CEhASVe0NERESd4XA4kJqaqnyOt4cBqZvkabWEhAQGJCIiojBzufIYFmkTERERtcKARERERNQKAxIRERFRKwxIRERERK0wIBERERG1woBERERE1AoDEhEREVErDEhERERErTAgEREREbXCgERERETUCgMSERERUSsMSEREREStMCARERERtcKARERERNSKQe0OUN/aVFp22Tb3Z6X1QU+IiIhCF0eQiIiIiFphQCIiIiJqhQGJiIiIqBUGJCIiIqJWGJCIiIiIWmFAIiIiImqFAYmIiIioFQYkIiIiolYYkIiIiIhaYUAiIiIiaoUBiYiIiKgVBiQiIiKiVhiQiIiIiFphQCIiIiJqhQGJiIiIqBUGpAhkdzRg94lKCCHU7goREVFIMqjdAep7W/edRlllHfQ6Cdel9VO7O0RERCGHI0gRRgiBckcDAKD0eKXKvSEiIgpNIRGQ1q1bh6FDh8JsNiMrKwu7du3qsP2WLVswYsQImM1mjB49Gtu2bQt4XAiBgoICJCcnIzo6Gjk5OTh8+HBAm6FDh0KSpIDbCy+8EPT3FmqcLi8aPT4AQFllHexNYYmIiIiaqR6Q3nnnHeTn52PlypXYu3cvxo4di9zcXFRUVLTZfufOnZg5cybmzp2Lffv2IS8vD3l5eTh48KDS5sUXX8TatWuxfv16lJaWIjY2Frm5uWhoCAwDzz77LM6ePavcnnjiiV59r6HgQm1jwJ8/5ygSERHRJVQPSKtXr8a8efMwZ84cXHPNNVi/fj1iYmLwxhtvtNn+5ZdfxtSpU7F48WKMHDkSzz33HMaNG4dXX30VgH/0aM2aNVi+fDmmT5+OMWPGYOPGjThz5gy2bt0a8Frx8fGw2WzKLTY2trffruouOF0AAJPB/1e/79RFuJpGlIiIiMhP1YDkcrmwZ88e5OTkKPfpdDrk5OSgpKSkzeeUlJQEtAeA3Nxcpf3x48dht9sD2lgsFmRlZV3ymi+88AIGDBiA6667Di+99BI8Hk+7fW1sbITD4Qi4haMLtf6ANHqwBf1iotDg9uHgmWqVe0VERBRaVA1I58+fh9frhdVqDbjfarXCbre3+Ry73d5he/nr5V7zpz/9KTZv3oyPPvoIDz/8MH75y19iyZIl7fa1sLAQFotFuaWmpnb+jYaQSqd/ii0pzoQxVyQCAE5ecKrYIyIiotATscv88/Pzle/HjBkDo9GIhx9+GIWFhTCZTJe0X7p0acBzHA5HWIYkeYqtf6wR9S4vAMBR3/7IGRERUSRSdQQpKSkJer0e5eXlAfeXl5fDZrO1+RybzdZhe/lrV14TALKysuDxeHDixIk2HzeZTEhISAi4hSN5im1AnBEJ0VEAAEeDW80uERERhRxVA5LRaMT48eNRXFys3Ofz+VBcXIzs7Ow2n5OdnR3QHgC2b9+utE9PT4fNZgto43A4UFpa2u5rAsD+/fuh0+kwaNCgnrylkFZd50a92z9qNCDWhIRo/wBidT0DEhERUUuqT7Hl5+dj9uzZmDBhAiZOnIg1a9bA6XRizpw5AIBZs2Zh8ODBKCwsBAAsWLAAU6ZMwapVqzBt2jRs3rwZu3fvxuuvvw4AkCQJCxcuxPPPP4+MjAykp6djxYoVSElJQV5eHgB/oXdpaSluvfVWxMfHo6SkBD/72c/wL//yL+jXT7s7S5+s9NcaxZsNMBp0sDSNINW5vHB7fYjSq76okYiIKCSoHpBmzJiBc+fOoaCgAHa7HZmZmSgqKlKKrMvKyqDTNX9wT5o0CZs2bcLy5cuxbNkyZGRkYOvWrRg1apTSZsmSJXA6nZg/fz6qqqowefJkFBUVwWw2A/BPl23evBk///nP0djYiPT0dPzsZz8LqDHSohMX6gD4648AIDpKD4NOgscnUNPgUe4nIiKKdJLgiaXd4nA4YLFYUF1dHTb1SK8UH8aq7f/EuLR++NH4KwAAqz48hAtOF+bddCXSk/z7QN2flaZmN4mIiHpNZz+/OacSQeQRpAFxzSNFSqE265CIiIgUDEgRRN7vaECLqbQEMwu1iYiIWmNAiiAnK5tGkGKb93mycKk/ERHRJRiQIkS9y4tzNf5dtFsWY8tTbBxBIiIiasaAFCGq6v0bROokwBzV/NeeYGYNEhERUWsMSBGitsF/nIg5Sg9JkpT7m6fYeNwIERGRjAEpQsgByGQI/CuXp9hqGtzwcccHIiIiAAxIEaO2sXkEqaU4kwESAJ9obkNERBTpGJAiRE3TKjWTITAg6XUS4puW+rMOiYiIyI8BKUI01yBd+lfOzSKJiIgCMSBFiPam2IDmQm0u9SciIvJjQIoQ7RVpAy2W+nMlGxEREQAGpIjRcpl/axxBIiIiCsSAFCGai7TbqkFikTYREVFLDEgRoqMapHhOsREREQVgQIoQNR3UIMWa/CNIdS4GJCIiIoABKWLUdDCCFGv031fv8nI3bSIiIjAgRQylBqmNfZCimwKSANDg9vZlt4iIiEISA1KEUFaxGS4dQTLodMrUW10jAxIREREDUoSo6WCZP9Bch+RkHRIREREDUiTweH2ob5o6a6tIGwBimqbZ6lwcQSIiImJAigDOFtNmbdUgAUCssWkEqZEjSERERAxIEcDRYpNIg44jSERERJfDgBQB5E0i5Q0h2yIHJNYgERERMSBFBLlAO95saLdN82aRHEEiIiJiQIoAtY3+KbaOAlJMUw1SHWuQiIiIGJAigTyCFGfqKCDJU2wcQSIiImJAigBdm2LjCBIREREDUgRoHkHqRJE2d9ImIiJiQIoEnatB8gekBrcXXh8PrCUiosjGgBQBOjPFJhdpCwDV9e6+6BYREVHIYkCKALWdKNLW6ySYm3bZrnS6+qRfREREoYoBKQLUdGKjSKD5uJGLdQxIREQU2RiQIkBN01EjcR1MsQHNdUgcQSIiokjHgBQBmo8a6TggyUv9LzIgERFRhGNAigBKkXYHNUhAixEkTrEREVGEY0CKALUNnatBkleycQSJiIgiHQNSBFA2irzcFFvTCNLFOi7zJyKiyMaApHGNHi9cXh+Ajpf5A0AMa5CIiIgAMCBpnjx6BFw+IMWyBomIiAgAA5LmyfVHsUY99Dqpw7asQSIiIvJjQNK4mk4WaANAjIn7IBEREQEMSJpX09i5TSKB5hEkR4MHnqa6JSIiokjEgKRxnTmHTRZj1EOehONKNiIiimQMSBpX5/ICAGKbps86opMkmKP87apYqE1ERBHs8sMKFFY2lZYF/PnzE5UAgAu1rksea0uMUY96txdV9RxBIiKiyMURJI1zN9USRek791ctHzfClWxERBTJGJA0zu3xByRjJwNStFGeYuMIEhERRS4GJI1zeQUAIMrQ8R5IMmUvJNYgERFRBGNA0jhlik3XxSk2jiAREVEEY0DSOCUgGboWkLiKjYiIIhkDksa55Sm2Thdpc4qNiIiIAUnjmlexdbYGiUXaREREDEgaJwekzq5ik0eQGJCIiCiSMSBpnKuL+yBFK0XanGIjIqLIFRIBad26dRg6dCjMZjOysrKwa9euDttv2bIFI0aMgNlsxujRo7Ft27aAx4UQKCgoQHJyMqKjo5GTk4PDhw+3+VqNjY3IzMyEJEnYv39/sN5SyPAoNUhdn2ITQvRav4iIiEKZ6gHpnXfeQX5+PlauXIm9e/di7NixyM3NRUVFRZvtd+7ciZkzZ2Lu3LnYt28f8vLykJeXh4MHDyptXnzxRaxduxbr169HaWkpYmNjkZubi4aGhkteb8mSJUhJSem196e27u6k7fL6lHPciIiIIo3qAWn16tWYN28e5syZg2uuuQbr169HTEwM3njjjTbbv/zyy5g6dSoWL16MkSNH4rnnnsO4cePw6quvAvCPHq1ZswbLly/H9OnTMWbMGGzcuBFnzpzB1q1bA17rgw8+wIcffohf//rXvf02VePydC0gGfU6pV6J02xERBSpVA1ILpcLe/bsQU5OjnKfTqdDTk4OSkpK2nxOSUlJQHsAyM3NVdofP34cdrs9oI3FYkFWVlbAa5aXl2PevHn4r//6L8TExFy2r42NjXA4HAG3cNDVfZAkSUJiTBQAFmoTEVHkUjUgnT9/Hl6vF1arNeB+q9UKu93e5nPsdnuH7eWvHbURQuChhx7CI488ggkTJnSqr4WFhbBYLMotNTW1U89Tm7uLNUgA0C/GCIAjSEREFLlUn2JTwyuvvIKamhosXbq0089ZunQpqqurldupU6d6sYfB09UaJAAcQSIiooinakBKSkqCXq9HeXl5wP3l5eWw2WxtPsdms3XYXv7aUZsdO3agpKQEJpMJBoMBw4cPBwBMmDABs2fPbvPnmkwmJCQkBNxCnU8IeHxd20kbaBmQOIJERESRSdWAZDQaMX78eBQXFyv3+Xw+FBcXIzs7u83nZGdnB7QHgO3btyvt09PTYbPZAto4HA6UlpYqbdauXYsDBw5g//792L9/v7JNwDvvvINf/OIXQX2PapKX+AOd3ygSaDnFxhEkIiKKTAa1O5Cfn4/Zs2djwoQJmDhxItasWQOn04k5c+YAAGbNmoXBgwejsLAQALBgwQJMmTIFq1atwrRp07B582bs3r0br7/+OgB/kfHChQvx/PPPIyMjA+np6VixYgVSUlKQl5cHAEhLSwvoQ1xcHABg2LBhuOKKK/ronfc+eZNIADB0oQYpkTVIREQU4VQPSDNmzMC5c+dQUFAAu92OzMxMFBUVKUXWZWVl0OmaRz8mTZqETZs2Yfny5Vi2bBkyMjKwdetWjBo1SmmzZMkSOJ1OzJ8/H1VVVZg8eTKKiopgNpv7/P2pSa4/Mugk6KSuFGmzBomIiCKbJLhdcrc4HA5YLBZUV1eHVD3SptIy5fsKRwPWFB9GdJQeK35wTadfw6CTsORPX+CWqwdiw5yJvdFNIiIiVXT28zsiV7FFCrev60v8geYibdYgERFRpGJA0jB3F3fRlvWL9dcgVbMGiYiIIhQDkobJNUjGTu6iLUuM5ggSERFFNgYkDevOJpFA8yo2R4MbXh9L1IiIKPIwIGmYqxvHjADNNUhCANX1HEUiIqLIw4CkYZ5ujiBF6XWIN/l3gOBeSEREFIkYkDTM1c2ABACJsTxuhIiIIhcDkoa5vV0/h02mHDfi5BQbERFFHgYkDWsu0u5aDRLQXKhdxRokIiKKQAxIGibvg9SVg2pl8lJ/TrEREVEkYkDSMKUGqYv7IAHN57GxSJuIiCIRA5KGeeQaJF33p9i4WSQREUUiBiQNC8YIEqfYiIgoEjEgaVh3d9IGms9j4yo2IiKKRAxIGtaTgNQ8xcYRJCIiijwMSBom74Nk7MYyf3mKjUeNEBFRJGJA0jB5BMnQrWX+HEEiIqLIxYCkYT2aYms6aqTB7UOD2xvUfhEREYU6BiQNa55i6/pfc7zJAEPT9gAcRSIiokjDgKRhLk/3jxqRJAmJ8maRXMlGREQRhgFJw9w92AcJaHEeG0eQiIgowjAgaZRPCHh8TTtpd2OKDWh53AhHkIiIKLIwIGmUfMwI0L0pNoB7IRERUeRiQNIoeXoN6PkIEvdCIiKiSMOApFHKHkg6CTqphyNITo4gERFRZGFA0ihXD/ZAkiWyBomIiCIUA5JGyXsgdbf+CAD6cRUbERFFKAYkjfIEYQSpeRUbAxIREUUWBiSNCs4UmzyCxCk2IiKKLAxIGuX2BG+KjSNIREQUaRiQNKqnu2gDgcv8fT5xmdZERETawYCkUXJA6s5BtTJLU0DyCaCmwROUfhEREYUDBiSNUvZB6kFAMhn0iDHqAXCajYiIIgsDkkbJy/yNPahBAliHREREkYkBSaOCsYoNaN4skivZiIgokjAgaZQ7SAGJI0hERBSJGJA0qjkg9WyKjceNEBFRJGJA0qjmo0aCM4JUzREkIiKKIAxIGhWMVWwAR5CIiCgyMSBplCcIh9UCzSNIlRxBIiKiCMKApFEeX1MNkq5nf8UD4vwB6UJtY4/7REREFC4YkDRKrkEy9HAEqX9s0wiSkyNIREQUORiQNCpYy/wZkIiIKBIZ1O4A9Q5PD0aQNpWWKd876v3F2ZVOF37/2UnoJAn3Z6UFp5NEREQhiiNIGqWMIPWwBinG5D+LzSeABpe3x/0iIiIKBwxIGuXxBWcfJINOB3OU/zVqXZ4e94uIiCgcMCBpVPM+SD0r0gaAWKN/JtbZyBEkIiKKDAxIGhWsIm0AiDXJAYkjSEREFBm69el57NixYPeDgkgI0aMi7dZijf46JCen2IiIKEJ0KyANHz4ct956K37/+9+joaEh2H2iHvIKAdH0fU+LtAGOIBERUeTp1qfn3r17MWbMGOTn58Nms+Hhhx/Grl27gt036iZ59AgI0giSiTVIREQUWboVkDIzM/Hyyy/jzJkzeOONN3D27FlMnjwZo0aNwurVq3Hu3Llg95O6QK4/kgAYdEEMSJxiIyKiCNGj+ReDwYC7774bW7Zswa9+9SscOXIEixYtQmpqKmbNmoWzZ88Gq5/UBS3rjyQpiDVInGIjIqII0aOAtHv3bvzkJz9BcnIyVq9ejUWLFuHo0aPYvn07zpw5g+nTpwern9QFyhL/INQfAZxiIyKiyNOtT9DVq1dj9OjRmDRpEs6cOYONGzfi5MmTeP7555Geno6bbroJGzZswN69ezv1euvWrcPQoUNhNpuRlZV12XqmLVu2YMSIETCbzRg9ejS2bdsW8LgQAgUFBUhOTkZ0dDRycnJw+PDhgDZ33nkn0tLSYDabkZycjAcffBBnzpzp2oUIUW5lk8iejx4BnGIjIqLI062A9Nprr+H+++/HyZMnsXXrVvzgBz+ArtVoxaBBg/Db3/72sq/1zjvvID8/HytXrsTevXsxduxY5ObmoqKios32O3fuxMyZMzF37lzs27cPeXl5yMvLw8GDB5U2L774ItauXYv169ejtLQUsbGxyM3NDVhxd+utt+IPf/gDDh06hD/96U84evQofvSjH3XncoQcj7JJZJBGkFpMsQkhLtOaiIgo/EmiG594J06cQFpa2iWhSAiBU6dOIS2t84eZZmVl4frrr8err74KAPD5fEhNTcUTTzyBp59++pL2M2bMgNPpxPvvv6/cd8MNNyAzMxPr16+HEAIpKSl48sknsWjRIgBAdXU1rFYrNmzYgPvuu6/NfvzlL39BXl4eGhsbERUVddl+OxwOWCwWVFdXIyEhodPvt7dtKi3DkYpavPGP47AmmLDgtqt6/Jpurw8r//IVAGDFtGsw96b0Hr8mERGRGjr7+d2tIYZhw4bh/Pnzl9xfWVmJ9PTOf3i6XC7s2bMHOTk5zR3S6ZCTk4OSkpI2n1NSUhLQHgByc3OV9sePH4fdbg9oY7FYkJWV1e5rVlZW4q233sKkSZM6FY5CnSeIu2jLr2M0+F+LhdpERBQJuvUJ2t6gU21tLcxmc6df5/z58/B6vbBarQH3W61W2O32Np9jt9s7bC9/7cxrPvXUU4iNjcWAAQNQVlaG9957r92+NjY2wuFwBNxClVyDFIwl/jLupk1ERJHE0JXG+fn5AABJklBQUICYmBjlMa/Xi9LSUmRmZga1g71p8eLFmDt3Lk6ePIlnnnkGs2bNwvvvv9/m0vjCwkI888wzKvSy64I9ggT4C7Uv1rk5gkRERBGhSwFp3759APwjSF9++SWMRqPymNFoxNixY5W6n85ISkqCXq9HeXl5wP3l5eWw2WxtPsdms3XYXv5aXl6O5OTkgDatw1tSUhKSkpJw1VVXYeTIkUhNTcVnn32G7OzsS37u0qVLlYAI+OcwU1NTO/1e+5Jb2QcpeAEpjkv9iYgognQpIH300UcAgDlz5uDll1/ucXGy0WjE+PHjUVxcjLy8PAD+Iu3i4mI8/vjjbT4nOzsbxcXFWLhwoXLf9u3blVCTnp4Om82G4uJiJRA5HA6Ulpbi0UcfbbcvPp9/1KWxsbHNx00mE0wmUxffoTqa90EK5hQbl/oTEVHk6FJAkr355ptB60B+fj5mz56NCRMmYOLEiVizZg2cTifmzJkDAJg1axYGDx6MwsJCAMCCBQswZcoUrFq1CtOmTcPmzZuxe/duvP766wD8038LFy7E888/j4yMDKSnp2PFihVISUlRQlhpaSk+//xzTJ48Gf369cPRo0exYsUKDBs2rM3Ro3DTO1Ns3E2biIgiR6cD0t13340NGzYgISEBd999d4dt33333U53YMaMGTh37hwKCgpgt9uRmZmJoqIipci6rKwsYDuBSZMmYdOmTVi+fDmWLVuGjIwMbN26FaNGjVLaLFmyBE6nE/Pnz0dVVRUmT56MoqIipYA8JiYG7777LlauXAmn04nk5GRMnToVy5cvD5tRoo4Ee6NIoHmzyFoGJCIiigCdDkgWi0UpXrZYLEHtxOOPP97ulNrHH398yX333nsv7r333nZfT5IkPPvss3j22WfbfHz06NHYsWNHt/oaDnplBEmZYmMNEhERaV+nA1LLabVgTrFR8ClF2sGsQeIUGxERRZBuDTHU19ejrq5O+fPJkyexZs0afPjhh0HrGHWfO8hHjQAtD6xlQCIiIu3r1ifo9OnTsXHjRgBAVVUVJk6ciFWrVmH69Ol47bXXgtpB6jpPL9QgtVzm7/PxPDYiItK2bgWkvXv34qabbgIA/PGPf4TNZsPJkyexceNGrF27NqgdpK5z90INkhyQvEKgut4dtNclIiIKRd36BK2rq0N8fDwA4MMPP8Tdd98NnU6HG264ASdPngxqB6nrPL1Qg2TQ62CO8v+6nK9te68oIiIirehWQBo+fDi2bt2KU6dO4W9/+xu+//3vAwAqKipC6mT7SNUbI0gAEGfyH+R7jgGJiIg0rlufoAUFBVi0aBGGDh2KrKwsZXPFDz/8ENddd11QO0hd1xs1SEDzNNv5WldQX5eIiCjUdGsn7R/96EeYPHkyzp49i7Fjxyr333bbbbjrrruC1jnqnt5YxQYA8eamgFTDESQiItK2bgUkwH8obOsDZSdOnNjjDlHPNR9W2zsjSJxiIyIiretWQHI6nXjhhRdQXFyMiooK5aBX2bFjx4LSOeoeZSdtXZBrkDiCREREEaJbAenHP/4x/v73v+PBBx9EcnKycgQJhYbms9iCXaQt1yAxIBERkbZ1KyB98MEH+O///m/ceOONwe4PBUHzWWzBDa7xLNImIqII0a0hhn79+qF///7B7gsFSW8VactTbOc4xUZERBrXrU/Q5557DgUFBQHnsVFo8PoE5JNAooK4USTQPMV2wdkIIXjcCBERaVe3pthWrVqFo0ePwmq1YujQoYiKigp4fO/evUHpHHWdp0XBfLBHkOQDa91e/3EjiTHGoL4+ERFRqOhWQMrLywtyNyhY5CX+QPCX+Uc1HTfS4PbhfG0jAxIREWlWtwLSypUrg90PChK5QFuvk6DrhdWFcaYoNLgbUVHTiOGD4oP++kRERKGg23MwVVVV+M///E8sXboUlZWVAPxTa6dPnw5a56jr5INqg72CTcbjRoiIKBJ0awTpiy++QE5ODiwWC06cOIF58+ahf//+ePfdd1FWVoaNGzcGu5/USW5f72wSKeNxI0REFAm69Sman5+Phx56CIcPH4bZbFbuv+OOO/DJJ58ErXPUdb11zIiMm0USEVEk6FZA+vzzz/Hwww9fcv/gwYNht9t73Cnqvt7aA0nGvZCIiCgSdOtT1GQyweFwXHL/P//5TwwcOLDHnaLu67saJAYkIiLSrm4FpDvvvBPPPvss3G43AECSJJSVleGpp57CPffcE9QOUte4e+mgWhmPGyEiokjQrU/RVatWoba2FgMHDkR9fT2mTJmC4cOHIz4+Hr/4xS+C3UfqAnmjyGAfVCuTp9g4gkRERFrWrVVsFosF27dvxz/+8Q8cOHAAtbW1GDduHHJycoLdP+qivizSFkJA6oW9loiIiNTW5YDk8/mwYcMGvPvuuzhx4gQkSUJ6ejpsNhs/MEOAp5eLtHncCBERRYIufYoKIXDnnXfixz/+MU6fPo3Ro0fj2muvxcmTJ/HQQw/hrrvu6q1+UifJI0jBPqhWFqXXIYEr2YiISOO6NIK0YcMGfPLJJyguLsatt94a8NiOHTuQl5eHjRs3YtasWUHtJHWevFFkb40gAYA1wQxHQy0qahqRYeVxI0REpD1d+hR9++23sWzZskvCEQB873vfw9NPP4233noraJ2jruvtZf4AMCjBBAAodzT02s8gIiJSU5cC0hdffIGpU6e2+/jtt9+OAwcO9LhT1H1yDVJvrWIDAGu8f/f0cgen2IiISJu69ClaWVkJq9Xa7uNWqxUXL17scaeo+3p7FRsADErwB6SKGo4gERGRNnUpIHm9XhgM7Zct6fV6eDyeHneKuq+3N4oEAGvTFFsFR5CIiEijulSkLYTAQw89BJPJ1ObjjY38wFSb29f7I0jWBHmKjSNIRESkTV0KSLNnz75sG65gU1ef1CDJRdqcYiMiIo3qUkB68803e6sfFCR9soqtRZE2NwclIiIt6r1hBlKFXINk6MUapIHx/hEkl8cHRz1rzoiISHsYkDTGrRxW23ujOuYoPRJjogBwmo2IiLSJAUljPMoy/979q23eC4kBiYiItIcBSWPcfVCkDbTcTZsrF4mISHsYkDTG3QdF2gCX+hMRkbYxIGlMn40gxcubRTIgERGR9jAgaUxfBSSrctwIp9iIiEh7GJA0xO31oWkjbRh7PSDJNUgcQSIiIu1hQNKQBrdX+b43jxoBmg+sZZE2ERFpEQOShtQ3BSQJgEHXN0XaFTUNEEL06s8iIiLqawxIGtLgaq4/6u3jPwbG+afY3F6Bi3XuXv1ZREREfY0BSUMaPP4RpN6eXgMAo0GH/rFGAP5RJCIiIi1hQNKQepc/IPV2gbZMXurPOiQiItIaBiQNkWuQenuJv0zZLLKaI0hERKQtDEga0qAEpN6fYgOAZIs/IJ1lQCIiIo1hQNKQhj4eQUq2RAMAzlbX98nPIyIi6isMSBqiTLEZ+uavNSXRP4J0uooBiYiItIUBSUMa3E3L/Ht5DyTZ4ET/CNIZBiQiItIYBiQNkVex9dUIUnKiPMXGzSKJiEhbGJA0pK9XsclF2nUuL6rruVkkERFpR0gEpHXr1mHo0KEwm83IysrCrl27Omy/ZcsWjBgxAmazGaNHj8a2bdsCHhdCoKCgAMnJyYiOjkZOTg4OHz6sPH7ixAnMnTsX6enpiI6OxrBhw7By5Uq4XK5eeX99pbGPV7GZo/RIivNvFsk6JCIi0hLVA9I777yD/Px8rFy5Env37sXYsWORm5uLioqKNtvv3LkTM2fOxNy5c7Fv3z7k5eUhLy8PBw8eVNq8+OKLWLt2LdavX4/S0lLExsYiNzcXDQ3+5ejffvstfD4ffvOb3+Crr77Cv/3bv2H9+vVYtmxZn7zn3tLXI0hAi5VsVVzqT0RE2iEJlYtHsrKycP311+PVV18FAPh8PqSmpuKJJ57A008/fUn7GTNmwOl04v3331fuu+GGG5CZmYn169dDCIGUlBQ8+eSTWLRoEQCguroaVqsVGzZswH333ddmP1566SW89tprOHbsWKf67XA4YLFYUF1djYSEhK6+7V6xfOuX+P1nZfjeiEHIGWnttZ9zf1aa8v3D/7Ubf/uqHM9Ovxazsof22s8kIiIKhs5+fqs6guRyubBnzx7k5OQo9+l0OuTk5KCkpKTN55SUlAS0B4Dc3Fyl/fHjx2G32wPaWCwWZGVltfuagD9E9e/fv93HGxsb4XA4Am6hRlnF1ocjSClNhdqcYiMiIi1RNSCdP38eXq8XVmvgaIfVaoXdbm/zOXa7vcP28teuvOaRI0fwyiuv4OGHH263r4WFhbBYLMotNTW14zengvo+rkECgBROsRERkQapXoOkttOnT2Pq1Km49957MW/evHbbLV26FNXV1crt1KlTfdjLzmno48NqgeYRJO6FREREWqJqQEpKSoJer0d5eXnA/eXl5bDZbG0+x2azddhe/tqZ1zxz5gxuvfVWTJo0Ca+//nqHfTWZTEhISAi4hZoGjz8gGfqySDuR57EREZH2qBqQjEYjxo8fj+LiYuU+n8+H4uJiZGdnt/mc7OzsgPYAsH37dqV9eno6bDZbQBuHw4HS0tKA1zx9+jRuueUWjB8/Hm+++SZ0uvAfTKtXRpD6bopN3k3b7miAx+vrs59LRETUmwxqdyA/Px+zZ8/GhAkTMHHiRKxZswZOpxNz5swBAMyaNQuDBw9GYWEhAGDBggWYMmUKVq1ahWnTpmHz5s3YvXu3MgIkSRIWLlyI559/HhkZGUhPT8eKFSuQkpKCvLw8AM3haMiQIfj1r3+Nc+fOKf1pb+QqHNSrUKQ9MM6EKL0Et1egoqZRmXIjIiIKZ6oHpBkzZuDcuXMoKCiA3W5HZmYmioqKlCLrsrKygNGdSZMmYdOmTVi+fDmWLVuGjIwMbN26FaNGjVLaLFmyBE6nE/Pnz0dVVRUmT56MoqIimM3+6aDt27fjyJEjOHLkCK644oqA/oTzkRnyRpF9OcWm00mwJpjx3cV6nK2uZ0AiIiJNUH0fpHAVivsgZRcW42x1Ax67ZTgG9+u9oNJyHyQA+L+/KcGu45VYO/M63Dk2pdd+LhERUU+FxT5IFFxqLPMHmuuQuJKNiIi0ggFJQxpUOGoEaD609iwDEhERaYTqNUgUHD6faN5J29C7AWlTaVnAn+VdtEuPVyqPtZ6GIyIiCiccQdKIRk/zEvu+nmLrH2sEAFxwuvr05xIREfUWBiSNkKfXgL6fYkuKNQEAKp0u+FjzT0REGsCApBFygbZeJ0En9e0IkiUmCnpJgtcn4Kh39+nPJiIi6g0MSBqh1go2ANBJEvpxmo2IiDSEAUkj1FrBJhsgB6RaBiQiIgp/DEgaoXpAipNHkBpV+flERETBxICkEfUu/yo2I0eQiIiIeowBSSMalHPY+r4GCQAGxDWvZCMiIgp3DEgaUa/2FFts8xQbl/oTEVG4Y0DSCDkgqTXFlhhjhE4C3F6BmgaPKn0gIiIKFgYkjWhUeYpNr5OQGMNCbSIi0gYGJI1QewQJAJKaVrJVslCbiIjCHAOSRsir2NSqQQKA/k1HjnCzSCIiCncMSBrR4FFvJ21Z81J/TrEREVF4Y0DSiHqXuqvYgJabRXIEiYiIwhsDkkYoO2kbVAxI8hRbrQs+H5f6ExFR+GJA0gglIOnUm2LrH2uEQSfB5fXh1MU61fpBRETUUwxIGlEfAiNIep2EQQn+UaRvztao1g8iIqKeYkDSiHq3+qvYACA5IRoA8K3doWo/iIiIeoIBSSMaQqBIGwBsFjMA4JuzDEhERBS+GJA0IhSW+QPNAelbO6fYiIgofDEgaUQoLPMHAFuCPyCdvFAHZyPPZCMiovDEgKQRSpG2ygEp1mRAgtkAADhUzlEkIiIKTwxIGtGgFGmrO8UGsA6JiIjCHwOSRjSEwGG1Mnma7Vsu9SciojCl/qcpBYU8xWYIhYBk4VJ/IiIKb+p/mlKPub0+eJuO9giJESRL8wiSEDxyhIiIwo/6n6bUY/LoERAaNUgD40yI0kuoafTgu4v1aneHiIioyxiQNEDeJFKS/Md9qE2vkzDClgAAOPBdlbqdISIi6gYGJA2QR5Cio/SQJPUDEgCMH9IPALDn5EWVe0JERNR1DEgaUNu0IWOsyaByT5qNY0AiIqIwxoCkAbUN/oAUF0IBSR5B+uqMA3Uu7qhNREThhQFJA5yu0AtIgxOjkWwxw+sTOHCqWu3uEBERdQkDkgbUNvprkGJNepV7EkieZttbxmk2IiIKLwxIGiAfChtKI0gAMKEpIO0+UalyT4iIiLqGAUkDnCFYpA001yHtLauCz8cNI4mIKHwwIGlAKK5iA4CRyQmIjtKjut6No+dq1e4OERFRpzEgaUCoTrFF6XUYm2oBAOzmcn8iIgojDEgaoBRpG0MrIAHAxKH9AQClxy6o3BMiIqLOC71PVOqy5hqk0FnFtqm0DABQ13QMSvG3FXjrs5PKTt/3Z6Wp1jciIqLL4QiSBoTqFBsApPaPgUEnoabBg3M1jWp3h4iIqFMYkDQgVIu0AX8d0pABMQCAo+edKveGiIiocxiQNCAUd9JuadjAOADAMa5kIyKiMMGApAFOZSft0AxIVyoByQmf4H5IREQU+hiQNKA2BIu0WxqcGA2TQYd6txf26ga1u0NERHRZDEgaUNvgD0jxpiiVe9I2vU7C0AGxAMANI4mIKCwwIIU5r0+g3h2ah9W2NGygPyAdO8dCbSIiCn0MSGFOLtAGQrcGCWiuQzp+wQkvz2UjIqIQx4AU5uQ9kAw6CSZD6P512ixmREfp4fL4cPpindrdISIi6lDofqJSpzhb7IEk71IdinSShCubptm4HxIREYU6BqQwJ5/DFqp7ILUk74fEQm0iIgp1DEhhLhTPYWuPPIJUdqEODU2F5URERKFI9YC0bt06DB06FGazGVlZWdi1a1eH7bds2YIRI0bAbDZj9OjR2LZtW8DjQggUFBQgOTkZ0dHRyMnJweHDhwPa/OIXv8CkSZMQExODxMTEYL+lPhXKx4y0NjDOhHizAR6fwN6yi2p3h4iIqF2qBqR33nkH+fn5WLlyJfbu3YuxY8ciNzcXFRUVbbbfuXMnZs6ciblz52Lfvn3Iy8tDXl4eDh48qLR58cUXsXbtWqxfvx6lpaWIjY1Fbm4uGhqaNyh0uVy499578eijj/b6e+xtoXxQbWuSJCnTbDuPXFC5N0RERO1TNSCtXr0a8+bNw5w5c3DNNddg/fr1iImJwRtvvNFm+5dffhlTp07F4sWLMXLkSDz33HMYN24cXn31VQD+0aM1a9Zg+fLlmD59OsaMGYONGzfizJkz2Lp1q/I6zzzzDH72s59h9OjRffE2e5UyxWYM/YAENO+HtPPoeZV7QkRE1D7VApLL5cKePXuQk5PT3BmdDjk5OSgpKWnzOSUlJQHtASA3N1dpf/z4cdjt9oA2FosFWVlZ7b5mZzU2NsLhcATcQkFtiJ/D1tqVSf4RpAPfVSvTg0RERKFGtYB0/vx5eL1eWK3WgPutVivsdnubz7Hb7R22l7925TU7q7CwEBaLRbmlpqb26PWCpXmKLfSLtAGgX6wR/WKi4PUJ7D5RqXZ3iIiI2qR6kXa4WLp0Kaqrq5XbqVOn1O4SgOYi7ThzeIwgAc2jSJ8dY0AiIqLQpFpASkpKgl6vR3l5ecD95eXlsNlsbT7HZrN12F7+2pXX7CyTyYSEhISAWyhwhtEqNll6Ux3SZ8dYqE1ERKFJtYBkNBoxfvx4FBcXK/f5fD4UFxcjOzu7zedkZ2cHtAeA7du3K+3T09Nhs9kC2jgcDpSWlrb7muGuNoxWscnSk/wB6cvTrEMiIqLQpOqnan5+PmbPno0JEyZg4sSJWLNmDZxOJ+bMmQMAmDVrFgYPHozCwkIAwIIFCzBlyhSsWrUK06ZNw+bNm7F79268/vrrAPzLyBcuXIjnn38eGRkZSE9Px4oVK5CSkoK8vDzl55aVlaGyshJlZWXwer3Yv38/AGD48OGIi4vr02vQU7VhtooNAPrFGJHaPxqnKuvx+YlK3Hr1ILW7REREFEDVT9UZM2bg3LlzKCgogN1uR2ZmJoqKipQi67KyMuh0zYNckyZNwqZNm7B8+XIsW7YMGRkZ2Lp1K0aNGqW0WbJkCZxOJ+bPn4+qqipMnjwZRUVFMJvNSpuCggL87ne/U/583XXXAQA++ugj3HLLLb38roMrHKfYACD7ygE4VfkdPjt2gQGJiIhCjiSEEGp3Ihw5HA5YLBZUV1erWo+U+2+f4FB5DX4/NwuTM5KwqbRMtb50hTlKh/w/HMDY1ES899iNaneHiIgiRGc/v7mKLczVhtFZbC3dcOUAAMDB09WoaXCr3BsiIqJADEhhzukKvyJtAEhJjMaQATFN+yHxXDYiIgotDEhhLlxrkADghnT/KBKX+xMRUahhQApjjR4v3F5/CVlYBqRh/QEwIBERUehhQApjzqZz2AAg1hheNUhAcx3Sl6er4WAdEhERhRAGpDAmT6+Zo3Qw6MPvrzLZEo2hA2LgE+C5bEREFFLC71OVFOG4i3Zr8igSz2UjIqJQwoAUxpwaCkglR1mHREREoYMBKYzVhvEKNpkckL46U43qetYhERFRaGBACmNaCEg2ixnpSbGsQyIiopDCgBTGtDDFBgA3XOlf7v/pkfMq94SIiMiPASmMVTr9U1KJMVEq96RnplzlP6z2f74pB48GJCKiUMCAFMYqnY0AgKQ4k8o96ZmbMpJgNOhwqrIehytq1e4OERERwntuJsJdqHUBAPrHGlXuSddtKi0L+HP6gFgcKq/Br/92CLdc7R9Ruj8rTY2uERERcQQpnF1w+gPSgDAMSK2NSI4HAHxz1qFyT4iIiBiQwtqFpim2AXEaCEi2BADAdxfrUcNjR4iISGUMSGGsslYeQQrvGiQAsERHYXBiNASAQ/YatbtDREQRjgEpTAkhcN4ZvjVIbRnZNM32NafZiIhIZQxIYcrp8sLl8QHQxhQbAFybYgEA/LO8BlV1LpV7Q0REkYwBKUxdqPXXH0VH6RFj1MZiRGtC867apce5qzYREamHASlMKSvYNDJ6JMtuOpvt8xOVaHB7Ve4NERFFKgakMHWhVjtL/FsamZwAS3QU6lxevP/FWbW7Q0REEYoBKUxVKkv8w38FW0t6nYSsdP/ZbBt2HufRI0REpAoGpDB1Pox30b6c64f2h0En4eBpB7bs/k7t7hARUQRiQApTlRqtQQKAWJMBOSOtAIBn3/8ap6vqVe4RERFFGgakMCWvYtNaDZJsckYSxqUlorbRg6f++AV8Pk61ERFR32FAClPN57BpqwZJppMkrPq/mTBH6fDpkfNY/t5BeBmSiIiojzAghSl5FVt/DU6xydKTYvHLu0ZDkoBNpWVYsHmfsjkmERFRb9LGDoMRSKlB0ugUG+APRQBw3/Vp+MPnp/D+F2fx5elqzJyYhgRzFO7PSlO5h0REpFUcQQpDQghc0Ogy/7aMHmzBg9lDYDLocPJCHV7dcQTHzteq3S0iItIwBqQwVNPogdvrr8fR8ghSS1dZ4/HYLcNhSzCjttGDNz49jt/8/Sj3SSIiol7BgBSG5PqjWKMe5ii9yr3pO0nxJjwyZRiuS02ETwCFH3yLR36/B3Uuj9pdIyIijWFACkPyLtpaLtBuj9Ggw4/GX4HpmSkw6nX421flmPkfpUpNFhERUTAwIIWh87XaXuJ/OZIkISt9AN6en4XEmCgcOFWFH722E6cq69TuGhERaQQDUhiKhBVsnTF+SH/88ZFsDE6MxrHzTtz92k58daZa7W4REZEGMCCFIWUX7QicYmtt+KB4/OnRSRhhi8e5mkbM+M1n2HnkvNrdIiKiMMeAFIbkXbT7R+gUm2xTaRk2lZZhx7cVuHd8KtKTYlHb6MGDv92Fp//0hdrdIyKiMMaAFIbO1fhHkJI4gqSINurx0KShGJWSAK8Q2Pz5KfzHJ8e4DQAREXULA1IYOnrOCQAYMiBW5Z6Elii9DvdNTMMNVw4AAPxi2zf46eb9cDZyGwAiIuoaBqQw4/UJHD3n30X6Kmucyr0JPTpJwg/HJGPa6GQYdBL+euAMpq/7Bw6cqlK7a0REFEYYkMLMyQtOuDw+mKN0SO0Xo3Z3QpIkSbhxeBI2z78B1gQTjlTU4q5//wd+ue0bbipJRESdwoAUZv5Z7h89Gj4oDjqdpHJvQtuEof3xwYKbkZeZAp8AXv/kGP7P6k9QdPAsa5OIiKhDBrU7QF1zuLwGAHDVoHiVexL6NpWWAQAmpg9AvDkKfz1wBqer6vHI7/ciY1AcfjgmBT/NyVC5l0REFIoYkMLMPyv8I0gZVgakrhiZnIBhA+Pw8T8r8L+Hz+NwRS1eLj6MBo8XP70tI6LOtCMiosvjFFuYUUaQWKDdZUaDDt+/xoYFt2XgKmscvELg3z8+itw1n+DTw9xckoiImjEghRGP14djTUv8r+IIUrclxZkwO3so/iUrDbYEM05eqMO//LYU+X/Yz0NviYgIAANSWDlZWQeX14foKD0GJ0ar3Z2wJkkSrkmxYHv+zXho0lBIEvDu3tO4bdXH+P1nJ+Hx+tTuIhERqYgBKYzI02tcwRY88eYo/PzOa/Fu03luF+vcWL71IHLXfIL//uIsvD6udiMiikQs0g4j8hL/DNYfBY280g0AHsgagl3HL6D42wocPefEY5v2Iq1/DB6aNBR51w1G/1ge7UJEFCkYkMLIP5UCbdYf9Qa9TkL2sCRcl9YPnx45j5KjF1BWWYdn3/8ahR98g++NGIQfjU/FLVcPRJSeg69ERFrGgBRGDpfziJG+YI7SI2ekFTdnDIReL+EPn5/Cl6er8bevyvG3r8oxINaIvOsG455xV+CalAS1u0tERL2AASlMHD/vxKHyGkgScG2KRe3uRASjwT9KNHNiGm6tbsDesovYf6oKF5wu/PbT4/jtp8dhSzDjwewh+OGYFKQN4NEvRERawYAUJt7e5a+VueWqgbAmmFXuTeSxWcy4Y3Qycq+14XBFDfaevIhv7DWwOxrw0t8O4aW/HULGoDjcOmIQJgzph+vS+mFgvEntbhMRUTcxIIWBBrcXW3afAuAvJCb16HUSRtgSMMKWgHqXF1+dqUZ5TQM+O1aJwxW1OFxRi9eb2l7RLxrXpfXD6MEJGJnsvyXFMTQREYUDBqQw8MHBs7hY50aKxYxbRwxSuzvUJNqox4Sh/QEA37vaisMVNThSUYtTF+tQ4WjEdxfr8d3Fevz1wBnlOUlxJoywxSO1fzRSLNFISYxGcqIZA+NMiDdHId5sQIxRD0nq/DYOHq8PLq8PjW4f9HoJ8SZDl55PRESXComAtG7dOrz00kuw2+0YO3YsXnnlFUycOLHd9lu2bMGKFStw4sQJZGRk4Fe/+hXuuOMO5XEhBFauXIn/+I//QFVVFW688Ua89tpryMhoPpi0srISTzzxBP76179Cp9Phnnvuwcsvv4y4uNArgP79Z/7ptZkT06Dn/kchKdqox5grEjHmikQA/lG/7y7W49TFOpytqsfZ6gZUOl04X9uIT480dvhaep2EeLPBfzNFwRSlQ5ROB4/PhzqXt+nmQZ3Li0aP75K9mvQ6CYnRUbDERKFfjBED40ywJpgwKMEMa4LZ/328GUlxRpii9IjSS4jS6bi3FhFRC6oHpHfeeQf5+flYv349srKysGbNGuTm5uLQoUMYNOjS0ZKdO3di5syZKCwsxA9+8ANs2rQJeXl52Lt3L0aNGgUAePHFF7F27Vr87ne/Q3p6OlasWIHc3Fx8/fXXMJv99TsPPPAAzp49i+3bt8PtdmPOnDmYP38+Nm3a1KfvvyNCCLxcfBh7Tl6EQSdhxvWpaneJOskcpcfwQXEYPqg5cLs8PpQ7GlDuaEBVvRvV9W5U17lxsc6FOpcXDW4vBACvT6Cqzo2qOjeA+i7/bK9P4ILThQtOFwBnp5+n10mQAAj4f/d8ApAkIM5oQJzZgDiT/2tidBSS4kxIijdhYJwJA+KMiDUaEG3UwxylR3SUHuYoHcxR+qabDmaDngGMiMKKJIRQdavgrKwsXH/99Xj11VcBAD6fD6mpqXjiiSfw9NNPX9J+xowZcDqdeP/995X7brjhBmRmZmL9+vUQQiAlJQVPPvkkFi1aBACorq6G1WrFhg0bcN999+Gbb77BNddcg88//xwTJkwAABQVFeGOO+7Ad999h5SUlMv22+FwwGKxoLq6GgkJwV3q7fb68PUZB17/5Bj++8uzAIAFt2XgZ//nqss+t+XGhxRehBBweX1ocPvQ4Pai0e1Fg8cHj1fAKwQkAKYoHUx6HYwGPYwGHQw6yX/T66DXSfAJgXp5lMntQV2jFzWNHtTUu+Fo8KCmwY2aBg8cDW7Uubx9+v76xfiD1cB4k/JVHs0a1PQ13uwPWtFR+jb3mhJCwOvzXw+vz3/z+QCPzz+S5mm6z+0N/LP/qw9ub+CfPS3+7GnxZ0kCdJIEvc5/kyQJekmCXocW30uQJH+w1EtNbXSdbCO1uF9uIzX/LDmg+oSATwgI5XvA5/P/2aCXYGga/Ytq+l2Iavo9IKL2dfbzW9URJJfLhT179mDp0qXKfTqdDjk5OSgpKWnzOSUlJcjPzw+4Lzc3F1u3bgUAHD9+HHa7HTk5OcrjFosFWVlZKCkpwX333YeSkhIkJiYq4QgAcnJyoNPpUFpairvuuuuSn9vY2IjGxuapkerqagD+Cx1Mi7ccwI5DFWh0+88Ci9JLWD5tJO4Zb+vUz6pz1gS1P9T3ogBE6YF4vXyP1HST+Zpuzd96PICn6VEDgASD/4ZoHQBj0y2QaBEy5JuQf5IEZTTJ7RFodPvQ4PWHtnqXF06XB7UNXtQ2elDn8sDl9cHj9QcQd1MQkcOG7EIjcOEicKiT18Ef/CR4BSCaQpG6/zsXHnQSoNfrEKUDDDodovQ6dJSZpBZBTidJ0EmATic1fy9JLQJac1gTaLrP1zziKJrub/578n8j/1m+W0Lzz9DrJOh0/oCo00kBQdHfP/lfgKR8D/mxFo/LP7f5R4sW97Xul7+vaPl4iz7K7wMtXk+0fj3R9uvJ11O+jnqp6Vrqmq+n/L4v6Qtavl6L79u4U7TZruPHW/+8S+9v43Xa6Idoox9oEeK9Pvl3RcDb9Pshtfrdau9/DvRS4LVblHsVxqb2a7O/3SV/ll5ufEjVgHT+/Hl4vV5YrdaA+61WK7799ts2n2O329tsb7fblcfl+zpq03r6zmAwoH///kqb1goLC/HMM89ccn9qau9Pe/2/VcD/6/WfQkREFFr+8mTvvXZNTQ0slvb3FVS9BilcLF26NGDkyufzobKyEgMGDFB9xZDD4UBqaipOnToV9Ok+LeN16x5et+7hdeseXrfu4XVrnxACNTU1ly2nUTUgJSUlQa/Xo7y8POD+8vJy2Gy2Np9js9k6bC9/LS8vR3JyckCbzMxMpU1FRUXAa3g8HlRWVrb7c00mE0ymwD1sEhMTO36DfSwhIYH/ELqB1617eN26h9ete3jduofXrW0djRzJVD1x02g0Yvz48SguLlbu8/l8KC4uRnZ2dpvPyc7ODmgPANu3b1fap6enw2azBbRxOBwoLS1V2mRnZ6Oqqgp79uxR2uzYsQM+nw9ZWVlBe39EREQUnlSfYsvPz8fs2bMxYcIETJw4EWvWrIHT6cScOXMAALNmzcLgwYNRWFgIAFiwYAGmTJmCVatWYdq0adi8eTN2796N11/3718sSRIWLlyI559/HhkZGcoy/5SUFOTl5QEARo4cialTp2LevHlYv3493G43Hn/8cdx3332dWsFGRERE2qZ6QJoxYwbOnTuHgoIC2O12ZGZmoqioSCmyLisrg07XPNA1adIkbNq0CcuXL8eyZcuQkZGBrVu3KnsgAcCSJUvgdDoxf/58VFVVYfLkySgqKlL2QAKAt956C48//jhuu+02ZaPItWvX9t0bDyKTyYSVK1deMgVIHeN16x5et+7hdeseXrfu4XXrOdX3QSIiIiIKNarWIBERERGFIgYkIiIiolYYkIiIiIhaYUAiIiIiaoUBKcytW7cOQ4cOhdlsRlZWFnbt2qV2l1Tz85//HFLTgaDybcSIEcrjDQ0NeOyxxzBgwADExcXhnnvuuWTT0bKyMkybNg0xMTEYNGgQFi9eDI/H0/pHhb1PPvkEP/zhD5GSkgJJkpSzDGVCCBQUFCA5ORnR0dHIycnB4cOHA9pUVlbigQceQEJCAhITEzF37lzU1tYGtPniiy9w0003wWw2IzU1FS+++GJvv7Vedbnr9tBDD13yOzh16tSANpF23QoLC3H99dcjPj4egwYNQl5eHg4dCjyRL1j/Nj/++GOMGzcOJpMJw4cPx4YNG3r77fWazly3W2655ZLft0ceeSSgTaRdt6ASFLY2b94sjEajeOONN8RXX30l5s2bJxITE0V5ebnaXVPFypUrxbXXXivOnj2r3M6dO6c8/sgjj4jU1FRRXFwsdu/eLW644QYxadIk5XGPxyNGjRolcnJyxL59+8S2bdtEUlKSWLp0qRpvp1dt27ZN/Ou//qt49913BQDx5z//OeDxF154QVgsFrF161Zx4MABceedd4r09HRRX1+vtJk6daoYO3as+Oyzz8T//u//iuHDh4uZM2cqj1dXVwur1SoeeOABcfDgQfH222+L6Oho8Zvf/Kav3mbQXe66zZ49W0ydOjXgd7CysjKgTaRdt9zcXPHmm2+KgwcPiv3794s77rhDpKWlidraWqVNMP5tHjt2TMTExIj8/Hzx9ddfi1deeUXo9XpRVFTUp+83WDpz3aZMmSLmzZsX8PtWXV2tPB6J1y2YGJDC2MSJE8Vjjz2m/Nnr9YqUlBRRWFioYq/Us3LlSjF27Ng2H6uqqhJRUVFiy5Ytyn3ffPONACBKSkqEEP4PP51OJ+x2u9LmtddeEwkJCaKxsbFX+66m1h/0Pp9P2Gw28dJLLyn3VVVVCZPJJN5++20hhBBff/21ACA+//xzpc0HH3wgJEkSp0+fFkII8e///u+iX79+AdfuqaeeEldffXUvv6O+0V5Amj59ervP4XUToqKiQgAQf//734UQwfu3uWTJEnHttdcG/KwZM2aI3Nzc3n5LfaL1dRPCH5AWLFjQ7nN43XqGU2xhyuVyYc+ePcjJyVHu0+l0yMnJQUlJiYo9U9fhw4eRkpKCK6+8Eg888ADKysoAAHv27IHb7Q64XiNGjEBaWppyvUpKSjB69Ghlk1IAyM3NhcPhwFdffdW3b0RFx48fh91uD7hWFosFWVlZAdcqMTEREyZMUNrk5ORAp9OhtLRUaXPzzTfDaDQqbXJzc3Ho0CFcvHixj95N3/v4448xaNAgXH311Xj00Udx4cIF5TFeN6C6uhoA0L9/fwDB+7dZUlIS8BpyG63897D1dZO99dZbSEpKwqhRo7B06VLU1dUpj/G69YzqO2lT95w/fx5erzfgFx8ArFYrvv32W5V6pa6srCxs2LABV199Nc6ePYtnnnkGN910Ew4ePAi73Q6j0XjJAcNWqxV2ux0AYLfb27ye8mORQn6vbV2Lltdq0KBBAY8bDAb0798/oE16evolryE/1q9fv17pv5qmTp2Ku+++G+np6Th69CiWLVuG22+/HSUlJdDr9RF/3Xw+HxYuXIgbb7xROf0gWP8222vjcDhQX1+P6Ojo3nhLfaKt6wYA999/P4YMGYKUlBR88cUXeOqpp3Do0CG8++67AHjdeooBiTTj9ttvV74fM2YMsrKyMGTIEPzhD3+I6H/k1Hfuu+8+5fvRo0djzJgxGDZsGD7++GPcdtttKvYsNDz22GM4ePAgPv30U7W7Elbau27z589Xvh89ejSSk5Nx22234ejRoxg2bFhfd1NzOMUWppKSkqDX6y9Z6VFeXg6bzaZSr0JLYmIirrrqKhw5cgQ2mw0ulwtVVVUBbVpeL5vN1ub1lB+LFPJ77eh3y2azoaKiIuBxj8eDyspKXs8WrrzySiQlJeHIkSMAIvu6Pf7443j//ffx0Ucf4YorrlDuD9a/zfbaJCQkhPX/ILV33dqSlZUFAAG/b5F63YKBASlMGY1GjB8/HsXFxcp9Pp8PxcXFyM7OVrFnoaO2thZHjx5FcnIyxo8fj6ioqIDrdejQIZSVlSnXKzs7G19++WXAB9j27duRkJCAa665ps/7r5b09HTYbLaAa+VwOFBaWhpwraqqqrBnzx6lzY4dO+Dz+ZT/SGdnZ+OTTz6B2+1W2mzfvh1XX311WE8TdcV3332HCxcuIDk5GUBkXjchBB5//HH8+c9/xo4dOy6ZPgzWv83s7OyA15DbhOt/Dy933dqyf/9+AAj4fYu06xZUaleJU/dt3rxZmEwmsWHDBvH111+L+fPni8TExIAVC5HkySefFB9//LE4fvy4+Mc//iFycnJEUlKSqKioEEL4lxKnpaWJHTt2iN27d4vs7GyRnZ2tPF9eEvv9739f7N+/XxQVFYmBAwdqcpl/TU2N2Ldvn9i3b58AIFavXi327dsnTp48KYTwL/NPTEwU7733nvjiiy/E9OnT21zmf91114nS0lLx6aefioyMjIDl6lVVVcJqtYoHH3xQHDx4UGzevFnExMSE7XJ1ITq+bjU1NWLRokWipKREHD9+XPzP//yPGDdunMjIyBANDQ3Ka0TadXv00UeFxWIRH3/8ccBy9Lq6OqVNMP5tysvVFy9eLL755huxbt26sF6ufrnrduTIEfHss8+K3bt3i+PHj4v33ntPXHnlleLmm29WXiMSr1swMSCFuVdeeUWkpaUJo9EoJk6cKD777DO1u6SaGTNmiOTkZGE0GsXgwYPFjBkzxJEjR5TH6+vrxU9+8hPRr18/ERMTI+666y5x9uzZgNc4ceKEuP3220V0dLRISkoSTz75pHC73X39VnrdRx99JABccps9e7YQwr/Uf8WKFcJqtQqTySRuu+02cejQoYDXuHDhgpg5c6aIi4sTCQkJYs6cOaKmpiagzYEDB8TkyZOFyWQSgwcPFi+88EJfvcVe0dF1q6urE9///vfFwIEDRVRUlBgyZIiYN2/eJf/DEmnXra3rBUC8+eabSptg/dv86KOPRGZmpjAajeLKK68M+Bnh5nLXraysTNx8882if//+wmQyieHDh4vFixcH7IMkRORdt2CShBCi78ariIiIiEIfa5CIiIiIWmFAIiIiImqFAYmIiIioFQYkIiIiolYYkIiIiIhaYUAiIiIiaoUBiYiIiKgVBiQiIiKiVhiQiIiIiFphQCIiIiJqhQGJiIiIqBUGJCIiIqJW/j+Ej67sy0B+zQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# check distribution of the answers' length\n",
    "doc_lengths = []\n",
    "\n",
    "for howto in howtos:\n",
    "\n",
    "    # get rough token count distribution\n",
    "    tokens = nltk.word_tokenize(howto)\n",
    "\n",
    "    doc_lengths.append(len(tokens))\n",
    "\n",
    "doc_lengths = np.array(doc_lengths)\n",
    "\n",
    "sns.distplot(doc_lengths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "831c9429",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4680344385418575"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The distribution of answers with the length of more than 128\n",
    "len(doc_lengths[doc_lengths > 128])/len(doc_lengths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "dde20559",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "172.86334493496977"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.average(doc_lengths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "04ce2135",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    }
   ],
   "source": [
    "# Load the GPT tokenizer and add special tokens.\n",
    "tokenizer = GPT2Tokenizer.from_pretrained('gpt2', \n",
    "                                          bos_token='<|startoftext|>', \n",
    "                                          eos_token='<|endoftext|>', \n",
    "                                          pad_token='<|pad|>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f96f2fc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7968530f",
   "metadata": {},
   "source": [
    "Defining a PyTorch dataset class called GPT2Dataset that prepares text data for use with a GPT-2 language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0fb98bd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "class GPT2Dataset(Dataset):\n",
    "    # Define the class constructor method\n",
    "    def __init__(self, txt_list, tokenizer, gpt2_type=\"gpt2\", max_length=512):\n",
    "        \n",
    "        # Store the tokenizer and initialize the input_ids and attn_masks lists\n",
    "        self.tokenizer = tokenizer\n",
    "        self.input_ids = []\n",
    "        self.attn_masks = []\n",
    "        \n",
    "        # Iterate through each text sample in txt_list\n",
    "        for txt in txt_list:\n",
    "            \n",
    "            # Tokenize the text sample using the tokenizer and truncate it if necessary\n",
    "            text = txt\n",
    "            encodings_dict = tokenizer('<|startoftext|>'+ text + '<|endoftext|>', truncation=True, max_length=max_length, padding=\"max_length\")\n",
    "            \n",
    "            # Add the tokenized input and attention mask to their respective lists\n",
    "            self.input_ids.append(torch.tensor(encodings_dict['input_ids']))\n",
    "            self.attn_masks.append(torch.tensor(encodings_dict['attention_mask']))\n",
    "            \n",
    "    # Define the __len__ method, which returns the number of input samples in the dataset\n",
    "    def __len__(self):\n",
    "        return len(self.input_ids)\n",
    "    \n",
    "    # Define the __getitem__ method, which returns the input sample and corresponding attention mask at a given index\n",
    "    def __getitem__(self, idx):\n",
    "        return self.input_ids[idx], self.attn_masks[idx] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9ba905ee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4,367 training samples\n",
      "1,092 validation samples\n"
     ]
    }
   ],
   "source": [
    "dataset = GPT2Dataset(howtos, tokenizer, max_length=512)\n",
    "\n",
    "# Split the dataset into training and validation sets with a 80/20 split\n",
    "train_size = int(0.8 * len(dataset))\n",
    "val_size = len(dataset) - train_size\n",
    "\n",
    "train_dataset, val_dataset = random_split(dataset, [train_size, val_size])\n",
    "\n",
    "print('{:>5,} training samples'.format(train_size))\n",
    "print('{:>5,} validation samples'.format(val_size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "77215b04",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a PyTorch DataLoader instance for the training set \n",
    "train_dataloader = DataLoader(\n",
    "            train_dataset,  # The training dataset\n",
    "            sampler = RandomSampler(train_dataset), # Randomly sample elements from the training dataset\n",
    "            batch_size = batch_size # The number of samples per batch to load\n",
    "        )\n",
    "\n",
    "# Create a PyTorch DataLoader instance for the validation set\n",
    "validation_dataloader = DataLoader(\n",
    "            val_dataset, # The validation dataset\n",
    "            sampler = SequentialSampler(val_dataset), # Iterate through the validation dataset sequentially\n",
    "            batch_size = batch_size # The number of samples per batch to load\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1faf320b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedding(50259, 768)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the GPT2Config from the 'gpt2' pre-trained model and set output_hidden_states to False\n",
    "configuration = GPT2Config.from_pretrained('gpt2', \n",
    "                                           output_hidden_states=False)\n",
    "# Instantiate the GPT2LMHeadModel with the 'gpt2' pre-trained weights and the configuration we just loaded\n",
    "model = GPT2LMHeadModel.from_pretrained(\"gpt2\", \n",
    "                                        config=configuration)\n",
    "\n",
    "# Resize the model's token embeddings to match the size of the tokenizer's vocabulary\n",
    "# This step is necessary if any special tokens have been added to the tokenizer\n",
    "model.resize_token_embeddings(len(tokenizer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6a000aaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tell pytorch to run this model on the GPU.\n",
    "device = torch.device(\"cuda\")\n",
    "model.cuda()\n",
    "\n",
    "# Set the seed value all over the place to make this reproducible.\n",
    "seed_val = 42\n",
    "\n",
    "# Set the random seed for the Python built-in `random` module.\n",
    "random.seed(seed_val)\n",
    "np.random.seed(seed_val)\n",
    "\n",
    "# Set the random seed for PyTorch to ensure reproducibility on the GPU.\n",
    "torch.manual_seed(seed_val)\n",
    "torch.cuda.manual_seed_all(seed_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "35220999",
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 2\n",
    "learning_rate = 3e-4\n",
    "warmup_steps = 1e2\n",
    "epsilon = 1e-5\n",
    "check_path = './checkpoints'\n",
    "sample_every = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b95b4b24",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "# Define the optimizer to use.\n",
    "# We're using the AdamW optimizer with the given learning rate, epsilon, and weight decay.\n",
    "# We're passing the model's parameters to the optimizer so that it can update them during training.\n",
    "optimizer = optim.AdamW(model.parameters(),\n",
    "                        lr=learning_rate,\n",
    "                        eps=epsilon,\n",
    "                       weight_decay = 0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "39716289",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the total number of training steps.\n",
    "# This is the number of batches in the training data multiplied by the number of epochs.\n",
    "total_steps = len(train_dataloader) * epochs\n",
    "\n",
    "# Set up the learning rate scheduler.\n",
    "# We're using the linear scheduler with warmup, which increases the learning rate\n",
    "# linearly from 0 to the initial learning rate over the first few training steps\n",
    "# (the warmup steps), and then decreases the learning rate linearly to zero over\n",
    "# the remaining training steps.\n",
    "scheduler = get_linear_schedule_with_warmup(optimizer, \n",
    "                                            num_warmup_steps = warmup_steps, \n",
    "                                            num_training_steps = total_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "cc40a577",
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_time(elapsed):\n",
    "    return str(datetime.timedelta(seconds=int(round((elapsed)))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e08426d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Release all unused memory that is held by the GPU memory cache\n",
    "torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "24c50cb1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "======== Epoch 1 / 2 ========\n",
      "Training...\n",
      "\n",
      "  Average training loss: 1.52\n",
      "  Training epoch took: 0:03:45\n",
      "\n",
      "Running Validation...\n",
      "  Validation Loss: 1.06\n",
      "  Validation took: 0:00:14\n",
      "\n",
      "======== Epoch 2 / 2 ========\n",
      "Training...\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_3431662/1494300529.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     74\u001b[0m         \u001b[0;31m# Backpropagate the loss and update the model parameters.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     75\u001b[0m         \u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 76\u001b[0;31m         \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     77\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     78\u001b[0m         \u001b[0;31m# Update the learning rate scheduler.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/cuda_tf2/lib/python3.7/site-packages/torch/optim/lr_scheduler.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     66\u001b[0m                 \u001b[0minstance\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_step_count\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     67\u001b[0m                 \u001b[0mwrapped\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__get__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minstance\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcls\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 68\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mwrapped\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     69\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     70\u001b[0m             \u001b[0;31m# Note that the returned function here is no longer a bound method,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/cuda_tf2/lib/python3.7/site-packages/torch/optim/optimizer.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    138\u001b[0m                 \u001b[0mprofile_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"Optimizer.step#{}.step\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    139\u001b[0m                 \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprofiler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecord_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprofile_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 140\u001b[0;31m                     \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    141\u001b[0m                     \u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_optimizer_step_code\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    142\u001b[0m                     \u001b[0;32mreturn\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/cuda_tf2/lib/python3.7/site-packages/torch/autograd/grad_mode.py\u001b[0m in \u001b[0;36mdecorate_context\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     25\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mdecorate_context\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclone\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 27\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     28\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mF\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdecorate_context\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     29\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/cuda_tf2/lib/python3.7/site-packages/torch/optim/adamw.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, closure)\u001b[0m\n\u001b[1;32m    174\u001b[0m                   \u001b[0mmaximize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'maximize'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    175\u001b[0m                   \u001b[0mforeach\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'foreach'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 176\u001b[0;31m                   capturable=group['capturable'])\n\u001b[0m\u001b[1;32m    177\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    178\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/cuda_tf2/lib/python3.7/site-packages/torch/optim/adamw.py\u001b[0m in \u001b[0;36madamw\u001b[0;34m(params, grads, exp_avgs, exp_avg_sqs, max_exp_avg_sqs, state_steps, foreach, capturable, amsgrad, beta1, beta2, lr, weight_decay, eps, maximize)\u001b[0m\n\u001b[1;32m    230\u001b[0m          \u001b[0meps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0meps\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    231\u001b[0m          \u001b[0mmaximize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmaximize\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 232\u001b[0;31m          capturable=capturable)\n\u001b[0m\u001b[1;32m    233\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    234\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/cuda_tf2/lib/python3.7/site-packages/torch/optim/adamw.py\u001b[0m in \u001b[0;36m_single_tensor_adamw\u001b[0;34m(params, grads, exp_avgs, exp_avg_sqs, max_exp_avg_sqs, state_steps, amsgrad, beta1, beta2, lr, weight_decay, eps, maximize, capturable)\u001b[0m\n\u001b[1;32m    268\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    269\u001b[0m         \u001b[0;31m# Perform stepweight decay\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 270\u001b[0;31m         \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmul_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mlr\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mweight_decay\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    271\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    272\u001b[0m         \u001b[0;31m# Decay the first and second moment running average coefficient\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "total_t0 = time.time()\n",
    "\n",
    "training_stats = []\n",
    "\n",
    "model = model.to(device)\n",
    "\n",
    "for epoch_i in range(0, epochs):\n",
    "\n",
    "    # ========================================\n",
    "    #               Training\n",
    "    # ========================================\n",
    "\n",
    "    print(\"\")\n",
    "    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))\n",
    "    print('Training...')\n",
    "\n",
    "    # Record the start time of the epoch.\n",
    "    t0 = time.time()\n",
    "\n",
    "    # Reset the total loss for this epoch.\n",
    "    total_train_loss = 0\n",
    "\n",
    "    # Put the model into training mode.\n",
    "    model.train()\n",
    "\n",
    "    # Loop over each batch of training data.\n",
    "    for step, batch in enumerate(train_dataloader):\n",
    "\n",
    "        # Get the input IDs, attention masks, and labels for this batch,\n",
    "        # and move them to the GPU if necessary.\n",
    "        b_input_ids = batch[0].to(device)\n",
    "        b_labels = batch[0].to(device)\n",
    "        b_masks = batch[1].to(device)\n",
    "\n",
    "        # Zero out any gradients that have accumulated from previous batches.\n",
    "        model.zero_grad()        \n",
    "\n",
    "        # Run this batch through the model to get the loss.\n",
    "        outputs = model(  b_input_ids,\n",
    "                          labels=b_labels, \n",
    "                          attention_mask = b_masks,\n",
    "                          token_type_ids=None\n",
    "                        )\n",
    "\n",
    "        loss = outputs[0]  \n",
    "\n",
    "        # Get the loss for this batch, and add it to the total loss for this epoch.\n",
    "        batch_loss = loss.item()\n",
    "        total_train_loss += batch_loss\n",
    "\n",
    "        # Print some generated output every `sample_every` batches.\n",
    "        if step % sample_every == 0 and not step == 0:\n",
    "            \n",
    "            # Measure how long this has taken so far.\n",
    "            elapsed = format_time(time.time() - t0)\n",
    "\n",
    "            # Put the model into evaluation mode.\n",
    "            model.eval()\n",
    "\n",
    "            # Generate some sample output from the model.\n",
    "            sample_outputs = model.generate(\n",
    "                                    bos_token_id=random.randint(1,30000),\n",
    "                                    do_sample=True,   \n",
    "                                    top_k=50, \n",
    "                                    max_length = 512,\n",
    "                                    top_p=0.95, \n",
    "                                    num_return_sequences=1,\n",
    "                                    pad_token_id=tokenizer.eos_token_id,\n",
    "                                    eos_token_id=-1,\n",
    "                                )\n",
    "            # Put the model back into training mode.\n",
    "            model.train()\n",
    "\n",
    "        # Backpropagate the loss and update the model parameters.\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        # Update the learning rate scheduler.\n",
    "        scheduler.step()\n",
    "\n",
    "    # Calculate the average loss over all of the batches.\n",
    "    avg_train_loss = total_train_loss / len(train_dataloader)       \n",
    "    \n",
    "    # Measure how long this epoch took.\n",
    "    training_time = format_time(time.time() - t0)\n",
    "\n",
    "    print(\"\")\n",
    "    print(\"  Average training loss: {0:.2f}\".format(avg_train_loss))\n",
    "    print(\"  Training epoch took: {:}\".format(training_time))\n",
    "        \n",
    "    # ========================================\n",
    "    #               Validation\n",
    "    # ========================================\n",
    "\n",
    "    print(\"\")\n",
    "    print(\"Running Validation...\")\n",
    "    # Perform validation on the model after training for each epoch.\n",
    "    t0 = time.time()\n",
    "\n",
    "    model.eval()\n",
    "\n",
    "    total_eval_loss = 0\n",
    "    nb_eval_steps = 0\n",
    "\n",
    "    # Evaluate data for one epoch\n",
    "    for batch in validation_dataloader:\n",
    "        \n",
    "        b_input_ids = batch[0].to(device)\n",
    "        b_labels = batch[0].to(device)\n",
    "        b_masks = batch[1].to(device)\n",
    "        \n",
    "        with torch.no_grad():        \n",
    "\n",
    "            outputs  = model(b_input_ids, \n",
    "#                            token_type_ids=None, \n",
    "                             attention_mask = b_masks,\n",
    "                            labels=b_labels)\n",
    "          \n",
    "            loss = outputs[0]  \n",
    "            \n",
    "        batch_loss = loss.item()\n",
    "        total_eval_loss += batch_loss        \n",
    "\n",
    "    avg_val_loss = total_eval_loss / len(validation_dataloader)\n",
    "    \n",
    "    validation_time = format_time(time.time() - t0)    \n",
    "\n",
    "    print(\"  Validation Loss: {0:.2f}\".format(avg_val_loss))\n",
    "    print(\"  Validation took: {:}\".format(validation_time))\n",
    "\n",
    "    # Record all statistics from this epoch.\n",
    "    training_stats.append(\n",
    "        {\n",
    "            'epoch': epoch_i + 1,\n",
    "            'Training Loss': avg_train_loss,\n",
    "            'Valid. Loss': avg_val_loss,\n",
    "            'Training Time': training_time,\n",
    "            'Validation Time': validation_time\n",
    "        }\n",
    "    )\n",
    "\n",
    "print(\"\")\n",
    "print(\"Training complete!\")\n",
    "print(\"Total training took {:} (h:mm:ss)\".format(format_time(time.time()-total_t0)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d29c28d",
   "metadata": {},
   "source": [
    "The increasing validation loss shows that the model is overfitting to the training data. Although the model is getting better at predicting the training data but cannot generalize well to the new data, which is the validation data.\n",
    "The reason is model is too complex, and it is able to learn the noise in the training data as well, which is not present in the validation data. We need more data for that. Let's see if we can improve it or not?!</p>\n",
    "(P.S. I have tried this configuration with full dataset and the results were satisfying. Since it was supposed to be small notebook, I shortened the dataset from 600.000 questions to less than 5500 samples.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3b081ac",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Display floats with two decimal places.\n",
    "pd.set_option('precision', 2)\n",
    "\n",
    "# Create a DataFrame from our training statistics.\n",
    "df_stats = pd.DataFrame(data=training_stats)\n",
    "\n",
    "# Use the 'epoch' as the row index.\n",
    "df_stats = df_stats.set_index('epoch')\n",
    "\n",
    "# Display the table.\n",
    "df_stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccb9e90a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use plot styling from seaborn.\n",
    "sns.set(style='darkgrid')\n",
    "\n",
    "# Set the plot size and font size.\n",
    "sns.set(font_scale=1.5)\n",
    "plt.rcParams[\"figure.figsize\"] = (12,6)\n",
    "\n",
    "# Plot the learning curve.\n",
    "plt.plot(df_stats['Training Loss'], 'b-o', label=\"Training\")\n",
    "plt.plot(df_stats['Valid. Loss'], 'g-o', label=\"Validation\")\n",
    "\n",
    "# Label the plot.\n",
    "plt.title(\"Training & Validation Loss\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()\n",
    "plt.xticks([1, 2, 3, 4])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d0fc956",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve a list of all the named parameters in the model\n",
    "params = list(model.named_parameters())\n",
    "\n",
    "# Print the total number of named parameters in the GPT-2 model\n",
    "print('The GPT-2 model has {:} different named parameters.\\n'.format(len(params)))\n",
    "\n",
    "# Print the size of the parameters in the embedding layer\n",
    "print('==== Embedding Layer ====\\n')\n",
    "\n",
    "for p in params[0:2]:\n",
    "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))\n",
    "\n",
    "# Print the size of the parameters in the first transformer block\n",
    "print('\\n==== First Transformer ====\\n')\n",
    "\n",
    "for p in params[2:14]:\n",
    "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))\n",
    "\n",
    "# Print the size of the parameters in the output layer\n",
    "print('\\n==== Output Layer ====\\n')\n",
    "\n",
    "for p in params[-2:]:\n",
    "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c716d5e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sets the directory to which the trained model will be saved\n",
    "output_dir = './models'\n",
    "\n",
    "# If the directory does not exist, it is created\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)\n",
    "\n",
    "#Prints the directory where the model will be saved\n",
    "print(\"Saving model to %s\" % output_dir)\n",
    "\n",
    "# This line ensures that the correct model is saved even if the code is running in a distributed/parallel environment\n",
    "model_to_save = model.module if hasattr(model, 'module') else model  # Take care of distributed/parallel training\n",
    "# Saves the trained model to the specified directory\n",
    "model_to_save.save_pretrained(output_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad0732d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "export_path = './models'\n",
    "torch.save(model, f'{export_path}/gpt2_v2.pt')\n",
    "tokenizer.save_pretrained(export_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2b6b44d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# This cell is just written for those who wants to adjust the number of responses and see the result for just one question \n",
    "model.eval()\n",
    "input_prompt = \"How to draw a circle\"\n",
    "\n",
    "prompt = f\"\\n<|startoftext|>[WP] {input_prompt} \\n[RESPONSE]\"\n",
    "\n",
    "# The input prompt is tokenized and converted to a tensor, which is then sent to the GPU.\n",
    "generated = torch.tensor(tokenizer.encode(prompt)).unsqueeze(0)\n",
    "generated = generated.to(device)\n",
    "\n",
    "# Generates 3 responses to a given input prompt using a pre-trained language model.\n",
    "sample_outputs = model.generate(\n",
    "                                generated, \n",
    "                                do_sample=True,   \n",
    "                                top_k=50, \n",
    "                                max_length = 512,\n",
    "                                top_p=0.95, \n",
    "                                num_return_sequences=3\n",
    "                                )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01cb8afb",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = pd.read_csv('data_test.csv')\n",
    "test_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b00e2f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generates the answer for the data test.\n",
    "model.eval()\n",
    "def generate_ans(question: str):\n",
    "    responses = []\n",
    "    input_prompt = question\n",
    "    prompt = f\"\\n<|startoftext|>[WP] {input_prompt} \\n[RESPONSE]\"\n",
    "\n",
    "    generated = torch.tensor(tokenizer.encode(prompt)).unsqueeze(0)\n",
    "    generated = generated.to(device)\n",
    "\n",
    "    sample_outputs = model.generate(\n",
    "                                    generated, \n",
    "                                    do_sample=True,   \n",
    "                                    top_k=50, \n",
    "                                    max_length = 512,\n",
    "                                    top_p=0.95, \n",
    "                                    num_return_sequences=2\n",
    "                                    )\n",
    "    for i, sample_output in enumerate(sample_outputs):\n",
    "        response_text = tokenizer.decode(sample_output, skip_special_tokens=True)\n",
    "        wp_responses = re.split(r\"\\[WP\\].*?\\n|\\[RESPONSE\\]\", response_text)[1:]\n",
    "        new_list = [elem for elem in wp_responses if elem != '']\n",
    "        responses.append(new_list[0])\n",
    "    return responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da9ec703",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Creates two arrays and store the first answer to answers1 and second answer for each question to answers2\n",
    "import re\n",
    "from tqdm import tqdm\n",
    "answers1 = []\n",
    "answers2 = []\n",
    "for _, row in tqdm(test_df.iterrows()):\n",
    "    _, question, _ = row\n",
    "    answer_gpt = generate_ans(question)\n",
    "    answers1.append(answer_gpt[0])\n",
    "    answers2.append(answer_gpt[1])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48a1ec35",
   "metadata": {},
   "source": [
    "BERTScore metric for evaluating the quality of generated text.It takes two lists of strings, `cands` and `refs`, where cands contains the generated text and refs contains the reference or ground-truth text. It returns three tensors: P, R, and F1, where P is precision, R is recall, and F1 is the harmonic mean of precision and recall. Each tensor has the same shape as the input cands and refs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cccb5f62",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Evaluate the generated answers\n",
    "import bert_score\n",
    "from evaluate import load\n",
    "bertscore = load('bertscore')\n",
    "\n",
    "bertscore_gpt2 = bertscore.compute(predictions=answers1, \n",
    "                                           references=test_df['answer'].to_list(), \n",
    "                                           lang='en')['f1']\n",
    "bertscore_gpt_2 = bertscore.compute(predictions=answers2, \n",
    "                                           references=test_df['answer'].to_list(), \n",
    "                                           lang='en')['f1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f48bea8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "bertscore_gpt2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a88555f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the generated answers and their scores   \n",
    "test_df['first_gpt2_answer'] = answers1\n",
    "test_df['second_gpt2_answer'] = answers2\n",
    "\n",
    "test_df['bert_score_first_gpt2'] = bertscore_gpt2\n",
    "test_df['bert_score_second_gpt2'] = bertscore_gpt_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0145cd8b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "test_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bdb6e9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean(test_df['bert_score_first_gpt2'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdaf92d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean(test_df['bert_score_second_gpt2'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d97f425",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df.to_csv('./data/eval_results.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd3aea8e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "results_df = pd.read_csv('./data/eval_results.csv')\n",
    "results_df.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76076274",
   "metadata": {},
   "outputs": [],
   "source": [
    "# creates a dataset dictionary containing training examples and labels for a binary classification task. \n",
    "# The goal of the classification task is to predict which of two generated responses is \n",
    "# better in terms of BERTScore.\n",
    "dataset = {}\n",
    "for _, row in results_df.iterrows():\n",
    "    _, _,_, answers1, answers2, bertscore_gpt2, bertscore_gpt2_2 = row\n",
    "    if bertscore_gpt2 > bertscore_gpt2_2:\n",
    "        dataset[answers1] = 1\n",
    "        dataset[answers2] = 0\n",
    "    elif bertscore_gpt2 < bertscore_gpt2_2:\n",
    "        dataset[answers1] = 0\n",
    "        dataset[answers2] = 1\n",
    "    else:\n",
    "        len_1 = len(answers1)\n",
    "        len_2 = len(answers2)\n",
    "        if len_1 < len_2:\n",
    "            dataset[answers1] = 1\n",
    "            dataset[answers2] = 0\n",
    "        elif len_1 > len_2:\n",
    "            dataset[answers1] = 0\n",
    "            dataset[answers2] = 1\n",
    "        else:\n",
    "            dataset[answers1] = 1\n",
    "            dataset[answers2] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d7c91d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_dataset = pd.DataFrame(list(dataset.items()), columns=['response', 'label'])\n",
    "clf_dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "939a4a47",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_dataset.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1dc5694e",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_dataset = clf_dataset.sample(frac=1).reset_index(drop=True)\n",
    "clf_dataset.to_csv('./data/clf_dataset.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46ceca08",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertForSequenceClassification\n",
    "from transformers import TrainingArguments\n",
    "from transformers import BertTokenizerFast\n",
    "from datasets import load_dataset\n",
    "from transformers import pipeline\n",
    "from transformers import set_seed\n",
    "from transformers import Trainer\n",
    "from datasets import DatasetDict\n",
    "import transformers\n",
    "import numpy as np\n",
    "import datasets\n",
    "import logging \n",
    "import torch\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ed1aa1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "N_GPUS = 1\n",
    "TRAIN_EPOCHS = 2\n",
    "TRAIN_BATCH_SIZE = 8\n",
    "EVAL_BATCH_SIZE = 8\n",
    "MAX_LEN = 128\n",
    "LOGGING_STEPS = 64\n",
    "SAVE_STEPS = 10240  # reduce it to a smaler value like 512 if you want to save checkpoints\n",
    "SAVE_TOTAL_LIMIT = 2\n",
    "\n",
    "set_seed(123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b981bd3b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# creates an instance of the BertForSequenceClassification class from the Hugging Face Transformers library.\n",
    "model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2,  force_download=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62f7dba7",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b78dd52",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = load_dataset('csv', \n",
    "                     data_files='./data/clf_dataset.csv', \n",
    "                     delimiter=',', \n",
    "                     split='train', \n",
    "                     cache_dir='/tmp/cache')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94d145cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_validation_test = data.train_test_split(shuffle=True, seed=123, test_size=0.1)\n",
    "data_splits = DatasetDict({'train': train_validation_test['train'],  \n",
    "                           'validation': train_validation_test['test']})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06ea87cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_function(examples):\n",
    "    return tokenizer(examples['response'], truncation=True, padding=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bb1c338",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenize datasets \n",
    "num_proc = int(os.cpu_count()/N_GPUS)\n",
    "tokenized_data = data_splits.map(preprocess_function, batched=True, num_proc=num_proc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c28fa717",
   "metadata": {},
   "source": [
    "# Finetune\n",
    "Define training hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45c70802",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set various training parameters such as the number of training epochs, batch size, and logging options.\n",
    "training_args = TrainingArguments(output_dir='./model', \n",
    "                                  overwrite_output_dir=True, \n",
    "                                  num_train_epochs=10,  \n",
    "                                  optim='adamw_torch', \n",
    "                                  save_strategy='steps', \n",
    "                                  evaluation_strategy='epoch',\n",
    "                                  per_device_train_batch_size=TRAIN_BATCH_SIZE, \n",
    "                                  per_device_eval_batch_size=EVAL_BATCH_SIZE, \n",
    "                                  warmup_steps=10, \n",
    "                                  weight_decay=0.01,\n",
    "                                  logging_steps=LOGGING_STEPS,\n",
    "                                  save_steps=SAVE_STEPS, \n",
    "                                  save_total_limit=SAVE_TOTAL_LIMIT,\n",
    "                                  logging_dir='logs')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77771316",
   "metadata": {},
   "source": [
    "### Training\n",
    "P.S: Since our dataset is small it has overfits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6ef9f3c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "trainer = Trainer(model=model, \n",
    "                  args=training_args, \n",
    "                  train_dataset=tokenized_data['train'], \n",
    "                  eval_dataset=tokenized_data['validation'], \n",
    "                  tokenizer=tokenizer)\n",
    "\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a7863de",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trl import AutoModelForCausalLMWithValueHead\n",
    "from transformers import GPT2Tokenizer\n",
    "from transformers import set_seed\n",
    "from datasets import load_dataset\n",
    "from transformers import pipeline\n",
    "from datasets import Dataset\n",
    "from random import choices\n",
    "from trl import PPOTrainer\n",
    "from trl import PPOConfig\n",
    "from evaluate import load\n",
    "from tqdm import tqdm\n",
    "import transformers \n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import bert_score\n",
    "import ipywidgets\n",
    "import datasets\n",
    "import evaluate\n",
    "import logging\n",
    "import jupyter\n",
    "import random\n",
    "import torch\n",
    "import wandb\n",
    "import time\n",
    "import trl\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61fc4b1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.display.max_colwidth = None\n",
    "np.random.seed(123)\n",
    "tqdm.pandas()\n",
    "set_seed(123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25cc2e44",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wandb login ee62107bfb3cfc4fb4a7087e8b9761437363721d"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be933cd3",
   "metadata": {},
   "source": [
    "### PPO\n",
    "Defining a configuration object for the Proximal Policy Optimization (PPO) algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a418b7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "config = PPOConfig(model_name='./model/finetuned', \n",
    "                   batch_size=4,\n",
    "                   learning_rate=1.41e-6,\n",
    "                   forward_batch_size=4, # This parameter is useful when using large models that don't fit into memory.\n",
    "                   remove_unused_columns=False, # All columns in the input data will be used during training.\n",
    "                   log_with='wandb'# For logging training metrics\n",
    "                  )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b0f5fbd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Provides an interface for using pre-trained models for language modeling. \n",
    "# This class can generate sequences of text based on a given prompt.\n",
    "# two pre-trained language models, loaded from the specified directories, \n",
    "# which will be used as the \"actor\" and \"critic\" in the PPO algorithm.\n",
    "# Note: for further improvement, we can use answer1 and create model and answer2 \n",
    "# and create another model and use them as actor critic. since the data is not big enough it is not a good idea\n",
    "active_model = AutoModelForCausalLMWithValueHead.from_pretrained('./models')\n",
    "ref_model = AutoModelForCausalLMWithValueHead.from_pretrained('./models')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "973b8b24",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = load_dataset('csv', \n",
    "                       data_files='./data_test.csv',  \n",
    "                       delimiter=',', \n",
    "                       split='train[:100%]',\n",
    "                       download_mode='force_redownload')\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccb12bb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(samples: list):\n",
    "    questions = samples['question']\n",
    "    ground_truth = samples['answer']\n",
    "    \n",
    "    input_ids = []\n",
    "    query = []\n",
    "    \n",
    "    for question in questions:\n",
    "        prompted_input = f'question: {question}\\nanswer:'\n",
    "        query.append(prompted_input)\n",
    "        tokenized_input = tokenizer(prompted_input, \n",
    "                                    truncation=True)\n",
    "        input_ids.append(torch.tensor(tokenized_input['input_ids'], dtype=torch.long))\n",
    "        \n",
    "    return {'input_ids': input_ids, 'query': query, 'ground_truth': ground_truth, 'questions': questions}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29e0bef6",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = dataset.map(tokenize, \n",
    "                      batched=True, \n",
    "                      #num_proc=num_proc, \n",
    "                      load_from_cache_file=False, \n",
    "                      remove_columns=['question', 'answer'])\n",
    "dataset.set_format('pt', \n",
    "                   columns=['input_ids', 'query', 'ground_truth'],\n",
    "                   output_all_columns=True)\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b282cb99",
   "metadata": {},
   "outputs": [],
   "source": [
    "def collator(dataset):\n",
    "    result = {}\n",
    "    for key in dataset[0]:\n",
    "        values = []\n",
    "        for d in dataset:\n",
    "            values.append(d[key])\n",
    "        result[key] = values\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7b586cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "997b4eb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# creates an instance of the PPOTrainer class\n",
    "ppo_trainer = PPOTrainer(config, active_model, ref_model, tokenizer, dataset=dataset, data_collator=collator)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1549a52d",
   "metadata": {},
   "source": [
    "pipeline: a high-level API provided by the transformers library, which makes it easy to use pre-trained models for a variety of natural language processing tasks, including text generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3daa291",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "bert_pipe = pipeline('text-generation', \n",
    "                     model='./models',\n",
    "                    max_length = MAX_LEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "640c966c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "prompt = \"How can I sum two integers in python?\"\n",
    "output = bert_pipe(prompt, max_length=512, num_return_sequences=1)\n",
    "output = output[0]['generated_text']\n",
    "response_text = output.split('[RESPONSE]')[1]\n",
    "formatted_text = \"```\\n\" + response_text + \"\\n```\"\n",
    "print(formatted_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b684e4ec",
   "metadata": {},
   "source": [
    "Tadda!! our model can now write quite reasonable python code! Enjoy :-)\n",
    "\n",
    "You can improve this by Just defining two models and use PPO trainer based on that. this way the model will be able to improve itself by comparing two responses and produce better results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4139fd8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "cuda_tf2",
   "language": "python",
   "name": "cuda_tf2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}