Data_Prep.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a8f78cd4-a4ec-4682-9766-3604fa1d1dca",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Data Prep\n",
    "This notebook extracts, preprocesses, and splits examples into training and test sets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eba19921-241e-4dc3-8e97-e5e5d63e9c7d",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Collect Data\n",
    "We extract sample sentences from the first half of the US English portion of the GloWbE corpus. After preprocessing, we search for instances where \"there\", \"their\", or \"theyre\" appears once."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bbe87e0-ea29-42ce-877b-e1bcd7e04857",
   "metadata": {},
   "source": [
    "### Data Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "25060e12-0eeb-44a3-b5f2-a564ee180ad2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "text = []\n",
    "with open('Euphemism Proj/glowbe_corpus/text_US_0-4.txt', encoding='utf8') as f:\n",
    "    for line in f:\n",
    "        for p in re.split(r'<p>', line):\n",
    "            p = p.strip()\n",
    "            text.append(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "c9cf2ee6-fbb4-4497-b38c-eb4555555a78",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 3711273 rows in df\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>##100040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Arts and cultural industries rake in an annual $1.2 billion and generate 19,500 jobs in Bernalillo County . UNM alumni in the sector are helping the university pave a smoother path for future arts managers and entrepreneurs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>##100060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>disabled or is not : About CDC.gov . &lt;h&gt; Cancer and Men</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Every year , cancer claims the lives of nearly 300,000 men in America . Men can reduce their risk for several of the most common kinds of cancer . &lt;h&gt; Lifestyle Changes &lt;h&gt; Tobacco Use and Secondhand Smoke</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711268</th>\n",
       "      <td>And I know from my experience with the boys that someday is coming fast ... inconceivably I will miss these nights of bad dreams and IN-adequate quality sleep .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711269</th>\n",
       "      <td>Its hard to imagine if you have n't parented through these years but trust me on this ... you will miss it all ... and it will gone before you know it .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711270</th>\n",
       "      <td>So I 'll have an extra cup of coffee and my eyes will be a little MORE puffy and I 'll struggle to stay awake in my meetings today but I will also thank the heavens above AGAIN that I have this amazing , wonderful privilege to be the ( exhausted ) mom of little ones again</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711271</th>\n",
       "      <td>##99914</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711272</th>\n",
       "      <td>Hilarious and heartwarming , this devilishly humorous show from the pens of the \" blaguards \" themselves , the great Frank McCourt ( Pulitzer Prize Winner , \" Angela 's Ashes \" ) and his brother Malachy ( \" A Monk Swimming \" ) delights . Liberally laced with music , parody and song , with a pinch of anarchy and a tall tale or two tossed in for good measure , this warm and irreverent comedy traces the brothers ' hilarious and unlikely journey from the streets of poverty-stricken Limerick to fame and fortune in</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3711273 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       text\n",
       "0        ##100040                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
       "1        Arts and cultural industries rake in an annual $1.2 billion and generate 19,500 jobs in Bernalillo County . UNM alumni in the sector are helping the university pave a smoother path for future arts managers and entrepreneurs                                                                                                                                                                                                                                                                                                   \n",
       "2        ##100060                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
       "3        disabled or is not : About CDC.gov . <h> Cancer and Men                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "4        Every year , cancer claims the lives of nearly 300,000 men in America . Men can reduce their risk for several of the most common kinds of cancer . <h> Lifestyle Changes <h> Tobacco Use and Secondhand Smoke                                                                                                                                                                                                                                                                                                                     \n",
       "...                                                                                                                                                                                                                ...                                                                                                                                                                                                                                                                                                                     \n",
       "3711268  And I know from my experience with the boys that someday is coming fast ... inconceivably I will miss these nights of bad dreams and IN-adequate quality sleep .                                                                                                                                                                                                                                                                                                                                                                  \n",
       "3711269  Its hard to imagine if you have n't parented through these years but trust me on this ... you will miss it all ... and it will gone before you know it .                                                                                                                                                                                                                                                                                                                                                                          \n",
       "3711270  So I 'll have an extra cup of coffee and my eyes will be a little MORE puffy and I 'll struggle to stay awake in my meetings today but I will also thank the heavens above AGAIN that I have this amazing , wonderful privilege to be the ( exhausted ) mom of little ones again                                                                                                                                                                                                                                                  \n",
       "3711271  ##99914                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           \n",
       "3711272  Hilarious and heartwarming , this devilishly humorous show from the pens of the \" blaguards \" themselves , the great Frank McCourt ( Pulitzer Prize Winner , \" Angela 's Ashes \" ) and his brother Malachy ( \" A Monk Swimming \" ) delights . Liberally laced with music , parody and song , with a pinch of anarchy and a tall tale or two tossed in for good measure , this warm and irreverent comedy traces the brothers ' hilarious and unlikely journey from the streets of poverty-stricken Limerick to fame and fortune in\n",
       "\n",
       "[3711273 rows x 1 columns]"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.set_option('display.max_colwidth', 0)\n",
    "\n",
    "# create a dataframe for the texts\n",
    "df = pd.DataFrame(text)\n",
    "df.columns = ['text']\n",
    "print(\"There are {} rows in df\".format(len(df)))\n",
    "df[['text']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "b580c544-a81a-4b3c-8ae4-8e2af6c46706",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\17322\\AppData\\Local\\Temp/ipykernel_18048/343950438.py:12: FutureWarning: The default value of regex will change from True to False in a future version.\n",
      "  df['clean_text'] = df['clean_text'].str.replace('(<.*?>|&lt;.*?&gt;)', '')\n",
      "C:\\Users\\17322\\AppData\\Local\\Temp/ipykernel_18048/343950438.py:14: FutureWarning: The default value of regex will change from True to False in a future version.\n",
      "  df['clean_text'] = df['clean_text'].str.replace('\\d+', '')\n",
      "C:\\Users\\17322\\AppData\\Local\\Temp/ipykernel_18048/343950438.py:17: FutureWarning: The default value of regex will change from True to False in a future version.\n",
      "  df['clean_text'] = df['clean_text'].str.replace(\"[^a-zA-Z\\s]\", \"\")\n"
     ]
    }
   ],
   "source": [
    "# lowercase all characters\n",
    "df['clean_text'] = df['text'].fillna('').apply(lambda x: x.lower())\n",
    "\n",
    "# keep only first sentence from each example to keep it simple\n",
    "def only_keep_first_sentence(x):\n",
    "    try:\n",
    "        text = re.search(r'^.*?[.?!]', x).group(0)\n",
    "    except:\n",
    "        text = '' # if no such sentence boundary can be find, discard this example by replacing it with ''\n",
    "    return text\n",
    "df['clean_text'] = df['clean_text'].apply(lambda x: only_keep_first_sentence(x))\n",
    "\n",
    "# Remove HTML tags\n",
    "df['clean_text'] = df['clean_text'].str.replace('(<.*?>|&lt;.*?&gt;)', '')\n",
    "\n",
    "# Remove digits\n",
    "df['clean_text'] = df['clean_text'].str.replace('\\d+', '')\n",
    "\n",
    "# Remove other special characters \n",
    "# df['clean_text'] = df['clean_text'].str.replace(\"( [^a-zA-Z ]+|#)\", \"\") # (most are punctuation marks with spaces)\n",
    "df['clean_text'] = df['clean_text'].str.replace(\"[^a-zA-Z\\s]\", \"\")\n",
    "\n",
    "# get rid of multiple white spaces + strip\n",
    "df['clean_text'] = df['clean_text'].apply(lambda x: ' '.join(x.split())) \n",
    "\n",
    "# Remove empty rows\n",
    "df['clean_text'].astype(bool) \n",
    "df= df[df['clean_text'].str.strip().astype(bool)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "b8c6a5fa-5aab-49d9-a21e-1ee59fe990e1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>orig_text</th>\n",
       "      <th>clean_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Arts and cultural industries rake in an annual $1.2 billion and generate 19,500 jobs in Bernalillo County . UNM alumni in the sector are helping the university pave a smoother path for future arts managers and entrepreneurs</td>\n",
       "      <td>arts and cultural industries rake in an annual</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>disabled or is not : About CDC.gov . &lt;h&gt; Cancer and Men</td>\n",
       "      <td>disabled or is not about cdc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Every year , cancer claims the lives of nearly 300,000 men in America . Men can reduce their risk for several of the most common kinds of cancer . &lt;h&gt; Lifestyle Changes &lt;h&gt; Tobacco Use and Secondhand Smoke</td>\n",
       "      <td>every year cancer claims the lives of nearly men in america</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>More men in the United States die from lung cancer than any other kind of cancer , and cigarette smoking causes most cases . Smoking also causes cancers of the esophagus , larynx ( voice box ) , mouth , throat , kidney , bladder , pancreas , stomach , and acute myeloid leukemia . Nonsmokers who are exposed to secondhand smoke at home or work increase their lung cancer risk by 20% -- 30% . Concentrations of many cancer-causing and toxic chemicals are higher in secondhand smoke than in the smoke inhaled by smokers .</td>\n",
       "      <td>more men in the united states die from lung cancer than any other kind of cancer and cigarette smoking causes most cases</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>One of the things you can do to lower your risk of cancer is @ @ @ @ @ @ @ @ @ @ smoke . &lt;h&gt; Obesity , Overweight , and Lack of Physical Activity</td>\n",
       "      <td>one of the things you can do to lower your risk of cancer is smoke</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711266</th>\n",
       "      <td>This morning I 'm feeling slightly nauseous from the lack of adequate quality sleep ( OK , I just snorted as I wrote that , \" adequate quality sleep \" , hysterical ) .</td>\n",
       "      <td>this morning i m feeling slightly nauseous from the lack of adequate quality sleep ok i just snorted as i wrote that adequate quality sleep hysterical</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711267</th>\n",
       "      <td>But somehow even through my sleep deprived nausea ( and this is the weird thing that is being a mom ) I ca n't help but look over at my girls , snuggled together on my bedroom floor and feel a crushing disappointment that @ @ @ @ @ @ @ @ @ @ they will be too old for this .</td>\n",
       "      <td>but somehow even through my sleep deprived nausea and this is the weird thing that is being a mom i ca nt help but look over at my girls snuggled together on my bedroom floor and feel a crushing disappointment that they will be too old for this</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711268</th>\n",
       "      <td>And I know from my experience with the boys that someday is coming fast ... inconceivably I will miss these nights of bad dreams and IN-adequate quality sleep .</td>\n",
       "      <td>and i know from my experience with the boys that someday is coming fast</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711269</th>\n",
       "      <td>Its hard to imagine if you have n't parented through these years but trust me on this ... you will miss it all ... and it will gone before you know it .</td>\n",
       "      <td>its hard to imagine if you have nt parented through these years but trust me on this</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711272</th>\n",
       "      <td>Hilarious and heartwarming , this devilishly humorous show from the pens of the \" blaguards \" themselves , the great Frank McCourt ( Pulitzer Prize Winner , \" Angela 's Ashes \" ) and his brother Malachy ( \" A Monk Swimming \" ) delights . Liberally laced with music , parody and song , with a pinch of anarchy and a tall tale or two tossed in for good measure , this warm and irreverent comedy traces the brothers ' hilarious and unlikely journey from the streets of poverty-stricken Limerick to fame and fortune in</td>\n",
       "      <td>hilarious and heartwarming this devilishly humorous show from the pens of the blaguards themselves the great frank mccourt pulitzer prize winner angela s ashes and his brother malachy a monk swimming delights</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2985654 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       orig_text  \\\n",
       "1        Arts and cultural industries rake in an annual $1.2 billion and generate 19,500 jobs in Bernalillo County . UNM alumni in the sector are helping the university pave a smoother path for future arts managers and entrepreneurs                                                                                                                                                                                                                                                                                                           \n",
       "3        disabled or is not : About CDC.gov . <h> Cancer and Men                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   \n",
       "4        Every year , cancer claims the lives of nearly 300,000 men in America . Men can reduce their risk for several of the most common kinds of cancer . <h> Lifestyle Changes <h> Tobacco Use and Secondhand Smoke                                                                                                                                                                                                                                                                                                                             \n",
       "5        More men in the United States die from lung cancer than any other kind of cancer , and cigarette smoking causes most cases . Smoking also causes cancers of the esophagus , larynx ( voice box ) , mouth , throat , kidney , bladder , pancreas , stomach , and acute myeloid leukemia . Nonsmokers who are exposed to secondhand smoke at home or work increase their lung cancer risk by 20% -- 30% . Concentrations of many cancer-causing and toxic chemicals are higher in secondhand smoke than in the smoke inhaled by smokers .   \n",
       "6        One of the things you can do to lower your risk of cancer is @ @ @ @ @ @ @ @ @ @ smoke . <h> Obesity , Overweight , and Lack of Physical Activity                                                                                                                                                                                                                                                                                                                                                                                         \n",
       "...                                                                                                                                                    ...                                                                                                                                                                                                                                                                                                                                                                                         \n",
       "3711266  This morning I 'm feeling slightly nauseous from the lack of adequate quality sleep ( OK , I just snorted as I wrote that , \" adequate quality sleep \" , hysterical ) .                                                                                                                                                                                                                                                                                                                                                                   \n",
       "3711267  But somehow even through my sleep deprived nausea ( and this is the weird thing that is being a mom ) I ca n't help but look over at my girls , snuggled together on my bedroom floor and feel a crushing disappointment that @ @ @ @ @ @ @ @ @ @ they will be too old for this .                                                                                                                                                                                                                                                         \n",
       "3711268  And I know from my experience with the boys that someday is coming fast ... inconceivably I will miss these nights of bad dreams and IN-adequate quality sleep .                                                                                                                                                                                                                                                                                                                                                                          \n",
       "3711269  Its hard to imagine if you have n't parented through these years but trust me on this ... you will miss it all ... and it will gone before you know it .                                                                                                                                                                                                                                                                                                                                                                                  \n",
       "3711272  Hilarious and heartwarming , this devilishly humorous show from the pens of the \" blaguards \" themselves , the great Frank McCourt ( Pulitzer Prize Winner , \" Angela 's Ashes \" ) and his brother Malachy ( \" A Monk Swimming \" ) delights . Liberally laced with music , parody and song , with a pinch of anarchy and a tall tale or two tossed in for good measure , this warm and irreverent comedy traces the brothers ' hilarious and unlikely journey from the streets of poverty-stricken Limerick to fame and fortune in        \n",
       "\n",
       "                                                                                                                                                                                                                                                   clean_text  \n",
       "1        arts and cultural industries rake in an annual                                                                                                                                                                                                        \n",
       "3        disabled or is not about cdc                                                                                                                                                                                                                          \n",
       "4        every year cancer claims the lives of nearly men in america                                                                                                                                                                                           \n",
       "5        more men in the united states die from lung cancer than any other kind of cancer and cigarette smoking causes most cases                                                                                                                              \n",
       "6        one of the things you can do to lower your risk of cancer is smoke                                                                                                                                                                                    \n",
       "...                                                                     ...                                                                                                                                                                                    \n",
       "3711266  this morning i m feeling slightly nauseous from the lack of adequate quality sleep ok i just snorted as i wrote that adequate quality sleep hysterical                                                                                                \n",
       "3711267  but somehow even through my sleep deprived nausea and this is the weird thing that is being a mom i ca nt help but look over at my girls snuggled together on my bedroom floor and feel a crushing disappointment that they will be too old for this  \n",
       "3711268  and i know from my experience with the boys that someday is coming fast                                                                                                                                                                               \n",
       "3711269  its hard to imagine if you have nt parented through these years but trust me on this                                                                                                                                                                  \n",
       "3711272  hilarious and heartwarming this devilishly humorous show from the pens of the blaguards themselves the great frank mccourt pulitzer prize winner angela s ashes and his brother malachy a monk swimming delights                                      \n",
       "\n",
       "[2985654 rows x 2 columns]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.rename(columns={'text': 'orig_text'}, inplace=True)\n",
    "# df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "80ce34dd-4130-4245-9251-5e4cc622bd36",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>orig_text</th>\n",
       "      <th>clean_text</th>\n",
       "      <th>char_count</th>\n",
       "      <th>word_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Arts and cultural industries rake in an annual $1.2 billion and generate 19,500 jobs in Bernalillo County . UNM alumni in the sector are helping the university pave a smoother path for future arts managers and entrepreneurs</td>\n",
       "      <td>arts and cultural industries rake in an annual</td>\n",
       "      <td>46</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>disabled or is not : About CDC.gov . &lt;h&gt; Cancer and Men</td>\n",
       "      <td>disabled or is not about cdc</td>\n",
       "      <td>28</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Every year , cancer claims the lives of nearly 300,000 men in America . Men can reduce their risk for several of the most common kinds of cancer . &lt;h&gt; Lifestyle Changes &lt;h&gt; Tobacco Use and Secondhand Smoke</td>\n",
       "      <td>every year cancer claims the lives of nearly men in america</td>\n",
       "      <td>59</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>More men in the United States die from lung cancer than any other kind of cancer , and cigarette smoking causes most cases . Smoking also causes cancers of the esophagus , larynx ( voice box ) , mouth , throat , kidney , bladder , pancreas , stomach , and acute myeloid leukemia . Nonsmokers who are exposed to secondhand smoke at home or work increase their lung cancer risk by 20% -- 30% . Concentrations of many cancer-causing and toxic chemicals are higher in secondhand smoke than in the smoke inhaled by smokers .</td>\n",
       "      <td>more men in the united states die from lung cancer than any other kind of cancer and cigarette smoking causes most cases</td>\n",
       "      <td>120</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>One of the things you can do to lower your risk of cancer is @ @ @ @ @ @ @ @ @ @ smoke . &lt;h&gt; Obesity , Overweight , and Lack of Physical Activity</td>\n",
       "      <td>one of the things you can do to lower your risk of cancer is smoke</td>\n",
       "      <td>66</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711266</th>\n",
       "      <td>This morning I 'm feeling slightly nauseous from the lack of adequate quality sleep ( OK , I just snorted as I wrote that , \" adequate quality sleep \" , hysterical ) .</td>\n",
       "      <td>this morning im feeling slightly nauseous from the lack of adequate quality sleep ok i just snorted as i wrote that adequate quality sleep hysterical</td>\n",
       "      <td>149</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711267</th>\n",
       "      <td>But somehow even through my sleep deprived nausea ( and this is the weird thing that is being a mom ) I ca n't help but look over at my girls , snuggled together on my bedroom floor and feel a crushing disappointment that @ @ @ @ @ @ @ @ @ @ they will be too old for this .</td>\n",
       "      <td>but somehow even through my sleep deprived nausea and this is the weird thing that is being a mom i ca nt help but look over at my girls snuggled together on my bedroom floor and feel a crushing disappointment that they will be too old for this</td>\n",
       "      <td>244</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711268</th>\n",
       "      <td>And I know from my experience with the boys that someday is coming fast ... inconceivably I will miss these nights of bad dreams and IN-adequate quality sleep .</td>\n",
       "      <td>and i know from my experience with the boys that someday is coming fast</td>\n",
       "      <td>71</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711269</th>\n",
       "      <td>Its hard to imagine if you have n't parented through these years but trust me on this ... you will miss it all ... and it will gone before you know it .</td>\n",
       "      <td>its hard to imagine if you have nt parented through these years but trust me on this</td>\n",
       "      <td>84</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3711272</th>\n",
       "      <td>Hilarious and heartwarming , this devilishly humorous show from the pens of the \" blaguards \" themselves , the great Frank McCourt ( Pulitzer Prize Winner , \" Angela 's Ashes \" ) and his brother Malachy ( \" A Monk Swimming \" ) delights . Liberally laced with music , parody and song , with a pinch of anarchy and a tall tale or two tossed in for good measure , this warm and irreverent comedy traces the brothers ' hilarious and unlikely journey from the streets of poverty-stricken Limerick to fame and fortune in</td>\n",
       "      <td>hilarious and heartwarming this devilishly humorous show from the pens of the blaguards themselves the great frank mccourt pulitzer prize winner angelas ashes and his brother malachy a monk swimming delights</td>\n",
       "      <td>207</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2985654 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       orig_text  \\\n",
       "1        Arts and cultural industries rake in an annual $1.2 billion and generate 19,500 jobs in Bernalillo County . UNM alumni in the sector are helping the university pave a smoother path for future arts managers and entrepreneurs                                                                                                                                                                                                                                                                                                           \n",
       "3        disabled or is not : About CDC.gov . <h> Cancer and Men                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   \n",
       "4        Every year , cancer claims the lives of nearly 300,000 men in America . Men can reduce their risk for several of the most common kinds of cancer . <h> Lifestyle Changes <h> Tobacco Use and Secondhand Smoke                                                                                                                                                                                                                                                                                                                             \n",
       "5        More men in the United States die from lung cancer than any other kind of cancer , and cigarette smoking causes most cases . Smoking also causes cancers of the esophagus , larynx ( voice box ) , mouth , throat , kidney , bladder , pancreas , stomach , and acute myeloid leukemia . Nonsmokers who are exposed to secondhand smoke at home or work increase their lung cancer risk by 20% -- 30% . Concentrations of many cancer-causing and toxic chemicals are higher in secondhand smoke than in the smoke inhaled by smokers .   \n",
       "6        One of the things you can do to lower your risk of cancer is @ @ @ @ @ @ @ @ @ @ smoke . <h> Obesity , Overweight , and Lack of Physical Activity                                                                                                                                                                                                                                                                                                                                                                                         \n",
       "...                                                                                                                                                    ...                                                                                                                                                                                                                                                                                                                                                                                         \n",
       "3711266  This morning I 'm feeling slightly nauseous from the lack of adequate quality sleep ( OK , I just snorted as I wrote that , \" adequate quality sleep \" , hysterical ) .                                                                                                                                                                                                                                                                                                                                                                   \n",
       "3711267  But somehow even through my sleep deprived nausea ( and this is the weird thing that is being a mom ) I ca n't help but look over at my girls , snuggled together on my bedroom floor and feel a crushing disappointment that @ @ @ @ @ @ @ @ @ @ they will be too old for this .                                                                                                                                                                                                                                                         \n",
       "3711268  And I know from my experience with the boys that someday is coming fast ... inconceivably I will miss these nights of bad dreams and IN-adequate quality sleep .                                                                                                                                                                                                                                                                                                                                                                          \n",
       "3711269  Its hard to imagine if you have n't parented through these years but trust me on this ... you will miss it all ... and it will gone before you know it .                                                                                                                                                                                                                                                                                                                                                                                  \n",
       "3711272  Hilarious and heartwarming , this devilishly humorous show from the pens of the \" blaguards \" themselves , the great Frank McCourt ( Pulitzer Prize Winner , \" Angela 's Ashes \" ) and his brother Malachy ( \" A Monk Swimming \" ) delights . Liberally laced with music , parody and song , with a pinch of anarchy and a tall tale or two tossed in for good measure , this warm and irreverent comedy traces the brothers ' hilarious and unlikely journey from the streets of poverty-stricken Limerick to fame and fortune in        \n",
       "\n",
       "                                                                                                                                                                                                                                                   clean_text  \\\n",
       "1        arts and cultural industries rake in an annual                                                                                                                                                                                                         \n",
       "3        disabled or is not about cdc                                                                                                                                                                                                                           \n",
       "4        every year cancer claims the lives of nearly men in america                                                                                                                                                                                            \n",
       "5        more men in the united states die from lung cancer than any other kind of cancer and cigarette smoking causes most cases                                                                                                                               \n",
       "6        one of the things you can do to lower your risk of cancer is smoke                                                                                                                                                                                     \n",
       "...                                                                     ...                                                                                                                                                                                     \n",
       "3711266  this morning im feeling slightly nauseous from the lack of adequate quality sleep ok i just snorted as i wrote that adequate quality sleep hysterical                                                                                                  \n",
       "3711267  but somehow even through my sleep deprived nausea and this is the weird thing that is being a mom i ca nt help but look over at my girls snuggled together on my bedroom floor and feel a crushing disappointment that they will be too old for this   \n",
       "3711268  and i know from my experience with the boys that someday is coming fast                                                                                                                                                                                \n",
       "3711269  its hard to imagine if you have nt parented through these years but trust me on this                                                                                                                                                                   \n",
       "3711272  hilarious and heartwarming this devilishly humorous show from the pens of the blaguards themselves the great frank mccourt pulitzer prize winner angelas ashes and his brother malachy a monk swimming delights                                        \n",
       "\n",
       "         char_count  word_count  \n",
       "1        46          8           \n",
       "3        28          6           \n",
       "4        59          11          \n",
       "5        120         22          \n",
       "6        66          15          \n",
       "...      ..          ..          \n",
       "3711266  149         25          \n",
       "3711267  244         48          \n",
       "3711268  71          14          \n",
       "3711269  84          17          \n",
       "3711272  207         31          \n",
       "\n",
       "[2985654 rows x 4 columns]"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['char_count'] = df['clean_text'].apply(len)\n",
    "df['word_count'] = df['clean_text'].apply(lambda x: len(x.split()))\n",
    "# df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bd1bdb1-6cc5-44a5-a712-e57b5fb95abf",
   "metadata": {},
   "source": [
    "### Searching for the keywords \"their\" \"there\" and \"theyre\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "da5edd73-e0cb-4299-a8ab-fcdd3952e275",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2985654it [01:56, 25684.71it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "115701\n",
      "139502\n",
      "11039\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm # module for keeping track of progress\n",
    "\n",
    "no_matches = [] # stores indices of rows to be deleted\n",
    "df['keyword'] = \"\" # column for indicating what PET is in this example\n",
    "their_examples, there_examples, theyre_examples = [], [], []\n",
    "\n",
    "# for each row, try to find and note down the PET, otherwise mark it for deletion\n",
    "for i, row in tqdm(df.iterrows(), position=0, leave=True):\n",
    "    text = df.loc[i, 'clean_text'] # locate the text\n",
    "    found_keyword = False # if this is still False after the PET search, mark this row for deletion\n",
    "    their_result = bool(re.search(r'their', text))\n",
    "    there_result = bool(re.search(r'there', text))\n",
    "    theyre_result = bool(re.search(r'they ?re', text))\n",
    "    if (their_result ^ there_result ^ theyre_result): # if only one of these cases is true (we don't want both 'their' and 'there' in an example, for instance)\n",
    "        if (their_result): \n",
    "            df.loc[i, 'keyword'] = 'their'\n",
    "            their_examples.append(i)\n",
    "        elif (there_result): \n",
    "            df.loc[i, 'keyword'] = 'there'\n",
    "            there_examples.append(i)\n",
    "        elif (theyre_result):\n",
    "            df.loc[i, 'keyword'] = 'theyre'\n",
    "            theyre_examples.append(i)\n",
    "            \n",
    "print(len(their_examples))\n",
    "print(len(there_examples))\n",
    "print(len(theyre_examples))\n",
    "#     for term in ['their', 'there', 'they\\'re']: \n",
    "#         if term in text:\n",
    "#             found_PET = True\n",
    "#             df.loc[i, 'PET'] = PET\n",
    "        \n",
    "#     if (found_PET == False):\n",
    "#         no_matches.append(i)\n",
    "        \n",
    "# result_df = df.drop(no_matches) # delete rows\n",
    "# display(result_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "901cc86f-50af-4166-898f-da245fa1ffb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# some more preprocessing: combining \"they re\" into \"theyre\", to be consistent with the other keywords\n",
    "for i, row in theyre_df.iterrows():\n",
    "    text = theyre_df.loc[i, 'clean_text'] \n",
    "    theyre_df.loc[i, 'clean_text'] = text.replace('they re', 'theyre')\n",
    "    \n",
    "theyre_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "3ffb6233-8548-4829-a5cd-d95dd3e40150",
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the matched examples\n",
    "matches_df = pd.concat([df.loc[their_examples], df.loc[there_examples], df.loc[theyre_examples]])\n",
    "matches_df.to_csv('their_there_theyre.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0d0a022-1005-4fa3-9385-0925060afdd5",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Form Training and Test Sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "id": "0bc08ae6-2554-49b4-8a23-84a9e82a0274",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14448\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "keyword\n",
       "there     131202\n",
       "their     110098\n",
       "theyre     10494\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 239,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "matches_df = pd.read_csv(\"their_there_theyre_data.csv\", index_col=0)\n",
    "\n",
    "# GloWbE sometimes provides duplicate examples, so let's take care of that now\n",
    "dup = matches_df.duplicated(subset=['clean_text'], keep=False)\n",
    "print(len(dup[dup].index))\n",
    "matches_df = matches_df.drop(dup[dup].index)\n",
    "matches_df['keyword'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "id": "e78bf268-d877-4de1-8aa3-75145ea3bfef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# sample 10k examples for each keyword\n",
    "their_df = matches_df.loc[matches_df['keyword'] == 'their'].sample(n=10000)\n",
    "there_df = matches_df.loc[matches_df['keyword'] == 'there'].sample(n=10000)\n",
    "theyre_df = matches_df.loc[matches_df['keyword'] == 'theyre'].sample(n=10000)\n",
    "\n",
    "# initialize column that will contain correct keyword (0 = their, 1 = there, 2 = theyre)\n",
    "their_df['label'] = 0\n",
    "there_df['label'] = 1\n",
    "theyre_df['label'] = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "id": "1df915d5-75df-4725-9255-f11cb9ac4d6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# randomly sample half of each dataset to be corrupted (i.e., the keyword will be replaced with 1 of the other 2 keywords to generate \"ungrammatical\" examples)\n",
    "their_corrupt_sample = their_df.sample(frac=0.5)\n",
    "their_regular_sample = their_df.drop(their_corrupt_sample.index)\n",
    "there_corrupt_sample = there_df.sample(frac=0.5)\n",
    "there_regular_sample = there_df.drop(there_corrupt_sample.index)\n",
    "theyre_corrupt_sample = theyre_df.sample(frac=0.5)\n",
    "theyre_regular_sample = theyre_df.drop(theyre_corrupt_sample.index)\n",
    "\n",
    "# initialize column that will indicate whether this example is ungrammatical (ie whether it has been corrupted)\n",
    "their_corrupt_sample['ungrammatical'] = 1\n",
    "their_regular_sample['ungrammatical'] = 0\n",
    "there_corrupt_sample['ungrammatical'] = 1\n",
    "there_regular_sample['ungrammatical'] = 0\n",
    "theyre_corrupt_sample['ungrammatical'] = 1\n",
    "theyre_regular_sample['ungrammatical'] = 0\n",
    "\n",
    "# for each of the corrupted datasets, randomly select half to be corrupted with one keyword; the other half, the remaining keyword\n",
    "their_there_corrupt_sample = their_corrupt_sample.sample(frac=0.5) # this means the examples with \"their\" that'll be corrupted with \"there\"\n",
    "their_theyre_corrupt_sample = their_corrupt_sample.drop(their_there_corrupt_sample.index) \n",
    "their_there_corrupt_sample['clean_text'] = their_there_corrupt_sample['clean_text'].apply(lambda x: x.replace('their', 'there'))\n",
    "their_theyre_corrupt_sample['clean_text'] = their_theyre_corrupt_sample['clean_text'].apply(lambda x: x.replace('their', 'theyre'))\n",
    "\n",
    "there_their_corrupt_sample = there_corrupt_sample.sample(frac=0.5) # this means the examples with \"their\" that'll be corrupted with \"there\"\n",
    "there_theyre_corrupt_sample = there_corrupt_sample.drop(there_their_corrupt_sample.index) \n",
    "there_their_corrupt_sample['clean_text'] = there_their_corrupt_sample['clean_text'].apply(lambda x: x.replace('there', 'their'))\n",
    "there_theyre_corrupt_sample['clean_text'] = there_theyre_corrupt_sample['clean_text'].apply(lambda x: x.replace('there', 'theyre'))\n",
    "\n",
    "theyre_their_corrupt_sample = theyre_corrupt_sample.sample(frac=0.5) # this means the examples with \"their\" that'll be corrupted with \"there\"\n",
    "theyre_there_corrupt_sample = theyre_corrupt_sample.drop(theyre_their_corrupt_sample.index) \n",
    "theyre_their_corrupt_sample['clean_text'] = theyre_their_corrupt_sample['clean_text'].apply(lambda x: x.replace('theyre', 'their'))\n",
    "theyre_there_corrupt_sample['clean_text'] = theyre_there_corrupt_sample['clean_text'].apply(lambda x: x.replace('theyre', 'there'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "id": "16763a20-1498-449c-8b9e-5be570a0daa4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clean_text</th>\n",
       "      <th>keyword</th>\n",
       "      <th>label</th>\n",
       "      <th>ungrammatical</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>818651</th>\n",
       "      <td>thanks joe for talking about what might off to...</td>\n",
       "      <td>their</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>410581</th>\n",
       "      <td>unfortunately societysometimes fails to recogn...</td>\n",
       "      <td>their</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1467069</th>\n",
       "      <td>we push browsers to their limits these days an...</td>\n",
       "      <td>their</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2318513</th>\n",
       "      <td>like most families the pga tour and the pga of...</td>\n",
       "      <td>their</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200065</th>\n",
       "      <td>my friend was suggesting that we need more stu...</td>\n",
       "      <td>their</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2779480</th>\n",
       "      <td>i ca nt believe they re going to strike this down</td>\n",
       "      <td>theyre</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2055532</th>\n",
       "      <td>by the way they re actually not closing comple...</td>\n",
       "      <td>theyre</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>340218</th>\n",
       "      <td>the units are in some ways a tacit acknowledge...</td>\n",
       "      <td>theyre</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1473756</th>\n",
       "      <td>when they reached its shores the tide was out</td>\n",
       "      <td>theyre</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2309623</th>\n",
       "      <td>so when ann romney describes living frugally a...</td>\n",
       "      <td>theyre</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>30000 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                clean_text keyword  label  \\\n",
       "818651   thanks joe for talking about what might off to...   their      0   \n",
       "410581   unfortunately societysometimes fails to recogn...   their      0   \n",
       "1467069  we push browsers to their limits these days an...   their      0   \n",
       "2318513  like most families the pga tour and the pga of...   their      0   \n",
       "200065   my friend was suggesting that we need more stu...   their      0   \n",
       "...                                                    ...     ...    ...   \n",
       "2779480  i ca nt believe they re going to strike this down  theyre      2   \n",
       "2055532  by the way they re actually not closing comple...  theyre      2   \n",
       "340218   the units are in some ways a tacit acknowledge...  theyre      2   \n",
       "1473756      when they reached its shores the tide was out  theyre      2   \n",
       "2309623  so when ann romney describes living frugally a...  theyre      2   \n",
       "\n",
       "         ungrammatical  \n",
       "818651               0  \n",
       "410581               0  \n",
       "1467069              0  \n",
       "2318513              0  \n",
       "200065               0  \n",
       "...                ...  \n",
       "2779480              1  \n",
       "2055532              1  \n",
       "340218               1  \n",
       "1473756              1  \n",
       "2309623              1  \n",
       "\n",
       "[30000 rows x 4 columns]"
      ]
     },
     "execution_count": 242,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df = pd.concat([their_regular_sample, there_regular_sample, theyre_regular_sample, \n",
    "           their_there_corrupt_sample, their_theyre_corrupt_sample, \n",
    "           there_their_corrupt_sample, there_theyre_corrupt_sample,\n",
    "           theyre_their_corrupt_sample, theyre_there_corrupt_sample])\n",
    "\n",
    "final_df[['clean_text', 'keyword', 'label', 'ungrammatical']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "id": "0fc26179-94b4-4998-b77e-61196f37dccc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5000 5000 5000 5000 5000 5000\n"
     ]
    }
   ],
   "source": [
    "# checks\n",
    "their_regular = final_df[(final_df['label'] == 0) & (final_df['ungrammatical'] == 0)]\n",
    "their_corrupt = final_df[(final_df['label'] == 0) & (final_df['ungrammatical'] == 1)]\n",
    "there_regular = final_df[(final_df['label'] == 1) & (final_df['ungrammatical'] == 0)]\n",
    "there_corrupt = final_df[(final_df['label'] == 1) & (final_df['ungrammatical'] == 1)]\n",
    "theyre_regular = final_df[(final_df['label'] == 2) & (final_df['ungrammatical'] == 0)]\n",
    "theyre_corrupt = final_df[(final_df['label'] == 2) & (final_df['ungrammatical'] == 1)]\n",
    "\n",
    "print(len(their_regular), len(their_corrupt), len(there_regular), len(there_corrupt), len(theyre_regular), len(theyre_corrupt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "id": "b9cee792-a3b7-4dbb-bf36-4d9729ddee14",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27000 3000\n"
     ]
    }
   ],
   "source": [
    "# form train test splits with balanced examples per category\n",
    "def split(df, split_index):\n",
    "    return df[0:split_index], df[split_index:len(df)]\n",
    "\n",
    "SPLIT_INDEX = int(0.9*5000)\n",
    "train_their_regular, test_their_regular = split(their_regular, SPLIT_INDEX)\n",
    "train_their_corrupt, test_their_corrupt = split(their_corrupt, SPLIT_INDEX)\n",
    "train_there_regular, test_there_regular = split(there_regular, SPLIT_INDEX)\n",
    "train_there_corrupt, test_there_corrupt = split(there_corrupt, SPLIT_INDEX)\n",
    "train_theyre_regular, test_theyre_regular = split(theyre_regular, SPLIT_INDEX)\n",
    "train_theyre_corrupt, test_theyre_corrupt = split(theyre_corrupt, SPLIT_INDEX)\n",
    "\n",
    "train_df = pd.concat([train_their_regular, train_their_corrupt, train_there_regular, train_there_corrupt, train_theyre_regular, train_theyre_corrupt])\n",
    "test_df = pd.concat([test_their_regular, test_their_corrupt, test_there_regular, test_there_corrupt, test_theyre_regular, test_theyre_corrupt])\n",
    "print(len(train_df), len(test_df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "id": "f96cde48-5ea7-4317-af78-d900f304b13c",
   "metadata": {},
   "outputs": [],
   "source": [
    "folder = 'Vantage_Labs_Project_Files'\n",
    "train_df.to_csv(f'{folder}/their_there_theyre_train.csv')\n",
    "test_df.to_csv(f'{folder}/their_there_theyre_test.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5041f73c-5025-40ee-9b96-a8056ad06c40",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "torch_env",
   "language": "python",
   "name": "torch_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}