yummy-nlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Yummy NLP (Yummy Opinion Advisor)\n",
    "\n",
    "## Latent Aspect Rating Analysis on Yelp Restaurant Reviews\n",
    "\n",
    "In this Jupyter Notebook, we will try to analyse the sentiment of restaurant reviews, do topic and aspect mining and perform latent aspect rating analysis.\n",
    "\n",
    "This project is contributed by:\n",
    "* Asad Bin Imtiaz (aimtiaz2@illinois.edu)\n",
    "* Karun Veluru (kveluru2@illinois.edu)\n",
    "* Ron Swan (rdswan2@illinois.edu)\n",
    "\n",
    "To perform NLP on Yelp reviews following libraries are used:\n",
    "\n",
    "* Spacy\n",
    "* Pandas\n",
    "* MongoDB\n",
    "* Numpy\n",
    "* Gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Data gathering and preprocessing\n",
    "\n",
    "In this step, we will gather the data and preprocess it for NLP operations. \n",
    "\n",
    "The data is downloaded from https://www.yelp.com/dataset/download. The data set zipped archive (tar-ball) is approximately 3.7GB, which uncompresses into around 8GB of Yelp data including Yelp business descriptions, their reviews, the users and tips.\n",
    "The data is composed of 6 json data files. These files should be extracted in the dataset directory. The JSON data files are ignored from Git versining by having an ignore entry in `gitignore` file. This data is already loadeded into an instance of Mongo DB. The instruction on loading the data into mongo DB can be found in project readme.\n",
    "\n",
    "The `prepare_data.ipynb` will read the data from mongo DB and combine reviews and businesses data into one csv file in `processed_data` directory (you may need to install pymongo). The businesses of category 'Restaurants' will be considered only and all restaurants will be selected which have atleast 50 reviews.\n",
    "\n",
    "Lets start by installing some libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pip install pymongo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pip install gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pip install pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we import the necessary libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary libs\n",
    "import pandas as pd\n",
    "import re\n",
    "import numpy as np\n",
    "import time\n",
    "from pprint import pprint\n",
    "from functools import reduce \n",
    "import operator\n",
    "import os\n",
    "import itertools\n",
    "\n",
    "# Gensim\n",
    "import gensim\n",
    "import gensim.corpora as corpora\n",
    "from gensim.utils import simple_preprocess\n",
    "from gensim.models import CoherenceModel\n",
    "\n",
    "# Spacy\n",
    "import spacy\n",
    "\n",
    "# Plotting\n",
    "import matplotlib.pyplot as plot\n",
    "%matplotlib inline\n",
    "\n",
    "# S3 Access\n",
    "import s3fs\n",
    "\n",
    "# Mongo DB\n",
    "from pymongo import MongoClient\n",
    "\n",
    "import seaborn as sns\n",
    "sns.set()\n",
    "import pickle as pickle\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from operator import itemgetter\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "import string\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.model_selection import train_test_split, cross_val_score\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score, roc_auc_score, roc_curve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load 2 instances for Spacy with large English model. The first one with Tokenizer, parser and NER. The second one is used only for parsing and lemmatization and therefore NER is disabled to gain some performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 29.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# SPACY\n",
    "# This is the large Spacy English Library\n",
    "nlp  = spacy.load('en_core_web_lg')\n",
    "nlp2 = spacy.load('en_core_web_lg', disable=[\"ner\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data pre-processing\n",
    "\n",
    "We have loaded the data in *Mongo DB* (with pandas we can also read jsons directly, but it needs **10x** more time). From MongoDB, we select relevant data and write to csv files to be used for next steps\n",
    "\n",
    "The selection criteria is:\n",
    "* Restaurant businesses\n",
    "* Businesses with 50 or more reviews \n",
    "* Select only following fields from businesses:\n",
    "    - 'business_id'\n",
    "    - 'name'\n",
    "    - 'city'\n",
    "    - 'state'\n",
    "    - 'stars'\n",
    "    - 'review_count'\n",
    "    - 'categories'\n",
    "* Select only following fields from reviews:\n",
    "    - 'review_id'\n",
    "    - 'user_id'\n",
    "    - 'business_id'\n",
    "    - 'stars'\n",
    "    - 'useful'\n",
    "    - 'text'\n",
    "    - 'date'\n",
    "    \n",
    "In the following, we connect to MongoDb and load data into Pandas Data Frames. The data from Restaurants and Reviews is merged in a single file, called **restaurant_reviews.csv**. All qualifying restaurant data is written to **restaurants.csv** file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Download and loading\n",
    "\n",
    "First the Yelp reviews need to be downloaded. There can be downloaded as a zippped achive from: https://www.yelp.com/dataset/download\n",
    "The data set zipped archive (tar-ball) as around 3.7GB, which uncompresses into around 8GB of Yelp data including Yelp business descriptions, their reviews, the users and tips.\n",
    "The data is composed of 6 jason data files. These files should be extracted in the dataset directory. The JSON data files are ignored from Git versining by having an ignore entry in `gitignore` file.\n",
    "\n",
    "Once the data is stored in dataset directory, it is needed to be read into an instance of mongo DB. Make sure the mongo DB is installed and running during the data preperation phase. Mongo DB can be downloaded (for windows and other OS) from: https://www.mongodb.com/download-center/community?jmp=docs. The x64 Windows msi file is around 250MB large. Install the software with default settings. For convinience, the service can be executed with network user. After installation, add the MongoDB executables in the system PATH variavle. On windows the installations is ususally found at `C:\\Program Files\\MongoDB\\Server\\4.2\\bin` directory. Once the MongoDB server is installed, is running and PATH variable is adjusted, the data can be loaded by executing `dataimport.bat` or `dataimport.sh` (depending on OS). Running this script will add all data in an instance named `yelp` (photos data will not be loaded).\n",
    "\n",
    "This notebook will read the data from mongo DB and combine reviews and businesses data into one csv file in processed_data directory (you may need to install pymongo). The businesses of catory 'Restaurants' will be considered only and all restaurants will be selected which have atleast 50 rewies. The data will be preprocessed with pandas into `processed_data/restaurant_reviews.csv` file (~2.5GB).\n",
    "\n",
    "Lets import the data from MongoDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import data from MongoDB\n",
    "DBClient = MongoClient()\n",
    "yelp_data = DBClient.yelp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our NLP tasks, we will only consider restaurants with a minimum of 50 reviews. Also, since the data is huge, sometime a sample is used to limit the number of records. For sampling, we used all restaurants in the state of Illinois. This is about 5% of the total data.\n",
    "\n",
    "The data is uploaded into an S3 bucket. The credentials of the bucket are defined below. Since the data is already public, the bucket is kept open."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select business having atleast 50 reviews\n",
    "min_review_count = 50\n",
    "\n",
    "# businesses to Analyse\n",
    "businesses_to_analyse = 'Restaurants'\n",
    "\n",
    "state_filter = 'IL'\n",
    "\n",
    "# S3 Bucket\n",
    "\n",
    "bucket =  's3://'+bucket_arn_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read all restaurants from Mongo DB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 336 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Get all restaurant businesses\n",
    "Restaurant_business = pd.DataFrame(yelp_data.business.find({\"categories\":{\"$regex\" :\".*\"+businesses_to_analyse+\".*\"}, \"review_count\":{\"$gte\":min_review_count} },  {'business_id':1, 'name':1, 'city':1, 'state':1, 'stars':1, 'review_count':1, 'categories':1, '_id': 0}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read all reviews from Mongo DB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 58.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Get all reviews\n",
    "All_reviews = pd.DataFrame(yelp_data.review.find({},{'review_id':1, 'user_id':1, 'business_id':1, 'stars':1, 'useful':1, 'text':1, 'date':1, '_id': 0}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Merge all restaurants and their reviews in single data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 14.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Find all restaurant reviews\n",
    "#Restaurant_reviews = All_reviews[All_reviews.business_id.isin(Restaurant_business.business_id.values)]\n",
    "Restaurant_reviews = pd.merge(Restaurant_business,All_reviews, on='business_id').rename(columns={'stars_x':'business_stars', 'stars_y':'review_stars'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets explore the data a little bit. We select top 5 records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>categories</th>\n",
       "      <th>city</th>\n",
       "      <th>name</th>\n",
       "      <th>review_count</th>\n",
       "      <th>stars</th>\n",
       "      <th>state</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tstimHoMcYbkSC4eBA1wEg</td>\n",
       "      <td>Mexican, Restaurants, Patisserie/Cake Shop, Fo...</td>\n",
       "      <td>Las Vegas</td>\n",
       "      <td>Maria's Mexican Restaurant &amp; Bakery</td>\n",
       "      <td>184</td>\n",
       "      <td>4.5</td>\n",
       "      <td>NV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gnKjwL_1w79qoiV3IC_xQQ</td>\n",
       "      <td>Sushi Bars, Restaurants, Japanese</td>\n",
       "      <td>Charlotte</td>\n",
       "      <td>Musashi Japanese Restaurant</td>\n",
       "      <td>170</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>QXAEGFB4oINsVuTFxEYKFQ</td>\n",
       "      <td>Specialty Food, Restaurants, Dim Sum, Imported...</td>\n",
       "      <td>Mississauga</td>\n",
       "      <td>Emerald Chinese Restaurant</td>\n",
       "      <td>128</td>\n",
       "      <td>2.5</td>\n",
       "      <td>ON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NDuUMJfrWk52RA-H-OtrpA</td>\n",
       "      <td>Juice Bars &amp; Smoothies, Food, Restaurants, Fas...</td>\n",
       "      <td>Toronto</td>\n",
       "      <td>Bolt Fresh Bar</td>\n",
       "      <td>57</td>\n",
       "      <td>3.0</td>\n",
       "      <td>ON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>dQj5DLZjeDK3KFysh1SYOQ</td>\n",
       "      <td>Nightlife, Bars, Polish, Modern European, Rest...</td>\n",
       "      <td>Pittsburgh</td>\n",
       "      <td>Apteka</td>\n",
       "      <td>242</td>\n",
       "      <td>4.5</td>\n",
       "      <td>PA</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              business_id                                         categories  \\\n",
       "0  tstimHoMcYbkSC4eBA1wEg  Mexican, Restaurants, Patisserie/Cake Shop, Fo...   \n",
       "1  gnKjwL_1w79qoiV3IC_xQQ                  Sushi Bars, Restaurants, Japanese   \n",
       "2  QXAEGFB4oINsVuTFxEYKFQ  Specialty Food, Restaurants, Dim Sum, Imported...   \n",
       "3  NDuUMJfrWk52RA-H-OtrpA  Juice Bars & Smoothies, Food, Restaurants, Fas...   \n",
       "4  dQj5DLZjeDK3KFysh1SYOQ  Nightlife, Bars, Polish, Modern European, Rest...   \n",
       "\n",
       "          city                                 name  review_count  stars state  \n",
       "0    Las Vegas  Maria's Mexican Restaurant & Bakery           184    4.5    NV  \n",
       "1    Charlotte          Musashi Japanese Restaurant           170    4.0    NC  \n",
       "2  Mississauga           Emerald Chinese Restaurant           128    2.5    ON  \n",
       "3      Toronto                       Bolt Fresh Bar            57    3.0    ON  \n",
       "4   Pittsburgh                               Apteka           242    4.5    PA  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "# Sample 5 Restaurant\n",
    "Restaurant_business.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the review text and restaurant names into UTF-8. Ignore all untranslatable unicode characters "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 8.96 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Convert text to Unicode\n",
    "Restaurant_reviews['text'] = Restaurant_reviews['text'].map(lambda x: x.encode('unicode-escape','strict').decode('utf-8').replace('\\\\u',''))\n",
    "\n",
    "#Restaurant_reviews['text'] = Restaurant_reviews['text'].map(lambda x: re.sub(r'[^\\x00-\\x7f]',r'', x.encode('unicode-escape','strict').decode('utf-8')).replace('\\\\u',''))\n",
    "#Restaurant_reviews['text'] = Restaurant_reviews[u'text'].map(lambda x: re.sub(r'[^\\x00-\\x7f]',r'', x.encode('ascii', 'ignore').decode('utf-8')))\n",
    "\n",
    "#  Stats\n",
    "## Wall time: 8.3 s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 2.85 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Convert name to Unicode\n",
    "Restaurant_reviews['name']  = Restaurant_reviews['name'].map(lambda x: x.encode('unicode-escape','strict').decode('utf-8').replace('\\\\u',''))\n",
    "Restaurant_business['name'] = Restaurant_business['name'].map(lambda x: x.encode('unicode-escape','strict').decode('utf-8').replace('\\\\u',''))\n",
    "\n",
    "\n",
    "#  Stats\n",
    "## Wall time: 2.63 s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets see a sample of 5 reviews and 5 restaurants again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>categories</th>\n",
       "      <th>city</th>\n",
       "      <th>name</th>\n",
       "      <th>review_count</th>\n",
       "      <th>business_stars</th>\n",
       "      <th>state</th>\n",
       "      <th>date</th>\n",
       "      <th>review_id</th>\n",
       "      <th>review_stars</th>\n",
       "      <th>text</th>\n",
       "      <th>useful</th>\n",
       "      <th>user_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tstimHoMcYbkSC4eBA1wEg</td>\n",
       "      <td>Mexican, Restaurants, Patisserie/Cake Shop, Fo...</td>\n",
       "      <td>Las Vegas</td>\n",
       "      <td>Maria's Mexican Restaurant &amp; Bakery</td>\n",
       "      <td>184</td>\n",
       "      <td>4.5</td>\n",
       "      <td>NV</td>\n",
       "      <td>2014-01-17 00:59:36</td>\n",
       "      <td>iWFBGYotfzwiLsOka0e1Rw</td>\n",
       "      <td>4.0</td>\n",
       "      <td>We found out about this gem from The Man's co-...</td>\n",
       "      <td>0</td>\n",
       "      <td>Gvv5hQR8BYj5f-APtKwImQ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>tstimHoMcYbkSC4eBA1wEg</td>\n",
       "      <td>Mexican, Restaurants, Patisserie/Cake Shop, Fo...</td>\n",
       "      <td>Las Vegas</td>\n",
       "      <td>Maria's Mexican Restaurant &amp; Bakery</td>\n",
       "      <td>184</td>\n",
       "      <td>4.5</td>\n",
       "      <td>NV</td>\n",
       "      <td>2014-08-24 21:52:27</td>\n",
       "      <td>QhiVo4F8MFSoz1T7YggcXA</td>\n",
       "      <td>3.0</td>\n",
       "      <td>The \"chips\" are a fried corn tortilla which we...</td>\n",
       "      <td>0</td>\n",
       "      <td>b-y736LEcRWXoNFj82AcOw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tstimHoMcYbkSC4eBA1wEg</td>\n",
       "      <td>Mexican, Restaurants, Patisserie/Cake Shop, Fo...</td>\n",
       "      <td>Las Vegas</td>\n",
       "      <td>Maria's Mexican Restaurant &amp; Bakery</td>\n",
       "      <td>184</td>\n",
       "      <td>4.5</td>\n",
       "      <td>NV</td>\n",
       "      <td>2011-09-26 01:53:09</td>\n",
       "      <td>FUv-c5CkLy71yFTGLrp2ag</td>\n",
       "      <td>4.0</td>\n",
       "      <td>The pollo mole is well worth the trip out here...</td>\n",
       "      <td>1</td>\n",
       "      <td>_7_Bh8VaUTUgFhmQ59ml0g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>tstimHoMcYbkSC4eBA1wEg</td>\n",
       "      <td>Mexican, Restaurants, Patisserie/Cake Shop, Fo...</td>\n",
       "      <td>Las Vegas</td>\n",
       "      <td>Maria's Mexican Restaurant &amp; Bakery</td>\n",
       "      <td>184</td>\n",
       "      <td>4.5</td>\n",
       "      <td>NV</td>\n",
       "      <td>2017-03-27 21:03:02</td>\n",
       "      <td>lJfGtkCc4in6Xcq-_Wtfwg</td>\n",
       "      <td>3.0</td>\n",
       "      <td>I've ordered Caldo de Rez from here twice and ...</td>\n",
       "      <td>0</td>\n",
       "      <td>PFdPopcevMX7_2cx27rJKA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>tstimHoMcYbkSC4eBA1wEg</td>\n",
       "      <td>Mexican, Restaurants, Patisserie/Cake Shop, Fo...</td>\n",
       "      <td>Las Vegas</td>\n",
       "      <td>Maria's Mexican Restaurant &amp; Bakery</td>\n",
       "      <td>184</td>\n",
       "      <td>4.5</td>\n",
       "      <td>NV</td>\n",
       "      <td>2014-02-06 03:59:50</td>\n",
       "      <td>BKoH9aG1ZjUjy2brzkxzJw</td>\n",
       "      <td>5.0</td>\n",
       "      <td>My boyfriend doesn't like Mexican food yet he ...</td>\n",
       "      <td>0</td>\n",
       "      <td>IxBVL9K2K9aur-_zKvIJiw</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              business_id                                         categories  \\\n",
       "0  tstimHoMcYbkSC4eBA1wEg  Mexican, Restaurants, Patisserie/Cake Shop, Fo...   \n",
       "1  tstimHoMcYbkSC4eBA1wEg  Mexican, Restaurants, Patisserie/Cake Shop, Fo...   \n",
       "2  tstimHoMcYbkSC4eBA1wEg  Mexican, Restaurants, Patisserie/Cake Shop, Fo...   \n",
       "3  tstimHoMcYbkSC4eBA1wEg  Mexican, Restaurants, Patisserie/Cake Shop, Fo...   \n",
       "4  tstimHoMcYbkSC4eBA1wEg  Mexican, Restaurants, Patisserie/Cake Shop, Fo...   \n",
       "\n",
       "        city                                 name  review_count  \\\n",
       "0  Las Vegas  Maria's Mexican Restaurant & Bakery           184   \n",
       "1  Las Vegas  Maria's Mexican Restaurant & Bakery           184   \n",
       "2  Las Vegas  Maria's Mexican Restaurant & Bakery           184   \n",
       "3  Las Vegas  Maria's Mexican Restaurant & Bakery           184   \n",
       "4  Las Vegas  Maria's Mexican Restaurant & Bakery           184   \n",
       "\n",
       "   business_stars state                 date               review_id  \\\n",
       "0             4.5    NV  2014-01-17 00:59:36  iWFBGYotfzwiLsOka0e1Rw   \n",
       "1             4.5    NV  2014-08-24 21:52:27  QhiVo4F8MFSoz1T7YggcXA   \n",
       "2             4.5    NV  2011-09-26 01:53:09  FUv-c5CkLy71yFTGLrp2ag   \n",
       "3             4.5    NV  2017-03-27 21:03:02  lJfGtkCc4in6Xcq-_Wtfwg   \n",
       "4             4.5    NV  2014-02-06 03:59:50  BKoH9aG1ZjUjy2brzkxzJw   \n",
       "\n",
       "   review_stars                                               text  useful  \\\n",
       "0           4.0  We found out about this gem from The Man's co-...       0   \n",
       "1           3.0  The \"chips\" are a fried corn tortilla which we...       0   \n",
       "2           4.0  The pollo mole is well worth the trip out here...       1   \n",
       "3           3.0  I've ordered Caldo de Rez from here twice and ...       0   \n",
       "4           5.0  My boyfriend doesn't like Mexican food yet he ...       0   \n",
       "\n",
       "                  user_id  \n",
       "0  Gvv5hQR8BYj5f-APtKwImQ  \n",
       "1  b-y736LEcRWXoNFj82AcOw  \n",
       "2  _7_Bh8VaUTUgFhmQ59ml0g  \n",
       "3  PFdPopcevMX7_2cx27rJKA  \n",
       "4  IxBVL9K2K9aur-_zKvIJiw  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sample 5 Reviews\n",
    "Restaurant_reviews.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>categories</th>\n",
       "      <th>city</th>\n",
       "      <th>name</th>\n",
       "      <th>review_count</th>\n",
       "      <th>stars</th>\n",
       "      <th>state</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tstimHoMcYbkSC4eBA1wEg</td>\n",
       "      <td>Mexican, Restaurants, Patisserie/Cake Shop, Fo...</td>\n",
       "      <td>Las Vegas</td>\n",
       "      <td>Maria's Mexican Restaurant &amp; Bakery</td>\n",
       "      <td>184</td>\n",
       "      <td>4.5</td>\n",
       "      <td>NV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>gnKjwL_1w79qoiV3IC_xQQ</td>\n",
       "      <td>Sushi Bars, Restaurants, Japanese</td>\n",
       "      <td>Charlotte</td>\n",
       "      <td>Musashi Japanese Restaurant</td>\n",
       "      <td>170</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>QXAEGFB4oINsVuTFxEYKFQ</td>\n",
       "      <td>Specialty Food, Restaurants, Dim Sum, Imported...</td>\n",
       "      <td>Mississauga</td>\n",
       "      <td>Emerald Chinese Restaurant</td>\n",
       "      <td>128</td>\n",
       "      <td>2.5</td>\n",
       "      <td>ON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NDuUMJfrWk52RA-H-OtrpA</td>\n",
       "      <td>Juice Bars &amp; Smoothies, Food, Restaurants, Fas...</td>\n",
       "      <td>Toronto</td>\n",
       "      <td>Bolt Fresh Bar</td>\n",
       "      <td>57</td>\n",
       "      <td>3.0</td>\n",
       "      <td>ON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>dQj5DLZjeDK3KFysh1SYOQ</td>\n",
       "      <td>Nightlife, Bars, Polish, Modern European, Rest...</td>\n",
       "      <td>Pittsburgh</td>\n",
       "      <td>Apteka</td>\n",
       "      <td>242</td>\n",
       "      <td>4.5</td>\n",
       "      <td>PA</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              business_id                                         categories  \\\n",
       "0  tstimHoMcYbkSC4eBA1wEg  Mexican, Restaurants, Patisserie/Cake Shop, Fo...   \n",
       "1  gnKjwL_1w79qoiV3IC_xQQ                  Sushi Bars, Restaurants, Japanese   \n",
       "2  QXAEGFB4oINsVuTFxEYKFQ  Specialty Food, Restaurants, Dim Sum, Imported...   \n",
       "3  NDuUMJfrWk52RA-H-OtrpA  Juice Bars & Smoothies, Food, Restaurants, Fas...   \n",
       "4  dQj5DLZjeDK3KFysh1SYOQ  Nightlife, Bars, Polish, Modern European, Rest...   \n",
       "\n",
       "          city                                 name  review_count  stars state  \n",
       "0    Las Vegas  Maria's Mexican Restaurant & Bakery           184    4.5    NV  \n",
       "1    Charlotte          Musashi Japanese Restaurant           170    4.0    NC  \n",
       "2  Mississauga           Emerald Chinese Restaurant           128    2.5    ON  \n",
       "3      Toronto                       Bolt Fresh Bar            57    3.0    ON  \n",
       "4   Pittsburgh                               Apteka           242    4.5    PA  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sample 5 Restaurants\n",
    "Restaurant_business.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the merged review data frame and restaurant data frame in CSV files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write selected Restaurants to file\n",
    "Restaurant_reviews.to_csv('processed_data/restaurant_reviews.csv',encoding='utf-8',line_terminator='\\r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write selected Restaurant-reviews to file\n",
    "Restaurant_business.to_csv('processed_data/restaurants.csv',encoding='utf-8',line_terminator='\\r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Some charts on the loaded data, just for fun\n",
    "\n",
    "Lets visualize the number of reviews / star rating in the selected data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Number of Reviews Per Rating of Restaurants')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg8AAAFNCAYAAACHXYxYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZhcVbX+8W93BkhIwhAaCSIBRFYCCmESlEFUEDGoKAIPRCAoAQSMyKwmKjiggiBwRblhCP5AHBgUgVy4IkiYlBkF8uJFCEOCxAiGQBISun9/7F2haHuo03R1VVe/n+fJk6pV+5yzancntWrvfc5pamtrw8zMzKxSzbVOwMzMzPoXFw9mZmZWiIsHMzMzK8TFg5mZmRXi4sHMzMwKcfFgZmZmhQyudQJmPRERGwJPAodJuqgsfgLwbkmTe+k4TwGfkXRvb+yvm2ONAmYBawDTJV1d9to3gaOB53KoCRgFXAMcL6nwOdcRsS1wiqTPvMXUKz3ercBY4N9AGzAUuBf4gqRX38J+ZwK7AwtyqBkYAfxU0g+62XYj4ExJ+0TEesCVkt7f01wqFRE/AT4K/FzS18riu5J+B1TWfCTwKDBZ0sIeHu/rwEOSftvjpHtBveRhb52LB+vPWoEfRsTtktRt6/o3AXibpE06ef2Xko4pPYmINYGHgRvzn0JyQdQnhUOZEyVdCRARTcCvgNOAE97ifs+WdGbpSURsADwWEddKmtPFdmOBAJA0D6h64ZAdAWwg6dkOXntC0oTSk4gYBFxF6qOv9PB4HyIVILVWL3nYW+TiwfqzJcAPgZ9HxPskvVb+Yv5G+tfSh0r58zyi8HPSf2ZrAj8AdgS2AZYDn8gfJgBHR8SWwCrADyVdnPf3cWAa6Rv0q8AJku7KowTvA9Yjfcv6bLu89ga+QfqG/DJwHOnb+MXA2yPiQeB9kpZ08/7fBgwHXsz7HQ+cA4wGBgHnSro4In4O3Cfph7ndF4BdgZ8A/yXp3RExFPg+8IG87QPAVOBzwDaSDoqIIcBC4EuSLomInXL/fxi4BHgXqaC7DzhCUmtXyUtqi4hbgI91k/+uOf4KaURhO0nLuumb9UmjMy/nfX8V+CQwDFiN9EF8LXAhqc9vJH2g/1XSiPwz3BAYQyowngM+K2l+RGyX+24o8ER+/ThJt5YnEBGbA/+V308b6XfnZxExO+c2KyKOkjS7m/cyCmgB7sj7XT33x3uAIcDNpKJsRUScCnwKeI30s5oMfBrYFjgjIl4HHgF+TBrRGAM8COwvaWlEtAEtkv6Zj9WWj/1u2v0MSP9mdsj7aSKNAt6R/50tyvm9g1TgHgwc0i6PBcBZpJ91G3C6pKu66QurE17zYP3dd4DFwHd7sO2qknYAvg78N3COpC2BZ0j/6ZYskbQ1aWj89IjYPCLelY/5MUlbAYcDV0fEanmbscBWHRQO44CfAvvkY30d+C0wHziM/K2zk8Jh/4h4MCL+FhELgfNIH9J/jojBwJWkaYhtSEXACRGxAzCj3fuZnGPlTgFWkAqFLYF5wPeAq4E9IqIZ2In04bF73uYTpG/EnwJG5m/L2+XXNu4g/zfJIyf7A7d0kz+kD68DJG3RSeHw5dw3f4+IfwInARMlPRcRY4HdgF0lbQF8DThN0uu80ed7dLDPnYF9JY3L7/vInOfVpGmlLYBzSSNG7d/bYFJxcl5utyfw3Vzk7pybfbCTwuGd+b08EhEvkIqDa4Ef5dfPJhWD2wBbAWsDx0XEO4BjScXVtsBNwPaSfkyaHjpR0jXAFODS/Lu/CbARMLGDPNpb+TMAtiYVx++TtBlwKel3qGQb0rTMeFIRtm8HeZwKnJXfx+dIhbz1Ey4erF/L324/CxwaEbt3176d0recJ4DnJT1U9nytsnYX5GPNI/2H/GHSB+gY4OY8UnA56Vt3acrhbkkrOjjmh4CbJf097/MPwAuk/2y788v8Ab15zn0Y8Lv82qbAO4GLcz5/zK9vBdwKrBoR20bEZqRvkje32/depG/mD+Tt9wY2k/Q0qZgqfRicDnwwTzmUiofbgc3zmoZTgB9J+r9O3sMZ+YPxoZzX7aRvtF3lD/CMpLld9M3ZuW/eA9wNLMv7J293MDApIr4HHEn69tydWyUtyo8fIP1OvCfvc1b++xbgrx1suympOL06t5tH6quPVnDcUgG5OXAy6ffs15KW59f3Ao7I/XQf8N6c13PAQ8D9EXEm8KCk33Sw/5OBBRFxEmkEZT0q64+VPwNJd5FG3Y7Ix/pMu338j6RlOee/8OZ/TyW/An4cEZeTfr++WkEOVidcPFi/J+kZ0pDzpaRvYSVtpOHUkqHtNi3/Bruczr1e9rg5tx1EKgImlP6QhnBLHySLO9lXaYi2XDNp+LkieXrmGGB14Iyy/f67g3wuyYspLyJ9gB4KXNTBAstBpOmI0rbv5Y31ENeQphY+QhodmEsaMVgi6QlJT5KKptNJQ+y/z1M6HTkxH2PL/OdrucjqNP+8XWf92b5vXgEOIo1cfBkgIrYG7sq53USanmnqbB9lykd/Sr9LKzrY9nX+01v+OQNIuoQ06vDrPJpR2ve+Zf20PXBMLqQ/QBpZWgicHREdLRi9gjRSNpc0inF/u/fUBJCnssqt/BlExETg+vz0t6TRtPJ9dNR37d/bBaSi53+BPYCHI2LVDvK1OuTiwRpCXoQ3izRsW7KANMdKXkn/gR7ufnLexwak4e+b85+P5GkIIuJjpLndYd3s62bSNMDGebsPkeaF/1QkoVxAfAE4KiK2Iq3OXxIRn837fQepkCmNaMwkjRTsyxsfyOVuBI6JiKF5imIGqRiANEx/INBcNvryA/LITV5DcQlwk6ST8762LvJ+Ksi/8h1JLwLHA6dGxNuBXYB7JZ1FGtHYm/QBDKkYKPKB/hiwLCI+mvMsfetvXyjMAZZHxKdzu/WAfUgflEWdTPodOTo/v5E0TdMUEauQiotj8rqcvwKPSTqdVBiUppHK3+cepGmbX+bn2/NGf6z8N0P6mXdmd+B3kn5Cmooo79OurMwjIu4kTe3NJBUzawDrVrAPqwMuHqyRTCV9myo5DxgTESJ9uP2hh/tdNSLuB24AvijpcUmPkv7D+0Uegv8WaZFll9+Q83ZHkdZH/JW0ruDjkv5dNClJt5OmS35MGg35JHBYRDxM+oCfLumO3PZ50jfMh/XGQtBy3wKeIg3PP0r6pnh8Wc5tvDHVcSPpw6w07fMz0gfHoxFxH2lE5NyC7+W1rvIvStLlpCH9M0nftNeOiMfye1sMrBURpVMgl0bEn6lgNCKPkuwDfDMiHiD10fOkBbPl7ZaTPlC/lN/P70kf2Lf04L28RCogTo2It5F+z1cjTQc8nP/+QZ52+xVwb0TcS1pHcFzezbWk9TqHkKYHromIv5Cm5P7IG9NtU0lTCfeT1ivM7yStnwK75n3cT5rq2ygXnl0pz+Mk4LTcj7cCp0p6qrJesVpr8i25zcwqFxFnkK4N8Y88QvIQsHH+kDcbEHyqpplZMXNJC2WX88Ypii4cbEDxyIOZmZkV4jUPZmZmVoinLSqzCmnV8nw6Pi3LzMyskQwiXWPkHt58Wjvg4qFS2wHdXULWzMys0exMupjbm7h4qMx8gBdffIXW1v61RmT06BEsXFjR9XWsh9zH1eX+rS73b3X11/5tbm5izTVXg05O13XxUJnXAVpb2/pd8QD0y5z7G/dxdbl/q8v9W139vH87nKr3gkkzMzMrxMWDmZmZFeLiwczMzApx8WBmZmaFuHgwMzOzQlw8mJmZWSEuHszMzKwQFw9mZmZWiIsHMzMzK8TFg5mZmRXiy1ObmdmAs9qoQQxfZXifHKulZWSfHOfVZa/yyqK+ufGziwczMxtwhq8ynKYpTbVOo1e1zWjjFV7uk2N52sLMzMwKcfFgZmZmhbh4MDMzs0KquuYhIkYBdwJ7SXqqLH4M8BlJu+bnGwCXAesAAiZJWhwRawCXAxsDC4D9JD0fEUOBi4BtgSXAgZLmREQTcAawF9AKTJF0Rz7G8cAUUsF0iqSrq/nezczMGlXVRh4iYnvgdmDTdvHNgFPaNT8fOF/SOOBeYHqOfxuYLWk8MAM4J8enAq/k+LHAzBzfBxgPbAbsDcyMiMERsR3wWWACsBNwRkSs1Utv1czMbECp5rTFFOBoYF4pEBGrABcAXy+LDQF2Aa7MoZnAvvnxRNLIA8AVwJ65/cq4pNuAljx6MRH4haRWSY8DTwPvBz4GXC1pqaQXgFtJoxNmZmZWUNWmLSQdBhAR5eHTgYuBJ8tiawOLJK3Iz+cD6+fH6+XnSFoREYuAlvJ4u226it/TQbyQ0aNHFN2kLvTVOcYDmfu4uty/1eX+bRx99bPss+s8RMTuwAaSjouIXcteagba2jVvzX+3Pwm3Kb/WfpuexgtZuHAxra3tU61vLS0jWbCgb877Hajcx9Xl/q2ugdq/jVow9dbPsrm5qcsvzH15tsUBwOYR8SBwIbBtRPwSeAFYPSIG5XZjeGOq4zlgXYCIGAyMBBYCz+Z2JevmbYrGzczMrKA+Kx4kfU7SeEkTgMOAeyXtL2k5MBvYPzc9GJiVH9+Qn5Nfn53br4xHxE7AUklP5/ikiBgUEZuQFmvek/e3T0QMj4gW4MPAzVV+y2ZmZg2pXi5PfRRwaURMIy1yPCDHp5POmHgEeAmYlOPnARfk+DLgoBy/EtgeeDg//7ykJcCfI+IyUiExGJgu6bkqvyczM7OG1NTW1r/m8GtkQ+BJr3mwjriPq8v9W10DtX9bWkY25L0tqrDmYSPgqf94vVeOYmZmZgOGiwczMzMrxMWDmZmZFeLiwczMzApx8WBmZmaFuHgwMzOzQlw8mJmZWSEuHszMzKwQFw9mZmZWiIsHMzMzK8TFg5mZmRXi4sHMzMwKcfFgZmZmhbh4MDMzs0JcPJiZmVkhLh7MzMysEBcPZmZmVoiLBzMzMyvExYOZmZkV4uLBzMzMCnHxYGZmZoW4eDAzM7NCXDyYmZlZIS4ezMzMrBAXD2ZmZlaIiwczMzMrxMWDmZmZFTK42geIiFHAncBekp6KiMOBqUAbcC9whKTXImICcCEwCrgNOFLSiojYALgMWAcQMEnS4ohYA7gc2BhYAOwn6fmIGApcBGwLLAEOlDQnIpqAM4C9gFZgiqQ7qv3+zczMGk1VRx4iYnvgdmDT/HxT4ETg/cAW+fhH5+aXAcdI2hRoAqbk+PnA+ZLGkYqN6Tn+bWC2pPHADOCcHJ8KvJLjxwIzc3wfYDywGbA3MDMiql48mZmZNZpqT1tMIRUH8/LzZcBRkhZJagP+AmwQEWOBYZLuzu1mAvtGxBBgF+DK8nh+PJE08gBwBbBnbr8yLuk2oCWPXkwEfiGpVdLjwNOkIsbMzMwKqOo3b0mHAURE6flcYG6OtQDHAJOB9YD5ZZvOB9YH1gYWSVrRLk75Nnl6YxHQ0sW+OotXbPToEUWa142WlpG1TqHhuY+ry/1bXe7fxtFXP8uaDNtHxNuBWcBFkm6NiB1JayBKmkjrEprbxcnxUptynW3TXbxiCxcuprW1fTr1raVlJAsWvFzrNBqa+7i63L/VNVD7t1ELpt76WTY3N3X5hbnPz7aIiHGkBZSXSvpWDj8LjClrti5pquMFYPWIGJTjY3hjCuS53I68dmEksLCLfXUWNzMzswL6tHiIiJHATcA0ST8sxfN0xtI8AgFwEDBL0nJgNrB/jh9MGrEAuCE/J78+O7dfGY+InYClkp7O8UkRMSgiNiEt4rynOu/UzMyscfX1tMVhwNuA4yPi+By7VtLXgUnAjHxq5/3Aufn1o4BLI2IaaZHjATk+nXTGxCPAS3l7gPOAC3J8GakQgbTocnvg4fz885KWVOE9mpmZNbSmtrb+NYdfIxsCT3rNg3XEfVxd7t/qGqj929IykqYp7ZfO9W9tM9qqseZhI+Cp/3i9V45iZmZmA4aLBzMzMyvExYOZmZkV4uLBzMzMCnHxYGZmZoW4eDAzM7NCXDyYmZlZIS4ezMzMrBAXD2ZmZlaIiwczMzMrxMWDmZmZFeLiwczMzApx8WBmZmaFuHgwMzOzQlw8mJmZWSEuHszMzKwQFw9mZmZWiIsHMzMzK8TFg5mZmRXi4sHMzMwKcfFgZmZmhbh4MDMzs0JcPJiZmVkhFRUPEbFx/ntiREyPiNWrm5aZmZnVq26Lh4i4ADg5IsYDM4CNgYurnZiZmZnVp0pGHrYBvgB8CrhU0qHA2KpmZWZmZnWrkuKhWVIrsDvwhxwbXr2UzMzMrJ4NrqDN/0XELNJ0xa0RcTnwUKUHiIhRwJ3AXpKeiojdgLOAYcAvJU3L7SYAFwKjgNuAIyWtiIgNgMuAdQABkyQtjog1gMtzXguA/SQ9HxFDgYuAbYElwIGS5kREE3AGsBfQCkyRdEel78PMzMySSkYeDiV9SH9A0nJgNvD5SnYeEdsDtwOb5ufDSOslPgmMB7aLiD1z88uAYyRtCjQBU3L8fOB8SeOAe4HpOf5tYLak0lqMc3J8KvBKjh8LzMzxffIxNwP2BmZGRCXFk5mZmZWppHi4GViX9IGOpJ9KerXC/U8Bjgbm5efvBf4m6UlJK0gFw74RMRYYJunu3G5mjg8BdgGuLI/nxxNJRQ3AFcCeuf3KuKTbgJY8ejER+IWkVkmPA08D76/wfZiZmVlWyTfv44FPADdExKukD/KrJam7DSUdBhARpdB6wPyyJvOB9buIrw0syoVGefxN+8rTG4uAlh4co2KjR48o0rxutLSMrHUKDc99XF3u3+py/zaOvvpZdls85HUBd5BO19wf+AFpymBQD47XDLSVPW8irT+oNE6Ol9qUK7qvUrxiCxcuprW1fTr1raVlJAsWvFzrNBqa+7i63L/VNVD7t1ELpt76WTY3N3X5hbnb4iEiDiGdafFBYC5p6uDGHubzLDCm7Pm6pCmNzuIvAKtHxCBJr+c2pSmQ53K7Z/PahZHAwrJ9PVHhMczMzKyAStY8nEsqHE4HPi3pG5Lu7OHx/gRERGwSEYOAA4FZkuYCSyNix9zuoBwvLdDcP8cPBmblxzfk5+TXZ+f2K+MRsROwVNLTOT4pIgZFxCakRZz39PB9mJmZDViVFA+jgUmk9QHXR8QDEXFGTw4maSkwGbgKeBSYwxuLIScBZ0fEHGAEqWgBOAo4PCIeBXYGpuX4dGCHiHgktzk6x88DVsnxc0mFCPk4jwAPA78FPi9pSU/eh5mZ2UDW1NZW2Rx+RGwLfIw0WtAqabNqJlZnNgSe9JoH64j7uLrcv9U1UPu3pWUkTVPaL53r39pmtFVjzcNGwFPtX69kzcPPgD1IpzZeA3xK0mO9kp2ZmZn1O5Wcqnkf8DVJz1Q7GTMzM6t/lRQPFwPfi4hxpAs0nQ4cL2lxVTMzMzOzulTJgslzgJeAtwFLSfee+O9qJmVmZmb1q5LiYStJXwOW58tSTwImVDctMzMzq1eVFA+vt3s+iIJXZjQzM7PGUUnxcFtEfB8YFhF7AFcDt1Q3LTMzM6tXlSyYPBk4Bfg38B3Spam/Vc2kzMwGuhFrrMKwIUP75Fh9dZ+HJctfY/FLy/rkWFZdldwYazmpWHDBYGbWR4YNGcrO1x1X6zR61ey9zmIxLh4aQafFQ0T8StJ+EfEX/vPOlkjaoqqZmZmZWV3qauTh+/nvY/oiETMzM+sfOi0eJN2XHx4J/LckL5I0MzOzihZM/hE4PSLWBi4CLpH0fHXTMjMzs3rV7amakn4qaQfg48CawJ0RcU3VMzMzM7O6VMl1HkqGAasATfznhaPMzMxsgKjkltzHAZNJhcNFwA6S/lHlvMzMzKxOVbLmYRtgqqRbq5yLmZmZ9QOVrHmYBLwaEUdExNCIeF8f5GVmZmZ1qtviISImA5cAJwFrAL+NiClVzsvMzMzqVCULJqcC7wMWSXqBNI1xbFWzMjMzs7pV0S25JS0qPZH0DLCieimZmZlZPaukePhXREwg398iIiYB/6pqVmZmZla3Kjnb4kvAlcA7I2I+sATYu6pZmZmZWd2q5JbccyJiS2BTYFAKaXnVMzMzM7O61GXxEBG7Ay9Kuhd4LMe2jIgfS9qpLxI0MzOz+tJp8RARZwL7AcMi4nDgOuBM4Cjgsr5Jz8zMzOpNVyMPnwK2ANYFzgW+TLrOw66S7uiD3MzMzKwOdVU8vCzpJeCliNiatGjyi72x3iEiPgt8JT+dJemEfEbHhcAo4DbgSEkrImID0kjHOoCASZIWR8QawOXAxsACYD9Jz0fEUNI9OLYlLe48MK/baALOAPYCWoEpLoLMzMyK6+pUzdayx/8CjumlwmE4aSTjA8CWwM4RsRupQDhG0qakO3eWrmJ5PnC+pHHAvcD0HP82MFvSeGAGcE6OTwVeyfFjgZk5vg8wHtiMdLbIzIio5GwTMzMzK1PpLbkXS+qtC0MNysddDRiS/ywHhkm6O7eZCewbEUOAXUijHivj+fFE0sgDwBXAnrn9yrik24CWPHoxEfiFpFZJjwNPA+/vpfdkZmY2YHT1zXv9iDi3g8cASJrakwNKejkipgNzgFeBPwKvAfPLms0H1gfWJl0We0W7OMB6pW3y9MYioKU83m6bzuIVGz16RJHmdaOlZWStU2h47uPqcv82Dv8sq6uv+rer4uHHnTx+SyJiC+BzwFjg36Tpio+Qr2CZNZGmTZrbxeGN6ZSmdvHOtukuXrGFCxfT2to+nfrW0jKSBQternUaDc19XF0DtX8b9UO2Xn6W7t+uNTc3dfmFudPiQdKpvZLBf9oDuDnfZIuImAmcAIwpa7MuMA94AVg9IgZJej23mZfbPJfbPZvXLowEFgLP5nZPtNtXKd7+GGZmZlZApWseetNDwG4RsVo+A+LjpKmLpRGxY25zEOksjOXAbGD/HD8YmJUf35Cfk1+fnduvjEfETsBSSU/n+KSIGBQRm5CumHlPFd+nmZlZQ+rzsw0k3RQRWwH3kRZK/hn4HnANMCMiRgH3k87IgHRRqksjYhppkeMBOT6ddMbEI8BLwKQcPw+4IMeXkQoRSIsutwcezs8/L2lJdd6lmZlZ4+rqCpN7S/pNRKwiaVlvHlTS94Hvtws/BLy3g7ZzgV07iP8L+EQH8aXAIR3E20jTIyf0KGkzMzMDup62+Fb++66+SMTMzMz6h66mLRZFxOPA2yPi4fYvStqiemmZmZlZveqqePgosBXpUs9f7Jt0zMzMrN51darmy8BtETGRdErjNqSrQf4pv2ZmZmYDUCWnaq4OPA78CDgLmBsRvqyzmZnZAFVJ8fBD0p0st8rrHD5DKiLMzMxsAKqkeBgp6ZbSE0l/AIZXLyUzMzOrZ5UUD20RMbb0JCI2BF6vWkZmZmZW1yq5wuRpwN0R8XvSjaX2IF310czMzAagbkceJP2GdIXHO0mXkt5V0lVVzsvMzMzqVEX3tpAkQFXOxczMzPqBWtxV08zMzPoxFw9mZmZWSLfFQ0T8rC8SMTMzs/6hkpGHCRHRVPVMzMzMrF+oZMHkPOCRiLgbWFwKSppatazMzMysblVSPNyV/5iZmZl1XzxIOjUihgGbAI8Aq0p6teqZmZmZWV2qZMHk9sATwPXAesAzvqummZnZwFXJgskzgd2AhZKeBQ4CzqlqVmZmZla3Kikehkt6tPRE0g1UeGVKMzMzazyVFA/LI2JN0k2xiIiobkpmZmZWzyoZQfg28EdgTERcAXwEOLyqWZmZmVndquRsi+siYg6wOzAIOE3SY1XPzMzMzOpSpfe2GEIqHJbnP2ZmZjZAVXKq5qHALcB2wM7A7IjYp9qJmZmZWX2qZM3DccBWkuYDRMQGwHXAVdVMzMzMzOpTJcXDa6XCAUDS0xHxlqYuIuLjwDeA1YCbJH0pInYDzgKGAb+UNC23nQBcCIwCbgOOlLQiFzGXAesAAiZJWhwRawCXAxsDC4D9JD0fEUOBi4BtgSXAgZLmvJX3YWZmNhB1Om0REVtHxNbAQxHxXxHxnojYLCJ+ANzR0wNGxMbAT4G9gS2ArSNiT+Bi4JPAeGC7HINUIBwjaVOgCZiS4+cD50saB9wLTM/xbwOzJY0HZvDGBa2mAq/k+LHAzJ6+BzMzs4GsqzUPV+U/HwQmAteSLlG9L7DXWzjmp0gjC89KWg7sD7wK/E3Sk5JWkAqGfSNiLDBM0t1525k5PgTYBbiyPJ4fTySNPABcAeyZ26+MS7oNaMmjF2ZmZlZAp9MWkjaq0jE3AV6LiGuB0vqJR4D5ZW3mA+uT7qXRUXxtYFEuNMrjlG+TpzcWAS1d7OvpShMfPXpEpU3rSkvLyFqn0PDcx9Xl/m0c/llWV1/1b7drHiJiXWAysFZ5XNJJb+GYuwC7AotJIxpLyFewzJqAVtLISCVxcrzUplx3+6rYwoWLaW1tf9j61tIykgULXq51Gg3NfVxdA7V/G/VDtl5+lu7frjU3N3X5hbmS6zxcC7yX9GFb/qenngd+L2mBpCXANaQbb40pa7MuMA94tpP4C8DqETEox8fkOMBzuR0RMRgYCSzsYl9mZmZWQCVnWwyV9OlePOZ1wKX5rIiXgT1JaxdOiYhNgCeBA4GLJc2NiKURsaOkO0h39JwlaXlEzCatl/g5cDAwK+//hvz8u/n12bl9KX57ROwELJVU8ZSFmZmZJZWMPNwXEe/urQNK+hPwA+B24FFgLvAT0tTIVTk2hzcWQ04Czs6XyB4BnJvjRwGHR8SjpItXTcvx6cAOEfFIbnN0jp8HrJLj55IKETMzMyuokpGHO4AHI2I+ZZemlrRxTw8q6WLSqZnlbga27KDtQ6Rpk/bxuaR1E+3j/wI+0UF8KXBIzzI2MzOzkkqKhxNJ0whPVDkXMzMz6wcqKR5ekvSrqmdiZmZm/UIlxcMfIuJM0nqEZaWgpPurlpWZmZnVrUqKhwPz3+V30mwj3TvCzMzMBphui4cqXmnSzMzM+qFKrjB5XEdxSWf1fjpmZmZW7yqZtnhP2eOhwAdIp1WamZnZAFTJtMWh5c8jYj3goqplZGZmZnWtkitMvomkecCGvZ+KmZmZ9QdF1zw0AduSbkxlZmZmA1DRNQ9twNOkq06amZnZAFR4zYOZmZkNbJ0WDxFxCWmkoeErzwwAAA9ASURBVCNtkj5fnZTMzMysnnU18vDXDmJrA8cCT1UlGzMzM6t7nRYPkn5Y/jwidgMuBS4HplY5LzMzM6tTlZxtMRg4HZgMHCnpqmonZWZmZvWry+IhIt4FXAEsBraS9GyfZGVmZmZ1q9OLREXEocCfgGsk7erCwczMzKDrkYeLgFbglIg4uSzeRDrbYlRVMzMzM7O61FXx4Ftxm1mHRq85iObBw/vseC0tI6t+jNYVr7LwxderfhyzRtDV2RZz+zIRM+s/mgcPhzlNtU6jVzWPawNernUaZv1C4RtjmZmZ2cDm4sHMzMwKcfFgZmZmhbh4MDMzs0JcPJiZmVkh3V6eupoi4kxgbUmTI2ICcCEwCriNdCnsFRGxAXAZsA4gYJKkxRGxBuk+GxsDC4D9JD0fEUNJ16jYFlgCHChpTkQ0AWcAe5GuXzFF0h19+obNzMwaQM1GHiLiw8AhZaHLgGMkbUq6ENWUHD8fOF/SOOBeYHqOfxuYLWk8MAM4J8enAq/k+LHAzBzfBxgPbAbsDczM9+0wMzOzAmpSPETEWsB3gO/m52OBYZLuzk1mAvtGxBBgF+DK8nh+PJE08gDp/ht75vYr45JuA1ry6MVE4BeSWiU9DjwNvL9a79HMzKxR1Wrk4QLga8CL+fl6wPyy1+cD6wNrA4skrWgXf9M2+fVFQEsX++osbmZmZgX0+bB9RBwGPCPp5oiYnMPNQFtZsybSuoT2cXK81KZcZ9t0F6/Y6NEjijSvG31xad+Bzn3cGPxzrD73cXX1Vf/WYs5/f2BMRDwIrAWMIH2ojylrsy4wD3gBWD0iBkl6PbeZl9s8l9s9m9cujAQWAs/mdk+021cp3v4YFVu4cDGtre1rmfrW0jKSBQt8yd1qGoh93KgfAPX0c3QfV5f7t2vNzU1dfmHu82kLSbtLerekCcDXgWslHQosjYgdc7ODgFmSlgOzSQUHwMHArPz4hvyc/Prs3H5lPCJ2ApZKejrHJ0XEoIjYBNgUuKea79XMzKwR1dPZBpOAGRExCrgfODfHjwIujYhppEWOB+T4dNIZE48AL+XtAc4DLsjxZaRCBNKiy+2Bh/Pzz0taUsX3Y2Zm1pCa2tr61zB8jWwIPNmb0xZrrjaEwcNX7ZV91YsVry7lxVeW1zqNPjdgpy0a7K6ajGurq59jS8tIdr7uuFqn0atm73VW3fRxS8tImqY01u9w24ze+x0um7bYCHiq/ev1NPIwoAwevipPtOxc6zR61TsXzIYBWDyYmQ00vjy1mZmZFeLiwczMzApx8WBmZmaFuHgwMzOzQlw8mJmZWSEuHszMzKwQFw9mZmZWiIsHMzMzK8QXibKGNHqNQTQPGd5nx+uLm+y0Ln+VhS+9XvXjmJl1x8WDNaTmIcPh54116dnmA9uA+ri0r5kNbJ62MDMzs0JcPJiZmVkhLh7MzMysEBcPZmZmVoiLBzMzMyvExYOZmZkV4uLBzMzMCnHxYGZmZoW4eDAzM7NCXDyYmZlZIS4ezMzMrBAXD2ZmZlaIiwczMzMrxMWDmZmZFeLiwczMzApx8WBmZmaFDK7FQSPiG8B++en1kk6KiN2As4BhwC8lTcttJwAXAqOA24AjJa2IiA2Ay4B1AAGTJC2OiDWAy4GNgQXAfpKej4ihwEXAtsAS4EBJc/roLZuZmTWMPh95yEXCR4CtgAnANhFxAHAx8ElgPLBdROyZN7kMOEbSpkATMCXHzwfOlzQOuBeYnuPfBmZLGg/MAM7J8anAKzl+LDCzam/SzMysgdVi2mI+cLyk1yQtBx4DNgX+JulJSStIBcO+ETEWGCbp7rztzBwfAuwCXFkez48nkkYeAK4A9sztV8Yl3Qa05NELMzMzK6DPpy0kPVJ6HBHvIk1fnEcqKkrmA+sD63USXxtYlAuN8jjl2+TpjUVASxf7errS3EePHlFp0wGrpWVkrVNoaO7f6nL/Vp/7uLr6qn9rsuYBICI2B64HTgRWkEYfSpqAVtLISFsFcXK81KZcd/uq2MKFi2ltbX/YnmnUf0ALFrxc6xQA92+1uX+rz31cXe7frjU3N3X5hbkmZ1tExI7AzcApki4FngXGlDVZF5jXRfwFYPWIGJTjY3Ic4LncjogYDIwEFnaxLzMzMyugFgsm3wH8hnS2wy9y+E/ppdgkFwQHArMkzQWW5mID4KAcXw7MBvbP8YOBWfnxDfk5+fXZuf3KeETsBCyVVPGUhZmZmSW1mLY4AVgVOCsiSrGfApOBq/JrN/DGYshJwIyIGAXcD5yb40cBl0bENNK6hQNyfDowMyIeAV7K20NaV3FBji8jFSJmZmZWUC0WTH4J+FInL2/ZQfuHgPd2EJ8L7NpB/F/AJzqILwUOKZiumZmZteMrTJqZmVkhLh7MzMysEBcPZmZmVoiLBzMzMyvExYOZmZkV4uLBzMzMCnHxYGZmZoW4eDAzM7NCXDyYmZlZIS4ezMzMrBAXD2ZmZlaIiwczMzMrxMWDmZmZFeLiwczMzApx8WBmZmaFuHgwMzOzQlw8mJmZWSEuHszMzKwQFw9mZmZWiIsHMzMzK8TFg5mZmRXi4sHMzMwKcfFgZmZmhbh4MDMzs0JcPJiZmVkhLh7MzMysEBcPZmZmVsjgWifQ1yLiQGAaMAT4kaQf1zglMzOzfmVAjTxExNuB7wA7AROAwyNis9pmZWZm1r8MtJGH3YA/SPoXQERcCXwGOK2b7QYBNDc39Woyg9+xbq/urx70dh+9JauNrXUGva6u+neI+7fa1h22Zq1T6HX11MdjR/t3uIL9DOro9aa2trZeOVB/EBFfAVaTNC0/Pwx4r6TDu9l0J2B2tfMzMzOrMzsDt7cPDrSRh2agvFpqAlor2O4eUgfOB16vQl5mZmb1ZBAwhvT59x8GWvHwLKkIKFkXmFfBdsvooPIyMzNrYE909sJAKx5+D3wzIlqAV4B9gO6mLMzMzKzMgDrbQtJzwNeAW4AHgZ9L+nNtszIzM+tfBtSCSTMzM3vrBtTIg5mZmb11Lh7MzMysEBcPZmZmVoiLBzMzMyvExYOZmZkVMtCu8zDgRMQo4E5gL0lP1TidhhIR3wD2y0+vl3RSLfNpNBFxGuneM23ARZLOqnFKDSkizgTWljS51rk0moi4BVgHWJ5DR0j6Uw1T6jUuHhpYRGwPzAA2rXUujSYidgM+AmxF+nD7n4j4lKRraptZY4iIDwAfArYAhgCPRsT1klTbzBpLRHwYOAS4vta5NJqIaCL93ztW0opa59PbPG3R2KYAR1PZJbitmPnA8ZJek7QceAzYoMY5NQxJfwQ+mP/TXYf0ReeV2mbVWCJiLeA7wHdrnUuDivz3TRHxUEQcU9NseplHHhqYpMMAIqK7plaQpEdKjyPiXaTpix1rl1HjkbQ8Ik4FTgB+DTxX45QazQWkK+6+o9aJNKg1gZuBL5JGz26NCEn639qm1Ts88mD2FkTE5sD/AidK+lut82k0kr4BtJA+4KbUOJ2GERGHAc9IurnWuTQqSXdJOljSvyX9E7gI+Fit8+otLh7MeigidiR9szhF0qW1zqeRRMS4iJgAIOlV4GrS+gfrHfsDH4mIB4HTgE9ExNk1zqmhRMROeU1JSRNvLJzs9zxtYdYDEfEO4DfA/pL+UOt8GtDGwKkRsRNpQeongYtrm1LjkLR76XFETAZ2lfTl2mXUkNYATouI95OmLQ4BjqxtSr3HIw9mPXMCsCpwVkQ8mP80zH8MtSbpBtIZAA8A9wF3SvpFbbMyq5yk63jz7/DFku6qbVa9x3fVNDMzs0I88mBmZmaFuHgwMzOzQlw8mJmZWSEuHszMzKwQFw9mZmZWiK/zYGZvEhE7AKcDo0lfMJ4BTihdkjsibgIOzFfNq8bxJwBfknRoRMwEdgcW5JebgRHATyX9oJv9bAScKWmfiFgPuFLS+3uY00jgV8CnJS3pyT7MGolHHsxspYhYBbiOdNOvLSS9G7gcmBURg3Kz3TvdwVs/fjPpMr7TysJnS5qQ/2wB7Ap8IyLGdbO7seSbE0ma19PCIW//MnAF8K2e7sOskXjkwczKDSddGW9EWexyYBEwKCIuzLFbIuJjwJbAV4GhpLtfXippekTsCpxDuhPmCGBn4ELgXUAr6aI5R0hqbXf8/YAnJXV1E6z1SZf6fRkgIr5KugLlMGA10gW8rs3He3tE3AgcAfxV0oiI+CawITCGVGA8B3xW0vyI2A74SX4/T+TXj5N0K2nk4fsRcYakf3TdjWaNzSMPZraSpBeBk4D/iYi/R8T/Aw4Ffp9vP35obvpB4FngeOAQSdsCOwBfiYi1c5t3Awfk0YJPACMlTQC2y69v3EEKnyGNfJT7cr6C598j4p85v4mSnouIscBupMsrb0G6S+Rpkl4HDgOekLRHB8fZGdhX0jhSgXNkRAwm3UNjet7XucCEsr5ZCtxDA93cyKynXDyY2ZtIOgt4GzAVmA+cDDwQEau3a9cGfBzYJiK+AZxFGhFYLTd5RtLc/Ph2YPOIuBU4BfiRpP/r4PDjgPbxs3PR8R7gbmAZcGvOYS5wMDApIr5HunfACLp3q6RF+fEDwFp5/0ialf++Bfhru+2eJE+FmA1kLh7MbKWI2DEiTpT0sqTrJJ0EbE66OdXu7dquRvrg3Rq4HziRdNfAptxkcamtpCeBTUgLMUcBv4+Ij3eQQhud/L8k6RXgIOADwJdzDlsDd+V93gR8v+z4XSlf9NiWt1nRwbavt3u+vIOY2YDj4sHMyi0ApuW7WZaMAVYH/pKfv066S+C7SB/a0yT9jrSQcRVgEO1ExBeAS4CbJJ0M3EgqOtoT8M7OksvTKseT7rj5dmAX4N48WvJHYO+y46/IeVbqMWBZRHw05/xe0mhE+Q2ANgLmFNinWUNy8WBmK0l6nPQB/N28xuBR0kLBQyUpN/s16YO6lbQ+YU5EPEaawniUNMLQ3s9IH+qPRsR9pGLk3A7aXQl8tJscLyctuDyTdAbE2vn4j5JGO9bKp1Y+CiyNiD9TwWiEpBXAPsA3I+IBUpHyPPAqQEQMJa3r+F13+zJrdL6rppnVjXw66H3kBZE1OP4ZpGtD/CMi3gE8BGws6aWImAxsLunEvs7LrN545MHM6kY+S2IK8N0apTAXuDmPPPwOOCwXDiOAA4Fv1igvs7rikQczMzMrxCMPZmZmVoiLBzMzMyvExYOZmZkV4uLBzMzMCnHxYGZmZoX8f2S6hq6X6yzBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot how many reviews we have of each star\n",
    "star_x = Restaurant_reviews.review_stars.value_counts().index\n",
    "star_y = Restaurant_reviews.review_stars.value_counts().values\n",
    "\n",
    "plot.figure(figsize=(8,5))\n",
    "# colors are in the order 5, 4, 3, 1, 2\n",
    "bar_colors = ['darkgreen', 'mediumseagreen', 'gold', 'crimson', 'orange']\n",
    "plot.bar(star_x, star_y, color=bar_colors, width=.6)\n",
    "plot.xlabel('Stars (Rating)')\n",
    "plot.ylabel('Number of Reviews')\n",
    "plot.title(f'Number of Reviews Per Rating of {businesses_to_analyse}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Breakdown of restaurants per state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>categories</th>\n",
       "      <th>city</th>\n",
       "      <th>name</th>\n",
       "      <th>review_count</th>\n",
       "      <th>stars</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AB</th>\n",
       "      <td>298</td>\n",
       "      <td>298</td>\n",
       "      <td>298</td>\n",
       "      <td>298</td>\n",
       "      <td>298</td>\n",
       "      <td>298</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AZ</th>\n",
       "      <td>5216</td>\n",
       "      <td>5216</td>\n",
       "      <td>5216</td>\n",
       "      <td>5216</td>\n",
       "      <td>5216</td>\n",
       "      <td>5216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IL</th>\n",
       "      <td>157</td>\n",
       "      <td>157</td>\n",
       "      <td>157</td>\n",
       "      <td>157</td>\n",
       "      <td>157</td>\n",
       "      <td>157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NC</th>\n",
       "      <td>1405</td>\n",
       "      <td>1405</td>\n",
       "      <td>1405</td>\n",
       "      <td>1405</td>\n",
       "      <td>1405</td>\n",
       "      <td>1405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NV</th>\n",
       "      <td>3989</td>\n",
       "      <td>3989</td>\n",
       "      <td>3989</td>\n",
       "      <td>3989</td>\n",
       "      <td>3989</td>\n",
       "      <td>3989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OH</th>\n",
       "      <td>1262</td>\n",
       "      <td>1262</td>\n",
       "      <td>1262</td>\n",
       "      <td>1262</td>\n",
       "      <td>1262</td>\n",
       "      <td>1262</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ON</th>\n",
       "      <td>3092</td>\n",
       "      <td>3092</td>\n",
       "      <td>3092</td>\n",
       "      <td>3092</td>\n",
       "      <td>3092</td>\n",
       "      <td>3092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PA</th>\n",
       "      <td>1063</td>\n",
       "      <td>1063</td>\n",
       "      <td>1063</td>\n",
       "      <td>1063</td>\n",
       "      <td>1063</td>\n",
       "      <td>1063</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QC</th>\n",
       "      <td>591</td>\n",
       "      <td>591</td>\n",
       "      <td>591</td>\n",
       "      <td>591</td>\n",
       "      <td>591</td>\n",
       "      <td>591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SC</th>\n",
       "      <td>82</td>\n",
       "      <td>82</td>\n",
       "      <td>82</td>\n",
       "      <td>82</td>\n",
       "      <td>82</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WI</th>\n",
       "      <td>524</td>\n",
       "      <td>524</td>\n",
       "      <td>524</td>\n",
       "      <td>524</td>\n",
       "      <td>524</td>\n",
       "      <td>524</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       business_id  categories  city  name  review_count  stars\n",
       "state                                                          \n",
       "AB             298         298   298   298           298    298\n",
       "AZ            5216        5216  5216  5216          5216   5216\n",
       "IL             157         157   157   157           157    157\n",
       "NC            1405        1405  1405  1405          1405   1405\n",
       "NV            3989        3989  3989  3989          3989   3989\n",
       "OH            1262        1262  1262  1262          1262   1262\n",
       "ON            3092        3092  3092  3092          3092   3092\n",
       "PA            1063        1063  1063  1063          1063   1063\n",
       "QC             591         591   591   591           591    591\n",
       "SC              82          82    82    82            82     82\n",
       "TX               1           1     1     1             1      1\n",
       "WI             524         524   524   524           524    524"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Restaurant_business.groupby('state').count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "restaurants_per_state = Restaurant_business.groupby('state').count()[['business_id']].rename(columns={'state': 'State', 'business_id': 'Restaurants'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1ffb4581e10>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAJWCAYAAACJconrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de5idVWEv/u8kA0k0CVIYBbVIvbDwaCuWmwIiKurhKKKtooIoqFC8tGppj3oA73rqT/FaOYciiJqiVjyWUqHSonJRsUrV1lKXqIAFooaI5iIgyeT3x97BMc1lT5hZe7/J5/M8eZi99rvf+e7NXL6z3vW+e2zdunUBAGD2zRl2AACA7YXiBQDQiOIFANCI4gUA0IjiBQDQyPiwAwxgXpL9kyxNsnbIWQAANmdukt2TfD3JnRve2YXitX+SK4cdAgBgGh6X5KoNB7tQvJYmyW23rc7k5Oxcc2yXXRZm+fJVs7Lv2dTV3El3s3c1d9Ld7F3NnXQ3e1dzJ93N3tXcSXezz1buOXPGsvPO9076/WVDXShea5NkcnLdrBWv9fvvoq7mTrqbvau5k+5m72rupLvZu5o76W72ruZOupt9lnNvdHmUxfUAAI0oXgAAjSheAACNdGGNFwAwxdq1a3LbbcuyZs2vhh0lP/3pnExOTg47xrTNRO7x8R2z884TmTt38DqleAFAx9x227LMn3+v3Pveu2VsbGyoWcbH52TNmu4Vr3uae926dVm9ekVuu21Zdt1194Ef51AjAHTMmjW/yr3vvXjopWt7NjY2lnvfe/G0Zx0VLwDoIKVr+Lbm/4FDjQDQcYsWL8j8eTP/K/2OO9dk5Yrbt7jd0qW35PnP/4PsueeDkyTr1k1m9erVOeKIp+clL/mjaX3Ov/u7z2bBggV58pP/+1ZlviduueXmfPSj5+T1r3/DrH0OxQsAOm7+vPEcecqFM77fi844KisH3HbXXSdy3nnn33371luX5XnPe1ae9KSnZM89f2fgz/lv//btPPrR+04z6cz48Y+X5uabb5rVz6F4AQAz7tZbb826detyr3vdKx//+Hn54hf/MWvXTubAAx+Tl73sT/LLX67Om950apYvX54kefGLT8y8efNz1VVX5Jprvp5ddtk1ExMTee9735Xbb789t932sxx33PF55jOfnXPOOStJ7p5Ne/azj8wHP3hWvvnNa3LJJX+fX/zi5zn44EPz5Cc/dZOPX7781vzoRzfmJz/5cZ7+9KPyohe9JO9//7tzyy0354wz3pnjjjs+b3nL6bn99tszZ85YXvWqP88jH/m79/h1UbwAgHvs1luX5fjjj8mvfnVnfvGLn2fvvR+Rd7zj3fnhD3+QWv8jZ5/9sYyNjeWtb31DLr30kkxOTma33e6fd73r/bnuuppLL/2HvOIVr8ohhxyaRz963xx44GPz/vefkRe96CXZb78DcvPNN+X444/JM5/57M3mWLbsp1my5NMZHx/f7OO///3r8qEPnZ1Vq1bm6KOfmT/4g6Pzqlf9Wc49969yyimvzbnn/lUOOuiQHHPMC3P11V/Jv/7rtxQvAGA0rD/UODk5mb/8y/fmhhuuz/77H5gzz/xArr32O3nJS45Lktx55x253/12y9Oe9oycddaHcuutP81jH3tIjj/+Jf9ln6985avzta99NR//+Efygx98P7ff/sst5thrr70zPj6+xcfvu+9+2WGHHbLzzr+VxYsXZ/Xq33zD7P32OyCnnvo/873v1Rx00CH5wz88+p68PHdTvACAGTNnzpy8/OWvygknHJNPfOLjmZxcm6OPfn6e97wXJElWrlyZuXPn5l73ulfOP/+CXH31V/PlL1+RT35ySZYs+fRv7OsNb3hdFi1anIMPflye9KSn5J/+6fNJemcTrlv36ze4XrNmzd0fz5s3b4uPT5Idd9zx7o833F+S/N7v7ZMlS/4mX/nKVbnssktz8cUX5X3vO/Oevz73eA8AAFOMj4/nFa94dc4775zstdfe+fznL84vf/nLrFmzJq9//Sn50pcuy2c+86mcc85ZeeITD88pp7wut912W1avXp25c+dm7dq1SZKvf/2f89KXnpzHPe6wXH31V5Ika9euzU473SfXX/+DJMm1134ny5ffutEcm3r8psydO373/Wee+f58/vOX5Igjnp7XvOa1+d736sy8NjOyFwCAKR7zmIPyyEf+br797W/m8Y9/Yk466fhMTq7NgQcelCOOePrdi+tf+MLnZu7cuXnFK/4kixYtyn77HZCzzjozCxcuzItffGJe9rKXZt68HfOQhzwsu+9+/yxdeksOP/wpufzyL+QFL3hOStk7D3tY2WiGTT1+U/bcc8+sWrUyb33r6TnppFfkzW8+LRdffFHmzJmT005784y8LmMbTq2NoD2TXL98+apMTs5O1omJRVm2bNATZkdHV3Mn3c3e1dxJd7N3NXfS3exdzZ10N/t0c//4xzdmt90edPftYV7Ha3t9y6D1Nvx/MWfOWHbZZWGS/E6SG/7L573HnxEAGKqVK24f+HpbDJc1XgAAjSheAACNKF4A0EEdWKO9zdua/weKFwB0zPj4jlm9eoXyNUTr1q3L6tUrMj6+45Y3nsLiegDomJ13nshtty3LqlU/H3aUzJkzJ5OT3TurcSZyj4/vmJ13npjeY+7RZwQAmps7dzy77rr7sGMk2X4u4TFTFC8AoPO25lpmExOLBt52kGuaDULxAgA6b/688Rx5yoWztv+LzjhqRq6VZnE9AEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAj44NsVEr5YpL7JrmrP/RHSR6S5LQkOyR5X631Q/1tD0/yniQLknyq1npaf3yfJB9OsjjJFUlOrrWumbmnAgAw2rY441VKGUuyV5JH1Vr3qbXuk+SmJG9PckiSfZKcVEr5b6WUBUnOTXJUkocn2b+UckR/V0uSvLLWuleSsSQnzvizAQAYYYPMeJX+fy8tpeyS5OwkK5N8odb6syQppVyQ5NlJLk9yXa31+v74kiTPKaVcm2RBrfXq/r7OS/LmJP9npp4IAMCoG2SN185JLkvyrCRPSnJykj2SLJ2yzdIkD0xy/2mOAwBsN7Y441Vr/WqSr66/XUo5J701XG+bstlYksn0ity6aYwPbJddFk5n82mbmFg0q/ufLV3NnXQ3e1dzJ93N3tXcSXezdzV30t3sXc2ddDv7dMzE89xi8SqlHJJkXq31sv7QWJIbkuw+ZbPdktyS3tqv6YwPbPnyVZmcXLflDbfCxMSiLFu2clb2PZu6mjvpbvau5k66m72ruZPuZu9q7qS72buaOxmd7C3K3yDPc86csc1OFg1yqPE+Sd5VSplfSlmU5EVJXpDkSaWUiVLKvZL8YZJ/SPK1JKWU8tBSytwkxyS5pNZ6Y5I7SikH9/d5XJJLBvjcAADbjC0Wr1rr3yf5XJJvJrkmybm11i8nOTXJF5N8K8n5tdZ/rrXekeT4JJ9Jcm2S7ya5oL+rY5O8t5Ty3SQLk3xgZp8KAMBoG+g6XrXW05OcvsHY+UnO38i2lyV51EbGv53kgK2LCQDQfa5cDwDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0Mj4oBuWUt6dZNda6/GllH2SfDjJ4iRXJDm51rqmlLJHkiVJ7pukJjm21rqqlHKfJH+d5MFJliU5utb64xl+LgAAI22gGa9SypOSvGjK0JIkr6y17pVkLMmJ/fEzk5xZa907yTeSnN4ff1uSK2utD09ydpL3z0B2AIBO2WLxKqX8VpK3J3lH//aDkiyotV7d3+S8JM8ppeyQ5NAkF0wd73/8tPRmvJLkE0mO6G8PALDdGGTG66wkpya5rX/7/kmWTrl/aZIHJtk1yYpa65oNxn/jMf37VySZuEfJAQA6ZrNrvEopL03yn7XWy0opx/eH5yRZN2WzsSSTGxlPf3z9NlONTblvILvssnA6m0/bxMSiWd3/bOlq7qS72buaO+lu9q7mTrqbvau5k+5m72rupNvZp2MmnueWFtc/N8nupZRvJfmtJAvTK1e7T9lmtyS3JPlpkp1KKXNrrWv729zS3+bm/nY3lVLGkyxKsnw6QZcvX5XJyQ173cyYmFiUZctWzsq+Z1NXcyfdzd7V3El3s3c1d9Ld7F3NnXQ3+yjlXrR4QebPG/jcu2m54841Wbni9lnZd4vyN8j/ozlzxjY7WbTZV7bW+uT1H/dnvA6rtZ5QSvlOKeXgWuuXkxyX5JJa612llCvTK2vnJ3lhkkv6D7+4f/sd/fuvrLXetcX0AEBT8+eN58hTLpyVfV90xlEZjXo5PFtbaY9NcnYpZXGSf0nygf74y5N8tJRyWpIfJXl+f/z0JOeVUv49yc/7jwcA2K4MXLxqreeld6Ziaq3fTnLARra5MclhGxn/WZJnbGVGAIBtgivXAwA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0Mj7sAACwrVm0eEHmz5ver9iJiUUDb3vHnWuycsXt043FCFC8AGCGzZ83niNPuXDW9n/RGUdl5aztndnkUCMAQCOKFwBAI4oXAEAjA63xKqW8Jcmzk6xLck6t9T2llMOTvCfJgiSfqrWe1t92nyQfTrI4yRVJTq61riml7JFkSZL7JqlJjq21rprpJwQAMKq2OONVSnl8kicm+b0k+yX541LKo5Kcm+SoJA9Psn8p5Yj+Q5YkeWWtda8kY0lO7I+fmeTMWuveSb6R5PSZfCIAAKNui8Wr1np5kifUWtekN1s1nuQ+Sa6rtV7fH1+S5DmllAclWVBrvbr/8PP64zskOTTJBVPHZ/KJAACMuoHWeNVa7yqlvDnJtUkuS3L/JEunbLI0yQM3M75rkhX9kjZ1HABguzHwdbxqrW8spbwzyUVJ9kpvvdd6Y0km0ytyg4ynPz6wXXZZOJ3Np206F64bJV3NnXQ3e1dzJ93N3tXcSXezdzV30u3s09HV59nV3MnMZN9i8Sql7J1kfq31W7XWX5ZS/l96C+3XTtlstyS3JLkpye4bGf9pkp1KKXNrrWv729wynaDLl6/K5OSG3W1mTEwsyrJl3bsUXVdzJ93N3tXcSXezdzV30t3sXc2djE72FuVitp7nbGfvau5ksOxz5oxtdrJokEOND05ydillXillx/QW1J+VpJRSHlpKmZvkmCSX1FpvTHJHKeXg/mOP64/fleTKJM/tj78wySUDfG4AgG3GIIvrL07yuSTfTHJNkq/UWj+Z5Pgkn0lv3dd38+uF88cmeW8p5btJFib5QH/85UlOKqVcm+RxSU6buacBADD6BlrjVWt9U5I3bTB2WZJHbWTbbyc5YCPjNyY5bCsyAgBsE1y5HgCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoBHFCwCgEcULAKARxQsAoJHxQTYqpbwxydH9m5+rtf7PUsrhSd6TZEGST9VaT+tvu0+SDydZnOSKJCfXWteUUvZIsiTJfZPUJMfWWlfN6LMBABhhW5zx6hespyR5dJJ9kuxbSnl+knOTHJXk4Un2L6Uc0X/IkiSvrLXulWQsyYn98TOTnFlr3TvJN5KcPpNPBABg1A1yqHFpklNqrb+qtd6V5D+S7JXkulrr9bXWNemVreeUUh6UZEGt9er+Y8/rj++Q5NAkF0wdn7mnAQAw+rZ4qLHW+u/rPy6lPCy9Q44fTK+Qrbc0yQOT3H8T47smWdEvaVPHAQC2GwOt8UqSUsojknwuyZ8nWZPerNd6Y0km05tBWzfAePrjA9tll4XT2XzaJiYWzer+Z0tXcyfdzd7V3El3s3c1d9Ld7F3NnXQ7+3R09Xl2NXcyM9kHXVx/cJLPJHl1rfWTpZTHJ9l9yia7JbklyU2bGP9pkp1KKXNrrWv729wynaDLl6/K5OSG3W1mTEwsyrJlK2dl37Opq7mT7mbvau6ku9m7mjvpbvau5k5GJ3uLcjFbz3O2s3c1dzJY9jlzxjY7WTTI4vrfTvK3SY6ptX6yP/y13l3loaWUuUmOSXJJrfXGJHf0i1qSHNcfvyvJlUme2x9/YZJLtpgeAGAbMsiM158lmZ/kPaWU9WP/N8nx6c2CzU9ycX69cP7YJGeXUhYn+ZckH+iPvzzJR0sppyX5UZLnz0B+AIDOGGRx/auSvGoTdz9qI9t/O8kBGxm/Mclh08wHALDNcOV6AIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARhQvAIBGFC8AgEYULwCARsaHHQAANmXR4gWZP296v6omJhYNtN0dd67JyhW3b00s2GqKFwAja/688Rx5yoWzsu+LzjgqK2dlz7BpDjUCADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADQyPuwAAMyuRYsXZP686f24n5hYNPC2d9y5JitX3D7dWLBdUrwAtnHz543nyFMunLX9X3TGUVk5a3uHbYtDjQAAjSheAACNKF4AAI0oXgAAjSheAACNKF4AAI0oXgAAjSheAACNKF4AAI0oXgAAjSheAACNKF4AAI0oXgAAjSheAACNKF4AAI0oXgAAjSheAACNjA87AEBXLFq8IPPnTe/H5sTEooG2u+PONVm54vatiQV0iOIFMKD588Zz5CkXzsq+LzrjqKyclT0Do8ShRgCARgae8SqlLE7ylSRPr7XeUEo5PMl7kixI8qla62n97fZJ8uEki5NckeTkWuuaUsoeSZYkuW+SmuTYWuuqGX02AAAjbKAZr1LKgUmuSrJX//aCJOcmOSrJw5PsX0o5or/5kiSvrLXulWQsyYn98TOTnFlr3TvJN5KcPlNPAgCgCwY91HhiklckuaV/+4Ak19Var6+1rkmvbD2nlPKgJAtqrVf3tzuvP75DkkOTXDB1/J7HBwDojoEONdZaX5okpZT1Q/dPsnTKJkuTPHAz47smWdEvaVPHAQC2G1t7VuOcJOum3B5LMjmN8fTHB7bLLgunn3IaBj3le9R0NXfS3exdzZ10N3tXc09Xl59nV7N3NXfS3exdzZ3MTPatLV43Jdl9yu3d0jsMuanxnybZqZQyt9a6tr/NLZmG5ctXZXJyw+42MyYmFmXZsu6dyN3V3El3s3c1d9Ld7KOUe7Z/YczW82zxi66r2buaO+lu9q7mTgbLPmfO2GYni7b2chJfS1JKKQ8tpcxNckySS2qtNya5o5RycH+74/rjdyW5Mslz++MvTHLJVn5uAIBO2qriVWu9I8nxST6T5Nok382vF84fm+S9pZTvJlmY5AP98ZcnOamUcm2SxyU5betjAwB0z7QONdZa95zy8WVJHrWRbb6d3lmPG47fmOSwaScEANhGuHI9AEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCOKFwBAI4oXAEAjihcAQCPjww4AbF8WLV6Q+fOm96NnYmLRwNveceearFxx+3RjATSheAFNzZ83niNPuXDW9n/RGUdl5aztHeCecagRAKARxQsAoBHFCwCgEcULAKARxQsAoBFnNUJHzeZlGVySAWB2KF7QUbN5WQaXZACYHQ41AgA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADSieAEANKJ4AQA0ongBADQyPuwAMEyLFi/I/HnT+zaYmFg08LZ33LkmK1fcPt1YAGyjFC+2a/PnjefIUy6ctf1fdMZRWTlrewegaxxqBABoRPECAGhE8QIAaETxAgBoRPECAGhE8QIAaETxAgBoRPECAGhE8QIAaETxAgBoRPECAGhE8QIAaMSbZDMjFi1ekPnzpvflNDGxaKDt7rhzTVauuH1rYgHASFG8mBHz543nyFMunJV9X3TGUVk5K3sGgLYcagQAaETxAgBoRPECAGhE8QIAaETxAgBoRPECAGjE5SRGyGxeCytxPSwAGDbFa4TM5rWwEtfDAoBhc6gRAKARxQsAoJGmhxpLKcckOS3JDkneV2v9UMvPDwAwTM1mvEopD0jy9iSHJNknyUmllP/W6vMDAAxby0ONhyf5Qq31Z7XW1UkuSPLshp8fAGCoWh5qvH+SpVNuL01ywACPm5skc+aMzUamu832/gd1350XzOr+Z/N5zmb2ruZOupu9q7mT7mbvau6ku9m7mjvpbvau5k4Gyz5lm7kbu39s3bp1Mxhp00oppyaZX2s9vX/7xCT71lpP3sJDD0ly5WznAwCYQY9LctWGgy1nvG7qh1hvtyS3DPC4r/cftzTJ2lnIBQAwU+Ym2T29/vJftJzxekB6ze+AJKuTfCXJSbXWf24SAABgyJotrq+13pzk1CRfTPKtJOcrXQDA9qTZjBcAwPbOlesBABpRvAAAGlG8AAAaUbwAABpRvAAAGlG8YECllN8adgYApqeUcmkp5aBN3Pf3rfNsV8WrlPKKTYzvVEr5ROs8JKWUM4edYWP6XxN/UUp5TSnlgaWUa5PcWkq5oZRy4LDz0Q2llD1LKe8Ydo5tVSll51LKrlNuP76UMjHMTNuyDv/xuU+Sz5RSXrOR+x7QOkzLtwwaBU8tpTwpyQm11l8kSSnlsCQfS/L5YQYbRCnlhCTfqbV+vX/7fye5rtZ67nCT3SMvSPLyYYfYiHOS/GeSRyR5dZL3J/lwksP7Hz9meNE2r5Tyws3dX2v9WKss01VKGUvypiSX11q/0B/7aJIbaq1vHGa2QZVS5iQ5Mskfpff1cuFwE21eKeUNm7u/1vqWVlmmo5Ty6CQXJzkhyT/0h5+S5PxSyhG11n8dWrgtKKV8JMkmL6JZa31xwzjT8b1SymVJzk1yaa21KxcCvTnJH6ZXvg5K8uJa68r+fc2fw3Y141VrfUaSryb5RinlsFLK/5fkk0n+uNZ64o1e10UAAAr5SURBVHDTbV4p5Y+TnJxkxZThS5K8rJTysuGkmhGz9zb198zetdbXJDkpyXit9T211hW11v+XZIchZ9uSJyQ5rP/vCRv8O2xYoQb05vT+Ov3ulLG3Jfn9LRWEYSulPKCU8uYkNyb5YJLHJim11ucMN9kWjU35d/IGt0f1+zNJ3p3k+bXW9aUrtdZTk7w4yXuGlmowX0py+Qb/bk5yTJKHDC/WFu2R5KIkf5rkh6WUt5RSfmfImQaxrtb6w/T+YP5FkmtKKY/s39f8a3x7m/FKrfVdpZSbk3whyY+T/H6tdZA36x62lyQ5tNZ6d/GqtV5RSjkiyWVJ/s/Qkt0zo/oX011JUmudLKX8ZIP7RvmXUdL7q391erO4/5jkC7XWVcONNLBnJtm/1nrn+oFa63WllOcnuTrJqM6+XJjkUenNbj0vvfei/WGt9fqhBhtArfXN6z8upTxz6u0Rt3Ot9UsbDtZaP19KeecQ8gys1vrRqbdLKX+S3gzpa5N8YCihBlBr/WWSJUmWlFJ2T3Jsks+WUpYnOafWev5QA25B/+fKS/tHBb5QSvnzDOF30HZXvPrHeF+X5GVJDk3yt6WUY2qt3x9usi2anFq61qu13lpKmRxGoEGVUr6YjX9xjyVZ0DjOoHYopfx2erPCO/Y/Xj8DsONQk21BrfUBpZSHJHlcekXmnaWUZemVsEtrrV8basDNWzu1dK1Xa11VSrlrGIEG9IAkNyVZnuTWWuu6Usqo/lGxOV3KvEMpZU6t9Td+/vUP9Y709+h6pZQHJ/lI/+ZBHfg9dLda69Ik7y6lfDK992H+SJJRLV4/n3qj1vqxUso1ST6dIcwwblfFq39sen6Sx/anHc/qr5u6qpRyaq31nOEm3Kw1pZT71lp/OnWwlHK/JHOHlGlQbxp2gK2wML3p//WzW1fk17+U5g8l0TTUWn+Q5AdJziul3CfJUUlOSXJaknnDzLYFq0spD+nnv1sp5aFJRvYPjFrrfqWU301vvdEVpZRbkuxUStmt1vrjIcfbVl2e5I39f1OdluQb7eNMT3+W69Qkb0/ywQ6tl0r/Z8pz0pvxul9666RH9pBjrfWJU2+XUsbTW7+7OkP4/bldvUl2f43I2zbyF1JJ8ola6+8PJ9mW9adG/yTJnyf5ZpI7kuyX5IwkH6u1fmiI8bZ5pZQdkvxBemtg9q+1LhxypE3q/1A5JMl/T/LU9GYV/6n/7wsbmzkdFaWUp6V3qOXtSf4lva/z/ZO8Icmptda/GWK8gfRf/yOTHJ/kyUk+N8rrvKacjDGW3qHc06feP6onY5RSFqW3uP63k3wrva+VfZP8JMkzaq0/G2K8zSqlXJ7kgPTWqV234f0j/Jofnd4JUQeld1j9vFrrlcNNNbj+erST0lsHeJ/0fs6cWWu9tWWO7WrGK8kzN3aGTq21llJG9iy15O6p0flJzkvvB826JD9M8q5a618NM9uW9A+FbupQ47pa68jO2E35Rj0hyc7pfaOO7C/RvtvSW2N0QZJn1VpvGG6cwdVaP1dKWZvkfyV5X3qzXN9I8sr01mWOpFLK/dP7JfqI9F7719daP9ufkX7BUMNt2RP6/12X3nrRw/q3x/pjI1kC0vt+fGF6+e+T3tfKX6Z3NvLCJCNbvJJcn97P7wfkNy9nMOqv+bvT+yPodUlWJUkpZY/1d9ZafzSkXJtVSnlWemvo9k3y2fS+J88e1hm721vx2qRa66+GnWEAtyR5YnrHqw9Nb8H9A0spO9RaR3b9S621c2fPbuQb9bgM8Rt1ms5K8qT0/qp7YCnl0iRf3XCmd4StTG/2Yl16hwHmJvlVkreVUi6vtV48zHCb8JEk/5bkr5M8O8l707tszU/Sm5UeWbXWE0opj09vpmv/9F73ryd5a631iqGG27zL08s69WSXdUl2T2+N18j+QVdrPb6jr/lkeod21/8hvb4ojvpr/pkkf5PeMqPvJ3dPCAzF9nao8c70Ttnd0PqZlwc3jjSwUsqfJXlukhelV5i/muRV6Z16v6bW+uohxtvm9L8p/ybJaVO+UX84yl8jG+qfdfTU/r/90ysGn6+1/t+hBtuM/nX1lqR3CYkr0jtM+pj01u18v9b6uOGl27RSyndqrY/sf7xDkm/VWh8x5FgDKaU8McnHk7w1yZXp/QI9KL1Zx2M3dubgKCqlLEyv5D41yYm11n8ccqRN8pq31b90xAnprUm7IcknkpxSa91jc4+bLdvbjNf3k/yPYYfYSsel19Z/WUr5iyR/V2v9cP+Ck9cOOdu26PfS+0a9qpRyQ3rfqJ36fqm1Li2lnJ/eGpKD0zssc2CSkS1e6Z2I8fRa67emjF1TSjkmo33G3d0z5rXWu0opXZhBX++NSZ62wWv+zVLK1enN3B06nFiD618Y++z0ztz93SkXxxxVXvOGaq3fSXJKKeW1SZ6e3vrL+5VSPpfkQ61n0Tv1i2QG/KrWeuOwQ2yldf1rqCS9NQ1nJkn/tPXhpdpGjdo36nSUUp6RXtE6JMmD07v+1ReSPLfW+u/DzDaAnTb4ZZTSe0uYz6Y329sVo1wSN7R4w9c8SWqt14z6W8SUUu6d3sVSR3rGZSO85kNQa12T5G/Tu4zURHp/jP7v9E7SaGZ7K15fHnaAe2BN/xTehUkeneTSJCmlPCjJmmEG25aNyjfqNK1fiP7qJNd0aG1Xkiwopcytta5dP9C/Vt370jvJYVQ9opTywym3H9C/PfLLGJIsLKWM97/W79Y/O3Nkf0dsMOPyyA5dJDjxmg9drXVZeodJm6/BHNn/wbOh1vrKYWe4B/4ivVOmx5N8uH8Y6egk70jvbVaYZcP8Rp2OWutThp3hHvhckveUUv50ffkqpcxN72yqUS67ew07wD3w+STvTO86b0nufs3fm97/j1H1j+m9w8RTkvzrlJn/LpRdr/l2bLsqXl1Wa72glPKVJLvWX7/566okL+3KQkwYwOnpzTD+oJTyzfQO2e2b5D/Suwr/SOrwEoak9zY1F5VSvp/epTvG07tG4L+nd+26UTWyF+wcgNd8O7ZdndUIdEMp5eD0LjA5luSfa61XDTnSNq9/eYP1lzb4mtd89nnNt0+KFwBAI527sCUAQFcpXgAAjShewDavlHJp/3pgM7IdwNZSvIDtwZNneDuArWJxPbDN6L933EeSPCy9N/S9Jr1T9Y9P8p303jLsUem9J96OSe6b5KO11tNLKR/ZYLvJJH+ZZI8kOyT5ZK31HQ2fDrANMuMFbEuelWRRrXWf9E7TT5K39//7hCQ3pXfRyhfVWvdL7w24X19K2bXWesL67Wqt/5nemxifW2vdN71LWxzev2gxwFZTvIBtyVXpvX3Pl5K8Lsn7aq3fX39nrXVdkiOT7FtKeWN67zk3luTeU3fSfz+6xyd5aynlW+m93+UeSfZp8SSAbZcr1wPbjFrr9aWUhyY5LMkTk/xTKeXu93jsF6pvpvem21cmOTe9K+KPbbCruf2xg9a/OX1/0f0ds/0cgG2bGS9gm1FKeVl6a7wurbW+Nr33xPv9JGvTW6f1sCSLk5xWa70ovYI2L72ilfXb1VpXpDfL9af9/d4nyZeTHNXsyQDbJMUL2JZ8LL0SdW0p5ZokOyX5QJJPJ7k8vQXzf5/ku6WU/0jvsOO1SR7af/ynk1xeSnlkkmOSPKaU8m9JvpbkE7XWv275ZIBtj7MaAQAaMeMFANCI4gUA0IjiBQDQiOIFANCI4gUA0IjiBQDQiOIFANCI4gUA0Mj/D/+RKXkDcXF5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "restaurants_per_state.sort_values(by='Restaurants').plot.bar(figsize=(10,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Until now, we have worked with full dataset and created CSV file for the data we need for our NLP tasks. But this data is huge, so we also prepare smaller files for testing purpose, by selecting data for Illinois. In the following, we repeat the same steps as above to create CSV files for data of restaurants in the state of Illinois"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "Restaurant_AZ = pd.DataFrame(yelp_data.business.find({\"categories\":{\"$regex\" :\".*\"+businesses_to_analyse+\".*\"}, \"review_count\":{\"$gte\":min_review_count}, \"state\":state_filter },  {'business_id':1, 'name':1, 'city':1, 'state':1, 'stars':1, 'review_count':1, 'categories':1, '_id': 0}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "Restaurant_AZ_reviews = pd.merge(Restaurant_AZ,All_reviews, on='business_id').rename(columns={'stars_x':'business_stars', 'stars_y':'review_stars'})\n",
    "Restaurant_AZ_reviews['text'] = Restaurant_AZ_reviews[u'text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "Restaurant_AZ_reviews['text']  = Restaurant_AZ_reviews['text'].map(lambda x: x.encode('unicode-escape','strict').decode('utf-8').replace('\\\\u',''))\n",
    "Restaurant_AZ_reviews['name']  = Restaurant_AZ_reviews['name'].map(lambda x: x.encode('unicode-escape','strict').decode('utf-8').replace('\\\\u',''))\n",
    "Restaurant_AZ['name'] = Restaurant_AZ['name'].map(lambda x: x.encode('unicode-escape','strict').decode('utf-8').replace('\\\\u',''))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the IL restaurant and review files. We use state abbreviation as prefix to file name. All these fies (for each state) are uploaded to project S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "Restaurant_AZ_reviews.to_csv('processed_data/'+state_filter+'_restaurant_reviews.csv',encoding='utf-8',line_terminator='\\r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "Restaurant_AZ.to_csv('processed_data/'+state_filter+'_restaurants.csv',encoding='utf-8',line_terminator='\\r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The illinois dataset contains only around 200k reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(19220, 13)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Restaurant_AZ_reviews.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Number of Reviews Per Rating of Restaurants')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfwAAAFNCAYAAADlxN3DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de5hdVX3/8fdMuAWSAIYoQQRF5AuiEETwggha8IbXKvIIgtAfNxW1XhCroQq2UhVBaUEtF8GCFgWvaKo1gqKClqvVwFdLMYLEksYLhJsJye+PtUYO05nMmXD2zOTs9+t55smcdfbZ+3vWTOaz19r77D2wevVqJElSfxuc7AIkSVLzDHxJklrAwJckqQUMfEmSWsDAlySpBQx8SZJaYL3JLkDtERGPB24FjszMczva3wU8JTMP79F2fgW8JjOv6cX6xtjWLGABsBlwYmZ+qeO5DwBvBn5TmwaAWcCXgXdm5rg/ExsRTwfek5mveYSld7u9K4BtgT8Cq4ENgGuAN2bmvY9gvecD+wNLa9MgMAP4VGZ+ZIzXPgE4NTNfHRFbAZdk5rPXtpZuRcQngRcBn8vM93W070v5HciOxWcCi4DDM3PZWm7vb4EbM/Ora110D0yVOvTIGfiaaKuAj0XEDzIzx1x66psHPCYztx/l+Ysz87ihBxGxOfBT4Fv1a1zqTsyEhH2H4zPzEoCIGAC+AJwMvOsRrvf0zDx16EFEbAPcFBFfy8yb1/C6bYEAyMw7gMbDvjoG2CYzbx/huVsyc97Qg4iYBlxK6aO/WcvtPZ+y0zDZpkodeoQMfE20+4CPAZ+LiGdl5p86n6wjv58NBUHn4zpy/xzlD9DmwEeAvYDdgRXAy2sAALw5InYFNgQ+lpnn1fW9DJhPGaneC7wrM6+qo/FnAVtRRjOvH1bXK4H3U0aidwPvoIx6zwMeGxE3AM/KzPvGeP+PATYGfl/XuxPwCWA2MA04IzPPi4jPAddm5sfqcm8E9gU+CfxTZj4lIjYAPgzsU197PfBW4K+A3TPz0IhYH1gGvC0zPxMRz6n9/xfAZ4AnUXbCrgWOycxVayo+M1dHxOXAS8aof9/afg9l5L5HZj4wRt9sTZkFubuu+73AK4DpwCaU8PwacA6lz79FCeGfZeaM+jN8PDCXslPwG+D1mbkkIvaofbcBcEt9/h2ZeUVnARGxM/BP9f2spvzufDYirqy1LYiIN2XmlWO8l1nAHOCHdb2b1v54KrA+sJCyI7UyIk4CXgX8ifKzOhz4S+DpwEcj4kHg58CZlJmDucANwEGZeX9ErAbmZOb/1m2trtt+CsN+BpT/M8+s6xmgzLb9sP4/u6vW9zjKTulhwBuG1bEUOI3ys14NnJKZl47RF5oiPIavyfD3wHLgQ2vx2o0y85nA3wL/DHwiM3cFbqP8oRxyX2Y+jTJtfEpE7BwRT6rbfElm7gYcDXwpIjapr9kW2G2EsN8R+BTw6rqtvwW+CiwBjqSO7kYJ+4Mi4oaI+GVELAP+kRKsP4mI9YBLKFP0u1OC+10R8Uzg7GHv5/Da1uk9wEpKuO8K3AH8A/Al4IURMQg8h/IHf//6mpdTRp6vAmbWUeke9bntRqj/YeoMxUHA5WPUDyVwXpeZu4wS9m+vffPfEfG/wLuBAzLzNxGxLbAfsG9m7gK8Dzg5Mx/koT5/4Qjr3Bs4MDN3rO/72FrnlyiHXHYBzqDMzAx/b+tRdij+sS73YuBDdcd077rY80YJ+yfW9/LziLiTEuhfAz5enz+dsgO3O7AbsAXwjoh4HPDXlB2ipwPfBp6RmWdSDp0cn5lfBo4CLqi/+9sDTwAOGKGO4f78MwCeRtmhfVZmPhm4gPI7NGR3yiGLnSg7TgeOUMdJwGn1ffwVZedb6wgDXxOujiJfDxwREfuPtfwwQ6OJW4DfZuaNHY8f1bHcp+u27qD8Ef0LSujNBRbWEflFlNHt0HT81Zm5coRtPh9YmJn/Xdf5XeBOyh/IsVxcQ3XnWvt04Ov1uR2AJwLn1Xq+V5/fDbgC2Cginh4RT6aM2BYOW/dLKSPg6+vrXwk8OTN/TdkBGvoDfgrwvDodPxT4PwB2rsfo3wN8PDP/a5T38NEaZjfWun5AGTmuqX6A2zJz8Rr65vTaN08FrgYeqOunvu4w4JCI+AfgWMoodSxXZOZd9fvrKb8TT63rXFD/vRz42Qiv3YGyQ/mlutwdlL56URfbHdrp2xk4gfJ79sXMXFGffylwTO2na4E9a12/AW4ErouIU4EbMvMrI6z/BGBpRLybMlOxFd31x59/Bpl5FWV265i6rdcMW8e/ZeYDteb/5OH/n4Z8ATgzIi6i/H69t4saNEUY+JoUmXkbZTr2AspoZ8hqylTjkA2GvbRzpLiC0T3Y8f1gXXYaJbjnDX1RpjeH/vgvH2VdQ9OXnQYpU7NdqYcujgM2BT7asd4/jlDPZ+oJfedSQu8I4NwRTvKbRpmqH3rtnjx0fP/LlGn3F1BG4YspI/P7MvOWzLyVsqNzCmX6+Tv1cMdIjq/b2LV+va/uGI1af33daP05vG/uAQ6lzBC8HSAingZcVWv7NuXQxcBo6+jQOcsy9Lu0coTXPsj/9Yh/zgCZ+RnK6P6LddZgaN0HdvTTM4Dj6s7vPpQZnGXA6REx0kmLn6fMSC2mzBZcN+w9DQDUwzyd/vwziIgDgG/Uh1+lzFp1rmOkvhv+3j5N2VH5d+CFwE8jYqMR6tUUZOBr0tQTwRZQpjSHLKUcM6Segb3PWq7+8LqObShTwwvr1wvqFD0R8RLKscrpY6xrIWWKfLv6uudTjnP+eDwF1dB/I/CmiNiNclb3fRHx+rrex1F2PoZmDs6njMgP5KEQ7fQt4LiI2KBO359NCXAoU9gHA4Mdsxwfoc6Q1HMCPgN8OzNPqOt62njeTxf1d7+izN8D7wROiojHAs8FrsnM0ygzB6+khCaUAB9PCN8EPBARL6p1Do2uh4f7zcCKiPjLutxWwKsp4TZeJ1B+R95cH3+LcghjICI2pOwQHFfPM/kZcFNmnkIJ86FDLJ3v84WUQxoX18fP4KH++PP/GcrPfDT7A1/PzE9Spuk7+3RN/lxHRPyIctjrfMoOyGbAll2sQ1OAga/J9lbKqGXIPwJzIyIpgfTdtVzvRhFxHfBN4C2Z+YvMXET5I/WvdXr6g5QT/dY4Eq2vexPleP/PKMfJX5aZfxxvUZn5A8qhhDMpsw6vAI6MiJ9SQvnEzPxhXfa3lJHcT/OhkxE7fRD4FWXqehFlRPbOjppX89BhgG9RAmjokMhnKX/sF0XEtZSZhzPG+V7+tKb6xyszL6JMd59KGdFuERE31fe2HHhURAx93O3+iPgJXYz662zEq4EPRMT1lD76LeWkzc7lVlBC8G31/XyHErKXr8V7+QMl9E+KiMdQfs83oUyV/7T++5F6SOoLwDURcQ3luPg76mq+Rjn/5A2UqfMvR8R/Ug5XfY+HDkW9lTLNfh3l+PuSUcr6FLBvXcd1lMNgT6g7i2vSWce7gZNrP14BnJSZv+quVzTZBrw9rqR+FxEfpXx2/3/qTMSNwHY1mKVW8GN5ktpgMeVkzRU89HE0w16t4ghfkqQW8Bi+JEkt0M9T+htSznZdwsgfwZEkqd9Mo1wH4j94+MeY+zrw9wDGuvylJEn9aG/KRbL+rJ8DfwnA739/D6tWrXvnKcyePYNly7q6bonWgv3bPPu4WfZvs9bV/h0cHGDzzTeBET6e2c+B/yDAqlWr18nAB9bZutcV9m/z7ONm2b/NWsf79/8cyvakPUmSWsDAlySpBQx8SZJawMCXJKkFDHxJklrAwJckqQUMfEmSWsDAlySpBQx8SZJawMCXJKkF+vnSupKkPrPJrGlsvOHGE7KtOXNmNr6Nex+4l3vumpgbuhr4kqR1xsYbbszAUQOTXUbPrD57Nfdw94Rsyyl9SZJawMCXJKkFGpvSj4gjgeM6mp4A/AvwFeA0YDpwcWbOr8vPA84BZgHfB47NzJURsQ1wIfBoIIFDMnPdu0mxJEmTqLERfmaek5nzMnMecAhwJ/Bh4DzgFcBOwB4R8eL6kguB4zJzB2AAOKq2nwWclZk7AtcAJzZVsyRJ/WqipvQ/CbwX2A74ZWbempkrKSF/YERsC0zPzKvr8ufX9vWB5wKXdLZPUM2SJPWNxs/Sj4j9KGH+xYh4HbCk4+klwNbAVqO0bwHcVXcOOtu7Nnv2jLUtfdJNxEdC2sz+bZ593Cz7tz9M1M9xIj6WdwzlmD2UGYXVHc8NAKvG0U5t79qyZctZtWr4Kqa+OXNmsnTpxHxUo43s3+bZx81qa//2405OL3+Og4MDow50G53Sj4gNgH2Ar9Wm24G5HYtsCdyxhvY7gU0jYlptn1vbJUnSODR9DH8X4BeZeU99/GMgImL7GuIHAwsyczFwf0TsVZc7tLavAK4EDqrthwELGq5ZkqS+03Tgb0cZvQOQmfcDhwOXAouAm3nohLxDgNMj4mZgBnBGbX8TcHRELAL2BuY3XLMkSX2n0WP4mfkF4AvD2hYCu46w7I3AniO0Lwb2bahESZJawSvtSZLUAga+JEktYOBLktQCBr4kSS1g4EuS1AIGviRJLWDgS5LUAga+JEktYOBLktQCBr4kSS1g4EuS1AIGviRJLWDgS5LUAga+JEktYOBLktQCBr4kSS1g4EuS1AIGviRJLWDgS5LUAga+JEktYOBLktQCBr4kSS1g4EuS1AIGviRJLWDgS5LUAga+JEktYOBLktQC6zW58oh4GfB+YBPg25n5tojYDzgNmA5cnJnz67LzgHOAWcD3gWMzc2VEbANcCDwaSOCQzFzeZN2SJPWbxkb4EbEd8CnglcAuwNMi4sXAecArgJ2APWoblFA/LjN3AAaAo2r7WcBZmbkjcA1wYlM1S5LUr5qc0n8VZQR/e2auAA4C7gV+mZm3ZuZKSsgfGBHbAtMz8+r62vNr+/rAc4FLOtsbrFmSpL7U5JT+9sCfIuJrwDbAZcDPgSUdyywBtga2GqV9C+CuunPQ2d612bNnrFXxU8GcOTMnu4S+Zv82zz5ulv3bHybq59hk4K9HGZ3vCywHvgbcB6zuWGYAWEWZaeimndretWXLlrNq1fBVTH1z5sxk6dK7J7uMvmX/Ns8+blZb+7cfd3J6+XMcHBwYdaDb5JT+b4HvZObSzLwP+DKwHzC3Y5ktgTuA20dpvxPYNCKm1fa5tV2SJI1Dk4F/GfDCiNisBvaLKcfiIyK2r20HAwsyczFwf0TsVV97aG1fAVxJOf4PcBiwoMGaJUnqS40Ffmb+GPgI8ANgEbAY+CRwOHBpbbuZh07IOwQ4PSJuBmYAZ9T2NwFHR8QiYG9gflM1S5LUrxr9HH5mnkf5GF6nhcCuIyx7I7DnCO2LKecBSJKkteSV9iRJagEDX5KkFjDwJUlqAQNfkqQWMPAlSWoBA1+SpBYw8CVJagEDX5KkFjDwJUlqAQNfkqQWMPAlSWoBA1+SpBYw8CVJagEDX5KkFjDwJUlqAQNfkqQWMPAlSWqB9Sa7AEnqFzM225Dp628wYdubM2dm49u4b8WfWP6HBxrfjppn4EtSj0xffwP2vuwdk11GT1350tNYjoHfD5zSlySpBQx8SZJawMCXJKkFDHxJklrAwJckqQW6CvyI2K7+e0BEnBgRmzZbliRJ6qUxAz8iPg2cEBE7AWcD2wHnNV2YJEnqnW5G+LsDbwReBVyQmUcA2zZalSRJ6qluLrwzmJmrImJ/4EO1beNuVh4RlwOPBlbUpmOAJwLzgfWBj2fmmXXZ/YDTgOnAxZk5v7bPA84BZgHfB47NzJXdbF+SJBXdjPD/KyIWUKbyr4iIi4Abx3pRRAwAOwC7Zua8zJwH3A78PfAcYB5wdEQ8OSKmUw4TvALYCdgjIl5cV3UhcFxm7gAMAEeN6x1KkqSuAv8I4CJgn8xcAVwJ/L8uXhf1329HxI0RcRywH/DdzPxdZt4DXAK8BtgT+GVm3lpH7xcCB0bEtsD0zLy6rut84MAu35skSaq6mdJfSAnmAYDM/FSX6968vvYtlOn7K4CLgSUdyyyhhP1WI7RvvYb2rs2ePWM8i08pE3FjjDazf5tnH/cHf47Nmqj+7Sbw3wm8HPhmRNxLCf8vZWau6UWZeRVw1dDjiDiXcoz+7zoWGwBWUWYaVo+jvWvLli1n1arVYy84xcyZM5OlS++e7DL6lv3bvDb2cb8G41T6OfZjH/eyfwcHB0Yd6I4Z+Jn5Q+CHlI/mHQR8hBLa09b0uoh4DrBhZi6sTQPAr4C5HYttCdxBObY/nnZJkjQOYwZ+RLwB2B94HrCYchz9W12sezPg5Ih4NmVK/w3A64ELI2IOcA/wauBo4KdlU7E9cCtwMHBeZi6OiPsjYq+643EosGB8b1GSJHUzpX8GsBw4BbgkM3/bzYoz87KIeAZwPWU24MzM/GFEvA+4HNgAOCczfwIQEYcDlwIbAd+kHDoAOAQ4OyJmAdfVeiRJ0jh0E/izKR+jexHwjYgYBL6TmceP9cLMPBE4cVjb54DPjbDsQmDXEdpvpJzYJ0mS1tKYH8vLzJWZeQVlxP1VyoVxDmi4LkmS1EPdHMP/LPBC4NfAl4FXZeZNTRcmSZJ6p5sp/WuB92XmbU0XI0mSmtFN4J8H/ENE7Ei5yt0pwDszc3mjlUmSpJ7p5tK6nwD+ADwGuJ9yE5t/brIoSZLUW90E/m6Z+T5gRWbeS/mY3Lxmy5IkSb3UTeA/OOzxNMZ5eVtJkjS5ugn870fEh4HpEfFC4EuUC+dIkqR1RDeBfwLlSnt/pNzL/qfAmBfdkSRJU0c3N89ZAXywfkmSpHXQqIEfEV/IzNdGxH/y8FvUApCZuzRamSRJ6pk1jfA/XP89biIKkSRJzRk18DPz2vrtscA/Z6Yn6kmStI7q5kp73wNOiYgtgHOBz3R7i1xJkjQ1dHO3vE9l5jOBlwGbAz+KiC83XpkkSeqZbj6WN2Q6sCEwwP+9GI8kSZrCurk97juAwylhfy7wzMz8n4brkiRJPdTNMfzdgbdm5hUN1yJJkhrSzTH8Q4B7I+KYiNggIp41AXVJkqQeGjPwI+Jw4DPAu4HNgK9GxFEN1yVJknqom5P23go8C7grM++kTPH/daNVSZKknurq9riZedfQg8y8DVjZXEmSJKnXugn830XEPOr19CPiEOB3jVYlSZJ6qpuz9N8GXAI8MSKWAPcBr2y0KkmS1FPd3B735ojYFdgBmFaackXjlUmSpJ5ZY+BHxP7A7zPzGuCm2rZrRJyZmc+ZiAIlSdIjN2rgR8SpwGuB6RFxNHAZcCrwJuDCiSlPkiT1wppG+K8CdgG2BM4A3k75HP6+mfnDbjdQdxy2yMzD68l/5wCzgO8Dx2bmyojYhrIT8WgggUMyc3lEbAZcBGwHLAVe6536JEkavzWdpX93Zv4hM28GngYsAnYfZ9j/BfCGjqYLgeMycwfKTXiGLuBzFnBWZu4IXAOcWNv/DrgyM3cCzgY+0e22JUnSQ9YU+Ks6vv8dJai7PlkvIh4F/D3wofp4W2B6Zl5dFzkfODAi1geeS/kkwJ/b6/cHUEb4AJ8HXlyXlyRJ49DNx/IAlmfmeC+282ngfcDj6uOtgCUdzy8Btga2oFzFb+Ww9oe9pk793wXMAe7otojZs2eMs+ypY86cmZNdQl+zf5tnH/cHf47Nmqj+XVPgbx0RZ4zwPQCZ+dbRXhgRRwK3ZebCei1+KLMJqzsWG6DMIgxvh4dmFwaGtQ/w8JmHMS1btpxVq4avfuqbM2cmS5fePdll9C37t3lt7ON+Dcap9HPsxz7uZf8ODg6MOtBdU+CfOcr33TgImBsRNwCPAmZQQn1uxzJbUkbqdwKbRsS0zHywLjM0gv9NXe72iFgPmAksG2ctkiS13qiBn5knre1KM3P/oe/rCH/fzDwiIn4WEXvVE/8OBRZk5oqIuJKyk/A54DBgQX35N+vjD9Xnr/SiP5IkjV+3x/B75RDg7IiYBVxH+bgflM/2XxAR84FfA6+r7ScC50fEz4E/1NdLkqRxajzwM/N8ypn3ZOaNwJ4jLLMY2HeE9t8BL2+0QEmSWmDUj+VFxCvrvxtOXDmSJKkJa/oc/gfrv1dNRCGSJKk5a5rSvysifgE8NiJ+OvzJzNylubIkSVIvrSnwXwTsBpwLvGViypEkSU1Y08fy7ga+HxEHUD4XvzuwPvDj+pwkSVpHrOkY/pBNgV8AHwdOAxZHxLMbrUqSJPVUN4H/Mcrtanerx+1fQwl+SZK0jugm8Gdm5uVDDzLzu8DGzZUkSZJ6rZvAX11vbQtARDweeLCxiiRJUs91c6W9k4GrI+I7lBvgvJByKVxJkrSOGHOEn5lfoVz29kfATyg3wrm04bokSVIPdXUt/cxMIBuuRZIkNaSbY/iSJGkdZ+BLktQCYwZ+RHx2IgqRJEnN6WaEPy8iBhqvRJIkNaabk/buAH4eEVcDy4caM/OtjVUlSZJ6qpvAv6p+SZKkddSYgZ+ZJ0XEdGB74OfARpl5b+OVSZKknunmpL1nALcA3wC2Am7zbnmSJK1bujlp71RgP2BZZt4OHAp8otGqJElST3UT+Btn5qKhB5n5Tbq8Qp8kSZoaugn8FRGxOeXGOURENFuSJEnqtW5G6n8HfA+YGxGfB14AHN1oVZIkqae6OUv/soi4GdgfmAacnJk3NV6ZJEnqmW6vpb8+JexX1C9JkrQO6eZjeUcAlwN7AHsDV0bEq5suTJIk9U43x/DfAeyWmUsAImIb4DLg0rFeGBEnA6+hnPB3bmaeFhH7AacB04GLM3N+XXYecA4wC/g+cGxmrqzbuxB4NJDAIZm5/P9uTZIkjaabKf0/DYU9QGb+mi6m9SNiH+D5wC7A04G3RMSuwHnAK4CdgD0i4sX1JRcCx2XmDsAAcFRtPws4KzN3BK4BTuzmjUmSpIeMOsKPiKfVb2+MiH8CPg08CBwO/HCsFWfm9yLieXWU/ti6rc2AX2bmrXUbFwIHRsQiYHpmXl1ffj5wUkScAzwXeGVH+/eAE8bzJiVJars1TekPn7I/oOP71cCYd8vLzBURcRLwLuCLlEvzLulYZAmw9RratwDuysyVw9q7Nnv2jPEsPqXMmTNzskvoa/Zv8+zj/uDPsVkT1b+jBn5mPqEXG8jM90fEh4GvAztQL+BTDQCrKIcWummntndt2bLlrFo1fBVT35w5M1m69O7JLqNv2b/Na2Mf92swTqWfYz/2cS/7d3BwYNSB7pgn7UXElpRp/Ed1tmfmu8d43Y6UO+vdkJn3RsSXKCfwPdix2JbAHcDtwNwR2u8ENo2IaZn5YF3mjrFqliRJD9fNSXtfA/akjLo7v8ayHXB2RGwYERtQTtT7NOXqvNtHxDTgYGBBZi4G7o+IveprD63tK4ArgYNq+2HAgu7emiRJGtLNx/I2yMy/HO+KM/ObEbEncD1lVH9pZv5rRCylnB+wEfBN4JL6kkMoOwizgOuAM2r7m4ALImI+8GvgdeOtRZKktusm8K+NiKdk5s/Gu/LM/ADwgWFtC4FdR1j2RspMwvD2xcC+4922JEl6SDeB/0PghohYQsfn7zNzu8aqkiRJPdVN4B9POdZ+S8O1SJKkhnQT+H/IzC80XokkSWpMN4H/3Yg4lXKi3QNDjZl5XWNVSZKknuom8A+u/3beIW815WN3ktYhszefxuB6G0/Y9ibiIimrVt7Lst8/OPaCUsuNGfi9uuKepMk3uN7GcHM3l9FYdwzuuBqYOleCk6aqbq60946R2jPztN6XI0mSmtDNlP5TO77fANgHWNhMOZIkqQndTOkf0fk4IrYCzm2sIkmS1HPdXEv/YTLzDuDxvS9FkiQ1ZbzH8AeAp1PuYidJktYR4z2Gv5pyA5vjmylHkiQ1YdzH8CVJ0rpn1MCPiM9QRvQjWZ2Z/6+ZkiRJUq+taYQ/0u1wtwD+GvhVI9VIkqRGjBr4mfmxzscRsR9wAXAR8NaG65IkST3UzVn66wGnAIcDx2bmpU0XJUmSemuNgR8RTwI+DywHdsvM2yekKkmS1FOjXngnIo4Afgx8OTP3NewlSVp3rWmEfy6wCnhPRJzQ0T5AOUt/VqOVSZKknllT4HtbXEmS+sSaztJfPJGFSJKk5oz75jmSJGndY+BLktQCBr4kSS1g4EuS1AIGviRJLTDmpXUfiYh4P/Da+vAbmfnuek3+04DpwMWZOb8uOw84B5gFfJ9yGd+VEbENcCHwaCCBQzJzeZN1S5LUbxob4ddgfwGwGzAP2D0iXgecB7wC2AnYIyJeXF9yIXBcZu5AubjPUbX9LOCszNwRuAY4samaJUnqV02O8JcA78zMPwFExE3ADsAvM/PW2nYhcGBELAKmZ+bV9bXnAydFxDnAc4FXdrR/D+i88p/6yOabrM96G280IduaM2fmhGxn5b338/t7VkzItiRpNI0Ffmb+fOj7ehOe1wL/SNkRGLIE2BrYapT2LYC7MnPlsPauzZ49Y9y1TxUTFUhTzS1z9p7sEnrqiUuvZM4E7cS0VVv/r0wU+7dZE9W/jR7DB4iInYFvAMcDKymj/CEDlOv1DwKru2intndt2bLlrFo1fBVT35w5M1m69O7JLmPC9esflqnys7R/m2X/Nq8f+7iX/Ts4ODDqQLfRs/QjYi9gIfCezLwAuB2Y27HIlsAda2i/E9g0IqbV9rm1XZIkjUOTJ+09DvgKcHBm/mtt/nF5KravIX4wsKBet//+uoMAcGhtXwFcCRxU2w8DFjRVsyRJ/arJKf13ARsBp0XEUNungMOBS+tz3wQuqc8dApwdEbOA64AzavubgAsiYj7wa+B1DdY8ptmbTWNw/Y0nZFsTNXW1asW9LPvDgxOyLUnS5GjypL23AW8b5eldR1j+RmDPEdoXA/v2tLhHYHD9jeFzA5NdRk8NHrwamDrH6CRJveeV9iRJagEDX5KkFjDwJUlqAQNfkqQWMPAlSWoBA1+SpBYw8CVJagEDX5KkFjDwJUlqAQNfkqQWMPAlSWoBA1+SpBYw8CVJagEDX5KkFjDwJUlqAQNfkqQWMPAlSWoBA1+SpBYw8CVJagEDX5KkFjDwJUlqAQNfkqQWMHpSrgUAAAoBSURBVPAlSWoBA1+SpBYw8CVJagEDX5KkFjDwJUlqgfWa3kBEzAJ+BLw0M38VEfsBpwHTgYszc35dbh5wDjAL+D5wbGaujIhtgAuBRwMJHJKZy5uuW5KkftLoCD8ingH8ANihPp4OnAe8AtgJ2CMiXlwXvxA4LjN3AAaAo2r7WcBZmbkjcA1wYpM1S5LUj5qe0j8KeDNwR328J/DLzLw1M1dSQv7AiNgWmJ6ZV9flzq/t6wPPBS7pbG+4ZkmS+k6jU/qZeSRARAw1bQUs6VhkCbD1Gtq3AO6qOwed7V2bPXvGuOtuozlzZk52CX3N/m2W/dss+7dZE9W/jR/DH2YQWN3xeABYNY52anvXli1bzqpVw1ex9vr1F3/p0rsnuwTA/m2a/dss+7d5/djHvezfwcGBUQe6E32W/u3A3I7HW1Km+0drvxPYNCKm1fa5PHR4QJIkdWmiA//HQETE9jXEDwYWZOZi4P6I2Ksud2htXwFcCRxU2w8DFkxwzZIkrfMmNPAz837gcOBSYBFwMw+dkHcIcHpE3AzMAM6o7W8Cjo6IRcDewPyJrFmSpH4wIcfwM/PxHd8vBHYdYZkbKWfxD29fDOzbYHmSJPU9r7QnSVILGPiSJLWAgS9JUgsY+JIktYCBL0lSCxj4kiS1gIEvSVILGPiSJLWAgS9JUgsY+JIktYCBL0lSCxj4kiS1gIEvSVILGPiSJLWAgS9JUgsY+JIktYCBL0lSCxj4kiS1gIEvSVILGPiSJLWAgS9JUgsY+JIktYCBL0lSCxj4kiS1gIEvSVILGPiSJLWAgS9JUgusN9kFdCMiDgbmA+sDH8/MMye5JEmS1ilTfoQfEY8F/h54DjAPODoinjy5VUmStG5ZF0b4+wHfzczfAUTEJcBrgJPHeN00gMHBgd5XtMm2vV/nJGukn9bSeo/bcrJL6Lmp1L+s7+9vk7acvvlkl9BzU6l/Abad3V+/w73s3451TRv+3MDq1at7tqEmRMTfAJtk5vz6+Ehgz8w8eoyXPge4sun6JEmagvYGftDZsC6M8AeBzr2SAWBVF6/7D8obXgI82EBdkiRNNdOAuZQMfJh1IfBvpwT3kC2BO7p43QMM27uRJKkFbhmpcV0I/O8AH4iIOcA9wKuBsabzJUlShyl/ln5m/gZ4H3A5cAPwucz8yeRWJUnSumXKn7QnSZIeuSk/wpckSY+cgS9JUgsY+JIktYCBL0lSCxj4kiS1wLrwOfzWiYhZwI+Al2bmrya5nL4TEe8HXlsffiMz3z2Z9fSbiDiZcr+L1cC5mXnaJJfUlyLiVGCLzDx8smvpJxFxOfBoYEVtOiYzfzyJJfWMgT/FRMQzgLOBHSa7ln4UEfsBLwB2owTSv0XEqzLzy5NbWX+IiH2A5wO7UG5nvSgivpGZObmV9ZeI+AvgDcA3JruWfhIRA5S/vdtm5srJrqfXnNKfeo4C3kx3lw/W+C0B3pmZf8rMFcBNwDaTXFPfyMzvAc+rfywfTRlU3DO5VfWXiHgU5ZbhH5rsWvpQ1H+/HRE3RsRxk1pNjznCn2Iy80iAiBhrUa2FzPz50PcR8STK1P5ek1dR/8nMFRFxEvAu4IvAbya5pH7zacrVRx832YX0oc2BhcBbKDNUV0REZua/T25ZveEIX60UETsD/w4cn5m/nOx6+k1mvh+YQwmloya5nL5Rbw9+W2YunOxa+lFmXpWZh2XmHzPzf4FzgZdMdl29YuCrdSJiL8pe/Hsy84LJrqefRMSOETEPIDPvBb5EOZ6v3jgIeEFE3ACcDLw8Ik6f5Jr6RkQ8p54fMWSAh07eW+c5pa9WiYjHAV8BDsrM7052PX1oO+CkiHgO5aTIVwDnTW5J/SMz9x/6PiIOB/bNzLdPXkV9ZzPg5Ih4NmVK/w3AsZNbUu84wlfbvAvYCDgtIm6oX33zH3qyZeY3KWeOXw9cC/woM/91cquSupOZl/Hw39/zMvOqya2qd7xbniRJLeAIX5KkFjDwJUlqAQNfkqQWMPAlSWoBA1+SpBbwc/hSn4iIZwKnALMpO/O3Ae8aupxwRHwbOLheQayJ7c8D3paZR0TE+cD+wNL69CAwA/hUZn5kjPU8ATg1M18dEVsBl2Tms9eyppnAF4C/zMz71mYdUr9whC/1gYjYELiMcmOgXTLzKcBFwIKImFYX23/UFTzy7Q9SLkM6v6P59MycV792AfYF3h8RO46xum2pNzHJzDvWNuzr6+8GPg98cG3XIfULR/hSf9iYcpWwGR1tFwF3AdMi4pzadnlEvATYFXgvsAHlrnYXZOaJEbEv8AnKHe5mAHsD5wBPAlZRLkZyTGauGrb91wK3ZuaabpSzNeVSpXcDRMR7KVfimw5sQrko0tfq9h4bEd8CjgF+lpkzIuIDwOOBuZSdgt8Ar8/MJRGxB/DJ+n5uqc+/IzOvoIzwPxwRH83M/1lzN0r9yxG+1Acy8/fAu4F/i4j/joh/AY4AvlNvBXxEXfR5wO3AO4E3ZObTgWcCfxMRW9RlngK8ro7KXw7MzMx5wB71+e1GKOE1lBmGTm+vVzL874j431rfAZn5m4jYFtiPcmnYXSh3fzs5Mx8EjgRuycwXjrCdvYEDM3NHyk7JsRGxHuWa/SfWdZ0BzOvom/uB/6CPboIirQ0DX+oTmXka8BjgrcAS4ATg+ojYdNhyq4GXAbtHxPuB0ygj703qIrdl5uL6/Q+AnSPiCuA9wMcz879G2PyOwPD20+uOwlOBq4EHgCtqDYuBw4BDIuIfKNcrn8HYrsjMu+r31wOPqusnMxfUfy8Hfjbsdbfy0L3OpVYy8KU+EBF7RcTxmXl3Zl6Wme8GdqbcwGb/YctuQgnLpwHXAcdT7gg2UBdZPrRsZt4KbE85GXAW8J2IeNkIJaxmlL8nmXkPcCiwD/D2WsPTgKvqOr8NfLhj+2vSeeLd6vqalSO89sFhj1eM0Ca1ioEv9YelwPx6l7ohc4FNgf+sjx+k3AHsSZSgnZ+ZX6ecTLchMI1hIuKNwGeAb2fmCcC3KDsKwyXwxNGKq4cc3km5k95jgecC19RZie8Br+zY/spaZ7duAh6IiBfVmvekjPo7bxTyBODmcaxT6jsGvtQHMvMXlND8UD1mvohystoRmZl1sS9SwnUV5Xj7zRFxE2V6fxFlJD/cZylBvCgirqXsQJwxwnKXAC8ao8aLKCf9nUo5c36Luv1FlFmFR9WP0S0C7o+In9DFqD8zVwKvBj4QEddTdix+C9wLEBEbUM5T+PpY65L6mXfLk/SI1Y/+XUs9KW8Stv9Rymf3/yciHgfcCGyXmX+o943fOTOPn+i6pKnEEb6kR6yeXX8U8KFJKmExsLCO8L8OHFnDfgZwMPCBSapLmjIc4UuS1AKO8CVJagEDX5KkFjDwJUlqAQNfkqQWMPAlSWqB/w8Axh4MLk4yKAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot how many reviews we have of each star\n",
    "star_x = Restaurant_AZ_reviews.review_stars.value_counts().index\n",
    "star_y = Restaurant_AZ_reviews.review_stars.value_counts().values\n",
    "\n",
    "plot.figure(figsize=(8,5))\n",
    "# colors are in the order 5, 4, 3, 1, 2\n",
    "bar_colors = ['darkgreen', 'mediumseagreen', 'gold', 'crimson', 'orange']\n",
    "plot.bar(star_x, star_y, color=bar_colors, width=.6)\n",
    "plot.xlabel('Stars (Rating)')\n",
    "plot.ylabel('Number of Reviews')\n",
    "plot.title(f'Number of Reviews Per Rating of {businesses_to_analyse}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Until now, we have gathered the necessary data. From now on, we start pre-processing the data for our later NLP tasks "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization and Parsing\n",
    "\n",
    "Read preprocessed restaurant and Review files.\n",
    "For testing, we read only the restaurants and reviews in **Illinois**\n",
    "\n",
    "We can also limit the data by selecting a sample of restaurants. But here we left this value high to get all Illinois restaurants and their reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for now we restrich Restaurants to this number to develop the code\n",
    "sample_restaurants_to_load = 100000\n",
    "\n",
    "# Only Arizona Businesses, Change if needed\n",
    "#restaurant_file='processed_data/restaurants.csv'\n",
    "#reviews_file   ='processed_data/restaurant_reviews.csv'\n",
    "restaurant_file='processed_data/'+state_filter+'_restaurants.csv'\n",
    "reviews_file   ='processed_data/'+state_filter+'_restaurant_reviews.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# SPACY\n",
    "# This is the large Spacy English Library\n",
    "#nlp  = spacy.load('en_core_web_lg')\n",
    "#nlp2 = spacy.load('en_core_web_lg', disable=[\"ner\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *All stopwords in restaurant reviews*\n",
    "\n",
    "These stopwords are manually prepared by us for this project. It includes some extra words such as 'restaurant', which we want to avoid for out Topic Modelling and Sentiment analysis. It also excludes words such as 'not' to do a better sentiment analysis  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stopwords for topic mining\n",
    "stopwords = [line.rstrip('\\n') for line in open('config/stopwords.txt', 'r', encoding='utf-8')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *All negations in restaurant reviews to be merged as phrases*\n",
    "\n",
    "These include words such as not, niether, rarely etc. which change the sense of sentiment and should be considered in phrases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "negations = [line.rstrip('\\n') for line in open('config/negations.txt', 'r', encoding='utf-8')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *All stopwords in restaurant names*\n",
    "\n",
    "If we do the topic modelling on reviews for a restaurant, the name of the restaurant may appear as frequet topic. This is not ideal, so we will remove it from the review. But some words such as chicken can be in the name and should be retained in the review text. These words are collected in another list and are loaded here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The words that appear in names of the Restaurants\n",
    "# Restaurants name may appear multiple time in review, increasing its word frequenty\n",
    "# For topic mining per restaurant, it is not useful and should be removed\n",
    "# However words such as 'chicken' when come in restaurant name should be retained\n",
    "stopnames = [line.rstrip('\\n').lower() for line in open('config/names.txt', 'r')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Utility functions\n",
    "\n",
    "- To get difference between 2 lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Difference between 2 lists: list1 - List2\n",
    "# \n",
    "def list_diff(list1,list2):\n",
    "    return list(itertools.filterfalse(set(list2).__contains__, list1)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To remove stopwords from a list of strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Remove Stop Words\n",
    "# \n",
    "def remove_stop_words(data):\n",
    "    return [list_diff(sent,stopwords) for sent in data]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To remove a list of words from a list of strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Remove List Words\n",
    "# \n",
    "def remove_list_words(data, alist):\n",
    "    return [list_diff(sent, alist) for sent in data]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To remove URLs from a list of strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Remove URLs from list of data\n",
    "#    \n",
    "def remove_urls (data):\n",
    "    start = time.time()\n",
    "    data = [re.sub(r'(https|http)?:\\/\\/(\\w|\\.|\\/|\\?|\\=|\\&|\\%)*\\b', ' ', str(sent).lower() , flags=re.MULTILINE) for sent in data]\n",
    "    return(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To lowercase text data, remove extra whitespaces & newlines and remove special characters (except period which marks the end of sentence). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Remove new lies and symbols & lowercase from list of data\n",
    "#\n",
    "def remove_newlines(data):\n",
    "    start = time.time()\n",
    "    data = [str(sent).replace('\\\\n',' ').replace('\\n',' ').replace('.',' . ').replace(',',' , ').replace('?',' . ').replace('!',' . ') for sent in data]\n",
    "    data = [str(sent).replace(';',' . ').replace('\\r',' ').replace(':',' . ').replace('/',' / ').replace('\"','').replace('$',' dollars ') for sent in data]\n",
    "    data = [str(sent).replace('~','').replace('(','').replace(')','').replace('+','').replace('#','').replace('-','_').replace('%',' dollars ') for sent in data]\n",
    "    data = [str(sent).strip('*').strip('-').replace('=',' ').replace('@',' ').replace('^',' ') for sent in data]\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To remove extra spaces in text data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Remove spaces and symbols from list of data\n",
    "#\n",
    "def remove_spaces (data):\n",
    "    start = time.time()\n",
    "    data = [re.sub('\\s+', ' '  ,  str(sent)) for sent in data]\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To change short form of not (n't) to long form (not). Remember not is important for sentiment analysis since it changes the sentiment sense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Convert n't to not in list of data\n",
    "#\n",
    "def remove_short_nots (data):\n",
    "    start = time.time()\n",
    "    data = [re.sub(\"n't\", ' not', str(sent)) for sent in data]\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define our tokenization function. It splits on space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# tokenize within list of data\n",
    "#\n",
    "def split_on_space (data):\n",
    "    start = time.time()\n",
    "    data = [sent.split() for sent in data]\n",
    "    #data = list(tokenize_docs(data))\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Cleansing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the data from files "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 9.97 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Read Businesses\n",
    "all_restaurants = pd.read_csv(restaurant_file).drop(labels='Unnamed: 0', axis=1).head(sample_restaurants_to_load)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 201 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Read all reviews\n",
    "all_reviews = pd.read_csv(reviews_file).drop(labels='Unnamed: 0', axis=1).drop(labels='city', axis=1).drop(labels='state', axis=1).drop(labels='categories', axis=1).drop(labels='user_id', axis=1).drop(labels='date', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 4 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Retain reviews of selected Businesses\n",
    "all_reviews = all_reviews[all_reviews.business_id.isin(all_restaurants.business_id)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>name</th>\n",
       "      <th>review_count</th>\n",
       "      <th>business_stars</th>\n",
       "      <th>review_id</th>\n",
       "      <th>review_stars</th>\n",
       "      <th>text</th>\n",
       "      <th>useful</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mofOjB6flg-eAWOFbOkHfQ</td>\n",
       "      <td>ChinaTown Buffet</td>\n",
       "      <td>72</td>\n",
       "      <td>2.5</td>\n",
       "      <td>8MTptiOpUeuPUFZgtfk9Vw</td>\n",
       "      <td>1.0</td>\n",
       "      <td>I would have given this restaurant zero stars ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mofOjB6flg-eAWOFbOkHfQ</td>\n",
       "      <td>ChinaTown Buffet</td>\n",
       "      <td>72</td>\n",
       "      <td>2.5</td>\n",
       "      <td>ofpfzn8LV4nJ2pE6IlTVdg</td>\n",
       "      <td>1.0</td>\n",
       "      <td>So...no. Just...no. Before I picked up a plate...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mofOjB6flg-eAWOFbOkHfQ</td>\n",
       "      <td>ChinaTown Buffet</td>\n",
       "      <td>72</td>\n",
       "      <td>2.5</td>\n",
       "      <td>O3yApCw95tbA0kodflQrpA</td>\n",
       "      <td>5.0</td>\n",
       "      <td>The best food I want to the a lot of the buffe...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mofOjB6flg-eAWOFbOkHfQ</td>\n",
       "      <td>ChinaTown Buffet</td>\n",
       "      <td>72</td>\n",
       "      <td>2.5</td>\n",
       "      <td>fFTVZE7EENdl66SX9cgc0A</td>\n",
       "      <td>1.0</td>\n",
       "      <td>It's impressive how filthy this place is. Some...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mofOjB6flg-eAWOFbOkHfQ</td>\n",
       "      <td>ChinaTown Buffet</td>\n",
       "      <td>72</td>\n",
       "      <td>2.5</td>\n",
       "      <td>pDlIw6DLLjZ5G1vJnMzr9A</td>\n",
       "      <td>5.0</td>\n",
       "      <td>I was visiting the Champaign area with my fami...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              business_id              name  review_count  business_stars  \\\n",
       "0  mofOjB6flg-eAWOFbOkHfQ  ChinaTown Buffet            72             2.5   \n",
       "1  mofOjB6flg-eAWOFbOkHfQ  ChinaTown Buffet            72             2.5   \n",
       "2  mofOjB6flg-eAWOFbOkHfQ  ChinaTown Buffet            72             2.5   \n",
       "3  mofOjB6flg-eAWOFbOkHfQ  ChinaTown Buffet            72             2.5   \n",
       "4  mofOjB6flg-eAWOFbOkHfQ  ChinaTown Buffet            72             2.5   \n",
       "\n",
       "                review_id  review_stars  \\\n",
       "0  8MTptiOpUeuPUFZgtfk9Vw           1.0   \n",
       "1  ofpfzn8LV4nJ2pE6IlTVdg           1.0   \n",
       "2  O3yApCw95tbA0kodflQrpA           5.0   \n",
       "3  fFTVZE7EENdl66SX9cgc0A           1.0   \n",
       "4  pDlIw6DLLjZ5G1vJnMzr9A           5.0   \n",
       "\n",
       "                                                text  useful  \n",
       "0  I would have given this restaurant zero stars ...       1  \n",
       "1  So...no. Just...no. Before I picked up a plate...       1  \n",
       "2  The best food I want to the a lot of the buffe...       0  \n",
       "3  It's impressive how filthy this place is. Some...       0  \n",
       "4  I was visiting the Champaign area with my fami...       0  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "# Top 5 Reviews\n",
    "all_reviews.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_docs(sentences):\n",
    "    for sentence in sentences:\n",
    "        yield(gensim.utils.simple_preprocess(str(sentence), deacc=True))  # deacc=True re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Remove new line and spaces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# String List cleaning, removes spaces, new lines\n",
    "def clean_string(data):\n",
    "    start = time.time()\n",
    "    data = remove_urls(data)\n",
    "    print(f\"URLs removed         - took {time.time() - start:9.4f} secs\")\n",
    "    \n",
    "    start = time.time()\n",
    "    data = remove_newlines(data)\n",
    "    print(f\"Removed line breaks  - took {time.time() - start:9.4f} secs\")\n",
    "    \n",
    "    start = time.time()\n",
    "    data = remove_spaces(data)\n",
    "    print(f\"Removed extra spaces - took {time.time() - start:9.4f} secs\")\n",
    "    \n",
    "    start = time.time()\n",
    "    data = remove_short_nots(data)\n",
    "    print(f\"Removed short Nots   - took {time.time() - start:9.4f} secs\")\n",
    "    \n",
    "    start = time.time()\n",
    "    data = split_on_space(data)\n",
    "    #data = list(tokenize_docs(data))\n",
    "    print(f\"Tokenized            - took {time.time() - start:9.4f} secs\")\n",
    "    \n",
    "    start = time.time()\n",
    "    data = remove_stop_words(data)\n",
    "    print(f\"Stopwords removed    - took {time.time() - start:9.4f} secs\")\n",
    "    \n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Remove stopwords from restaurant names\n",
    "Ee need to remove restaurant names from reviews, otherwise these may potentially become topics (most frequent *nouns*). But restaurant names can have other words, such as chinese, grill etc. which should not be removed from reviews\n",
    "In below function, we cleanse restautant name so that only valid parts should be removed. This consists of proper nouns which are not in stopwords for restaurant names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_name(name):\n",
    "    name_toks = []\n",
    "    \n",
    "    # Nlp doc from Name\n",
    "    name_doc = nlp2(name)\n",
    "    for token in name_doc:\n",
    "        \n",
    "        # Retain Proper nouns in Name\n",
    "        if token.pos_ == 'PROPN' or token.like_num:\n",
    "        \n",
    "            # Lose stop words in Name\n",
    "            if token.text.lower() not in stopnames:\n",
    "            \n",
    "                # All Restaurant name tokens to be remoed from reviews of this reataurant\n",
    "                name_toks.append(token.text.lower())\n",
    "    \n",
    "    #for noun_phrase in list(name_doc.noun_chunks):\n",
    "        #if(len(str(noun_phrase).split())<2):\n",
    "            #noun_phrase.merge(noun_phrase.root.tag_, noun_phrase.root.lemma_, noun_phrase.root.ent_type_)\n",
    "    \n",
    "    \n",
    "    for chunk in name_doc.ents:\n",
    "        name_toks.append(chunk.text.lower())\n",
    "    \n",
    "    return name_toks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View a sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>categories</th>\n",
       "      <th>city</th>\n",
       "      <th>name</th>\n",
       "      <th>review_count</th>\n",
       "      <th>stars</th>\n",
       "      <th>state</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mofOjB6flg-eAWOFbOkHfQ</td>\n",
       "      <td>Sushi Bars, Buffets, Restaurants, Chinese</td>\n",
       "      <td>Champaign</td>\n",
       "      <td>ChinaTown Buffet</td>\n",
       "      <td>72</td>\n",
       "      <td>2.5</td>\n",
       "      <td>IL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>o9nGD3J0btXjE9M6IdgZTA</td>\n",
       "      <td>Salad, Delis, Restaurants, Sandwiches, Food</td>\n",
       "      <td>Champaign</td>\n",
       "      <td>McAlister's Deli</td>\n",
       "      <td>50</td>\n",
       "      <td>3.0</td>\n",
       "      <td>IL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>y2V42mxWvVoH4NvUT_n8eg</td>\n",
       "      <td>American (Traditional), Restaurants, African, ...</td>\n",
       "      <td>Champaign</td>\n",
       "      <td>Red Robin Gourmet Burgers</td>\n",
       "      <td>67</td>\n",
       "      <td>2.5</td>\n",
       "      <td>IL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>o5Mofj5KJkYAMs_fhxftpg</td>\n",
       "      <td>Soul Food, Mexican, Asian Fusion, Korean, Food...</td>\n",
       "      <td>Champaign</td>\n",
       "      <td>Seoul Taco</td>\n",
       "      <td>70</td>\n",
       "      <td>3.5</td>\n",
       "      <td>IL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>vqEtZ7CtNB9G8WMzjwJA2g</td>\n",
       "      <td>Mexican, Restaurants, Nightlife, Bars</td>\n",
       "      <td>Champaign</td>\n",
       "      <td>Wedge Tequila Bar &amp; Grill</td>\n",
       "      <td>125</td>\n",
       "      <td>3.5</td>\n",
       "      <td>IL</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              business_id                                         categories  \\\n",
       "0  mofOjB6flg-eAWOFbOkHfQ          Sushi Bars, Buffets, Restaurants, Chinese   \n",
       "1  o9nGD3J0btXjE9M6IdgZTA        Salad, Delis, Restaurants, Sandwiches, Food   \n",
       "2  y2V42mxWvVoH4NvUT_n8eg  American (Traditional), Restaurants, African, ...   \n",
       "3  o5Mofj5KJkYAMs_fhxftpg  Soul Food, Mexican, Asian Fusion, Korean, Food...   \n",
       "4  vqEtZ7CtNB9G8WMzjwJA2g              Mexican, Restaurants, Nightlife, Bars   \n",
       "\n",
       "        city                       name  review_count  stars state  \n",
       "0  Champaign           ChinaTown Buffet            72    2.5    IL  \n",
       "1  Champaign           McAlister's Deli            50    3.0    IL  \n",
       "2  Champaign  Red Robin Gourmet Burgers            67    2.5    IL  \n",
       "3  Champaign                 Seoul Taco            70    3.5    IL  \n",
       "4  Champaign  Wedge Tequila Bar & Grill           125    3.5    IL  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_restaurants.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "convert to list of reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = (all_reviews['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"I would have given this restaurant zero stars if the app allowed it. This is a disgusting restaurant. You can find roaches lurching around the food. The 'clean' dishes next to buffet items are mostly dirty. Just look at the bottom of the plate. This restaurant should really be inspected by authorities.\""
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URLs removed         - took    0.4758 secs\n",
      "Removed line breaks  - took    0.1496 secs\n",
      "Removed extra spaces - took    0.6483 secs\n",
      "Removed short Nots   - took    0.0328 secs\n",
      "Tokenized            - took    0.1397 secs\n",
      "Stopwords removed    - took    0.5526 secs\n",
      "Wall time: 2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = clean_string(data)\n",
    "\n",
    "#  Stats\n",
    "## URLs removed         - took   120.258 secs\n",
    "## Removed line breaks  - took   42.2213 secs\n",
    "## Removed extra spaces - took   164.281 secs\n",
    "## Removed short Nots   - took   8.88918 secs\n",
    "## Tokenized            - took   67.6205 secs\n",
    "## Stopwords removed    - took   174.829 secs\n",
    "## Wall time: 9min 58s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['given',\n",
       " 'zero',\n",
       " 'stars',\n",
       " 'allowed',\n",
       " '.',\n",
       " 'disgusting',\n",
       " '.',\n",
       " 'find',\n",
       " 'roaches',\n",
       " 'lurching',\n",
       " 'food',\n",
       " '.',\n",
       " \"'clean'\",\n",
       " 'dishes',\n",
       " 'buffet',\n",
       " 'items',\n",
       " 'dirty',\n",
       " '.',\n",
       " 'look',\n",
       " 'plate',\n",
       " '.',\n",
       " 'inspected',\n",
       " 'authorities',\n",
       " '.']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phrase creation\n",
    "\n",
    "We create bi-grams and tri-grams for frequent sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 5.93 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "bigram  = gensim.models.Phrases(data, min_count=4, threshold=50) # higher threshold fewer phrases.\n",
    "trigram = gensim.models.Phrases(bigram[data],min_count=3, threshold=100)  \n",
    "\n",
    "#  Stats\n",
    "## Wall time: 18min 9s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 7.23 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "bigram_mod  = gensim.models.phrases.Phraser(bigram)\n",
    "trigram_mod = gensim.models.phrases.Phraser(trigram)\n",
    "\n",
    "#  Stats\n",
    "## Wall time: 5min 43s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 5.85 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "bigrams  = [bigram_mod[doc] for doc in data]\n",
    "trigrams = [trigram_mod[bigram_mod[doc]] for doc in data]\n",
    "\n",
    "#  Stats\n",
    "## Wall time: 21min 44s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%time\n",
    "#with open (\"processed_data/vocab.csv\",\"w\",encoding='utf-8')as vocab:\n",
    "#    vocab.write('\\n'.join(list(sorted(set(reduce(operator.concat, trigrams))))))\n",
    "#    \n",
    "#  Stats\n",
    "## Several Hours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 39.9 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "all_reviews['topic_text'] = [\" \".join(trigram).replace(\" .\",\".\\n\") for trigram in trigrams]\n",
    "\n",
    "#  Stats\n",
    "## Wall time: 37.4 s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Checkpoint\n",
    "#all_reviews.to_csv('processed_data/trigram_reviews.csv',encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "#all_reviews     = pd.read_csv('processed_data/trigram_reviews.csv').drop(labels='Unnamed: 0', axis=1)\n",
    "#all_restaurants = pd.read_csv('processed_data/restaurants.csv').drop(labels='Unnamed: 0', axis=1).head(sample_restaurants_to_load)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Remove stopwords restaurant reviews\n",
    "In below function, we cleanse restaurant reviews for **Topic Modelling**. We revove all stop words, keep only nouns, verbs, adjectives and adverbs, and remove restaurant references in reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_doc(doc):\n",
    "    \n",
    "    # Remove punctuation, symbols (#) and stopwords\n",
    "    topic_allowed_postags=['PROPN', 'NOUN', 'VERB']\n",
    "    sent_allowed_postags=['PROPN', 'NOUN', 'ADJ', 'VERB', 'ADV', 'DET', 'ADP']\n",
    "    \n",
    "    topics = []\n",
    "    sentis = []\n",
    "    \n",
    "    skip = False\n",
    "    for sent in doc.sents:\n",
    "        sent_words = []\n",
    "        for i,token in enumerate(sent):\n",
    "            if skip:\n",
    "                skip = False\n",
    "            else:\n",
    "                lemma = token.lemma_.strip().replace('_',' ')\n",
    "                word  = token.text.replace('_',' ')\n",
    "                pos   = token.pos_\n",
    "                if pos in topic_allowed_postags:\n",
    "                    topics.append(lemma.replace(\" \",\"_\"))\n",
    "                if pos in sent_allowed_postags:\n",
    "                    if i+1<len(sent) and pos in ['ADJ', 'ADV'] and sent[i+1].pos_ in ['NOUN', 'VERB']:\n",
    "                        sent_words.append(lemma+\"_\"+sent[i+1].lemma_)\n",
    "                        skip = True\n",
    "                    elif i+1<len(sent) and lemma in negations and sent[i+1].pos_ in ['ADJ', 'ADV'] and sent[i+1].lemma_ not in negations:\n",
    "                        sent_words.append(lemma+\"_\"+sent[i+1].lemma_)\n",
    "                        skip = True\n",
    "                    elif len(lemma.replace(\".\",\"\"))>1:\n",
    "                        sent_words.append(lemma.replace(\" \",\"_\"))\n",
    "        if len(sent_words)>0:\n",
    "            sent_words.append('.')\n",
    "        sentis.append(\" \".join(sent_words).replace(\" .\",\".\"))\n",
    "            \n",
    "    topic_text = str(\" \".join(topics).replace(\"\\n\",\" \").replace(\"..\",\".\"))\n",
    "    \n",
    "    sentiment_text = str(\" \".join((value for value in sentis if value != '.'))).replace(\"\\n\",\" \").replace(\"..\",\".\")\n",
    "    \n",
    "    return [topic_text, sentiment_text]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this function, we lemmatize the reviews and remove stop words and restaurant names from reviews. Finally we add a period at the end of each sentence to delimit the sentences from topic and sentiment mining"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_doc(doc, name_toks):\n",
    "    \n",
    "    sents  = []\n",
    "    \n",
    "    for sent in doc.sents:\n",
    "        sent_words = []\n",
    "        for i,token in enumerate(sent) :\n",
    "            if token.lemma_ in list_diff([token.lemma_.lower() for token in sent],stopwords+name_toks) and token.lemma_ != \"PUNCT\":\n",
    "                sent_words.append(str(token.lemma_))\n",
    "        if len(sent_words)>0:\n",
    "            sent_words.append('.')\n",
    "        sents.append(\" \".join(sent_words))\n",
    "    \n",
    "    new_doc = str(\" \".join(sents).replace(\"  \",\" \").replace(\" .\",\"\").replace(\" .\",\".\").replace(\" .\",\".\").replace(\"..\",\".\"))\n",
    "    \n",
    "    return split_doc(nlp2(new_doc))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets call the above function of the cleaned review data list and process all reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cleaning reviews [   11/  157 ] -    11.825 secs\n",
      "Cleaning reviews [   21/  157 ] -    14.656 secs\n",
      "Cleaning reviews [   31/  157 ] -    16.951 secs\n",
      "Cleaning reviews [   41/  157 ] -    11.582 secs\n",
      "Cleaning reviews [   51/  157 ] -    17.158 secs\n",
      "Cleaning reviews [   61/  157 ] -    18.175 secs\n",
      "Cleaning reviews [   71/  157 ] -    25.335 secs\n",
      "Cleaning reviews [   81/  157 ] -    19.513 secs\n",
      "Cleaning reviews [   91/  157 ] -    35.970 secs\n",
      "Cleaning reviews [  101/  157 ] -    17.006 secs\n",
      "Cleaning reviews [  111/  157 ] -    25.744 secs\n",
      "Cleaning reviews [  121/  157 ] -    20.446 secs\n",
      "Cleaning reviews [  131/  157 ] -    28.275 secs\n",
      "Cleaning reviews [  141/  157 ] -    16.203 secs\n",
      "Cleaning reviews [  151/  157 ] -    17.959 secs\n",
      "Cleand [  157 ] reviews successfully\n",
      "Wall time: 5min 10s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "total = len(all_restaurants)\n",
    "cleansed_text = []\n",
    "start = time.time()\n",
    "for index, restaurant in all_restaurants.iterrows():\n",
    "    #print(f'Cleaning reviews for restaurant: \"{restaurant[\"name\"]:<{40}}\" [{index+1:>{5}}/{total:>{5}}]')\n",
    "    if index % 10 == 0 and index > 0:\n",
    "        end = time.time()\n",
    "        print(f'Cleaning reviews [{index+1:>{5}}/{total:>{5}} ] - {str(end-start):>{9.6}} secs')\n",
    "        #with open (\"processed_data/data/cleansed_text_\"+str(index)+\".csv\",\"w\",encoding='utf-8')as f:\n",
    "        #   f.write('\\n'.join([sublist[0].replace('\\n',' ')+\",\"+sublist[1].replace('\\n',' ')for sublist in cleansed_text]))\n",
    "        #cleansed_text = []\n",
    "        start = time.time()\n",
    "    \n",
    "    # Convert to list\n",
    "    \n",
    "    data = all_reviews.query(' business_id == \"'+restaurant['business_id']+'\" ')['topic_text']\n",
    "    data = [u''+str(txt) for txt in data]\n",
    "    \n",
    "    # iterate list, clean sentences\n",
    "    for parsed_review in nlp2.pipe(iter(data), batch_size=5000, n_threads=20):\n",
    "        #[noun_phrase.merge(noun_phrase.root.tag_, noun_phrase.root.lemma_, noun_phrase.root.ent_type_) for noun_phrase in parsed_review.noun_chunks if len(str(noun_phrase).split())>1 and len(str(noun_phrase).split())<4]\n",
    "        cleansed_text.append(clean_doc(parsed_review,clean_name(restaurant[\"name\"])))\n",
    "        #pprint(parsed_review)\n",
    "\n",
    "print(f'Cleand [{total:>{5}} ] reviews successfully')\n",
    "#\"\"\"\n",
    "#Cleaning reviews [    1/17680 ] -    0.0060 secs\n",
    "#Cleaning reviews [  501/17680 ] -    1363.8 secs\n",
    "#Cleaning reviews [ 1001/17680 ] -    1490.7 secs\n",
    "#Cleaning reviews [ 1501/17680 ] -    1380.7 secs\n",
    "#Cleaning reviews [ 2001/17680 ] -    1484.2 secs\n",
    "#Cleaning reviews [ 2501/17680 ] -    1319.1 secs\n",
    "#Cleaning reviews [ 3001/17680 ] -    1504.2 secs\n",
    "#Cleaning reviews [ 3501/17680 ] -    1454.6 secs\n",
    "#Cleaning reviews [ 4001/17680 ] -    1410.8 secs\n",
    "#Cleaning reviews [ 4501/17680 ] -    1407.9 secs\n",
    "#Cleaning reviews [ 5001/17680 ] -    1453.4 secs\n",
    "#Cleaning reviews [ 5501/17680 ] -    1414.1 secs\n",
    "#\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we write the cleansed data to CSV files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "print(\"Done\")\n",
    "\n",
    "all_reviews['topic_text']     = [el[0] for el in cleansed_text]\n",
    "all_reviews['sentiment_text'] = [el[1] for el in cleansed_text]\n",
    "\n",
    "all_reviews.to_csv('processed_data/'+state_filter+'_cleaned_reviews.csv',encoding='utf-8')\n",
    "all_restaurants.to_csv('processed_data/'+state_filter+'_cleaned_restaurants.csv',encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "#sorted(set(reduce(operator.concat, all_reviews['topic_text']).split()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets have a look at the data after cleaning.\n",
    "For the `review text` of first review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>name</th>\n",
       "      <th>review_count</th>\n",
       "      <th>business_stars</th>\n",
       "      <th>review_id</th>\n",
       "      <th>review_stars</th>\n",
       "      <th>text</th>\n",
       "      <th>useful</th>\n",
       "      <th>topic_text</th>\n",
       "      <th>sentiment_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mofOjB6flg-eAWOFbOkHfQ</td>\n",
       "      <td>ChinaTown Buffet</td>\n",
       "      <td>72</td>\n",
       "      <td>2.5</td>\n",
       "      <td>8MTptiOpUeuPUFZgtfk9Vw</td>\n",
       "      <td>1.0</td>\n",
       "      <td>I would have given this restaurant zero stars ...</td>\n",
       "      <td>1</td>\n",
       "      <td>zero_star allow find roach lurch food buffet i...</td>\n",
       "      <td>zero_star allow disgusting find roach lurch fo...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              business_id              name  review_count  business_stars  \\\n",
       "0  mofOjB6flg-eAWOFbOkHfQ  ChinaTown Buffet            72             2.5   \n",
       "\n",
       "                review_id  review_stars  \\\n",
       "0  8MTptiOpUeuPUFZgtfk9Vw           1.0   \n",
       "\n",
       "                                                text  useful  \\\n",
       "0  I would have given this restaurant zero stars ...       1   \n",
       "\n",
       "                                          topic_text  \\\n",
       "0  zero_star allow find roach lurch food buffet i...   \n",
       "\n",
       "                                      sentiment_text  \n",
       "0  zero_star allow disgusting find roach lurch fo...  "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_reviews.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the cleaned topic text looks like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'zero_star allow find roach lurch food buffet item look plate inspect authority'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_reviews['topic_text'].head(1).iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and the cleaned sentiment text looks like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'zero_star allow disgusting find roach lurch food clean_dish buffet item. dirty. look. plate inspect authority.'"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_reviews['sentiment_text'].head(1).iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally We also save the vocabulary in a CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 8.13 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "with open ('processed_data/'+state_filter+'_Topicp_Vocab.csv',\"w\",encoding='utf-8')as vocab:\n",
    "    vocab.write('\\n'.join(list(sorted(set(reduce(operator.concat, all_reviews['topic_text']).split())))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "#END"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topic Mining\n",
    "\n",
    "In the following we do the topic mining. We take the cleansed text and try to find \n",
    "- The overall theme in the review\n",
    "- The positive topic if any\n",
    "- The negative topic if any\n",
    "\n",
    "Lets read the cleansed files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 214 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Read the Pre-Processed data\n",
    "restaurant_file = pd.read_csv('processed_data/'+state_filter+'_cleaned_restaurants.csv')\n",
    "restaurant_file = restaurant_file.drop(labels=\"Unnamed: 0\", axis=1)\n",
    "review_file     = pd.read_csv('processed_data/'+state_filter+'_cleaned_reviews.csv')\n",
    "review_file     = review_file.drop(labels=\"Unnamed: 0\", axis=1) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the loaded data to unicode and remove periods and newlines. These are not needed for topic modelling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 44.8 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Convert in Unicode\n",
    "# seperate topic relavant data from sentiment relavent data\n",
    "review_file['topic_text'] = [u''+str(txt).replace('.','').replace('\\\\n','') for txt in review_file.topic_text]\n",
    "review_file['sentiment_text'] = [u''+str(txt).replace('.','').replace('\\\\n','') for txt in review_file.sentiment_text]\n",
    "review_file['text'] = [u''+str(txt) for txt in review_file.text]\n",
    "review_file['review_length'] = review_file.text.map(len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the distribution of the reviews by length of comments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Text(-0.04000000000001336, 0.5, 'Number of Reviews')]"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABDoAAADMCAYAAACSqcOuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de1yUdd7/8TcCIoqum4GwZHWnppvmMQ+UaXqraDBSaqWZhzXTzKyw9ZCarqap5QnzkLqudnu7a5aah8rNQ1pCtZ1WqtvWHuUhDyGBJxB0YL6/P/w1RjCMA8NwMbyej0ePnIuL63ozw5sZPlxzXQHGGCMAAAAAAAA/UKW8AwAAAAAAAHgLgw4AAAAAAOA3GHQAAAAAAAC/waADAAAAAAD4DQYdAAAAAADAbzDoAAAAAAAAfoNBBwAAAAAA8BsMOvzAP/7xD61YsaJc9v3GG29o3bp15bLvoiQlJWn69OkuP37gwAH16dNHPXv21ODBg3X69GkfpoPV0B3pww8/VO/evZWQkKD7779f+/fvL3K9I0eOaMCAAbr33nvVt29fff/99z5OCiuhO9K7776rXr16yWazadCgQTpy5EiR6/G8g1+jO1elpqaqadOmyszMLPLjPO/gt+iPtHbtWt11111KSEhQQkKCHn744SLXoz+SDFAK48ePN3/961/LO4Y5deqUGT16tGnevLmZNm1aketcunTJdOzY0Xz22WfGGGPWrVtnhg0b5suYgJMVunP+/HnTtm1bc+jQIWOMMQcPHjStW7c2Fy5cKLRunz59zNatW40xxuzdu9fExcUZh8Ph07yAMdbozunTp02bNm3MqVOnjDHGrF271gwdOrTQejzvwEqs0J1fZGRkmPvvv9/ceuutJiMjo8h1eN6BlVilP4mJic5eFIf+GBNU3oOWyu6TTz7RzJkzVb16dWVnZ2vjxo3av3+/li1bJrvdrmrVqmn8+PFq1qyZunTpoiVLlqhp06aSpGeeeUZt27ZVRkaGzpw5oylTpigtLU3Tp0/XqVOnZLfbFRcXp8cff1xPPPGEOnfurAceeEBffvml+vXrp127dqlevXpaunSpsrOzNXbsWJc5//73v2v9+vUKDg5WSEiIpk+frsOHD2vPnj1KTk5WtWrVFBsbqylTpigjI0Pp6emKjo7WwoULVadOHXXp0kXNmjXTf/7zH40ZM0bp6emFttegQYMC+1yxYoXefvvtQlnWrFmj3//+9wWWvfnmm2rbtq3q16+vc+fOFfk1fPXVVwoLC1Pr1q0lSX379tWLL76oM2fOFNoerI/ulL47drtdU6dOVcOGDSVJDRo0kDFGZ86cUVhYmHO9tLQ0/fDDD4qLi5MkderUSdOmTdP//d//qUmTJiV/EFEu6E7puxMeHq7k5GQFBwcrLy9PJ06cUO3atQt9Hs87/oXueOc1m8Ph0NixY5WYmKhhw4YV+TXwvON/6I93+vPll18qKytLK1asUEREhMaNG6dGjRoVWIf+/H/lPWmp7D7++GPTuHFjc/z4cWOMMYcPHzbx8fEmMzPTGGPMoUOHzF133WWys7NNUlKS82iFs2fPmrZt25rz58+bRYsWOZcPHDjQ7N692xhjTG5urhk4cKB5++23zebNm83o0aONMcYkJSWZu+66y6xfv94Yc2Xid+DAAZcZ8/LyTJMmTUxaWpoxxpjNmzc7P/fX0801a9aY5cuXG2OMcTgcZtiwYWbVqlXGGGM6d+5sFi9e7HZ7pfXr++K3tm/fXugvbnfffbc5ePCgV/YN36I73u2OMcbMmzfP9O7du9DyL7/80sTGxhZY1q9fP7Nr1y6v7Ru+Q3e8153U1FRz5513mlatWpkvvvii0Md53vEvdMc73Zk/f75ZuHChMca4PKKD5x3/Q39K35/s7GwzdOhQ869//csYY8zbb79t7r77bpOVlVVgPfpzBUd0WEBUVJSio6MlScnJyTp9+rSGDBni/HhAQICOHTumPn36qG/fvpowYYK2b9+uLl26qGbNms71Ll68qE8//VTnzp1TUlKSc9m3336rRx99VLNmzVJeXp7279+vkSNHKjk5Wffcc48yMzN1++23u8wXGBioHj16qF+/frrnnnvUoUMHderUqdB6gwcP1meffabVq1fryJEj+u6779S8eXPnx++44w6PtufJdPNaOBwOBQQEFFhmjFFgYKDH24I10B3vdCcvL0+zZ8/WBx98oDVr1hT6ON3xP3THO925/fbblZycrA8++EAjRozQrl27VKtWLefH6Y7/oTul687evXuVmpqqVatWufwaJLrjr+hP6fpTvXr1At259957tWzZMn311Vdq3769czn9uYJBhwVUr17d+W+Hw6GYmBgtXLjQuezUqVOKiIhQYGCgbrvtNu3du1ebNm3SxIkTC2zH4XDIGKP169crNDRUkpSZmamQkBDVqFFDf/zjH/X+++8rKytLCQkJWrp0qXbt2qWuXbsWKsNvzZ07V4cOHVJKSopWrFihLVu2OH+w/OLll19Wamqq+vTpo3bt2ikvL0/GmCK/zmvZ3vDhwzV8+PBrvBfdi4qKKnASOLvdrrNnz6pu3bpe2wd8i+6Uvjvnzp3TU089JWOMXn/99SJ/mfvDH/6g9PR0GWOcX+/p06cVGRl5TfuA9dCd0nUnLS1Nhw4d0t133y1J6tixo8LCwnTs2DHnodYSzzv+iO6UrjsbN27UTz/9pPvvv9+5bPDgwXrxxRcL/ALK845/oj+l68+JEye0Z88eDRw40LnMGKOgoIK/0tOfK7jqisXExMQoOTnZeWbcffv2qVevXsrNzZUkPfjgg1q5cqVycnKc7/n9RVhYmFq0aKHVq1dLks6fP6/+/ftr9+7dkqRu3bpp/vz5iomJUVhYmG6++WatXLlS3bt3LzZTZmamOnXqpNq1a2vIkCF65pln9NVXX0m6MqnMy8uTJO3fv1+DBw/Wfffdpzp16iglJUX5+fkeba8sNW/eXGfPntUXX3wh6cqTbYsWLQr89Q0VF93xXH5+voYPH64bbrhBf/vb31weKRUZGakbb7xR77zzjqQrV2qpUqWKbr311hLvG9ZBdzx3+fJljRkzRkePHpUkffzxx8rLy1P9+vULrMfzjn+jO5575ZVX9O6772rLli3asmWLJOm1114r9Fd2nnf8H/3xXGhoqBYuXKjU1FTnfZaTk6NmzZoVWI/+XMERHRbToEEDTZ8+XWPGjHFO6JYtW6YaNWpIkrp06aJp06bpscceK/Lz586dqxdeeEE2m02XL19WfHy8evXqJUnq2rWrXnjhBf35z3+WJHXo0EHr1q1Tq1atis103XXXaeTIkRoyZIiqVaumwMBAzZgxQ9KVv2LNnj1bkjRq1Ci99NJLSkpKUnBwsFq1aqVjx455tD1vS0tL0/Dhw7VixQrVrVtXixcv1vTp05WTk6PatWtrzpw5ZbJf+B7d8dy7776rf//737p48aL69OnjXP7SSy+pUaNGSkhI0IwZM3T77bdr/vz5ev7557Vs2TJVrVpVSUlJqlKFWbk/oDueq1evnmbMmKHRo0crICBAtWrV0quvvqrQ0FCedyoRuuN9PO9UHvTHc9ddd50WLlyoKVOmyG63KywsTEuWLFHVqlUl0Z/fCjC/Ps4GAAAAAACgAuOIDkiStm7d6vLEUDabzeXlv4DKju4AJUN3gJKhO0DJ0Z/KgyM6AAAAAACA36hcb9QBAAAAAAB+rcwHHVlZWYqPj9fx48clSc8995y6d++uhIQEJSQkaOfOnZKklJQU2Ww2de/eXQsWLHB+/sGDB9W7d2/FxsZq0qRJzrPdAgAAAAAA/FaZnqPjwIEDmjx5so4cOeJc9vXXX+t///d/FRER4VyWm5uriRMnau3atYqKitKIESO0b98+derUSWPHjtWMGTPUokULTZw4URs2bNDDDz98zRkyMrLkcFjj3Tm//311nTlzsbxjOJHHNStlkaTw8Jo+3yfdcY08xbNSnvLojmSd/ljpsZDI447V8vDcY53Hw0pZJPK4Q3es9XhYKY+VskjWy+PN7pTpER0bNmzQ1KlTnUONnJwcnTx5UhMnTpTNZtOiRYvkcDiUmpqqm266SfXq1VNQUJBsNpt27NihEydOKDc3Vy1atJAk9e7dWzt27CjLyGUqKCiwvCMUQB7XrJQF1ns8yFM8q+WpzKz2WJCneFbLU9lZ6fGwUhaJPCie1R4PK+WxUhbJenm8qUyP6Jg5c2aB2z///LPat2+vqVOnqmbNmhoxYoTefPNNVa9eXeHh4c71IiIilJaWptOnTxdYHh4errS0tLKMDAAAAAAAKjCfXl62Xr16WrJkifP2wIED9dZbbyk2NlYBAQHO5cYYBQQEyOFwFLncE3XqhJU+uBeV12HUrpDHNStlKQ90p3jkKZ7V8vialfpjtceCPMWzWh5fs1J3JGs9HlbKIpHHauhO8ayUx0pZJOvl8RafDjr+85//6MiRI4qNjZV0ZXARFBSkyMhIpaenO9dLT09XREREoeU///xzgXN7XAsrvV8tPLym0tMvlHcMJ/K4ZqUsEu/1tOLjQR7XrJSnsp+jw0qPhUQed6yYx9es0h3JWo+HlbJI5HGH7ljv8bBKHitlkayZx1t8enlZY4xefPFFnTt3Tna7Xa+//rq6deum5s2b6/Dhwzp69Kjy8/O1fft2dezYUdHR0QoJCdHnn38uSdqyZYs6duzoy8gAAAAAAKAC8ekRHY0bN9bw4cPVv39/5eXlqXv37oqPj5ckzZ49W6NHj9alS5fUqVMn9ejRQ5I0d+5cTZ48WVlZWWrSpIkGDRrky8gAAAAAAKAC8cmgY8+ePc5/DxgwQAMGDCi0TkxMjLZu3VpoeePGjfXmm2+WaT4AAAAAgFSzVqiqhZTu18Tw8JrKvZSnC+dzvJQK8IxPj+gAAAAAAFhXtZAg2Z7dUurtbJuXIOuc/QGVjU/P0QEAAAAAAFCWGHQAAAAAAAC/waADAAAAAAD4DQYdAAAAAADAbzDoAAAAAAAAfoNBBwAAAAAA8BsMOgAAAAAAgN9g0AEAAAAAAPwGgw4AAAAAAOA3GHQAAAAAAAC/waADAAAAAAD4jWsadPz444+SpL1792rJkiW6cOFCmYYCAAAAAAAoCbeDjilTpmjlypX6/vvvNXnyZB0/flwTJ070RTYAAAAAAACPuB10fP311/rLX/6inTt36v7779esWbN04sQJX2QDAAAAAADwiNtBhzFGVapUUXJystq3by9Jys3NLfNgAAAAAAAAngpyt8KNN96oYcOG6fjx42rbtq2effZZNW7c2BfZAAAAAAAV0GV7vsLDa5ZqG7mX8nThfI6XEqEycTvomDVrlnbu3KnWrVsrODhYd9xxh+677z5fZAMAAAAAVEBVgwNle3ZLqbaxbV6CuAwGSsLtW1eGDBmin3/+WcYYSVL//v0VGhpa5sEAAAAAAAA85XbQMX78eGVmZmr48OHq3bu3li9frh9++MEX2QAAAAAAADzidtDRunVrjR07Vu+++66GDRum9evXKy4uzhfZAAAAAAAAPOL2HB2bN29WcnKyPvnkE0VHR+v+++9Xhw4dfJENAAAAAADAI24HHTNmzFD16tU1YsQIxcbGKjw83Be5AAAAAAAAPOZ20PHJJ5/o888/14cffqgRI0bI4XAoJiZG48eP90U+AAAAAACAa+b2HB1BQUFq166dYmNj1aVLF+Xm5mrfvn2+yAYAAAAAAOARt0d0jBs3Tvv371dUVJS6deumJUuWqH79+r7IBgAAAAAA4BG3g44mTZooMTFRUVFRvsgDAAAAAABQYm7futKnTx+tWLFCgwcP1tmzZzVlyhRlZ2f7IhsAAAAAAIBH3A46Zs6cqVq1aikjI0MhISHKysrSlClTfJENAAAAAADAI24HHQcPHlRiYqKCgoIUGhqquXPn6uDBg77IBgAAAAAA4BG3g44qVQqukp+fX2gZAAAAAACAFbg9GWmbNm308ssvKzc3Vx9++KHWrVundu3a+SIbAAAAAACAR9wemvHnP/9Z1atXV82aNbVgwQI1atRI48aN80U2AAAAAAAAj7g9oiM4OFijRo3SqFGjfJEHAAAAAACgxFwOOp5++mklJSXJZrMV+fFt27aVWSgAAAAAAICScDnoeOyxxyRJzz//vM/CAAAAAAAAlIbLQUfTpk0lSevXr9eDDz6o9u3b+ywUAAAAAABASbg9GWmbNm00f/58devWTcuXL1d6erovcgEAAAAAAHjM7aCjf//+2rBhg1599VWdO3dO/fr1u+YTk2ZlZSk+Pl7Hjx+XJKWkpMhms6l79+5asGCBc72DBw+qd+/eio2N1aRJk5SXlydJOnnypAYMGKAePXpo5MiRys7OLsnXCAAAAAAAKgm3g45f5Obm6vLlyzLGKDAw0O36Bw4cUP/+/XXkyBHn50+cOFFLly7VO++8o6+//lr79u2TJI0dO1ZTpkzRP//5TxljtGHDBknStGnT9PDDD2vHjh1q2rSpli5dWoIvEQAAAAAAVBZuBx2rV6+WzWbTmDFjVLduXW3YsEGLFi1yu+ENGzZo6tSpioiIkCSlpqbqpptuUr169RQUFCSbzaYdO3boxIkTys3NVYsWLSRJvXv31o4dO2S32/Xpp58qNja2wHIAAAAAAABXXJ6M9Bdff/21Jk+erHbt2nm04ZkzZxa4ffr0aYWHhztvR0REKC0trdDy8PBwpaWl6cyZMwoLC1NQUFCB5Z6qUyfM488pS+HhNcs7QgHkcc1KWcoD3SkeeYpntTy+ZqX+WO2xIE/xrJbH16zUHclaj4eVskjksRqrdcebvPHYWun7w0pZJOvl8Ra3g4558+YpNTVV69evV+/evfXNN9+oZcuWHu/I4XAoICDAedsYo4CAAJfLf/n/r/329rXIyMiSw2E8/ryyEB5eU+npF8o7hhN5XLNSFql8fgDRHdfIUzwr5SmvJ2+r9MdKj4VEHnesmMfXrNIdyVqPh5WySORxp6J3x0q/+F6256tqsPvTJriTeylPF87neCFR6Vjxe9VqebzF7aBj06ZNWrVqlS5duqRu3brpiSeeUGJioh588EGPdhQZGVngii3p6emKiIgotPznn39WRESErrvuOl24cEH5+fkKDAx0rg8AAAAA8H9VgwNle3ZLqbezbV6CrPPrPHzB7Tk61q5dq9dff11hYWGqU6eONm3apNdee83jHTVv3lyHDx/W0aNHlZ+fr+3bt6tjx46Kjo5WSEiIPv/8c0nSli1b1LFjRwUHB+uOO+7QO++8I0l666231LFjR4/3CwAAAAAAKg+3R3RUqVJFYWFX3/MVFRV1TVdd+a2QkBDNnj1bo0eP1qVLl9SpUyf16NFDkjR37lxNnjxZWVlZatKkiQYNGiRJmjp1qiZMmKBly5YpKipK8+fP93i/AAAAAACg8nA76Khdu7YOHjzoPD/G1q1b9bvf/e6ad7Bnzx7nv2NiYrR169ZC6zRu3FhvvvlmoeXR0dFau3btNe8LAAAAAABUbm4HHRMnTtTTTz+tY8eOqUOHDgoJCdGSJUt8kQ0AAAAAAMAjbgcd9evX15YtW3TkyBHl5+frv/7rvxQcHOyLbAAAAAAAAB4p9mSkycnJ+uqrrxQYGKj69evr1ltv1ffff6/+/fv7Kh8AAAAAAMA1cznomDNnjiZNmqTHHntMO3fulN1u14wZM9SnTx/dfPPNPowIAAAAAABwbVy+dWXnzp3aunWr0tPTNXPmTK1Zs0bnz5/X//zP/6h169a+zAgAAAAAAHBNXB7RUaNGDdWqVUv169fXN998o/r162vTpk0MOQAAAAAAgGW5PKKjSpWrM5DatWtrypQpCgpye+5SAAAAAACAclPsyUh/Ub16dYYcAAAAAADA8lxOL3766SfNmDGj0L9/MXny5LJNBgAAAAAA4CGXg44BAwYU+W8AAAAAAACrcjnoePLJJ32ZAwAAAAAAoNSu6RwdAAAAAAAAFQGDDgAAAAAA4DdcvnVl165d6tq1qy5fvqyqVav6MhMAAABQajVrhapaSMmvHFizVqgunM/xYiIAgC+4/MmflJSkrl276qGHHtLmzZt9mQkAAAAotWohQbI9u6XEn79tXoIueDEPAMA3XA46atSoodjYWKWlpclmsxX6+LZt28o0GAAAAFCeLtvzFR5es1TbyL2Ux1EhAOBjLgcdf/3rX3Xw4EFNmjRJzz//vC8zAQAAAOWuanBgqY4IkTgqBADKg8tBR1hYmNq0aaPly5crIiJC33zzjfLy8tSsWTOFhYX5MiMAAAAAAMA1cXt2pgsXLmjgwIG6/vrrlZ+fr7S0NL366qtq1aqVL/IBAAAAAABcM7eDjjlz5mju3Llq3769JOmjjz7S7NmztWHDhjIPBwAAAAAA4Ikq7lbIzs52DjkkKSYmRjk5nFAJAAAAAABYj9sjOgICAnTixAlFR0dLko4fP67AwMAyDwYAAAAAQGl54wpKEldRqkjcDjpGjRqlhx56SDExMQoICND+/fs1depUX2QDAAAAKjRv/IJ12Z7vpTRA5eSNKyhJXEWpInE76OjatatuueUWffzxx3I4HBoxYoTq16/vi2wAAABAheatS9QCAK6d20GHJN1yyy265ZZbyjoLAAAAAABAqbg9GSkAAAAAAEBFwaADAAAAAAD4DbeDjnHjxvkiBwAAAAAAQKm5HXQcPHhQxhhfZAEAAAAAACgVtycjjYiIUFxcnJo3b64aNWo4l0+ePLlMgwEAAAAAAHjK7aCjZcuWatmypS+yAAAAAAAAlIrbQceTTz6p3NxcHT16VA0bNtSlS5cUGhrqi2wAAAAAAAAecXuOjgMHDqhr164aMWKETp8+rXvuuUdffPGFL7IBAAAAAAB4xO2gY86cOVqzZo1q166tyMhIvfTSS5o5c6YvsgEAAAAAAHjE7aAjNzdXDRo0cN7u1KmT8vPzyzQUAAAAAABASbgddAQFBencuXMKCAiQJP3www9lHgoAAAAAAKAk3J6MdOTIkXrkkUeUnp6uMWPGKDk5WdOnT/dFNgAAAAAAAI+4HXR07txZt9xyi5KTk+VwODRq1CjVr1+/VDsdOHCgMjMzFRR0ZffTp0/XsWPHtGzZMuXl5Wnw4MEaMGCAJCklJUWzZs3SpUuX1LNnTyUmJpZq3wAAAEBFctmer/DwmqXaRu6lPF04n+OlRABgbW4HHZKUl5cnh8OhoKAg53CipIwxOnLkiN5//33nttLS0pSYmKhNmzapatWq6tevn9q1a6cbbrhBEydO1Nq1axUVFaURI0Zo37596tSpU6kyAAAAABVF1eBA2Z7dUqptbJuXoAteygMAVud2arFx40bNnz9fHTp0UH5+vhYvXqznn39esbGxJdrhL+f4GDp0qM6ePasHH3xQNWrUUPv27VW7dm1JUmxsrHbs2KG2bdvqpptuUr169SRJNptNO3bsYNABAAAAAL9Ss1aoqoWU7o/SgL9w24Q1a9Zo8+bNioiIkCSdPHlSI0aMKPGg4/z584qJidHzzz8vu92uQYMGqWfPngoPD3euExERodTUVJ0+fbrQ8rS0tBLtFwAAAAD8VbWQoFIf+SNdOfoHRfPG28jCw2vyVjIfcDvoCA4Odg45JOkPf/iDgoODS7zDli1bqmXLls7bffv21axZszRy5EjnMmOMAgIC5HA4nFd7+fVyT9SpE1birGWhtMXwNvK4ZqUs5YHuFI88xbNaHl+zUn+s9liQp3hWy+NrVuqOP/Lm95fVvletlsfX6I5veONtZNKVYVI1i3zP+mt3XA46vvnmG0lSo0aNNH36dD300EMKDAzUpk2b1KpVqxLv8LPPPpPdbldMTIykK8OL6OhopaenO9dJT09XRESEIiMji1zuiYyMLDkcpsR5vSk8vKbS063z7kjyuGalLFL5/ACiO66Rp3hWylNeT95W6Y+VHguJPO5YMY+vebs7/voCvqS89f1lxe9Vq+XxtYyMLIYdFYwVvmf9uTsuBx2jR48ucHvv3r3OfwcEBGjy5Mkl2uGFCxe0aNEirV+/Xna7XZs3b9bLL7+ssWPHKjMzU6GhoXrvvff0wgsvqFGjRjp8+LCOHj2qG264Qdu3b1efPn1KtF8AAAAAAOD/XA469uzZUyY77Ny5sw4cOKD77rtPDodDDz/8sFq3bq3ExEQNGjRIdrtdffv2VbNmzSRJs2fP1ujRo3Xp0iV16tRJPXr0KJNcAAAAAACg4nN7jo709HRt3rxZZ8+eLbB83LhxJd7pM888o2eeeabAMpvNJpvNVmjdmJgYbd26tcT7AgAAAAAAlYfbQcfIkSMVGRnpvMQrAAAAgIrFG1eL4EoRACoKt4MOu92uxYsX+yILAAAAgDLgjatFbJuXIOucthAAXKviboUmTZro0KFDvsgCAAAAAABQKm6P6GjVqpXuu+8+hYeHKyjo6uq7d+8u02AAAAAAAACecjvoWLVqlebOnasbb7zRF3kAAAAAAABKzO2go1atWrr33nt9kQUAAAAAAKBU3A462rdvrzlz5qh79+6qWrWqc3mTJk3KNBgAAAAAAICn3A46tm3bJkn65z//6VwWEBDAOToAAAAAAIDluB107Nmzxxc5AAAAAADwe5ft+QoPr1nq7eReytOF8zleSOR/3A46Vq9eXeTyP/3pT14PAwAACqtZK1TVQtw+ZRfplxdSvBgCAMAaqgYHyvbsllJvZ9u8BF3wQh5/5PZV06FDh5z/vnz5sj799FPFxMSUaSgAAHBVtZCgUr8g2jg7nr8eoUIpzYAPAFC5uX32mDVrVoHbaWlpmjRpUpkFAgAA3sdfj1DReGPAt21egpfSQCp4uH1JB6cMSwH4gsdj8rp16+rEiRNlkQUAAFhcad9XzFtpgIrLGwNThqUAfMGjc3QYY/T111+rTp06ZRoKAABYE0eGAAAAq/PoHB2SFBUVpXHjxpVZIAAAAAAAgJLy+BwdAAAApcWl9QAAKB1vPJeGh9f0y+dSl4OO5557zuUnBQQE6MUXXyyTQAAAwP/xFhigcvLGL2b++EsZUBI8l7rmctDRsGHDQsvOnDmj1157TdHR0WUaCgAAAID/4YSmAJAGSR0AABB8SURBVHzB5aBj6NChBW6npKRo/Pjxstlsmjx5cpkH80ccpgsAgHd54yowPK8CAOBf3J6jIy8vT/PmzdPmzZs1bdo0xcbG+iKXpdSsFapqIR5fibdIHFoEXOFpr1z9IuONX1BK0vGi8vDLEuB7/HUYAIDS8cc/yBf7yv7IkSMaM2aMatSoobfeekuRkZG+ymUp1UKCvDag8AZvvbcRKE/e6tXG2fFe+cHMEBIAAACVkT+e68PloGPjxo2aM2eO/vSnP2nkyJG+zAQ3vPXXK8Af0AcAAAAAv+Zy0DFp0iRVqVJFK1as0MqVK53LjTEKCAjQF1984ZOAAAAAAPCLoo5u9vToTisdYg/A+1wOOnbv3u3LHPCxy/Z8VQ0O5PJegBf54/sbAQCwGs7NA8Adl4MOLiHr3/zxfVhAeaNXAAAAQPnzzqVEAAAAAKCC8NbJ/TkCE7AmBh0oFW8eqg/gipL0isvdAiXnreeyS5fzFVI1sFTbCA+vSXcBH+DtL4D3eev51BsYdKBUvHmoPoAreAsM4Fve7BzdBQBUVqV9PvXm74RVvLYlAAAAAACAcsYRHbAEb1wFhkN9AVhNzVqhqhbCUy0A+CMrHaYPoCBefcESeJ8k4H3led6B3+63sg4iq4UE8fY+eIxLVQMVg6vXr/zMBsofgw4A8FNWOu8Ag0jg2nGeHgAASodBB/wGV4DxDQ7FB4CKgcPqAQCVFb+twG946y9gG2fHSyr6cp2e8NdDhjkUHyXBofiA71np7PcAAPgSgw7gN7w5MOEIE+AKDsUHAACArzDoAMqIN3+xA3AFh+IDAADAHQYdgMX9culdAJ4PEBkUAgAAVD5+P+ioUyesvCMApcKQAwAAAACuXYUYdGzbtk3Lli1TXl6eBg8erAEDBlzz5z464z2dPlO6E9fxF0EAAAAAACoGyw860tLStGDBAm3atElVq1ZVv3791K5dOzVo0KC8owEAAAAAAIux/KAjJSVF7du3V+3atSVJsbGx2rFjh5588slr+vw6v6vmlRwRvw/1u+1YKYu3tmOlLN7cTnmoUiXA5cesdv/wvVMxtmOlLGXt1/2x0tdtpSxW246VslhxO77y2+ceKzy2VsjANvx7G97wS3es9rPDSvcx26lY2ymtAGOMKe8QxVm+fLkuXryoxMRESdIbb7yh1NRUvfDCC+WcDAAAAAAAWE2V8g7gjsPhUEDA1em+MabAbQAAAAAAgF9YftARGRmp9PR05+309HRFRESUYyIAAAAAAGBVlh903Hnnnfroo4+UmZmpnJwcvffee+rYsWN5xwIAAAAAABZk+ZOR1q1bV4mJiRo0aJDsdrv69u2rZs2alXcsAAAAAABgQZY/GSkAAAAAAMC1svxbVwAAAAAAAK4Vgw4AAAAAAOA3GHQAAAAAAAC/waADAAAAAAD4Db8ddGzbtk333nuvunfvrnXr1vlkn4sXL1ZcXJzi4uL00ksvSZJSUlJks9nUvXt3LViwwLnuwYMH1bt3b8XGxmrSpEnKy8srs1xz5szRhAkTit3vyZMnNWDAAPXo0UMjR45Udna213Ps2bNHvXv3Vs+ePTVjxgxJ5Xv/bNmyxfl4zZkzp9j9ltX9k5WVpfj4eB0/flyS5/dHWeSiO1fRnaLRnaKVR3cka/aH7hTNCt2RrNcfunMV3Ska3XGN121X0Z/CKnV3jB/66aefTOfOnc2ZM2dMdna2sdls5rvvvivTfSYnJ5uHHnrIXLp0yVy+fNkMGjTIbNu2zXTq1MkcO3bM2O12M3ToULN3715jjDFxcXHmyy+/NMYY89xzz5l169aVSa6UlBTTrl07M378+GL3O3z4cLN9+3ZjjDGLFy82L730kldzHDt2zHTo0MGcOnXKXL582fTv39/s3bu33O6fixcvmjZt2piMjAxjt9tN3759TXJysk/vn3//+98mPj7eNGnSxPz4448mJyfH4/vD27nozlV0p2h0p2jl0R1jrNkfulM0K3THGOv1h+5cRXeKRndc43XbVfSnsMreHb88oiMlJUXt27dX7dq1Vb16dcXGxmrHjh1lus/w8HBNmDBBVatWVXBwsOrXr68jR47opptuUr169RQUFCSbzaYdO3boxIkTys3NVYsWLSRJvXv3LpN8Z8+e1YIFC/T4449Lksv92u12ffrpp4qNjS2zPDt37tS9996ryMhIBQcHa8GCBQoNDS23+yc/P18Oh0M5OTnKy8tTXl6egoKCfHr/bNiwQVOnTlVERIQkKTU11aP7oyxy0Z0r6I5rdKdo5dEdyXr9oTuuWaE7kvX6Q3euoDuu0R3XeN12Bf0pWmXvTpBX0lvM6dOnFR4e7rwdERGh1NTUMt1nw4YNnf8+cuSI3n33XT3yyCOFcqSlpRXKFx4errS0NK9nmjJlihITE3Xq1ClJhe+XX/Z75swZhYWFKSgoqMzyHD16VMHBwXr88cd16tQp3XPPPWrYsGG53T9hYWF6+umn1bNnT4WGhqpNmzYKDg726f0zc+bMAreL+r4t7v4oi1x05wq64xrdKVp5dEeyXn/ojmtW6I5kvf7QnSvojmt0xzVet11Bf4pW2bvjl0d0OBwOBQQEOG8bYwrcLkvfffedhg4dqnHjxqlevXpF5vBFvjfeeENRUVGKiYlxLnO136L27+08+fn5+uijj/Tiiy/q9ddfV2pqqn788cdyu3++/fZbbdy4Ue+//74+/PBDValSRcnJyeV2/0iuHx9fPm50h+64Q3c8y+ArVugP3SmeFbsjlX9/6A7dcYfueJ7BF6zQHYn+FKeyd8cvj+iIjIzUZ5995rydnp7uPFSmLH3++ed66qmnNHHiRMXFxelf//qX0tPTC+WIjIwssPznn3/2er533nlH6enpSkhI0Llz53Tx4kUFBAQUud/rrrtOFy5cUH5+vgIDA8vk/rr++usVExOj6667TpLUtWtX7dixQ4GBgc51fHn/7N+/XzExMapTp46kK4dArVq1qtzuH0mFvm5390dZ5KI7dMcduuM6Q3l0R7JOf+hO8azYHan8+0N36I47dKf4DLxuoz+uVPbu+OURHXfeeac++ugjZWZmKicnR++99546duxYpvs8deqURo0apblz5youLk6S1Lx5cx0+fFhHjx5Vfn6+tm/fro4dOyo6OlohISH6/PPPJV05G663861evVrbt2/Xli1b9NRTT6lLly6aNWtWkfsNDg7WHXfcoXfeeUeS9NZbb3k9T+fOnbV//36dP39e+fn5+vDDD9WjR49yu38aN26slJQUXbx4UcYY7dmzR23bti23+0fy/PulLHLRHbrjDt0pWnl0R7JWf+hO8azYHan8+0N36I47dMc1XrfRn+JU+u64PV1pBbV161YTFxdnunfvblasWFHm+3vhhRdMixYtTK9evZz//f3vfzcpKSnGZrOZ7t27m5kzZxqHw2GMMebgwYOmT58+JjY21owZM8ZcunSpzLJt3LjReQZiV/s9fvy4eeSRR0zPnj3N0KFDzdmzZ72e44033nA+JtOmTTP5+fnlev8sX77cxMbGmvj4ePPcc8+Z3Nzccrl/OnfubH788UdjjPH4/iiLXHTnKrpTNLpTNF93xxjr9ofuFM0q3THGWv2hO1fRnaLRHdd43XYV/SmsMncnwBhjymRUAwAAAAAA4GN++dYVAAAAAABQOTHoAAAAAAAAfoNBBwAAAAAA8BsMOgAAAAAAgN9g0AEAAAAAAPwGg45K5h//+IdWrFhRpvvo0qWLvvrqqzLZ9tChQ5WZmVnm+wGKQn+AkqE7QMnQHaBk6A6CyjsAfKt///7lHaFUkpOTyzsCKjH6A5QM3QFKhu4AJUN3wKCjgvnkk080c+ZMVa9eXdnZ2Xr66ae1fPly2e12VatWTePHj1ezZs3UpUsXLVmyRE2bNpUkPfPMM2rbtq0yMjJ05swZTZkyRWlpaZo+fbpOnTolu92uuLg4Pf7443riiSfUuXNnPfDAA/ryyy/Vr18/7dq1S/Xq1dPSpUuVnZ2tsWPHXlPePXv2aNmyZQXytWzZUq+88opOnDih9PR0nThxQnXr1tXLL7+siIgIpaam6i9/+YvsdrtuvPFGnTx5UhMmTNBbb70lSRo8eLBzQvv6669r6tSpyszMVEJCghITE8vmjodfoD/0ByVDd+gOSobu0B2UDN2hO6VmUKF8/PHHpnHjxub48ePm8OHDJj4+3mRmZhpjjDl06JC56667THZ2tklKSjLTpk0zxhhz9uxZ07ZtW3P+/HmzaNEi5/KBAwea3bt3G2OMyc3NNQMHDjRvv/222bx5sxk9erQxxpikpCRz1113mfXr1xtjjOnTp485cOBAsRk7d+5sUlNTi823aNEi89///d/mwoULxhhjRowYYZKSkozdbjcdO3Y0e/fuNcYY89FHH5lGjRqZjz/+2BhjzK233moyMjKc+5k+fboxxpjTp0+bpk2bmpMnT3rhXoa/oj/0ByVDd+gOSobu0B2UDN2hO6XFER0VUFRUlKKjo7Vu3TqdPn1aQ4YMcX4sICBAx44dU58+fdS3b19NmDBB27dvV5cuXVSzZk3nehcvXtSnn36qc+fOKSkpybns22+/1aOPPqpZs2YpLy9P+/fv18iRI5WcnKx77rlHmZmZuv32268pZ3Jysst8ktS2bVuFhYVJkm677TadO3dOhw4dkiR16tRJktS+fXs1bNjQ5T7i4+MlSeHh4br++uuVkZGhqKioa8qHyon+XEV/4Am6cxXdgSfozlV0B56gO1fRHc8x6KiAqlevLklyOByKiYnRwoULnR87deqUIiIiFBgYqNtuu0179+7Vpk2bNHHixALbcDgcMsZo/fr1Cg0NlSRlZmYqJCRENWrU0B//+Ee9//77ysrKUkJCgpYuXapdu3apa9euCggIuKacxeXbuXOnqlWr5lweEBAgY4wCAwNljCmwncDAQJf7CAq6+i38yzaA4tCfq+gPPEF3rqI78ATduYruwBN05yq64zmuulKBxcTEKDk5Wd9//70kad++ferVq5dyc3MlSQ8++KBWrlypnJwctW7dusDnhoWFqUWLFlq9erUk6fz58+rfv792794tSerWrZvmz5+vmJgYhYWF6eabb9bKlSvVvXt3r+UrSv369VW1alV98MEHkqTU1FQdOnTI+YMmMDBQeXl515wBcIX+ACVDd4CSoTtAydAdlASDjgqsQYMGmj59usaMGaNevXopKSlJy5YtU40aNSRduRTRiRMn9MADDxT5+XPnztWBAwdks9n0wAMPKD4+Xr169ZIkde3aVYcPH9add94pSerQoYPy8vLUqlUrr+UrSlBQkF555RUtXrxY9913n/72t7/p+uuvd05Ce/TooYEDBzoP9wJKiv4AJUN3gJKhO0DJ0B2URIDhuBdYzJw5c/Too4/q+uuv16lTp5SQkKBdu3apVq1a5R0NsDz6A5QM3QFKhu4AJUN3yhbn6IDHtm7dqlWrVhX5MZvNpmHDhpVq+9HR0RoyZIiCgoJkjNGMGTMoPPwG/QFKhu4AJUN3gJKhOxUbR3QAAAAAAAC/wTk6AAAAAACA32DQAQAAAAAA/AaDDgAAAAAA4DcYdAAAAAAAAL/BoAMAAAAAAPgNBh0AAAAAAMBv/D8O10rjFqCMRQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x216 with 5 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#let's check the distribution of the reviews by length of comments\n",
    "ax = sns.FacetGrid(data=review_file, col='review_stars', xlim=(0, 1000)).map(plt.hist, 'review_length', bins=50)\n",
    "ax.axes[0][0].set(ylabel='Number of Reviews')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### It's interesting that five star reivews are more verbose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Postive  Reviews: 12,552\n",
      "Negative Reviews: 3,927\n"
     ]
    }
   ],
   "source": [
    "pos_reviews_sent  = review_file.sentiment_text[review_file.review_stars>3].values\n",
    "neg_reviews_sent  = review_file.sentiment_text[review_file.review_stars<3].values\n",
    "pos_reviews_topic = review_file.topic_text[review_file.review_stars>3].values\n",
    "neg_reviews_topic = review_file.topic_text[review_file.review_stars<3].values\n",
    "\n",
    "all_reviews_topic = review_file.topic_text.values\n",
    "all_reviews_sent  = review_file.sentiment_text.values\n",
    "\n",
    "print('Postive  Reviews: {:,}'.format(len(pos_reviews_topic)))\n",
    "print('Negative Reviews: {:,}'.format(len(neg_reviews_topic)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_reviews_topic = pos_reviews_sent\n",
    "neg_reviews_topic = neg_reviews_sent\n",
    "all_reviews_topic = all_reviews_sent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Take our additional stopwords\n",
    "#from sklearn.feature_extraction import text\n",
    "#extra_words = ['like','did','said','ok', 've', 'got']\n",
    "#stop_words = text.ENGLISH_STOP_WORDS.union(extra_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load our additional stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_words = [line.rstrip('\\n') for line in open('config/stopwords.txt', 'r', encoding='utf-8')] \n",
    "neg_words  = [line.rstrip('\\n') for line in open('config/stopwords.txt', 'r', encoding='utf-8')]\n",
    "stop_words = stop_words + neg_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets see how many words and features are in our vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32683"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = list(sorted(set(reduce(operator.concat, all_reviews['topic_text']).split())))\n",
    "len(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we take all words and phrases in vocab, this will take a lot of time in building the topic model. Instead we will take only the top 2500 words/phrases as our features for topic model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set max features for data vertorication\n",
    "max_features=5000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our tokenizer will split on space and consider words/phrases with length > 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenize the text\n",
    "def tokenizer(x):\n",
    "    return ( w for w in str(x).split() if len(w) >3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Create a vectorizer object to generate term document counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#TF-IDF\n",
    "# Create a vectorizer object to generate term document counts\n",
    "tfidf_pos = TfidfVectorizer(stop_words=stop_words, min_df=10, max_df=0.5,\n",
    "                        tokenizer = tokenizer,ngram_range=(1,1), max_features=max_features)\n",
    "                        #ngram_range=(1,1), token_pattern='[a-z][a-z]+', max_features=max_features)\n",
    "\n",
    "tfidf_neg = TfidfVectorizer(stop_words=stop_words, min_df=10, max_df=0.5, \n",
    "                        tokenizer = tokenizer,ngram_range=(1,1), max_features=max_features)\n",
    "                        #ngram_range=(1,1), token_pattern='[a-z][a-z]+', max_features=max_features)\n",
    "\n",
    "tfidf_all = TfidfVectorizer(stop_words=stop_words, min_df=10, max_df=0.5, \n",
    "                        tokenizer = tokenizer,ngram_range=(1,1), max_features=max_features)\n",
    "                        #ngram_range=(1,1), token_pattern='[a-z][a-z]+', max_features=max_features)\n",
    "\n",
    "#  Stats\n",
    "## Wall time: 2min 47s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1.05 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Vectorize Data\n",
    "neg_vectors  = tfidf_neg.fit_transform(neg_reviews_topic)\n",
    "pos_vectors  = tfidf_pos.fit_transform(pos_reviews_topic)\n",
    "all_vectors  = tfidf_all.fit_transform(all_reviews_topic)\n",
    "\n",
    "#  Stats\n",
    "## Wall time: 1min 25s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "neg_feature_names = np.array(tfidf_neg.get_feature_names())\n",
    "pos_feature_names = np.array(tfidf_pos.get_feature_names())\n",
    "all_feature_names = np.array(tfidf_all.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We pickle dump time to time to resume later from that checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/topic_term_vector_all.pk','wb') as f:\n",
    "#    pickle.dump(tfidf_all, f)\n",
    "#with open('pickles/topic_term_vector_pos.pk','wb') as f:\n",
    "#    pickle.dump(tfidf_pos, f)\n",
    "#with open('pickles/topic_term_vector_neg.pk','wb') as f:\n",
    "#    pickle.dump(tfidf_neg, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/neg_topic_vectors_'+str(max_features)+'.pk', 'wb') as fin:\n",
    "#    pickle.dump(neg_vectors, fin)\n",
    "#with open('pickles/neg_topic_features_'+str(max_features)+'.pk', 'wb') as fin:\n",
    "#    pickle.dump(np.array(tfidf_neg.get_feature_names()), fin)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/pos_topic_vectors_'+str(max_features)+'.pk', 'wb') as fin:\n",
    "#    pickle.dump(pos_vectors, fin)\n",
    "#with open('pickles/pos_topic_features_'+str(max_features)+'.pk', 'wb') as fin:\n",
    "#    pickle.dump(np.array(tfidf_pos.get_feature_names()), fin)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/all_topic_vectors_'+str(max_features)+'.pk', 'wb') as fin:\n",
    "#    pickle.dump(all_vectors, fin)\n",
    "#with open('pickles/all_topic_features_'+str(max_features)+'.pk', 'wb') as fin:\n",
    "#    pickle.dump(np.array(tfidf_all.get_feature_names()), fin) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/neg_topic_vectors_'+str(max_features)+'.pk', 'rb') as fin:\n",
    "#    neg_vectors = pickle.load(fin)\n",
    "#with open('pickles/neg_topic_features_'+str(max_features)+'.pk', 'rb') as fin:\n",
    "#    neg_feature_names = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/pos_topic_vectors_'+str(max_features)+'.pk', 'rb') as fin:\n",
    "#    pos_vectors = pickle.load(fin)\n",
    "#with open('pickles/pos_topic_features_'+str(max_features)+'.pk', 'rb') as fin:\n",
    "#    pos_feature_names = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/all_topic_vectors_'+str(max_features)+'.pk', 'rb') as fin:\n",
    "#    all_vectors = pickle.load(fin)\n",
    "#with open('pickles/all_topic_features_'+str(max_features)+'.pk', 'rb') as fin:\n",
    "#    all_feature_names = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(19220, 5000)"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_vectors.get_shape()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of topics is set to 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_TOPICS = 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 565 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# LSA\n",
    "from gensim import corpora, models, similarities, matutils\n",
    "\n",
    "# Convert sparse matrix of counts to a gensim corpus\n",
    "# Need to transpose it for gensim which wants terms by docs instead of docs by terms\n",
    "tfidf_pos_corpus = matutils.Sparse2Corpus(pos_vectors.transpose())\n",
    "tfidf_neg_corpus = matutils.Sparse2Corpus(neg_vectors.transpose())\n",
    "tfidf_all_corpus = matutils.Sparse2Corpus(all_vectors.transpose())\n",
    "\n",
    "# Row indices\n",
    "id2word_pos = dict((v,k) for k,v in tfidf_pos.vocabulary_.items())\n",
    "id2word_neg = dict((v,k) for k,v in tfidf_neg.vocabulary_.items())\n",
    "id2word_all = dict((v,k) for k,v in tfidf_all.vocabulary_.items())\n",
    "\n",
    "# This is a hack for Python 3!\n",
    "id2word_pos = corpora.Dictionary.from_corpus(tfidf_pos_corpus, id2word=id2word_pos)\n",
    "id2word_neg = corpora.Dictionary.from_corpus(tfidf_neg_corpus, id2word=id2word_neg)\n",
    "id2word_all = corpora.Dictionary.from_corpus(tfidf_all_corpus, id2word=id2word_all)\n",
    "\n",
    "#  Stats\n",
    "## Wall time: 1min 21s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Create LSA models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 2.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# create LSA model\n",
    "lsi_pos = models.LsiModel(tfidf_pos_corpus, id2word=id2word_pos, num_topics=NUM_TOPICS)\n",
    "lsi_neg = models.LsiModel(tfidf_neg_corpus, id2word=id2word_neg, num_topics=NUM_TOPICS)\n",
    "lsi_all = models.LsiModel(tfidf_all_corpus, id2word=id2word_all, num_topics=NUM_TOPICS)\n",
    "\n",
    "#  Stats\n",
    "## Wall time: 13min 3s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**LSA:** look at top positive topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 5 topics for POSITIVE reviews\n",
      "---------------------------------\n",
      "[(0,\n",
      "  '0.287*\"good\" + 0.286*\"food\" + 0.203*\"like\" + 0.199*\"love\" + 0.181*\"time\" + '\n",
      "  '0.172*\"great\" + 0.161*\"service\" + 0.137*\"delicious\" + 0.127*\"chicken\" + '\n",
      "  '0.125*\"pretty\"'),\n",
      " (1,\n",
      "  '-0.819*\"pizza\" + 0.355*\"food\" + 0.137*\"service\" + -0.112*\"papa_del\" + '\n",
      "  '-0.107*\"good_pizza\" + -0.107*\"slice\" + -0.106*\"love\" + -0.093*\"cheese\" + '\n",
      "  '-0.089*\"crust\" + 0.085*\"great\"'),\n",
      " (2,\n",
      "  '0.709*\"love\" + -0.563*\"good\" + -0.160*\"pretty\" + 0.138*\"great\" + '\n",
      "  '-0.121*\"like\" + 0.101*\"delicious\" + 0.084*\"staff\" + 0.080*\"amazing\" + '\n",
      "  '0.060*\"atmosphere\" + -0.059*\"mexican_food\"'),\n",
      " (3,\n",
      "  '-0.545*\"love\" + -0.481*\"good\" + 0.326*\"food\" + 0.230*\"pizza\" + '\n",
      "  '0.208*\"great\" + -0.208*\"chicken\" + 0.146*\"service\" + 0.119*\"wait\" + '\n",
      "  '0.111*\"time\" + 0.108*\"staff\"'),\n",
      " (4,\n",
      "  '0.408*\"good\" + 0.382*\"food\" + 0.291*\"pizza\" + 0.221*\"great\" + -0.219*\"like\" '\n",
      "  '+ 0.200*\"love\" + 0.198*\"service\" + -0.195*\"sauce\" + -0.175*\"dollar\" + '\n",
      "  '-0.157*\"sandwich\"')]\n"
     ]
    }
   ],
   "source": [
    "topics_to_print = 5\n",
    "\n",
    "# look at top positive topics\n",
    "print('Top %d topics for POSITIVE reviews' %(topics_to_print))\n",
    "print('-'*33)\n",
    "pprint(lsi_pos.show_topics()[0:topics_to_print])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**LSA:** look at top negative topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 5 topics for NEGATIVE reviews\n",
      "---------------------------------\n",
      "[(0,\n",
      "  '0.333*\"food\" + 0.209*\"time\" + 0.199*\"like\" + 0.190*\"wait\" + 0.172*\"service\" '\n",
      "  '+ 0.158*\"table\" + 0.156*\"because\" + 0.143*\"good\" + 0.136*\"dollar\" + '\n",
      "  '0.130*\"pizza\"'),\n",
      " (1,\n",
      "  '0.423*\"wait\" + 0.296*\"pizza\" + 0.238*\"table\" + -0.216*\"taste\" + '\n",
      "  '-0.211*\"chicken\" + -0.204*\"like\" + 0.195*\"hour\" + -0.174*\"dollar\" + '\n",
      "  '0.144*\"tell\" + -0.140*\"price\"'),\n",
      " (2,\n",
      "  '-0.820*\"pizza\" + 0.326*\"food\" + 0.144*\"wait\" + -0.124*\"crust\" + '\n",
      "  '-0.115*\"sauce\" + -0.113*\"cheese\" + -0.103*\"slice\" + 0.097*\"table\" + '\n",
      "  '-0.088*\"like\" + 0.084*\"service\"'),\n",
      " (3,\n",
      "  '0.668*\"food\" + 0.260*\"pizza\" + -0.234*\"dollar\" + -0.200*\"table\" + '\n",
      "  '-0.156*\"sandwich\" + -0.146*\"steak\" + -0.129*\"burger\" + -0.127*\"server\" + '\n",
      "  '-0.123*\"drink\" + 0.111*\"service\"'),\n",
      " (4,\n",
      "  '0.562*\"dollar\" + -0.556*\"chicken\" + 0.296*\"price\" + 0.162*\"sushi\" + '\n",
      "  '-0.144*\"wait\" + 0.116*\"charge\" + 0.114*\"not_worth\" + -0.110*\"sauce\" + '\n",
      "  '-0.110*\"taste\" + 0.105*\"food\"')]\n"
     ]
    }
   ],
   "source": [
    "# look at top positive topics\n",
    "print('Top %d topics for NEGATIVE reviews' %(topics_to_print))\n",
    "print('-'*33)\n",
    "pprint(lsi_neg.show_topics()[0:topics_to_print])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### We do **NMF (Nonnegative Matrix Factorization)** to get rid of negative word probabilities\n",
    "\n",
    "Negative values in LSA topics are annoying, let's see if we can fix that with NMF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 34.9 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# negative values in LSA topics are annoying\n",
    "# let's see if we can fix that with NMF\n",
    "from sklearn.decomposition import NMF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# change num_topics\n",
    "num_topics = NUM_TOPICS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 5.98 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "nmf_pos = NMF(n_components=num_topics)\n",
    "W_pos = nmf_pos.fit_transform(pos_vectors)\n",
    "H_pos = nmf_pos.components_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 2.35 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "nmf_neg = NMF(n_components=num_topics)\n",
    "W_neg = nmf_neg.fit_transform(neg_vectors)\n",
    "H_neg = nmf_neg.components_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 18.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "nmf_all = NMF(n_components=num_topics)\n",
    "W_all = nmf_all.fit_transform(all_vectors)\n",
    "H_all = nmf_all.components_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Checkpointing: dump processed models in pickles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save models for future use\n",
    "#with open('pickles/neg_topic_model.pk', 'wb') as fin:\n",
    "#    pickle.dump(lsi_neg, fin)\n",
    "#with open('pickles/pos_topic_model.pk', 'wb') as fin:\n",
    "#    pickle.dump(lsi_pos, fin)\n",
    "#with open('pickles/all_topic_model.pk', 'wb') as fin:\n",
    "#    pickle.dump(lsi_all, fin)        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save models for future use\n",
    "#with open('pickles/neg_nmf_model.pk', 'wb') as fin:\n",
    "#    pickle.dump(nmf_neg, fin)\n",
    "#with open('pickles/pos_nmf_model.pk', 'wb') as fin:\n",
    "#    pickle.dump(nmf_pos, fin)\n",
    "#with open('pickles/all_nmf_model.pk', 'wb') as fin:\n",
    "#    pickle.dump(nmf_all, fin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Load saved pickles and resume"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/all_topic_model.pk', 'rb') as fin:\n",
    "#    lsi_all = pickle.load(fin)\n",
    "#with open('pickles/pos_topic_model.pk', 'rb') as fin:\n",
    "#    lsi_pos = pickle.load(fin)\n",
    "#with open('pickles/neg_topic_model.pk', 'rb') as fin:\n",
    "#    lsi_neg = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/neg_nmf_model.pk', 'rb') as fin:\n",
    "#    nmf_neg = pickle.load(fin)\n",
    "#with open('pickles/pos_nmf_model.pk', 'rb') as fin:\n",
    "#    nmf_pos = pickle.load(fin)\n",
    "#with open('pickles/all_nmf_model.pk', 'rb') as fin:\n",
    "#    nmf_all = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/all_topic_features_'+str(max_features)+'.pk', 'rb') as fin:\n",
    "#    all_feature_names = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/pos_topic_features_'+str(max_features)+'.pk', 'rb') as fin:\n",
    "#    pos_feature_names = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/neg_topic_features_'+str(max_features)+'.pk', 'rb') as fin:\n",
    "#    neg_feature_names = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('pickles/topic_term_vector_all.pk', 'rb') as fin:\n",
    "#    term_vector = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Utility functions to get the top-n topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_list(model, feature_names, num_topics, no_top_words):\n",
    "    lst = []\n",
    "    for topic_idx, topic in enumerate(model.components_):\n",
    "        if topic_idx < num_topics:\n",
    "            #print(\"{:11}\".format(\"Topic %d:\" %(topic_idx)), end='')\n",
    "            #print(\", \".join(['{:04.3f}*'.format(topic[i])+feature_names[i] for i in topic.argsort()[:-no_top_words-1:-1]]))\n",
    "            lst.append('/'.join(sort_list([feature_names[i]+\" \" for i in topic.argsort()[:-no_top_words-1:-1]])))\n",
    "    return((list((set(lst)))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_map(model, feature_names, num_topics, no_top_words):\n",
    "    dicty = {}\n",
    "    for topic_idx, topic in enumerate(model.components_):\n",
    "        if topic_idx < num_topics:\n",
    "            dicty[topic_idx]='/'.join(sort_list([feature_names[i]+\" \" for i in topic.argsort()[:-no_top_words-1:-1]]))\n",
    "    return dicty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_topics(model, feature_names, num_topics, no_top_words):\n",
    "    for topic_idx, topic in enumerate(model.components_):\n",
    "        if topic_idx < num_topics:\n",
    "            print(\"{:11}\".format(\"Topic %d:\" %(topic_idx)), end='')\n",
    "            print(\", \".join(['{:04.3f}*'.format(topic[i])+feature_names[i] \\\n",
    "                             for i in topic.argsort()[:-no_top_words-1:-1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Utility function to display the topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "def topic_to_text(topic):\n",
    "    str = ''\n",
    "    for elem in list(topic)[1].split('+')[:6]:\n",
    "        str+= '/'.join(re.findall(r'\"([^\"]*)\"', elem))+' '\n",
    "    topics.append(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sort_list(lst):\n",
    "    return list(sorted(set(reduce(operator.concat, lst).split())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets display the top 4 topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_list = []\n",
    "no_topics = num_topics\n",
    "no_top_words = 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top topics + words for POSITIVE reviews\n",
      "---------------------------------------\n",
      "Topic 0:   11.950*like, 1.565*feel, 0.400*look, 0.326*not_feel\n",
      "Topic 1:   2.951*pizza, 0.381*papa_del, 0.363*good_pizza, 0.345*slice\n",
      "Topic 2:   4.251*love, 0.134*atmosphere, 0.068*indian_food, 0.065*salmon\n",
      "Topic 3:   3.772*good, 0.187*mexican_food, 0.148*chinese_food, 0.130*thai_food\n",
      "Topic 4:   3.814*food, 0.397*price, 0.286*quality, 0.116*atmosphere\n",
      "Topic 5:   3.454*great, 0.071*server, 0.061*small, 0.046*slider\n",
      "Topic 6:   2.729*sushi, 1.798*roll, 0.592*good_sushi, 0.456*ramen\n",
      "Topic 7:   5.091*chicken, 0.353*wing, 0.198*waffle, 0.195*grill\n",
      "Topic 8:   7.601*time, 0.474*visit, 0.369*year, 0.361*couple\n",
      "Topic 9:   6.381*service, 0.555*fast, 0.483*quick, 0.402*customer\n",
      "Topic 10:  4.832*taco, 1.602*mexican_food, 1.120*fish, 1.032*salsa\n",
      "Topic 11:  3.960*sauce, 1.760*burn, 1.338*meat, 1.057*pulled_pork\n",
      "Topic 12:  4.081*amazing, 0.233*absolutely, 0.149*burn, 0.133*simply\n",
      "Topic 13:  4.109*dollar, 0.721*price, 0.391*meal, 0.327*special\n",
      "Topic 14:  3.236*delicious, 0.382*absolutely, 0.155*flavor, 0.124*fresh\n",
      "Topic 15:  4.333*menu, 0.874*item, 0.527*option, 0.222*change\n",
      "Topic 16:  4.133*staff, 0.281*helpful, 0.219*attentive, 0.202*highly_recommend\n",
      "Topic 17:  3.132*burger, 0.457*good_burger, 0.223*bacon, 0.186*meathead\n",
      "Topic 18:  4.325*pretty, 0.366*much, 0.288*tasty, 0.265*overall\n",
      "Topic 19:  3.586*town, 0.217*good_sushi, 0.187*stop, 0.185*college\n",
      "Topic 20:  6.131*friendly, 0.706*fast, 0.639*attentive, 0.569*stop\n",
      "Topic 21:  4.257*champaign, 0.350*stop, 0.311*visit, 0.304*chicago\n",
      "Topic 22:  5.178*great_food, 0.256*great_atmosphere, 0.239*wonderful, 0.234*great_price\n",
      "Topic 23:  4.731*wait, 1.385*worth, 0.767*seat, 0.715*long\n",
      "Topic 24:  3.887*sandwich, 0.614*cheese, 0.303*bread, 0.200*pulled_pork\n",
      "Topic 25:  4.444*salad, 0.920*slider, 0.516*salmon, 0.276*pasta\n",
      "Topic 26:  4.889*because, 3.388*star, 0.502*review, 0.438*reason\n",
      "Topic 27:  2.971*definitely, 0.236*worth, 0.129*worth_wait, 0.093*worth_try\n",
      "Topic 28:  4.622*lunch, 1.897*dinner, 0.708*special, 0.351*buffet\n",
      "Topic 29:  3.544*awesome, 0.168*server, 0.153*dessert, 0.133*cheese\n",
      "Topic 30:  3.115*breakfast, 1.499*pancake, 0.983*brunch, 0.724*cheese\n",
      "Topic 31:  4.030*dish, 0.622*thai, 0.578*chinese_food, 0.520*pad_thai\n",
      "Topic 32:  3.788*good_food, 0.451*good_service, 0.413*price, 0.353*atmosphere\n",
      "Topic 33:  2.882*nice, 0.214*inside, 0.191*outside, 0.167*clean\n",
      "Topic 34:  3.946*favorite, 0.298*campus, 0.269*fantastic, 0.223*probably\n",
      "Topic 35:  3.858*more, 0.626*much, 0.485*wish, 0.408*often\n",
      "Topic 36:  3.371*drink, 0.620*night, 0.230*bartender, 0.164*dessert\n",
      "Topic 37:  2.764*beer, 0.372*selection, 0.287*wine, 0.194*appetizer\n",
      "Topic 38:  2.945*super, 0.623*tasty, 0.376*fresh, 0.330*fast\n",
      "Topic 39:  2.950*steak, 0.849*cook, 0.328*ribeye, 0.215*grill\n",
      "Topic 40:  3.167*coffee, 0.449*work, 0.416*espresso, 0.402*study\n",
      "Topic 41:  2.915*recommend, 0.093*salmon, 0.091*sure, 0.088*look\n",
      "Topic 42:  2.237*enjoy, 0.458*meal, 0.420*atmosphere, 0.279*experience\n",
      "Topic 43:  2.545*area, 0.652*champaign_urbana, 0.265*visit, 0.149*find\n",
      "Topic 44:  2.602*great_service, 0.166*thank, 0.122*fantastic, 0.114*good_sushi\n",
      "Topic 45:  1.406*want, 1.404*think, 1.275*table, 1.096*people\n",
      "Topic 46:  2.501*taste, 0.574*fresh, 0.162*meat, 0.138*flavor\n",
      "Topic 47:  2.479*excellent, 0.168*atmosphere, 0.158*wife, 0.100*perfect\n",
      "Topic 48:  1.322*soup, 0.983*beef, 0.899*noodle, 0.663*rice\n",
      "Topic 49:  2.609*friend, 0.893*visit, 0.428*family, 0.184*group\n",
      "\n",
      "Top topics + words for NEGATIVE reviews\n",
      "---------------------------------------\n",
      "Topic 0:   8.578*like, 2.388*look, 0.717*smell, 0.520*flavor\n",
      "Topic 1:   3.181*table, 0.397*leave, 0.273*water, 0.225*plate\n",
      "Topic 2:   2.619*pizza, 0.320*crust, 0.229*slice, 0.163*papa_del\n",
      "Topic 3:   2.632*food, 0.162*poisoning, 0.085*atmosphere, 0.082*average\n",
      "Topic 4:   3.243*dollar, 0.555*charge, 0.279*cost, 0.169*spend\n",
      "Topic 5:   4.371*wait, 0.327*minute, 0.218*long, 0.200*staff\n",
      "Topic 6:   2.201*sushi, 1.285*roll, 0.416*fish, 0.212*piece\n",
      "Topic 7:   3.081*steak, 1.018*cook, 0.481*ribeye, 0.347*medium_rare\n",
      "Topic 8:   3.320*sandwich, 0.270*fish, 0.206*bread, 0.192*soggy\n",
      "Topic 9:   2.922*good, 0.198*average, 0.197*bland, 0.167*indian_food\n",
      "Topic 10:  2.760*chicken, 0.177*wing, 0.153*chewy, 0.138*buffalo\n",
      "Topic 11:  3.211*time, 0.176*second_time, 0.154*waste, 0.152*year\n",
      "Topic 12:  3.314*service, 0.636*slow, 0.347*customer, 0.327*terrible\n",
      "Topic 13:  2.294*drink, 0.175*bartender, 0.154*horrible, 0.139*minute\n",
      "Topic 14:  2.427*burger, 0.235*patty, 0.138*bland, 0.136*meathead\n",
      "Topic 15:  3.072*dish, 0.255*flavor, 0.241*beef, 0.207*fish\n",
      "Topic 16:  2.914*star, 0.562*review, 0.265*not_know, 0.257*reason\n",
      "Topic 17:  3.012*price, 0.468*quality, 0.336*not_great, 0.312*expect\n",
      "Topic 18:  3.816*waitress, 0.434*check, 0.420*bring, 0.243*bad_service\n",
      "Topic 19:  3.333*salad, 0.349*dress, 0.275*dinner, 0.251*ribeye\n",
      "Topic 20:  4.710*hour, 0.819*delivery, 0.744*half, 0.357*deliver\n",
      "Topic 21:  2.436*sauce, 0.515*meat, 0.224*cook, 0.219*flavor\n",
      "Topic 22:  2.527*serve, 2.286*staff, 0.277*customer, 0.195*kitchen\n",
      "Topic 23:  3.676*server, 0.610*bring, 0.579*meal, 0.532*check\n",
      "Topic 24:  5.646*because, 0.387*reason, 0.346*decide, 0.323*away\n",
      "Topic 25:  2.493*manager, 1.098*customer, 0.632*owner, 0.477*leave\n",
      "Topic 26:  2.497*waiter, 0.469*meal, 0.305*bring, 0.304*water\n",
      "Topic 27:  3.159*people, 0.652*work, 0.345*know, 0.246*group\n",
      "Topic 28:  2.841*coffee, 0.422*breakfast, 0.244*espresso_royale, 0.244*burn\n",
      "Topic 29:  2.711*rude, 1.024*extremely, 0.298*horrible, 0.297*not_care\n",
      "Topic 30:  2.633*taste, 0.144*definitely, 0.137*thai, 0.105*not_fresh\n",
      "Topic 31:  3.590*friend, 0.262*find, 0.244*not_recommend, 0.240*group\n",
      "Topic 32:  2.856*rice, 0.485*bean, 0.370*bowl, 0.248*veggie\n",
      "Topic 33:  1.935*taco, 0.780*chip, 0.710*fish, 0.602*burrito\n",
      "Topic 34:  3.098*pretty, 0.229*decent, 0.223*find, 0.181*probably\n",
      "Topic 35:  2.655*menu, 0.397*item, 0.166*change, 0.163*kind\n",
      "Topic 36:  2.524*tell, 0.160*phone, 0.146*know, 0.127*ready\n",
      "Topic 37:  2.004*not_good, 0.426*experience, 0.184*beef, 0.133*not_sure\n",
      "Topic 38:  2.141*cheese, 1.121*bread, 0.373*onion, 0.321*grill\n",
      "Topic 39:  2.313*think, 0.231*not_think, 0.221*find, 0.157*flavor\n",
      "Topic 40:  2.149*want, 0.177*not_want, 0.146*look, 0.115*buffet\n",
      "Topic 41:  1.946*seat, 1.304*reservation, 0.761*arrive, 0.598*party\n",
      "Topic 42:  2.353*lunch, 0.364*buffet, 0.250*today, 0.215*special\n",
      "Topic 43:  1.932*walk, 1.381*never, 0.431*street, 0.310*stand\n",
      "Topic 44:  1.704*soup, 0.318*bowl, 0.264*noodle, 0.255*not_sure\n",
      "Topic 45:  1.732*not_worth, 0.529*money, 0.433*definitely, 0.155*super\n",
      "Topic 46:  1.611*more, 0.666*much, 0.179*location, 0.177*expect\n",
      "Topic 47:  1.947*beer, 0.289*bartender, 0.144*bacon, 0.120*slow_service\n",
      "Topic 48:  1.821*cold, 0.217*not_recommend, 0.162*disappointed, 0.154*love\n",
      "Topic 49:  1.988*feel, 1.005*great, 0.674*wing, 0.244*portion\n",
      "\n",
      "Top topics + words for ALL reviews\n",
      "---------------------------------------\n",
      "Topic 0:   10.244*because, 1.858*want, 1.455*know, 1.349*people\n",
      "Topic 1:   3.169*pizza, 0.345*crust, 0.325*slice, 0.302*papa_del\n",
      "Topic 2:   5.824*good, 0.160*thai_food, 0.154*chinese_food, 0.122*indian_food\n",
      "Topic 3:   4.060*love, 0.131*atmosphere, 0.108*favorite, 0.082*people\n",
      "Topic 4:   3.460*food, 0.143*quality, 0.098*not_good, 0.089*not_great\n",
      "Topic 5:   3.395*sushi, 0.373*good_sushi, 0.330*fish, 0.297*ramen\n",
      "Topic 6:   3.100*table, 1.262*server, 1.124*tell, 1.001*leave\n",
      "Topic 7:   5.488*great, 0.134*think, 0.114*atmosphere, 0.103*slider\n",
      "Topic 8:   4.045*chicken, 0.300*wing, 0.177*naan, 0.163*buffalo\n",
      "Topic 9:   4.657*like, 0.834*feel, 0.620*look, 0.246*think\n",
      "Topic 10:  4.870*sandwich, 0.377*bread, 0.165*fish, 0.165*turkey\n",
      "Topic 11:  5.526*delicious, 0.613*absolutely, 0.326*fresh, 0.209*perfect\n",
      "Topic 12:  3.209*time, 0.210*visit, 0.175*year, 0.147*couple\n",
      "Topic 13:  4.053*taco, 0.988*fish, 0.779*chip, 0.765*salsa\n",
      "Topic 14:  3.563*burger, 0.370*good_burger, 0.242*bacon, 0.229*meathead\n",
      "Topic 15:  4.322*steak, 1.305*cook, 0.551*ribeye, 0.305*grill\n",
      "Topic 16:  3.543*sauce, 0.253*flavor, 0.145*sweet, 0.141*bowl\n",
      "Topic 17:  3.692*pretty, 0.309*much, 0.287*decent, 0.270*overall\n",
      "Topic 18:  5.086*service, 0.494*excellent, 0.466*customer, 0.432*slow\n",
      "Topic 19:  4.436*wait, 0.836*hour, 0.505*seat, 0.480*long\n",
      "Topic 20:  4.496*drink, 0.420*night, 0.351*bartender, 0.189*dinner\n",
      "Topic 21:  4.424*staff, 3.324*friendly, 0.530*helpful, 0.523*attentive\n",
      "Topic 22:  3.846*menu, 0.713*item, 0.360*option, 0.242*change\n",
      "Topic 23:  3.292*amazing, 0.186*absolutely, 0.113*highly_recommend, 0.106*worth\n",
      "Topic 24:  4.273*town, 0.341*college, 0.251*good_sushi, 0.232*visit\n",
      "Topic 25:  3.553*lunch, 1.190*dinner, 0.574*special, 0.341*buffet\n",
      "Topic 26:  3.641*dish, 0.385*thai, 0.368*pad_thai, 0.286*curry\n",
      "Topic 27:  3.476*champaign, 0.542*visit, 0.420*location, 0.313*chicago\n",
      "Topic 28:  4.805*salad, 0.730*slider, 0.422*salmon, 0.348*pasta\n",
      "Topic 29:  3.728*coffee, 0.492*espresso, 0.408*work, 0.402*study\n",
      "Topic 30:  3.545*dollar, 0.299*charge, 0.277*cost, 0.199*meal\n",
      "Topic 31:  4.319*definitely, 0.394*worth, 0.165*worth_wait, 0.130*worth_visit\n",
      "Topic 32:  4.310*taste, 0.236*bland, 0.207*not_good, 0.193*flavor\n",
      "Topic 33:  4.437*beer, 0.460*selection, 0.328*wine, 0.313*bartender\n",
      "Topic 34:  3.108*friend, 0.469*visit, 0.398*recommend, 0.183*group\n",
      "Topic 35:  3.361*nice, 0.229*atmosphere, 0.229*clean, 0.225*inside\n",
      "Topic 36:  3.238*great_food, 1.768*great_service, 0.186*great_atmosphere, 0.170*night\n",
      "Topic 37:  2.903*area, 1.841*authentic, 1.682*mexican_food, 1.046*favorite\n",
      "Topic 38:  3.138*more, 0.657*much, 0.373*wish, 0.268*often\n",
      "Topic 39:  2.618*cheese, 0.528*grill, 0.416*bread, 0.304*plate\n",
      "Topic 40:  2.776*enjoy, 1.815*meal, 0.592*atmosphere, 0.283*experience\n",
      "Topic 41:  2.734*price, 0.553*quality, 0.224*not_worth, 0.212*portion\n",
      "Topic 42:  2.875*star, 0.316*review, 0.198*reason, 0.121*think\n",
      "Topic 43:  1.785*soup, 1.276*rice, 1.099*beef, 1.024*noodle\n",
      "Topic 44:  3.131*awesome, 0.327*recommend, 0.167*dessert, 0.119*highly_recommend\n",
      "Topic 45:  1.825*burn, 1.339*meat, 0.975*black_dog, 0.813*pulled_pork\n",
      "Topic 46:  2.282*breakfast, 1.052*pancake, 0.595*brunch, 0.469*bacon\n",
      "Topic 47:  2.652*good_food, 0.300*good_service, 0.160*atmosphere, 0.145*want\n",
      "Topic 48:  2.747*roll, 0.330*salmon, 0.191*tuna, 0.148*fish\n",
      "Topic 49:  2.586*super, 0.486*tasty, 0.387*fast, 0.302*fresh\n"
     ]
    }
   ],
   "source": [
    "print('Top topics + words for POSITIVE reviews')\n",
    "print('-'*39)\n",
    "display_topics(nmf_pos, tfidf_pos.get_feature_names(), no_topics, no_top_words)\n",
    "\n",
    "print('\\nTop topics + words for NEGATIVE reviews')\n",
    "print('-'*39)\n",
    "display_topics(nmf_neg, tfidf_neg.get_feature_names(), no_topics, no_top_words)\n",
    "\n",
    "print('\\nTop topics + words for ALL reviews')\n",
    "print('-'*39)\n",
    "display_topics(nmf_all, tfidf_all.get_feature_names(), no_topics, no_top_words)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gather all topics together. This will be used to map the topic top-n words with a theme"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_list.extend(get_topic_list(nmf_all, all_feature_names, no_topics, no_top_words))\n",
    "topic_list.extend(get_topic_list(nmf_pos, pos_feature_names, no_topics, no_top_words))\n",
    "topic_list.extend(get_topic_list(nmf_neg, neg_feature_names, no_topics, no_top_words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write the topic map in a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('config/topic_map.csv','w+') as f:\n",
    "#    f.write('\\n'.join(topic_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic 0:   10.244*because, 1.858*want, 1.455*know, 1.349*people\n",
      "Topic 1:   3.169*pizza, 0.345*crust, 0.325*slice, 0.302*papa_del\n",
      "Topic 2:   5.824*good, 0.160*thai_food, 0.154*chinese_food, 0.122*indian_food\n",
      "Topic 3:   4.060*love, 0.131*atmosphere, 0.108*favorite, 0.082*people\n",
      "Topic 4:   3.460*food, 0.143*quality, 0.098*not_good, 0.089*not_great\n",
      "Topic 5:   3.395*sushi, 0.373*good_sushi, 0.330*fish, 0.297*ramen\n",
      "Topic 6:   3.100*table, 1.262*server, 1.124*tell, 1.001*leave\n",
      "Topic 7:   5.488*great, 0.134*think, 0.114*atmosphere, 0.103*slider\n",
      "Topic 8:   4.045*chicken, 0.300*wing, 0.177*naan, 0.163*buffalo\n",
      "Topic 9:   4.657*like, 0.834*feel, 0.620*look, 0.246*think\n",
      "Topic 10:  4.870*sandwich, 0.377*bread, 0.165*fish, 0.165*turkey\n",
      "Topic 11:  5.526*delicious, 0.613*absolutely, 0.326*fresh, 0.209*perfect\n",
      "Topic 12:  3.209*time, 0.210*visit, 0.175*year, 0.147*couple\n",
      "Topic 13:  4.053*taco, 0.988*fish, 0.779*chip, 0.765*salsa\n",
      "Topic 14:  3.563*burger, 0.370*good_burger, 0.242*bacon, 0.229*meathead\n",
      "Topic 15:  4.322*steak, 1.305*cook, 0.551*ribeye, 0.305*grill\n",
      "Topic 16:  3.543*sauce, 0.253*flavor, 0.145*sweet, 0.141*bowl\n",
      "Topic 17:  3.692*pretty, 0.309*much, 0.287*decent, 0.270*overall\n",
      "Topic 18:  5.086*service, 0.494*excellent, 0.466*customer, 0.432*slow\n",
      "Topic 19:  4.436*wait, 0.836*hour, 0.505*seat, 0.480*long\n",
      "Topic 20:  4.496*drink, 0.420*night, 0.351*bartender, 0.189*dinner\n",
      "Topic 21:  4.424*staff, 3.324*friendly, 0.530*helpful, 0.523*attentive\n",
      "Topic 22:  3.846*menu, 0.713*item, 0.360*option, 0.242*change\n",
      "Topic 23:  3.292*amazing, 0.186*absolutely, 0.113*highly_recommend, 0.106*worth\n",
      "Topic 24:  4.273*town, 0.341*college, 0.251*good_sushi, 0.232*visit\n",
      "Topic 25:  3.553*lunch, 1.190*dinner, 0.574*special, 0.341*buffet\n",
      "Topic 26:  3.641*dish, 0.385*thai, 0.368*pad_thai, 0.286*curry\n",
      "Topic 27:  3.476*champaign, 0.542*visit, 0.420*location, 0.313*chicago\n",
      "Topic 28:  4.805*salad, 0.730*slider, 0.422*salmon, 0.348*pasta\n",
      "Topic 29:  3.728*coffee, 0.492*espresso, 0.408*work, 0.402*study\n",
      "Topic 30:  3.545*dollar, 0.299*charge, 0.277*cost, 0.199*meal\n",
      "Topic 31:  4.319*definitely, 0.394*worth, 0.165*worth_wait, 0.130*worth_visit\n",
      "Topic 32:  4.310*taste, 0.236*bland, 0.207*not_good, 0.193*flavor\n",
      "Topic 33:  4.437*beer, 0.460*selection, 0.328*wine, 0.313*bartender\n",
      "Topic 34:  3.108*friend, 0.469*visit, 0.398*recommend, 0.183*group\n",
      "Topic 35:  3.361*nice, 0.229*atmosphere, 0.229*clean, 0.225*inside\n",
      "Topic 36:  3.238*great_food, 1.768*great_service, 0.186*great_atmosphere, 0.170*night\n",
      "Topic 37:  2.903*area, 1.841*authentic, 1.682*mexican_food, 1.046*favorite\n",
      "Topic 38:  3.138*more, 0.657*much, 0.373*wish, 0.268*often\n",
      "Topic 39:  2.618*cheese, 0.528*grill, 0.416*bread, 0.304*plate\n",
      "Topic 40:  2.776*enjoy, 1.815*meal, 0.592*atmosphere, 0.283*experience\n",
      "Topic 41:  2.734*price, 0.553*quality, 0.224*not_worth, 0.212*portion\n",
      "Topic 42:  2.875*star, 0.316*review, 0.198*reason, 0.121*think\n",
      "Topic 43:  1.785*soup, 1.276*rice, 1.099*beef, 1.024*noodle\n",
      "Topic 44:  3.131*awesome, 0.327*recommend, 0.167*dessert, 0.119*highly_recommend\n",
      "Topic 45:  1.825*burn, 1.339*meat, 0.975*black_dog, 0.813*pulled_pork\n",
      "Topic 46:  2.282*breakfast, 1.052*pancake, 0.595*brunch, 0.469*bacon\n",
      "Topic 47:  2.652*good_food, 0.300*good_service, 0.160*atmosphere, 0.145*want\n",
      "Topic 48:  2.747*roll, 0.330*salmon, 0.191*tuna, 0.148*fish\n",
      "Topic 49:  2.586*super, 0.486*tasty, 0.387*fast, 0.302*fresh\n"
     ]
    }
   ],
   "source": [
    "display_topics(nmf_all, all_feature_names, no_topics, no_top_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'because/know/people/want',\n",
       " 1: 'crust/papa_del/pizza/slice',\n",
       " 2: 'chinese_food/good/indian_food/thai_food',\n",
       " 3: 'atmosphere/favorite/love/people',\n",
       " 4: 'food/not_good/not_great/quality',\n",
       " 5: 'fish/good_sushi/ramen/sushi',\n",
       " 6: 'leave/server/table/tell',\n",
       " 7: 'atmosphere/great/slider/think',\n",
       " 8: 'buffalo/chicken/naan/wing',\n",
       " 9: 'feel/like/look/think',\n",
       " 10: 'bread/fish/sandwich/turkey',\n",
       " 11: 'absolutely/delicious/fresh/perfect',\n",
       " 12: 'couple/time/visit/year',\n",
       " 13: 'chip/fish/salsa/taco',\n",
       " 14: 'bacon/burger/good_burger/meathead',\n",
       " 15: 'cook/grill/ribeye/steak',\n",
       " 16: 'bowl/flavor/sauce/sweet',\n",
       " 17: 'decent/much/overall/pretty',\n",
       " 18: 'customer/excellent/service/slow',\n",
       " 19: 'hour/long/seat/wait',\n",
       " 20: 'bartender/dinner/drink/night',\n",
       " 21: 'attentive/friendly/helpful/staff',\n",
       " 22: 'change/item/menu/option',\n",
       " 23: 'absolutely/amazing/highly_recommend/worth',\n",
       " 24: 'college/good_sushi/town/visit',\n",
       " 25: 'buffet/dinner/lunch/special',\n",
       " 26: 'curry/dish/pad_thai/thai',\n",
       " 27: 'champaign/chicago/location/visit',\n",
       " 28: 'pasta/salad/salmon/slider',\n",
       " 29: 'coffee/espresso/study/work',\n",
       " 30: 'charge/cost/dollar/meal',\n",
       " 31: 'definitely/worth/worth_visit/worth_wait',\n",
       " 32: 'bland/flavor/not_good/taste',\n",
       " 33: 'bartender/beer/selection/wine',\n",
       " 34: 'friend/group/recommend/visit',\n",
       " 35: 'atmosphere/clean/inside/nice',\n",
       " 36: 'great_atmosphere/great_food/great_service/night',\n",
       " 37: 'area/authentic/favorite/mexican_food',\n",
       " 38: 'more/much/often/wish',\n",
       " 39: 'bread/cheese/grill/plate',\n",
       " 40: 'atmosphere/enjoy/experience/meal',\n",
       " 41: 'not_worth/portion/price/quality',\n",
       " 42: 'reason/review/star/think',\n",
       " 43: 'beef/noodle/rice/soup',\n",
       " 44: 'awesome/dessert/highly_recommend/recommend',\n",
       " 45: 'black_dog/burn/meat/pulled_pork',\n",
       " 46: 'bacon/breakfast/brunch/pancake',\n",
       " 47: 'atmosphere/good_food/good_service/want',\n",
       " 48: 'fish/roll/salmon/tuna',\n",
       " 49: 'fast/fresh/super/tasty'}"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_topic_map(nmf_all, all_feature_names, no_topics, no_top_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save all topics top-n words in a file for later use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('config/all_topics.txt', 'w') as f:\n",
    "#    f.write('\\n'.join([str(key)+\";\"+str(value) for key, value in get_topic_map(nmf_all, all_feature_names, no_topics, no_top_words).items()]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('config/neg_topics.txt', 'w') as f:\n",
    "#    f.write('\\n'.join([str(key)+\";\"+str(value) for key, value in get_topic_map(nmf_neg, neg_feature_names, no_topics, no_top_words).items()]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('config/pos_topics.txt', 'w') as f:\n",
    "#    f.write('\\n'.join([str(key)+\";\"+str(value) for key, value in get_topic_map(nmf_pos, pos_feature_names, no_topics, no_top_words).items()]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "#with open('config/topics.csv', 'w+') as f:\n",
    "#    f.write('\\n'.join(topic_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "#END"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load everything related to topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 253 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Read the Pre-Processed data\n",
    "#restaurant_file = pd.read_csv('processed_data/cleaned_restaurants.csv')\n",
    "#restaurant_file = restaurant_file.drop(labels=\"Unnamed: 0\", axis=1)\n",
    "restaurant_file = pd.read_csv('processed_data/'+state_filter+'_cleaned_restaurants.csv')\n",
    "restaurant_file = restaurant_file.drop(labels=\"Unnamed: 0\", axis=1)\n",
    "review_file     = pd.read_csv('processed_data/'+state_filter+'_cleaned_reviews.csv')\n",
    "review_file     = review_file.drop(labels=\"Unnamed: 0\", axis=1) \n",
    "\n",
    "review_file['topic_text'] = [u''+str(txt).replace('.','').replace('\\\\n','') for txt in review_file.topic_text]\n",
    "review_file['sentiment_text'] = [u''+str(txt).replace('.','').replace('\\\\n','') for txt in review_file.sentiment_text]\n",
    "review_file['text'] = [u''+str(txt) for txt in review_file.text]\n",
    "review_file['review_length'] = review_file.text.map(len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenizer(x):\n",
    "    return ( w for w in str(x).split() if len(w) >3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "term_vector = tfidf_all\n",
    "neg_vectors = tfidf_neg\n",
    "pos_vectors = tfidf_pos\n",
    "all_vectors = tfidf_all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we read all the saved topic models and vectors based on negative, positine and combined revies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('pickles/neg_topic_features_'+str(max_features)+'.pk', 'rb') as fin:\n",
    "    neg_feature_names = pickle.load(fin)\n",
    "with open('pickles/pos_topic_features_'+str(max_features)+'.pk', 'rb') as fin:\n",
    "    pos_feature_names = pickle.load(fin)\n",
    "with open('pickles/all_topic_features_'+str(max_features)+'.pk', 'rb') as fin:\n",
    "    all_feature_names = pickle.load(fin)\n",
    "\n",
    "with open('pickles/topic_term_vector_all.pk', 'rb') as fin:\n",
    "    term_vector = pickle.load(fin)\n",
    "with open('pickles/topic_term_vector_neg.pk', 'rb') as fin:\n",
    "    neg_vectors = pickle.load(fin)\n",
    "with open('pickles/topic_term_vector_pos.pk', 'rb') as fin:\n",
    "    pos_vectors = pickle.load(fin)\n",
    "with open('pickles/topic_term_vector_all.pk', 'rb') as fin:\n",
    "    all_vectors = pickle.load(fin)\n",
    "    \n",
    "with open('pickles/all_nmf_model.pk', 'rb') as fin:\n",
    "    nmf_all = pickle.load(fin)\n",
    "with open('pickles/pos_nmf_model.pk', 'rb') as fin:\n",
    "    nmf_pos = pickle.load(fin)\n",
    "with open('pickles/neg_nmf_model.pk', 'rb') as fin:\n",
    "    nmf_neg = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also retreive the topic map we created earlier. This mapping is between top-n (4) topic words and a short description of the topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_desc_map = {}\n",
    "with open('config/topic_map.txt', 'r') as f:\n",
    "    topic_desc_map = {line.split(';')[0]: line.split(';')[1].replace('\\n','') for line in f.readlines()}\n",
    "\n",
    "neg_topic_map = {}\n",
    "pos_topic_map = {}\n",
    "all_topic_map = {}\n",
    "\n",
    "with open('config/pos_topics.txt', 'r') as f:\n",
    "    pos_topic_map = {line.split(';')[0]: line.split(';')[1].replace('\\n','') for line in f.readlines()}\n",
    "with open('config/neg_topics.txt', 'r') as f:\n",
    "    neg_topic_map = {line.split(';')[0]: line.split(';')[1].replace('\\n','') for line in f.readlines()}\n",
    "with open('config/all_topics.txt', 'r') as f:\n",
    "    all_topic_map = {line.split(';')[0]: line.split(';')[1].replace('\\n','') for line in f.readlines()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'0': 'reservation/seat/table/waiter',\n",
      " '1': 'atmosphere/favorite/husband/love',\n",
      " '10': 'burrito/chip/fish/salsa',\n",
      " '11': 'feel/like/more/much',\n",
      " '12': 'chicken/rice/waffle/wing',\n",
      " '13': 'couple/second-time/time/visit',\n",
      " '14': 'customer/great-customer/service/slow',\n",
      " '15': 'bread/pulled-pork/sandwich/turkey',\n",
      " '16': 'attentive/friendly/helpful/staff',\n",
      " '17': 'absolutely/amazing/highly-recommend/not-wait',\n",
      " '18': 'atmosphere/enjoy/experience/meal',\n",
      " '19': 'absolutely/definitely/delicious/highly-recommend',\n",
      " '2': 'atmosphere/food/good-service/quality',\n",
      " '20': 'great-atmosphere/great-food/great-service/highly-recommend',\n",
      " '21': 'leave/manager/people/tell',\n",
      " '22': 'dinner/lunch/special/today',\n",
      " '23': 'caesar/dress/pasta/salad',\n",
      " '24': 'dish/noodle/rice/soup',\n",
      " '25': 'hour/line/minute/wait',\n",
      " '26': 'breakfast/brunch/coffee/pancake',\n",
      " '27': 'item/menu/offer/option',\n",
      " '28': 'bartender/drink/night/refill',\n",
      " '29': 'beer/selection/wine/wing',\n",
      " '3': 'crust/good-pizza/pizza/slice',\n",
      " '30': 'attentive/bring/check/server',\n",
      " '31': 'decent/much/not-bad/pretty',\n",
      " '32': 'awesome/definitely/highly-recommend/thank',\n",
      " '33': 'portion/price/quality/reasonable',\n",
      " '34': 'appetizer/happy-hour/special/wing',\n",
      " '35': 'clean/inside/nice/outside',\n",
      " '36': 'area/drive/location/visit',\n",
      " '37': 'cook/dinner/medium-rare/steak',\n",
      " '38': 'bacon/cheese/grill/mac-n',\n",
      " '39': 'fast/friendly-staff/super/yummy',\n",
      " '4': 'good/mexican-food/not-great/town',\n",
      " '40': 'because/reason/review/star',\n",
      " '41': 'friendly-staff/good-food/good-service/recommend',\n",
      " '42': 'lobster/roll/salmon/shrimp',\n",
      " '43': 'hotel/night/room/stay',\n",
      " '44': 'bland/flavor/not-good/taste',\n",
      " '45': 'family/friend/night/recommend',\n",
      " '46': 'buffet/dessert/line/vegas',\n",
      " '47': 'dinner/excellent/highly-recommend/wife',\n",
      " '48': 'flavor/meat/sauce/wing',\n",
      " '49': 'clean/fast/fresh/tasty',\n",
      " '5': 'bacon/burger/good-burger/shake',\n",
      " '6': 'atmosphere/definitely/great/highly-recommend',\n",
      " '7': 'chef/fish/sashimi/sushi',\n",
      " '8': 'bill/charge/dollar/spend',\n",
      " '9': 'shrimp/street/taco/tuesday'}\n"
     ]
    }
   ],
   "source": [
    "pprint(all_topic_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These topics are mapped with title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'absolute/favorite/mexican/mexican-food': 'Mexican Food',\n",
      " 'absolutely/amazing/definitely/not-wait': 'Dining Experience',\n",
      " 'absolutely/amazing/highly-recommend/not-wait': 'Good Service',\n",
      " 'absolutely/definitely/delicious/highly-recommend': 'Tasty food',\n",
      " 'absolutely/definitely/delicious/husband': 'Dining experience',\n",
      " 'amazing-food/delicious-food/great-service/thank': 'Good Food, Good service',\n",
      " 'amazing-food/excellent-service/fantastic/highly-recommend': 'Amazing Food & '\n",
      "                                                              'Service',\n",
      " 'appetizer/definitely/excellent/wife': 'Food & Service',\n",
      " 'appetizer/happy-hour/special/wing': 'Happy Hour / Specials',\n",
      " 'area/drive/location/visit': 'Location & Area',\n",
      " 'area/find/room/stay': 'Dining area & Location',\n",
      " 'arrive/later/minute/ready': 'Reception & Wait',\n",
      " 'atmosphere/bartender/coffee/drink': 'Bar / Breakfast Experience',\n",
      " 'atmosphere/definitely/friendly-service/great': 'Atmosphere',\n",
      " 'atmosphere/definitely/great/highly-recommend': 'Atmosphere & Food',\n",
      " 'atmosphere/enjoy/experience/meal': 'Atmosphere & Food',\n",
      " 'atmosphere/favorite/husband/love': 'Atmosphere',\n",
      " 'atmosphere/food/good-service/quality': 'Atmosphere & Food',\n",
      " 'atmosphere/friendly-staff/great-atmosphere/great-food': 'Atmosphere & Staff',\n",
      " 'atmosphere/husband/love/yummy': 'Food & Environment',\n",
      " 'attentive/bring/check/server': 'Staff & Service',\n",
      " 'attentive/experience/server/thank': 'Staff & Management',\n",
      " 'attentive/friendly/helpful/staff': 'Reception & Staff',\n",
      " 'average/good/mediocre/not-taste': 'Average dining Experience',\n",
      " 'away/door/stand/walk': 'Dining Experience',\n",
      " 'awesome/bartender/definitely/not-wait': 'Staff & Service',\n",
      " 'awesome/definitely/highly-recommend/thank': 'Good dining experience',\n",
      " 'bacon/breakfast/coffee/pancake': 'Breakfast / Coffee',\n",
      " 'bacon/burger/cook/patty': 'Burgers & Fast food',\n",
      " 'bacon/burger/good-burger/shake': 'burgers / Shakes',\n",
      " 'bacon/cheese/grill/mac-n': 'Grilled Food',\n",
      " 'bartender/beer/happy-hour/selection': 'Beer/Drink Selection',\n",
      " 'bartender/drink/happy-hour/refill': 'Drinks in general',\n",
      " 'bartender/drink/night/refill': 'Bar & Drinks',\n",
      " 'bean/burrito/rice/shrimp': 'Food in general',\n",
      " 'because/not-want/reason/star': 'Review & Stars',\n",
      " 'because/reason/review/star': 'Review & Star Rating',\n",
      " 'beef/bowl/noodle/soup': 'Soups & Noodles',\n",
      " 'beef/noodle/rice/soup': 'Soups & Noodles',\n",
      " 'beer/selection/wine/wing': 'Food veriety',\n",
      " 'bill/bring/check/server': 'Bill & Service',\n",
      " 'bill/bring/check/waiter': 'Waiter & Bill',\n",
      " 'bill/bring/check/waitress': 'Waiter/Waitress/Bill',\n",
      " 'bill/charge/dollar/spend': 'Bill & Money',\n",
      " 'birthday/friend/group/night': 'Birthday / Group Dining',\n",
      " 'bland/flavor/not-good/taste': 'Food flavors & Taste',\n",
      " 'bland/horrible/money/taste': 'Bad taste / Value',\n",
      " 'bland/much/nice/pretty': 'Food Misc',\n",
      " 'bread/cold/plate/serve': 'Bread / Food quality',\n",
      " 'bread/lunch/sandwich/turkey': 'Sandwitches',\n",
      " 'bread/meat/sandwich/turkey': 'Sandwitches & Meat',\n",
      " 'bread/pulled-pork/sandwich/turkey': 'Sandwitches / Meat',\n",
      " 'breakfast/brunch/coffee/pancake': 'Breakfast',\n",
      " 'breakfast/burrito/coffee/pancake': 'Breakfast & Service',\n",
      " 'breast/chicken/piece/waffle': 'Chicken / Waffle',\n",
      " 'bring/friend/meet/recommend': 'Dining Experience',\n",
      " 'bring/glass/refill/water': 'Drinks',\n",
      " 'buffalo/cold/sauce/wing': 'Buffalo Wings / Sauces',\n",
      " 'buffet/dessert/line/vegas': 'Buffet & Deserts',\n",
      " 'buffet/dinner/lunch/special': 'Buffet Service & Specials ',\n",
      " 'buffet/dollar/include/person': 'Price & Value ',\n",
      " 'buffet/line/lunch/vegas': 'Buffet',\n",
      " 'buffet/strip/trip/vegas': 'Vegas Buffet Experience',\n",
      " 'burrito/chip/fish/salsa': 'Maxican Style Food',\n",
      " 'caesar/dress/lettuce/salad': 'Salads',\n",
      " 'caesar/dress/pasta/salad': 'Salads / Pasta',\n",
      " 'caesar/dress/potato/salad': 'Salads & dressings',\n",
      " 'chair/dirty/hostess/table': 'Dining Area',\n",
      " 'change/item/lunch/menu': 'Lunch / Food',\n",
      " 'check/hotel/room/stay': 'Hotel Stay',\n",
      " 'cheese/grill/mac-n/steak': 'Stakes / Mac-n-Cheese',\n",
      " 'chef/fish/sashimi/sushi': 'Seafood / Sushi',\n",
      " 'chicken/rice/waffle/wing': 'Chicken / Wings / Deserts',\n",
      " 'chinese-food/good/mexican-food/not-good': 'Chinese / Maxican Food',\n",
      " 'chip/fish/salsa/taco': 'Finger food',\n",
      " 'clean/fast/fresh/tasty': 'Atmosphere & Food',\n",
      " 'clean/fresh/taste/tasty': 'Environment & Taste',\n",
      " 'clean/inside/nice/outside': 'Ambiance',\n",
      " 'clean/inside/nice/owner': 'Ambiance & Decor',\n",
      " 'close/delivery/half/hour': 'Delivery & Delays',\n",
      " 'cold/food/poisoning/quality': 'Food Quality',\n",
      " 'cook/dinner/medium-rare/steak': 'Stake Style Food',\n",
      " 'cook/medium/medium-rare/steak': 'Stakes',\n",
      " 'cool/much/pretty/tasty': 'Good Food',\n",
      " 'couple/second-time/time/visit': 'Couple dining experience',\n",
      " 'couple/time/today/week': 'Couple Date',\n",
      " 'crust/delivery/pizza/slice': 'Pizza delivery',\n",
      " 'crust/good-pizza/pizza/slice': 'Pizzeria dining',\n",
      " 'customer/employee/owner/rude': 'Bad Staff Behavior',\n",
      " 'customer/fast/great-customer/service': 'Customer Service',\n",
      " 'customer/great-customer/service/slow': 'Service ',\n",
      " 'decent/much/not-bad/pretty': 'Decent Food experience',\n",
      " 'decide/feel/hungry/leave': 'Food in general',\n",
      " 'dinner/excellent/highly-recommend/wife': 'Family dining experience',\n",
      " 'dinner/husband/meal/night': 'Dinner Experience',\n",
      " 'dinner/lunch/special/today': 'Food in general',\n",
      " 'dinner/night/steak/wine': 'Steak & Wine ',\n",
      " 'dish/flavor/meat/sauce': 'Food flavor & taste',\n",
      " 'dish/meat/sauce/serve': 'Food & Service',\n",
      " 'dish/noodle/rice/soup': 'Soups / Noodles',\n",
      " 'drive/employee/location/love': 'Location & Area',\n",
      " 'experience/not-bad/not-good/not-recommend': 'Dining Experience',\n",
      " 'family/friend/night/recommend': 'Dining Experience',\n",
      " 'family/spot/town/year': 'Family Dining',\n",
      " 'fast/friendly-staff/super/yummy': 'Staff & Food',\n",
      " 'feel/like/more/much': 'Food in gereral',\n",
      " 'feel/like/more/smell': 'Smell',\n",
      " 'feel/like/more/taste': 'Food in general',\n",
      " 'fish/roll/sashimi/sushi': 'Fish & Sushi',\n",
      " 'flavor/meat/sauce/wing': 'Food veriety & flavors',\n",
      " 'friendly-staff/good-food/good-price/good-service': 'Overall good dining & '\n",
      "                                                     'service',\n",
      " 'friendly-staff/good-food/good-service/recommend': 'Good food & service',\n",
      " 'friendly/member/rude/staff': 'Management & Staff',\n",
      " 'good/mexican-food/not-great/town': 'Mexican Food / Location',\n",
      " 'great-atmosphere/great-food/great-service/highly-recommend': 'Overall great '\n",
      "                                                               'dining '\n",
      "                                                               'experience',\n",
      " 'group/line/people/work': 'People & Location',\n",
      " 'high/not-worth/price/quality': 'Value for money',\n",
      " 'horrible/service/slow/terrible': 'Bad dining experience',\n",
      " 'hostess/party/reservation/seat': 'Service & Reservation',\n",
      " 'hotel/night/room/stay': 'Visit/Stay',\n",
      " 'hour/line/minute/wait': 'Service Delays',\n",
      " 'item/menu/offer/option': 'Menu & options',\n",
      " 'leave/manager/people/tell': 'Management & Service',\n",
      " 'line/long/not-worth/wait': 'Wait & Delay',\n",
      " 'lobster/roll/salmon/shrimp': 'Sea Food / Rolls',\n",
      " 'location/not-wait/phoenix/visit': 'Location & Area',\n",
      " 'love/more/review/year': 'Review & Rating',\n",
      " 'manager/phone/speak/tell': 'management in general',\n",
      " 'minute/table/wait/worth': 'Wait & Service',\n",
      " 'money/not-waste/time/waste': 'Pricing & Value (Negative)',\n",
      " 'portion/price/quality/reasonable': 'Food Quantity & Quality',\n",
      " 'reservation/seat/table/waiter': 'Reservation & Environment ',\n",
      " 'shrimp/street/taco/tuesday': 'Food & Environment in general'}\n"
     ]
    }
   ],
   "source": [
    "pprint(topic_desc_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "review_topics = review_file[['business_id','name','review_id','review_stars','text']]\n",
    "review_topics = review_topics.reindex(review_topics.columns.tolist() + ['topic','pos_topic','neg_topic'], axis=1)\n",
    "review_topics = review_topics.fillna('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 358 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "text = review_file[\"sentiment_text\"].tolist()\n",
    "text = [u''+str(txt) for txt in text]\n",
    "star = review_file[\"review_stars\"].tolist()\n",
    "vec = all_vectors.transform(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1.32 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "avec = all_vectors.transform(text)\n",
    "pvec = pos_vectors.transform(text)\n",
    "nvec = neg_vectors.transform(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 331 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "txt_topic = list(itemgetter(*itemgetter(*([str(np.argmax(res)) for res in nmf_all.transform(avec)]))(all_topic_map))(topic_desc_map))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 320 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "pos_topic = list(itemgetter(*itemgetter(*([str(np.argmax(res)) for res in nmf_pos.transform(pvec)]))(pos_topic_map))(topic_desc_map))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 321 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "neg_topic = list(itemgetter(*itemgetter(*([str(np.argmax(res)) for res in nmf_neg.transform(nvec)]))(neg_topic_map))(topic_desc_map))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is the topic assignment for our 50 topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Buffet & Deserts',\n",
       " 'Buffet & Deserts',\n",
       " 'Buffet & Deserts',\n",
       " 'Maxican Style Food',\n",
       " 'Buffet & Deserts',\n",
       " 'Bill & Money',\n",
       " 'Buffet & Deserts',\n",
       " 'Seafood / Sushi',\n",
       " 'Decent Food experience',\n",
       " 'Dining Experience',\n",
       " 'Management & Service',\n",
       " 'Management & Service',\n",
       " 'Good Service',\n",
       " 'Atmosphere & Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Sea Food / Rolls',\n",
       " 'Atmosphere & Food',\n",
       " 'Food in gereral',\n",
       " 'Mexican Food / Location',\n",
       " 'Couple dining experience',\n",
       " 'Atmosphere & Food',\n",
       " 'Soups / Noodles',\n",
       " 'Food Quantity & Quality',\n",
       " 'Buffet & Deserts',\n",
       " 'Buffet & Deserts',\n",
       " 'Couple dining experience',\n",
       " 'Management & Service',\n",
       " 'Buffet & Deserts',\n",
       " 'Management & Service',\n",
       " 'Atmosphere & Food',\n",
       " 'Staff & Service',\n",
       " 'Management & Service',\n",
       " 'Buffet & Deserts',\n",
       " 'Bill & Money',\n",
       " 'Buffet & Deserts',\n",
       " 'Decent Food experience',\n",
       " 'Food in gereral',\n",
       " 'Stake Style Food',\n",
       " 'Bill & Money',\n",
       " 'Soups / Noodles',\n",
       " 'Atmosphere & Food',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Couple dining experience',\n",
       " 'Atmosphere & Food',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Service Delays',\n",
       " 'Pizzeria dining',\n",
       " 'Atmosphere & Food',\n",
       " 'Food in gereral',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Buffet & Deserts',\n",
       " 'Mexican Food / Location',\n",
       " 'Couple dining experience',\n",
       " 'Buffet & Deserts',\n",
       " 'Atmosphere & Food',\n",
       " 'Service ',\n",
       " 'Food veriety & flavors',\n",
       " 'Buffet & Deserts',\n",
       " 'Buffet & Deserts',\n",
       " 'Buffet & Deserts',\n",
       " 'Management & Service',\n",
       " 'Bill & Money',\n",
       " 'Stake Style Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Management & Service',\n",
       " 'Management & Service',\n",
       " 'Food veriety & flavors',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Atmosphere & Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Seafood / Sushi',\n",
       " 'Couple dining experience',\n",
       " 'Food Quantity & Quality',\n",
       " 'Buffet & Deserts',\n",
       " 'Tasty food',\n",
       " 'Sandwitches / Meat',\n",
       " 'Atmosphere & Food',\n",
       " 'Mexican Food / Location',\n",
       " 'Decent Food experience',\n",
       " 'Family dining experience',\n",
       " 'Sandwitches / Meat',\n",
       " 'Reception & Staff',\n",
       " 'Atmosphere & Food',\n",
       " 'Mexican Food / Location',\n",
       " 'Sandwitches / Meat',\n",
       " 'Salads / Pasta',\n",
       " 'Food Quantity & Quality',\n",
       " 'Food flavors & Taste',\n",
       " 'Grilled Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Food in general',\n",
       " 'Mexican Food / Location',\n",
       " 'Sandwitches / Meat',\n",
       " 'Tasty food',\n",
       " 'Salads / Pasta',\n",
       " 'Service ',\n",
       " 'Salads / Pasta',\n",
       " 'Decent Food experience',\n",
       " 'Management & Service',\n",
       " 'Ambiance',\n",
       " 'Menu & options',\n",
       " 'Couple dining experience',\n",
       " 'Bill & Money',\n",
       " 'Sandwitches / Meat',\n",
       " 'Staff & Food',\n",
       " 'Food in general',\n",
       " 'Sandwitches / Meat',\n",
       " 'Mexican Food / Location',\n",
       " 'Management & Service',\n",
       " 'Grilled Food',\n",
       " 'Good Service',\n",
       " 'Maxican Style Food',\n",
       " 'Salads / Pasta',\n",
       " 'Menu & options',\n",
       " 'Salads / Pasta',\n",
       " 'Decent Food experience',\n",
       " 'Menu & options',\n",
       " 'Service Delays',\n",
       " 'Sandwitches / Meat',\n",
       " 'Salads / Pasta',\n",
       " 'Sandwitches / Meat',\n",
       " 'Food in general',\n",
       " 'Management & Service',\n",
       " 'Maxican Style Food',\n",
       " 'Salads / Pasta',\n",
       " 'Location & Area',\n",
       " 'Food veriety & flavors',\n",
       " 'burgers / Shakes',\n",
       " 'Service ',\n",
       " 'Bill & Money',\n",
       " 'Decent Food experience',\n",
       " 'Bill & Money',\n",
       " 'Bar & Drinks',\n",
       " 'Atmosphere & Food',\n",
       " 'burgers / Shakes',\n",
       " 'Good dining experience',\n",
       " 'burgers / Shakes',\n",
       " 'Dining Experience',\n",
       " 'Location & Area',\n",
       " 'burgers / Shakes',\n",
       " 'Maxican Style Food',\n",
       " 'Service ',\n",
       " 'burgers / Shakes',\n",
       " 'Location & Area',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'Sandwitches / Meat',\n",
       " 'Family dining experience',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'Service ',\n",
       " 'Atmosphere & Food',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'Bar & Drinks',\n",
       " 'Location & Area',\n",
       " 'Decent Food experience',\n",
       " 'Food veriety',\n",
       " 'Management & Service',\n",
       " 'burgers / Shakes',\n",
       " 'Food Quantity & Quality',\n",
       " 'Family dining experience',\n",
       " 'burgers / Shakes',\n",
       " 'Bill & Money',\n",
       " 'burgers / Shakes',\n",
       " 'Couple dining experience',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'Location & Area',\n",
       " 'Management & Service',\n",
       " 'Food veriety',\n",
       " 'burgers / Shakes',\n",
       " 'Bill & Money',\n",
       " 'Management & Service',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'Atmosphere & Food',\n",
       " 'burgers / Shakes',\n",
       " 'Mexican Food / Location',\n",
       " 'burgers / Shakes',\n",
       " 'Service Delays',\n",
       " 'Couple dining experience',\n",
       " 'burgers / Shakes',\n",
       " 'Atmosphere & Food',\n",
       " 'Staff & Service',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'Food veriety',\n",
       " 'Food & Environment in general',\n",
       " 'Decent Food experience',\n",
       " 'Location & Area',\n",
       " 'Staff & Food',\n",
       " 'Maxican Style Food',\n",
       " 'Couple dining experience',\n",
       " 'Maxican Style Food',\n",
       " 'Food flavors & Taste',\n",
       " 'Stake Style Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Ambiance',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Maxican Style Food',\n",
       " 'Mexican Food / Location',\n",
       " 'Good dining experience',\n",
       " 'Atmosphere & Food',\n",
       " 'Food & Environment in general',\n",
       " 'Mexican Food / Location',\n",
       " 'Dining Experience',\n",
       " 'Food veriety & flavors',\n",
       " 'Food & Environment in general',\n",
       " 'Food Quantity & Quality',\n",
       " 'Stake Style Food',\n",
       " 'Stake Style Food',\n",
       " 'Food & Environment in general',\n",
       " 'Atmosphere & Food',\n",
       " 'Mexican Food / Location',\n",
       " 'Food veriety & flavors',\n",
       " 'Good dining experience',\n",
       " 'Food veriety & flavors',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Food veriety & flavors',\n",
       " 'Location & Area',\n",
       " 'Staff & Food',\n",
       " 'Location & Area',\n",
       " 'Maxican Style Food',\n",
       " 'Salads / Pasta',\n",
       " 'Bill & Money',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Staff & Food',\n",
       " 'Food & Environment in general',\n",
       " 'Stake Style Food',\n",
       " 'Food & Environment in general',\n",
       " 'Reception & Staff',\n",
       " 'Family dining experience',\n",
       " 'Food & Environment in general',\n",
       " 'Good dining experience',\n",
       " 'Decent Food experience',\n",
       " 'Management & Service',\n",
       " 'Bill & Money',\n",
       " 'Location & Area',\n",
       " 'Atmosphere',\n",
       " 'Couple dining experience',\n",
       " 'Food & Environment in general',\n",
       " 'Decent Food experience',\n",
       " 'Atmosphere & Food',\n",
       " 'Maxican Style Food',\n",
       " 'Staff & Food',\n",
       " 'Food & Environment in general',\n",
       " 'Food veriety & flavors',\n",
       " 'Atmosphere & Food',\n",
       " 'Soups / Noodles',\n",
       " 'Good Service',\n",
       " 'Tasty food',\n",
       " 'Food & Environment in general',\n",
       " 'Maxican Style Food',\n",
       " 'Maxican Style Food',\n",
       " 'Food veriety & flavors',\n",
       " 'Decent Food experience',\n",
       " 'Family dining experience',\n",
       " 'Maxican Style Food',\n",
       " 'Bar & Drinks',\n",
       " 'Atmosphere & Food',\n",
       " 'Good Service',\n",
       " 'Atmosphere & Food',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Bar & Drinks',\n",
       " 'Food & Environment in general',\n",
       " 'Bar & Drinks',\n",
       " 'Maxican Style Food',\n",
       " 'Dining Experience',\n",
       " 'Dining Experience',\n",
       " 'Service ',\n",
       " 'Good Service',\n",
       " 'Bar & Drinks',\n",
       " 'Stake Style Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Ambiance',\n",
       " 'Couple dining experience',\n",
       " 'Bill & Money',\n",
       " 'Atmosphere & Food',\n",
       " 'Family dining experience',\n",
       " 'Stake Style Food',\n",
       " 'Food veriety',\n",
       " 'Management & Service',\n",
       " 'Good dining experience',\n",
       " 'Management & Service',\n",
       " 'Mexican Food / Location',\n",
       " 'Menu & options',\n",
       " 'Bar & Drinks',\n",
       " 'Couple dining experience',\n",
       " 'Sea Food / Rolls',\n",
       " 'Atmosphere',\n",
       " 'Dining Experience',\n",
       " 'Atmosphere & Food',\n",
       " 'Couple dining experience',\n",
       " 'Dining Experience',\n",
       " 'Food veriety & flavors',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Good dining experience',\n",
       " 'Family dining experience',\n",
       " 'Couple dining experience',\n",
       " 'Staff & Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Bar & Drinks',\n",
       " 'Ambiance',\n",
       " 'Salads / Pasta',\n",
       " 'Bar & Drinks',\n",
       " 'Maxican Style Food',\n",
       " 'Atmosphere',\n",
       " 'Bar & Drinks',\n",
       " 'Bill & Money',\n",
       " 'Maxican Style Food',\n",
       " 'Bar & Drinks',\n",
       " 'Ambiance',\n",
       " 'Grilled Food',\n",
       " 'Food veriety & flavors',\n",
       " 'Staff & Food',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Bar & Drinks',\n",
       " 'Good Service',\n",
       " 'Bar & Drinks',\n",
       " 'Family dining experience',\n",
       " 'Food & Environment in general',\n",
       " 'Good dining experience',\n",
       " 'Good dining experience',\n",
       " 'Maxican Style Food',\n",
       " 'Salads / Pasta',\n",
       " 'Bar & Drinks',\n",
       " 'Bar & Drinks',\n",
       " 'Food & Environment in general',\n",
       " 'Stake Style Food',\n",
       " 'Food flavors & Taste',\n",
       " 'Food veriety',\n",
       " 'Stake Style Food',\n",
       " 'Couple dining experience',\n",
       " 'Good dining experience',\n",
       " 'Reception & Staff',\n",
       " 'Food & Environment in general',\n",
       " 'Service ',\n",
       " 'Menu & options',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Mexican Food / Location',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Atmosphere & Food',\n",
       " 'Location & Area',\n",
       " 'Bar & Drinks',\n",
       " 'Mexican Food / Location',\n",
       " 'Dining Experience',\n",
       " 'Bar & Drinks',\n",
       " 'Mexican Food / Location',\n",
       " 'Bar & Drinks',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Dining Experience',\n",
       " 'Maxican Style Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Sea Food / Rolls',\n",
       " 'Mexican Food / Location',\n",
       " 'Food & Environment in general',\n",
       " 'Bar & Drinks',\n",
       " 'Bar & Drinks',\n",
       " 'Bar & Drinks',\n",
       " 'Food flavors & Taste',\n",
       " 'Food & Environment in general',\n",
       " 'Food & Environment in general',\n",
       " 'Couple dining experience',\n",
       " 'Food & Environment in general',\n",
       " 'Location & Area',\n",
       " 'Bar & Drinks',\n",
       " 'Food flavors & Taste',\n",
       " 'Bar & Drinks',\n",
       " 'Dining Experience',\n",
       " 'Bill & Money',\n",
       " 'Good Service',\n",
       " 'Food & Environment in general',\n",
       " 'Food Quantity & Quality',\n",
       " 'Good dining experience',\n",
       " 'burgers / Shakes',\n",
       " 'Good dining experience',\n",
       " 'Menu & options',\n",
       " 'Good Service',\n",
       " 'burgers / Shakes',\n",
       " 'Review & Star Rating',\n",
       " 'burgers / Shakes',\n",
       " 'Couple dining experience',\n",
       " 'Visit/Stay',\n",
       " 'Food Quantity & Quality',\n",
       " 'Service ',\n",
       " 'Dining Experience',\n",
       " 'Atmosphere & Food',\n",
       " 'Couple dining experience',\n",
       " 'Management & Service',\n",
       " 'Staff & Service',\n",
       " 'Ambiance',\n",
       " 'Good dining experience',\n",
       " 'Couple dining experience',\n",
       " 'Couple dining experience',\n",
       " 'Staff & Food',\n",
       " 'Couple dining experience',\n",
       " 'Decent Food experience',\n",
       " 'Atmosphere & Food',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'Ambiance',\n",
       " 'Food flavors & Taste',\n",
       " 'Good Service',\n",
       " 'Review & Star Rating',\n",
       " 'Couple dining experience',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'burgers / Shakes',\n",
       " 'Management & Service',\n",
       " 'burgers / Shakes',\n",
       " 'Food & Environment in general',\n",
       " 'Decent Food experience',\n",
       " 'Breakfast',\n",
       " 'Couple dining experience',\n",
       " 'Food veriety',\n",
       " 'Staff & Service',\n",
       " 'burgers / Shakes',\n",
       " 'Food in gereral',\n",
       " 'Bill & Money',\n",
       " 'Ambiance',\n",
       " 'burgers / Shakes',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Review & Star Rating',\n",
       " 'Staff & Service',\n",
       " 'Dining Experience',\n",
       " 'Couple dining experience',\n",
       " 'Food veriety',\n",
       " 'burgers / Shakes',\n",
       " 'Salads / Pasta',\n",
       " 'Management & Service',\n",
       " 'Service ',\n",
       " 'burgers / Shakes',\n",
       " 'Food flavors & Taste',\n",
       " 'Couple dining experience',\n",
       " 'Good dining experience',\n",
       " 'Food in gereral',\n",
       " 'Bar & Drinks',\n",
       " 'Sandwitches / Meat',\n",
       " 'Good Service',\n",
       " 'Staff & Food',\n",
       " 'Pizzeria dining',\n",
       " 'Service ',\n",
       " 'Sandwitches / Meat',\n",
       " 'Pizzeria dining',\n",
       " 'Ambiance',\n",
       " 'Reception & Staff',\n",
       " 'Staff & Food',\n",
       " 'Mexican Food / Location',\n",
       " 'Visit/Stay',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Pizzeria dining',\n",
       " 'Food Quantity & Quality',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Atmosphere & Food',\n",
       " 'Couple dining experience',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Food in general',\n",
       " 'Pizzeria dining',\n",
       " 'Dining Experience',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Food in general',\n",
       " 'Food veriety & flavors',\n",
       " 'Stake Style Food',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Dining Experience',\n",
       " 'Reception & Staff',\n",
       " 'Bar & Drinks',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Food veriety & flavors',\n",
       " 'Food flavors & Taste',\n",
       " 'Pizzeria dining',\n",
       " 'Atmosphere & Food',\n",
       " 'Stake Style Food',\n",
       " 'Pizzeria dining',\n",
       " 'Service Delays',\n",
       " 'Pizzeria dining',\n",
       " 'Service ',\n",
       " 'Mexican Food / Location',\n",
       " 'Mexican Food / Location',\n",
       " 'Mexican Food / Location',\n",
       " 'Atmosphere & Food',\n",
       " 'Food in general',\n",
       " 'Sandwitches / Meat',\n",
       " 'Bill & Money',\n",
       " 'Mexican Food / Location',\n",
       " 'Staff & Food',\n",
       " 'Bill & Money',\n",
       " 'Management & Service',\n",
       " 'Pizzeria dining',\n",
       " 'Dining Experience',\n",
       " 'Food veriety & flavors',\n",
       " 'Sandwitches / Meat',\n",
       " 'Food veriety & flavors',\n",
       " 'Pizzeria dining',\n",
       " 'Food veriety & flavors',\n",
       " 'Grilled Food',\n",
       " 'Staff & Food',\n",
       " 'Couple dining experience',\n",
       " 'Service ',\n",
       " 'Food Quantity & Quality',\n",
       " 'Atmosphere & Food',\n",
       " 'Couple dining experience',\n",
       " 'Good dining experience',\n",
       " 'Food veriety & flavors',\n",
       " 'Couple dining experience',\n",
       " 'Ambiance',\n",
       " 'Atmosphere & Food',\n",
       " 'Grilled Food',\n",
       " 'Grilled Food',\n",
       " 'Good dining experience',\n",
       " 'Atmosphere & Food',\n",
       " 'Grilled Food',\n",
       " 'Staff & Food',\n",
       " 'Sandwitches / Meat',\n",
       " 'Grilled Food',\n",
       " 'Grilled Food',\n",
       " 'Bill & Money',\n",
       " 'Grilled Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Food in gereral',\n",
       " 'Family dining experience',\n",
       " 'Reception & Staff',\n",
       " 'Couple dining experience',\n",
       " 'Family dining experience',\n",
       " 'Couple dining experience',\n",
       " 'Grilled Food',\n",
       " 'Atmosphere',\n",
       " 'Good dining experience',\n",
       " 'Grilled Food',\n",
       " 'Dining Experience',\n",
       " 'Grilled Food',\n",
       " 'Food veriety & flavors',\n",
       " 'Sandwitches / Meat',\n",
       " 'Sandwitches / Meat',\n",
       " 'Food veriety & flavors',\n",
       " 'Good dining experience',\n",
       " 'Atmosphere & Food',\n",
       " 'Grilled Food',\n",
       " 'Sandwitches / Meat',\n",
       " 'Atmosphere',\n",
       " 'Grilled Food',\n",
       " 'Buffet & Deserts',\n",
       " 'Grilled Food',\n",
       " 'Grilled Food',\n",
       " 'Good Service',\n",
       " 'Grilled Food',\n",
       " 'Sandwitches / Meat',\n",
       " 'Grilled Food',\n",
       " 'Couple dining experience',\n",
       " 'Sandwitches / Meat',\n",
       " 'Good dining experience',\n",
       " 'Grilled Food',\n",
       " 'Grilled Food',\n",
       " 'Good Service',\n",
       " 'Reception & Staff',\n",
       " 'Sandwitches / Meat',\n",
       " 'Grilled Food',\n",
       " 'Seafood / Sushi',\n",
       " 'Grilled Food',\n",
       " 'Management & Service',\n",
       " 'Sandwitches / Meat',\n",
       " 'Grilled Food',\n",
       " 'Grilled Food',\n",
       " 'Grilled Food',\n",
       " 'Dining Experience',\n",
       " 'Review & Star Rating',\n",
       " 'Sandwitches / Meat',\n",
       " 'Good Service',\n",
       " 'Maxican Style Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Food veriety',\n",
       " 'Couple dining experience',\n",
       " 'Food Quantity & Quality',\n",
       " 'burgers / Shakes',\n",
       " 'Review & Star Rating',\n",
       " 'Mexican Food / Location',\n",
       " 'Stake Style Food',\n",
       " 'Food veriety',\n",
       " 'Food in general',\n",
       " 'Location & Area',\n",
       " 'Couple dining experience',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Couple dining experience',\n",
       " 'Good Service',\n",
       " 'Atmosphere & Food',\n",
       " 'Staff & Service',\n",
       " 'Stake Style Food',\n",
       " 'burgers / Shakes',\n",
       " 'Salads / Pasta',\n",
       " 'Mexican Food / Location',\n",
       " 'Bar & Drinks',\n",
       " 'Stake Style Food',\n",
       " 'Stake Style Food',\n",
       " 'Staff & Service',\n",
       " 'Visit/Stay',\n",
       " 'Mexican Food / Location',\n",
       " 'Atmosphere & Food',\n",
       " 'Salads / Pasta',\n",
       " 'Food veriety',\n",
       " 'Service ',\n",
       " 'Bar & Drinks',\n",
       " 'Couple dining experience',\n",
       " 'Food in gereral',\n",
       " 'burgers / Shakes',\n",
       " 'Bar & Drinks',\n",
       " 'Review & Star Rating',\n",
       " 'Food veriety & flavors',\n",
       " 'Salads / Pasta',\n",
       " 'Soups / Noodles',\n",
       " 'Service Delays',\n",
       " 'Bar & Drinks',\n",
       " 'Salads / Pasta',\n",
       " 'Bar & Drinks',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Service ',\n",
       " 'Review & Star Rating',\n",
       " 'burgers / Shakes',\n",
       " 'Family dining experience',\n",
       " 'Stake Style Food',\n",
       " 'Stake Style Food',\n",
       " 'Location & Area',\n",
       " 'Food veriety',\n",
       " 'Service Delays',\n",
       " 'Food veriety & flavors',\n",
       " 'Service ',\n",
       " 'Reception & Staff',\n",
       " 'Food in gereral',\n",
       " 'Dining Experience',\n",
       " 'Decent Food experience',\n",
       " 'Dining Experience',\n",
       " 'Couple dining experience',\n",
       " 'Stake Style Food',\n",
       " 'Tasty food',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Food veriety & flavors',\n",
       " 'Bill & Money',\n",
       " 'Pizzeria dining',\n",
       " 'Atmosphere',\n",
       " 'Staff & Food',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Management & Service',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Food in general',\n",
       " 'Atmosphere & Food',\n",
       " 'Mexican Food / Location',\n",
       " 'Pizzeria dining',\n",
       " 'Mexican Food / Location',\n",
       " 'Pizzeria dining',\n",
       " 'Grilled Food',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Staff & Food',\n",
       " 'Salads / Pasta',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Atmosphere & Food',\n",
       " 'Food Quantity & Quality',\n",
       " 'Bill & Money',\n",
       " 'Pizzeria dining',\n",
       " 'Menu & options',\n",
       " 'Bill & Money',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Couple dining experience',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Ambiance',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Food veriety & flavors',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Mexican Food / Location',\n",
       " 'Bill & Money',\n",
       " 'Pizzeria dining',\n",
       " 'Ambiance',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Food veriety & flavors',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Visit/Stay',\n",
       " 'Pizzeria dining',\n",
       " 'Staff & Food',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Bill & Money',\n",
       " 'Staff & Service',\n",
       " 'Atmosphere & Food',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Food Quantity & Quality',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Mexican Food / Location',\n",
       " 'Pizzeria dining',\n",
       " 'Atmosphere & Food',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Mexican Food / Location',\n",
       " 'Ambiance',\n",
       " 'Food veriety',\n",
       " 'Food flavors & Taste',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Atmosphere & Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Dining Experience',\n",
       " 'Food veriety & flavors',\n",
       " 'Pizzeria dining',\n",
       " 'Food veriety',\n",
       " 'Review & Star Rating',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Atmosphere & Food',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Food in gereral',\n",
       " 'Mexican Food / Location',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Food Quantity & Quality',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Mexican Food / Location',\n",
       " 'Mexican Food / Location',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Family dining experience',\n",
       " 'Ambiance',\n",
       " 'Sea Food / Rolls',\n",
       " 'Family dining experience',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Management & Service',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Atmosphere & Food',\n",
       " 'Decent Food experience',\n",
       " 'Pizzeria dining',\n",
       " 'Couple dining experience',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Mexican Food / Location',\n",
       " 'Location & Area',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Family dining experience',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Atmosphere & Food',\n",
       " 'Staff & Service',\n",
       " 'Pizzeria dining',\n",
       " 'Staff & Food',\n",
       " 'Pizzeria dining',\n",
       " 'Good Service',\n",
       " 'Staff & Food',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Food veriety & flavors',\n",
       " 'Menu & options',\n",
       " 'Pizzeria dining',\n",
       " 'Good Service',\n",
       " 'Pizzeria dining',\n",
       " 'Service ',\n",
       " 'Pizzeria dining',\n",
       " 'Salads / Pasta',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Family dining experience',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Pizzeria dining',\n",
       " 'Food in gereral',\n",
       " 'Atmosphere',\n",
       " 'Bill & Money',\n",
       " 'Couple dining experience',\n",
       " 'Bill & Money',\n",
       " 'Couple dining experience',\n",
       " 'Salads / Pasta',\n",
       " 'Salads / Pasta',\n",
       " 'Good Service',\n",
       " 'Food in gereral',\n",
       " 'Couple dining experience',\n",
       " 'Review & Star Rating',\n",
       " 'Soups / Noodles',\n",
       " 'Decent Food experience',\n",
       " 'Atmosphere & Food',\n",
       " 'Food flavors & Taste',\n",
       " 'Menu & options',\n",
       " 'Atmosphere & Food',\n",
       " 'Buffet & Deserts',\n",
       " 'Salads / Pasta',\n",
       " 'Atmosphere & Food',\n",
       " 'Food flavors & Taste',\n",
       " 'Food in general',\n",
       " 'Food in general',\n",
       " 'Dining Experience',\n",
       " 'Food in general',\n",
       " 'Food in general',\n",
       " 'Review & Star Rating',\n",
       " 'Visit/Stay',\n",
       " 'Reservation & Environment ',\n",
       " 'Family dining experience',\n",
       " 'Grilled Food',\n",
       " 'Menu & options',\n",
       " 'Decent Food experience',\n",
       " 'Review & Star Rating',\n",
       " 'Good dining experience',\n",
       " 'Atmosphere',\n",
       " 'Atmosphere & Food',\n",
       " 'Dining Experience',\n",
       " 'Good dining experience',\n",
       " 'Mexican Food / Location',\n",
       " 'Bill & Money',\n",
       " 'Mexican Food / Location',\n",
       " 'Food veriety & flavors',\n",
       " 'Salads / Pasta',\n",
       " 'Sandwitches / Meat',\n",
       " 'Atmosphere & Food',\n",
       " 'Reception & Staff',\n",
       " 'Service ',\n",
       " 'Food in gereral',\n",
       " 'Staff & Food',\n",
       " 'Review & Star Rating',\n",
       " 'Family dining experience',\n",
       " 'Food in gereral',\n",
       " 'Pizzeria dining',\n",
       " 'Couple dining experience',\n",
       " 'Sandwitches / Meat',\n",
       " 'Atmosphere & Food',\n",
       " 'Sandwitches / Meat',\n",
       " 'Ambiance',\n",
       " 'Food in gereral',\n",
       " 'Staff & Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Good Service',\n",
       " 'Atmosphere & Food',\n",
       " 'Atmosphere & Food',\n",
       " 'Decent Food experience',\n",
       " 'Dining Experience',\n",
       " 'Ambiance',\n",
       " 'Mexican Food / Location',\n",
       " 'Management & Service',\n",
       " 'Food in general',\n",
       " 'Food in general',\n",
       " 'Decent Food experience',\n",
       " 'Food veriety & flavors',\n",
       " 'Mexican Food / Location',\n",
       " 'Family dining experience',\n",
       " 'Ambiance',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Food in gereral',\n",
       " 'Atmosphere & Food',\n",
       " 'Food veriety & flavors',\n",
       " 'Food veriety & flavors',\n",
       " 'Mexican Food / Location',\n",
       " 'Food veriety & flavors',\n",
       " 'Decent Food experience',\n",
       " 'Reception & Staff',\n",
       " 'Mexican Food / Location',\n",
       " 'Family dining experience',\n",
       " 'Food veriety & flavors',\n",
       " 'Staff & Food',\n",
       " 'Food veriety & flavors',\n",
       " 'Mexican Food / Location',\n",
       " 'Maxican Style Food',\n",
       " 'Food flavors & Taste',\n",
       " 'Soups / Noodles',\n",
       " 'Couple dining experience',\n",
       " 'Decent Food experience',\n",
       " 'Staff & Food',\n",
       " 'Food in gereral',\n",
       " 'Good Service',\n",
       " 'Decent Food experience',\n",
       " 'Food in general',\n",
       " 'Bill & Money',\n",
       " 'Soups / Noodles',\n",
       " 'Food veriety & flavors',\n",
       " 'Sea Food / Rolls',\n",
       " 'Mexican Food / Location',\n",
       " 'Couple dining experience',\n",
       " 'Mexican Food / Location',\n",
       " 'Food Quantity & Quality',\n",
       " 'Ambiance',\n",
       " 'Mexican Food / Location',\n",
       " 'Dining Experience',\n",
       " 'Food veriety & flavors',\n",
       " 'Bill & Money',\n",
       " 'Couple dining experience',\n",
       " 'Family dining experience',\n",
       " 'Couple dining experience',\n",
       " 'Food in general',\n",
       " 'Dining Experience',\n",
       " 'Food Quantity & Quality',\n",
       " 'Food in general',\n",
       " 'Food in gereral',\n",
       " 'Bill & Money',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Food in general',\n",
       " 'Ambiance',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Good dining experience',\n",
       " 'Food flavors & Taste',\n",
       " 'Atmosphere & Food',\n",
       " 'Chicken / Wings / Deserts',\n",
       " 'Food in general',\n",
       " 'Food in general',\n",
       " 'Atmosphere & Food',\n",
       " 'Mexican Food / Location',\n",
       " 'Decent Food experience',\n",
       " 'Food in general',\n",
       " 'Food in gereral',\n",
       " 'Sea Food / Rolls',\n",
       " ...]"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt_topic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [],
   "source": [
    "for index in range(len(review_file)):\n",
    "    if star[index]<3:\n",
    "        pos_topic[index]=''\n",
    "    if star[index]>3:\n",
    "        neg_topic[index]=''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [],
   "source": [
    "review_topics['topic']     = txt_topic\n",
    "review_topics['pos_topic'] = pos_topic\n",
    "review_topics['neg_topic'] = neg_topic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [],
   "source": [
    "##### Lets look at the topics fount for a text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a 5-Star (positive review)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_topics.review_stars.iloc[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The best food I want to the a lot of the buffet restaurant this is the best place to eat and Great service'"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_topics.text.iloc[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obviously it has no negative topic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "''"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_topics.neg_topic.iloc[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Vegas Buffet Experience'"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_topics.pos_topic.iloc[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets take another one a 3 Star review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_topics.review_stars.iloc[9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Review_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'My friend took me there. Price was ok. Everything is just within the normal range. Nothing special.'"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_topics.text.iloc[9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Overall Theme:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Dining Experience'"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_topics.topic.iloc[9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Positive Theme:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Food Quantity & Quality'"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_topics.pos_topic.iloc[9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Negative Theme:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Birthday / Group Dining'"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_topics.neg_topic.iloc[9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the data in CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [],
   "source": [
    "#review_topics.to_csv('processed_data/review_topics.csv',encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topics_texts(str_text, nmf_all, nmf_pos, nmf_neg, all_topic_map, pos_topic_map, neg_topic_map, topic_desc_map, all_vectors, pos_vectors, neg_vectors):\n",
    "    text = [(u''+str(str_text))]\n",
    "    \n",
    "    avec = all_vectors.transform(text)\n",
    "    pvec = pos_vectors.transform(text)\n",
    "    nvec = neg_vectors.transform(text)\n",
    "    \n",
    "    txt_topic_text = (topic_desc_map[all_topic_map[str(np.argmax(nmf_all.transform(avec)))]])\n",
    "    pos_topic_text = (topic_desc_map[pos_topic_map[str(np.argmax(nmf_pos.transform(pvec)))]])\n",
    "    neg_topic_text = (topic_desc_map[neg_topic_map[str(np.argmax(nmf_neg.transform(nvec)))]])\n",
    "    \n",
    "    return [txt_topic_text,pos_topic_text,neg_topic_text]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_term_vector():\n",
    "    with open('pickles/topic_term_vector_all.pk', 'rb') as fin:\n",
    "        return pickle.load(fin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_nmf_all():\n",
    "    with open('pickles/all_nmf_model.pk', 'rb') as fin:\n",
    "        return pickle.load(fin)\n",
    "\n",
    "def get_nmf_pos():\n",
    "    with open('pickles/pos_nmf_model.pk', 'rb') as fin:\n",
    "        return pickle.load(fin)\n",
    "\n",
    "def get_nmf_neg():\n",
    "    with open('pickles/neg_nmf_model.pk', 'rb') as fin:\n",
    "        return pickle.load(fin)\n",
    "    \n",
    "def get_pos_topics_map():\n",
    "    with open('config/pos_topics.txt', 'r') as f:\n",
    "        ret = {line.split(';')[0]: line.split(';')[1].replace('\\n','') for line in f.readlines()}\n",
    "    return ret\n",
    "\n",
    "def get_neg_topics_map():\n",
    "    with open('config/neg_topics.txt', 'r') as f:\n",
    "        ret =  {line.split(';')[0]: line.split(';')[1].replace('\\n','') for line in f.readlines()}\n",
    "    return ret\n",
    "    \n",
    "def get_all_topics_map():\n",
    "    with open('config/all_topics.txt', 'r') as f:\n",
    "         ret = {line.split(';')[0]: line.split(';')[1].replace('\\n','') for line in f.readlines()}\n",
    "    return ret\n",
    "\n",
    "def get_topic_desc_map():\n",
    "    with open('config/topic_map.txt', 'r') as f:\n",
    "        ret = {line.split(';')[0]: line.split(';')[1].replace('\\n','') for line in f.readlines()}\n",
    "    return ret\n",
    "\n",
    "def get_all_term_vec():\n",
    "    with open('pickles/topic_term_vector_all.pk','rb') as f:\n",
    "        return pickle.load(f)\n",
    "\n",
    "def get_pos_term_vec():\n",
    "    with open('pickles/topic_term_vector_pos.pk','rb') as f:\n",
    "        return pickle.load(f)\n",
    "\n",
    "def get_neg_term_vec():\n",
    "    with open('pickles/topic_term_vector_neg.pk','rb') as f:\n",
    "        return pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [],
   "source": [
    "nmf_all     = get_nmf_all()\n",
    "nmf_pos     = get_nmf_pos()\n",
    "nmf_neg     = get_nmf_neg()\n",
    "all_topic_map = get_all_topics_map()\n",
    "pos_topic_map = get_pos_topics_map()\n",
    "neg_topic_map = get_neg_topics_map()\n",
    "topic_desc_map = get_topic_desc_map()    \n",
    "pos_vec     = get_pos_term_vec()\n",
    "neg_vec     = get_neg_term_vec()\n",
    "all_vec     = get_all_term_vec()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [],
   "source": [
    "str_text = review_file.sentiment_text.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The text for review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'zero_star allow disgusting find roach lurch food clean_dish buffet item dirty look plate inspect authority'"
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets find out the overall theme, the negative & positive topics in the review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Buffet & Deserts', 'Atmosphere & Food', 'Buffet']"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_topics_texts(str_text, nmf_all, nmf_pos, nmf_neg, all_topic_map, pos_topic_map, neg_topic_map, topic_desc_map, all_vec, pos_vec, neg_vec)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The is the end of *'Topic Mining'* section. No we proceed to *'Sentiment analysis'*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [],
   "source": [
    "#END"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment analysis\n",
    "\n",
    "In this sention of the notebook, we try to use the pre-processed text find out the sentiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets start by importing libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "import string\n",
    "import math\n",
    "import pickle\n",
    "import os\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.model_selection import train_test_split, cross_val_score\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score, roc_auc_score, roc_curve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Install missing packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 0 == 1:\n",
    "    import sys\n",
    "    !conda install --yes --prefix {sys.prefix} s3fs seaborn scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a utility function to retreive the data from S3 project bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_s3_bucket(bucket, data_key):\n",
    "    data_location = 's3://{}/{}'.format(bucket, data_key)\n",
    "\n",
    "    chunksize = 1000000\n",
    "    chunk_list = []\n",
    "    df_chunk = pd.read_csv(data_location, chunksize=chunksize)\n",
    "    for chunk in df_chunk:\n",
    "        chunk_list.append(chunk)\n",
    "\n",
    "    df = pd.concat(chunk_list)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets fo ahead and retreive the data from the Illinois file we created earlier in the fisr section of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 240 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "if 1 == 1:\n",
    "    \n",
    "    bucket = 'cs410-yelp'\n",
    "    data_key = 'processed_data/'+state_filter+'_cleaned_reviews.csv'\n",
    "\n",
    "    #df = read_s3_bucket(bucket, data_key)\n",
    "    df = pd.read_csv(data_key, encoding='utf-8')\n",
    "    df = df.drop(labels='Unnamed: 0', axis=1)\n",
    "    df['review_stars']   = df['review_stars'].astype(int)\n",
    "    df['sentiment_text'] = df['sentiment_text'].astype(str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets see some sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>name</th>\n",
       "      <th>review_count</th>\n",
       "      <th>business_stars</th>\n",
       "      <th>review_id</th>\n",
       "      <th>review_stars</th>\n",
       "      <th>text</th>\n",
       "      <th>useful</th>\n",
       "      <th>topic_text</th>\n",
       "      <th>sentiment_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mofOjB6flg-eAWOFbOkHfQ</td>\n",
       "      <td>ChinaTown Buffet</td>\n",
       "      <td>72</td>\n",
       "      <td>2.5</td>\n",
       "      <td>8MTptiOpUeuPUFZgtfk9Vw</td>\n",
       "      <td>1</td>\n",
       "      <td>I would have given this restaurant zero stars ...</td>\n",
       "      <td>1</td>\n",
       "      <td>zero_star allow find roach lurch food buffet i...</td>\n",
       "      <td>zero_star allow disgusting find roach lurch fo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mofOjB6flg-eAWOFbOkHfQ</td>\n",
       "      <td>ChinaTown Buffet</td>\n",
       "      <td>72</td>\n",
       "      <td>2.5</td>\n",
       "      <td>ofpfzn8LV4nJ2pE6IlTVdg</td>\n",
       "      <td>1</td>\n",
       "      <td>So...no. Just...no. Before I picked up a plate...</td>\n",
       "      <td>1</td>\n",
       "      <td>pick plate start food notice roach buffet act ...</td>\n",
       "      <td>no pick plate start. food notice. roach buff...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mofOjB6flg-eAWOFbOkHfQ</td>\n",
       "      <td>ChinaTown Buffet</td>\n",
       "      <td>72</td>\n",
       "      <td>2.5</td>\n",
       "      <td>O3yApCw95tbA0kodflQrpA</td>\n",
       "      <td>5</td>\n",
       "      <td>The best food I want to the a lot of the buffe...</td>\n",
       "      <td>0</td>\n",
       "      <td>want lot buffet</td>\n",
       "      <td>good_food want lot buffet best_eat great_service.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mofOjB6flg-eAWOFbOkHfQ</td>\n",
       "      <td>ChinaTown Buffet</td>\n",
       "      <td>72</td>\n",
       "      <td>2.5</td>\n",
       "      <td>fFTVZE7EENdl66SX9cgc0A</td>\n",
       "      <td>1</td>\n",
       "      <td>It's impressive how filthy this place is. Some...</td>\n",
       "      <td>0</td>\n",
       "      <td>puke bathroom mess smell condition floor area ...</td>\n",
       "      <td>impressive filthy puke bathroom mess smell asi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mofOjB6flg-eAWOFbOkHfQ</td>\n",
       "      <td>ChinaTown Buffet</td>\n",
       "      <td>72</td>\n",
       "      <td>2.5</td>\n",
       "      <td>pDlIw6DLLjZ5G1vJnMzr9A</td>\n",
       "      <td>5</td>\n",
       "      <td>I was visiting the Champaign area with my fami...</td>\n",
       "      <td>0</td>\n",
       "      <td>visit champaign area family recently_move sugg...</td>\n",
       "      <td>visit. champaign area family recently_move. su...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              business_id              name  review_count  business_stars  \\\n",
       "0  mofOjB6flg-eAWOFbOkHfQ  ChinaTown Buffet            72             2.5   \n",
       "1  mofOjB6flg-eAWOFbOkHfQ  ChinaTown Buffet            72             2.5   \n",
       "2  mofOjB6flg-eAWOFbOkHfQ  ChinaTown Buffet            72             2.5   \n",
       "3  mofOjB6flg-eAWOFbOkHfQ  ChinaTown Buffet            72             2.5   \n",
       "4  mofOjB6flg-eAWOFbOkHfQ  ChinaTown Buffet            72             2.5   \n",
       "\n",
       "                review_id  review_stars  \\\n",
       "0  8MTptiOpUeuPUFZgtfk9Vw             1   \n",
       "1  ofpfzn8LV4nJ2pE6IlTVdg             1   \n",
       "2  O3yApCw95tbA0kodflQrpA             5   \n",
       "3  fFTVZE7EENdl66SX9cgc0A             1   \n",
       "4  pDlIw6DLLjZ5G1vJnMzr9A             5   \n",
       "\n",
       "                                                text  useful  \\\n",
       "0  I would have given this restaurant zero stars ...       1   \n",
       "1  So...no. Just...no. Before I picked up a plate...       1   \n",
       "2  The best food I want to the a lot of the buffe...       0   \n",
       "3  It's impressive how filthy this place is. Some...       0   \n",
       "4  I was visiting the Champaign area with my fami...       0   \n",
       "\n",
       "                                          topic_text  \\\n",
       "0  zero_star allow find roach lurch food buffet i...   \n",
       "1  pick plate start food notice roach buffet act ...   \n",
       "2                                    want lot buffet   \n",
       "3  puke bathroom mess smell condition floor area ...   \n",
       "4  visit champaign area family recently_move sugg...   \n",
       "\n",
       "                                      sentiment_text  \n",
       "0  zero_star allow disgusting find roach lurch fo...  \n",
       "1    no pick plate start. food notice. roach buff...  \n",
       "2  good_food want lot buffet best_eat great_service.  \n",
       "3  impressive filthy puke bathroom mess smell asi...  \n",
       "4  visit. champaign area family recently_move. su...  "
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and the length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19220"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets analyse the averages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review_count</th>\n",
       "      <th>business_stars</th>\n",
       "      <th>useful</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>review_stars</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>150.897895</td>\n",
       "      <td>3.413684</td>\n",
       "      <td>1.790526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>166.664529</td>\n",
       "      <td>3.553527</td>\n",
       "      <td>1.516527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>182.922291</td>\n",
       "      <td>3.634258</td>\n",
       "      <td>0.883619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>204.313246</td>\n",
       "      <td>3.781063</td>\n",
       "      <td>0.832649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>245.093020</td>\n",
       "      <td>3.934302</td>\n",
       "      <td>0.934650</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              review_count  business_stars    useful\n",
       "review_stars                                        \n",
       "1               150.897895        3.413684  1.790526\n",
       "2               166.664529        3.553527  1.516527\n",
       "3               182.922291        3.634258  0.883619\n",
       "4               204.313246        3.781063  0.832649\n",
       "5               245.093020        3.934302  0.934650"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stval = df.groupby('review_stars').mean()\n",
    "stval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the stopwords for sentiment analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_words = [line.rstrip('\\n') for line in open('config/stopwords.txt', 'r', encoding='utf-8')] \n",
    "stopwords = frozenset(stop_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets start building the classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2713, 10)\n",
      "0     zero_star allow disgusting find roach lurch fo...\n",
      "1       no pick plate start. food notice. roach buff...\n",
      "5     absolutely disgusting arrive lunch sunday. not...\n",
      "12    good_food east building outside absolutely dis...\n",
      "23    general not_fan chinese_buffet. start amazing_...\n",
      "Name: sentiment_text, dtype: object\n",
      "0     1\n",
      "1     1\n",
      "5     1\n",
      "12    1\n",
      "23    3\n",
      "Name: review_stars, dtype: int32\n",
      "Wall time: 9.97 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# CLASSIFICATION\n",
    "df_classes = df[(df['review_stars'] == 1) | (df['review_stars'] == 3) | (df['review_stars'] == 5)]\n",
    "df_classes = df_classes[(df_classes['useful'] == 1)]\n",
    "df_classes.head()\n",
    "print(df_classes.shape)\n",
    "\n",
    "# Seperate the data set into X and Y for prediction\n",
    "x = df_classes['sentiment_text']\n",
    "y = df_classes['review_stars']\n",
    "print(x.head())\n",
    "print(y.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_process(text):\n",
    "    nopunc = [char for char in text if char not in string.punctuation]\n",
    "    nopunc = ''.join(nopunc)\n",
    "    return [word for word in nopunc.split() if word.lower() not in stopwords]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Create the Vectorizor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20535\n",
      "Wall time: 1.14 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "vocab = CountVectorizer(analyzer=text_process,stop_words=stopwords).fit(x)\n",
    "print(len(vocab.vocabulary_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Vectorization of the whole review set and and checking the sparse matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of the sparse matrix: (2713, 20535)\n",
      "Non-Zero occurences: 80558\n",
      "Density of the matrix: 0.14459862877392093\n",
      "Wall time: 111 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "x = vocab.transform(x)\n",
    "#Shape of the matrix:\n",
    "print(\"Shape of the sparse matrix: {}\".format(x.shape))\n",
    "#Non-zero occurences:\n",
    "print(\"Non-Zero occurences: {}\".format(x.nnz))\n",
    "\n",
    "# DENSITY OF THE MATRIX\n",
    "density = (x.nnz / (x.shape[0] * x.shape[1])) * 100\n",
    "print(\"Density of the matrix: {}\".format(density))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Splitting data set into training and testing set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train,x_test,y_train,y_test = train_test_split(x, y, test_size=0.2, random_state=101)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_results(y_true, y_pred, classifier_name):\n",
    "    print(\"Confusion Matrix for {}:\".format(classifier_name))\n",
    "    print(confusion_matrix(y_true, y_pred))\n",
    "    print(\"\\nClassification Report:\")\n",
    "    print(classification_report(y_true, y_pred, target_names=['1-Star', '3-Star', '5-Star']))\n",
    "    print(\"\\nScore: {}\".format(round(accuracy_score(y_true, y_pred)*100, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we are ready to build and test some classifiers.\n",
    "\n",
    "In the following we try and test some classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multinomial Naive Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix for Multinomial Naive Bayes:\n",
      "[[ 39  21  12]\n",
      " [  4  70  59]\n",
      " [  2  23 313]]\n",
      "\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "      1-Star       0.87      0.54      0.67        72\n",
      "      3-Star       0.61      0.53      0.57       133\n",
      "      5-Star       0.82      0.93      0.87       338\n",
      "\n",
      "   micro avg       0.78      0.78      0.78       543\n",
      "   macro avg       0.77      0.66      0.70       543\n",
      "weighted avg       0.77      0.78      0.77       543\n",
      "\n",
      "\n",
      "Score: 77.72\n",
      "Wall time: 12.9 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "mnb = MultinomialNB()\n",
    "mnb.fit(x_train, y_train)\n",
    "predmnb = mnb.predict(x_test)\n",
    "print_results(y_test, predmnb, \"Multinomial Naive Bayes\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\asad_\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:246: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix for Random Forest Classifier:\n",
      "[[ 23  21  28]\n",
      " [ 16  39  78]\n",
      " [  7  25 306]]\n",
      "\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "      1-Star       0.50      0.32      0.39        72\n",
      "      3-Star       0.46      0.29      0.36       133\n",
      "      5-Star       0.74      0.91      0.82       338\n",
      "\n",
      "   micro avg       0.68      0.68      0.68       543\n",
      "   macro avg       0.57      0.51      0.52       543\n",
      "weighted avg       0.64      0.68      0.65       543\n",
      "\n",
      "\n",
      "Score: 67.77\n",
      "Wall time: 409 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "if 1 == 1:\n",
    "    from sklearn.ensemble import RandomForestClassifier\n",
    "    rmfr = RandomForestClassifier()\n",
    "    rmfr.fit(x_train, y_train)\n",
    "    p = rmfr.predict(x_test)\n",
    "    print_results(y_test, p, \"Random Forest Classifier\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Tree Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix for Decision Tree:\n",
      "[[ 32  28  12]\n",
      " [ 29  54  50]\n",
      " [ 25  56 257]]\n",
      "\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "      1-Star       0.37      0.44      0.41        72\n",
      "      3-Star       0.39      0.41      0.40       133\n",
      "      5-Star       0.81      0.76      0.78       338\n",
      "\n",
      "   micro avg       0.63      0.63      0.63       543\n",
      "   macro avg       0.52      0.54      0.53       543\n",
      "weighted avg       0.65      0.63      0.64       543\n",
      "\n",
      "\n",
      "Score: 63.17\n",
      "Wall time: 309 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "if 1 == 1:\n",
    "    from sklearn.tree import DecisionTreeClassifier\n",
    "    dt = DecisionTreeClassifier()\n",
    "    dt.fit(x_train,y_train)\n",
    "    p = dt.predict(x_test)\n",
    "    print_results(y_test, p, \"Decision Tree\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Support Vector Machines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\asad_\\Anaconda3\\lib\\site-packages\\sklearn\\svm\\base.py:196: FutureWarning: The default value of gamma will change from 'auto' to 'scale' in version 0.22 to account better for unscaled features. Set gamma explicitly to 'auto' or 'scale' to avoid this warning.\n",
      "  \"avoid this warning.\", FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix for SVM:\n",
      "[[  0   0  72]\n",
      " [  0   0 133]\n",
      " [  0   0 338]]\n",
      "\n",
      "Classification Report:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\asad_\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "C:\\Users\\asad_\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "C:\\Users\\asad_\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "      1-Star       0.00      0.00      0.00        72\n",
      "      3-Star       0.00      0.00      0.00       133\n",
      "      5-Star       0.62      1.00      0.77       338\n",
      "\n",
      "   micro avg       0.62      0.62      0.62       543\n",
      "   macro avg       0.21      0.33      0.26       543\n",
      "weighted avg       0.39      0.62      0.48       543\n",
      "\n",
      "\n",
      "Score: 62.25\n",
      "Wall time: 1.18 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "if 1 == 1:\n",
    "    from sklearn.svm import SVC\n",
    "    svm = SVC(random_state=101)\n",
    "    svm.fit(x_train,y_train)\n",
    "    p = svm.predict(x_test)\n",
    "    print_results(y_test, p, \"SVM\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K - Nearest Neighbor Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix for kNN:\n",
      "[[ 14   5  53]\n",
      " [ 21  14  98]\n",
      " [ 18   1 319]]\n",
      "\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "      1-Star       0.26      0.19      0.22        72\n",
      "      3-Star       0.70      0.11      0.18       133\n",
      "      5-Star       0.68      0.94      0.79       338\n",
      "\n",
      "   micro avg       0.64      0.64      0.64       543\n",
      "   macro avg       0.55      0.41      0.40       543\n",
      "weighted avg       0.63      0.64      0.57       543\n",
      "\n",
      "\n",
      "Score: 63.9\n",
      "Wall time: 47.9 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "if 1 == 1:\n",
    "    from sklearn.neighbors import KNeighborsClassifier\n",
    "    knn = KNeighborsClassifier(n_neighbors=10)\n",
    "    knn.fit(x_train,y_train)\n",
    "    p = knn.predict(x_test)\n",
    "    print_results(y_test, p, \"kNN\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multilayer Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix for Multilayer Perceptron:\n",
      "[[ 45  17  10]\n",
      " [ 12  67  54]\n",
      " [  3  32 303]]\n",
      "\n",
      "Classification Report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "      1-Star       0.75      0.62      0.68        72\n",
      "      3-Star       0.58      0.50      0.54       133\n",
      "      5-Star       0.83      0.90      0.86       338\n",
      "\n",
      "   micro avg       0.76      0.76      0.76       543\n",
      "   macro avg       0.72      0.68      0.69       543\n",
      "weighted avg       0.75      0.76      0.76       543\n",
      "\n",
      "\n",
      "Score: 76.43\n",
      "Wall time: 59.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "if 1 == 1:\n",
    "    from sklearn.neural_network import MLPClassifier\n",
    "    mlp = MLPClassifier()\n",
    "    mlp.fit(x_train,y_train)\n",
    "    p = mlp.predict(x_test)\n",
    "    print_results(y_test, p, \"Multilayer Perceptron\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In our data scplit of 80% train, 20% test, we got the best overall accuracy and precision from *_Multilayer Perceptron_* classifier. We save this for the next steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle_dir = os.path.join('pickles/sentiment_analysis')\n",
    "cleaned_reviews_file = os.path.join(pickle_dir, 'cleaned_reviews_df.pkl')\n",
    "df_classes_file = os.path.join(pickle_dir, 'df_classes.pkl')\n",
    "vocab_file = os.path.join(pickle_dir, 'cleaned_reviews_vocab.pkl')\n",
    "transformed_sentiment_file = os.path.join(pickle_dir, 'cleaned_reviews_x_sentiment.pkl')\n",
    "classifier_file = os.path.join(pickle_dir, 'mnb_classifier.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 0 ns\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# DO Not Overwrite\n",
    "#with open(cleaned_reviews_file, 'wb') as file:\n",
    "#    pickle.dump(df, file)\n",
    "\n",
    "#with open(df_classes_file, 'wb') as file:\n",
    "#    pickle.dump(df_classes, file)\n",
    "\n",
    "#with open(vocab_file, 'wb') as file:\n",
    "#    pickle.dump(vocab, file)\n",
    "\n",
    "#with open(transformed_sentiment_file, 'wb') as file:\n",
    "#    pickle.dump(x, file)\n",
    "\n",
    "#with open(classifier_file, 'wb') as file:\n",
    "#    pickle.dump(mnb, file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [],
   "source": [
    "# END"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Aspect Segmentation\n",
    "\n",
    "In this section, we will segment out review text to segments and assign an aspect with each segments\n",
    "\n",
    "As usual, we start by importing necessary libs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports needed libs\n",
    "import gensim \n",
    "import pandas as pd\n",
    "import pickle\n",
    "import itertools\n",
    "import spacy\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the spacy library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_lg', disable=[\"ner\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the data from stored CSV file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_topic_data():\n",
    "    return pd.read_csv('processed_data/'+state_filter+'_cleaned_reviews.csv', usecols = [\"business_id\", \"name\", \"review_id\",\"text\",\"topic_text\",\"sentiment_text\",\"review_stars\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = read_topic_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Read our list of **stop words**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_words = [line.rstrip('\\n') for line in open('config/stopwords.txt', 'r', encoding='utf-8')] \n",
    "neg_words  = [line.rstrip('\\n') for line in open('config/negations.txt', 'r', encoding='utf-8')]\n",
    "stopwords = stop_words + neg_words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some **Utility functions**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Differece between list1 and List2 (list1 - list2)\n",
    "#\n",
    "def list_diff(list1,list2):\n",
    "    return list(itertools.filterfalse(set(list2).__contains__, list1)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Phrases are connect by '-'. We split phrases into words and add these words to our vocab for better accuracy\n",
    "#\n",
    "def split_phrases(str_list):\n",
    "    new_list = []\n",
    "    for tok in str_list:\n",
    "        if '-' in tok:\n",
    "            new_list += tok.split('-')\n",
    "        new_list.append(tok)\n",
    "    return list_diff(new_list,stop_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets read the pre-processed review text for **topic modelling** & **sentiment analysis**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_set = [split_phrases((u''+str(txt)).replace('.','').split()) for txt in data.topic_text]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [],
   "source": [
    "senti_set = [split_phrases((u''+str(txt)).replace('.','').split()) for txt in data.sentiment_text]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build a **Word2Vec** model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1.45 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#build vocabulary and train model\n",
    "topic_model = gensim.models.Word2Vec(\n",
    "        topic_set,\n",
    "        size=100,\n",
    "        window=10,\n",
    "        min_count=100,\n",
    "        workers=100,\n",
    "        iter=10)\n",
    "\n",
    "#  Stats\n",
    "## NC (0250 MB): Wall time:   14.8 s /   22 s\n",
    "## ALL         : 4min 34s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 4.08 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#build vocabulary and train model\n",
    "senti_model = gensim.models.Word2Vec(\n",
    "        senti_set,\n",
    "        size=100,\n",
    "        window=15,\n",
    "        min_count=100,\n",
    "        workers=100,\n",
    "        iter=20)\n",
    "\n",
    "#  Stats\n",
    "## NC (0250 MB): Wall time:   14.8 s /   22 s\n",
    "## ALL         : Wall time: 17min 19s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pickle dump the models for later use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do Not Overwrite\n",
    "#with open('pickles/aspect_topic_model.pk', 'wb') as fin:\n",
    "#    pickle.dump(topic_model, fin)\n",
    "#with open('pickles/aspect_senti_model.pk', 'wb') as fin:\n",
    "#    pickle.dump(senti_model, fin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the saved models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [],
   "source": [
    "senti_model = None\n",
    "topic_model = None\n",
    "with open('pickles/aspect_topic_model.pk', 'rb') as fin:\n",
    "    topic_model = pickle.load(fin)\n",
    "with open('pickles/aspect_senti_model.pk', 'rb') as fin:\n",
    "    senti_model = pickle.load(fin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing\n",
    "\n",
    "#### Find word similarities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('cuisine', 0.7098367810249329),\n",
       " ('thailand', 0.6114474534988403),\n",
       " ('laos', 0.607699453830719),\n",
       " ('authenticity', 0.5937446355819702),\n",
       " ('region', 0.5881584882736206)]"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_model.wv.most_similar(positive='cusine',topn=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('superior-food', 0.6407245397567749),\n",
       " ('superior-service', 0.5888615846633911)]"
      ]
     },
     "execution_count": 215,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "senti_model.wv.most_similar(positive='far',topn=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aspects\n",
    "\n",
    "##### Lets define out Aspects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 294,
   "metadata": {},
   "outputs": [],
   "source": [
    "Aspect_Terms = ['atmosphere', 'food','service','staff','location','value']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Read the **topic map** from config\n",
    "\n",
    "This mis the 50 topics we generated in **Topic Mining** section"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_all_topics_map():\n",
    "    with open('config/all_topics.txt', 'r') as f:\n",
    "         ret = {line.split(';')[0]: line.split(';')[1].replace('\\n','') for line in f.readlines()}\n",
    "    return ret"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_map = get_all_topics_map()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'0': 'reservation/seat/table/waiter',\n",
       " '1': 'atmosphere/favorite/husband/love',\n",
       " '2': 'atmosphere/food/good-service/quality',\n",
       " '3': 'crust/good-pizza/pizza/slice',\n",
       " '4': 'good/mexican-food/not-great/town',\n",
       " '5': 'bacon/burger/good-burger/shake',\n",
       " '6': 'atmosphere/definitely/great/highly-recommend',\n",
       " '7': 'chef/fish/sashimi/sushi',\n",
       " '8': 'bill/charge/dollar/spend',\n",
       " '9': 'shrimp/street/taco/tuesday',\n",
       " '10': 'burrito/chip/fish/salsa',\n",
       " '11': 'feel/like/more/much',\n",
       " '12': 'chicken/rice/waffle/wing',\n",
       " '13': 'couple/second-time/time/visit',\n",
       " '14': 'customer/great-customer/service/slow',\n",
       " '15': 'bread/pulled-pork/sandwich/turkey',\n",
       " '16': 'attentive/friendly/helpful/staff',\n",
       " '17': 'absolutely/amazing/highly-recommend/not-wait',\n",
       " '18': 'atmosphere/enjoy/experience/meal',\n",
       " '19': 'absolutely/definitely/delicious/highly-recommend',\n",
       " '20': 'great-atmosphere/great-food/great-service/highly-recommend',\n",
       " '21': 'leave/manager/people/tell',\n",
       " '22': 'dinner/lunch/special/today',\n",
       " '23': 'caesar/dress/pasta/salad',\n",
       " '24': 'dish/noodle/rice/soup',\n",
       " '25': 'hour/line/minute/wait',\n",
       " '26': 'breakfast/brunch/coffee/pancake',\n",
       " '27': 'item/menu/offer/option',\n",
       " '28': 'bartender/drink/night/refill',\n",
       " '29': 'beer/selection/wine/wing',\n",
       " '30': 'attentive/bring/check/server',\n",
       " '31': 'decent/much/not-bad/pretty',\n",
       " '32': 'awesome/definitely/highly-recommend/thank',\n",
       " '33': 'portion/price/quality/reasonable',\n",
       " '34': 'appetizer/happy-hour/special/wing',\n",
       " '35': 'clean/inside/nice/outside',\n",
       " '36': 'area/drive/location/visit',\n",
       " '37': 'cook/dinner/medium-rare/steak',\n",
       " '38': 'bacon/cheese/grill/mac-n',\n",
       " '39': 'fast/friendly-staff/super/yummy',\n",
       " '40': 'because/reason/review/star',\n",
       " '41': 'friendly-staff/good-food/good-service/recommend',\n",
       " '42': 'lobster/roll/salmon/shrimp',\n",
       " '43': 'hotel/night/room/stay',\n",
       " '44': 'bland/flavor/not-good/taste',\n",
       " '45': 'family/friend/night/recommend',\n",
       " '46': 'buffet/dessert/line/vegas',\n",
       " '47': 'dinner/excellent/highly-recommend/wife',\n",
       " '48': 'flavor/meat/sauce/wing',\n",
       " '49': 'clean/fast/fresh/tasty'}"
      ]
     },
     "execution_count": 220,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read the Aspect Description Map from the config\n",
    "\n",
    "This map relates some important terms to pur 6 Aspect terms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_aspect_map():\n",
    "    with open('config/aspect_map.txt', 'r') as f:\n",
    "        ret = {line.split(';')[0]: line.split(';')[1].replace('\\n','') for line in f.readlines()}\n",
    "    return ret"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [],
   "source": [
    "aspect_map = get_aspect_map()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Ingredient': 'food',\n",
      " 'ambience': 'atmosphere',\n",
      " 'appetizer': 'food',\n",
      " 'area': 'atmosphere',\n",
      " 'atmosphere': 'atmosphere',\n",
      " 'attentive': 'staff',\n",
      " 'attitude': 'staff',\n",
      " 'bacon': 'food',\n",
      " 'bad': 'value',\n",
      " 'bar': 'location',\n",
      " 'bartender': 'staff',\n",
      " 'beer': 'food',\n",
      " 'bill': 'value',\n",
      " 'bland': 'food',\n",
      " 'bread': 'food',\n",
      " 'breakfast': 'food',\n",
      " 'brunch': 'food',\n",
      " 'buffet': 'food',\n",
      " 'burger': 'food',\n",
      " 'burrito': 'food',\n",
      " 'caesar': 'food',\n",
      " 'cafe': 'location',\n",
      " 'cafeteria': 'location',\n",
      " 'cent': 'value',\n",
      " 'chair': 'atmosphere',\n",
      " 'charge': 'value',\n",
      " 'check': 'value',\n",
      " 'cheese': 'food',\n",
      " 'chef': 'staff',\n",
      " 'chicken': 'food',\n",
      " 'city': 'location',\n",
      " 'clean': 'atmosphere',\n",
      " 'coffee': 'food',\n",
      " 'cook': 'staff',\n",
      " 'couple': 'atmosphere',\n",
      " 'crust': 'food',\n",
      " 'customer': 'atmosphere',\n",
      " 'delicious': 'food',\n",
      " 'dessert': 'food',\n",
      " 'dinner': 'food',\n",
      " 'dish': 'food',\n",
      " 'dollar': 'value',\n",
      " 'door': 'location',\n",
      " 'downtown': 'location',\n",
      " 'dress': 'atmosphere',\n",
      " 'drink': 'food',\n",
      " 'drive': 'location',\n",
      " 'employee': 'staff',\n",
      " 'environment': 'atmosphere',\n",
      " 'fast': 'service',\n",
      " 'fish': 'food',\n",
      " 'flavor': 'food',\n",
      " 'food': 'food',\n",
      " 'friendly': 'staff',\n",
      " 'friendly-staff': 'staff',\n",
      " 'great-atmosphere': 'atmosphere',\n",
      " 'great-customer': 'service',\n",
      " 'great-food': 'service',\n",
      " 'great-service': 'service',\n",
      " 'grill': 'food',\n",
      " 'happy-hour': 'atmosphere',\n",
      " 'helpful': 'staff',\n",
      " 'hotel': 'location',\n",
      " 'hour': 'service',\n",
      " 'inside': 'location',\n",
      " 'job': 'staff',\n",
      " 'kitchen': 'atmosphere',\n",
      " 'lady': 'staff',\n",
      " 'lobster': 'food',\n",
      " 'location': 'location',\n",
      " 'lunch': 'food',\n",
      " 'manager': 'staff',\n",
      " 'meal': 'food',\n",
      " 'meat': 'food',\n",
      " 'minute': 'service',\n",
      " 'money': 'value',\n",
      " 'noodle': 'food',\n",
      " 'not-wait': 'service',\n",
      " 'offer': 'service',\n",
      " 'option': 'service',\n",
      " 'outside': 'location',\n",
      " 'overprice': 'value',\n",
      " 'pancake': 'location',\n",
      " 'parking': 'location',\n",
      " 'pasta': 'food',\n",
      " 'people': 'atmosphere',\n",
      " 'pizza': 'food',\n",
      " 'place': 'location',\n",
      " 'portion': 'service',\n",
      " 'price': 'value',\n",
      " 'quality': 'value',\n",
      " 'refill': 'service',\n",
      " 'reservation': 'service',\n",
      " 'restaurant': 'location',\n",
      " 'review': 'value',\n",
      " 'rude': 'staff',\n",
      " 'selection': 'value',\n",
      " 'server': 'staff',\n",
      " 'service': 'service',\n",
      " 'slow': 'service',\n",
      " 'special': 'value',\n",
      " 'staff': 'staff',\n",
      " 'star': 'value',\n",
      " 'street': 'location',\n",
      " 'table': 'atmosphere',\n",
      " 'tasty': 'food',\n",
      " 'toilet': 'atmosphere',\n",
      " 'town': 'location',\n",
      " 'value': 'value',\n",
      " 'vegas': 'location',\n",
      " 'visit': 'location',\n",
      " 'wait': 'service',\n",
      " 'waiter': 'staff',\n",
      " 'washroom': 'atmosphere',\n",
      " 'wine': 'food',\n",
      " 'wing': 'food',\n",
      " 'yummy': 'food'}\n"
     ]
    }
   ],
   "source": [
    "pprint(aspect_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Start segmenting text to segments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = []\n",
    "for x in topic_map.values():\n",
    "    for y in x.split('/'):\n",
    "        vocab.append(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {},
   "outputs": [],
   "source": [
    "def assign_aspect(sent,aspect_map):\n",
    "    dicty = {}\n",
    "    \n",
    "    for word in nlp(sent.lower()):\n",
    "        if word.lemma_ not in stopwords:\n",
    "            score = 0\n",
    "            aspec = None\n",
    "            for asp in aspect_map.keys():\n",
    "                try:\n",
    "                    sco = senti_model.wv.similarity(w1=asp,w2=word.lemma_)\n",
    "                    if sco > score:\n",
    "                        score=sco\n",
    "                        aspec=asp\n",
    "                        dicty[aspec]= dicty.setdefault(asp, 0) + score    \n",
    "                except:\n",
    "                    continue\n",
    "            \n",
    "    if len(dicty)==0:\n",
    "        return None\n",
    "    \n",
    "    ret = max(dicty.items(), key=lambda k: k[1])\n",
    "    if ret[1]<0.3:\n",
    "        return None\n",
    "    \n",
    "    return aspect_map[ret[0]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets performs the segmentation on a sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_size = 10000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total: 10000\n",
      ".......... Clustered [ 1001/10000 ] -    97.089 secs\n",
      ".......... Clustered [ 2001/10000 ] -    101.92 secs\n",
      ".......... Clustered [ 3001/10000 ] -    102.78 secs\n",
      ".......... Clustered [ 4001/10000 ] -    98.101 secs\n",
      ".......... Clustered [ 5001/10000 ] -    97.584 secs\n",
      ".......... Clustered [ 6001/10000 ] -    97.874 secs\n",
      ".......... Clustered [ 7001/10000 ] -    102.54 secs\n",
      ".......... Clustered [ 8001/10000 ] -    92.615 secs\n",
      ".......... Clustered [ 9001/10000 ] -    104.06 secs\n",
      ".........\n",
      " Clustered [10000 ] -    -0.000 secs\n",
      "Wall time: 16min 38s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lst = []\n",
    "start = time.time()\n",
    "end = time.time()\n",
    "total = len(data.head(sample_size))\n",
    "print(f'Total: {total}')\n",
    "\n",
    "for index,txt in enumerate(data.head(sample_size).text):\n",
    "    mydict = {}\n",
    "    for w in Aspect_Terms:\n",
    "        mydict[w] = mydict.setdefault(w, '')  \n",
    "    for x in txt.replace('\\\\n','.').replace('\\n','.').split('.'):\n",
    "        if(len(x)>0):\n",
    "            y = assign_aspect(x,aspect_map)\n",
    "            mydict[y] = mydict.setdefault(y, '') + x +'.\\n' \n",
    "    mydict.pop(None, None)\n",
    "    lst.append(mydict)\n",
    "    \n",
    "    if index%100 == 0 and index > 0:\n",
    "        print(f\".\", end='')\n",
    "            \n",
    "    if index%1000 == 0 and index > 0:\n",
    "        end = time.time()\n",
    "        print(f' Clustered [{index+1:>{5}}/{total:>{5}} ] - {str(end-start):>{9.6}} secs')\n",
    "        start = time.time()\n",
    "        \n",
    "print(f'\\n Clustered [{total:>{5}} ] - {str(end-start):>{9.6}} secs')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 295,
   "metadata": {},
   "outputs": [],
   "source": [
    "aspect_df = pd.DataFrame(lst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 296,
   "metadata": {},
   "outputs": [],
   "source": [
    "aspect_df = aspect_df.fillna('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 297,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Atmosphere</th>\n",
       "      <th>atmosphere</th>\n",
       "      <th>food</th>\n",
       "      <th>location</th>\n",
       "      <th>service</th>\n",
       "      <th>staff</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>This is a disgusting restaurant.\\n You can fi...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>This restaurant should really be inspected by...</td>\n",
       "      <td>I would have given this restaurant zero stars ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>I quietly walked back to my seat to retrieve ...</td>\n",
       "      <td>Before I picked up a plate to start in on the...</td>\n",
       "      <td>I will be calling the Champaign-Urbana Public...</td>\n",
       "      <td></td>\n",
       "      <td>I alerted a waitress, who acted as if it was ...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>The best food I want to the a lot of the buffe...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td>It's impressive how filthy this place is.\\n So...</td>\n",
       "      <td>The only worthwhile feature is that the food ...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>You would think the staff would pick up trash...</td>\n",
       "      <td>The quality ranges from poor to mediocre.\\n I...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>And they suggested that I try this buffet.\\n...</td>\n",
       "      <td>I was visiting the Champaign area with my fami...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>absolutely disgusting.\\n arrived for lunch on ...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>this is by far the worst sushi I have ever se...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td>I have no idea what sort of food was being pr...</td>\n",
       "      <td>I was hungry and against my better judgement ...</td>\n",
       "      <td>Another restaurants make the DNV list in Champ...</td>\n",
       "      <td>It was a weeknight so luckily there was no li...</td>\n",
       "      <td>Even the water that they were boiled in looke...</td>\n",
       "      <td>Restaurant review:.\\n This is bad news.\\n Okay...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>It is a Chinese buffet!! They have a lot of su...</td>\n",
       "      <td>They have a wide range of things to eat and th...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>I was looking for reviews and I saw how bad th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Not necessarily the best Chinese food in town,...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>My friend took me there.\\n</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Price was ok.\\n Everything is just within the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>If you are looking for some gourmet Chinese f...</td>\n",
       "      <td>STAY AWAY OF THIS PLACE!!!!!.\\n</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>The worst buffet I've ever had.\\n Not worth of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>But it did what it was supposed to which was ...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Good food east side of building  outside is ab...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td></td>\n",
       "      <td>Nothing really fun about this place.\\n</td>\n",
       "      <td>Kinda just \"meh\".\\n They need to change the fo...</td>\n",
       "      <td>Looks run down from the outside.\\n Would reco...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Nothing really special about this place.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>While the price may be a little steep dependin...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>The manager is also extremely unprofessional ...</td>\n",
       "      <td>I just can't shake the substandard health sta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Came here for Valentines Day when it was busy,...</td>\n",
       "      <td></td>\n",
       "      <td>Come here when it's busy, and you won't be dis...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>I would give them more stars the food is great...</td>\n",
       "      <td></td>\n",
       "      <td>im sorry but who wants to be waited on as soon...</td>\n",
       "      <td></td>\n",
       "      <td>so heres a tip if your reading this yall bette...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td></td>\n",
       "      <td>While I was there I walked into the bathroom a...</td>\n",
       "      <td>Went there last night for a family dinner.\\n T...</td>\n",
       "      <td></td>\n",
       "      <td>You have to chase down a server to get a drin...</td>\n",
       "      <td>Makes me wonder if the employees ever wash th...</td>\n",
       "      <td>I use to really like this place but over the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>This is not the worst Chinese buffet I've ever...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>I have had my fair share of Chinese buffet.\\n ...</td>\n",
       "      <td>Over the last four years, I went to this rest...</td>\n",
       "      <td></td>\n",
       "      <td>Horrible, slam dunk horrible.\\n</td>\n",
       "      <td>This was definitely the worst among all.\\n Ea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Food was very good! Also very fresh!! The staf...</td>\n",
       "      <td>They could cut in front of you and stand righ...</td>\n",
       "      <td>We never got a refill on drinks.\\n</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>My husband likes it because there is a lot of ...</td>\n",
       "      <td>Oh, and I do like \"fried biscuits\" the donut...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>As far as cleanliness, choices and prices, thi...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Our waitress was very sweet as well.\\n We did...</td>\n",
       "      <td>However, we brought a coupon that we printed ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>In general, I am not a fan of Chinese buffets....</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>If you have that as your mindset, you won't n...</td>\n",
       "      <td>There's plenty of seating here and you genera...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Had lunch here thje other day.\\n Lots of buffe...</td>\n",
       "      <td>The outside needs attention, and the restroom...</td>\n",
       "      <td>very poor shape.\\n</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>like seriously first of all I'm not small I lo...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>It's my favorite place to eat other then the f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td></td>\n",
       "      <td>I've only come here when I'm craving Chinese,...</td>\n",
       "      <td></td>\n",
       "      <td>Not the best Chinese in town, but I think my p...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Therefore, I appreciate the decent selection ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Why are you reading reviews of a buffet? There...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Rude came in8:45 close at 9 told us we could e...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>I spent the longest time walking around with ...</td>\n",
       "      <td>The Chinese restaurant life has been my famil...</td>\n",
       "      <td>Walked in, not greeted, was asked \"How many\" a...</td>\n",
       "      <td>Short answer: Avoid this place.\\n Waitress did...</td>\n",
       "      <td>Not trying to tear down this restaurant becaus...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9970</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Awesome food and beer selection.\\n Wish i coul...</td>\n",
       "      <td></td>\n",
       "      <td>Usually a wait and often run out of certain t...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9971</th>\n",
       "      <td></td>\n",
       "      <td>hands down.\\n</td>\n",
       "      <td></td>\n",
       "      <td>best barbecue joint in central Illinois.\\n a M...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>one of the best hard-to-find bourbon selectio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9972</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Awesome gourmet barbecue, would stand up to an...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9973</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>They offer appetizers, salads sandwiches and ...</td>\n",
       "      <td>My wife and I headed to the Black Dog on a rec...</td>\n",
       "      <td>Seating is very limited and they don't take r...</td>\n",
       "      <td>The folks behind the bar and the serving staff...</td>\n",
       "      <td>The prices seemed reasonable to me with nothi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9974</th>\n",
       "      <td></td>\n",
       "      <td>Its a little cramped when it gets busy but t...</td>\n",
       "      <td>Been to this place a couple of times and the f...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9975</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Smoked catfish sandwich was probably the best...</td>\n",
       "      <td>Finally was able to get in to this place; ther...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9976</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>We split way too many apps, but everything loo...</td>\n",
       "      <td>I felt like I was cheating as we drove by Hick...</td>\n",
       "      <td>But we got right in, only waiting 5 minutes ...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9977</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Great Burnt Ends and Peach BBQ Sauce.\\n Love t...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9978</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>I like their beef ribs; not quite like the du...</td>\n",
       "      <td>Visited with a friend of mine when she was in ...</td>\n",
       "      <td>Good place to go; waiting time was very long.\\n</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9979</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>I can't even begin to describe how incredibl...</td>\n",
       "      <td>Driving through central Illinois--let's be hon...</td>\n",
       "      <td></td>\n",
       "      <td>This is the kind of BBQ that makes you swear...</td>\n",
       "      <td>Next to a diploma from U of I, there is only...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9980</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>The burnt ends get all the rave but if you hav...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Pair anything with the milos spicy mustard a...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9981</th>\n",
       "      <td></td>\n",
       "      <td>Hands down the best BBQ I've had in the area, ...</td>\n",
       "      <td>My wife and I usually start with the wings, an...</td>\n",
       "      <td>There are two locations, but the Urbana spot...</td>\n",
       "      <td>You may have to put your name down and wait ...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9982</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Started out with the chips and salsa which we...</td>\n",
       "      <td>Overall, would recommend a visit.\\n</td>\n",
       "      <td>Decided to try this place out on March 3rd and...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9983</th>\n",
       "      <td></td>\n",
       "      <td>Black Dog is the best barbeque in the Champaig...</td>\n",
       "      <td>The pulled pork sandwich and burnt ends are t...</td>\n",
       "      <td></td>\n",
       "      <td>No reservations, so go at an off time, around...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9984</th>\n",
       "      <td></td>\n",
       "      <td>It's pretty big (I usually split with one oth...</td>\n",
       "      <td>Their ribs and burnt ends are my favorite.\\n I...</td>\n",
       "      <td>Might be the best BBQ joint in the Central Ill...</td>\n",
       "      <td>It's a close 2 or 3 in my BBQ book (OK Joe's ...</td>\n",
       "      <td></td>\n",
       "      <td>For just being in a college town, I wish this...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9985</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Burnt ends are to die for! Love their spicy sa...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9986</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Yummy food and drink! I am not in this part of...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9987</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>The ribs, pulled pork and collard greens wer...</td>\n",
       "      <td>Neat little bar with good BBQ.\\n</td>\n",
       "      <td>around 3 and there was very little wait (5min...</td>\n",
       "      <td>Wait staff are super.\\n</td>\n",
       "      <td>Well I had heard and read all the hype.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9988</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>It's a little disappointed that they ran out o...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9989</th>\n",
       "      <td></td>\n",
       "      <td>and then it burns and lingers for a long time.\\n</td>\n",
       "      <td>Surprisingly good BBQ in a small town!  The sa...</td>\n",
       "      <td>IF you're in Urbana-Champaign, try this spot!.\\n</td>\n",
       "      <td>I've had spicy food/sauces that burn all the ...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9990</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>favorites are rib tips, burnt ends, pulled p...</td>\n",
       "      <td>the place is always pretty crowded, which ma...</td>\n",
       "      <td>in one year i have tried everything on the men...</td>\n",
       "      <td></td>\n",
       "      <td>they also have the best draft selection as e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9991</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>I missed out on the burnt ends but the pulled...</td>\n",
       "      <td>It is a destination spot in Urbana!.\\n</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Without question, met expectations and its fin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9992</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Very delicious! Everything was very good.\\n My...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>The applesauce and meatloaf special were over...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9993</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>I could go on and on about how fantastic this ...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Read the other reviews.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9994</th>\n",
       "      <td></td>\n",
       "      <td>Nearly everything is made from scratch and has...</td>\n",
       "      <td></td>\n",
       "      <td>This is by far the best bbq in East-Central Il...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9995</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>smoke duck is running out pretty soon on Thurs...</td>\n",
       "      <td>The service here is very nice.\\n</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9996</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>I love the burnt ends, but give the salmon san...</td>\n",
       "      <td>One of the best places to eat in town!.\\n</td>\n",
       "      <td>I love all the sides (although the portions o...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9997</th>\n",
       "      <td></td>\n",
       "      <td>Hope they keep doing what they do now.\\n</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Worth the wait every time!  Brought visitors f...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9998</th>\n",
       "      <td></td>\n",
       "      <td>Definitely go here if you're in the area!.\\n</td>\n",
       "      <td>First of all, let me say I am not the biggest ...</td>\n",
       "      <td></td>\n",
       "      <td>The portions are also pretty great and the si...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9999</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>One of my favorite restaurants in town! Not on...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10000 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Atmosphere                                         atmosphere  \\\n",
       "0                                                                    \n",
       "1                 I quietly walked back to my seat to retrieve ...   \n",
       "2                                                                    \n",
       "3                It's impressive how filthy this place is.\\n So...   \n",
       "4                                                                    \n",
       "5                                                                    \n",
       "6                 I have no idea what sort of food was being pr...   \n",
       "7                                                                    \n",
       "8                                                                    \n",
       "9                                                                    \n",
       "10                                                                   \n",
       "11                                                                   \n",
       "12                                                                   \n",
       "13                          Nothing really fun about this place.\\n   \n",
       "14                                                                   \n",
       "15                                                                   \n",
       "16                                                                   \n",
       "17               While I was there I walked into the bathroom a...   \n",
       "18                                                                   \n",
       "19                                                                   \n",
       "20                                                                   \n",
       "21                                                                   \n",
       "22                                                                   \n",
       "23                                                                   \n",
       "24                                                                   \n",
       "25                                                                   \n",
       "26                I've only come here when I'm craving Chinese,...   \n",
       "27                                                                   \n",
       "28                                                                   \n",
       "29                                                                   \n",
       "...         ...                                                ...   \n",
       "9970                                                                 \n",
       "9971                                                 hands down.\\n   \n",
       "9972                                                                 \n",
       "9973                                                                 \n",
       "9974               Its a little cramped when it gets busy but t...   \n",
       "9975                                                                 \n",
       "9976                                                                 \n",
       "9977                                                                 \n",
       "9978                                                                 \n",
       "9979                                                                 \n",
       "9980                                                                 \n",
       "9981             Hands down the best BBQ I've had in the area, ...   \n",
       "9982                                                                 \n",
       "9983             Black Dog is the best barbeque in the Champaig...   \n",
       "9984              It's pretty big (I usually split with one oth...   \n",
       "9985                                                                 \n",
       "9986                                                                 \n",
       "9987                                                                 \n",
       "9988                                                                 \n",
       "9989              and then it burns and lingers for a long time.\\n   \n",
       "9990                                                                 \n",
       "9991                                                                 \n",
       "9992                                                                 \n",
       "9993                                                                 \n",
       "9994             Nearly everything is made from scratch and has...   \n",
       "9995                                                                 \n",
       "9996                                                                 \n",
       "9997                      Hope they keep doing what they do now.\\n   \n",
       "9998                  Definitely go here if you're in the area!.\\n   \n",
       "9999                                                                 \n",
       "\n",
       "                                                   food  \\\n",
       "0      This is a disgusting restaurant.\\n You can fi...   \n",
       "1      Before I picked up a plate to start in on the...   \n",
       "2     The best food I want to the a lot of the buffe...   \n",
       "3      The only worthwhile feature is that the food ...   \n",
       "4       And they suggested that I try this buffet.\\n...   \n",
       "5     absolutely disgusting.\\n arrived for lunch on ...   \n",
       "6      I was hungry and against my better judgement ...   \n",
       "7     It is a Chinese buffet!! They have a lot of su...   \n",
       "8     Not necessarily the best Chinese food in town,...   \n",
       "9                            My friend took me there.\\n   \n",
       "10     If you are looking for some gourmet Chinese f...   \n",
       "11     But it did what it was supposed to which was ...   \n",
       "12    Good food east side of building  outside is ab...   \n",
       "13    Kinda just \"meh\".\\n They need to change the fo...   \n",
       "14    While the price may be a little steep dependin...   \n",
       "15    Came here for Valentines Day when it was busy,...   \n",
       "16    I would give them more stars the food is great...   \n",
       "17    Went there last night for a family dinner.\\n T...   \n",
       "18    This is not the worst Chinese buffet I've ever...   \n",
       "19    I have had my fair share of Chinese buffet.\\n ...   \n",
       "20    Food was very good! Also very fresh!! The staf...   \n",
       "21    My husband likes it because there is a lot of ...   \n",
       "22    As far as cleanliness, choices and prices, thi...   \n",
       "23    In general, I am not a fan of Chinese buffets....   \n",
       "24    Had lunch here thje other day.\\n Lots of buffe...   \n",
       "25    like seriously first of all I'm not small I lo...   \n",
       "26                                                        \n",
       "27    Why are you reading reviews of a buffet? There...   \n",
       "28                                                        \n",
       "29     I spent the longest time walking around with ...   \n",
       "...                                                 ...   \n",
       "9970  Awesome food and beer selection.\\n Wish i coul...   \n",
       "9971                                                      \n",
       "9972                                                      \n",
       "9973   They offer appetizers, salads sandwiches and ...   \n",
       "9974  Been to this place a couple of times and the f...   \n",
       "9975   Smoked catfish sandwich was probably the best...   \n",
       "9976  We split way too many apps, but everything loo...   \n",
       "9977  Great Burnt Ends and Peach BBQ Sauce.\\n Love t...   \n",
       "9978   I like their beef ribs; not quite like the du...   \n",
       "9979    I can't even begin to describe how incredibl...   \n",
       "9980  The burnt ends get all the rave but if you hav...   \n",
       "9981  My wife and I usually start with the wings, an...   \n",
       "9982   Started out with the chips and salsa which we...   \n",
       "9983   The pulled pork sandwich and burnt ends are t...   \n",
       "9984  Their ribs and burnt ends are my favorite.\\n I...   \n",
       "9985  Burnt ends are to die for! Love their spicy sa...   \n",
       "9986  Yummy food and drink! I am not in this part of...   \n",
       "9987    The ribs, pulled pork and collard greens wer...   \n",
       "9988  It's a little disappointed that they ran out o...   \n",
       "9989  Surprisingly good BBQ in a small town!  The sa...   \n",
       "9990    favorites are rib tips, burnt ends, pulled p...   \n",
       "9991   I missed out on the burnt ends but the pulled...   \n",
       "9992  Very delicious! Everything was very good.\\n My...   \n",
       "9993  I could go on and on about how fantastic this ...   \n",
       "9994                                                      \n",
       "9995                                                      \n",
       "9996  I love the burnt ends, but give the salmon san...   \n",
       "9997                                                      \n",
       "9998  First of all, let me say I am not the biggest ...   \n",
       "9999  One of my favorite restaurants in town! Not on...   \n",
       "\n",
       "                                               location  \\\n",
       "0                                                         \n",
       "1      I will be calling the Champaign-Urbana Public...   \n",
       "2                                                         \n",
       "3                                                         \n",
       "4     I was visiting the Champaign area with my fami...   \n",
       "5                                                         \n",
       "6     Another restaurants make the DNV list in Champ...   \n",
       "7     They have a wide range of things to eat and th...   \n",
       "8                                                         \n",
       "9                                                         \n",
       "10                      STAY AWAY OF THIS PLACE!!!!!.\\n   \n",
       "11                                                        \n",
       "12                                                        \n",
       "13     Looks run down from the outside.\\n Would reco...   \n",
       "14                                                        \n",
       "15                                                        \n",
       "16                                                        \n",
       "17                                                        \n",
       "18                                                        \n",
       "19     Over the last four years, I went to this rest...   \n",
       "20     They could cut in front of you and stand righ...   \n",
       "21      Oh, and I do like \"fried biscuits\" the donut...   \n",
       "22                                                        \n",
       "23                                                        \n",
       "24     The outside needs attention, and the restroom...   \n",
       "25                                                        \n",
       "26    Not the best Chinese in town, but I think my p...   \n",
       "27                                                        \n",
       "28                                                        \n",
       "29     The Chinese restaurant life has been my famil...   \n",
       "...                                                 ...   \n",
       "9970                                                      \n",
       "9971  best barbecue joint in central Illinois.\\n a M...   \n",
       "9972  Awesome gourmet barbecue, would stand up to an...   \n",
       "9973  My wife and I headed to the Black Dog on a rec...   \n",
       "9974                                                      \n",
       "9975  Finally was able to get in to this place; ther...   \n",
       "9976  I felt like I was cheating as we drove by Hick...   \n",
       "9977                                                      \n",
       "9978  Visited with a friend of mine when she was in ...   \n",
       "9979  Driving through central Illinois--let's be hon...   \n",
       "9980                                                      \n",
       "9981    There are two locations, but the Urbana spot...   \n",
       "9982                Overall, would recommend a visit.\\n   \n",
       "9983                                                      \n",
       "9984  Might be the best BBQ joint in the Central Ill...   \n",
       "9985                                                      \n",
       "9986                                                      \n",
       "9987                   Neat little bar with good BBQ.\\n   \n",
       "9988                                                      \n",
       "9989   IF you're in Urbana-Champaign, try this spot!.\\n   \n",
       "9990    the place is always pretty crowded, which ma...   \n",
       "9991             It is a destination spot in Urbana!.\\n   \n",
       "9992                                                      \n",
       "9993                                                      \n",
       "9994  This is by far the best bbq in East-Central Il...   \n",
       "9995  smoke duck is running out pretty soon on Thurs...   \n",
       "9996          One of the best places to eat in town!.\\n   \n",
       "9997                                                      \n",
       "9998                                                      \n",
       "9999                                                      \n",
       "\n",
       "                                                service  \\\n",
       "0                                                         \n",
       "1                                                         \n",
       "2                                                         \n",
       "3                                                         \n",
       "4                                                         \n",
       "5                                                         \n",
       "6      It was a weeknight so luckily there was no li...   \n",
       "7                                                         \n",
       "8                                                         \n",
       "9                                                         \n",
       "10                                                        \n",
       "11                                                        \n",
       "12                                                        \n",
       "13                                                        \n",
       "14                                                        \n",
       "15    Come here when it's busy, and you won't be dis...   \n",
       "16    im sorry but who wants to be waited on as soon...   \n",
       "17     You have to chase down a server to get a drin...   \n",
       "18                                                        \n",
       "19                                                        \n",
       "20                   We never got a refill on drinks.\\n   \n",
       "21                                                        \n",
       "22                                                        \n",
       "23                                                        \n",
       "24                                   very poor shape.\\n   \n",
       "25                                                        \n",
       "26                                                        \n",
       "27                                                        \n",
       "28                                                        \n",
       "29    Walked in, not greeted, was asked \"How many\" a...   \n",
       "...                                                 ...   \n",
       "9970   Usually a wait and often run out of certain t...   \n",
       "9971                                                      \n",
       "9972                                                      \n",
       "9973   Seating is very limited and they don't take r...   \n",
       "9974                                                      \n",
       "9975                                                      \n",
       "9976    But we got right in, only waiting 5 minutes ...   \n",
       "9977                                                      \n",
       "9978    Good place to go; waiting time was very long.\\n   \n",
       "9979                                                      \n",
       "9980                                                      \n",
       "9981    You may have to put your name down and wait ...   \n",
       "9982  Decided to try this place out on March 3rd and...   \n",
       "9983   No reservations, so go at an off time, around...   \n",
       "9984   It's a close 2 or 3 in my BBQ book (OK Joe's ...   \n",
       "9985                                                      \n",
       "9986                                                      \n",
       "9987   around 3 and there was very little wait (5min...   \n",
       "9988                                                      \n",
       "9989   I've had spicy food/sauces that burn all the ...   \n",
       "9990  in one year i have tried everything on the men...   \n",
       "9991                                                      \n",
       "9992                                                      \n",
       "9993                                                      \n",
       "9994                                                      \n",
       "9995                   The service here is very nice.\\n   \n",
       "9996   I love all the sides (although the portions o...   \n",
       "9997  Worth the wait every time!  Brought visitors f...   \n",
       "9998   The portions are also pretty great and the si...   \n",
       "9999                                                      \n",
       "\n",
       "                                                  staff  \\\n",
       "0      This restaurant should really be inspected by...   \n",
       "1      I alerted a waitress, who acted as if it was ...   \n",
       "2                                                         \n",
       "3      You would think the staff would pick up trash...   \n",
       "4                                                         \n",
       "5                                                         \n",
       "6      Even the water that they were boiled in looke...   \n",
       "7                                                         \n",
       "8                                                         \n",
       "9                                                         \n",
       "10                                                        \n",
       "11                                                        \n",
       "12                                                        \n",
       "13                                                        \n",
       "14     The manager is also extremely unprofessional ...   \n",
       "15                                                        \n",
       "16                                                        \n",
       "17     Makes me wonder if the employees ever wash th...   \n",
       "18                                                        \n",
       "19                      Horrible, slam dunk horrible.\\n   \n",
       "20                                                        \n",
       "21                                                        \n",
       "22     Our waitress was very sweet as well.\\n We did...   \n",
       "23     If you have that as your mindset, you won't n...   \n",
       "24                                                        \n",
       "25                                                        \n",
       "26                                                        \n",
       "27                                                        \n",
       "28    Rude came in8:45 close at 9 told us we could e...   \n",
       "29    Short answer: Avoid this place.\\n Waitress did...   \n",
       "...                                                 ...   \n",
       "9970                                                      \n",
       "9971                                                      \n",
       "9972                                                      \n",
       "9973  The folks behind the bar and the serving staff...   \n",
       "9974                                                      \n",
       "9975                                                      \n",
       "9976                                                      \n",
       "9977                                                      \n",
       "9978                                                      \n",
       "9979    This is the kind of BBQ that makes you swear...   \n",
       "9980    Pair anything with the milos spicy mustard a...   \n",
       "9981                                                      \n",
       "9982                                                      \n",
       "9983                                                      \n",
       "9984                                                      \n",
       "9985                                                      \n",
       "9986                                                      \n",
       "9987                            Wait staff are super.\\n   \n",
       "9988                                                      \n",
       "9989                                                      \n",
       "9990                                                      \n",
       "9991                                                      \n",
       "9992                                                      \n",
       "9993                                                      \n",
       "9994                                                      \n",
       "9995                                                      \n",
       "9996                                                      \n",
       "9997                                                      \n",
       "9998                                                      \n",
       "9999                                                      \n",
       "\n",
       "                                                  value  \n",
       "0     I would have given this restaurant zero stars ...  \n",
       "1                                                        \n",
       "2                                                        \n",
       "3      The quality ranges from poor to mediocre.\\n I...  \n",
       "4                                                        \n",
       "5      this is by far the worst sushi I have ever se...  \n",
       "6     Restaurant review:.\\n This is bad news.\\n Okay...  \n",
       "7     I was looking for reviews and I saw how bad th...  \n",
       "8                                                        \n",
       "9      Price was ok.\\n Everything is just within the...  \n",
       "10    The worst buffet I've ever had.\\n Not worth of...  \n",
       "11                                                       \n",
       "12                                                       \n",
       "13           Nothing really special about this place.\\n  \n",
       "14     I just can't shake the substandard health sta...  \n",
       "15                                                       \n",
       "16    so heres a tip if your reading this yall bette...  \n",
       "17     I use to really like this place but over the ...  \n",
       "18                                                       \n",
       "19     This was definitely the worst among all.\\n Ea...  \n",
       "20                                                       \n",
       "21                                                       \n",
       "22     However, we brought a coupon that we printed ...  \n",
       "23     There's plenty of seating here and you genera...  \n",
       "24                                                       \n",
       "25    It's my favorite place to eat other then the f...  \n",
       "26     Therefore, I appreciate the decent selection ...  \n",
       "27                                                       \n",
       "28                                                       \n",
       "29    Not trying to tear down this restaurant becaus...  \n",
       "...                                                 ...  \n",
       "9970                                                     \n",
       "9971   one of the best hard-to-find bourbon selectio...  \n",
       "9972                                                     \n",
       "9973   The prices seemed reasonable to me with nothi...  \n",
       "9974                                                     \n",
       "9975                                                     \n",
       "9976                                                     \n",
       "9977                                                     \n",
       "9978                                                     \n",
       "9979    Next to a diploma from U of I, there is only...  \n",
       "9980                                                     \n",
       "9981                                                     \n",
       "9982                                                     \n",
       "9983                                                     \n",
       "9984   For just being in a college town, I wish this...  \n",
       "9985                                                     \n",
       "9986                                                     \n",
       "9987          Well I had heard and read all the hype.\\n  \n",
       "9988                                                     \n",
       "9989                                                     \n",
       "9990    they also have the best draft selection as e...  \n",
       "9991  Without question, met expectations and its fin...  \n",
       "9992   The applesauce and meatloaf special were over...  \n",
       "9993                          Read the other reviews.\\n  \n",
       "9994                                                     \n",
       "9995                                                     \n",
       "9996                                                     \n",
       "9997                                                     \n",
       "9998                                                     \n",
       "9999                                                     \n",
       "\n",
       "[10000 rows x 7 columns]"
      ]
     },
     "execution_count": 297,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aspect_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [],
   "source": [
    "aspects_df = data[['review_id','text','review_stars']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\asad_\\AppData\\Roaming\\Python\\Python37\\site-packages\\ipykernel_launcher.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "for asp in Aspect_Terms:\n",
    "    aspects_df[asp] = aspect_df[asp]\n",
    "aspects_df = aspects_df.fillna('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review_id</th>\n",
       "      <th>text</th>\n",
       "      <th>review_stars</th>\n",
       "      <th>Atmosphere</th>\n",
       "      <th>food</th>\n",
       "      <th>service</th>\n",
       "      <th>staff</th>\n",
       "      <th>location</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8MTptiOpUeuPUFZgtfk9Vw</td>\n",
       "      <td>I would have given this restaurant zero stars ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td>This is a disgusting restaurant.\\n You can fi...</td>\n",
       "      <td></td>\n",
       "      <td>This restaurant should really be inspected by...</td>\n",
       "      <td></td>\n",
       "      <td>I would have given this restaurant zero stars ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ofpfzn8LV4nJ2pE6IlTVdg</td>\n",
       "      <td>So...no. Just...no. Before I picked up a plate...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td>Before I picked up a plate to start in on the...</td>\n",
       "      <td></td>\n",
       "      <td>I alerted a waitress, who acted as if it was ...</td>\n",
       "      <td>I will be calling the Champaign-Urbana Public...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>O3yApCw95tbA0kodflQrpA</td>\n",
       "      <td>The best food I want to the a lot of the buffe...</td>\n",
       "      <td>5.0</td>\n",
       "      <td></td>\n",
       "      <td>The best food I want to the a lot of the buffe...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>fFTVZE7EENdl66SX9cgc0A</td>\n",
       "      <td>It's impressive how filthy this place is. Some...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td>The only worthwhile feature is that the food ...</td>\n",
       "      <td></td>\n",
       "      <td>You would think the staff would pick up trash...</td>\n",
       "      <td></td>\n",
       "      <td>The quality ranges from poor to mediocre.\\n I...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>pDlIw6DLLjZ5G1vJnMzr9A</td>\n",
       "      <td>I was visiting the Champaign area with my fami...</td>\n",
       "      <td>5.0</td>\n",
       "      <td></td>\n",
       "      <td>And they suggested that I try this buffet.\\n...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>I was visiting the Champaign area with my fami...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>xwJizYjeafDbsJVmPmBSgQ</td>\n",
       "      <td>absolutely disgusting. arrived for lunch on a ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td>absolutely disgusting.\\n arrived for lunch on ...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>this is by far the worst sushi I have ever se...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>mvAvUVOkSuEr6EHZ97sg0Q</td>\n",
       "      <td>Restaurant review:\\nAnother restaurants make t...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td>I was hungry and against my better judgement ...</td>\n",
       "      <td>It was a weeknight so luckily there was no li...</td>\n",
       "      <td>Even the water that they were boiled in looke...</td>\n",
       "      <td>Another restaurants make the DNV list in Champ...</td>\n",
       "      <td>Restaurant review:.\\n This is bad news.\\n Okay...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ticGaTnIjhkijFqWmww5uw</td>\n",
       "      <td>I was looking for reviews and I saw how bad th...</td>\n",
       "      <td>5.0</td>\n",
       "      <td></td>\n",
       "      <td>It is a Chinese buffet!! They have a lot of su...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>They have a wide range of things to eat and th...</td>\n",
       "      <td>I was looking for reviews and I saw how bad th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>4ftuoeacaUQ1EyiW_xt1QA</td>\n",
       "      <td>Not necessarily the best Chinese food in town,...</td>\n",
       "      <td>4.0</td>\n",
       "      <td></td>\n",
       "      <td>Not necessarily the best Chinese food in town,...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>aDCXNKvZpoD9s4GICWdqUg</td>\n",
       "      <td>My friend took me there. Price was ok. Everyth...</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "      <td>My friend took me there.\\n</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Price was ok.\\n Everything is just within the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>r-4_NT2Ko232o2jb57xamQ</td>\n",
       "      <td>The worst buffet I've ever had. Not worth of t...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td>If you are looking for some gourmet Chinese f...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>STAY AWAY OF THIS PLACE!!!!!.\\n</td>\n",
       "      <td>The worst buffet I've ever had.\\n Not worth of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>yLwPBITQnIe3MMU6Gkm1Tw</td>\n",
       "      <td>This place was just ok. But it did what it was...</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "      <td>But it did what it was supposed to which was ...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>wRfxg0eGbauSElkZy4AObQ</td>\n",
       "      <td>Good food east side of building  outside is ab...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td>Good food east side of building  outside is ab...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>QDaM70gzI4XLZFHrPffDVA</td>\n",
       "      <td>Kinda just \"meh\". Nothing really fun about thi...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td>Kinda just \"meh\".\\n They need to change the fo...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Looks run down from the outside.\\n Would reco...</td>\n",
       "      <td>Nothing really special about this place.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>JTISByuUwaZwfmTH2nLVLQ</td>\n",
       "      <td>While the price may be a little steep dependin...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td>While the price may be a little steep dependin...</td>\n",
       "      <td></td>\n",
       "      <td>The manager is also extremely unprofessional ...</td>\n",
       "      <td></td>\n",
       "      <td>I just can't shake the substandard health sta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>tBmosR1h73EC3zkbztkZ4Q</td>\n",
       "      <td>Came here for Valentines Day when it was busy,...</td>\n",
       "      <td>5.0</td>\n",
       "      <td></td>\n",
       "      <td>Came here for Valentines Day when it was busy,...</td>\n",
       "      <td>Come here when it's busy, and you won't be dis...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>TlUl5Mp7DXMLp9oCvsu_9g</td>\n",
       "      <td>I would give them more stars the food is great...</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "      <td>I would give them more stars the food is great...</td>\n",
       "      <td>im sorry but who wants to be waited on as soon...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>so heres a tip if your reading this yall bette...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>ca3Weqo3gE5BfWySyYyYqw</td>\n",
       "      <td>Went there last night for a family dinner. I u...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td>Went there last night for a family dinner.\\n T...</td>\n",
       "      <td>You have to chase down a server to get a drin...</td>\n",
       "      <td>Makes me wonder if the employees ever wash th...</td>\n",
       "      <td></td>\n",
       "      <td>I use to really like this place but over the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>85JphyjHRvq9SsEz5Q7e2A</td>\n",
       "      <td>This is not the worst Chinese buffet I've ever...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td>This is not the worst Chinese buffet I've ever...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>t1Rm0I05qmAdYSWfQwsmHg</td>\n",
       "      <td>I have had my fair share of Chinese buffet. Th...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td>I have had my fair share of Chinese buffet.\\n ...</td>\n",
       "      <td></td>\n",
       "      <td>Horrible, slam dunk horrible.\\n</td>\n",
       "      <td>Over the last four years, I went to this rest...</td>\n",
       "      <td>This was definitely the worst among all.\\n Ea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>qsLHeRDEq9E_Cj-Bz_59qQ</td>\n",
       "      <td>Food was very good! Also very fresh!! The staf...</td>\n",
       "      <td>4.0</td>\n",
       "      <td></td>\n",
       "      <td>Food was very good! Also very fresh!! The staf...</td>\n",
       "      <td>We never got a refill on drinks.\\n</td>\n",
       "      <td></td>\n",
       "      <td>They could cut in front of you and stand righ...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>K6J0JPnPoEdbDPeRmfBbxw</td>\n",
       "      <td>My husband likes it because there is a lot of ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td>My husband likes it because there is a lot of ...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Oh, and I do like \"fried biscuits\" the donut...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>YpzcpYhvMopaCrVcnAmQSQ</td>\n",
       "      <td>As far as cleanliness, choices and prices, thi...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td>As far as cleanliness, choices and prices, thi...</td>\n",
       "      <td></td>\n",
       "      <td>Our waitress was very sweet as well.\\n We did...</td>\n",
       "      <td></td>\n",
       "      <td>However, we brought a coupon that we printed ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>ucKQ8OeJyZlLeyhruHM3bQ</td>\n",
       "      <td>In general, I am not a fan of Chinese buffets....</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "      <td>In general, I am not a fan of Chinese buffets....</td>\n",
       "      <td></td>\n",
       "      <td>If you have that as your mindset, you won't n...</td>\n",
       "      <td></td>\n",
       "      <td>There's plenty of seating here and you genera...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>6Hhm7bjMyZecqU5kZZIyBQ</td>\n",
       "      <td>Had lunch here thje other day. Lots of buffet ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td>Had lunch here thje other day.\\n Lots of buffe...</td>\n",
       "      <td>very poor shape.\\n</td>\n",
       "      <td></td>\n",
       "      <td>The outside needs attention, and the restroom...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>nfIH54SxLB77fcQpFTKbOw</td>\n",
       "      <td>It's my favorite place to eat other then the f...</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "      <td>like seriously first of all I'm not small I lo...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>It's my favorite place to eat other then the f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>x3b3EsZJEhhSjsz8abD8fQ</td>\n",
       "      <td>Not the best Chinese in town, but I think my p...</td>\n",
       "      <td>4.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Not the best Chinese in town, but I think my p...</td>\n",
       "      <td>Therefore, I appreciate the decent selection ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Sp82eEDhnOivtxKSQFfAIA</td>\n",
       "      <td>Why are you reading reviews of a buffet? There...</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "      <td>Why are you reading reviews of a buffet? There...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>wKa4a-vfzH_I7-joGAhLFw</td>\n",
       "      <td>Rude came in8:45 close at 9 told us we could e...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Rude came in8:45 close at 9 told us we could e...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>jFlrexBLigmpybhXcqtGzA</td>\n",
       "      <td>Short answer: Avoid this place. The Chinese re...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td>I spent the longest time walking around with ...</td>\n",
       "      <td>Walked in, not greeted, was asked \"How many\" a...</td>\n",
       "      <td>Short answer: Avoid this place.\\n Waitress did...</td>\n",
       "      <td>The Chinese restaurant life has been my famil...</td>\n",
       "      <td>Not trying to tear down this restaurant becaus...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19190</th>\n",
       "      <td>qCX-c0kPNh2Up9CpJkZLow</td>\n",
       "      <td>Friendly staff, good bar food and great drank ...</td>\n",
       "      <td>5.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19191</th>\n",
       "      <td>1KbcRXRxMZ4x89HYMTCACw</td>\n",
       "      <td>I have been here twice on a Monday afternoon a...</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19192</th>\n",
       "      <td>HYIFnjHnFRykeMS6cdP95g</td>\n",
       "      <td>Food: Ordered a Prime Rib Sandwich, and by no ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19193</th>\n",
       "      <td>W2NtkgFFYuC3uYvieFNzSg</td>\n",
       "      <td>I've had a mix of experiences here but one thi...</td>\n",
       "      <td>4.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19194</th>\n",
       "      <td>mRtHozxg42Uxxi3zmKynYA</td>\n",
       "      <td>An overall pretty good bar in downtown Champai...</td>\n",
       "      <td>4.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19195</th>\n",
       "      <td>_DQzRTv2kR9l9pYyI2_eBA</td>\n",
       "      <td>Decent food and service, but I can honestly sa...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19196</th>\n",
       "      <td>sbmErmzLymQs-pMqsJg6MQ</td>\n",
       "      <td>Owner reached out to me after I wrote this rev...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19197</th>\n",
       "      <td>512VpYv0w5OFgwJoFVCc8Q</td>\n",
       "      <td>Over the course of a week, there comes a time ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19198</th>\n",
       "      <td>NwzSuCpvTEn9hcFbv7nzGw</td>\n",
       "      <td>This is a bar that i have been to a lot of net...</td>\n",
       "      <td>4.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19199</th>\n",
       "      <td>mruCbb3pjDxy7Zy6ybJMTg</td>\n",
       "      <td>In general, I like Guidos.  They have a good s...</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19200</th>\n",
       "      <td>7ksbP0FSK2RB4kpQuXndvA</td>\n",
       "      <td>Great place for eating and enjoying some reall...</td>\n",
       "      <td>5.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19201</th>\n",
       "      <td>X5vbDohuLAZ4Jl_gkapiHA</td>\n",
       "      <td>Going a few more times since last review, I th...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19202</th>\n",
       "      <td>3QkKg1yTovNtlL-T6Qv5UQ</td>\n",
       "      <td>The food is very good, but I've been to lunch ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19203</th>\n",
       "      <td>GrW96p-AjJWYN7TWF05pbw</td>\n",
       "      <td>I like their decor, and I like their onion rin...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19204</th>\n",
       "      <td>mTWz1did2_9yPfzqbuKn_w</td>\n",
       "      <td>If you are in the mood for nachos, head here. ...</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19205</th>\n",
       "      <td>q7LlT84AkXJwoadj_fslCg</td>\n",
       "      <td>This is not a place to write home about for th...</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19206</th>\n",
       "      <td>ibm1FOvNxkytJbIF2nzDSw</td>\n",
       "      <td>What comes to mind when I think of Guido's? Th...</td>\n",
       "      <td>4.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19207</th>\n",
       "      <td>GosGbgFc6ua_IClskgLG7Q</td>\n",
       "      <td>Outstanding food for the price, but the staff ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19208</th>\n",
       "      <td>ic1hB_83A0Pwa8aeU-N1QA</td>\n",
       "      <td>Food just never up to par. Walk across the str...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19209</th>\n",
       "      <td>llINukf2uGHLm0LW-yKbIA</td>\n",
       "      <td>Very disappointing tonight. I know it's pub fo...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19210</th>\n",
       "      <td>RWJwyzSInsGEmXfeo0GfLA</td>\n",
       "      <td>Had a grilled ham &amp; cheese sandwich that was g...</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19211</th>\n",
       "      <td>Qir-QA3QA_djQVFfiWbIFA</td>\n",
       "      <td>Do not open a tab here.. very dishonest, quest...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19212</th>\n",
       "      <td>wUQngfXMmiSaGTKbjrhowg</td>\n",
       "      <td>Guido's is ok. \\nPros: great burgers (even the...</td>\n",
       "      <td>3.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19213</th>\n",
       "      <td>refeAKpRj7EQfU2eIvIFJQ</td>\n",
       "      <td>I am very impressed. After a bad experience at...</td>\n",
       "      <td>5.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19214</th>\n",
       "      <td>ZGTjk4wT7FbKa5UdYRSmLg</td>\n",
       "      <td>I keep coming back over and over for a good dr...</td>\n",
       "      <td>5.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19215</th>\n",
       "      <td>8ePzpg9tHddHi9WMKehrAA</td>\n",
       "      <td>The burgers are good! They claim double patty ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19216</th>\n",
       "      <td>5GCqbhpFF_omSFsYxbRVUg</td>\n",
       "      <td>It did have one good experience, but last nigh...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19217</th>\n",
       "      <td>sVztDMZzAsAP-4qTATTF2A</td>\n",
       "      <td>I go to Guido's occasionally and always have a...</td>\n",
       "      <td>5.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19218</th>\n",
       "      <td>HuuKbMdougFJwvZ8H8Pz9g</td>\n",
       "      <td>The management went out of their way to make u...</td>\n",
       "      <td>5.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19219</th>\n",
       "      <td>FeudNHXnbF5LCedgZLJdeQ</td>\n",
       "      <td>A bar with bar food.  There were four of us.  ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>19220 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                    review_id  \\\n",
       "0      8MTptiOpUeuPUFZgtfk9Vw   \n",
       "1      ofpfzn8LV4nJ2pE6IlTVdg   \n",
       "2      O3yApCw95tbA0kodflQrpA   \n",
       "3      fFTVZE7EENdl66SX9cgc0A   \n",
       "4      pDlIw6DLLjZ5G1vJnMzr9A   \n",
       "5      xwJizYjeafDbsJVmPmBSgQ   \n",
       "6      mvAvUVOkSuEr6EHZ97sg0Q   \n",
       "7      ticGaTnIjhkijFqWmww5uw   \n",
       "8      4ftuoeacaUQ1EyiW_xt1QA   \n",
       "9      aDCXNKvZpoD9s4GICWdqUg   \n",
       "10     r-4_NT2Ko232o2jb57xamQ   \n",
       "11     yLwPBITQnIe3MMU6Gkm1Tw   \n",
       "12     wRfxg0eGbauSElkZy4AObQ   \n",
       "13     QDaM70gzI4XLZFHrPffDVA   \n",
       "14     JTISByuUwaZwfmTH2nLVLQ   \n",
       "15     tBmosR1h73EC3zkbztkZ4Q   \n",
       "16     TlUl5Mp7DXMLp9oCvsu_9g   \n",
       "17     ca3Weqo3gE5BfWySyYyYqw   \n",
       "18     85JphyjHRvq9SsEz5Q7e2A   \n",
       "19     t1Rm0I05qmAdYSWfQwsmHg   \n",
       "20     qsLHeRDEq9E_Cj-Bz_59qQ   \n",
       "21     K6J0JPnPoEdbDPeRmfBbxw   \n",
       "22     YpzcpYhvMopaCrVcnAmQSQ   \n",
       "23     ucKQ8OeJyZlLeyhruHM3bQ   \n",
       "24     6Hhm7bjMyZecqU5kZZIyBQ   \n",
       "25     nfIH54SxLB77fcQpFTKbOw   \n",
       "26     x3b3EsZJEhhSjsz8abD8fQ   \n",
       "27     Sp82eEDhnOivtxKSQFfAIA   \n",
       "28     wKa4a-vfzH_I7-joGAhLFw   \n",
       "29     jFlrexBLigmpybhXcqtGzA   \n",
       "...                       ...   \n",
       "19190  qCX-c0kPNh2Up9CpJkZLow   \n",
       "19191  1KbcRXRxMZ4x89HYMTCACw   \n",
       "19192  HYIFnjHnFRykeMS6cdP95g   \n",
       "19193  W2NtkgFFYuC3uYvieFNzSg   \n",
       "19194  mRtHozxg42Uxxi3zmKynYA   \n",
       "19195  _DQzRTv2kR9l9pYyI2_eBA   \n",
       "19196  sbmErmzLymQs-pMqsJg6MQ   \n",
       "19197  512VpYv0w5OFgwJoFVCc8Q   \n",
       "19198  NwzSuCpvTEn9hcFbv7nzGw   \n",
       "19199  mruCbb3pjDxy7Zy6ybJMTg   \n",
       "19200  7ksbP0FSK2RB4kpQuXndvA   \n",
       "19201  X5vbDohuLAZ4Jl_gkapiHA   \n",
       "19202  3QkKg1yTovNtlL-T6Qv5UQ   \n",
       "19203  GrW96p-AjJWYN7TWF05pbw   \n",
       "19204  mTWz1did2_9yPfzqbuKn_w   \n",
       "19205  q7LlT84AkXJwoadj_fslCg   \n",
       "19206  ibm1FOvNxkytJbIF2nzDSw   \n",
       "19207  GosGbgFc6ua_IClskgLG7Q   \n",
       "19208  ic1hB_83A0Pwa8aeU-N1QA   \n",
       "19209  llINukf2uGHLm0LW-yKbIA   \n",
       "19210  RWJwyzSInsGEmXfeo0GfLA   \n",
       "19211  Qir-QA3QA_djQVFfiWbIFA   \n",
       "19212  wUQngfXMmiSaGTKbjrhowg   \n",
       "19213  refeAKpRj7EQfU2eIvIFJQ   \n",
       "19214  ZGTjk4wT7FbKa5UdYRSmLg   \n",
       "19215  8ePzpg9tHddHi9WMKehrAA   \n",
       "19216  5GCqbhpFF_omSFsYxbRVUg   \n",
       "19217  sVztDMZzAsAP-4qTATTF2A   \n",
       "19218  HuuKbMdougFJwvZ8H8Pz9g   \n",
       "19219  FeudNHXnbF5LCedgZLJdeQ   \n",
       "\n",
       "                                                    text  review_stars  \\\n",
       "0      I would have given this restaurant zero stars ...           1.0   \n",
       "1      So...no. Just...no. Before I picked up a plate...           1.0   \n",
       "2      The best food I want to the a lot of the buffe...           5.0   \n",
       "3      It's impressive how filthy this place is. Some...           1.0   \n",
       "4      I was visiting the Champaign area with my fami...           5.0   \n",
       "5      absolutely disgusting. arrived for lunch on a ...           1.0   \n",
       "6      Restaurant review:\\nAnother restaurants make t...           1.0   \n",
       "7      I was looking for reviews and I saw how bad th...           5.0   \n",
       "8      Not necessarily the best Chinese food in town,...           4.0   \n",
       "9      My friend took me there. Price was ok. Everyth...           3.0   \n",
       "10     The worst buffet I've ever had. Not worth of t...           1.0   \n",
       "11     This place was just ok. But it did what it was...           3.0   \n",
       "12     Good food east side of building  outside is ab...           1.0   \n",
       "13     Kinda just \"meh\". Nothing really fun about thi...           2.0   \n",
       "14     While the price may be a little steep dependin...           2.0   \n",
       "15     Came here for Valentines Day when it was busy,...           5.0   \n",
       "16     I would give them more stars the food is great...           3.0   \n",
       "17     Went there last night for a family dinner. I u...           1.0   \n",
       "18     This is not the worst Chinese buffet I've ever...           2.0   \n",
       "19     I have had my fair share of Chinese buffet. Th...           1.0   \n",
       "20     Food was very good! Also very fresh!! The staf...           4.0   \n",
       "21     My husband likes it because there is a lot of ...           2.0   \n",
       "22     As far as cleanliness, choices and prices, thi...           2.0   \n",
       "23     In general, I am not a fan of Chinese buffets....           3.0   \n",
       "24     Had lunch here thje other day. Lots of buffet ...           2.0   \n",
       "25     It's my favorite place to eat other then the f...           3.0   \n",
       "26     Not the best Chinese in town, but I think my p...           4.0   \n",
       "27     Why are you reading reviews of a buffet? There...           3.0   \n",
       "28     Rude came in8:45 close at 9 told us we could e...           1.0   \n",
       "29     Short answer: Avoid this place. The Chinese re...           1.0   \n",
       "...                                                  ...           ...   \n",
       "19190  Friendly staff, good bar food and great drank ...           5.0   \n",
       "19191  I have been here twice on a Monday afternoon a...           3.0   \n",
       "19192  Food: Ordered a Prime Rib Sandwich, and by no ...           4.0   \n",
       "19193  I've had a mix of experiences here but one thi...           4.0   \n",
       "19194  An overall pretty good bar in downtown Champai...           4.0   \n",
       "19195  Decent food and service, but I can honestly sa...           2.0   \n",
       "19196  Owner reached out to me after I wrote this rev...           1.0   \n",
       "19197  Over the course of a week, there comes a time ...           2.0   \n",
       "19198  This is a bar that i have been to a lot of net...           4.0   \n",
       "19199  In general, I like Guidos.  They have a good s...           3.0   \n",
       "19200  Great place for eating and enjoying some reall...           5.0   \n",
       "19201  Going a few more times since last review, I th...           2.0   \n",
       "19202  The food is very good, but I've been to lunch ...           2.0   \n",
       "19203  I like their decor, and I like their onion rin...           2.0   \n",
       "19204  If you are in the mood for nachos, head here. ...           3.0   \n",
       "19205  This is not a place to write home about for th...           3.0   \n",
       "19206  What comes to mind when I think of Guido's? Th...           4.0   \n",
       "19207  Outstanding food for the price, but the staff ...           4.0   \n",
       "19208  Food just never up to par. Walk across the str...           2.0   \n",
       "19209  Very disappointing tonight. I know it's pub fo...           1.0   \n",
       "19210  Had a grilled ham & cheese sandwich that was g...           3.0   \n",
       "19211  Do not open a tab here.. very dishonest, quest...           1.0   \n",
       "19212  Guido's is ok. \\nPros: great burgers (even the...           3.0   \n",
       "19213  I am very impressed. After a bad experience at...           5.0   \n",
       "19214  I keep coming back over and over for a good dr...           5.0   \n",
       "19215  The burgers are good! They claim double patty ...           2.0   \n",
       "19216  It did have one good experience, but last nigh...           1.0   \n",
       "19217  I go to Guido's occasionally and always have a...           5.0   \n",
       "19218  The management went out of their way to make u...           5.0   \n",
       "19219  A bar with bar food.  There were four of us.  ...           1.0   \n",
       "\n",
       "      Atmosphere                                               food  \\\n",
       "0                  This is a disgusting restaurant.\\n You can fi...   \n",
       "1                  Before I picked up a plate to start in on the...   \n",
       "2                 The best food I want to the a lot of the buffe...   \n",
       "3                  The only worthwhile feature is that the food ...   \n",
       "4                   And they suggested that I try this buffet.\\n...   \n",
       "5                 absolutely disgusting.\\n arrived for lunch on ...   \n",
       "6                  I was hungry and against my better judgement ...   \n",
       "7                 It is a Chinese buffet!! They have a lot of su...   \n",
       "8                 Not necessarily the best Chinese food in town,...   \n",
       "9                                        My friend took me there.\\n   \n",
       "10                 If you are looking for some gourmet Chinese f...   \n",
       "11                 But it did what it was supposed to which was ...   \n",
       "12                Good food east side of building  outside is ab...   \n",
       "13                Kinda just \"meh\".\\n They need to change the fo...   \n",
       "14                While the price may be a little steep dependin...   \n",
       "15                Came here for Valentines Day when it was busy,...   \n",
       "16                I would give them more stars the food is great...   \n",
       "17                Went there last night for a family dinner.\\n T...   \n",
       "18                This is not the worst Chinese buffet I've ever...   \n",
       "19                I have had my fair share of Chinese buffet.\\n ...   \n",
       "20                Food was very good! Also very fresh!! The staf...   \n",
       "21                My husband likes it because there is a lot of ...   \n",
       "22                As far as cleanliness, choices and prices, thi...   \n",
       "23                In general, I am not a fan of Chinese buffets....   \n",
       "24                Had lunch here thje other day.\\n Lots of buffe...   \n",
       "25                like seriously first of all I'm not small I lo...   \n",
       "26                                                                    \n",
       "27                Why are you reading reviews of a buffet? There...   \n",
       "28                                                                    \n",
       "29                 I spent the longest time walking around with ...   \n",
       "...          ...                                                ...   \n",
       "19190                                                                 \n",
       "19191                                                                 \n",
       "19192                                                                 \n",
       "19193                                                                 \n",
       "19194                                                                 \n",
       "19195                                                                 \n",
       "19196                                                                 \n",
       "19197                                                                 \n",
       "19198                                                                 \n",
       "19199                                                                 \n",
       "19200                                                                 \n",
       "19201                                                                 \n",
       "19202                                                                 \n",
       "19203                                                                 \n",
       "19204                                                                 \n",
       "19205                                                                 \n",
       "19206                                                                 \n",
       "19207                                                                 \n",
       "19208                                                                 \n",
       "19209                                                                 \n",
       "19210                                                                 \n",
       "19211                                                                 \n",
       "19212                                                                 \n",
       "19213                                                                 \n",
       "19214                                                                 \n",
       "19215                                                                 \n",
       "19216                                                                 \n",
       "19217                                                                 \n",
       "19218                                                                 \n",
       "19219                                                                 \n",
       "\n",
       "                                                 service  \\\n",
       "0                                                          \n",
       "1                                                          \n",
       "2                                                          \n",
       "3                                                          \n",
       "4                                                          \n",
       "5                                                          \n",
       "6       It was a weeknight so luckily there was no li...   \n",
       "7                                                          \n",
       "8                                                          \n",
       "9                                                          \n",
       "10                                                         \n",
       "11                                                         \n",
       "12                                                         \n",
       "13                                                         \n",
       "14                                                         \n",
       "15     Come here when it's busy, and you won't be dis...   \n",
       "16     im sorry but who wants to be waited on as soon...   \n",
       "17      You have to chase down a server to get a drin...   \n",
       "18                                                         \n",
       "19                                                         \n",
       "20                    We never got a refill on drinks.\\n   \n",
       "21                                                         \n",
       "22                                                         \n",
       "23                                                         \n",
       "24                                    very poor shape.\\n   \n",
       "25                                                         \n",
       "26                                                         \n",
       "27                                                         \n",
       "28                                                         \n",
       "29     Walked in, not greeted, was asked \"How many\" a...   \n",
       "...                                                  ...   \n",
       "19190                                                      \n",
       "19191                                                      \n",
       "19192                                                      \n",
       "19193                                                      \n",
       "19194                                                      \n",
       "19195                                                      \n",
       "19196                                                      \n",
       "19197                                                      \n",
       "19198                                                      \n",
       "19199                                                      \n",
       "19200                                                      \n",
       "19201                                                      \n",
       "19202                                                      \n",
       "19203                                                      \n",
       "19204                                                      \n",
       "19205                                                      \n",
       "19206                                                      \n",
       "19207                                                      \n",
       "19208                                                      \n",
       "19209                                                      \n",
       "19210                                                      \n",
       "19211                                                      \n",
       "19212                                                      \n",
       "19213                                                      \n",
       "19214                                                      \n",
       "19215                                                      \n",
       "19216                                                      \n",
       "19217                                                      \n",
       "19218                                                      \n",
       "19219                                                      \n",
       "\n",
       "                                                   staff  \\\n",
       "0       This restaurant should really be inspected by...   \n",
       "1       I alerted a waitress, who acted as if it was ...   \n",
       "2                                                          \n",
       "3       You would think the staff would pick up trash...   \n",
       "4                                                          \n",
       "5                                                          \n",
       "6       Even the water that they were boiled in looke...   \n",
       "7                                                          \n",
       "8                                                          \n",
       "9                                                          \n",
       "10                                                         \n",
       "11                                                         \n",
       "12                                                         \n",
       "13                                                         \n",
       "14      The manager is also extremely unprofessional ...   \n",
       "15                                                         \n",
       "16                                                         \n",
       "17      Makes me wonder if the employees ever wash th...   \n",
       "18                                                         \n",
       "19                       Horrible, slam dunk horrible.\\n   \n",
       "20                                                         \n",
       "21                                                         \n",
       "22      Our waitress was very sweet as well.\\n We did...   \n",
       "23      If you have that as your mindset, you won't n...   \n",
       "24                                                         \n",
       "25                                                         \n",
       "26                                                         \n",
       "27                                                         \n",
       "28     Rude came in8:45 close at 9 told us we could e...   \n",
       "29     Short answer: Avoid this place.\\n Waitress did...   \n",
       "...                                                  ...   \n",
       "19190                                                      \n",
       "19191                                                      \n",
       "19192                                                      \n",
       "19193                                                      \n",
       "19194                                                      \n",
       "19195                                                      \n",
       "19196                                                      \n",
       "19197                                                      \n",
       "19198                                                      \n",
       "19199                                                      \n",
       "19200                                                      \n",
       "19201                                                      \n",
       "19202                                                      \n",
       "19203                                                      \n",
       "19204                                                      \n",
       "19205                                                      \n",
       "19206                                                      \n",
       "19207                                                      \n",
       "19208                                                      \n",
       "19209                                                      \n",
       "19210                                                      \n",
       "19211                                                      \n",
       "19212                                                      \n",
       "19213                                                      \n",
       "19214                                                      \n",
       "19215                                                      \n",
       "19216                                                      \n",
       "19217                                                      \n",
       "19218                                                      \n",
       "19219                                                      \n",
       "\n",
       "                                                location  \\\n",
       "0                                                          \n",
       "1       I will be calling the Champaign-Urbana Public...   \n",
       "2                                                          \n",
       "3                                                          \n",
       "4      I was visiting the Champaign area with my fami...   \n",
       "5                                                          \n",
       "6      Another restaurants make the DNV list in Champ...   \n",
       "7      They have a wide range of things to eat and th...   \n",
       "8                                                          \n",
       "9                                                          \n",
       "10                       STAY AWAY OF THIS PLACE!!!!!.\\n   \n",
       "11                                                         \n",
       "12                                                         \n",
       "13      Looks run down from the outside.\\n Would reco...   \n",
       "14                                                         \n",
       "15                                                         \n",
       "16                                                         \n",
       "17                                                         \n",
       "18                                                         \n",
       "19      Over the last four years, I went to this rest...   \n",
       "20      They could cut in front of you and stand righ...   \n",
       "21       Oh, and I do like \"fried biscuits\" the donut...   \n",
       "22                                                         \n",
       "23                                                         \n",
       "24      The outside needs attention, and the restroom...   \n",
       "25                                                         \n",
       "26     Not the best Chinese in town, but I think my p...   \n",
       "27                                                         \n",
       "28                                                         \n",
       "29      The Chinese restaurant life has been my famil...   \n",
       "...                                                  ...   \n",
       "19190                                                      \n",
       "19191                                                      \n",
       "19192                                                      \n",
       "19193                                                      \n",
       "19194                                                      \n",
       "19195                                                      \n",
       "19196                                                      \n",
       "19197                                                      \n",
       "19198                                                      \n",
       "19199                                                      \n",
       "19200                                                      \n",
       "19201                                                      \n",
       "19202                                                      \n",
       "19203                                                      \n",
       "19204                                                      \n",
       "19205                                                      \n",
       "19206                                                      \n",
       "19207                                                      \n",
       "19208                                                      \n",
       "19209                                                      \n",
       "19210                                                      \n",
       "19211                                                      \n",
       "19212                                                      \n",
       "19213                                                      \n",
       "19214                                                      \n",
       "19215                                                      \n",
       "19216                                                      \n",
       "19217                                                      \n",
       "19218                                                      \n",
       "19219                                                      \n",
       "\n",
       "                                                   value  \n",
       "0      I would have given this restaurant zero stars ...  \n",
       "1                                                         \n",
       "2                                                         \n",
       "3       The quality ranges from poor to mediocre.\\n I...  \n",
       "4                                                         \n",
       "5       this is by far the worst sushi I have ever se...  \n",
       "6      Restaurant review:.\\n This is bad news.\\n Okay...  \n",
       "7      I was looking for reviews and I saw how bad th...  \n",
       "8                                                         \n",
       "9       Price was ok.\\n Everything is just within the...  \n",
       "10     The worst buffet I've ever had.\\n Not worth of...  \n",
       "11                                                        \n",
       "12                                                        \n",
       "13            Nothing really special about this place.\\n  \n",
       "14      I just can't shake the substandard health sta...  \n",
       "15                                                        \n",
       "16     so heres a tip if your reading this yall bette...  \n",
       "17      I use to really like this place but over the ...  \n",
       "18                                                        \n",
       "19      This was definitely the worst among all.\\n Ea...  \n",
       "20                                                        \n",
       "21                                                        \n",
       "22      However, we brought a coupon that we printed ...  \n",
       "23      There's plenty of seating here and you genera...  \n",
       "24                                                        \n",
       "25     It's my favorite place to eat other then the f...  \n",
       "26      Therefore, I appreciate the decent selection ...  \n",
       "27                                                        \n",
       "28                                                        \n",
       "29     Not trying to tear down this restaurant becaus...  \n",
       "...                                                  ...  \n",
       "19190                                                     \n",
       "19191                                                     \n",
       "19192                                                     \n",
       "19193                                                     \n",
       "19194                                                     \n",
       "19195                                                     \n",
       "19196                                                     \n",
       "19197                                                     \n",
       "19198                                                     \n",
       "19199                                                     \n",
       "19200                                                     \n",
       "19201                                                     \n",
       "19202                                                     \n",
       "19203                                                     \n",
       "19204                                                     \n",
       "19205                                                     \n",
       "19206                                                     \n",
       "19207                                                     \n",
       "19208                                                     \n",
       "19209                                                     \n",
       "19210                                                     \n",
       "19211                                                     \n",
       "19212                                                     \n",
       "19213                                                     \n",
       "19214                                                     \n",
       "19215                                                     \n",
       "19216                                                     \n",
       "19217                                                     \n",
       "19218                                                     \n",
       "19219                                                     \n",
       "\n",
       "[19220 rows x 9 columns]"
      ]
     },
     "execution_count": 239,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aspects_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets save the data in file for later "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT OVERWRITE\n",
    "#aspects_df.to_csv('processed_data/aspect_review.csv',encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test Aspect Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'location'"
      ]
     },
     "execution_count": 241,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assign_aspect('A bar with bar food. There were four of us.',aspect_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'staff'"
      ]
     },
     "execution_count": 242,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assign_aspect('Our waiter was a nice guy.',aspect_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'location'"
      ]
     },
     "execution_count": 243,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assign_aspect('The restaurant was too far from the downtown. And there was no parking!',aspect_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'atmosphere'"
      ]
     },
     "execution_count": 248,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assign_aspect('The decor was aewsome. The music was ok.',aspect_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'service'"
      ]
     },
     "execution_count": 250,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assign_aspect('It took 40 minutes to prepared our order. And it was cold.',aspect_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {},
   "outputs": [],
   "source": [
    "# END"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LARA\n",
    "\n",
    "## Latent Aspect Rating Analysis\n",
    "\n",
    "Now that we have all the pieces read, lets do Latent Aspect Rating Analysis\n",
    "We start by loading the Sentiment Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_pickle(filename_with_path):\n",
    "    pk = None\n",
    "    with open(filename_with_path, 'rb') as f:\n",
    "        pk = pickle.load(f)\n",
    "    return pk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_words = [line.rstrip('\\n') for line in open('config/stopwords.txt', 'r', encoding='utf-8')] \n",
    "stopwords = frozenset(stop_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_process(text):\n",
    "    nopunc = [char for char in text if char not in string.punctuation]\n",
    "    nopunc = ''.join(nopunc)\n",
    "    return [word for word in nopunc.split() if word.lower() not in stopwords]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = read_pickle('pickles/senti_vocab.pk')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the **Multinomial Perceptron Classifier**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp   = read_pickle('pickles/senti_mlp_classifier.pk')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_sentiment_score(str_txt):\n",
    "    \n",
    "    pr_t = vocab.transform([str_txt])\n",
    "    return mlp.predict(pr_t)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "metadata": {},
   "outputs": [],
   "source": [
    "senti_model = read_pickle('pickles/aspect_senti_model.pk')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Lets test the classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment Score: 2\n"
     ]
    }
   ],
   "source": [
    "score = get_sentiment_score(\"We've always been there on a Sunday so we were hoping that Saturday dim sum would be less busy. No such luck. We were surprised that some of the dishes were cold because it was so packed; I could understand if it was empty and the carts weren't circulating but every table was full. It took a while to get drinks and other items (napkins). The dishes were not of the same quality as they had been on other visits, but they were acceptable.\")\n",
    "print(f\"Sentiment Score: {score}\" )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "metadata": {},
   "outputs": [],
   "source": [
    "senti_text = data['sentiment_text'].head(1).iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'zero_star allow disgusting find roach lurch food clean_dish buffet item. dirty. look. plate inspect authority.'"
      ]
     },
     "execution_count": 269,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "senti_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 276,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = data['text'].head(1).iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"I would have given this restaurant zero stars if the app allowed it. This is a disgusting restaurant. You can find roaches lurching around the food. The 'clean' dishes next to buffet items are mostly dirty. Just look at the bottom of the plate. This restaurant should really be inspected by authorities.\""
      ]
     },
     "execution_count": 277,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 328,
   "metadata": {},
   "outputs": [],
   "source": [
    "def assign_aspect(sent,aspect_map):\n",
    "    dicty = {}\n",
    "    \n",
    "    try:\n",
    "        stopwords.remove('restaurant')\n",
    "        stopwords.remove('place')\n",
    "    except:\n",
    "        pass\n",
    "        \n",
    "    for word in nlp(sent.lower()):\n",
    "        \n",
    "        if word.lemma_ not in stopwords:\n",
    "            score = 0\n",
    "            aspec = None\n",
    "            for asp in aspect_map.keys():\n",
    "                try:\n",
    "                    sco = senti_model.wv.similarity(w1=asp,w2=word.lemma_)\n",
    "                    if sco > score:\n",
    "                        score=sco\n",
    "                        aspec=asp\n",
    "                        dicty[aspec]= dicty.setdefault(asp, 0) + score    \n",
    "                except:\n",
    "                    continue\n",
    "            #print(f\"word {word} got {score} score\")\n",
    "            \n",
    "    if len(dicty)==0:\n",
    "        return None\n",
    "    \n",
    "    ret = max(dicty.items(), key=lambda k: k[1])\n",
    "    if ret[1]<0.3:\n",
    "        return None\n",
    "    \n",
    "    return aspect_map[ret[0]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 329,
   "metadata": {},
   "outputs": [],
   "source": [
    "Aspects = Aspect_Terms\n",
    "txt = senti_text\n",
    "vocab = []\n",
    "for x in topic_map.values():\n",
    "    for y in x.split('/'):\n",
    "        vocab.append(y)\n",
    "    \n",
    "mydict = {}\n",
    "\n",
    "for w in Aspect_Terms:\n",
    "    mydict[w.lower()] = mydict.setdefault(w, '')  \n",
    "    \n",
    "for x in txt.replace('\\\\n','.').replace('\\n','.').split('.'):\n",
    "    if(len(x)>0):\n",
    "        y = str(assign_aspect(x.replace('restaurant','location').replace('place','location'),aspect_map)).lower()\n",
    "        mydict[y.lower()] = mydict.setdefault(y.lower(), '') + x +'.\\n' \n",
    "    \n",
    "mydict.pop(None, None)\n",
    "\n",
    "for asp in Aspects:\n",
    "    mydict[asp.lower()]=str(mydict[asp]).strip().replace('\\n','').replace('..','.').replace(';','.').replace('.','.\\n')\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 330,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'atmosphere': 'dirty.\\n',\n",
      " 'food': 'zero_star allow disgusting find roach lurch food clean_dish buffet '\n",
      "         'item.\\n',\n",
      " 'location': '',\n",
      " 'none': ' look.\\n',\n",
      " 'service': '',\n",
      " 'staff': 'plate inspect authority.\\n',\n",
      " 'value': ''}\n"
     ]
    }
   ],
   "source": [
    "pprint(mydict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 331,
   "metadata": {},
   "outputs": [],
   "source": [
    "for asp in Aspects:\n",
    "    mydict[asp+'Score'] = None\n",
    "    mydict[asp+'Score'] = 0\n",
    "        \n",
    "    if str(mydict[asp]).replace('\\n','') != '':\n",
    "        try:\n",
    "            mydict[asp+'Score'] = get_sentiment_score(str(mydict[asp]).replace('\\n',''))\n",
    "        except:\n",
    "            mydict[asp+'Score'] = 3\n",
    "\n",
    "vocab = read_pickle('pickles/senti_vocab.pk')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 332,
   "metadata": {},
   "outputs": [],
   "source": [
    "mydict['0_Review_Score'] = get_sentiment_score(str(senti_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 333,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'0_Review_Score': 1,\n",
      " 'atmosphere': 'dirty.\\n',\n",
      " 'atmosphereScore': 3,\n",
      " 'food': 'zero_star allow disgusting find roach lurch food clean_dish buffet '\n",
      "         'item.\\n',\n",
      " 'foodScore': 3,\n",
      " 'location': '',\n",
      " 'locationScore': 0,\n",
      " 'none': ' look.\\n',\n",
      " 'service': '',\n",
      " 'serviceScore': 0,\n",
      " 'staff': 'plate inspect authority.\\n',\n",
      " 'staffScore': 3,\n",
      " 'value': '',\n",
      " 'valueScore': 0}\n"
     ]
    }
   ],
   "source": [
    "pprint(mydict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adjust the LARA Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 334,
   "metadata": {},
   "outputs": [],
   "source": [
    "def adjust_aspect_scores(dicty):\n",
    "    prdScore = int(dicty['0_Review_Score'])\n",
    "    fodScore = int(dicty['foodScore'])\n",
    "    ambScore = int(dicty['atmosphereScore'])\n",
    "    locScore = int(dicty['locationScore'])\n",
    "    srvScore = int(dicty['serviceScore'])\n",
    "    stfScore = int(dicty['staffScore'])\n",
    "    valScore = int(dicty['valueScore'])\n",
    "    \n",
    "    totScore = 30\n",
    "    newScore = 0\n",
    "    \n",
    "    try:\n",
    "        totScore = int(round(statistics.mean(filter(None, [fodScore,ambScore,srvScore,stfScore,locScore,valScore]))))\n",
    "    except Exception as e:\n",
    "        pass\n",
    "    \n",
    "    dicty['0_Review_Score']       = 0\n",
    "    dicty['foodScore']       = 0\n",
    "    dicty['atmosphereScore'] = 0\n",
    "    dicty['locationScore']   = 0\n",
    "    dicty['serviceScore']    = 0\n",
    "    dicty['staffScore']      = 0\n",
    "    dicty['valueScore']      = 0\n",
    "    \n",
    "    try:\n",
    "        dicty['0_Review_Score']       = int(round(prdScore))\n",
    "        dicty['foodScore']       = int(round(newScore* fodScore/totScore))\n",
    "        dicty['atmosphereScore'] = int(round(newScore* ambScore/totScore))\n",
    "        dicty['locationScore']   = int(round(newScore* locScore/totScore))\n",
    "        dicty['serviceScore']    = int(round(newScore* srvScore/totScore))\n",
    "        dicty['staffScore']      = int(round(newScore* stfScore/totScore))\n",
    "        dicty['valueScore']      = int(round(newScore* valScore/totScore))\n",
    "        \n",
    "                \n",
    "        \n",
    "    except Exception as e:\n",
    "        pass\n",
    "    \n",
    "    for asp in Aspects:\n",
    "        if dicty[asp+'Score']>5:\n",
    "            dicty[asp+'Score']=5\n",
    "        if dicty[asp+'Score']<1 and dicty[asp] != '':\n",
    "            dicty[asp+'Score']=1\n",
    "    \n",
    "    for asp in Aspects:\n",
    "        if dicty[asp+'Score']==0:\n",
    "            dicty[asp+'Score']=''\n",
    "        else:\n",
    "            dicty[asp+'Score']=str(dicty[asp+'Score'])\n",
    "    return dicty\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The Final Scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 335,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"I would have given this restaurant zero stars if the app allowed it. This is a disgusting restaurant. You can find roaches lurching around the food. The 'clean' dishes next to buffet items are mostly dirty. Just look at the bottom of the plate. This restaurant should really be inspected by authorities.\""
      ]
     },
     "execution_count": 335,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 336,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'atmosphere': 'dirty.\\n',\n",
       " 'food': 'zero_star allow disgusting find roach lurch food clean_dish buffet item.\\n',\n",
       " 'service': '',\n",
       " 'staff': 'plate inspect authority.\\n',\n",
       " 'location': '',\n",
       " 'value': '',\n",
       " 'none': ' look.\\n',\n",
       " 'atmosphereScore': '1',\n",
       " 'foodScore': '1',\n",
       " 'serviceScore': '',\n",
       " 'staffScore': '1',\n",
       " 'locationScore': '',\n",
       " 'valueScore': '',\n",
       " '0_Review_Score': 1}"
      ]
     },
     "execution_count": 336,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adjust_aspect_scores(mydict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 337,
   "metadata": {},
   "outputs": [],
   "source": [
    "# END"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### End Of Notebook ###"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}