Tutorial1_Introduction_to_Python_and_Sklearn.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/pgmikhael/mit_deeplearning_bootcamp/blob/master/Tutorial1_Introduction_to_Python_and_Sklearn.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VNic6xm-HFj-"
      },
      "source": [
        "# NLP Task: Beer Review Sentiment Analysis\n",
        "\n",
        "Given long and detailed beer reviews, we want to predict if the reviewed ranked it as an bad, okay or good.\n",
        "\n",
        "# Step 1: Downloading and Preprocessing the Data\n",
        "\n",
        "To start off, we're going to load the data from some pickle files and do some simple preprocessing. We'll throw away non-alphanumeric characters and lowercase everything.\n",
        "\n",
        "i.e \"Best Beer ever!!!\" -> \"best beer ever\"\n",
        "\n",
        "The sanity check the data, we'll look at a few examples."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "bAhHi9YmHFkA"
      },
      "outputs": [],
      "source": [
        "import pickle\n",
        "\n",
        "!apt-get install wget\n",
        "!wget https://raw.githubusercontent.com/pgmikhael/mit_deeplearning_bootcamp/master/data/beer/overall_train.p\n",
        "!wget https://raw.githubusercontent.com/pgmikhael/mit_deeplearning_bootcamp/master/data/beer/overall_dev.p\n",
        "!wget https://raw.githubusercontent.com/pgmikhael/mit_deeplearning_bootcamp/master/data/beer/overall_test.p\n",
        "\n",
        "train_set =  pickle.load(open(\"overall_train.p\", \"rb\"))\n",
        "dev_set =  pickle.load(open(\"overall_dev.p\", \"rb\"))\n",
        "test_set =  pickle.load(open(\"overall_test.p\", \"rb\"))\n",
        "\n",
        "# Extract tweets and labels into 2 lists\n",
        "def preprocess_data(data):\n",
        "    for indx, sample in enumerate(data):\n",
        "        text, label = sample['text'], sample['y']\n",
        "        text = text.lower().strip()\n",
        "        data[indx] = text, label\n",
        "    return data\n",
        "\n",
        "# Preprocess all the data splits.\n",
        "train_set = preprocess_data(train_set)\n",
        "dev_set = preprocess_data(dev_set)\n",
        "test_set =  preprocess_data(test_set)\n",
        "\n",
        "# Separate components into X and Y lists.\n",
        "trainText = [t[0] for t in train_set]\n",
        "trainY = [t[1] for t in train_set]\n",
        "\n",
        "devText = [t[0] for t in dev_set]\n",
        "devY = [t[1] for t in dev_set]\n",
        "\n",
        "testText = [t[0] for t in test_set]\n",
        "testY = [t[1] for t in test_set]\n",
        "\n",
        "# Sanity check:\n",
        "print(\"EXAMPLE INPUTS\")\n",
        "print(trainText[0])\n",
        "print(trainY[0])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ugasiGZhmBjy"
      },
      "source": [
        "# Exercise 1:\n",
        "\n",
        "Make sure you have run the first cell to download and read the train, dev, and test data files.\n",
        "\n",
        "1.   How many examples each are in train, dev, and test?\n",
        "2.   What are the different fields of each example?\n",
        "3.   What is the input? What is the output? How many types of outputs are there?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "znaxvFDSnZFC"
      },
      "outputs": [],
      "source": [
        "# Write code here!"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "33oCxlT1mmwR"
      },
      "source": [
        "# Step 2: Feature Engineering \n",
        "\n",
        "How do we represent a review? We're going to use a simple bag of words representation. Meaning we'll represent each review as a vector, and the whole set of reviews as a large matrix.\n",
        "\n",
        "For example, consider our vocabulary is ```[best, ever, beer, cat, good, dog]```.\n",
        "The bag of words representation for:\n",
        "```\"best beer ever\"``` is ```[1, 1, 1, 0, 0, 0]```\n",
        "Where one indicates that the vocab words did appear and 0 indicates the words that did not. S\n",
        "\n",
        "With sklearn, we can do this very easily with ```sklearn.feature_extraction.text.CountVectorizer```\n",
        "\n",
        "<img src=\"https://github.com/yala/MLCodeLab/blob/master/lab1/vectorizer.png?raw=true\">"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "q1CzWuiuHFkc"
      },
      "outputs": [],
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "# Set that word has to appear at least 5 times to be in vocab\n",
        "min_df = 5\n",
        "max_features = 1000\n",
        "countVec = CountVectorizer(min_df=min_df, max_features=max_features )\n",
        "\n",
        "# Learn vocabulary from train set\n",
        "countVec.fit(trainText)\n",
        "\n",
        "# Transform list of review to matrix of bag-of-word vectors\n",
        "trainX = countVec.transform(trainText)\n",
        "devX = countVec.transform(devText)\n",
        "testX = countVec.transform(testText)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BLUarFjsHFkb"
      },
      "source": [
        "## Exercise 2:\n",
        "\n",
        "1. What is the size of the vocabulary?\n",
        "2. What if you change the mininum token frequency to 500?\n",
        "3. What is the index of \"beer\"?\n",
        "\n",
        "Hint: Use the documentation!\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "9r-u1Q4NnT-O"
      },
      "outputs": [],
      "source": [
        "# Write code here!"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1i1PprZnnJYy"
      },
      "source": [
        "# Step 3: Pick a Model\n",
        "\n",
        "Here we'll explore various types of linear models, namely Logistic Regression, Passive Aggressive, and Perceptron. It's very straight-forward\n",
        "to fit a new classifier and get preliminary results!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "DFgurZU1HFkq"
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.linear_model import Perceptron\n",
        "\n",
        "lr = LogisticRegression()\n",
        "lr.fit(trainX, trainY)\n",
        "print(\"Logistic Regression Train:\", lr.score(trainX, trainY))\n",
        "print(\"Logistic Regression Dev:\", lr.score(devX, devY))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5ZeK-LcYHFko"
      },
      "source": [
        "## Exercise 3:\n",
        "\n",
        "1. What is the *test* score of the logistic regression model?\n",
        "2. What are the maximum and minimum weight values?\n",
        "3. What are the train/dev scores if you use a perceptron?\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Zy_rqdpwnS5p"
      },
      "outputs": [],
      "source": [
        "# Write code here!"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2jz04-8dHFlI"
      },
      "source": [
        "# Step 4: Analysis, Debugging the Model\n",
        "To understand how to make the model better, it's important understand what the model is learning, and what it's getting wrong.\n",
        "\n",
        "To do this, we can inspect the highest weighted features of our best LR model and look at some examples the model got wrong on the development set. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "y2DuosInHFlK"
      },
      "outputs": [],
      "source": [
        "lr = LogisticRegression()\n",
        "lr.fit(trainX, trainY)\n",
        "print(\"Logistic Regression Train:\", lr.score(trainX, trainY))\n",
        "print(\"Logistic Regression Dev:\", lr.score(devX, devY))\n",
        "print(\"--\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Gsyo-PLwHFlQ"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "print(\"Interpreting LR\")\n",
        "for label in range(3):\n",
        "    coefs = lr.coef_[label]\n",
        "    vocab = np.array(countVec.get_feature_names_out())\n",
        "    num_features = 10\n",
        "\n",
        "    top = np.argpartition(coefs, -num_features)[-num_features:]\n",
        "    # Sort top\n",
        "    top = top[np.argsort(coefs[top])]\n",
        "    s_coef = coefs[top]\n",
        "    scored_vocab = list(zip(vocab[top], s_coef))\n",
        "    print(\"Top weighted features for label {}:\\n \\n {}\\n -- \\n\".format(label, scored_vocab))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "7yGk01kBHFlW"
      },
      "outputs": [],
      "source": [
        "# Find erronous dev errors\n",
        "devPred = lr.predict(devX)\n",
        "errors = []\n",
        "for indx in range(len(devText)):\n",
        "    if devPred[indx] != devY[indx]:\n",
        "        error = \"Review: \\n {} \\n Predicted: {} \\n Correct: {} \\n ---\".format(\n",
        "            devText[indx],\n",
        "            devPred[indx],\n",
        "            devY[indx])\n",
        "        errors.append(error)\n",
        "\n",
        "np.random.seed(2)\n",
        "print(\"Random dev error: \\n {} \\n \\n {} \\n \\n{}\".format(\n",
        "        np.random.choice(errors,1),\n",
        "        np.random.choice(errors,1),\n",
        "        np.random.choice(errors,1))\n",
        "     )"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Go11wytYosDq"
      },
      "source": [
        "# Exercise 4:\n",
        "\n",
        "1. Count the number of false positives in the dev set.\n",
        "2. Count the number of false negatives in the dev set."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_pGVb1i7HFlb"
      },
      "source": [
        "## Step 5: Play with regularization\n",
        "\n",
        "We can see that LogisticRegression so far works the best so far, but it is greatly over fitting. Meaning that it does much better on train than development. A common strategy to dealing with this is adding an extra penalty for model complexity, like the square sum of the model weights. We call this idea regularization. \n",
        "\n",
        "In sklearn, it is very easy to test out various regularization amounts and tune the model. The smaller the parameter `C`, the stronger the regularization cost."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "-edhCWbOHFld"
      },
      "outputs": [],
      "source": [
        "lr = LogisticRegression(C=.5)\n",
        "lr.fit(trainX, trainY)\n",
        "\n",
        "\n",
        "print(\"Logistic Regression Train:\", lr.score(trainX, trainY))\n",
        "print(\"Logistic Regression Dev:\", lr.score(devX, devY))\n",
        "print(\"--\")\n",
        "\n",
        "lr = LogisticRegression(C=.1)\n",
        "lr.fit(trainX, trainY)\n",
        "\n",
        "\n",
        "print(\"Logistic Regression Train:\", lr.score(trainX, trainY))\n",
        "print(\"Logistic Regression Dev:\", lr.score(devX, devY))\n",
        "print(\"--\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Kp6127uOHFlj"
      },
      "outputs": [],
      "source": [
        "lr = LogisticRegression(C=.01)\n",
        "lr.fit(trainX, trainY)\n",
        "\n",
        "\n",
        "print(\"Logistic Regression Train:\", lr.score(trainX, trainY))\n",
        "print(\"Logistic Regression Dev:\", lr.score(devX, devY))\n",
        "print(\"--\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jipGDZXiHFlq"
      },
      "source": [
        "# Step 6: Adding in Ngrams\n",
        "\n",
        "How does our model distinguish between the sentiment phrase that says:\n",
        "```\"great flavor and too bad there isn't more.\"```\n",
        "versus\n",
        "```\"bad flavor and too great there isn't more.\"```\n",
        "\n",
        "In our bag of words model, both have the same vector. In order to capture some of these ordering depency, we generalize the bag-of-words model to take \"n-grams\" of words that occur in the training set. a \"bi-gram\" is a pair of words, \"tri-gram\" triple, etc.\n",
        "\n",
        "Let see how this imporves our model \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "stptfRWrHFls"
      },
      "outputs": [],
      "source": [
        "# Set that word has to appear at least 5 times to be in vocab\n",
        "min_df = 5\n",
        "ngram_range = (1,3)\n",
        "max_features = 5000\n",
        "countVecNgram = CountVectorizer(min_df = min_df, ngram_range = ngram_range, max_features=max_features)\n",
        "\n",
        "# Learn vocabulary from train set\n",
        "countVecNgram.fit(trainText)\n",
        "\n",
        "# Transform list of review to matrix of bag-of-word vectors\n",
        "trainXNgram = countVecNgram.transform(trainText)\n",
        "devXNgram = countVecNgram.transform(devText)\n",
        "testXNgram = countVecNgram.transform(testText)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "43xCmdRYHFly"
      },
      "outputs": [],
      "source": [
        "lrNgram = LogisticRegression(C=1)\n",
        "lrNgram.fit(trainXNgram, trainY)\n",
        "print(\"Logistic Regression Train:\", lrNgram.score(trainXNgram, trainY))\n",
        "print(\"Logistic Regression Dev:\", lrNgram.score(devXNgram, devY))\n",
        "print(\"--\")\n",
        "\n",
        "lrNgram = LogisticRegression(C=.5)\n",
        "lrNgram.fit(trainXNgram, trainY)\n",
        "print(\"Logistic Regression Train:\", lrNgram.score(trainXNgram, trainY))\n",
        "print(\"Logistic Regression Dev:\", lrNgram.score(devXNgram, devY))\n",
        "print(\"--\")\n",
        "\n",
        "lrNgram = LogisticRegression(C=.1)\n",
        "lrNgram.fit(trainXNgram, trainY)\n",
        "print(\"Logistic Regression Train:\", lrNgram.score(trainXNgram, trainY))\n",
        "print(\"Logistic Regression Dev:\", lrNgram.score(devXNgram, devY))\n",
        "print(\"--\")\n",
        "\n",
        "lrNgram = LogisticRegression(C=.01)\n",
        "lrNgram.fit(trainXNgram, trainY)\n",
        "print(\"Logistic Regression Train:\", lrNgram.score(trainXNgram, trainY))\n",
        "print(\"Logistic Regression Dev:\", lrNgram.score(devXNgram, devY))\n",
        "print(\"--\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_mp-B3tVHFl7"
      },
      "source": [
        "## Step 7: Take best model, and report results on Test"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "jKz302S-HFl-"
      },
      "outputs": [],
      "source": [
        "print(\"Logistic Regression Test:\", lrNgram.score(testXNgram, testY))"
      ]
    }
  ],
  "metadata": {
    "accelerator": "TPU",
    "colab": {
      "collapsed_sections": [],
      "include_colab_link": true,
      "name": "Introduction to Python and Sklearn",
      "private_outputs": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.9.1 64-bit ('3.9.1')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.1"
    },
    "vscode": {
      "interpreter": {
        "hash": "6e1c477bbb4741057bb356a8a4cfea3fd85ab06b5d35fee50af9f87046f40a78"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}