lab_2_backprop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DVL7_bgmIAPR"
   },
   "source": [
    "# Backpropagation Lab\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "6ZbYjZZZ_yLV"
   },
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator, ClassifierMixin\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sCcEPx5VIORj"
   },
   "source": [
    "## 1. (40%) Correctly implement and submit your own code for the backpropagation algorithm. \n",
    "\n",
    "## Code requirements \n",
    "- Ability to create a network structure with at least one hidden layer and an arbitrary number of nodes.\n",
    "- Random weight initialization with small random weights with mean of 0 and a variance of 1.\n",
    "- Use Stochastic/On-line training updates: Iterate and update weights after each training instance (i.e. do not attempt batch updates)\n",
    "- Implement a validation set based stopping criterion.\n",
    "- Shuffle training set at each epoch.\n",
    "- Option to include a momentum term\n",
    "\n",
    "You may use your own random train/test split or use the scikit-learn version if you want.\n",
    "\n",
    "Use your Backpropagation algorithm to solve the Debug data. We provide you with several parameters, and you should be able to replicate our results every time. When you are confident it is correct, run your script on the Evaluation data with the same parameters, and print your final weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "_a2KSZ_7AN0G"
   },
   "outputs": [],
   "source": [
    "class MLP(BaseEstimator,ClassifierMixin):\n",
    "\n",
    "    def __init__(self,lr=.1, momentum=0, shuffle=True,hidden_layer_widths=None):\n",
    "        \"\"\" Initialize class with chosen hyperparameters.\n",
    "\n",
    "        Args:\n",
    "            lr (float): A learning rate / step size.\n",
    "            shuffle(boolean): Whether to shuffle the training data each epoch. DO NOT SHUFFLE for evaluation / debug datasets.\n",
    "            momentum(float): The momentum coefficent \n",
    "        Optional Args (Args we think will make your life easier):\n",
    "            hidden_layer_widths (list(int)): A list of integers which defines the width of each hidden layer if hidden layer is none do twice as many hidden nodes as input nodes. (and then one more for the bias node)\n",
    "            For example: input width 1, then hidden layer will be 3 nodes\n",
    "        Example:\n",
    "            mlp = MLP(lr=.2,momentum=.5,shuffle=False,hidden_layer_widths = [3,3]),  <--- this will create a model with two hidden layers, both 3 nodes wide\n",
    "        \"\"\"\n",
    "        self.hidden_layer_widths\n",
    "        self.lr = lr\n",
    "        self.momentum = momentum\n",
    "        self.shuffle = shuffle\n",
    "\n",
    "\n",
    "    def fit(self, X, y, initial_weights=None):\n",
    "        \"\"\" Fit the data; run the algorithm and adjust the weights to find a good solution\n",
    "\n",
    "        Args:\n",
    "            X (array-like): A 2D numpy array with the training data, excluding targets\n",
    "            y (array-like): A 2D numpy array with the training targets\n",
    "        Optional Args (Args we think will make your life easier):\n",
    "            initial_weights (array-like): allows the user to provide initial weights\n",
    "        Returns:\n",
    "            self: this allows this to be chained, e.g. model.fit(X,y).predict(X_test)\n",
    "\n",
    "        \"\"\"\n",
    "        self.weights = self.initialize_weights() if not initial_weights else initial_weights\n",
    "\n",
    "        return self\n",
    "\n",
    "    def predict(self, X):\n",
    "        \"\"\" Predict all classes for a dataset X\n",
    "        Args:\n",
    "            X (array-like): A 2D numpy array with the training data, excluding targets\n",
    "        Returns:\n",
    "            array, shape (n_samples,)\n",
    "                Predicted target values per element in X.\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "    def initialize_weights(self):\n",
    "        \"\"\" Initialize weights for perceptron. Don't forget the bias!\n",
    "\n",
    "        Returns:\n",
    "\n",
    "        \"\"\"\n",
    "\n",
    "        return [0]\n",
    "\n",
    "    def score(self, X, y):\n",
    "        \"\"\" Return accuracy of model on a given dataset. Must implement own score function.\n",
    "\n",
    "        Args:\n",
    "            X (array-like): A 2D numpy array with data, excluding targets\n",
    "            y (array-like): A 2D numpy array with targets\n",
    "\n",
    "        Returns:\n",
    "            score : float\n",
    "                Mean accuracy of self.predict(X) wrt. y.\n",
    "        \"\"\"\n",
    "\n",
    "        return 0\n",
    "\n",
    "    def _shuffle_data(self, X, y):\n",
    "        \"\"\" Shuffle the data! This _ prefix suggests that this method should only be called internally.\n",
    "            It might be easier to concatenate X & y and shuffle a single 2D array, rather than\n",
    "             shuffling X and y exactly the same way, independently.\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "    ### Not required by sk-learn but required by us for grading. Returns the weights.\n",
    "    def get_weights(self):\n",
    "        pass\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KibCIXIThpbE"
   },
   "source": [
    "## 1.1 \n",
    "\n",
    "Debug your model using the following parameters:\n",
    "\n",
    "Learning Rate = 0.1\\\n",
    "Momentum = 0.5\\\n",
    "Deterministic = 10 [This means run it 10 epochs and should be the same everytime you run it]\\\n",
    "Shuffle = False\\\n",
    "Validation size = 0\\\n",
    "Initial Weights = All zeros\\\n",
    "Hidden Layer Widths = [4]\n",
    "\n",
    "---\n",
    "\n",
    "### 1.1.1 Debug \n",
    "\n",
    "Debug your model by running it on the [Debug Dataset](https://raw.githubusercontent.com/cs472ta/CS472/master/datasets/linsep2nonorigin.arff)\n",
    "\n",
    "\n",
    "Expected Results for Binary Classification (i.e. 1 output node): [debug_bp_0.csv](https://github.com/cs472ta/CS472/blob/master/debug_solutions/debug_bp_0.csv) \n",
    "\n",
    "$$ \\text{Layer 1} = \\begin{bmatrix} -8.81779797\\text{e}-05 & -8.81779797\\text{e}-05 & -8.81779797\\text{e}-05 & -8.81779797\\text{e}-05 \\\\ 7.82757731\\text{e}-04 & 7.82757731\\text{e}-04 & 7.82757731\\text{e}-04 & 7.82757731\\text{e}-04 \\\\ -3.94353645\\text{e}-03 & -3.94353645\\text{e}-03 & -3.94353645\\text{e}-03 & -3.94353645\\text{e}-03 \\end{bmatrix}$$\n",
    "                                             \n",
    "$$ \\text{Layer 2} = \\begin{bmatrix} -0.01060888 \\\\ -0.01060888 \\\\ -0.01060888 \\\\ -0.01060888 \\\\ -0.02145495 \\end{bmatrix}$$\n",
    "\n",
    "(The weights do not need to be in this order or shape.)\n",
    "\n",
    "Expected Results for One Hot Vector Classification (i.e. 2 output nodes): [debug_bp_2outs.csv](https://github.com/cs472ta/CS472/blob/master/debug_solutions/debug_bp_2outs.csv) \n",
    "\n",
    "$$ \\text{Layer 1} = \\begin{bmatrix} -0.00018149 & -0.00018149 & -0.00018149 & -0.00018149 \\\\ 0.00157468 & 0.00157468 & 0.00157468 & 0.00157468 \\\\ -0.00788218 & -0.00788218 & -0.00788218 & -0.00788218 \\end{bmatrix}$$\n",
    "                          \n",
    "$$ \\text{Layer 2} = \\begin{bmatrix} 0.01050642 & -0.01050642 \\\\ 0.01050642 & -0.01050642 \\\\ 0.01050642 & -0.01050642 \\\\ 0.01050642 & -0.01050642 \\\\ 0.02148778 & -0.02148778 \\end{bmatrix}$$\n",
    "\n",
    "(The weights do not need to be in this order or shape.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KgAyy82gixIF"
   },
   "outputs": [],
   "source": [
    "# Load debug data\n",
    "\n",
    "# Train on debug data\n",
    "\n",
    "# Print weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kY3VNB1ui03N"
   },
   "source": [
    "### 1.1.2 Evaluation\n",
    "\n",
    "Evaluate your model using the following parameters:\n",
    "\n",
    "Learning Rate = 0.1\\\n",
    "Momentum = 0.5\\\n",
    "Deterministic = 10 [This means run it 10 epochs and should be the same everytime you run it]\\\n",
    "Shuffle = False\\\n",
    "Validation size = 0\\\n",
    "Initial Weights = All zeros\\\n",
    "Hidden Layer Widths = [4]\n",
    "\n",
    "We will evaluate your model based on printed weights after training on the [Evaluation Dataset](https://raw.githubusercontent.com/cs472ta/CS472/master/datasets/data_banknote_authentication.arff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "2yAxA78QjDh2"
   },
   "outputs": [],
   "source": [
    "# Load evaluation data\n",
    "\n",
    "# Train on evaluation data\n",
    "\n",
    "# Print weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9vWiTdlbR2Xh"
   },
   "source": [
    "## 2. (10%) Backpropagation on the Iris Classification problem.\n",
    "\n",
    "Load the Iris Dataset [Iris Dataset](https://raw.githubusercontent.com/cs472ta/CS472/master/datasets/iris.arff)\n",
    "\n",
    "Parameters:\n",
    "- One layer of hidden nodes with the number of hidden nodes being twice the number of inputs.\n",
    "- Use a 80/20 split of the data for the training/test set.\n",
    "- Use a learning rate of 0.1\n",
    "- Use a validation set (15% of the training set) taken from the training set for your stopping criteria\n",
    "- Create one graph with MSE (mean squared error) over epochs from the training set and validation set\n",
    "- Create one graph with classification accuracy (% classified correctly) over epochs from the training set and validation set\n",
    "- Print out your test set accuracy\n",
    "\n",
    "The results for the different measurables should be shown with a different color, line type, etc. Typical backpropagation accuracies for the Iris data set are 85-95%.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4SSoasDQSKXb"
   },
   "outputs": [],
   "source": [
    "# Iris Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SIRG42TgSR4x"
   },
   "source": [
    "## 3. (10%) Working with the Vowel Dataset - Learning Rate\n",
    "\n",
    "Load the Vowel Dataset [Vowel Dataset](https://raw.githubusercontent.com/cs472ta/CS472/master/datasets/vowel.arff)\n",
    "\n",
    "- Use one layer of hidden nodes with the number of hidden nodes being twice the number of inputs.\n",
    "- Use random 80/20 splits of the data for the training/test set.\n",
    "- Use a validation set (15% of the training set) taken from the training set for your stopping criteria\n",
    "- Try some different learning rates (LR). Note that each LR will probably require a different number of epochs to learn. \n",
    "\n",
    "- For each LR you test, plot their validation's set MSE over Epochs on the same graph. Graph 4-5 different LRs and make them different enough to see a difference between them.\n",
    "\n",
    "In general, whenever you are testing a parameter such as LR, # of hidden nodes, etc., test values until no more improvement is found. For example, if 20 hidden nodes did better than 10, you would not stop at 20, but would try 40, etc., until you no longer get improvement.\n",
    "\n",
    "If you would like you may average the results of multiple initial conditions (e.g. 3) per LR, and that obviously would give more accurate results.\n",
    "\n",
    "<img src=https://raw.githubusercontent.com/cs472ta/CS472/master/images/backpropagation/backprop_val_set_MSE_vs_epochs.png width=500 height=500  align=\"left\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "KBGUn43ASiXW"
   },
   "outputs": [],
   "source": [
    "# Train on each dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mOteTlV6S0bq"
   },
   "source": [
    "## 3.1 (5%) Working with the Vowel Dataset - Intuition\n",
    "- Discuss the effect of varying learning rates. \n",
    "- Discuss why the vowel data set might be more difficult than Iris\n",
    "    - Report both datasets' baseline accuracies and best **test** set accuracies. \n",
    "- Consider which of the vowel dataset's given input features you should actually use (Train/test, speaker, gender, ect) and discuss why you chose the ones you did.\n",
    "\n",
    "Typical backpropagation accuracies for the Vowel data set are above 75%.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qmq9GSbJS8k2"
   },
   "source": [
    "*Discuss Intuition here*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SIRG42TgSR4x"
   },
   "source": [
    "## 3.2 (10%) Working with the Vowel Dataset - Hidden Layer Nodes\n",
    "\n",
    "Using the best LR you discovered, experiment with different numbers of hidden nodes.\n",
    "\n",
    "- Start with 1 hidden node, then 2, and then double them for each test until you get no more improvement in accuracy. \n",
    "- For each number of hidden nodes find the best validation set solution (in terms of validation set MSE).  \n",
    "- Create one graph with MSE for the training set and validation set on the y-axis and # of hidden nodes on the x-axis.\n",
    "- Report the final test set accuracy for every # of hidden nodes you experimented on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qmq9GSbJS8k2"
   },
   "source": [
    "*Discuss Hidden Layer Nodes here*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SIRG42TgSR4x"
   },
   "source": [
    "## 3.3 (10%) Working with the Vowel Dataset - Momentum\n",
    "\n",
    "Try some different momentum terms using the best number of hidden nodes and LR from your earlier experiments.\n",
    "\n",
    "- Create a graph similar to step 3.2, but with momentum on the x-axis and number of epochs until validation set convergence on the y-axis.\n",
    "- For each momentum term, print the test set accuracy. \n",
    "- You are trying to see how much momentum speeds up learning and how it affects accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qmq9GSbJS8k2"
   },
   "source": [
    "*Discuss Momentum here*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oBBmeNQ7jvcQ"
   },
   "source": [
    "## 4.1 (10%) Use the scikit-learn (SK) version of the MLP classifier on the Iris and Vowel data sets.  \n",
    "\n",
    "You do not need to go through all the steps above, nor graph results. Compare results (accuracy and learning speed) between your version and theirs for some selection of hyper-parameters. Try different hyper-parameters and comment on their effect.\n",
    "\n",
    "At a minimum, try\n",
    "\n",
    "- number of hidden nodes and layers\n",
    "- different activation functions\n",
    "- learning rate\n",
    "- regularization and parameters\n",
    "- momentum (and try nesterov)\n",
    "- early stopping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "OFQv70W2VyqJ"
   },
   "outputs": [],
   "source": [
    "# Load sklearn perceptron\n",
    "\n",
    "# Train on voting dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BqSFAXwlk3Ms"
   },
   "source": [
    "*Record impressions*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oBBmeNQ7jvcQ"
   },
   "source": [
    "## 4.2 (5%) Using the Iris Dataset automatically adjust hyper-parameters using your choice of grid/random search\n",
    "- Use a grid or random search approach across a reasonable subset of hyper-parameters from the above \n",
    "- Report your best accuracy and hyper-parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "OFQv70W2VyqJ"
   },
   "outputs": [],
   "source": [
    "# Load sklearn perceptron\n",
    "\n",
    "# Train on voting dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cTlK-kijk8Mg"
   },
   "source": [
    "## 5. (Optional 5% Extra credit) For the vowel data set, use the other hyper-parameter approach that you did not use in part 4.2 to find LR, # of hidden nodes, and momentum.  \n",
    "\n",
    "- Compare and discuss the values found with the ones you found in part 3.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BqSFAXwlk3Ms"
   },
   "source": [
    "*Discuss findings here*"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "lab 1 - perceptron",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}