12.ErrorMeasures.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Error Measures\n",
    "\n",
    "We have already gotten into the innards of a popular machine learning algorithm, KNN, and we have shown how to evaluate its performance. What we have not shown is what that evaluation of performance means.\n",
    "\n",
    "In evaluating a ML algorithm we compare the predicted results to the actual results, but that comparison is not prespecified. We have thusfar been using accuracy (which is useful for classification). But we can use many different forms. Let's start off by cleaning the data and doing a prediction on a new class: was political."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "breast_cancer_data = load_breast_cancer()\n",
    "\n",
    "# we make our test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(breast_cancer_data['data'], breast_cancer_data['target'], test_size=0.2, random_state=1)\n",
    "\n",
    "# and we make our validation set\n",
    "X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(364, 30) (91, 30) (114, 30)\n"
     ]
    }
   ],
   "source": [
    "print X_train.shape, X_val.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.950549450549\n",
      "0.934065934066\n"
     ]
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "cls = KNeighborsClassifier(n_neighbors=3, weights='uniform')\n",
    "\n",
    "cls.fit(X_train, y_train)\n",
    "\n",
    "print cls.score(X_train, y_train)\n",
    "print cls.score(X_val, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Pretty great accuracy right? We are correct in our predictions 93.4% of the time. Of course we will need to validate on our test set in order to get a tight bound (though we did only test one hypothesis here with our validation set).\n",
    "\n",
    "The problem here is that accuracy only tells one side of the story. \n",
    "\n",
    "Here is a question for you: what if 93% of the data was benign cancer?\n",
    "\n",
    "In order to investigate the rest of the story, we will need to look at a confusion matrix, so let's make one below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x109718890>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAboAAAFJCAYAAADt8egyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE/1JREFUeJzt3X9wVeWdx/HPuQkhkBBZf6CohPIbIxbqZgJuIVTXGH8W\nXHEDunEtoo4VNC1gCAQCogTKimIUVHDrDMgA/qZ21dUMJZU4sbACSxCtCCHyIwiIhEDz8+w/nbiW\ncgl57uU8eXy/Zu4Mucl55us/fub7PN9zjuf7vi8AABwVCroAAACiiaADADiNoAMAOI2gAwA4jaAD\nADiNoAMAOC02mou/O3lRNJcHzoprpmcFXQIQEXFJ50Vt7R93H97qa7dUrItgJSeLatABAH4YPM8L\nuoRTYusSAOA0OjoAgDHPs7dvsrcyAAAigI4OAGAsJHvP6Ag6AIAxm4dRCDoAgLGQxWd0BB0AwJjN\nHZ29EQwAQAQQdAAAp7F1CQAw5jF1CQBwGcMoAACn2TyMQtABAIyFLA46e3tNAAAigKADADiNrUsA\ngDHP4r6JoAMAGGMYBQDgNJuHUQg6AIAxm28Yt3dTFQCACCDoAABOY+sSAGCMR4ABAJzG1CUAwGlM\nXQIAnMbUJQAAAaGjAwAYs3kYxd7KAACIADo6AICxaE5d3nrrrUpMTJQkXXrppcrKytLjjz+umJgY\nDR06VOPHjw97PUEHADAWranL2tpa+b6vZcuWNX83YsQIFRUVqVu3brrvvvu0bds2paSknHINgg4A\nYCxaU5fbt2/XiRMnNHbsWDU0NGjChAmqq6tTcnKyJGno0KEqLS0l6AAAbVN8fLzuuece3X777dq1\na5fuvfdeJSUlNf8+ISFBlZWVYdcg6AAAxqJ1RtejRw91795dnuepR48e6tSpk44cOdL8+5qamu8F\n39/D1CUAwFjI81r9CefVV1/V3LlzJUlVVVU6ceKEOnbsqN27d8v3fX344YdKTU0NuwYdHQDAWqNG\njVJeXp7GjBkjz/M0Z84chUIhTZo0SY2NjRo6dKgGDhwYdg2CDgBgLFrDKHFxcXriiSdO+n716tUt\nXoOgAwAY48koAAAEhI4OAGCM99EBAJzG++gAAE7jfXQAAASEjg4AYMzmrUs6OgCA0+joAADGmLoE\nADjN5q1Lgg4AYMzmqUuCDgBgzOaOjmEUAIDTCDoAgNPYugQAGGPqEgDgNJvP6Ag6AIAxpi4BAE6z\nuaNjGAUA4DSCDgDgNLYuAQDGmLoEADjN5jM6gg4AYIyODgDgNJtvL2AYBQDgNDo6AICxkL0NHR0d\nAMBtdHQAAGMMowAAnMbtBQAAp9nc0XFGBwBwGh0dAMBYyOL76Ai6NqTrlX3VY/ggSVJjXYM+feuP\nqt53SCkjh+ncXpeosa5eB7bt0hfv/0nyAy4WaKHf/de7emn5CnnyFB/fXnmTfqXLUy4LuiycIZu3\nLgm6NiLhgs7qd9NV+uipV1RbfVzn90/WT+66Xns2bFf8P3TS+gWr1NTYqMtv+5mSrxqg3aVbgy4Z\nOK2duyq04OlntXr5b3XB+eerZH2pch6ZqvfffiPo0uCQFp/RNTU1RbMOnEZTQ6PKX/mDaquPS5KO\nVn6t9p066pxLu2j/pj+rqaFR8qUDW3fqoh/3CrhaoGXi4uI0K3+KLjj/fEnS5Zf118FDh1RfXx9w\nZThTIc9r9SfawnZ0lZWVKiws1NatWxUbG6umpib17dtXeXl56tGjR9SLw3dOfFOtE99UN//c/+c/\n1YFtu1S976AuGthb+7fsUFNjk7r+pI/ad+oYYKVAy11ycVddcnFXSZLv+5r/5NO6On2o2rVrF3Bl\nOFMW71yGD7pp06Zp4sSJGjhwYPN3mzZtUl5enlauXBn14nCymHaxuiLrGsV3TtSGpW+rsb5BfW8Y\noiHjb1P9iVrt3/yFOnU9L+gygTNy/MQJ5c96TFVVB7T46QVBlwPHhA26urq674WcJA0aNCiqBeHU\n4jsn6spf3KiaA9/o4+feUlNDo+I6ddCudZv02dulkqSLBvbW8UPfBlwp0HL79u/X+F8/op4/6q4X\nFz+j+Pj2QZeEVmizN4z369dPeXl5GjZsmDp16qSamhqtW7dO/fr1O1v14a/adWivtAdGas+G7drx\n/obm77uk9FCXy7rrf156RzFxsfpR+kDt/MMnAVYKtNy33x7VL+5/UCNuvlEP3HtP0OXAgM2v6Qkb\ndDNnztQHH3ygjRs36tixY0pMTNTVV1+tjIyMs1Uf/qrbVQPUoXOiLhzQUxcO6Nn8/YYlv1Pn5Av1\n04mj5YU8fVW2TVX/+2WAlQItt+q117Vvf5WK15aoeG1J8/dLFz2tzp3PCbAynCmbby/wfN+P2h1X\n705eFK2lgbPmmulZQZcARERcUvTO76dm5rX62jnvFUawkpNxHx0AwFibPaMDAKAlLM45HuoMAHAb\nHR0AwBhblwAAp7XZ2wsAAGgJmzs6zugAANY7dOiQhg8frh07dqiiokJjxozRHXfcoYKCgtO+dICg\nAwAY87zWf06nvr5eM2bMUHx8vCSpsLBQOTk5WrFihXzfV3FxcdjrCToAgNXmzZun0aNHq0uXLpKk\n8vJypaWlSZLS09NVWloa9nqCDgBgzPO8Vn/Cef3113Xuuedq2LBhzd/5vt98XUJCgqqrq091uSSG\nUQAAERCtYZTXXntNnufpo48+0qeffqrc3FwdPny4+fc1NTVKSkoKuwZBBwAwFq2hy5dffrn539nZ\n2Zo5c6bmz5+vsrIyDR48WCUlJRoyZEjYNdi6BAAYC3leqz9nKjc3V0VFRcrKylJ9fb0yMzPD/j0d\nHQCgTVi2bFnzv5cvX97i6+joAABOo6MDABjjEWAAAKfZ/IZxgg4AYCxkb84RdAAAczZ3dAyjAACc\nRtABAJzG1iUAwJjNW5cEHQDAGMMoAACn0dEBAJxmcc4xjAIAcBsdHQDAWLTeRxcJdHQAAKfR0QEA\njPFQZwCA0yzeuSToAADmOKMDACAgdHQAAGPcMA4AcJrFOcfWJQDAbXR0AABjbF0CAJxm89sL2LoE\nADiNjg4AYIytSwCA0yzOOYIOAGCOJ6MAABAQOjoAgDGbz+jo6AAATqOjAwAYs7ihI+gAAOZs3rok\n6AAAxizOOYIOAGCO2wsAAAgIQQcAcBpblwAAYxbvXBJ0AABzTF0CAJxmcc4RdAAAczZ3dAyjAACc\nRtABAJzG1iUAwJjFO5cEHQDAnM1PRiHoAADGLM45gg4AYI6pSwAAAkJHBwAwZnFDR9ABAOzV2Nio\n/Px87dy5U57nadasWWrfvr2mTJkiz/PUp08fFRQUKBQ69QYlQQcAMBatM7q1a9dKklauXKmysjI9\n+eST8n1fOTk5Gjx4sGbMmKHi4mJlZGSccg3O6AAAxjyv9Z9wrr32Ws2ePVuStHfvXiUlJam8vFxp\naWmSpPT0dJWWloZdg6ADABjzPK/Vn9OJjY1Vbm6uZs+erVtuuUW+7zdfl5CQoOrq6rDXE3QAAOvN\nmzdP7733nqZPn67a2trm72tqapSUlBT2WoIOAGAsWluXb775pp5//nlJUocOHeR5ngYMGKCysjJJ\nUklJiVJTU8OuwTAKAMBYtIZRrrvuOuXl5enOO+9UQ0ODpk6dql69emn69OlasGCBevbsqczMzLBr\nEHQAAGt17NhRCxcuPOn75cuXt3gNgg4AYOwHe8N4+qSbo7k8cFakXvEvQZcARMSWinVRW5u3FwAA\nnGZxzjF1CQBwGx0dAMCYza/pIegAAMYszjm2LgEAbqOjAwAY80L2tnQEHQDAGFuXAAAEhI4OAGCM\nqUsAgNMszjmCDgBgzuaOjjM6AIDT6OgAAMYsbujo6AAAbqOjAwCYs7ilI+gAAMZsHkYh6AAAxizO\nOYIOAGDO5mddMowCAHAaQQcAcBpblwAAY5zRAQCcxtQlAMBpFuccQQcAMGdzR8cwCgDAaQQdAMBp\nbF0CAIxZvHNJ0AEAzNl8RkfQAQDMWXwQRtABAIzZ3NFZnMEAAJgj6AAATmPrEgBgzOKdS4IOAGDO\n5jM6gg4AYMzinCPoAAARYHHSMYwCAHAaHR0AwJgXoqMDACAQdHQAAGMWH9ERdAAAc9xeAABwmsU5\nxxkdAMBtdHQAAHMWt3QEHQDAmM23FxB0AABr1dfXa+rUqdqzZ4/q6ur0wAMPqHfv3poyZYo8z1Of\nPn1UUFCgUOjUJ3EEHQDAWLR2LtesWaPOnTtr/vz5OnLkiEaOHKn+/fsrJydHgwcP1owZM1RcXKyM\njIxTrsEwCgDAnOe1/hPG9ddfr4cffliS5Pu+YmJiVF5errS0NElSenq6SktLw65B0AEArJWQkKDE\nxEQdO3ZMDz30kHJycuT7fvN9ewkJCaqurg67BkEHADAWpYZOkrRv3z7dddddGjFihG655ZbvncfV\n1NQoKSkp7PUEHQDAmBfyWv0J5+DBgxo7dqwmT56sUaNGSZJSUlJUVlYmSSopKVFqamrYNRhGAQAY\ni9YjwJ577jkdPXpUixYt0qJFiyRJ06ZN02OPPaYFCxaoZ8+eyszMDF+b7/t+VKqTdLxqd7SWBs6a\nIWnZQZcARMSWinVRW/uzl15p9bX97r49gpWcjI4OAGDO3vvFOaMDALiNjg4AYIzX9AAAnEbQAQDc\nZvFBGEEHADBmc0dncQYDAGCOoAMAOI2tSwCAMZu3Lgk6AIA5e3OOoAMAmDvdw5mDRNABAMxZvHXJ\nMAoAwGkEHQDAaWxdAgCMWbxzSdC1dWv/uF7TH/+NPnz3raBLAVpsYv4vdd2NP9O3R45KknZ9WalH\nxs/StTeka9yD/6a4uHbau6dK0341p/lvYDduL0BUVFR+pScXvaAmvynoUoAzMugfB+iRCbO0eWN5\n83cpV/RT3qM5yr71l9r71X5Nnv6gJkwep8emLQiwUrQYU5eItBN/+YvyH5unXz94v6bOLgy6HKDF\n2sW1U/+U3vr3+0arW/LFqqzYo988+oxuvjVDb6z6vfZ+tV+StPipl3RO56SAq0VL0dEh4h7/j6d0\n289vUt9ePYMuBTgjXS48Tx9/9ImenveCdn1ZqbvvH62FS+fo0NeH9dm2L7RwyeO6+NKL9OfPvtT8\nR58Julw4gKnLNmj1G2sUExOjkTddH3QpwBnbU7lfD96dq11fVkqSXnp+pbolX6yYmBgNv/af9OjU\nJ/SvN47Toa8Pq2Du5ICrRYt5Bp8oI+jaoDXv/LfKt3+urLH3a/wj01RbW6essffrwMGDQZcGnFaf\n/j11863Xfe87z/MUCnkqLfmTDn19WL7v683V72jglZcHVCVcEnbrMjs7W/X19d/7zvd9eZ6nlStX\nRrUwnNryF77bztm7b79G3X2vVv3n8wFWBLSc3+RryqyH9MmGLdpTuV9Z2SP1+fYdWv7iq5o47QEt\nKVqmb48c1T/fkK6tW7YHXS5aqM2e0U2aNEn5+fl69tlnFRMTc7ZqAuCwLz7fqcKChSp6sVChmBhV\n7ftauRMe1f69B3Rh1wv029UL5YVC2renSgWT5wVdLlrI5mdder7v++H+YOnSperevbsyMjLOePHj\nVbtbXRhgiyFp2UGXAETElop1UVu78vfvtPrabjfdEMFKTnbaqctx48ZFtQAAQNtn89YlwygAAKdx\nHx0AwJy9DR0dHQDAbXR0AABjNk9dEnQAAHMWD6MQdAAAY0xdAgAQEDo6AIA5zugAAC5j6xIAgIDQ\n0QEAzNnb0BF0AABzbF0CABAQOjoAgDmmLgEALrN565KgAwCYszjoOKMDADiNjg4AYMzmrUs6OgCA\n0+joAADmmLoEALjM5q1Lgg4AYM7ioOOMDgBgzAt5rf60xObNm5WdnS1Jqqio0JgxY3THHXeooKBA\nTU1NYa8l6AAAVluyZIny8/NVW1srSSosLFROTo5WrFgh3/dVXFwc9nqCDgBgteTkZBUVFTX/XF5e\nrrS0NElSenq6SktLw15P0AEAzHle6z+nkZmZqdjY70ZKfN9vHn5JSEhQdXV12OsZRgEAGDubU5eh\n0Hc9Wk1NjZKSksL/fbQLAgD8AESxo/tbKSkpKisrkySVlJQoNTU17N8TdAAAY9Geuvz/cnNzVVRU\npKysLNXX1yszMzN8bb7v+639Dzud41W7o7U0cNYMScsOugQgIrZUrIva2oc3fdzqa88dlBbBSk5G\nRwcAcBrDKAAAcxY/GYWgAwCYI+gAAC7joc4AALdZ/JoehlEAAE6jowMAGPM8e/smeysDACAC6OgA\nAOYYRgEAuIypSwCA25i6BAAgGHR0AABjbF0CANxmcdCxdQkAcBodHQDAnMU3jBN0AABjrXlT+Nli\nbwQDABABdHQAAHMWD6MQdAAAY9xeAABwm8XDKPZWBgBABNDRAQCMMXUJAEBA6OgAAOYYRgEAuIyp\nSwCA2yyeuiToAADmGEYBACAYBB0AwGlsXQIAjDGMAgBwG8MoAACX0dEBANxmcUdnb2UAAEQAQQcA\ncBpblwAAYza/vYCgAwCYYxgFAOAyz+JhFIIOAGDO4o7O833fD7oIAACixd5eEwCACCDoAABOI+gA\nAE4j6AAATiPoAABOI+gAAE4j6NqopqYmzZgxQ1lZWcrOzlZFRUXQJQGttnnzZmVnZwddBhzFDeNt\n1AcffKC6ujqtWrVKmzZt0ty5c7V48eKgywLO2JIlS7RmzRp16NAh6FLgKDq6Nmrjxo0aNmyYJGnQ\noEHaunVrwBUBrZOcnKyioqKgy4DDCLo26tixY0pMTGz+OSYmRg0NDQFWBLROZmamYmPZXEL0EHRt\nVGJiompqapp/bmpq4n8WAPB3EHRt1JVXXqmSkhJJ0qZNm9S3b9+AKwIAO9ECtFEZGRlav369Ro8e\nLd/3NWfOnKBLAgAr8fYCAIDT2LoEADiNoAMAOI2gAwA4jaADADiNoAMAOI2gAwA4jaADADiNoAMA\nOO3/AKGa57QJ7+Z6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10cae0790>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "sns.heatmap(confusion_matrix(cls.predict(X_val), y_val), annot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Okay, what are we looking at here? So notice that the matrix has four entries: two rows and two columns, where the rows and the columns have the same labels. The columns represent the true class of the data point. The rows represent the predicted value.\n",
    "\n",
    "So anything along the diagonal of the matrix was correctly predicted and anything off the diagonal was incorrectly predicted. And this is where we get to see something terrible about the predictions that we have made thusfar. While we get most of the 'not political' billionaires correct, we barely get any of the political billionaires correct!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "One of the other tools that is extremely useful to determine the true accuracy of your algorithm is the classification report:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.88      0.94      0.91        31\n",
      "          1       0.97      0.93      0.95        60\n",
      "\n",
      "avg / total       0.94      0.93      0.93        91\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print classification_report(cls.predict(X_val), y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here we get the second best way to evaluate the performance of a classification model: precision and recall. \n",
    "\n",
    "Don't be too scared here as the terms are simply ways to summarize the above confusion matrix. Each class has its own precision, recall, f1 and support. \n",
    "\n",
    "* Precision: precision is a measure of how accurate your predictions are of the particular class. High precision means that when you predict a data point is this class, you should be pretty sure it is. Technically it is the number of correctly predicted instances of that class over total instances predicted of that class.\n",
    "* Recall: recall is a measure of how well you can find members of this class in the dataset. High recall means that you will be confident that you can identify a member of this class if you see it. Technically it is the number of correctly predicted instances of that class over total instances of that class.\n",
    "* F1 Score: this is simply an average of precision and recall. It is generally better to present both precision and recall rather than just F1.\n",
    "* Support: This is simple number of data points in this class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "What you will look for is large imbalances in the class or in the percision to recall. Fortunately we don't have large ones in our dataset above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Other topics\n",
    "\n",
    "We could dwell on this topic for a full class, but it is good to mention some other very important techniques before we proceed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### ROC/Precision Recall Curve\n",
    "\n",
    "One of the ways that we can further see how well our model does is change the threshold at which we predict certain classes. Previously we would claim a specific class only if we thought the probability it was that class was over 50%, but for very important decisions (cancer screened perhaps), you would want to be alerted when the chance was only 10%. The Precision Recall Curve (and ROC) is a way to visualize performance over these different thresholds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Class Imbalance\n",
    "\n",
    "Today we have seen that our algorithm does much better at predicting the more represented class. This is a common problem and there are many solutions to this problem (including sub-sampling and class reweighting) but we will not have the time to go over them during this class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Learning Objectives\n",
    "\n",
    "Today we are going to learn about different error measures, specifically those for qualitative outputs. You need to be very careful with choosing an error measure for a specific dataset, and today we will learn why"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Comprehension Questions\n",
    "\n",
    "1.\tWhy is accuracy a bad measure of how well an algorithm performs?\n",
    "2.\tIs MSE a good measure? \n",
    "3.\tWhat measure would you use for self driving cars?\n",
    "4.\tHow would you describe precision and recall to a non-technical person?\n",
    "5.\tWhat is a dataset where you would only care about recall? What about precision? \n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}