lectures/08_hyperparameter-optimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Lecture 8: Hyperparameter Optimization and Optimization Bias\n",
    "\n",
    "UBC 2022 Summer\n",
    "\n",
    "Instructor: Mehrdad Oveisi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "sys.path.append(\"code/.\")\n",
    "\n",
    "import IPython\n",
    "import ipywidgets as widgets\n",
    "import matplotlib.pyplot as plt\n",
    "import mglearn\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from IPython.display import HTML, display\n",
    "from ipywidgets import interact, interactive\n",
    "from plotting_functions import *\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.model_selection import cross_val_score, cross_validate, train_test_split\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from utils import *\n",
    "\n",
    "%matplotlib inline\n",
    "pd.set_option(\"display.max_colwidth\", 200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Learning outcomes \n",
    "\n",
    "From this lecture, you will be able to \n",
    "\n",
    "- explain the need for hyperparameter optimization  \n",
    "- carry out hyperparameter optimization using `sklearn`'s `GridSearchCV` and `RandomizedSearchCV` \n",
    "- explain different hyperparameters of `GridSearchCV`\n",
    "- explain the importance of selecting a good range for the values. \n",
    "- explain optimization bias\n",
    "- identify and reason when to trust and not trust reported accuracies "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Hyperparameter optimization motivation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Motivation\n",
    "\n",
    "- Remember that the **fundamental goal** of supervised machine learning is to **generalize** beyond what we see in the training examples. \n",
    "- We have been using data splitting and cross-validation to provide a framework to **approximate generalization error**.  \n",
    "- With this framework, we can improve the model's generalization performance by ***tuning model hyperparameters*** using cross-validation on the training set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Hyperparameters: the problem\n",
    "\n",
    "- In order to improve the generalization performance, finding the best values for the important hyperparameters of a model is necessary for almost all models and datasets. \n",
    "- Picking **good hyperparameters** is important because if we don't do it, we might end up with an **underfit** or **overfit** model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Some ways to pick hyperparameters:\n",
    "- **Manual** or expert knowledge or heuristics based optimization \n",
    "- Data-driven or **automated** optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Manual hyperparameter optimization\n",
    "\n",
    "- Advantage: we may have some intuition about what might work.\n",
    "  - E.g. if I'm massively overfitting, try decreasing `max_depth` or `C`.\n",
    "- Disadvantages\n",
    "    - it takes a lot of work\n",
    "    - not reproducible\n",
    "    - in very complicated cases, our intuition might be worse than a data-driven approach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Automated hyperparameter optimization \n",
    "\n",
    "- Formulate the hyperparamter optimization as a **one big search problem**. \n",
    "- Often we have many hyperparameters of different types: Categorical, integer, and continuous.\n",
    "- Often, the search space is quite big and systematic search for optimal values is infeasible. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In homework assignments, we have been carrying out hyperparameter search by exhaustively trying different possible combinations of the hyperparameters of interest. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApYAAAGnCAYAAAANPBefAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAArEAAAKxAFmbYLUAABvFUlEQVR4nO3dd3hUZfYH8O87kzbpDYRQQgs9VAU0QBJAqqELiCDNBgq7Nn66upS1IK662MFGUTCuyNIFBZIgKoig0hEICRiEEEhI7+f3B86YYSaQhEnuzOT7eZ55lHvvvHOm3Xty7rnvKBGBkVIqAMDwoKCgCSLSSETcQWQbpXq9Pj07O/ur/Pz8VSJyWOuAiOgvSqmOOp3uuYCAgP6lpaUGADqtY3Jgotfri/Py8o7n5OQsArBcyh5siZyYMn7WlVKDDAbD6kGDBpWOHz/eu3nz5jAYDBqHR86iuLgYaWlp+OqrrwqXLVtWmJeXtzEzM3OCiJRoHRtRbefm5rbCzc1t4siRIzF27Fg0atQI7u6sK1RVSUkJMjIysH37dixduhSXLl26kpmZ2VRE0rWOjai6KRGBUirKz89vY1xcnFfnzp21jomcXH5+PgYNGpTz888/f5mRkTFJ63iIajN3d/d3AwICpu/evRtNmjTROhynU1paikmTJmH9+vVXMjMzA1i5JGenRATBwcG7Fi9eHDF69Git46FaIj8/H3Xq1MnLzs5uJiLntY6HqDZSSik/P7+ibdu26W+99Vatw3FapaWlaN68OZKSkiaLyHKt4yGqTjqlVGBeXl7XIUOGaB0L1SIeHh4YPny4KKVGaR0LUS02zNPTU9+1a1et43BqOp0OU6ZMgaen52Nax0JU3XQAOnbo0CGf/ZRU0wYOHOgZGBjYR+s4iGqxmH79+kEppXUcTq9v377w9PRspXUcRNVNB8A/KCiIexWqcQEBAVBKBWodB1EtFhQcHKx1DLVCQEAASkpKXLSOg6i66QAovV7PxJKua/LkyVBKISkpqULbJyUlQSmFyZMnl7uNXq+HUkpvmwiJqAqUi0vVcp2KfMfpL3q9HgB4rCWnx3nKbIg7WiKiylFKISoqSuswqiQ+Ph5KKcybN0/rUIjsBhNLqpAFCxbg6NGjaNCggdahEBERkZ1ivwdVSP369VG/fn2twyAiIiI7dsOKZdlS/86dOxEZGQlvb28EBgZi/Pjx+P333y3uExcXh6lTp6JVq1bw9vaGt7c3br31Vrz//vtWH8N4KiQlJQWTJ09GvXr1oNPpEB8ff9PjjR8/HsHBwfDx8cGQIUOQmJgIADh+/DhGjBiBwMBA+Pj44O6770ZqaqrV8Q4cOIBx48ahfv36cHNzQ2hoKGbOnIlLly6Ztlm2bBmaNm0KAFi+fDmUUqab8XkAgIjg448/RkREBHx9feHp6Ylbb70VH3/8scXjzps3z3T/5cuXo2vXrvD09KzQaaPi4mIsWLAAzZs3h4eHB1q0aIEFCxYgMTHR6un6Jk2aoEmTJsjIyMCsWbPQqFEjuLi4YNmyZQDK77EsKSnBwoUL0aJFC7PHKS0tvWGMRGT/Kvsdr+j+2nhsAYCEhASzfaZxv3PlyhUsXLgQkZGRCAkJgZubG0JCQnDffffh1KlTFX4OpaWl+PDDD9GtWzcEBgbC09MTTZo0wfDhw7Fz506L7Xfu3ImYmBgEBwfD3d0dYWFheO6555Cbm2vaZt68eYiOjgYAzJ8/3yz+ivaiEzmjClcsd+/ejQULFmDIkCGYNWsW9u/fj88++wy7du3C3r17ccstt5i2XbhwIU6ePIkePXpgxIgRyMjIwJYtW/DQQw/h+PHjeO211yzGv3TpEm6//XYEBgZi7NixKCwshK+vb5XHS09PR8+ePVGvXj1MmjQJv/32GzZu3Ihjx45h/fr16NWrF7p06YKpU6di3759WL16NTIyMvDNN9+YjbN+/XqMGTMGer0eQ4cORaNGjXDkyBG8/fbb2Lp1K/bs2YOAgAB06tQJf/vb3/DGG2+gY8eOGD58uGkM469ZiAgmTJiAVatWoWXLlhg/fjzc3NzwzTffYNq0aThy5AheffVVi+fy73//G3FxcRg6dCjuvPNOVKTZfurUqfjkk0/QvHlzPPLIIygoKMCiRYvwww8/lHufgoIC9OnTB1lZWYiJiYGbm5vZ+2rNgw8+iI8//hhNmzbFI488gvz8fLz++uv4/vvvbxgjEdm/yn7HK7q/btKkCebOnYv58+cjNDTU7I/dTp06AQCOHj2KOXPmIDo6GiNGjICXlxeOHTuGVatWYdOmTdi/fz9CQ0Nv+ByeeeYZvPLKK2jevDnGjx8PHx8fpKSk4Ntvv8WOHTvQu3dv07aLFy/GjBkzEBAQgJiYGNSpUwd79+7Fiy++iLi4OMTFxcHNzQ1RUVFISkrC8uXLERkZafYHv7+/f6VfZyJnMnLo0KFXpBxxcXECQADIhx9+aLZu/vz5AkCmTp1qtjwxMdFinKKiIrnzzjtFr9dLcnKy2Trj+FOmTJHi4mKL+1Z1vMcee8xs+cMPPywAxN/fXxYtWmRaXlpaKoMHDxYAsn//ftPytLQ08fX1lYYNG1o8xqpVqwSAPProo6Zlp0+fFgAyadIki3hFRN5//30BINOmTZOioiLT8oKCAomJiREA8tNPP5mWz507VwCIl5eXHDhwwOqY1mzbtk0AyK233iq5ubmm5X/88YfUq1fPaoyhoaECQPr37292H6NJkyYJADl9+rRpmfGz0bFjR8nOzjYt//333yU4OPi6r4WIyJYtW6ROnTo7RQS88cZbzd8ArHvqqaekPFX5jldlfx0ZGWn18TMyMuTSpUsWy3fs2CE6nU7uv//+cmMvKzAwUBo0aCA5OTlmy0tLS83GP3z4sLi4uEjnzp0tHnfBggUCQF599VXTMuPrM3fu3BvGcOzYMQkICCgWO3jfeeOtOm8VvninVatWmDp1qtmyp556CnXq1MFnn32GwsJC03LjKeGyXFxc8PDDD6OkpARxcXEW693c3PDKK68Yp2QwU5XxvL298fzzz5stGz9+PAAgKCgIs2bNMi1XSmHcuHEAgF9//dW0fMWKFcjMzMSCBQvQuHFjs7HuuecedOnSBbGxsRaPXZ63334bXl5eePvtt82qjm5ubnjxxRcBAJ999pnF/R588EGEh4dX+HE+/fRTAMA///lPlJ34vl69evjb3/523fv++9//RkUny1+xYgUAYM6cOfDy8jItb9CgwQ0fh4jsX1W+41XZX5fHz88PgYGWU91GR0ejXbt22LZtW4XHcnNzszjbo5QyG3/JkiUoLi7Gm2++afG4s2fPNh3viKh8FT4VHhERYfHrDAaDAV27dsWWLVvw22+/oX379gCArKwsvPrqq1i7di1OnTqFnJwcs/udO3fOYvymTZuivIl6qzJeWFiY2Y4QgOnikw4dOlg8F+O6lJQU07Ldu3eb/nvy5EmLx8jPz0daWhrS0tLKjd0oNzcXBw8eREhICF5++WWL9UVFRQCAY8eOWazr1q3bdce+ljE5vuOOOyzWWVtm5OHhUakE1vg4vXr1slhnbRkROZaqfMersr++nvj4eCxatAh79uxBWloaiouLTevc3NwqNMaYMWOwePFitG/fHmPHjkVkZCRuv/12i2OEcZ+/ZcsWq0mrq6ur1X00Ef2lwoll3bp1rS439uBduXIFAFBYWIioqCjs378fnTt3xsSJExEUFAQXFxdTP0pBQUG541yrquMZ+zPNnuyff61eb50xwQOAy5cvAwDeeecdq7EZ5eTk3DCxTE9Ph4ggJSUF8+fPv+5Y17pRn+O1MjMzodPpEBQUVKmx6tatW6mfdrty5Qp0Op3V517ZmInI/lT2O17V/XV5vvjiC4wdOxbe3t4YMGAAmjRpAk9PT9MFPsnJyRUa580330SzZs2wbNkyvPDCC3jhhRfg4eGBMWPG4LXXXjM9P+M+33gGiYgqr8KJZXlXTF+4cAHA1VMWALBu3Trs378f999/Pz744AOzbWNjY7F8+XKr45SX0FR1PFswJqAHDx40VWNvdqyuXbvip59+qtR9K/s7vr6+vigtLcWlS5csDgjG98sWj+Pn54fS0lKkpaWhTp06FX4cInIMlf2O23p/PW/ePHh4eGDfvn0ICwuzGK+iXF1d8dRTT+Gpp57CuXPnkJCQgKVLl2LFihU4f/48tm7dCuCv/XRmZiZ8fHwqFSsRXVXhHsvvvvsOImK2LC8vD/v27YPBYEDLli0BwDQFxNChQy3G+PbbbysdoK3Hq4zu3bsDwHWvpC7L2B9aUlJisc7Hxwdt2rTB0aNHkZGRYbMYrenYsSMAWL1q05ZXaxsfx9r7UN3vDRFVv8p+x6uyv9bpdFb3mcbx2rRpY5FUnjt3rlLTDZUVEhKCe+65B1u2bEFYWBi2bduGvLw8AH/t842nxG/kevt8otqqwonl8ePHLeZa/Pe//42LFy/innvuMfW6GKd+2LVrl9m2CQkJFn/BVoStx6uMKVOmwMfHB88++ywOHz5ssT43N9dsBxQQEACllNW5PQFg1qxZyM3NxQMPPGD1lPfp06dtMv/ZvffeCwB4/vnnkZ+fb1p+/vx5vPHGGzc9vtF9990HAPjXv/5l9nxSUlJs+jhEpI3Kfsersr8ODAwsd58ZGhqKkydPmlVH8/PzMX36dLNey+spKCjAjh07LAojOTk5yMrKgqurqylBnDFjBlxcXDBz5kycPXvWYqyMjAz8/PPPZrEDKDd+otqowqfC+/fvjxkzZmDTpk1o3bo19u/fj61bt6JRo0Z46aWXTNvFxMSgSZMmeOWVV3Do0CG0b98ex48fx8aNGzF8+HB8+eWXlQrQ1uNVhvEKwLvvvhsdO3bEwIED0bp1a+Tn5yM5ORkJCQm44447sGXLFgBXr0S/7bbbsHPnTkyZMgVhYWHQ6XQYP348GjdujIceegi7d+/G8uXL8d1336Ffv34ICQnBhQsXcOzYMezZswerVq0yzXtZVf369cO9996LlStXIjw8HMOGDUNBQQH++9//onv37tiwYQN0upv/Nc+oqChMmTIFS5cuRXh4OEaMGIGCggJ8/vnn6NGjBzZu3HjTj0FE2qnsd7wq++s+ffrgv//9L0aPHo3OnTtDr9djyJAhCA8Px8yZMzFz5kx07twZo0ePRnFxMb755huICDp27Gg2i0d58vLy0LdvXzRr1gzdu3dH48aNkZ2djY0bN+L8+fP4v//7P1NhpH379nj33Xcxffp0tGrVCoMHD0bz5s2RmZmJxMREJCQkYPLkyVi8eDEAoHXr1ggJCUFsbCw8PT3RsGFDKKUwffp0U3sYUW1UoXks586dKwkJCdKrVy/x9PQUf39/GTdunJw5c8biPomJiTJq1CipU6eOeHp6ym233SaxsbHlzvmF68xjZsvxrjfP5PXmIzt27JhMmzZNQkNDxc3NTQICAiQ8PFxmzZolP/74o9m2x48fl8GDB4u/v78opQSAxMXFmW3z+eefS79+/SQgIEBcXV2lQYMGEhUVJa+99ppcvHjRtJ1xHstr718RRUVF8vzzz0vTpk3Fzc1NmjVrJi+99JLs2bNHAMjf/vY3s+1DQ0MlNDS03PGszWMpIlJcXCwLFiyQZs2amT3OyZMnOY8lb7zZ+Q03mMdSpPLf8crur//44w8ZM2aMBAcHi06nEwCydOlSEbk6z+TixYulXbt24uHhIfXq1ZNp06bJhQsXJDIyUgBcN3YRkcLCQlm4cKH0799fGjZsKG5ubnLLLbdIZGSkxMbGWr3Pjz/+KOPGjZOQkBBxdXWV4OBg6dKlizz99NNy9OhRs213794tkZGR4uPjY5pD+dr9pAjnseSt9twqlViS4/vggw8EgLz77rtah8LEkjfeNL5VJLEk22BiyVttud38+VCyS+fPn4eIeU9RSkoKXnjhBej1etx1110aRUZERETOqsI9luRYXn75ZWzatAm9evVC3bp1cebMGWzcuBFZWVmYN28eGjVqpHWIRERE5GRcAEhJSYnccEtyKAMHDsSRI0ewadMmpKenw8PDAx06dMCMGTNMP22ptZKSEogI5+kg0o5U9Opqujl/TknEYy05PRcAGZcuXSr3wx4VFWVxSpXs38CBAzFw4ECtw7iuP3+N6LLWcRDVYpfS0tK0jqFWSE9Ph16vZxZPTk8H4JcDBw54GCeIJaopW7Zsyb18+fIOreMgqsU2bNu2jcWDGrB9+3bk5uYe1zoOouqmE5F0g8Gwb9OmTVrHQrVIfn4+1q5dq0Sk+iYiJaIbWZebm1uyb98+reNwaqWlpVi6dClyc3P/o3UsRNVNBwCXLl167v77788p+4sCRNUlPz8fgwYNytHr9V+IyHmt4yGqrURE8vLy3r/rrrts8qtfZKm0tBSTJk3C5cuXrwBYoXU8RNVNGU+BKKUGGQyG1QMHDiy99957vZs1awaDwQCllMYhkjMoLi5GWloaNm/eXLh8+fLCvLy8jZmZmRN48Q6R9tzc3Fa4ublNHDFiBMaOHYtGjRrBw8ND67AcVklJCdLT07F9+3YsXboUly9fvpKZmdlURNK1jo2ouqmyvTVKqQAAw4KCgiaKSCMRcdcuNHIyJXq9PiM7O3tzfn7+ZyJi+ePrRKQZpVRHpdQ/AgMDB4qIQUQ4z3HViV6vL87NzT3+5+nvFcJGVqolFD/rRERkC0qpNgC6iMhKrWO5WUqpdgA6AIhlUkhUcfyL1MaUUl2UUg9oHQcRUU1RSumVUrMBxAEo0DoeG7kC4G8ANiqlGmgdDJGjYGJpe6UAHtE6CCKimvBnlfI7AJEAuorIao1DsgkR+R1ATwAJAPYppaYoXnRAdEM8FW5jSikdgFQALTn5NxE5K6WUHsATAB4H8DSA5c56yvjP5HkZgDQAD4pIirYREdkvVixtTERKAewC0FvrWIiIqoOVKuUyZ00qAUBEjgKIAKuXRDfExLJ6xAOI0jgGIiKbuqaXcjGAu2pL9U5EikXkFQDRAB4Gey+JrGJiWT3iwcSSiJxIbatSlofVS6LrY49lNWCfJRE5i9rUS1lZ7L0kssSKZTVgnyUROQNWKa+P1UsiS0wsq088eDqciBxQbe6lrCz2XhKZY2JZfeLBxJKIHAyrlFXD6iXRVeyxrCbssyQiR8JeStth7yXVZqxYVhP2WRKRo2CV0rZYvaTajIll9YoHT4cTkZ1iL2X1Ye8l1VZMLKtXPJhYEpEdYpWyZrB6SbUNeyyrEfssicjesJdSO+y9pNqAFctqxD5LIrInrFJqi9VLqg2YWFa/ePB0OBFpiL2U9oO9l+TsmFhWv3gwsSQijbBKaZ9YvSRnxR7LasY+SyLSAnspHQd7L8mZsGJZzdhnSUQ1jVVKx8LqJTkTJpY1Ix48HU5E1Yy9lI6LvZfkLJhY1owEMLEkomrEKqVzYPWSHB17LGvAn71OqQDC2GdJRLbEXkrnxd5LckSsWNYAESkB+yyJyMZYpXRurF6SI2JiWXPiwdPhRGQD7KWsPdh7SY6GiWXNiQcTSyK6SaxS1k6sXpKjYI9lDWGfJRHdDPZSkhF7L8mesWJZQ9hnSURVxSollcXqJdkzJpY1Kx48HU5EFcReSioPey/JXjGxrFnxYGJJRBXAKiVVBKuXZG/YY1mD2GdJRDfCXkqqKvZekj1gxbIGsc+SiK6HVUq6Gaxekj1gYlnz4sHT4URUhlLKhb2UZAvsvSStMbGsefFgYklEf7qmSnkrq5RkC6xeklbYY1nD2GdJRMDVKiWu9lI+BvZSUjVi7yXVJFYsaxj7LImoTJWyN9hLSdWM1UuqSUwstREPng4nqnX+7KX8P1ztpXwP7KWkGsLeS6opTCy1EQ8mlkS1CquUZA9YvaTqxh5LDbDPkqj2YC8l2Sv2XlJ1YMVSA+yzJKodWKUke8bqJVUHJpbaiQdPhxM5JfZSkqNg7yXZGhNL7cSDiSWR02GVkhwRq5dkK+yx1Aj7LImcC3spyVmw95JuBiuWGmGfJZHzYJWSnAmrl3QzmFhqKx48HU7ksNhLSc6KvZdUVUwstRUPJpZEDolVSqoNWL2kymKPpYbYZ0nkeNhLSbUVey+pIlix1BD7LIkcC6uUVJuxekkVwcRSe/Hg6XAiu8ZeSqKr2HtJN8LEUnvxYGJJZLdYpSSyxOollYc9lhpjnyWRfWIvJVHFsPeSymLFUmPssySyP6xSElUcq5dUFhNL+xAPng4n0hx7KYmqhr2XZMTE0j7Eg4klkaZYpSS6eaxeEnss7QD7LIm0w15KourB3svaiRVLO8A+SyJtsEpJVH1YvaydmFjaj3jwdDhRjWAvJVHNYO9l7cPE0n7Eg4klkU0operodLp7ylnHKiVRDato9VIp1U6v1/er8QDJZthjaSfYZ0lkO/7+/p9lZWWNLS0tjRaRBIC9lET2orzeS6WUh4+Pz/GSkpKA3NzcUBFJ1zJOqhpWLO0E+yyJbEMp1UWv1w9dvXq18vb2jlVKebFKSWQ/yqteent7Lxg4cGDwfffd5+rj4/OixmFSFbFiaUeUUo8BCBWRv2sdC5EjUkopPz+/n958880u9913HyZPnpz3xRdfHMjNzW0CVimJ7E6Z6mVRQEBAl5MnTxoAoGnTprmZmZmdReQ3TQOkSmPF0r7Eg32WRFWm0+lGNmjQoOWECRMAAG+++abBy8vrVgAzWKUksj9/Vi/7eHt7t1u2bJkhMDAQgYGBeP755z38/f0Xax0fVR4TS/tyAEAjpVSg1oEQORqllIenp+e777//vrdOd3XX5uvri5UrV+q9vb3fUUp5aRwiEVnh7e39wqBBg9yGDh1qWjZ9+nSdn5/fbUqpOzUMjaqAiaUdYZ8lUdUZDIYn+/Xr5xMREWG2/M4778SoUaP8fH19X9coNCIqh1Kqm6ur64OLFy/2LLvc1dUVixcv9vbx8fngzwvvyEGwx9LOsM+SqPKUUg08PDySVq5c6SIiSEpKwokTJ/KOHz9ecPr0aXX+/HlPpdSF/Pz8UBEp1TpeIrrK09Pz5eLi4sf9/f0LGjduXBwWFubSqlUrz6ZNm+qaNGmCRx99tPT48eNzCgsLeTGPg2BiaWeUUp0BLBWRTlrHQuQolFJLPDw8pnp6eibrdLrfcnJyjubl5Z0AkATgNIAzIpKnbZREZM2f0+3VB9AEQBMXF5dmPj4+bfR6fcuCgoKWxcXFhbm5uUHaRkkVxcTSznA+SyIiInJU7FuwMyJSopQy9lmu1TgcqiKlVBMA/wwKChohIt4iwn7m2kd0Ol1RTk7O/vz8/FdEZL3WARFVN6WUL4ChQUFBEwA0LS0t9dA6JidWotPpLmdnZ28oKCj4zF6mZmLF0g6xz9KxKaX+aTAY/jVo0CCMHz8ezZs3h8Fg0DosqmHFxcVIS0vD5s2bsXz5cuTm5iZnZWWFiUiR1rERVQelVG+DwbCpT58+MmHCBJ9WrVrBYDDAyi83kg0UFRUhNTUVGzZsKPjkk0+KCwsLV2RlZT2i9bRqTCztEPssHZdS6u9+fn7/iYuLQ+fOnbUOh+xEfn4+Bg0ahP3795+6cuVKC63jIbI1pdStPj4+cZs3b/bu2bOn1uHUOtnZ2YiKiso5ceLEh1euXPm7lrEwsbRD7LN0XEFBQVeWLFniO3r0aK1DITuTn5+POnXqIDs7u4OIHNQ6HiJbCgwMXP/KK6/E3H///VqHUmtduXIF9evXz8/Ly7tFRDK1ioN9X3aI81k6JqVUs/z8fN8hQ4ZoHQrZIQ8PDwwfPhxKqee0joXIlpRShvz8/L6jRo3SOpRazc/PD9HR0UUAYrSMg4ml/YoHf97R0Qzr0KGDsJ+SyjNw4EAEBATwPCE5m9ahoaGFAQEBWsdR6w0ePNjHx8dH06IUE0v7FQ8mlo7mlsBA/honlS8gIAA6nc7zxlsSORR/f39/rWMgXN3HuLq6ajrnJxNL+8XfDXc8Sq/Xax2Dybx586CUQnx8/E2NExUV5TBXddrqOVeXPz8fjvFiElWcXe37qtPkyZOhlEJSUpLWoVil1+uN12lohomlnWKfpfOJj4+HUgrz5s3TOhS6jmXLlkEphWXLlmkdCpFT0GrfZ69/aNprXLbCxNK+xYOnw6mKHn30URw9ehTdunW7qXFWrFiBo0eP2igqIiJyZvzlHfsWD2Cp1kGQYwoODkZwcPBNj9O4cWMbRENERLUBK5b2jX2WTmLevHmIjo4GAMyfPx9KKdPN2Ktj7N1JTEzEf/7zH7Rr1w7u7u6YPHkyAODcuXOYO3cuevTogbp168Ld3R1NmjTBjBkzkJqaavUxrz3dkpSUBKUUJk+ejMTERIwePRoBAQHw8vJCv3798Ouvv1qMY63Hsuzp4u3bt6Nnz57w8vJCUFAQJk2ahEuXLll9HZYsWYJ27drBw8MDjRo1wuzZs5Gfnw+lFKKioir8ep49exb33HMPAgMD4e3tjcjISOzcudPqtoWFhXjrrbcwYMAANGrUCO7u7qhbty5GjhyJn3/+2WzbyZMnY8qUKQCAKVOmmL1PRvv27cOjjz6K9u3bw8/PDwaDAeHh4Xj55ZdRVMQf1SEqqyL7PuDq9/T1119Hly5d4OXlBR8fH/Tq1Qvr11v+EuqVK1cwZ84ctG3bFt7e3vDz80Pr1q0xZcoUnD17FsDV/db8+fMBANHR0abHbNKkSYXiPnz4MO666y74+PjAz88PgwcPxqFDh6xue+XKFSxcuBCRkZEICQmBm5sbQkJCcN999+HUqVNm21Ykrri4OEydOhWtWrWCt7c3vL29ceutt+L999+vUOxaY8XSjvF3w51HVFQUkpKSsHz5ckRGRpolUddeTTlz5kzs3r0bQ4YMwV133YVbbrkFALBz50689tpr6Nu3L7p37w5XV1f8/PPPeO+997B161bs378ffn5+FYonKSkJ3bt3R9u2bTF16lScOnUK69atQ3R0NI4ePWp6zBvZsGEDNm7ciJiYGEyfPh07d+7EihUrcOrUKezatcts2zlz5uD5559H/fr18eCDD8LFxQVffPEFjh07VqHHMvrjjz9w++23IyUlBQMGDECXLl1w9OhR3HnnnaYDWFmXL1/G3//+d/Tq1QuDBw9GQEAAEhMTsX79enz11VfYuXMnbrvtNgDA8OHDkZGRgXXr1mHYsGHo1KmTxXgffPABNmzYgN69e2Pw4MHIzc1FfHw8nnnmGezduxdffvllpZ4PkTOryL6voKAAAwcORHx8PDp37oxp06ahqKgImzZtwrBhw/DWW2/h0UcfBQCICAYMGIA9e/YgIiICAwcOhE6nQ1JSEv73v/9h0qRJaNSokekP8oSEBEyaNMmUuFXk6vVDhw4hIiIC2dnZGDlyJMLCwvDjjz8iIiICHTt2tNj+6NGjmDNnDqKjozFixAh4eXnh2LFjWLVqFTZt2oT9+/cjNDQUACoU18KFC3Hy5En06NEDI0aMQEZGBrZs2YKHHnoIx48fx2uvvVbh118TIsKbHd8APAZgkdZx8Fah92phTExMqZQjLi5OAMjcuXOtrp80aZIAkIYNG0pycrLF+gsXLkhWVpbF8uXLlwsAeeGFF8yWz507VwBIXFycadnp06cFgACQl19+2Wz75557TgDIggULzJZHRkbK1V3FX5YuXSoAxMXFRXbt2mVaXlxcLFFRUQJAfvjhB9Py48ePi16vl8aNG0taWpppeVZWlrRr104ASGRkpNXX5VrG1+na57tkyRLTcyv7nPPz8+X333+3GOfQoUPi7e0t/fr1s/rcli5davXxk5KSpLi42GxZaWmpTJ06VQCYvR7X2rJliwQHB2eIHXxeeePNVjcAfSIiItKlHDfa9/3jH/8QADJv3jwpLf1rF5qZmSm33nqruLm5SUpKioiIHDhwQADIiBEjLMbJz88320da2wdWhHGf9+mnn5otf+aZZ0z7mNOnT5uWZ2RkyKVLlyzG2bFjh+h0Orn//vvNlt8orsTERItlRUVFcuedd4per7d6fDCKjY2VoKCg/4mGnweeCrd/8eAFPLXKU089ZbWvsW7duvD29rZYPnHiRPj6+mLbtm0VfoymTZviqaeeMls2bdo0AMDevXsrPM748eMRERFh+rder8ekSZMsxvnss89QUlKCJ554AkFBf02x5u3tjeeeq/gP0RQWFuLzzz9H3bp18cQTT5itu//++9GyZUuL+7i7u6NBgwYWy9u1a4fo6Gjs3LmzUqewQ0NDce3UKkopPPLIIwBQqfeBqLYrLS3Fe++9hxYtWmDOnDlmbSc+Pj6YM2cOCgsLsWbNGrP7WfshCnd3d6v7yMo4c+YMEhIS0KFDB9x7771m6/7xj39YrXj6+fnB2hzG0dHRaNeuXaX3CU2bNrVY5uLigocffhglJSWIi4ur1Hg1jafC7Z+xzzJIRKw3rpFTud5V3GvWrMGSJUuwf/9+pKeno6SkxLTu3LlzFX6Mjh07Qqcz/7uyYcOGAICMjIwKj9OlSxeLZdbGMfZu3nHHHRbbW1tWnuPHjyM/Px99+vSBh4eH2TqdToc77rgDv/32m8X9fvnlF7zyyivYtWsXzp8/b5FIpqWloX79+hWKobCwEG+//TZiY2Nx7NgxZGdnG6s2ACr3PhDVdsePH0d6ejpCQkJMvYdlXbx4EQBMLTNt2rRBeHg4Vq1ahbNnz2L48OHo1asXunTpYvEHX1UY91U9e1r+QJa3tzc6depkdZqg+Ph4LFq0CHv27EFaWhqKi4tN69zc3CoVQ1ZWFl599VWsXbsWp06dQk5Ojtl6e9/HMLG0c/JXn2UvsM+yViivv/G1117Dk08+iTp16qB///5o2LCh6a/2RYsWoaCgoMKPYa0X08Xl6u6gbLJqq3EyMzMBAHXq1LHYvqL9nMDVJnngavXWGmtjff/99+jTpw8AoH///ggLC4O3tzeUUli7di1+/fXXSr12o0ePxoYNG9CyZUuMHTsWdevWhaurKzIyMvDGG29Uaiyi2u7y5csArl4sc/jw4XK3MyZXLi4u2LFjB+bNm4c1a9aYzlwEBwdj5syZePbZZ28qwazKPuaLL77A2LFj4e3tjQEDBqBJkybw9PQ0XeCYnJxc4ccvLCxEVFQU9u/fj86dO2PixIkICgqCi4uLqVfV3vcxTCwdQzyung5fq2kUVCOs/cpNcXExnn/+eYSEhOCXX34xS9BEBK+88kpNhlhpvr6+AK5WH4xN7EYXLlyo8DjGRNbaVfDljfXiiy+ioKAAu3btMjttDwC7d++2eiV8efbu3YsNGzZgwIAB2LRpk9kBbPfu3XjjjTcqPBYR/bVvGDVqFFavXl2h+wQHB+Ptt9/GW2+9hWPHjmHHjh146623MHfuXLi6uuKZZ56pcjxV2cfMmzcPHh4e2LdvH8LCwszWxcbGVurx161bh/379+P+++/HBx98YDHW8uXLKzWeFthj6RjiwT5Lh2dMQipTETRKS0vDlStX0KNHD4uq308//YS8vDybxFhdjFdSfv/99xbrrC0rT6tWreDh4YGffvoJ+fn5ZutKS0utjnXq1CkEBgZaJJW5ubnYv3+/xfbXe5+MU4cMGTLEoiry7bffVvh5ENUm1/tOtWnTBr6+vvjpp58qPV2XUgpt2rTBI488gm+++QYAzKYnqso+17ivunZWCwDIzs7GL7/8YrH81KlTaNOmjUVSee7cOYvphm4Ul3H7oUOHWqxzlH0ME0vHwPksnYCxufv333+v9H3r1q0Lg8GA/fv3Izc317Q8PT0dM2fOtFmM1WXcuHHQ6XR4/fXXzea4zMnJwYsvvljhcdzc3DBmzBikpqZaTLnx4YcfWu2vDA0NRXp6utlptpKSEjz55JOm/q2yrvc+Gaut1x50Dh8+jAULFlT4eRDVJtf7Trm4uGD69OlITk7Gk08+aTW5PHTokKmCePr0aRw5csRiG2MlsexFPVXZ5zZu3Bi9e/fGgQMHsHLlSrN1L730ktUe9NDQUJw8edKsmpmfn4/p06eb9VpWJK7y9jEJCQkWFUx7xVPhDkA4n6VTaN26NUJCQhAbGwtPT080bNgQSilMnz79hvNP6nQ6zJgxA6+99ho6duyImJgYZGZm4quvvkJoaChCQkJq6FlUTatWrfD000/jpZdeQnh4OO6++264uLhgzZo1CA8Px6FDhywuJirPyy+/jO3bt+O5557Drl270LlzZxw9ehSbN29G//798fXXX5ttP3PmTHz99dfo2bMnxowZAw8PD8THxyMlJQVRUVEWjfi33347DAYDFi1ahMzMTFOF+Omnn0a3bt3QrVs3/Pe//8Uff/yBHj164MyZM1i/fj2GDBlS4VN5RLXJjfZ98+fPx/79+/Hmm29i06ZNiIyMRJ06dZCSkoKDBw/i119/xQ8//IC6devi119/xYgRI3Dbbbehffv2qFevHlJSUrB27Vro9Xqz2SKME5A/++yzOHbsGPz8/ODn54fp06dfN9533nkHERERuO+++7B27VqEhYVh7969+PHHH9GrVy+LyuHMmTMxc+ZMdO7cGaNHj0ZxcTG++eYbiAg6duxo0W5zvbhiYmLQpEkTvPLKKzh06BDat2+P48ePY+PGjRg+fLhjzJOr5VxHvFX8Bs5nafc33GAeSxGR3bt3S2RkpPj4+FjMh2acn7Hs/GhlFRYWyosvvihhYWHi7u4ujRs3lscff1yysrIkNDRUQkNDzba/3jyWkyZNsvoYsDKf5PXmsbQ21+P15qx79913pU2bNuLm5iYNGzaUJ598Us6ePSsAZNiwYVZjsiY5OVnGjh0r/v7+4unpKb169ZKEhIRy54dbvXq1dOnSRTw9PSU4OFjGjBkjp06dKvc137Rpk9x2221iMBhM75NRamqqTJ06VUJCQsTDw0PCw8PlnXfekcTExOu+tiKcx5I357zhBvNYilx/3ydydQ7cJUuWSEREhPj6+pr2cQMHDpT33ntPsrOzRUTk7Nmz8vTTT0uPHj2kbt264ubmJo0bN5bRo0fLnj17LB532bJlEh4eLu7u7gLAYj9ZnoMHD8rgwYPF29tbfHx8ZNCgQXLw4EGr+4zS0lJZvHixtGvXTjw8PKRevXoybdo0uXDhgtX9543iSkxMlFGjRkmdOnXE09NTbrvtNomNjb3hfKAi9jGPpRL5a5oMsl9Kqc4AlopIJ61jIeuUUgtjYmKeWr9+veXVN1Subdu24c4778Ts2bOxcOFCrcOpVlu3bsWECROuXLx40V/rWIhsRSnVJyIi4stdu3b5ax1Lbff555/jkUceWZuWljZCqxjYY+k42GdJDu3ixYsWzeoZGRmmKziHDx+uQVRERGRL7LF0EMI+S3JwK1euxKuvvoo+ffogJCQEf/zxB7Zs2YLU1FRMnjwZt99+u9YhEhHRTWJi6Vjiwfks7ZlUZSqh2uKOO+5A165dsW3bNly+fBl6vR5t2rTBP//5T8yYMUPr8GrEn58P9h+Rs+G+z06UlJRARDR9M5hYOpZ4AEu1DoLKdcH4KxJkqVu3bli3bp3WYWgqPT0dpaWluTfeksihZFTmp2Cp+qSnp6OoqEjTn39mj6VjYZ+lffvfgQMHlL1PVk7a2bJlC9LT0y1nXiZybMeSk5Pd0tPTtY6j1tu8eXNWVlbWTi1jYGLpQP4sbxv7LMnOiEiSh4dH5qZNm7QOhexQfn4+1q5dCxF5QetYiGxJRPI8PDy2O8Qci07sypUriIuLcwWwQcs4mFg6nnjw5x3t1uXLl+fef//9+Pnnn7UOhexIfn4+Bg0aBJ1Od0pEDmodD5Gtpaen/+vxxx/PtvZTiFT9srOz0bdv3xxXV9clIpKpZSycx9LBcD5L+6eU+qfBYPjXgAEDMGHCBDRr1gwGgwFKcXrL2qS4uBhpaWnYtGkTVqxYgdzc3OSsrKwwEancDyITOQilVG+DwbApOjpaJk6c6NOyZUt4enpy31dNioqKkJqaivXr1xd8+umnxYWFhSuysrIeEY0TOyaWDkYppQeQCiBMRHiliJ1SSjUB8FxQUNBIEfEWEZ4dqH1Ep9MV5eTk7M/Pz39FRNZrHRBRdVNK+QKICQwMnKiUalJaWmq44Z2oqkr0ev3lzMzM9YWFhbEi8pvWAQFMLB2SUmodrlYt12odCzkmpVQbAF1EZKXWsRARkfNgFcUxxYN9llQFSim9Umo2gDgABVrHAwBKqS1/VjmIiDSjlHpKKRWgdRyOjomlY4oHE0uqpD+rlN8BiATQVURWaxyS0e8AZmodBBHVemMBNNA6CEfHxNIxcT5LqrBrqpSLAdwlIikah1XWSwBmsWpJROT4mFg6IM5nSRVlpUq5TOsrBq8lIom4Ou8aq5ZERA6OiaXjigdPh1M5HKBKeS1WLYmInAATS8cVDyaWZIUjVCmvxaolEZFzYGLpuNhnSWYcsEp5LVYtiYgcHBNLB8U+SyrLEauU12LVkojI8TGxdGzx4OnwWs0JqpTXYtWSiMiBMbF0bPFgYllrOUOV8lqsWhIROTYmlo6NfZa1kBNWKa/FqiURkYNiYunA2GdZ+zhjlfJarFoSETkuJpaOLx48He70akGV8lqsWhIROSAmlo4vHkwsnVptqFJei1VLIiLHxMTS8bHP0knVwirltVi1JCJyMEwsHRz7LJ1TbaxSXotVSyIix8PE0jnEg6fDnQKrlBZYtSQiciBMLJ1DPJhYOjxWKS2xaklE5FiYWDoH9lk6MFYpb4hVSyIiB8HE0gmwz9JxsUp5Y6xaEhE5DiaWziMePB3uMFilrDRWLYmIHAATS+cRDyaWDoFVyspj1ZKIyDEwsXQe7LO0c6xS3jRWLYmI7BwTSyfBPkv7xirlzWPVkojI/jGxdC7x4Olwu8Iqpc2xaklEZMeYWDqXeDCxtBusUtoeq5ZERPaNiaVzYZ+lHWCVstqxaklEZKeYWDoR9llqj1XK6seqJRGR/WJi6XziwdPhNY5VyhrHqiURkR1iYul84sHEskaxSlnzWLUkIrJPTCydD/ssawirlJpj1ZKIyM4wsXQy7LOsGaxSao9VSyIi+8PE0jnFg6fDq4VSyoVVSrvCqiURkR1hYumc4sHE0uauqVLeyiql9li1JCKyL0wsnRP7LG3ozyrl/+FqlfI9XK1S/q5xWPQXVi2JiOwEE0snxD5L2ylTpewN9lLaJVYtiYjsBxNL5xUPng6vsnKqlOyltF+sWhIR2QEmls4rHkwsq4RVSsfDqiURkX1gYum82GdZSaxSOjxWLYmINMbE0kmxz7JyWKV0fKxaEhFpj4mlc4sHT4dfF6uUTodVSyIiDTGxdG7xYGJZLlYpnQ+rlkRE2mJi6dzYZ2kFq5ROj1VLIiKNMLF0YuyztMQqpfNj1ZKISDtMLJ1fHHg6nFXK2odVSyIiDTCxdH7xqOWJJauUtQ+rlkRE2mBi6fwOAGhYG/ssWaWs9Vi1JCKqYUwsnZyIlKIW9lmySkmsWhIR1TwmlrVDPGrJ6XBWKekarFoSEdUgJpa1QzxqQWLJKiVdi1VLIqKaxcSydnDqPktWKekGWLUkIqohTCxrAWfus2SVkm6EVUsioprDxLL2iIcTnQ5nlZIqiVVLIqIawMSy9oiHkySWrFJSZbFqSURUM5hY1h4O32fJKiXdJFYtiYiqGRPLWqJMn2WkUipcr9fPCggI2KiU6qx1bBXBKiXdLFYtiYiqHxNLJ6eU0hkTSX9//zYGg+Hz0NDQXffff/9CAHcCcNE6xuthlZJsjFVLIqJqpFj0cV5KKR83N7ek4OBgt7vuusvlzjvv9Ojduzfq1q2LvLw8+Pv7FxQWFvqKSKGGMdZRSvUrLS39zMq6NgCWAUgD8CATSrIFpdSHAE6LyItax0JE9kMp9ROAySJySOtYHBkrlk5MRLLc3d0/b9mypXrnnXc8Ro8ejbp16wIA9u3bBx8fnxNaJpUA4Ofn96ZSaqVSKtK4jFVKqmasWhIRVRMmlk4uKytr5v79+3+YPn16ftnq9Pfffy/5+fk7NAwNSqkuer1+6OrVq5W3t3esUsqLvZRU3dhrSURUfZhYOjkRKcnMzBz++eefJ7/++uslxuU7duzIysnJSdAqLqWU8vPz++A///mP54gRIzBq1Cg/T0/P7WCVkmoGq5ZERNWAPZa1hFIqxNvb+9dPPvkkeNiwYQgICMi9cuVKCxH5Q4t49Hr9qNatWy87ePCgt06nQ2ZmJlq0aFFy8eLFMSKyRouYqHZhryURlcUeS9tgxbKWEJFz2dnZfSdOnJi9Zs0alJSU5GuVVCqlPDw9Pd99//33vXW6qx9BX19frFy5Uu/t7f2OUspLi7io1mHVkojIxphY1iIiciA3N3fMuHHj4OLikqhVHAaD4cl+/fr5REREmC2/8847MWrUKD9fX9/XNQqNahH2WhIR2R5PhddC7u7u/y4qKjpfWlr6Wk0/tlKqgYeHR9LKlStdRARJSUk4ceJE3vHjxwtOnz6tzp8/76mUupCfnx/656TuRNVGKdUMwA8AwkQkU+t4iEg7PBVuG0wsqUYppZZ4eHhM9fT0TNbpdL/l5OQczcvLOwEgCcBpAGdEJE/bKKk2Ya8lEQFMLG2FiSUR1WqsWhIRwMTSVuz65/wqQykVAGB4UFDQBBFpJCLuWsdETq9Ur9enZ2dnf5Wfn79KRA5rHRBVnogkKqWMvZasWhIR3QSnqFgqpQYZDIbVgwYNKh0/frx38+bNYTAYtA6LnFxxcTHS0tLw1VdfFS5btqwwLy9vY2Zm5gQRKbnxvcmesGpJRKxY2obDJ5ZKqSg/P7+NcXFxXp07d9Y6HKql8vPzMWjQoJyff/75y4yMjElax0OVx15LotqNiaVtOPx0Q0FBQS98+OGHTCpJUx4eHvjqq6+8SkpK7lZK1dM6HqoSzmtJRHSTHDqxVEoF5uXldR0yZIjWoRDBw8MDw4cPF6XUKK1jocrjvJZERDfPoRNLAB07dOiQz35KshcDBw70DAwM7KN1HFRlrFoSEd0ER08s/YOCgpTWQRAZBQQEQCkVqHUcVDWsWhIR3RxHTyyVXq9nYllNli1bBqUUli1bZra8SZMmaNKkyU2PY0vz5s2DUgrx8fHV9hgVodfroZTSaxoE3SxWLYmIqsjRE0uqJeLj46GUwrx587QOhZwcq5ZERFXnNBOkU83Zvn271iFYePTRRzFu3Dg0btxY61DIObwE4Ael1Fuc15KIqOKYWFKlNW/eXOsQLAQHByM4OFjrMMhJ8Nd4iIiqplacCv/2228xYsQI3HLLLXB3d0ejRo0wcuRI7Nq1C4B5f97y5cvRtWtXeHp6IioqyjTGmTNnMG3aNDRo0ABubm5o2LAhpk2bhrNnz1o83h9//IG//e1vCAsLg8FgQGBgIMLDwzFjxgxkZv5V/Lhy5QrmzJmDtm3bwtvbG35+fmjdujWmTJliddxrNW/eHD4+PsjNzbW6/s4774ROp8OZM2dMj7dw4UJERkYiJCQEbm5uCAkJwX333YdTp05V+PUsr8fy8uXLePjhh3HLLbfA09MTt912G/73v/+VO87HH3+MYcOGoUmTJvDw8EBgYCAGDBiAuLg4s+3mzZuH6OhoAMD8+fOhlDLdkpKSTNuU12O5ceNGREdHw8/PDwaDAZ06dcKiRYtQUmL+AzlJSUlQSmHy5MlITEzE6NGjERAQAC8vL/Tr1w+//vprhV8jcgrstSQiqiSnr1i+8847mDlzJgwGA0aMGIHGjRsjJSUFu3btwurVq9GzZ0/Ttv/+978RFxeHoUOH4s4774SLy9WX58SJE+jZsydSU1MRExODdu3a4fDhw/j444+xceNGfPfdd2jRogUAIDc3FxEREUhKSkL//v0xYsQIFBYWIjExEcuWLcPs2bPh6+sLEcGAAQOwZ88eREREYODAgdDpdEhKSsL//vc/TJo0CY0aNbruc5swYQL+9a9/Yd26dbjnnnvM1v3xxx/YsWMHevfubTo9fPToUcyZMwfR0dEYMWIEvLy8cOzYMaxatQqbNm3C/v37ERoaWqXXOTc3F1FRUTh48CBuv/12REZG4uzZsxg7diz69+9v9T6PPPIIOnbsiH79+qFOnTpISUnB2rVr0a9fP6xZswbDhg0DAERFRSEpKQnLly9HZGSkWcLv7+9/3bjeeOMN/P3vf0dgYCDGjx8PLy8vbNiwAY899hi+/fZbrF69GkqZX/+VlJSE7t27o23btpg6dSpOnTqFdevWITo6GkePHsUtt9xSpdeIHAurlkREVSAiDnsDMHLo0KFXpBwHDhwQvV4vISEhcvr0abN1paWlkpKSIiIic+fOFQDi5eUlBw4csBinT58+AkCWLFlitnzJkiUCQPr27Wtatn79egEgjz32mMU4mZmZUlBQYIoNgIwYMcJiu/z8fMnKyirvaZmcOHFCAMjgwYMt1r366qsCQD788EPTsoyMDLl06ZLFtjt27BCdTif333+/2fKlS5cKAFm6dKnZ8tDQUAkNDTVbZnwNH3jgAbPlW7duFQBWx0lMTLSI5dy5cxISEiJhYWFmy+Pi4gSAzJ071+I+ZR8/Li7OtOzUqVPi4uIidevWlTNnzpiWFxQUSGRkpACQTz75xLT89OnTplhffvlls/Gfe+45ASALFiyw+vhGW7ZskTp16uwUO/h+8GaTfUwzABcA+GodC2+88Va9NwA/AWivdRyOfnPqU+GLFy9GSUkJXnjhBYtTt0ophISEmC178MEHER4ebrbs7Nmz2LFjB9q2bYsHHnjAbN0DDzyANm3aYPv27Ranrq1N2u7j4wM3N7cbbufu7g5vb+8bPr8WLVqge/fu+Prrr3Hx4kWzdZ9++ik8PDwwevRo0zI/Pz8EBlpOsRgdHY127dph27ZtN3zM8qxYsQJubm7417/+Zba8f//+6Nu3r9X7NG3a1GJZ/fr1MWrUKJw4cQLJyclVjgcAVq5cieLiYjzxxBNm1V83Nze8/PLLAGB1CqSmTZviqaeeMls2bdo0AMDevXtvKiZyLMIrxImIKsWpT4X/+OOPAFDuqdhrdevWzWLZzz//DACIjIy0OGWqlELv3r1x9OhR/Prrr2jUqBF69+6NevXqYcGCBfjll18wZMgQ9OzZE+Hh4Wb3b9OmDcLDw7Fq1SqcPXsWw4cPR69evdClSxfo9X9Ng5iUlGSR/Pj7++Pvf/87AGDixInYs2cPYmNjMXPm1WPf4cOH8csvv+Duu++Gn5+f2X3j4+OxaNEi7NmzB2lpaSguLjatuzbpraisrCycPn0abdu2Rb16lj+T3atXL6tXkicmJmLBggXYsWMHUlJSUFBQYLb+3LlzVT41D/z13pU9dW7Uo0cPGAwG/PLLLxbrOnbsCJ3O/G+uhg0bAgAyMjKqHA85rBcB7OYV4kREN+bUiWVGRgaUUqhfv36FtrfWO2e82Ka8vjpjInXlyhUAV6uCP/zwA+bOnYsNGzZg8+bNAK4mJs888wxmzJgBAHBxccGOHTswb948rFmzBk888QSAq1c3z5w5E88++yz0ej2SkpIwf/58s8cMDQ01JZbjxo3DY489hpUrV5oSy08++QTA1aSzrC+++AJjx46Ft7c3BgwYgCZNmsDT09M0eXlVK4TG5163bl2r6629didPnkS3bt2QmZmJ6OhoxMTEwNfXFzqdDvHx8UhISLBINCvrRu9d3bp1kZKSYrH82mQcgKnf9toLfsj5ichp9loSEVWMUyeW/v7+EBH88ccfaNCgwQ23v7YiCQC+vlcvCL1w4YLV+xiXG7cDrl41vXz5cpSUlODgwYP4+uuv8eabb+KRRx5BQECA6UKb4OBgvP3223jrrbdw7Ngx7NixA2+99Rbmzp0LV1dXPPPMM4iKijL2flgVFBSEQYMGYf369Th58iSaN2+OVatWITg4GAMHDjTbdt68efDw8MC+ffsQFhZmti42NvaGr095jM89NTXV6nprr91//vMfpKen49NPP8W9995rtu7hhx9GQkJCleO5Nq4LFy5YrXympqaavW9E18F5LYmIKsCpeyyNp7a//vrrKo/RqVMnAMDOnTstEjwRwbfffmu2XVl6vR6dOnXC7Nmz8dlnnwEA1q9fb7GdUgpt2rTBI488gm+++abc7cozYcIEAFf7KhMSEkxXY7u6upptd+rUKbRp08YiqTx37lylphu6lq+vL5o2bYqTJ0/i/PnzFuuNr9G1sQDA0KFDzZaXlpbiu+++s9je2B5QmYph586dAcDqFEQ//vgj8vLyrL5vRNdiryURUcU4dWL58MMPQ6/X47nnnrM4zWusZN5I48aNER0dbZpeqKyPP/4Yhw8fRp8+fUwXhxw6dMjqKWVj1c54sc7p06dx5MiRG25XETExMfDz88PKlSvLPQ0OXD2FfvLkSbMKYn5+PqZPn27Wa1kVEydORGFhIebMmWO2/Ouvv7baX2msIBrnEjVauHAhDh06ZLG98aKj33//vcIxjR8/Hi4uLnj99ddx7tw50/KioiI8/fTTAIDJkydXeDyq9TivJRHRDTj1qfDw8HAsWrQIs2bNQrt27TB8+HCEhobi/Pnz2LlzJ4YMGYJFixbdcJz33nsPPXv2xAMPPIANGzagbdu2OHLkCNavX486dergvffeM227bds2PPHEE4iIiEDr1q0RFBSExMRErF+/HgaDAY8++igA4Ndff8WIESNw2223oX379qhXr55pHke9Xm/quawIDw8P3H333fjwww+RnJyMsLAwdO/e3WK7mTNnYubMmejcuTNGjx6N4uJifPPNNxARdOzY8aYmAJ89ezbWrFmDDz74AIcPH0bv3r1x9uxZ/Pe//8WQIUOwadMms+0ffvhhLF26FCNHjsTYsWMRFBSE3bt3Y//+/Va3b926NUJCQhAbGwtPT080bNgQSilMnz7dak8kcHUC+YULF+KJJ55Ahw4dMGbMGHh5eWHjxo04duwYhg0bZqr2Et2IcF5LIqIb03q+o5u54QbzWBrFxcXJXXfdJYGBgeLm5iYNGzaUUaNGyXfffSci1udAvFZSUpJMmTJF6tevLy4uLlK/fn2ZMmWKJCUlmW135MgR+dvf/iadO3eWoKAgcXd3l2bNmsnkyZPlyJEjpu3Onj0rTz/9tPTo0UPq1q0rbm5u0rhxYxk9erTs2bPnRk/JQkJCgmkOxvnz51vdprS0VBYvXizt2rUTDw8PqVevnkybNk0uXLhgmtexrMrMYykicunSJXnwwQelTp064uHhIV27dpU1a9aUO05cXJxERESIj4+P+Pv7y+DBg2Xfvn3lvh+7d++WyMhI8fHxMT1X4/yk13sP161bZ7qfu7u7hIeHy2uvvSZFRUVm2xnnsZw0aZLV1w+AREZGWl1nxHksnfsGzmvJG29OewPnsbTJTf35YjokpdTIoUOHLl23bh1PTZFd2Lp1KyZOnPhtampqb61joeqhlPoQwGkRYdWSyIkopX4CMFlELPuxqMKcuseSiKgasNeSiKgcTCyJiCpBeIU4EVG5HD2xlJKSEsc9l09Op6SkBCLCWdSdH6uWRERWOHpimXHp0iUmlmQ30tPTISKXtY6DqherlkRE1jl6YvnLgQMHPPLy8rSOgwgAsGXLltzLly/v0DoOqhGsWhIRXcOhE0sRSTcYDPuunfOQSAv5+flYu3atEpEvtY6Fqh+rlkRElhw6sQSAS5cuPXf//ffn/Pzzz1qHQrVYfn4+Bg0alKPX678QEcvftSRnxaolEVEZDj2PpZFSapDBYFg9cODA0nvvvde7WbNmMBgMUEppHRo5seLiYqSlpWHz5s2Fy5cvL8zLy9uYmZk5gRfv1C6c15LIOXAeS9twisQSAJRSAQCGBQUFTRSRRiLirnVM5PRK9Hp9RnZ29ub8/PzPROSw1gFRzVNKNQPwA4AwEcnUOh4iqhomlrbhNIklEZFWWLUkcnxMLG3D4XssyTaUUk/+WfUlospjryUREZhY0l/GAWigdRBEjohXiBMRXcXEkojINli1JKJaj4klEZENsGpJRMTEkojIlli1JKJajYklEZGNsGpJRLUdE0siItti1ZKIai0mlkRENsSqJRHVZkwsiYhsj1VLIqqVmFgSEdkYq5ZEVFsxsSQiqh6sWhJRrcPEkoioGrBqSUS1ERNLIqLqw6olEdUqTCyJiKoJq5ZEVNswsSQiql6sWhJRrcHEkoioGrFqSUS1CRNLIqLqx6olEdUKTCyJiKoZq5ZEVFswsSQiqhmsWhKR02NiSURUA1i1JKLagIklEVHNYdWSiJwaE0siohrCqiUROTsmlkRENYtVSyJyWkwsiYhqEKuWROTMmFgSEdU8Vi2JyCkxsSQiqmGsWhKRs2JiSUSkDVYticjpMLEkItIAq5ZE5IyYWBIRaYdVSyJyKkwsiYg0wqolETkbJpZERNpi1ZKInAYTSyIiDbFqSUTOhIklEZH2WLUkIqfAxJKISGOsWhKRs2BiSURkH1i1JCKHx8SSiMgOsGpJRM6AiSURkf1g1ZKIHBoTSyIiO8GqJRE5OiaWRET2hVVLInJYTCyJiOwIq5ZE5MiYWBIR2R9WLYnIITGxJCKyM6xaEpGjYmJJRGSfWLUkIofDxJKIyA6xaklEjoiJJRGR/WLVkogcChNLIiI7xaolETkaJpZERPaNVUsichhMLImI7BirlkTkSJhYEhHZP1YticghMLEkIrJzrFoSkaNgYklE5BhYtSQiu8fEkojIAbBqSUSOgIklEZHjYNWSiOwaE0siIgfBqiUR2TsmlkREjoVVSyKyWy5aB0DaUEp5AahTZpEbgBClVPaf/84VkdSaj4yIrkdEEpVSxqrli0opnVJqlJeXV6+srKxZWsdH5Cj+PA7WLbPIDUADpVTOn//OFZELNR+ZY1MionUMpAEPD4/FSqmpgYGBeQAgIt4A8pRSJQUFBS4ZGRmqpKTES/gBIbI7SqlmAH5QSj3l4+Pzr0aNGgUnJSUVZ2dn+2sdG5Gj8PDwWKzT6UzHwdLSUm+lVK5SqjQ/P5/HwSriqfBaqqCg4D0vL6/CxMRE35SUFN9z587pzp0755WSkuJ73333uXh6en7ELxOR/fmzQtnVx8fHs1OnTu+vXbs2ND4+3kvruIgcTUFBwXuenp6Fp06d8v399999z507p0tJSfH+/fffeRy8CUwsaykR+bWkpOS7jz76yOxLk5qaiiVLlhRnZWW9oFVsRGSdXq8f7uvrm9ilS5eP1q1b571v3z736OhoKKW0Do3I4RiPgx9++CGPgzbExLIWy8jImD1nzpzcgoIC07KXX365UK/Xf8y+EiL7opRSHh4eC9q2bRvy7bff+jChJLp5GRkZs+fOncvjoA0xsazFrq1a8q80IvslIpKbm3vr0aNH13bs2DHnt99+0zokIod3bdWSx8Gbx8SylitbtVywYEER/0ojsl8ikpORkTEmOTl5VteuXXNiY2O1DonI4ZWtWvI4ePOYWNZyxr/WXnrpJbz//vsl/CuNyP4VFBR8nJ2d3f3BBx88M23atIL8/HwA4HlxoiowHgdffPFFHgdtgNMNEZRSHQH84uXl9Vl2dvZ4reMhoopRSnn5+fkt9fX1HXn58uXi7OxsD61jInJEPA7aDhNLAgDo9fpPS0tLnxORJK1jIaLKcXNzm+3i4vJUbm5unRtvTUTW6PX6T/48DiZrHYsjY2JJRERERDbBHksiIiIisgmH/a1wpZQvgKFBQUETADQtLS1lbxFprUSn013Ozs7eUFBQ8JmIcD4YG1FXJ2y8zcvLa4LBYIgsKSnxAy9WoRqk0+nyROTU5cuXlwPYJCI5N7yTBpRSOgB3+Pj4THBzc4soLS31Ab8rVA10Ol0+gNOXLl36FMB6EckEHPRUuFKqt8Fg2NSnTx+ZMGGCT6tWrWAwGDhZMGmqqKgIqamp2LBhQ8Enn3xSXFhYuCIrK+sR/iTYzVFKGXx9fbcGBQV1mjJlimffvn31/v7+0Ov1WodGtYSIIDc3F0eOHMGKFSsyv/vuO8nNze0rIvu0jq0spZSPr69vQr169ZpPmzbNu3fv3jo/Pz/odDw5SbYlIsjLy8Px48fx6aefZu3YsUPl5eUNEZGdDpdYKqVu9fHxidu8ebN3z549tQ6HyKrs7GxERUXlnDhx4sMrV678Xet4HJVSSvn6+iYMHTr01uXLlxt4gCR7sHXrVowePTozOzv7dhE5onU8AKCUcvH19d173333tX7zzTc9WGihmrRr1y4MHjw4OysrK9rh9tIBAQFzXn/9dSaVZNe8vb2xfft2r6Kioof+bNugqrktKCioE5NKsicDBgzAnDlzvH19fZ/QOpYy+jRr1qwZk0rSQs+ePfH66697BwQE/NOh9tRKKUN+fn7fUaNGaR0K0Q35+fkhOjq6CECM1rE4Ki8vrwlTpkzxZFJJ9mbcuHG64uLi0Uopu+jJ8PPzmzxt2jQfJpWklZEjRyI/P7+fo+2tW4eGhhYGBARoHQdRhQwePNjHx8ent9ZxOCqDwRDZt29fuzhwE5XVqFEj+Pr6KgCNtY4FAHQ63R19+vRhVkmaCQwMROPGjYscLbH09/f31zoGogoLCAiAq6trkNZxOKqSkhI/fufJXvn5+ZUA8NM6DgAoKiry5neFtBYQECCOlliq610JGh8fD6UU5s2bV3MREV2HXq+HvZwqc1DX/c47sqioKLubyUIphaioKK3DuCmTJ0+GUgpJSUlVHqOix5I/P5v2chy97nelsLAQzz33HJo3bw43NzcopRAfH6/ZcdMW75NW7DH2ZcuWQSmFZcuWaRqHXq+3my+E07HHg4azadKkCZo0aaJ1GOQk5s2bZzrYEjmbV199FS+++CIaN26M2bNnY+7cudx/OpCkpCQopTB58mStQ7khh50gnYjIka1YsQK5ublah0G1xObNm+Ht7Y2vv/4arq6upuV169bF0aNHERwcrGF0dLNGjBiBHj16oH79+lqHwsSSiEgLjRvbxTUfVEucO3cOQUFBZkklAHh6eqJ169YaRUW24ufnBz8/u2j3dd5T4Tt37kRkZCS8vb0RGBiI8ePH4/fff7e6bWpqKh577DG0aNEC7u7uCA4OxqhRo3Do0CGLbU+cOIEpU6agadOm8PDwQHBwMLp06YInnvhrOjOlFBISEkz/b7xVpIRt7HE6e/Ysxo4di6CgIHh5eSEqKgrff/+9xfa//fYbZs+ejS5duiAoKAgeHh5o2bIlnn76aWRnZ1tsbzxFX1BQgDlz5qBFixZwdXU19dfczHj/+Mc/0LhxYxgMBnTt2hXbtm0DAGRlZWHWrFlo0KABPDw8cPvtt+Onn36q8nthPCWQnJyM5ORks9f42j6hnTt3IiYmBsHBwXB3d0dYWBiee+45i0pR2T6jH374AQMGDIC/v79ZO0NcXBwGDRqEkJAQuLu7IyQkBFFRUfjwww+tv5lkQSmlU0r1U0rd9B+1FXnPRAQff/wxIiIi4OvrC09PT9x66634+OOPzcaKiorC/PnzAQDR0dGmz1PZU4XG1ouMjAzMmjULjRo1gouLi6mnad++fXj00UfRvn17+Pn5wWAwIDw8HC+//DKKioos4rfWLlO2T2r79u3o2bMnvLy8EBQUhEmTJuHSpUtWX4sDBw5g3LhxqF+/Ptzc3BAaGoqZM2eWu/2HH36I9u3bw8PDA40aNcLs2bORn59/w9e8LGOfWWJiIl599VW0bNkSBoMBbdu2RWxsLICrv0Y1Z84c0/6yQ4cO2Lp1q9Xxzpw5g2nTpqFBgwZwc3NDw4YNMW3aNJw9e9bq9ocPH8Zdd90FHx8f+Pn5YfDgwVb32WWtW7cOffv2RUBAADw8PNC+fXu8+uqrKCkpqdRzrwlKKRelVJ8/f6KxyowtHqdPnzbbXxp7acvrsTR+3nNycvD444+jQYMGcHd3R4cOHbB69WqLx6nssaMqKvI5T05Ohk6nQ9++fa2OkZ+fDz8/P7Ro0cKmsV+vx7G81/h///sf7rnnHrRo0QKenp7w8/NDr1698OWXX1qM3bRpUwDA8uXLzY55xtad6z3+999/jyFDhiAwMBAeHh5o3bo15s2bZ/WMifGzcfHiRUydOhV169aFwWBAjx49Ktwm5JQVy927d2PBggUYMmQIZs2ahf379+Ozzz7Drl27sHfvXtxyyy2mbU+dOoWoqCikpKSgf//+GD58OFJTU/Hll19i69at2L59O7p37w7g6l983bp1Q05ODoYMGYKxY8ciOzsbJ06cwFtvvYXXXnsNADB37lwsW7YMycnJmDt3rumxOnXqVKH409PTERERgfr16+PBBx9ESkoKPv/8c0RHR2Pr1q1mzfVr1qzBRx99hOjoaERFRaG0tBS7d+/GwoULkZCQgJ07d1r8hQpcnW/q119/xYABAxAYGIhmzZrd1Hhjx47FwYMHMXToUOTl5WHlypW466678P333+Ohhx5Cfn4+Ro8ejYsXL+Lzzz/HgAEDcPr0afj6/jV3eEXfC39/f8ydOxeLFi0CAPz97383jVH2tVm8eDFmzJiBgIAAxMTEoE6dOti7dy9efPFFxMXFIS4uDm5ubmbP4/vvv8dLL72E6OhoPPjggzhz5gwAYNOmTYiJiYG/vz+GDRuG+vXr4+LFi/jll1+wcuVK3H///RV6bwlNAXzt4+Pzh4uLy9MlJSWfiUjxzQxY3nsmIpgwYQJWrVqFli1bYvz48XBzc8M333yDadOm4ciRI3j11VcBwPRHX0JCAiZNmmRKKK+9yragoAB9+vRBVlYWYmJi4ObmZtqffPDBB9iwYQN69+6NwYMHIzc3F/Hx8XjmmWewd+9ei4PF9WzYsAEbN25ETEwMpk+fjp07d2LFihU4deoUdu3aZbbt+vXrMWbMGOj1egwdOhSNGjXCkSNH8Pbbb2Pr1q3Ys2cPyk7R9vzzz2POnDm45ZZb8MADD8DV1RWff/45jh49WpmX3eTxxx/Hnj17EBMTA71ej9jYWIwfPx4BAQF45513cOjQIQwePBj5+flYtWoVhg4dimPHjpkOlMDVP9h79uyJ1NRUxMTEoF27djh8+DA+/vhjbNy4Ed99951ZInDo0CFEREQgOzsbI0eORFhYGH788UdERESgY8eOVuP8xz/+gQULFqBhw4YYNWoUfH19sXPnTjz11FPYs2cPvvjiiyo9/2oUrpTa7uPjk6zX658sLS1dIyKllR3EuE+8dn9Zkf7KoqIi9O/fH5cvX8bIkSORm5uL2NhYjBkzBlu2bEH//v1N21b12FFRFf2ch4aGolevXoiPj0dKSgoaNGhgNs66deuQmZmJxx57rMZiL88zzzwDNzc39OzZ03RMWb9+PUaPHo0333wTM2fOBHA1d/jb3/6GN954Ax07dsTw4cNNY9zoffzyyy8xbtw4uLm5YezYsahbty62bduG+fPn4+uvv0ZcXBzc3d3N7pORkWH6Y/zee+9Famqq6bi9b98+tG/f/vpPTEQc5gagT0RERLqUIy4uTgAIAPnwww/N1s2fP18AyNSpU82W33HHHeLi4iJff/212fLjx4+Lj4+PhIeHm5a9+eabAkDeeOMNi8e+ePGi2b8jIyPl6stbOcb4J06cKKWlpabl8fHxopSSFi1aSElJiWn577//LgUFBRbjGJ/vp59+ajWuTp06yaVLlyzuV9XxIiIiJDs727Q8NjZWAIi/v7/cfffdUlRUZFq3cOFCASCvv/662ViVeS9EREJDQyU0NNQiVhGRw4cPi4uLi3Tu3NnieS5YsEAAyKuvvmpaVvaz89FHH1mMN3LkSAEgv/76q8W6tLQ0qzGIXH0dgoKC/id28P2xhxuA5g0bNrzy7bffyh133JHl4+OTotfrJwJwsbZ9QEBA8rFjx6y+tjd6z95//30BINOmTTP7/BUUFEhMTIwAkJ9++sm0fO7cuQJA4uLirD5eaGioAJD+/ftLbm6uxfqkpCQpLi42W1ZaWipTp04VALJr1y6zddb2EUuXLhUA4uLiYrZ9cXGxREVFCQD54YcfTMvT0tLE19dXGjZsKMnJyWZjrVq1SgDIo48+alp24sQJcXFxkQYNGsiFCxdMy69cuSKtWrUSABIZGWn1+V9r0qRJAkDCwsIkNTXVtHz37t2m737Pnj3N9guff/65AJBZs2aZjdWnTx8BIEuWLDFbvmTJEgEgffv2NVtufO2u3R8988wzps/E6dOnTcu//vprASCDBg2SnJwc0/LS0lJ5+OGHBYCsXr3atNz42Zo7d+51X4O2bdumA+gi1fNd6dymTZv07du3S9euXTN9fX1P63S60QB01rb39va+dP78+XJjLW9/Wd5zNX7ehw0bZnZM2LZtmwCQAQMGmG1f2WOH8fNT9n0qT2U/5x988IEAkFdeecVirLvuuksAyIkTJ2wau/G7u3TpUotxynuNT506ZbFtVlaWhIeHi5+fn9ln9fTp0wJAJk2aZHGf8h4/MzNT/P39xd3d3ezYVVpaKuPHjxcA8vzzz5uNY/z+zJgxwyzX+PDDDwWAPPTQQ1Yf3ygiIiJd8wNNZW4VTSxbtWpllpSJiOTm5kqdOnXEYDCYPkD79+83HXisefzxxwWAHDx4UET+Sizff//9672uInJziaVer5czZ85YrBsyZIgAkG+//faG41y6dEkAyOTJk63GtW7dukrFdaPx4uPjzZYXFxeLq6urALDYEZw5c8biC1LZ90Lk+onlrFmzyn2tSkpKpE6dOtK1a1fTMuNnp3PnzlbHMyaWv/32m9X15WFiafEdbt6wYcMrxtfnRglmRRLL8t6zDh06iJeXl+Tl5VmsO3DggACQJ554wrSsoomltT8urmffvn0CQObNm2e2/HqJ5X333WcxjnHdm2++aVr2+uuvCwD55JNPrD52ly5dJDg42PRv44Hytddes9j2k08+qVJiuWzZMot1zZo1EwCSkJBgtty4Xyj7GMb9Qdu2bS3226WlpdKmTRsBYNonJicnCwDp0KGDxeNmZWWJv7+/xUF/6NChZmOUlZGRIUopGTVqlGmZvSSWbdu2TTe+DjdKMKsrsUxMTLQ6VmBg4HVfG6Pyjh2VSSwr+znPyMgQd3d3i8/IxYsXxdXVVXr06GHz2KuSWJbntddesziuViWxXLFihQCQ6dOnW2x/5swZcXFxkebNm5stByBeXl6SlZVltryoqEhcXFykS5cu1409IiIi3SlPhUdERFj0Lhn7/rZs2YLffvsN7du3x+7duwEA58+ftzqH17Fjx0z/bd++Pe666y48/fTTeOSRR/DNN99g4MCB6NmzJ1q2bGnT+ENDQ9GoUSOL5b169cKmTZvwyy+/wPhb6SKCpUuXYtmyZTh06BCuXLmC0tK/zpacO3fO6mN069bN6vKqjte5c2ezf+v1etStWxc5OTkWFykYr1pLSUkxLavse3EjxvG2bNli6vUsy9XV1TRmWeW9LmPGjMGaNWvQvXt33HPPPejTpw969eqFunXr3jAWI6XUvbh6Krg2CxQR03mXnj174rvvvvP+7rvvvGfPnv3hgQMH3nFxcfm/4uLi9yo6oLX3LDc3FwcPHkRISAhefvlli/XGnkdrn4Hr8fDwQHh4uNV1hYWFePvttxEbG4tjx44hOzvbmCAAKP+7Y02XLl0sljVs2BDA1dNURsbP+e7du3Hy5EmL++Tn5yMtLQ1paWkIDg7Gr7/+CuDqvuRa1pZVxLXffeDqdzwxMdGi/ce4Xyj73f/5558BAJGRkRb7baUUevfujaNHj+LXX39Fo0aNTM/BuA8sy9vbG506dbLoBdu9eze8vLzw0UcfWX0OBoOh0p+Fa+K8BcD9AGw5x1z90tJSjz/HR58+fbB3716fuLg4nyeffPLTkydP5un1+kdLSkpW2vAxzfj7+5u1LBg1bNgQP/zwg9myqh47KqKyn3M/Pz/ExMRg9erVOHjwoOk7Gxsbi6KiIkycOLHGYr+e1NRUvPzyy/jqq6+QnJyMvLw8s/U3+7jG75a1uWkbNWqE5s2b4/jx48jKyoKPj49pXVhYGLy9vc22d3FxwS233GK2/ymPUyaW5R3sjb1QV65cAQBcvnwZwNX+uU2bNpU7Xk5ODgCgadOm+OGHHzB//nx89dVXpp6cVq1a4fnnn8fdd99do/EDwKxZs/D222+jUaNGGDp0KOrXr2/ql5g/fz4KCgquO9a1qjpe2V5JIxcXF6tXqbm4XP3Ylb2gobLvxY0Yx3vxxRcrtL1Rea/L2LFj4erqikWLFmHJkiV49913TU3Or7/+eoX7Z8k6+atKA1w9FVNh1t6z9PR0iAhSUlJMF+VYU9HPk1HdunXLnZ929OjR2LBhA1q2bGnqZXJ1dUVGRgbeeOONcr871lzve1P2QhPj5/ydd9657ng5OTkIDg427Tus7WPK++zfSHnf/eutK/vdz8zMvO7j16tXD8Bf+73rPYfyxrl8+TKKi4tt+lmoDcq7ytjFxcUs8QKqfuyoiMp+zgFg4sSJWL16NVauXGn64/LTTz+Fq6srxo4dW2Oxl+fy5cu47bbbcObMGURERKBfv37w9/eHXq/HL7/8gnXr1t3041bku3X8+HFkZmaaJZbXe98rcqGbUyaWqampVpdfuHABwF8vmnGn99Zbb+HRRx+t0NgdOnTAl19+iaKiIuzbtw9fffUV3nzzTYwdOxYhISGIiIiosfhTU1PxzjvvoEOHDvjhhx/g6elp2vb8+fPX3YlaOzjezHg3qyrvRUXGu/YLcyPXm9R+5MiRGDlyJDIzM/H999+bGr4HDBiA48ePW1zocS0RqbbKgqNQSjVXSk0D4A4Au3btwv/93/9lHzx4MDM3N/f//ryYp1KX6Fp7z4zvf9euXcudgaAqyvt87N27Fxs2bMCAAQOwadMmlP0FlN27d+ONN96wWQxlGZ/nwYMHK1TJL7vvCA0NNVtn3L/UNONzKO/xjcuN25V9Dtfb/trHUEohLS3tpuO1RkQuAKjcX7E3oJTqrNPpxgPwEBHExcVh9uzZWSdOnLiUnZ39VFUv5qkO1X3sqOznHAAGDRqE4OBgrFq1CgsWLMCpU6ewZ88eDBs2DEFBf/3Krq1i1+muXrxfXGx5LWLZYpDRRx99hDNnzuCFF17As88+a7bu5Zdfxrp16yr0uNdT2e+WrTjldEPfffed2SkoAMjLy8O+fftgMBhMp66NV3tfW9KvCFdXV/To0QPz58/Hm2++CRHBxo0bTeuNB5aqTGORnJxsdYqNb7/9FsBfV5cnJiZCRNCvXz+zL0PZbSvD1uNVRlXeC71eX+7raxzPeArFlnx9fTFw4EC8//77mDx5MlJTU7Fnzx6bP44z27VrFyIiIrIHDx58bs+ePdOzsrIaFxcXf1rZpLI8Pj4+aNOmDY4ePVqhUzfAzX1nT506BQAYMmQIrv1Zver87lT2e2O8YtpaTNX9HS+PcX+2c+dOi/22iFjs94zP4dqr4wEgOzsbv/zyi8Xy7t2749KlSzhx4oTtAq8BIoIdO3bgtttuyxoxYkTSzz//PDUzM7N5SUnJantJKoHqP3ZU5fjg6uqKMWPG4OzZs0hISMCnn34KAJgwYYLZdraK3TjzQtk2DyPjKemyjPuMoUOHWqyz9rhV2T8Z21SsTROUkpKCU6dOoVmzZpUqvlSEUyaWx48ft5in7t///jcuXryIe+65xzTFTLdu3dC9e3d89tln+Pzzzy3GKS0tNc1HCVytSlj7K9mY9RsMBtOywMBAACh37szrKSkpwbPPPmu2k01ISMDmzZvRokUL3HHHHQBgqjh8//33Zqclfv/9dzz99NOVflxbj1cZlX0vgKuvcVpamtX592bMmAEXFxfMnDnTapKekZFh9ctenu3bt1t9HOPnoex7T9f3+++/+1RXQlnWrFmzkJubiwceeMDqac7Tp0+b/dbvzXxnjd+da5Odw4cPY8GCBZUer6KmTJkCHx8fPPvsszh8+LDF+tzcXLM/rsaPHw+9Xo/XX3/dbF+WmZmJF154odrivJ7GjRsjOjraNL1QWR9//DEOHz6MPn36mPrOGzdujN69e+PAgQNYudL8JMBLL71k9Q+JWbNmAQCmTp1qdW7P8+fPV3m6pep07Ngxf3tOKI2q+9hR2c+5kbGX8tNPP8XKlSvh7++PmJiYaom9S5cuUEohNjbW7Fhx4sQJq2csyttnrFq1Cps3b7bYPiAgAEqpSu2fhg0bBj8/PyxdutTsdRMRPPPMMygqKqqWn4h0ylPh/fv3x4wZM7Bp0ya0bt0a+/fvx9atW9GoUSO89NJLZtt+9tlniI6Oxrhx47Bo0SJ07doVHh4eOHPmDH744QdcvHjR9CFZuXIl3n33XURFRaFFixbw9fXFkSNHsHnzZgQHB2Pq1Kmmcfv06YPVq1fj7rvvxuDBg01N/0OGDLlh/B06dEB8fDx69OiBPn364Ny5c4iNjYWrqys++OADU8m9fv36GDVqFL788kvceuut6Nu3Ly5cuICNGzeiT58+SExMrNTrZuvxKqsy7wVw9TX+6aefEBMTg169epnmA+vZsyfat2+Pd999F9OnT0erVq0wePBgNG/eHJmZmUhMTERCQgImT56MxYsXVyi2J554AmfOnEFUVBSaNGkCpRR27dqFH3/8EXfccYdNWiBqidMA7szKyoqvjmSyrIceegi7d+/G8uXL8d1336Ffv34ICQnBhQsXcOzYMezZswerVq0yzQNnnBj92WefxbFjx0y/ZDF9+vQbPla3bt3QrVs3/Pe//8Uff/yBHj164MyZM1i/fj2GDBlidUJpW6hTpw4+++wz3H333ejYsSMGDhyI1q1bIz8/H8nJyUhISMAdd9yBLVu2AABatGiBOXPmYO7cuejQoQPGjBkDFxcXfPnllwgPD8fx48erJc4bee+999CzZ0888MAD2LBhA9q2bYsjR45g/fr1qFOnDt57z/xarnfeeQcRERG47777sHbtWoSFhWHv3r348ccf0atXL4uKz8CBA/HPf/4Tzz//PFq0aIGBAwciNDQUly5dwsmTJ/Htt9/ihRdeQJs2bWryad/IQRHpm5mZGW+PyWRZ1X3sqOzn3KhHjx4ICwvDihUrUFRUhAceeMBizkZbxd6gQQOMHTsWsbGx6Nq1KwYOHIjU1FT873//w8CBAy3msZ04cSIWLlyImTNnIi4uDqGhoThw4AC2bduGkSNHYs2aNWbbe3t747bbbsPOnTsxZcoUhIWFQafTYfz48eX+ipevry8++OAD3HPPPejevTvGjh2LOnXqYPv27fjpp5/QrVs3PPXUUxV6fpUiYvtpEqrrhgpONzR37lxJSEiQXr16iaenp/j7+8u4ceOsTjUhInL58mV57rnnpH379mIwGMTb21vCwsJk/PjxsmbNGtN2u3fvloceekjat28v/v7+YjAYJCwsTGbNmmUxdlFRkcyePVsaN24sLi4u150moCz8Od1HcnKy3H333RIQECAGg0F69+5tMQ+eyNXpNZ544glp0qSJuLu7S1hYmDz//PNSWFhodeqQG02DZMvxrjcdkLWxRCr+XhhjfeCBB6R+/fqi0+msTufw448/yrhx4yQkJERcXV0lODhYunTpIk8//bQcPXrUtN2NpoOIjY2VMWPGSPPmzcXT01P8/PykU6dO8sorr5jN02ftfpxuqOq3ikw3dKMpPD7//HPp16+fBAQEiKurqzRo0ECioqLktddes5h/dtmyZRIeHi7u7u4CwOzze73Ps4hIamqqTJ06VUJCQsTDw0PCw8PlnXfekcTERKvf/+tNN1TZKUuOHTsm06ZNk9DQUHFzc5OAgAAJDw+XWbNmyY8//mix/QcffCBt27YVNzc3adiwoTz55JOSm5tbpemGrE0XU5X9QlJSkkyZMkXq168vLi4uUr9+fZkyZYokJSVZHefgwYMyePBg8fb2Fh8fHxk0aJAcPHjwunF98803EhMTI3Xq1BFXV1epV6+e3H777fL888+b7cPtYbqhyt6qY7qh8j7v1t7fyh47KjPdkFFlP+cif02xBSvTX9k69pycHJk5c6bccsstpumOVq5cWe5r/Msvv0j//v0lICBAfHx8JDIyUrZt21bufuD48eMyePBg8ff3F6WU2fRo19t37Ny5UwYNGiT+/v7i5uYmLVu2lH/+859Wj13X2wfcaB8ocnW6IXV1HMeglOoTERHx5a5du/y1jqW6KKUQGRlZ4Z9OIvv2+eef45FHHlmblpY2QutYHFFgYGDyDz/80LhVq1Zah0JkoV27dhlHjhzpKyL7tY7Fx8fn0smTJwOrenU/kS307Nkzwyl7LImIiIio5jGxJCIiIiKbcLTEUqoyFQiRVkpKSiDVfJGKk+N3nuzWn59Ne7mwht8V0lxJSYnDXRWeUdE56RyVI/W80o2lp6ejqKjIcn4TqhC9Xn/F2b/z5LiuXLmiB2A5+7UGXF1dszMyMoJCQkK0DoVqsfT0dOVoFctjycnJbunp6VrHQVQhmzdvzsrKytqpdRyOKi8vL2H79u0sw5DdOXv2LDIzMwXAGa1jAYDS0tLvd+zYwcoEaeby5ctITk52c6jEUkTyPDw8tl87HxSRPbpy5Qri4uJcAWzQOhZHlZOT8+nSpUtzr/1dYiKtxcbGlrq4uKy2l1aXK1euLPvoo4+yeNaLtLJmzRoYDIZvHCqxBID09PR/Pf7449nWfs6LyF5kZ2ejb9++Oa6urktEJFPreBzY3kuXLv0yadKkPCaXZC+2bt2Kf/3rX9mZmZmvaR1LGTsSExMTZ82alc/kkmrarl278Pjjj2enp6c/71DzWBoppXobDIZN0dHRMnHiRJ+WLVvC09MTSimtQ6NarKioCKmpqVi/fn3Bp59+WlxYWLgiKyvrEXHEL5kdUUoZfH19twYGBnaaMmWKoW/fvi4BAQEWv8lNVF1EBDk5OTh69CiWL1+e+f3330tubm5fEdmndWxlKaV8fH19E+rVq9d8ypQpXpGRkXp/f3/Tr7UR2YqIIDc3F7/99hs++eSTrLi4OJWXlzdERHY6ZGIJAEopXwAxgYGBE5VSTUpLS/ljzaS1Er1efzkzM3N9YWFhrIj8pnVAzkJd/avxNi8vr3sNBkNUSUmJLxxvVgtyYDqdLldETl6+fHkFgM0iYvkD9HZAKaUDcLuPj88ENze3iNLSUl8ArLqQzel0ujwRSbp8+fInADYYz845bGJJRERERPaFf/ETERERkU0wsSQiIiIim2BiSUREREQ2wcSSiIiIiGyCiSURERER2QQTSyIiIiKyCSaWRERERGQTTCyJiIiIyCaYWBIRERGRTTCxJCIiIiKbYGJJRERERDbBxJKIiIiIbIKJJRERERHZBBNLIiIiIrIJJpZEREREZBNMLImIiIjIJphYEhEREZFNMLEkIiIiIptgYklERERENsHEkoiIiIhsgoklEREREdkEE0siIiIisgkmlkRERERkE0wsiYiIiMgmmFgSERERkU0wsSQiIiIim2BiSUREREQ2wcSSiIiIiGyCiSURERER2QQTSyIiIiKyCSaWRERERGQTTCyJiIiIyCaYWBIRERGRTTCxJCIiIiKbYGJJRERERDbBxJKIiIiIbIKJJRERERHZBBNLIiIiIrIJJpZEREREZBNMLImIiIjIJphYEhEREZFNMLEkIiIiIptgYklERERENsHEkoiIiIhsgoklEREREdkEE0siIiIisgkmlkRERERkE0wsiYiIiMgmmFgSERERkU0wsSQiIiIim2BiSUREREQ2wcSSiIiIiGyCiSURERER2QQTSyIiIiKyCSaWRERERGQTTCyJiIiIyCaYWBIRERGRTTCxJCIiIiKbYGJJRERERDbBxJKIiIiIbIKJJRERERHZBBNLIiIiIrIJJpZEREREZBNMLImIiIjIJphYEhEREZFNMLEkIiIiIptgYklERERENsHEkoiIiIhsgoklEREREdkEE0siIiIisgkmlkRERERkE0wsiYiIiMgmmFgSERERkU0wsSQiIiIim2BiSUREREQ2wcSSiIiIiGyCiSURERER2QQTSyIiIiKyCSaWRERERGQTTCyJiIiIyCaYWBIRERGRTTCxJCIiIiKbYGJJRERERDbBxJKIiIiIbIKJJRERERHZBBNLIiIiIrIJJpZEREREZBNMLImIiIjIJphYEhEREZFNMLEkIiIiIpv4f2Od3zM8ALtJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 700x210 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mglearn.plots.plot_grid_search_overview()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's look at an example of tuning `max_depth` of the `DecisionTreeClassifier` on the Spotify dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>acousticness</th>\n",
       "      <th>danceability</th>\n",
       "      <th>duration_ms</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>key</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>mode</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>time_signature</th>\n",
       "      <th>valence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0102</td>\n",
       "      <td>0.833</td>\n",
       "      <td>204600</td>\n",
       "      <td>0.434</td>\n",
       "      <td>0.021900</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1650</td>\n",
       "      <td>-8.795</td>\n",
       "      <td>1</td>\n",
       "      <td>0.4310</td>\n",
       "      <td>150.062</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.1990</td>\n",
       "      <td>0.743</td>\n",
       "      <td>326933</td>\n",
       "      <td>0.359</td>\n",
       "      <td>0.006110</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1370</td>\n",
       "      <td>-10.401</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0794</td>\n",
       "      <td>160.083</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0344</td>\n",
       "      <td>0.838</td>\n",
       "      <td>185707</td>\n",
       "      <td>0.412</td>\n",
       "      <td>0.000234</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1590</td>\n",
       "      <td>-7.148</td>\n",
       "      <td>1</td>\n",
       "      <td>0.2890</td>\n",
       "      <td>75.044</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.6040</td>\n",
       "      <td>0.494</td>\n",
       "      <td>199413</td>\n",
       "      <td>0.338</td>\n",
       "      <td>0.510000</td>\n",
       "      <td>5</td>\n",
       "      <td>0.0922</td>\n",
       "      <td>-15.236</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0261</td>\n",
       "      <td>86.468</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.1800</td>\n",
       "      <td>0.678</td>\n",
       "      <td>392893</td>\n",
       "      <td>0.561</td>\n",
       "      <td>0.512000</td>\n",
       "      <td>5</td>\n",
       "      <td>0.4390</td>\n",
       "      <td>-11.648</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0694</td>\n",
       "      <td>174.004</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.904</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   acousticness  danceability  duration_ms  energy  instrumentalness  key  \\\n",
       "0        0.0102         0.833       204600   0.434          0.021900    2   \n",
       "1        0.1990         0.743       326933   0.359          0.006110    1   \n",
       "2        0.0344         0.838       185707   0.412          0.000234    2   \n",
       "3        0.6040         0.494       199413   0.338          0.510000    5   \n",
       "4        0.1800         0.678       392893   0.561          0.512000    5   \n",
       "\n",
       "   liveness  loudness  mode  speechiness    tempo  time_signature  valence  \n",
       "0    0.1650    -8.795     1       0.4310  150.062             4.0    0.286  \n",
       "1    0.1370   -10.401     1       0.0794  160.083             4.0    0.588  \n",
       "2    0.1590    -7.148     1       0.2890   75.044             4.0    0.173  \n",
       "3    0.0922   -15.236     1       0.0261   86.468             4.0    0.230  \n",
       "4    0.4390   -11.648     0       0.0694  174.004             4.0    0.904  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spotify_df = pd.read_csv(\"data/spotify.csv\", index_col=0)\n",
    "X_spotify = spotify_df.drop(columns=[\"target\", \"song_title\", \"artist\"])\n",
    "y_spotify = spotify_df[\"target\"]\n",
    "X_spotify.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X_spotify, y_spotify, test_size=0.2, random_state=123\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "best_score = 0\n",
    "\n",
    "param_grid = {\"max_depth\": np.arange(1, 20, 2)}\n",
    "\n",
    "results_dict = {\"max_depth\": [], \"mean_cv_score\": []}\n",
    "\n",
    "for depth in param_grid[\"max_depth\"]:\n",
    "    dt = DecisionTreeClassifier(max_depth=depth)\n",
    "    scores = cross_val_score(dt, X_train, y_train)  # perform cross-validation\n",
    "    mean_score = np.mean(scores)  # compute mean cross-validation accuracy\n",
    "    if mean_score > best_score:  # if we got a better score, store the score and parameters\n",
    "        best_score = mean_score\n",
    "        best_params = {\"max_depth\": depth}\n",
    "    results_dict[\"max_depth\"].append(depth)\n",
    "    results_dict[\"mean_cv_score\"].append(mean_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': 5}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.71792204295906"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's try SVM RBF and tuning `C` and `gamma` on the same dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('standardscaler', StandardScaler()), ('svc', SVC())])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_svm = make_pipeline(StandardScaler(), SVC())  # We need scaling for SVM RBF\n",
    "pipe_svm.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's try cross-validation with **default hyperparameters** of SVC. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.199797</td>\n",
       "      <td>0.06765</td>\n",
       "      <td>0.738998</td>\n",
       "      <td>0.814011</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      fit_time  score_time  test_score  train_score\n",
       "mean  0.199797     0.06765    0.738998     0.814011"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = cross_validate(pipe_svm, X_train, y_train, return_train_score=True)\n",
    "pd.DataFrame(scores).mean().rename('mean').to_frame().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Now let's try **exhaustive hyperparameter search** using for loops. \n",
    "\n",
    "This is what we have been doing for this:\n",
    "\n",
    "```python\n",
    "for gamma in [0.01, 1, 10, 100]: # for some values of gamma\n",
    "    for C in [0.01, 1, 10, 100]: # for some values of C\n",
    "        for fold in folds:\n",
    "            fit within training portion using the given combination (gamma, C)\n",
    "            score on validation portion\n",
    "        compute average score\n",
    "\n",
    "pick hyperparameter values (gamma, C), yielding best average score\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "param_grid = {\n",
    "    \"C\": [0.001, 0.01, 0.1, 1, 10, 100],\n",
    "    \"gamma\": [0.001, 0.01, 0.1, 1, 10, 100],\n",
    "}\n",
    "\n",
    "results_dict = {\"C\": [], \"gamma\": [], \"mean_cv_score\": []}\n",
    "\n",
    "for gamma in param_grid[\"gamma\"]:\n",
    "    for C in param_grid[\"C\"]:  # for each combination of parameters, train an SVC\n",
    "        pipe_svm = make_pipeline(StandardScaler(), SVC(gamma=gamma, C=C))\n",
    "        scores = cross_val_score(pipe_svm, X_train, y_train)  # perform cross-validation\n",
    "        mean_score = np.mean(scores)  # compute mean cross-validation accuracy\n",
    "        # we can find the best_score here, or later using results_dict\n",
    "        results_dict[\"C\"].append(C)\n",
    "        results_dict[\"gamma\"].append(gamma)\n",
    "        results_dict[\"mean_cv_score\"].append(mean_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>C</th>\n",
       "      <th>gamma</th>\n",
       "      <th>mean_cv_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.743961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>100.0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.732792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>10.0</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.729091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>10.0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.720391</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>100.0</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.711715</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        C  gamma  mean_cv_score\n",
       "15    1.0   0.10       0.743961\n",
       "11  100.0   0.01       0.732792\n",
       "16   10.0   0.10       0.729091\n",
       "10   10.0   0.01       0.720391\n",
       "17  100.0   0.10       0.711715"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df = pd.DataFrame(results_dict)\n",
    "results_df.sort_values(by=\"mean_cv_score\", ascending=False).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7439609253312309"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_score_index = results_df['mean_cv_score'].argmax()\n",
    "best_score = results_df.loc[best_score_index, 'mean_cv_score']\n",
    "best_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 1.0, 'gamma': 0.1}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_parameters = results_df.loc[best_score_index, ['C', 'gamma']].to_dict()\n",
    "best_parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARkAAAEGCAYAAABRkOFZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAv6klEQVR4nO3dd3hUZdrH8e8zkw4kIT0kVAnRwEZKCNhBAQFRXBuir7uyrIgF3RUVF1ZFWRbdpqu4iwXELupKU6oUBaQTlK6BkDCESW+kZ+Z5/5iYMEmASSbHhHh/rosrc865n5lfMuSe55yTOaO01gghhFFMLR1ACNG2SZMRQhhKmowQwlDSZIQQhpImI4QwlEdLB/g5eClv7UO7lo4hRJtVRF621jq0oW2/iCbjQzsGqetaOoYQbdZX+rPUs22T3SUhhKGkyQghDCVNRghhKGkyQghDSZMRQhhKmowQwlDSZIQQhpImI4QwlDQZIYShpMkIIQwlTUYIYShpMkIIQ/0i3iDpioTr+/LgyxMwmU2snL+ORS8ucdoef00czy+ZhjUlE4DNi7fz/qzPAJg6/wEG3TCA/MwCJsVPlTySp8XztKZMLd5klFILgDFApta6T/W6IGAR0A04Dtyhtc6r3vYnYCJgAx7RWq92N4PJZGLK3IlMGzGLbEsuc3fMYeuyXaQdsjjV7dt0iKdveqHe+DULN7J07iqefOdhd6NIHsnT5jK1ht2lhcDIOuueAtZprWOAddXLKKXigDuB3tVj/qOUMrsbIDaxJ+nJVqwpmVRVVrFx0RYuH5vg8vh9mw5RlHva3RiSR/K0yUwt3mS01t8AuXVWjwXeqb79DnDzGes/1lqXa61TgGQg0d0MIVFBZFlyapazLbmERAXXq4u7rBfzkv7O7C+n0zUu2t2HlTyS5xeRqcV3l84iXGt9CkBrfUopFVa9PgrYdkadpXpdPUqpScAkAB/8zvlgStVfV/fzqJL3pHB3twcpKy4jcVQ/nlv8JPfGPuLSN9NYkkfytKVMLT6TaaQGfnQ0+Ol0Wus3tNYJWusET7zPeadZllxCo2u7fEh0EDnpzpOrkqJSyorLANixMgmzpxn/4A6NjO8aySN52lKm1tpkMpRSkQDVXzOr11uAzmfURQPp7j7YkZ3JRMVEEtEtDA9PD4aMu4Kty3Y51XQMD6y5HTuwJyaTicKcIncfWvJInjafqbXuLi0Dfgu8UP116RnrP1RK/QvoBMQAO9x9MLvNztwp85mzagYms4nVb28g9aCFMfcPB+CL19dy9W2DGTN5BLYqGxWlFcwe/1LN+OkfPEr8kN4EhHTgw7R5vDvzE1YtWC95JE+L5GltmVRLfxa2UuojYAgQAmQAzwJLgE+ALkAacLvWOre6fgbwO6AK+IPWeuX5HsNfBWm5kLgQxvlKf7Zba93g6asWn8lorcefZVODXUFrPRuYbVwiIURzaq3HZIQQbYQ0GSGEoaTJCCEMJU1GCGEoaTJCCENJkxFCGEqajBDCUNJkhBCGkiYjhDCUNBkhhKGkyQghDCVNRghhKGkyQghDtfi7sIWoy+Tl1dIRnKhWlqdVOse1rmQmI4QwlDQZIYShpMkIIQwlTUYIYShpMkIIQ0mTEUIYSpqMEMJQ0mSEEIaSJiOEMJQ0GSGEoaTJCCEMJU1GCGEoaTJCCEPJu7CrJVzflwdfnoDJbGLl/HUsenGJ0/b4a+J4fsk0rCmZAGxevJ33Z30GwNT5DzDohgHkZxYwKX6q5Pk58gyPZ/I/78FsNrHy7Y188o/lznmuvoSZnz6G9XgWAFuW7uSDvy4G4LHX72PQqH7kZxVy/4CnmiXPgGF9eODFuzCZTax65xs+eWmFc54rY3n2o0ewpmY78izfzYcvLiMkKognXv89HcMD0HbNioVfs/S/a9tUplbbZJRSC4AxQKbWuk/1uiBgEdANOA7cobXOc/exTCYTU+ZOZNqIWWRbcpm7Yw5bl+0i7ZDFqW7fpkM8fdML9cavWbiRpXNX8eQ7D7sbRfK4lEfx0L/v5U83zCHbksurW2ax7Ys9pB0+6VS3f8sRnrnlH/XzvLeJZf9dyxPzJzdfnn/ew/Sx/yD7ZC6vbHyGbSv2knYk3TnP1h949o5/O62zV9l4c8Yikr9Lxbe9D69+8yxJ6w/UG3shZ2rNu0sLgZF11j0FrNNaxwDrqpfdFpvYk/RkK9aUTKoqq9i4aAuXj01wefy+TYcoyj3dHFEkjyt5Bl5E+tEMrClZVFXa2PjpNi67cYDL4/dvPkxRXjPmSejBqWOZWI878nz9vx1cdkM/l8bmZhSQ/F0qAKWnyzhx5BTBnQLbVKZW22S01t8AuXVWjwXeqb79DnBzczxWSFQQWZacmuVsSy4hUcH16uIu68W8pL8z+8vpdI2Lbo6HljxNENypTp6TuYR06liv7pJBPfnvjr/yl6VP0vWSKOPyRHYky1L7XzU7PZfghvIk9uQ/W55j1v/+SNeLO9XbHt4lmIviu3Bk17E2lanV7i6dRbjW+hSA1vqUUirsbIVKqUnAJAAf/M55p0rVX6e1dlpO3pPC3d0epKy4jMRR/Xhu8ZPcG/tIo78BV0ieZsiTdJx7ej1KWXE5A6+/lGc/fYzf9Wme40FNyvNdKr/p/bgjz4h4nvnoESb2q52I+7Tz5s/vPczrT31ESVFZm8rUamcy7tJav6G1TtBaJ3jifc7aLEsuodG1r8wh0UHkpDtPokqKSikrdvygd6xMwuxpxj+4Q/MHlzznlX2yTp6oIHJO5TeQpxyAnau/q87T3pg86XmERgfV5ukURG69PGW1edZ8j4eHGf8gRx6zh5mn33+YDZ9sZcvy3W0u04XWZDKUUpEA1V8zm+NOj+xMJiomkohuYXh4ejBk3BVsXbbLqaZjeGDN7diBPTGZTBTmnOPCppLHuDy7jhHVM4LwbqF4eJoZcvtgtn3h/IvQMTygNk9CD0wmRWFO8x2HccqzO4VOPcII7xqCh6eZa25NZNuKJOc8Yf41t3sN6I4yKQqrj1P98bUJpB1J5/PX1rTJTBfa7tIy4LfAC9VflzbHndptduZOmc+cVTMwmU2sfnsDqQctjLl/OABfvL6Wq28bzJjJI7BV2agorWD2+Jdqxk//4FHih/QmIKQDH6bN492Zn7BqwXrJY2Ce1/6wkL8un4bJbGLNO1+TeugkN/z+OgC+fGsdV/06kTGThmGrslFeWsmce+bWjH/q3YeIv+oSAkI68H7yq7z3l89YvfBrt/L854kPmL14qiPPe5tIPZzO6N8NAWDFgo1cefNAxkwc6shTVsmcCfMA6D04hmHjryBl/wle2/wcAAuf/x8713zf5DytLZOqu5/WWiilPgKGACFABvAssAT4BOgCpAG3a63rHhyux18F6UHqOsOyiuYln1Zw4VldtHC31rrBU46tdiajtR5/lk3SLYS4gFxox2SEEBcYaTJCCENJkxFCGEqajBDCUNJkhBCGkiYjhDCUNBkhhKGkyQghDCVNRghhKGkyQghDSZMRQhhKmowQwlCt9g2S4per+AvjLpXZFJ4mW0tHaP3O8bZlmckIIQwlTUYIYShpMkIIQ0mTEUIYSpqMEMJQ0mSEEIaSJiOEMJQ0GSGEoaTJCCEMJU1GCGEoaTJCCENJkxFCGEqajBDCUPIu7GoJ1/flwZcnYDKbWDl/HYteXOK0Pf6aOJ5fMg1rSiYAmxdv5/1ZnwEwdf4DDLphAPmZBUyKnyp5foY8g4J78WivsZiU4ouTO3g/dWO9mn4de/BIr5vwUCbyK0uYsnsenf1Cef5Xd9fUdPIN4q2ja/j0xGa38gwMiuXhmJsxKxNfntrOR6nr69VcGngRD8eMxUOZKags5g9J/6GzXyjP9L6npibSN5i3j63if5ZNbuVpTZkuyCajlFoAjAEytdZ93L0/k8nElLkTmTZiFtmWXObumMPWZbtIO2Rxqtu36RBP3/RCvfFrFm5k6dxVPPnOw+5GkTyu5EHxWOyv+WPSm2SWFfBW4hQ2Zx/keHFmTU17Dx8ei/01jyfNJ6M8n0DPdgCcKMliwvaXa+5n8VV/5pus/W7neTT2Fp5Iep2s8gLmJfyBb7MOkFqSUVPTzsOHP8TewrS9b5JZnk+gZ/uaPPft/FfN/Xx6xTNsznYvT2vLdKHuLi0ERjbXncUm9iQ92Yo1JZOqyio2LtrC5WMTXB6/b9MhinJPN1ccyXMelwR0xlKaTXppLlXaxlcZ33FlaG+nmuER/fgmaz8Z5fkA5FcW17ufAUE9OVmaQ0ZZvlt5LvbvQnpJDqfKHHnWZyZxRZ08w8L7sylrH5k1eer/PPoHxZBemkNGWZ5beVpbpguyyWitvwFym+v+QqKCyLLk1CxnW3IJiQquVxd3WS/mJf2d2V9Op2tcdHM9vORppFDvADLLCmqWs8oKCPX2d6rp7BdCBw9fXh1wP/MTH2FkZP969zMsoi9fWfe6nSfEO6DmFxUgq7yAEO8Ap5pov1A6ePjxUr8HeD3hD4yIGFDvfq4N68e6jCS387S2TBfk7pIrlFKTgEkAPvidp7b+Oq2103LynhTu7vYgZcVlJI7qx3OLn+Te2EeaLa/kaUSeBtbpOstmZSLWP4pHd7+Bt9mTeQMf5kBBGidKsgHwUGauCIljXvJKY/LUCWRWJnp1iGZq0jy8zJ68NmAKBwtSsZTW5rk8pDdvHv3S7TytLdMFOZNxhdb6Da11gtY6wRPvc9ZmWXIJja59ZQ6JDiIn3XmiVFJUSllxGQA7ViZh9jTjH9yh+YNLnvPKLC8gzKf2VTnUJ4Ds8kLnzGUFbM/5gTJ7JQWVJXyXd4ye7SNrtg8OieWHopPkVbi/G5dVXkCYd2BtHu8AcioK6tXsyD1Mmb2Cwspivs8/xkXtO9VsHxR8MT+ctpDXwC7LhZ6pzTaZxjiyM5momEgiuoXh4enBkHFXsHXZLqeajuGBNbdjB/bEZDJRmFMkeVogz+FCC519Q4j06YiHMjMs/FK2ZB10qtmUdZD4wG6YlQlvkydxAV2cDgwPC2+eXSWAw0UniPILIcInCA9l5tqwfnybfcCpZkvWfuIDemCqznOJfxdSS2rzXBvej/XNtKvU2jK5tLuklBoMvApcAngBZqBYa+1/zoEXCLvNztwp85mzagYms4nVb28g9aCFMfcPB+CL19dy9W2DGTN5BLYqGxWlFcwe/1LN+OkfPEr8kN4EhHTgw7R5vDvzE1YtqH+6UPI0Tx6btvOvI0v5V7/fY1ImvkzfSUpxBmOjBgOw9OQ2Uksy2Z7zAwsH/RGNZvnJHaQUO86seJs8GRgUw98Pfd7kDGeyazuv/PA5f+s7CZNSrEzfwfHiDG7sdBkAy9O3klaSyY7cw8xPnIrWmi/Tt3O82FqTZ0BQL/51+LNmydPaMqm6+9YNFim1C7gT+BRIAH4D9NRaz3A7QRMopT4ChgAhQAbwrNZ6/tnq/VWQHqTOcTl10aqUrune0hGcyKcVnN/G6/61W2vd4ClHlw/8aq2TlVJmrbUNeFsp9W2zJWwkrfX4lnpsIUTjuNpkSpRSXsBepdTfgFNAO+NiCSHaClcP/N6D4zjMw0Ax0Bm41ahQQoi2w6WZjNY6tfpmKfCccXGEEG2NSzMZpdQYpVSSUipXKVWolCpSShWef6QQ4pfO1WMyLwO3APu0K6ejhBCimqvHZE4A+6XBCCEay9WZzJPACqXU10D5Tyu11v8yJJUQos1wtcnMBk4DPjj+4lcIIVziapMJ0lqPMDSJEKJNcvWYzFdKKWkyQohGc7XJPASsUkqVyilsIURjuPrHeMZcGEQI0ea5/AZJpVQ80O3MMVrr5nmvvBCizXL1ejILgHjgAGCvXq0BaTJtwPUHWtee72Mdl7R0BCeHG7gIeUvbW27cNZSbYuM5trk6kxmstY5rhixCiF8YVw/8blVKSZMRQjSaqzOZd3A0GiuOv/hVgNZaxxuWTAjRJrjaZBbguKbMPmqPyQghxHm52mTStNbLDE0ihGiTXG0yh5VSHwLLcX6DpJxdEkKck6tNxhdHcznzrQVyClsIcV6u/sXvBKODCCHaJlf/GM8HmAj0xnG5BwC01r8zKJcQoo1w9e9k3gMigOuBr4FowJjPIBVCtCmuNpmeWuuncXw07TvADcCvjIslhGgrXG0yldVf85VSfYAAHG+WFEKIc3L17NIbSqmOwJ+BZUB74GnDUrWAhOv78uDLEzCZTaycv45FLy5x2h5/TRzPL5mGNSUTgM2Lt/P+LMeHkU+d/wCDbhhAfmYBk+Kntsk8hzdls2zOYew2TeJt0Vx7n/PnVW+cn8KeLxwf1m632ck8VszMzUPxC/Q879imWLW+mD8+k4XNBhPv8mfalCCn7f/4Tx4ffu5442dVFRz6sYKM/T0I6mgGwGbTJI48QacIM8vfi3I7z6aNZbwwswCbDW6904/7HnK+OsqCeaf5YkmJ47Gr4FhyFZv2RuDro/jN7dlUVGhsVTBitA8PT/V3Ow/Ad9/k895f0rDbNEPuCOWm+zs5bf/izVNsWZYDgN2mOXm0lHnb+9M+0IM3njpG0oZ8/IM9eXGFezstrjaZAOCnM0yvVX+tUkr11VrvdStBHUqpkcC/cXxi5Vta6xfqbL8YeBvoD8zQWv/D3cc0mUxMmTuRaSNmkW3JZe6OOWxdtou0Qxanun2bDvH0TS/UG79m4UaWzl3Fk+887G6UVpnHbtMs/sshJr01gIBwH14Zt43eQ0MJ79m+pmbIxO4MmehoHgc3ZPLNu6n4BXq6NLaxbDbNlOlZrF4URXSkB4NGpXHjiHbExXrX1Dz+YEcef7AjAMvXnObfb+TXNBiAV97M5+IYTwqL3P8DdptNM/vPBbz5QTDhkWbG3ZjF0OE+9OzlWVPzu8nt+d1kx/e8YW0Z784/TWCgCa01Cz4Opl07E5WVmntuzeaqoRVc2t+9S2nbbZqFM1P508JYgiK8ePrWA/S/tiPRMb41NWPui2TMfZEA7FmXx8qFVtoHOlrCVbeEMPyecOY9ccytHOD67tIAYDIQBXQC7gOGAG8qpZ50O0U1pZQZRxMbBcQB4xt4Y2Yu8AjgdnP5SWxiT9KTrVhTMqmqrGLjoi1cPjbB5fH7Nh2iKPd0c8VpdXnS9hUQ0sWP4M5+eHiZ6DsqggPrM89an7TCSr/RkU0a64odSWVc1M2THl098fJSjBvbgWWrz345ho+XFDHu5tqZhSW9khXripl4V4BbOX6yb28lnbt50LmrB15eitE3+rJhTdlZ61csK2X0TY5fdqUU7do5fg2rqjRVVaCU+5mOfn+a8K7ehHXxwcPLxOAbgtm9Lu+s9d9+kcNlY4Jrli9J9Kd9gMuXmzonV5tMMNBfaz1Vaz0VSABCgauBe5sliUMikKy1Pqa1rgA+BsaeWaC1ztRa76T2OJHbQqKCyLLk1CxnW3IJiQquVxd3WS/mJf2d2V9Op2uccdfzaG15CjPKCIyo+csFAiJ8KMgsb7C2otTGkU3Z/Gp4eKPHuuqktYrOUbW/AFGRHpy0VjVYW1JiZ/WGEm69oXbm9MdnsnnhzyGYXP3ffx4ZVhuRnWpnSeGRZjIybA3Wlpba2byxjOGja2cUNpvmlpGZXNUvg8uu9Ca+n/sfCJJrrSQ4snZmFxThRV5GRYO15aU2vt9UQOL1QQ1ud5erP+YuwJkJK4GuWutSznibQTOIwvFBcj+xVK9rNKXUJKXULqXUrsrzRGzolaPu59gl70nh7m4PMrnfEyydu5LnFjfbBK7V52noI/3O9mJ7cGMW3foH4hfo2eixbuU5y50uX1vM5QN9a3aVvlh7mrAQMwMu9Wl4QJMCNZSn4UAb15bTL8GLwMDaXz2zWfH5qjDWbw9n33cV/Hik2V4/62RqeP2e9fn06t+hZlepubnaZD4EtimlnlVKPQtsAT5SSrUDDjZjnoZ+DE361Eqt9Rta6wStdYIn3ueszbLkEhpdO1MIiQ4iJz3XqaakqJSyYscUeMfKJMyeZvyDjbn0cWvLExDhQ761dvpfYC3DP6zhn+neM3aVGjvWVdGRHpw4WTtzOXmqik7hDf+CLFpSxJ03185ivt1RxvI1xfQYmMJdk61s2FzKPQ9Z3coTHmnmVHrtzCXjlI2wsIZ/tVYuL2X0WN8Gt/kHmEgc7M3mje6/bgdFeJJzqvZ+cq0VBIY1PEPa9mUul40xZhYDLjYZrfUsHMdh8oECYLLW+nmtdbHW+u5mzGMBOp+xHA2kN+P9N+jIzmSiYiKJ6BaGh6cHQ8ZdwdZlu5xqOoYH1tyOHdgTk8lEYY4xf4/Y2vJ07uNPdmoJuZYSqirs7F1pJW5oWL260qJKju3Mpfe1oY0e2xgD+/qQnFJBSlolFRWaRUuLuPH6dvXqCgptfLOtlLEja5vMX2eEkLanO8d2dufDeREMvdKX916LcCtPn0s9SUupwpJWRUWFZsXyUoYOrz9TKiq0s3NbOdeOqN2Wm2OjsMBx8LmsTLN1czndL3J/RtHjV+2xHi8n80Q5VRV2tn2Zw4DrAuvVlRRVcWhHIQOGdXT7Mc/G5e9Ga70b2G1YEoedQIxSqjtwErgTuMvgx8RuszN3ynzmrJqByWxi9dsbSD1oYcz9wwH44vW1XH3bYMZMHoGtykZFaQWzx79UM376B48SP6Q3ASEd+DBtHu/O/IRVC9a3mTxmDxM3z7iYN+/bg92uSfx1FBEx7dn6sWPP9rI7Ha8L+7/KpNcVIXj5eZx3rDs8PBSv/DWMUeNPYrPBhDv96R3rzbx38gGY/NtAABavLGb4NX6082umgy/nyDNjVgCT7snBboNfj/OjZ6wni95zHIwed4+jAX61uowrrvbG74w8WZl2pj+Wh90GdjtcP8aXIcPc35UzeyjufbYrL/7uMHYbXHNbKNExfnz1oeOg+7C7HI1+55o8fnVlAD5+Zqfxc/+QzKEdRRTlVfHwlUnc9mg0Q24Prfc4rlB19/VbmlJqNPAyjlPYC7TWs5VSkwG01vOUUhHALsAfxwW0TgNxWuuzXg3bXwXpQeo6w7NfqFrfhcRTWjqCE7mQ+PndHbNjt9a6wVOgxhzpcYPWegWwos66eWfctuLYjRJCXACMnUcKIX7xpMkIIQwlTUYIYShpMkIIQ0mTEUIYSpqMEMJQ0mSEEIaSJiOEMJQ0GSGEoaTJCCEMJU1GCGEoaTJCCENJkxFCGKrVvQtb/Py25PZs6QhOCqr8WjqCk6EdmvPij83jcp+0lo7gMpnJCCEMJU1GCGEoaTJCCENJkxFCGEqajBDCUNJkhBCGkiYjhDCUNBkhhKGkyQghDCVNRghhKGkyQghDSZMRQhhK3iBZLeH6vjz48gRMZhMr569j0YtLnLbHXxPH80umYU1xfGD55sXbeX/WZwBMnf8Ag24YQH5mAZPip7bJPP0C45jY4zZMmPgqYwufn1xbr6a3fwwTu9+G2WSmqPI0f97/MgA3dhrKsPArQGtSS9J59cf3qNRVbuU5ujmD1S/uQ9s1fW/pyhUTezlt3/r2j+xfcQIAe5UmO6WIx74ejW+A13nHNsWOr4t57blM7HYYPS6A8Q8EOW1f9Hou65YWAWCzadKSK/jf7osoK7HzwlQreVk2lAluGB/ArRM6up0H4OsN5Tw3sxC7DcaN9+WBh9o7bX99XjFLF5c6MlVBcnIVu/eG4euruOO2XCoqNDYbjBrtzR+ndmhyjhZpMkqpkcC/ATPwltb6hTrbVfX20UAJcK/Wek/1tgXAGCBTa92nOfKYTCamzJ3ItBGzyLbkMnfHHLYu20XaIYtT3b5Nh3j6phfqjV+zcCNL567iyXcebo44rS8Pikk97mDmgVfJqcjnb5c+yY7cfVhKrTU1fmZf7r9oHM8feI3sijwCPB3/oYO8ArghcgiPJP2FCnslj8dO5MrQBDZkbmtyHrtNs/Kv33H3G1fgH+7L/PEb6TUkgtCL/GtqLpsQw2UTYgD4YeMptr93FN8AL5fGNpbNpnnlmUz+9l4UoRGePDg2lcuGtaNbjHdNzbj7gxh3v6PxfPvVaf63IA//QDOVFZrJM0Lp1ceHktN2Jt+YyoAr/ZzGNjXTM38u5L0POxIRaWbsmByGDfchplftr/z9k9tx/+R2AHy1towFb5UQ2NGE1poPF3WkXTsTlZWa22/JZcjQCvr192pSlp99d0kpZQZeA0YBccB4pVRcnbJRQEz1v0nAf8/YthAY2ZyZYhN7kp5sxZqSSVVlFRsXbeHysQkuj9+36RBFuafbbJ6YDt04VZZFRnkOVdrG5qzdJAbFO9VcHZrAtpy9ZFfkAVBQWfv4ZmXGy+SJCRPeJk9yK/LdypO+P4+gLu3pGN0Os6eJ3iOj+WGD9az1B1aepPeo6CaNdcXh78qI6upJpy5eeHopht7oz7dri89av2F5Edfe6JgZBId50KuPDwB+7U107elFttW9WR7Ad3sr6drNTJeuHnh5KW68yYe1a8rOWr98aRk3jnXkUErRrp2jNVRVQVWVBtX0LC1xTCYRSNZaH9NaVwAfA2Pr1IwF3tUO24BApVQkgNb6GyC3OQOFRAWRZcmpWc625BISFVyvLu6yXsxL+juzv5xO17jo5ozQqvMEeQXWNA+AnIp8gr0DnWo6+YbR3sOPWX0e5R+XTmNIaCIAuRUFLD35FW8k/IUFiX+l2FbGd/mH3cpTlFGKf7hvzXKHcB+KMksbrK0sreLolgwuGd6p0WNdlW2tIjSydoYQGuFBtrWywdqyUjs7vy7mqlH1dz+slkqSD5ZzSV8ft/IAWK12IjuZa5YjIs1YrfYGa0tLNV9vLGfUqNrHtdk0o6/PJqFvJlde5U2/fk2bxUDL7C5FASfOWLYAg1yoiQJOufogSqlJOGZB+HDuiyCpBrq01tppOXlPCnd3e5Cy4jISR/XjucVPcm/sI67GaZRWl6eBdXXzmJWZHu278Oz+V/AyefJC/OP8UHScgsoiEoPimbzrGYptJTwR+3uuCR3I11k7m5xHN7TyLK+0P3xtpXPfIHwDvBo91p1AqqEnEdi6rpjeA3zxDzQ7rS8ttjPzgXQefDqUdh3MDY5tVKQGMzVcu25tGQMGehHYsXbOYTYrVqwOobDAzv335XPkcCWxF3s2KUtLzGQa/D/bhJpz0lq/obVO0FoneHLu/dssSy6h0bUzhZDoIHLSnSdLJUWllBU7pps7ViZh9jTjH9z0g2EXUp6cinxCvGoPRgZ7BZJbUeBcU55HUt5Byu0VFFUVc7AwmW7torg08GIyynMorDqNTdvZlrOX2A493MrjH+5LYUbt7KMoo4wOob4N1h5cVbur1NixrgqJ9CDrVO0uTpa1iuDwhl+/Nywv5NqbnJ+nqkrNzAfSuW6sP1eNbJ7nMDLSxKl0W82y9ZSN8PCGf92XLyvjppsanj35B5gYfJkXX2+saHKWlmgyFqDzGcvRQHoTaprNkZ3JRMVEEtEtDA9PD4aMu4Kty3Y51XQMD6y5HTuwJyaTicKcol9Enh+LUon0DSPMOxgPZebK0AHszN3nVLMj93vi/HtiwoSXyZNe7bthKbWSVZ5Hrw7d8TI5XgXjA2OdDhg3RafegeSmnibPUoyt0s6BVRZ6DYmoV1dWVEnqrmx6DY1s9NjGuDjeh5PHKzl1opLKCs2G5YVcPqxdvbrThTa+317K5cNrz/JorfnHNCtdenpx+++b56wSQPylnhw/buNEWhUVFZrly8oYNrz+i21hoZ3t2yoYfn3ttpwcO4UFjl2rslLN5k3lXNSz6bOrlthd2gnEKKW6AyeBO4G76tQsAx5WSn2MY1eqQGvt8q5SY9ltduZOmc+cVTMwmU2sfnsDqQctjLl/OABfvL6Wq28bzJjJI7BV2agorWD2+Jdqxk//4FHih/QmIKQDH6bN492Zn7Bqwfq2kwc7bx77hGd7P4QJE+syt3Ki9BTXR1wJwGrrZiylGSTlH+TlftPRWrM241vSShxP2dbsJP556VPYtZ1jxRbWWLc0OQuAycPEyOnxfPTAt9htmr43dyW0pz+7P0kBYMAd3QE4sj6dHpeH4eXncd6x7jB7KKY8F8q031iw22HU7f506+XN8g/yAbjx7kAANq85zYCr2uHrV/vavn9XGWsXF9E91otJo1MBmPhEMIOGtq/7MI3i4aF4bpY/v/m/POw2uH2cL71iPfngvRIA7r7HcQhhzaoyrrraG78zMmVm2nj8jwXYbKDtcMONPlw3rOnHiVTdfeufg1JqNPAyjlPYC7TWs5VSkwG01vOqT2HPxXEWqQSYoLXeVT32I2AIEAJkAM9qreef6/H8VZAepK4z6Lu58LXfFNbSEZz08Tds0tokrfFC4j08Cls6gpPuna27tdYNngJtkb+T0VqvAFbUWTfvjNsaeOgsY8cbm04I0ZzkbQVCCENJkxFCGEqajBDCUNJkhBCGkiYjhDCUNBkhhKGkyQghDCVNRghhKGkyQghDSZMRQhhKmowQwlDSZIQQhpImI4QwlHwkiuD0VZktHcHJok/7t3QEJ9+HR7V0hHqGBP/Q0hHqOPuFyGQmI4QwlDQZIYShpMkIIQwlTUYIYShpMkIIQ0mTEUIYSpqMEMJQ0mSEEIaSJiOEMJQ0GSGEoaTJCCEMJU1GCGEoaTJCCEPJu7CrJVzflwdfnoDJbGLl/HUsenGJ0/b4a+J4fsk0rCmOdyxvXryd92d9BsDU+Q8w6IYB5GcWMCl+quT5GfJcFX4RMy69HrNSfJqSxBs/fFuvJjGkKzMuHYGHyUxeeQn/9827AKwfOYXiqgrs2k6VtnPr+vlu5+kXeAn39bgNkzKxNuNb/mdZW6+mT0AME7vfiocyU1h1mhn7/g3ATZ2GMjz8cjSa1JJ0XvnhfSp1lduZDm3KZsmcI9htmsG3RXHdfd2dtq+ff5w9X5wCwG7TZBwr5vnNQ2gX6HnesY1haJNRSo0E/g2Ygbe01i/U2a6qt48GSoB7tdZ7zjVWKXU7MBO4BEjUWu9yN6fJZGLK3IlMGzGLbEsuc3fMYeuyXaQdsjjV7dt0iKdveqHe+DULN7J07iqefOdhd6NIHlfyoHi270gmbP4Aa0kh/7v296w79QNHi7Jrajp4ejOz3ygmbv6QU6WFBHn7Od3Hb755l7yK0mbLc/9Fd/Ds/rnkVOTzj75PsCNnHydKay9/0M7sy+SL7mDmgf+QXZ5HgGd7AIK8AhjT6Roe3jObCnslT8T+jqtCB7A+c7tbmew2zed/Oczkt/oTEO7DS+O203toKBE929fUXDuxG9dO7AbAgQ1ZfP1uKu0CPV0a2xiG7S4ppczAa8AoIA4Yr5SKq1M2Coip/jcJ+K8LY/cDtwDfNFfW2MSepCdbsaZkUlVZxcZFW7h8bILL4/dtOkRR7unmiiN5ziM+qBOpxXmcKM6nUtv50nKAYZ1inWpu7NyHNScPc6q0EIDc8pJme/y6Yjp0w1qWTUZ5DlXaxqasPSQGxzvVXB2awNbs78guzwOgoLL252FWZrxMnpgw4W32IreiwO1MafsKCOniR3BnPzy8TPQbFcH+9Vlnrd+zwkq/0RFNGns+Rh6TSQSStdbHtNYVwMfA2Do1Y4F3tcM2IFApFXmusVrrQ1rrI80ZNCQqiCxLTs1ytiWXkKjgenVxl/ViXtLfmf3ldLrGRTdnBMnTCOG+/lhLCmuWraWFhPt2cKrp1j6YAC8f3rv6Hj6/9vfc3KX2l16jWXDl3Xx+7e8Z172f23mCvQJqmgdATnkewV4BTjWdfMNo7+HHX371KP/s+yRDwxIByK0oYPHJdbw1cBYLB82mpKqUvfmH3c5UkFFOYIR3zXJghDcFmeUN1laU2ji8KZv44eGNHusKI3eXooATZyxbgEEu1ES5OPaclFKTcMyO8MHvPLX112mtnZaT96Rwd7cHKSsuI3FUP55b/CT3xj7SmEgukzznydPAurp5PEwmegdG8ttN7+Nj9mDR0AnszbVw/HQu4zcuJLPsNEHefiy88v84WpTDruy0Zk2k6yyblYmL2nfm6f2v4mXy5G+XTuVI4XEKKosYFPQrJu18lmJbCU9ePJFrQgfyddZON/KArhugwZQOBzZm0b1/IO0CPRs91hVGzmQa/L/gYo0rY89Ja/2G1jpBa53gifc5a7MsuYRG174yh0QHkZOe61RTUlRKWXEZADtWJmH2NOMf7Pzq2Vwkz7lZSwuJ8POvWY7w9SezzHl3zFpSyKaMo5TaKsmrKGVnVhoXBzheqX+qzS0vYW36YeI7dnIrT05FPiHeHWuWg7071tvlyanIZ0/+IcrtFRRVFXOgIJlu7aK4NPBiMspyKKw6jU3b2ZbzHRf7N/0g608CI7zJt9bOPvKt5fiHNfx7kHTGrlJjx7rCyCZjATqfsRwNpLtY48rYZnNkZzJRMZFEdAvDw9ODIeOuYOsy5+PJHcMDa27HDuyJyWSiMKdI8rRAnn156XRrH0S0XyCeysQN0b1Zl+58zdt1p34gIaQLZqXwMXtwaVAUR4uy8TV70s7DCwBfsydXhPfgx8KmH28A+LEolUjfUMK8g/FQZq4K7c+O3O+darbnfE+c/0WYMOFl8qRXh25YSq1kl+cS26E7XibHLCI+IBZLSYZbeQA69/EnK7WEHEspVRV2klZa6TM0tF5daVElR3fm0efasEaPdZWRu0s7gRilVHfgJHAncFedmmXAw0qpj3HsDhVorU8ppbJcGNts7DY7c6fMZ86qGZjMJla/vYHUgxbG3D8cgC9eX8vVtw1mzOQR2KpsVJRWMHv8SzXjp3/wKPFDehMQ0oEP0+bx7sxPWLVgveQxKI9Na57fu4r5V96FWSk+O/4dyUVZ3NndcQHyj1P2cLQom28yjrJ82P3YtebT40n8WJhF53aBvDb4DgDMJhPL0/azKeNok7MA2LHzxtFPmNnnIUwo1mVs40SJlZERVwKwyroZS2kGSXkHeaX/n7BrzdqMb0krcZw+/jYniZf6TsOm7RwrtrDausWtPABmDxO3zIjljfv2YLdrEn/diYiY9nz7seMoxOV3Ol7D932VRewVwXj7mc87tqlU3X3Z5qSUGg28jOM09AKt9Wyl1GQArfW86lPYc4GROE5hT/jplHRDY6vX/xp4FQgF8oG9Wuvrz5XDXwXpQeq6Zv/+hDFSP40/f9HP6JJw92cWza21fVrBY3Frd2utGzzlaOjfyWitVwAr6qybd8ZtDTzk6tjq9YuBxc2bVAhhFHlbgRDCUNJkhBCGkiYjhDCUNBkhhKGkyQghDCVNRghhKGkyQghDSZMRQhhKmowQwlDSZIQQhpImI4QwlDQZIYShDH0XdmtRfemI1Ga4qxAg+7xVP5/WlgdaXybJc27Nlaer1rrBi878IppMc1FK7Trb29lbQmvLA60vk+Q5t58jj+wuCSEMJU1GCGEoaTKN80ZLB6ijteWB1pdJ8pyb4XnkmIwQwlAykxFCGEqajBDCUL/YJqOUGqmUOqKUSlZKPdXAdqWUeqV6+/dKqf7nG6uUul0pdUApZVdKuXVa0M18C5RSmUqp/e5kcCPbxUqprUqpcqXU40ZkcCGjoT+DpmZQSgUppdYqpX6s/trxXPfxc2dQSv2p+nk9opQ656eAuExr/Yv7h+NjVo4CPQAv4Dsgrk7NaGAljk+zHAxsP99Y4BIgFtgIJLREvuptVwP9gf0t9LMLAwYCs4HHW+g5Nuxn4E4G4G/AU9W3nwJebC0ZgLjq59Mb6F79PJvdzfBLnckkAsla62Na6wrgY2BsnZqxwLvaYRsQqJSKPNdYrfUhrfWRFs6H1vobIBdjnDeb1jpTa70TqDQow3kZ/DNwJ8NY4J3q2+8AN7eiDGOBj7XW5VrrFCAZx/Ptll9qk4kCTpyxbKle50qNK2NbMp/RWupx24pwrfUpgOqvYeep/zkzGPLc/lKbjGpgXd1z+WercWWsu9zJZ7SWelxhPEOe219qk7EAnc9YjgbSXaxxZWxL5jNaSz1uW5Hx025t9dfMVpTBkOf2l9pkdgIxSqnuSikv4E5gWZ2aZcBvqs/iDAYKqqeWroxtyXxG+zm+/7ZsGfDb6tu/BZa2ogzLgDuVUt5Kqe5ADLDD7UdrqSPvLf0Px9mZH3AcQZ9RvW4yMLn6tgJeq96+jzPOFjU0tnr9r3G8GpQDGcDqFsr3EXAKx4FXCzDxZ/7ZRVQ/biGQX33b/2d+fg39GTQ1AxAMrAN+rP4a1JoyADOqn9cjwKjmyCBvKxBCGOqXurskhPiZSJMRQhhKmowQwlDSZIQQhpImI4QwlDQZ0aoopSKUUh8rpY4qpQ4qpVYopXq1dC7RdNJkRKuhlFLAYmCj1voirXUcMB0Ib9lkwh0eLR1AiDMMBSq11vN+WqG13ttycURzkJmMaE36ALtbOoRoXtJkhBCGkiYjWpMDwICWDiGalzQZ0ZqsB7yVUvf9tEIpNVApdU0LZhJukjdIilZFKdUJeBnHjKYMOA78QWv9YwvGEm6QJiOEMJTsLgkhDCVNRghhKGkyQghDSZMRQhhKmowQwlDSZIQQhpImI4Qw1P8DEufguw0RB80AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "scores = np.array(results_df.mean_cv_score).reshape(6, 6)\n",
    "\n",
    "mglearn.tools.heatmap(\n",
    "    scores,\n",
    "    xlabel=\"C\",\n",
    "    xticklabels=param_grid[\"C\"],\n",
    "    ylabel=\"gamma\",\n",
    "    yticklabels=param_grid[\"gamma\"],\n",
    "    cmap=\"viridis\",\n",
    ");\n",
    "# plot the mean cross-validation scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Each point in the heat map corresponds to one run of cross-validation, with a particular setting\n",
    "- Colour encodes cross-validation accuracy\n",
    "    - Lighter colour means high accuracy\n",
    "    - Darker colour means low accuracy\n",
    "- SVC is quite sensitive to hyperparameter settings.\n",
    "- Adjusting hyperparameters can change the accuracy from 0.51 to 0.74! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Grid size: 36\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'C': [0.001, 0.01, 0.1, 1, 10, 100], 'gamma': [0.001, 0.01, 0.1, 1, 10, 100]}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Grid size:\", np.prod([len(v) for v in param_grid.values()]))\n",
    "param_grid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We have 6 possible values for `C` and 6 possible values for `gamma`. \n",
    "- In 5-fold cross-validation, for each combination of parameter values, five accuracies are computed.\n",
    "- So to evaluate the accuracy of the SVM using 6 values of `C` and 6 values of `gamma` using five-fold cross-validation, we need to train 6 * 6 * 5 = 180 models!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "__Python Side Note__\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use `best_parameters` as an example dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 1.0, 'gamma': 0.1}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example 1: unpacking a dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'A': 'abc', 'C': 1.0, 'gamma': 0.1, 'D': 'def'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{'A': 'abc', **best_parameters, 'D': 'def'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example 2: unpacking a dictionary used as function arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C is 1.0 and gamma is 0.1\n"
     ]
    }
   ],
   "source": [
    "def test1(C, gamma):\n",
    "    print(f\"C is {C} and gamma is {gamma}\")\n",
    "\n",
    "test1(**best_parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example 3: unpacking using a single asterisk can be used for lists and tuples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['C', 'gamma']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[*best_parameters]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('C', 'gamma')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(*best_parameters,)  # note use of comma ',' here to enforce () be interpreted as a tuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('C', 'gamma', 'Something else', 5)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(*best_parameters, 'Something else', 5)  # last comma is not required anymore (but still optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This single and double asterisks operators can be used in function definitions too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Positional arguments: (11, 123) which also can be unpacked: 11 123\n",
      "Keyword arguments: {'a_kw_param': 456, 'C': 1.0, 'gamma': 0.1, 'another_kw_param': 789}\n"
     ]
    }
   ],
   "source": [
    "def test2(*args, **kwargs):\n",
    "    print(\"Positional arguments:\", args, \"which also can be unpacked:\", *args)\n",
    "    print(\"Keyword arguments:\", kwargs)  # but cannot unpack here as **kwargs; why? Give it a try!\n",
    "\n",
    "test2(11, 123, a_kw_param=456, **best_parameters, another_kw_param=789)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Python Side Note End__\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Once we have optimized hyperparameters, we retrain a model on the full training set with these optimized hyperparameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 1.0, 'gamma': 0.1}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('standardscaler', StandardScaler()), ('svc', SVC(gamma=0.1))])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Retrain a model with optimized hyperparameters on the combined training and validation set\n",
    "pipe_svm = make_pipeline(StandardScaler(), SVC(**best_parameters))\n",
    "pipe_svm.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally evaluate the performance of this model on the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7376237623762376"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_svm.score(X_test, y_test)  # Final evaluation on the test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This process is so common that there are some **standard methods** in `scikit-learn` where we can carry out all of this in a more compact way. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApYAAAGnCAYAAAANPBefAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAArEAAAKxAFmbYLUAABvFUlEQVR4nO3dd3hUZfYH8O87kzbpDYRQQgs9VAU0QBJAqqELiCDNBgq7Nn66upS1IK662MFGUTCuyNIFBZIgKoig0hEICRiEEEhI7+f3B86YYSaQhEnuzOT7eZ55lHvvvHOm3Xty7rnvKBGBkVIqAMDwoKCgCSLSSETcQWQbpXq9Pj07O/ur/Pz8VSJyWOuAiOgvSqmOOp3uuYCAgP6lpaUGADqtY3Jgotfri/Py8o7n5OQsArBcyh5siZyYMn7WlVKDDAbD6kGDBpWOHz/eu3nz5jAYDBqHR86iuLgYaWlp+OqrrwqXLVtWmJeXtzEzM3OCiJRoHRtRbefm5rbCzc1t4siRIzF27Fg0atQI7u6sK1RVSUkJMjIysH37dixduhSXLl26kpmZ2VRE0rWOjai6KRGBUirKz89vY1xcnFfnzp21jomcXH5+PgYNGpTz888/f5mRkTFJ63iIajN3d/d3AwICpu/evRtNmjTROhynU1paikmTJmH9+vVXMjMzA1i5JGenRATBwcG7Fi9eHDF69Git46FaIj8/H3Xq1MnLzs5uJiLntY6HqDZSSik/P7+ibdu26W+99Vatw3FapaWlaN68OZKSkiaLyHKt4yGqTjqlVGBeXl7XIUOGaB0L1SIeHh4YPny4KKVGaR0LUS02zNPTU9+1a1et43BqOp0OU6ZMgaen52Nax0JU3XQAOnbo0CGf/ZRU0wYOHOgZGBjYR+s4iGqxmH79+kEppXUcTq9v377w9PRspXUcRNVNB8A/KCiIexWqcQEBAVBKBWodB1EtFhQcHKx1DLVCQEAASkpKXLSOg6i66QAovV7PxJKua/LkyVBKISkpqULbJyUlQSmFyZMnl7uNXq+HUkpvmwiJqAqUi0vVcp2KfMfpL3q9HgB4rCWnx3nKbIg7WiKiylFKISoqSuswqiQ+Ph5KKcybN0/rUIjsBhNLqpAFCxbg6NGjaNCggdahEBERkZ1ivwdVSP369VG/fn2twyAiIiI7dsOKZdlS/86dOxEZGQlvb28EBgZi/Pjx+P333y3uExcXh6lTp6JVq1bw9vaGt7c3br31Vrz//vtWH8N4KiQlJQWTJ09GvXr1oNPpEB8ff9PjjR8/HsHBwfDx8cGQIUOQmJgIADh+/DhGjBiBwMBA+Pj44O6770ZqaqrV8Q4cOIBx48ahfv36cHNzQ2hoKGbOnIlLly6Ztlm2bBmaNm0KAFi+fDmUUqab8XkAgIjg448/RkREBHx9feHp6Ylbb70VH3/8scXjzps3z3T/5cuXo2vXrvD09KzQaaPi4mIsWLAAzZs3h4eHB1q0aIEFCxYgMTHR6un6Jk2aoEmTJsjIyMCsWbPQqFEjuLi4YNmyZQDK77EsKSnBwoUL0aJFC7PHKS0tvWGMRGT/Kvsdr+j+2nhsAYCEhASzfaZxv3PlyhUsXLgQkZGRCAkJgZubG0JCQnDffffh1KlTFX4OpaWl+PDDD9GtWzcEBgbC09MTTZo0wfDhw7Fz506L7Xfu3ImYmBgEBwfD3d0dYWFheO6555Cbm2vaZt68eYiOjgYAzJ8/3yz+ivaiEzmjClcsd+/ejQULFmDIkCGYNWsW9u/fj88++wy7du3C3r17ccstt5i2XbhwIU6ePIkePXpgxIgRyMjIwJYtW/DQQw/h+PHjeO211yzGv3TpEm6//XYEBgZi7NixKCwshK+vb5XHS09PR8+ePVGvXj1MmjQJv/32GzZu3Ihjx45h/fr16NWrF7p06YKpU6di3759WL16NTIyMvDNN9+YjbN+/XqMGTMGer0eQ4cORaNGjXDkyBG8/fbb2Lp1K/bs2YOAgAB06tQJf/vb3/DGG2+gY8eOGD58uGkM469ZiAgmTJiAVatWoWXLlhg/fjzc3NzwzTffYNq0aThy5AheffVVi+fy73//G3FxcRg6dCjuvPNOVKTZfurUqfjkk0/QvHlzPPLIIygoKMCiRYvwww8/lHufgoIC9OnTB1lZWYiJiYGbm5vZ+2rNgw8+iI8//hhNmzbFI488gvz8fLz++uv4/vvvbxgjEdm/yn7HK7q/btKkCebOnYv58+cjNDTU7I/dTp06AQCOHj2KOXPmIDo6GiNGjICXlxeOHTuGVatWYdOmTdi/fz9CQ0Nv+ByeeeYZvPLKK2jevDnGjx8PHx8fpKSk4Ntvv8WOHTvQu3dv07aLFy/GjBkzEBAQgJiYGNSpUwd79+7Fiy++iLi4OMTFxcHNzQ1RUVFISkrC8uXLERkZafYHv7+/f6VfZyJnMnLo0KFXpBxxcXECQADIhx9+aLZu/vz5AkCmTp1qtjwxMdFinKKiIrnzzjtFr9dLcnKy2Trj+FOmTJHi4mKL+1Z1vMcee8xs+cMPPywAxN/fXxYtWmRaXlpaKoMHDxYAsn//ftPytLQ08fX1lYYNG1o8xqpVqwSAPProo6Zlp0+fFgAyadIki3hFRN5//30BINOmTZOioiLT8oKCAomJiREA8tNPP5mWz507VwCIl5eXHDhwwOqY1mzbtk0AyK233iq5ubmm5X/88YfUq1fPaoyhoaECQPr37292H6NJkyYJADl9+rRpmfGz0bFjR8nOzjYt//333yU4OPi6r4WIyJYtW6ROnTo7RQS88cZbzd8ArHvqqaekPFX5jldlfx0ZGWn18TMyMuTSpUsWy3fs2CE6nU7uv//+cmMvKzAwUBo0aCA5OTlmy0tLS83GP3z4sLi4uEjnzp0tHnfBggUCQF599VXTMuPrM3fu3BvGcOzYMQkICCgWO3jfeeOtOm8VvninVatWmDp1qtmyp556CnXq1MFnn32GwsJC03LjKeGyXFxc8PDDD6OkpARxcXEW693c3PDKK68Yp2QwU5XxvL298fzzz5stGz9+PAAgKCgIs2bNMi1XSmHcuHEAgF9//dW0fMWKFcjMzMSCBQvQuHFjs7HuuecedOnSBbGxsRaPXZ63334bXl5eePvtt82qjm5ubnjxxRcBAJ999pnF/R588EGEh4dX+HE+/fRTAMA///lPlJ34vl69evjb3/523fv++9//RkUny1+xYgUAYM6cOfDy8jItb9CgwQ0fh4jsX1W+41XZX5fHz88PgYGWU91GR0ejXbt22LZtW4XHcnNzszjbo5QyG3/JkiUoLi7Gm2++afG4s2fPNh3viKh8FT4VHhERYfHrDAaDAV27dsWWLVvw22+/oX379gCArKwsvPrqq1i7di1OnTqFnJwcs/udO3fOYvymTZuivIl6qzJeWFiY2Y4QgOnikw4dOlg8F+O6lJQU07Ldu3eb/nvy5EmLx8jPz0daWhrS0tLKjd0oNzcXBw8eREhICF5++WWL9UVFRQCAY8eOWazr1q3bdce+ljE5vuOOOyzWWVtm5OHhUakE1vg4vXr1slhnbRkROZaqfMersr++nvj4eCxatAh79uxBWloaiouLTevc3NwqNMaYMWOwePFitG/fHmPHjkVkZCRuv/12i2OEcZ+/ZcsWq0mrq6ur1X00Ef2lwoll3bp1rS439uBduXIFAFBYWIioqCjs378fnTt3xsSJExEUFAQXFxdTP0pBQUG541yrquMZ+zPNnuyff61eb50xwQOAy5cvAwDeeecdq7EZ5eTk3DCxTE9Ph4ggJSUF8+fPv+5Y17pRn+O1MjMzodPpEBQUVKmx6tatW6mfdrty5Qp0Op3V517ZmInI/lT2O17V/XV5vvjiC4wdOxbe3t4YMGAAmjRpAk9PT9MFPsnJyRUa580330SzZs2wbNkyvPDCC3jhhRfg4eGBMWPG4LXXXjM9P+M+33gGiYgqr8KJZXlXTF+4cAHA1VMWALBu3Trs378f999/Pz744AOzbWNjY7F8+XKr45SX0FR1PFswJqAHDx40VWNvdqyuXbvip59+qtR9K/s7vr6+vigtLcWlS5csDgjG98sWj+Pn54fS0lKkpaWhTp06FX4cInIMlf2O23p/PW/ePHh4eGDfvn0ICwuzGK+iXF1d8dRTT+Gpp57CuXPnkJCQgKVLl2LFihU4f/48tm7dCuCv/XRmZiZ8fHwqFSsRXVXhHsvvvvsOImK2LC8vD/v27YPBYEDLli0BwDQFxNChQy3G+PbbbysdoK3Hq4zu3bsDwHWvpC7L2B9aUlJisc7Hxwdt2rTB0aNHkZGRYbMYrenYsSMAWL1q05ZXaxsfx9r7UN3vDRFVv8p+x6uyv9bpdFb3mcbx2rRpY5FUnjt3rlLTDZUVEhKCe+65B1u2bEFYWBi2bduGvLw8AH/t842nxG/kevt8otqqwonl8ePHLeZa/Pe//42LFy/innvuMfW6GKd+2LVrl9m2CQkJFn/BVoStx6uMKVOmwMfHB88++ywOHz5ssT43N9dsBxQQEACllNW5PQFg1qxZyM3NxQMPPGD1lPfp06dtMv/ZvffeCwB4/vnnkZ+fb1p+/vx5vPHGGzc9vtF9990HAPjXv/5l9nxSUlJs+jhEpI3Kfsersr8ODAwsd58ZGhqKkydPmlVH8/PzMX36dLNey+spKCjAjh07LAojOTk5yMrKgqurqylBnDFjBlxcXDBz5kycPXvWYqyMjAz8/PPPZrEDKDd+otqowqfC+/fvjxkzZmDTpk1o3bo19u/fj61bt6JRo0Z46aWXTNvFxMSgSZMmeOWVV3Do0CG0b98ex48fx8aNGzF8+HB8+eWXlQrQ1uNVhvEKwLvvvhsdO3bEwIED0bp1a+Tn5yM5ORkJCQm44447sGXLFgBXr0S/7bbbsHPnTkyZMgVhYWHQ6XQYP348GjdujIceegi7d+/G8uXL8d1336Ffv34ICQnBhQsXcOzYMezZswerVq0yzXtZVf369cO9996LlStXIjw8HMOGDUNBQQH++9//onv37tiwYQN0upv/Nc+oqChMmTIFS5cuRXh4OEaMGIGCggJ8/vnn6NGjBzZu3HjTj0FE2qnsd7wq++s+ffrgv//9L0aPHo3OnTtDr9djyJAhCA8Px8yZMzFz5kx07twZo0ePRnFxMb755huICDp27Gg2i0d58vLy0LdvXzRr1gzdu3dH48aNkZ2djY0bN+L8+fP4v//7P1NhpH379nj33Xcxffp0tGrVCoMHD0bz5s2RmZmJxMREJCQkYPLkyVi8eDEAoHXr1ggJCUFsbCw8PT3RsGFDKKUwffp0U3sYUW1UoXks586dKwkJCdKrVy/x9PQUf39/GTdunJw5c8biPomJiTJq1CipU6eOeHp6ym233SaxsbHlzvmF68xjZsvxrjfP5PXmIzt27JhMmzZNQkNDxc3NTQICAiQ8PFxmzZolP/74o9m2x48fl8GDB4u/v78opQSAxMXFmW3z+eefS79+/SQgIEBcXV2lQYMGEhUVJa+99ppcvHjRtJ1xHstr718RRUVF8vzzz0vTpk3Fzc1NmjVrJi+99JLs2bNHAMjf/vY3s+1DQ0MlNDS03PGszWMpIlJcXCwLFiyQZs2amT3OyZMnOY8lb7zZ+Q03mMdSpPLf8crur//44w8ZM2aMBAcHi06nEwCydOlSEbk6z+TixYulXbt24uHhIfXq1ZNp06bJhQsXJDIyUgBcN3YRkcLCQlm4cKH0799fGjZsKG5ubnLLLbdIZGSkxMbGWr3Pjz/+KOPGjZOQkBBxdXWV4OBg6dKlizz99NNy9OhRs213794tkZGR4uPjY5pD+dr9pAjnseSt9twqlViS4/vggw8EgLz77rtah8LEkjfeNL5VJLEk22BiyVttud38+VCyS+fPn4eIeU9RSkoKXnjhBej1etx1110aRUZERETOqsI9luRYXn75ZWzatAm9evVC3bp1cebMGWzcuBFZWVmYN28eGjVqpHWIRERE5GRcAEhJSYnccEtyKAMHDsSRI0ewadMmpKenw8PDAx06dMCMGTNMP22ptZKSEogI5+kg0o5U9Opqujl/TknEYy05PRcAGZcuXSr3wx4VFWVxSpXs38CBAzFw4ECtw7iuP3+N6LLWcRDVYpfS0tK0jqFWSE9Ph16vZxZPTk8H4JcDBw54GCeIJaopW7Zsyb18+fIOreMgqsU2bNu2jcWDGrB9+3bk5uYe1zoOouqmE5F0g8Gwb9OmTVrHQrVIfn4+1q5dq0Sk+iYiJaIbWZebm1uyb98+reNwaqWlpVi6dClyc3P/o3UsRNVNBwCXLl167v77788p+4sCRNUlPz8fgwYNytHr9V+IyHmt4yGqrURE8vLy3r/rrrts8qtfZKm0tBSTJk3C5cuXrwBYoXU8RNVNGU+BKKUGGQyG1QMHDiy99957vZs1awaDwQCllMYhkjMoLi5GWloaNm/eXLh8+fLCvLy8jZmZmRN48Q6R9tzc3Fa4ublNHDFiBMaOHYtGjRrBw8ND67AcVklJCdLT07F9+3YsXboUly9fvpKZmdlURNK1jo2ouqmyvTVKqQAAw4KCgiaKSCMRcdcuNHIyJXq9PiM7O3tzfn7+ZyJi+ePrRKQZpVRHpdQ/AgMDB4qIQUQ4z3HViV6vL87NzT3+5+nvFcJGVqolFD/rRERkC0qpNgC6iMhKrWO5WUqpdgA6AIhlUkhUcfyL1MaUUl2UUg9oHQcRUU1RSumVUrMBxAEo0DoeG7kC4G8ANiqlGmgdDJGjYGJpe6UAHtE6CCKimvBnlfI7AJEAuorIao1DsgkR+R1ATwAJAPYppaYoXnRAdEM8FW5jSikdgFQALTn5NxE5K6WUHsATAB4H8DSA5c56yvjP5HkZgDQAD4pIirYREdkvVixtTERKAewC0FvrWIiIqoOVKuUyZ00qAUBEjgKIAKuXRDfExLJ6xAOI0jgGIiKbuqaXcjGAu2pL9U5EikXkFQDRAB4Gey+JrGJiWT3iwcSSiJxIbatSlofVS6LrY49lNWCfJRE5i9rUS1lZ7L0kssSKZTVgnyUROQNWKa+P1UsiS0wsq088eDqciBxQbe6lrCz2XhKZY2JZfeLBxJKIHAyrlFXD6iXRVeyxrCbssyQiR8JeStth7yXVZqxYVhP2WRKRo2CV0rZYvaTajIll9YoHT4cTkZ1iL2X1Ye8l1VZMLKtXPJhYEpEdYpWyZrB6SbUNeyyrEfssicjesJdSO+y9pNqAFctqxD5LIrInrFJqi9VLqg2YWFa/ePB0OBFpiL2U9oO9l+TsmFhWv3gwsSQijbBKaZ9YvSRnxR7LasY+SyLSAnspHQd7L8mZsGJZzdhnSUQ1jVVKx8LqJTkTJpY1Ix48HU5E1Yy9lI6LvZfkLJhY1owEMLEkomrEKqVzYPWSHB17LGvAn71OqQDC2GdJRLbEXkrnxd5LckSsWNYAESkB+yyJyMZYpXRurF6SI2JiWXPiwdPhRGQD7KWsPdh7SY6GiWXNiQcTSyK6SaxS1k6sXpKjYI9lDWGfJRHdDPZSkhF7L8mesWJZQ9hnSURVxSollcXqJdkzJpY1Kx48HU5EFcReSioPey/JXjGxrFnxYGJJRBXAKiVVBKuXZG/YY1mD2GdJRDfCXkqqKvZekj1gxbIGsc+SiK6HVUq6Gaxekj1gYlnz4sHT4URUhlLKhb2UZAvsvSStMbGsefFgYklEf7qmSnkrq5RkC6xeklbYY1nD2GdJRMDVKiWu9lI+BvZSUjVi7yXVJFYsaxj7LImoTJWyN9hLSdWM1UuqSUwstREPng4nqnX+7KX8P1ztpXwP7KWkGsLeS6opTCy1EQ8mlkS1CquUZA9YvaTqxh5LDbDPkqj2YC8l2Sv2XlJ1YMVSA+yzJKodWKUke8bqJVUHJpbaiQdPhxM5JfZSkqNg7yXZGhNL7cSDiSWR02GVkhwRq5dkK+yx1Aj7LImcC3spyVmw95JuBiuWGmGfJZHzYJWSnAmrl3QzmFhqKx48HU7ksNhLSc6KvZdUVUwstRUPJpZEDolVSqoNWL2kymKPpYbYZ0nkeNhLSbUVey+pIlix1BD7LIkcC6uUVJuxekkVwcRSe/Hg6XAiu8ZeSqKr2HtJN8LEUnvxYGJJZLdYpSSyxOollYc9lhpjnyWRfWIvJVHFsPeSymLFUmPssySyP6xSElUcq5dUFhNL+xAPng4n0hx7KYmqhr2XZMTE0j7Eg4klkaZYpSS6eaxeEnss7QD7LIm0w15KourB3svaiRVLO8A+SyJtsEpJVH1YvaydmFjaj3jwdDhRjWAvJVHNYO9l7cPE0n7Eg4klkU0operodLp7ylnHKiVRDato9VIp1U6v1/er8QDJZthjaSfYZ0lkO/7+/p9lZWWNLS0tjRaRBIC9lET2orzeS6WUh4+Pz/GSkpKA3NzcUBFJ1zJOqhpWLO0E+yyJbEMp1UWv1w9dvXq18vb2jlVKebFKSWQ/yqteent7Lxg4cGDwfffd5+rj4/OixmFSFbFiaUeUUo8BCBWRv2sdC5EjUkopPz+/n958880u9913HyZPnpz3xRdfHMjNzW0CVimJ7E6Z6mVRQEBAl5MnTxoAoGnTprmZmZmdReQ3TQOkSmPF0r7Eg32WRFWm0+lGNmjQoOWECRMAAG+++abBy8vrVgAzWKUksj9/Vi/7eHt7t1u2bJkhMDAQgYGBeP755z38/f0Xax0fVR4TS/tyAEAjpVSg1oEQORqllIenp+e777//vrdOd3XX5uvri5UrV+q9vb3fUUp5aRwiEVnh7e39wqBBg9yGDh1qWjZ9+nSdn5/fbUqpOzUMjaqAiaUdYZ8lUdUZDIYn+/Xr5xMREWG2/M4778SoUaP8fH19X9coNCIqh1Kqm6ur64OLFy/2LLvc1dUVixcv9vbx8fngzwvvyEGwx9LOsM+SqPKUUg08PDySVq5c6SIiSEpKwokTJ/KOHz9ecPr0aXX+/HlPpdSF/Pz8UBEp1TpeIrrK09Pz5eLi4sf9/f0LGjduXBwWFubSqlUrz6ZNm+qaNGmCRx99tPT48eNzCgsLeTGPg2BiaWeUUp0BLBWRTlrHQuQolFJLPDw8pnp6eibrdLrfcnJyjubl5Z0AkATgNIAzIpKnbZREZM2f0+3VB9AEQBMXF5dmPj4+bfR6fcuCgoKWxcXFhbm5uUHaRkkVxcTSznA+SyIiInJU7FuwMyJSopQy9lmu1TgcqiKlVBMA/wwKChohIt4iwn7m2kd0Ol1RTk7O/vz8/FdEZL3WARFVN6WUL4ChQUFBEwA0LS0t9dA6JidWotPpLmdnZ28oKCj4zF6mZmLF0g6xz9KxKaX+aTAY/jVo0CCMHz8ezZs3h8Fg0DosqmHFxcVIS0vD5s2bsXz5cuTm5iZnZWWFiUiR1rERVQelVG+DwbCpT58+MmHCBJ9WrVrBYDDAyi83kg0UFRUhNTUVGzZsKPjkk0+KCwsLV2RlZT2i9bRqTCztEPssHZdS6u9+fn7/iYuLQ+fOnbUOh+xEfn4+Bg0ahP3795+6cuVKC63jIbI1pdStPj4+cZs3b/bu2bOn1uHUOtnZ2YiKiso5ceLEh1euXPm7lrEwsbRD7LN0XEFBQVeWLFniO3r0aK1DITuTn5+POnXqIDs7u4OIHNQ6HiJbCgwMXP/KK6/E3H///VqHUmtduXIF9evXz8/Ly7tFRDK1ioN9X3aI81k6JqVUs/z8fN8hQ4ZoHQrZIQ8PDwwfPhxKqee0joXIlpRShvz8/L6jRo3SOpRazc/PD9HR0UUAYrSMg4ml/YoHf97R0Qzr0KGDsJ+SyjNw4EAEBATwPCE5m9ahoaGFAQEBWsdR6w0ePNjHx8dH06IUE0v7FQ8mlo7mlsBA/honlS8gIAA6nc7zxlsSORR/f39/rWMgXN3HuLq6ajrnJxNL+8XfDXc8Sq/Xax2Dybx586CUQnx8/E2NExUV5TBXddrqOVeXPz8fjvFiElWcXe37qtPkyZOhlEJSUpLWoVil1+uN12lohomlnWKfpfOJj4+HUgrz5s3TOhS6jmXLlkEphWXLlmkdCpFT0GrfZ69/aNprXLbCxNK+xYOnw6mKHn30URw9ehTdunW7qXFWrFiBo0eP2igqIiJyZvzlHfsWD2Cp1kGQYwoODkZwcPBNj9O4cWMbRENERLUBK5b2jX2WTmLevHmIjo4GAMyfPx9KKdPN2Ktj7N1JTEzEf/7zH7Rr1w7u7u6YPHkyAODcuXOYO3cuevTogbp168Ld3R1NmjTBjBkzkJqaavUxrz3dkpSUBKUUJk+ejMTERIwePRoBAQHw8vJCv3798Ouvv1qMY63Hsuzp4u3bt6Nnz57w8vJCUFAQJk2ahEuXLll9HZYsWYJ27drBw8MDjRo1wuzZs5Gfnw+lFKKioir8ep49exb33HMPAgMD4e3tjcjISOzcudPqtoWFhXjrrbcwYMAANGrUCO7u7qhbty5GjhyJn3/+2WzbyZMnY8qUKQCAKVOmmL1PRvv27cOjjz6K9u3bw8/PDwaDAeHh4Xj55ZdRVMQf1SEqqyL7PuDq9/T1119Hly5d4OXlBR8fH/Tq1Qvr11v+EuqVK1cwZ84ctG3bFt7e3vDz80Pr1q0xZcoUnD17FsDV/db8+fMBANHR0abHbNKkSYXiPnz4MO666y74+PjAz88PgwcPxqFDh6xue+XKFSxcuBCRkZEICQmBm5sbQkJCcN999+HUqVNm21Ykrri4OEydOhWtWrWCt7c3vL29ceutt+L999+vUOxaY8XSjvF3w51HVFQUkpKSsHz5ckRGRpolUddeTTlz5kzs3r0bQ4YMwV133YVbbrkFALBz50689tpr6Nu3L7p37w5XV1f8/PPPeO+997B161bs378ffn5+FYonKSkJ3bt3R9u2bTF16lScOnUK69atQ3R0NI4ePWp6zBvZsGEDNm7ciJiYGEyfPh07d+7EihUrcOrUKezatcts2zlz5uD5559H/fr18eCDD8LFxQVffPEFjh07VqHHMvrjjz9w++23IyUlBQMGDECXLl1w9OhR3HnnnaYDWFmXL1/G3//+d/Tq1QuDBw9GQEAAEhMTsX79enz11VfYuXMnbrvtNgDA8OHDkZGRgXXr1mHYsGHo1KmTxXgffPABNmzYgN69e2Pw4MHIzc1FfHw8nnnmGezduxdffvllpZ4PkTOryL6voKAAAwcORHx8PDp37oxp06ahqKgImzZtwrBhw/DWW2/h0UcfBQCICAYMGIA9e/YgIiICAwcOhE6nQ1JSEv73v/9h0qRJaNSokekP8oSEBEyaNMmUuFXk6vVDhw4hIiIC2dnZGDlyJMLCwvDjjz8iIiICHTt2tNj+6NGjmDNnDqKjozFixAh4eXnh2LFjWLVqFTZt2oT9+/cjNDQUACoU18KFC3Hy5En06NEDI0aMQEZGBrZs2YKHHnoIx48fx2uvvVbh118TIsKbHd8APAZgkdZx8Fah92phTExMqZQjLi5OAMjcuXOtrp80aZIAkIYNG0pycrLF+gsXLkhWVpbF8uXLlwsAeeGFF8yWz507VwBIXFycadnp06cFgACQl19+2Wz75557TgDIggULzJZHRkbK1V3FX5YuXSoAxMXFRXbt2mVaXlxcLFFRUQJAfvjhB9Py48ePi16vl8aNG0taWpppeVZWlrRr104ASGRkpNXX5VrG1+na57tkyRLTcyv7nPPz8+X333+3GOfQoUPi7e0t/fr1s/rcli5davXxk5KSpLi42GxZaWmpTJ06VQCYvR7X2rJliwQHB2eIHXxeeePNVjcAfSIiItKlHDfa9/3jH/8QADJv3jwpLf1rF5qZmSm33nqruLm5SUpKioiIHDhwQADIiBEjLMbJz88320da2wdWhHGf9+mnn5otf+aZZ0z7mNOnT5uWZ2RkyKVLlyzG2bFjh+h0Orn//vvNlt8orsTERItlRUVFcuedd4per7d6fDCKjY2VoKCg/4mGnweeCrd/8eAFPLXKU089ZbWvsW7duvD29rZYPnHiRPj6+mLbtm0VfoymTZviqaeeMls2bdo0AMDevXsrPM748eMRERFh+rder8ekSZMsxvnss89QUlKCJ554AkFBf02x5u3tjeeeq/gP0RQWFuLzzz9H3bp18cQTT5itu//++9GyZUuL+7i7u6NBgwYWy9u1a4fo6Gjs3LmzUqewQ0NDce3UKkopPPLIIwBQqfeBqLYrLS3Fe++9hxYtWmDOnDlmbSc+Pj6YM2cOCgsLsWbNGrP7WfshCnd3d6v7yMo4c+YMEhIS0KFDB9x7771m6/7xj39YrXj6+fnB2hzG0dHRaNeuXaX3CU2bNrVY5uLigocffhglJSWIi4ur1Hg1jafC7Z+xzzJIRKw3rpFTud5V3GvWrMGSJUuwf/9+pKeno6SkxLTu3LlzFX6Mjh07Qqcz/7uyYcOGAICMjIwKj9OlSxeLZdbGMfZu3nHHHRbbW1tWnuPHjyM/Px99+vSBh4eH2TqdToc77rgDv/32m8X9fvnlF7zyyivYtWsXzp8/b5FIpqWloX79+hWKobCwEG+//TZiY2Nx7NgxZGdnG6s2ACr3PhDVdsePH0d6ejpCQkJMvYdlXbx4EQBMLTNt2rRBeHg4Vq1ahbNnz2L48OHo1asXunTpYvEHX1UY91U9e1r+QJa3tzc6depkdZqg+Ph4LFq0CHv27EFaWhqKi4tN69zc3CoVQ1ZWFl599VWsXbsWp06dQk5Ojtl6e9/HMLG0c/JXn2UvsM+yViivv/G1117Dk08+iTp16qB///5o2LCh6a/2RYsWoaCgoMKPYa0X08Xl6u6gbLJqq3EyMzMBAHXq1LHYvqL9nMDVJnngavXWGmtjff/99+jTpw8AoH///ggLC4O3tzeUUli7di1+/fXXSr12o0ePxoYNG9CyZUuMHTsWdevWhaurKzIyMvDGG29Uaiyi2u7y5csArl4sc/jw4XK3MyZXLi4u2LFjB+bNm4c1a9aYzlwEBwdj5syZePbZZ28qwazKPuaLL77A2LFj4e3tjQEDBqBJkybw9PQ0XeCYnJxc4ccvLCxEVFQU9u/fj86dO2PixIkICgqCi4uLqVfV3vcxTCwdQzyung5fq2kUVCOs/cpNcXExnn/+eYSEhOCXX34xS9BEBK+88kpNhlhpvr6+AK5WH4xN7EYXLlyo8DjGRNbaVfDljfXiiy+ioKAAu3btMjttDwC7d++2eiV8efbu3YsNGzZgwIAB2LRpk9kBbPfu3XjjjTcqPBYR/bVvGDVqFFavXl2h+wQHB+Ptt9/GW2+9hWPHjmHHjh146623MHfuXLi6uuKZZ56pcjxV2cfMmzcPHh4e2LdvH8LCwszWxcbGVurx161bh/379+P+++/HBx98YDHW8uXLKzWeFthj6RjiwT5Lh2dMQipTETRKS0vDlStX0KNHD4uq308//YS8vDybxFhdjFdSfv/99xbrrC0rT6tWreDh4YGffvoJ+fn5ZutKS0utjnXq1CkEBgZaJJW5ubnYv3+/xfbXe5+MU4cMGTLEoiry7bffVvh5ENUm1/tOtWnTBr6+vvjpp58qPV2XUgpt2rTBI488gm+++QYAzKYnqso+17ivunZWCwDIzs7GL7/8YrH81KlTaNOmjUVSee7cOYvphm4Ul3H7oUOHWqxzlH0ME0vHwPksnYCxufv333+v9H3r1q0Lg8GA/fv3Izc317Q8PT0dM2fOtFmM1WXcuHHQ6XR4/fXXzea4zMnJwYsvvljhcdzc3DBmzBikpqZaTLnx4YcfWu2vDA0NRXp6utlptpKSEjz55JOm/q2yrvc+Gaut1x50Dh8+jAULFlT4eRDVJtf7Trm4uGD69OlITk7Gk08+aTW5PHTokKmCePr0aRw5csRiG2MlsexFPVXZ5zZu3Bi9e/fGgQMHsHLlSrN1L730ktUe9NDQUJw8edKsmpmfn4/p06eb9VpWJK7y9jEJCQkWFUx7xVPhDkA4n6VTaN26NUJCQhAbGwtPT080bNgQSilMnz79hvNP6nQ6zJgxA6+99ho6duyImJgYZGZm4quvvkJoaChCQkJq6FlUTatWrfD000/jpZdeQnh4OO6++264uLhgzZo1CA8Px6FDhywuJirPyy+/jO3bt+O5557Drl270LlzZxw9ehSbN29G//798fXXX5ttP3PmTHz99dfo2bMnxowZAw8PD8THxyMlJQVRUVEWjfi33347DAYDFi1ahMzMTFOF+Omnn0a3bt3QrVs3/Pe//8Uff/yBHj164MyZM1i/fj2GDBlS4VN5RLXJjfZ98+fPx/79+/Hmm29i06ZNiIyMRJ06dZCSkoKDBw/i119/xQ8//IC6devi119/xYgRI3Dbbbehffv2qFevHlJSUrB27Vro9Xqz2SKME5A/++yzOHbsGPz8/ODn54fp06dfN9533nkHERERuO+++7B27VqEhYVh7969+PHHH9GrVy+LyuHMmTMxc+ZMdO7cGaNHj0ZxcTG++eYbiAg6duxo0W5zvbhiYmLQpEkTvPLKKzh06BDat2+P48ePY+PGjRg+fLhjzJOr5VxHvFX8Bs5nafc33GAeSxGR3bt3S2RkpPj4+FjMh2acn7Hs/GhlFRYWyosvvihhYWHi7u4ujRs3lscff1yysrIkNDRUQkNDzba/3jyWkyZNsvoYsDKf5PXmsbQ21+P15qx79913pU2bNuLm5iYNGzaUJ598Us6ePSsAZNiwYVZjsiY5OVnGjh0r/v7+4unpKb169ZKEhIRy54dbvXq1dOnSRTw9PSU4OFjGjBkjp06dKvc137Rpk9x2221iMBhM75NRamqqTJ06VUJCQsTDw0PCw8PlnXfekcTExOu+tiKcx5I357zhBvNYilx/3ydydQ7cJUuWSEREhPj6+pr2cQMHDpT33ntPsrOzRUTk7Nmz8vTTT0uPHj2kbt264ubmJo0bN5bRo0fLnj17LB532bJlEh4eLu7u7gLAYj9ZnoMHD8rgwYPF29tbfHx8ZNCgQXLw4EGr+4zS0lJZvHixtGvXTjw8PKRevXoybdo0uXDhgtX9543iSkxMlFGjRkmdOnXE09NTbrvtNomNjb3hfKAi9jGPpRL5a5oMsl9Kqc4AlopIJ61jIeuUUgtjYmKeWr9+veXVN1Subdu24c4778Ts2bOxcOFCrcOpVlu3bsWECROuXLx40V/rWIhsRSnVJyIi4stdu3b5ax1Lbff555/jkUceWZuWljZCqxjYY+k42GdJDu3ixYsWzeoZGRmmKziHDx+uQVRERGRL7LF0EMI+S3JwK1euxKuvvoo+ffogJCQEf/zxB7Zs2YLU1FRMnjwZt99+u9YhEhHRTWJi6Vjiwfks7ZlUZSqh2uKOO+5A165dsW3bNly+fBl6vR5t2rTBP//5T8yYMUPr8GrEn58P9h+Rs+G+z06UlJRARDR9M5hYOpZ4AEu1DoLKdcH4KxJkqVu3bli3bp3WYWgqPT0dpaWluTfeksihZFTmp2Cp+qSnp6OoqEjTn39mj6VjYZ+lffvfgQMHlL1PVk7a2bJlC9LT0y1nXiZybMeSk5Pd0tPTtY6j1tu8eXNWVlbWTi1jYGLpQP4sbxv7LMnOiEiSh4dH5qZNm7QOhexQfn4+1q5dCxF5QetYiGxJRPI8PDy2O8Qci07sypUriIuLcwWwQcs4mFg6nnjw5x3t1uXLl+fef//9+Pnnn7UOhexIfn4+Bg0aBJ1Od0pEDmodD5Gtpaen/+vxxx/PtvZTiFT9srOz0bdv3xxXV9clIpKpZSycx9LBcD5L+6eU+qfBYPjXgAEDMGHCBDRr1gwGgwFKcXrL2qS4uBhpaWnYtGkTVqxYgdzc3OSsrKwwEancDyITOQilVG+DwbApOjpaJk6c6NOyZUt4enpy31dNioqKkJqaivXr1xd8+umnxYWFhSuysrIeEY0TOyaWDkYppQeQCiBMRHiliJ1SSjUB8FxQUNBIEfEWEZ4dqH1Ep9MV5eTk7M/Pz39FRNZrHRBRdVNK+QKICQwMnKiUalJaWmq44Z2oqkr0ev3lzMzM9YWFhbEi8pvWAQFMLB2SUmodrlYt12odCzkmpVQbAF1EZKXWsRARkfNgFcUxxYN9llQFSim9Umo2gDgABVrHAwBKqS1/VjmIiDSjlHpKKRWgdRyOjomlY4oHE0uqpD+rlN8BiATQVURWaxyS0e8AZmodBBHVemMBNNA6CEfHxNIxcT5LqrBrqpSLAdwlIikah1XWSwBmsWpJROT4mFg6IM5nSRVlpUq5TOsrBq8lIom4Ou8aq5ZERA6OiaXjigdPh1M5HKBKeS1WLYmInAATS8cVDyaWZIUjVCmvxaolEZFzYGLpuNhnSWYcsEp5LVYtiYgcHBNLB8U+SyrLEauU12LVkojI8TGxdGzx4OnwWs0JqpTXYtWSiMiBMbF0bPFgYllrOUOV8lqsWhIROTYmlo6NfZa1kBNWKa/FqiURkYNiYunA2GdZ+zhjlfJarFoSETkuJpaOLx48He70akGV8lqsWhIROSAmlo4vHkwsnVptqFJei1VLIiLHxMTS8bHP0knVwirltVi1JCJyMEwsHRz7LJ1TbaxSXotVSyIix8PE0jnEg6fDnQKrlBZYtSQiciBMLJ1DPJhYOjxWKS2xaklE5FiYWDoH9lk6MFYpb4hVSyIiB8HE0gmwz9JxsUp5Y6xaEhE5DiaWziMePB3uMFilrDRWLYmIHAATS+cRDyaWDoFVyspj1ZKIyDEwsXQe7LO0c6xS3jRWLYmI7BwTSyfBPkv7xirlzWPVkojI/jGxdC7x4Olwu8Iqpc2xaklEZMeYWDqXeDCxtBusUtoeq5ZERPaNiaVzYZ+lHWCVstqxaklEZKeYWDoR9llqj1XK6seqJRGR/WJi6XziwdPhNY5VyhrHqiURkR1iYul84sHEskaxSlnzWLUkIrJPTCydD/ssawirlJpj1ZKIyM4wsXQy7LOsGaxSao9VSyIi+8PE0jnFg6fDq4VSyoVVSrvCqiURkR1hYumc4sHE0uauqVLeyiql9li1JCKyL0wsnRP7LG3ozyrl/+FqlfI9XK1S/q5xWPQXVi2JiOwEE0snxD5L2ylTpewN9lLaJVYtiYjsBxNL5xUPng6vsnKqlOyltF+sWhIR2QEmls4rHkwsq4RVSsfDqiURkX1gYum82GdZSaxSOjxWLYmINMbE0kmxz7JyWKV0fKxaEhFpj4mlc4sHT4dfF6uUTodVSyIiDTGxdG7xYGJZLlYpnQ+rlkRE2mJi6dzYZ2kFq5ROj1VLIiKNMLF0YuyztMQqpfNj1ZKISDtMLJ1fHHg6nFXK2odVSyIiDTCxdH7xqOWJJauUtQ+rlkRE2mBi6fwOAGhYG/ssWaWs9Vi1JCKqYUwsnZyIlKIW9lmySkmsWhIR1TwmlrVDPGrJ6XBWKekarFoSEdUgJpa1QzxqQWLJKiVdi1VLIqKaxcSydnDqPktWKekGWLUkIqohTCxrAWfus2SVkm6EVUsioprDxLL2iIcTnQ5nlZIqiVVLIqIawMSy9oiHkySWrFJSZbFqSURUM5hY1h4O32fJKiXdJFYtiYiqGRPLWqJMn2WkUipcr9fPCggI2KiU6qx1bBXBKiXdLFYtiYiqHxNLJ6eU0hkTSX9//zYGg+Hz0NDQXffff/9CAHcCcNE6xuthlZJsjFVLIqJqpFj0cV5KKR83N7ek4OBgt7vuusvlzjvv9Ojduzfq1q2LvLw8+Pv7FxQWFvqKSKGGMdZRSvUrLS39zMq6NgCWAUgD8CATSrIFpdSHAE6LyItax0JE9kMp9ROAySJySOtYHBkrlk5MRLLc3d0/b9mypXrnnXc8Ro8ejbp16wIA9u3bBx8fnxNaJpUA4Ofn96ZSaqVSKtK4jFVKqmasWhIRVRMmlk4uKytr5v79+3+YPn16ftnq9Pfffy/5+fk7NAwNSqkuer1+6OrVq5W3t3esUsqLvZRU3dhrSURUfZhYOjkRKcnMzBz++eefJ7/++uslxuU7duzIysnJSdAqLqWU8vPz++A///mP54gRIzBq1Cg/T0/P7WCVkmoGq5ZERNWAPZa1hFIqxNvb+9dPPvkkeNiwYQgICMi9cuVKCxH5Q4t49Hr9qNatWy87ePCgt06nQ2ZmJlq0aFFy8eLFMSKyRouYqHZhryURlcUeS9tgxbKWEJFz2dnZfSdOnJi9Zs0alJSU5GuVVCqlPDw9Pd99//33vXW6qx9BX19frFy5Uu/t7f2OUspLi7io1mHVkojIxphY1iIiciA3N3fMuHHj4OLikqhVHAaD4cl+/fr5REREmC2/8847MWrUKD9fX9/XNQqNahH2WhIR2R5PhddC7u7u/y4qKjpfWlr6Wk0/tlKqgYeHR9LKlStdRARJSUk4ceJE3vHjxwtOnz6tzp8/76mUupCfnx/656TuRNVGKdUMwA8AwkQkU+t4iEg7PBVuG0wsqUYppZZ4eHhM9fT0TNbpdL/l5OQczcvLOwEgCcBpAGdEJE/bKKk2Ya8lEQFMLG2FiSUR1WqsWhIRwMTSVuz65/wqQykVAGB4UFDQBBFpJCLuWsdETq9Ur9enZ2dnf5Wfn79KRA5rHRBVnogkKqWMvZasWhIR3QSnqFgqpQYZDIbVgwYNKh0/frx38+bNYTAYtA6LnFxxcTHS0tLw1VdfFS5btqwwLy9vY2Zm5gQRKbnxvcmesGpJRKxY2obDJ5ZKqSg/P7+NcXFxXp07d9Y6HKql8vPzMWjQoJyff/75y4yMjElax0OVx15LotqNiaVtOPx0Q0FBQS98+OGHTCpJUx4eHvjqq6+8SkpK7lZK1dM6HqoSzmtJRHSTHDqxVEoF5uXldR0yZIjWoRDBw8MDw4cPF6XUKK1jocrjvJZERDfPoRNLAB07dOiQz35KshcDBw70DAwM7KN1HFRlrFoSEd0ER08s/YOCgpTWQRAZBQQEQCkVqHUcVDWsWhIR3RxHTyyVXq9nYllNli1bBqUUli1bZra8SZMmaNKkyU2PY0vz5s2DUgrx8fHV9hgVodfroZTSaxoE3SxWLYmIqsjRE0uqJeLj46GUwrx587QOhZwcq5ZERFXnNBOkU83Zvn271iFYePTRRzFu3Dg0btxY61DIObwE4Ael1Fuc15KIqOKYWFKlNW/eXOsQLAQHByM4OFjrMMhJ8Nd4iIiqplacCv/2228xYsQI3HLLLXB3d0ejRo0wcuRI7Nq1C4B5f97y5cvRtWtXeHp6IioqyjTGmTNnMG3aNDRo0ABubm5o2LAhpk2bhrNnz1o83h9//IG//e1vCAsLg8FgQGBgIMLDwzFjxgxkZv5V/Lhy5QrmzJmDtm3bwtvbG35+fmjdujWmTJliddxrNW/eHD4+PsjNzbW6/s4774ROp8OZM2dMj7dw4UJERkYiJCQEbm5uCAkJwX333YdTp05V+PUsr8fy8uXLePjhh3HLLbfA09MTt912G/73v/+VO87HH3+MYcOGoUmTJvDw8EBgYCAGDBiAuLg4s+3mzZuH6OhoAMD8+fOhlDLdkpKSTNuU12O5ceNGREdHw8/PDwaDAZ06dcKiRYtQUmL+AzlJSUlQSmHy5MlITEzE6NGjERAQAC8vL/Tr1w+//vprhV8jcgrstSQiqiSnr1i+8847mDlzJgwGA0aMGIHGjRsjJSUFu3btwurVq9GzZ0/Ttv/+978RFxeHoUOH4s4774SLy9WX58SJE+jZsydSU1MRExODdu3a4fDhw/j444+xceNGfPfdd2jRogUAIDc3FxEREUhKSkL//v0xYsQIFBYWIjExEcuWLcPs2bPh6+sLEcGAAQOwZ88eREREYODAgdDpdEhKSsL//vc/TJo0CY0aNbruc5swYQL+9a9/Yd26dbjnnnvM1v3xxx/YsWMHevfubTo9fPToUcyZMwfR0dEYMWIEvLy8cOzYMaxatQqbNm3C/v37ERoaWqXXOTc3F1FRUTh48CBuv/12REZG4uzZsxg7diz69+9v9T6PPPIIOnbsiH79+qFOnTpISUnB2rVr0a9fP6xZswbDhg0DAERFRSEpKQnLly9HZGSkWcLv7+9/3bjeeOMN/P3vf0dgYCDGjx8PLy8vbNiwAY899hi+/fZbrF69GkqZX/+VlJSE7t27o23btpg6dSpOnTqFdevWITo6GkePHsUtt9xSpdeIHAurlkREVSAiDnsDMHLo0KFXpBwHDhwQvV4vISEhcvr0abN1paWlkpKSIiIic+fOFQDi5eUlBw4csBinT58+AkCWLFlitnzJkiUCQPr27Wtatn79egEgjz32mMU4mZmZUlBQYIoNgIwYMcJiu/z8fMnKyirvaZmcOHFCAMjgwYMt1r366qsCQD788EPTsoyMDLl06ZLFtjt27BCdTif333+/2fKlS5cKAFm6dKnZ8tDQUAkNDTVbZnwNH3jgAbPlW7duFQBWx0lMTLSI5dy5cxISEiJhYWFmy+Pi4gSAzJ071+I+ZR8/Li7OtOzUqVPi4uIidevWlTNnzpiWFxQUSGRkpACQTz75xLT89OnTplhffvlls/Gfe+45ASALFiyw+vhGW7ZskTp16uwUO/h+8GaTfUwzABcA+GodC2+88Va9NwA/AWivdRyOfnPqU+GLFy9GSUkJXnjhBYtTt0ophISEmC178MEHER4ebrbs7Nmz2LFjB9q2bYsHHnjAbN0DDzyANm3aYPv27Ranrq1N2u7j4wM3N7cbbufu7g5vb+8bPr8WLVqge/fu+Prrr3Hx4kWzdZ9++ik8PDwwevRo0zI/Pz8EBlpOsRgdHY127dph27ZtN3zM8qxYsQJubm7417/+Zba8f//+6Nu3r9X7NG3a1GJZ/fr1MWrUKJw4cQLJyclVjgcAVq5cieLiYjzxxBNm1V83Nze8/PLLAGB1CqSmTZviqaeeMls2bdo0AMDevXtvKiZyLMIrxImIKsWpT4X/+OOPAFDuqdhrdevWzWLZzz//DACIjIy0OGWqlELv3r1x9OhR/Prrr2jUqBF69+6NevXqYcGCBfjll18wZMgQ9OzZE+Hh4Wb3b9OmDcLDw7Fq1SqcPXsWw4cPR69evdClSxfo9X9Ng5iUlGSR/Pj7++Pvf/87AGDixInYs2cPYmNjMXPm1WPf4cOH8csvv+Duu++Gn5+f2X3j4+OxaNEi7NmzB2lpaSguLjatuzbpraisrCycPn0abdu2Rb16lj+T3atXL6tXkicmJmLBggXYsWMHUlJSUFBQYLb+3LlzVT41D/z13pU9dW7Uo0cPGAwG/PLLLxbrOnbsCJ3O/G+uhg0bAgAyMjKqHA85rBcB7OYV4kREN+bUiWVGRgaUUqhfv36FtrfWO2e82Ka8vjpjInXlyhUAV6uCP/zwA+bOnYsNGzZg8+bNAK4mJs888wxmzJgBAHBxccGOHTswb948rFmzBk888QSAq1c3z5w5E88++yz0ej2SkpIwf/58s8cMDQ01JZbjxo3DY489hpUrV5oSy08++QTA1aSzrC+++AJjx46Ft7c3BgwYgCZNmsDT09M0eXlVK4TG5163bl2r6629didPnkS3bt2QmZmJ6OhoxMTEwNfXFzqdDvHx8UhISLBINCvrRu9d3bp1kZKSYrH82mQcgKnf9toLfsj5ichp9loSEVWMUyeW/v7+EBH88ccfaNCgwQ23v7YiCQC+vlcvCL1w4YLV+xiXG7cDrl41vXz5cpSUlODgwYP4+uuv8eabb+KRRx5BQECA6UKb4OBgvP3223jrrbdw7Ngx7NixA2+99Rbmzp0LV1dXPPPMM4iKijL2flgVFBSEQYMGYf369Th58iSaN2+OVatWITg4GAMHDjTbdt68efDw8MC+ffsQFhZmti42NvaGr095jM89NTXV6nprr91//vMfpKen49NPP8W9995rtu7hhx9GQkJCleO5Nq4LFy5YrXympqaavW9E18F5LYmIKsCpeyyNp7a//vrrKo/RqVMnAMDOnTstEjwRwbfffmu2XVl6vR6dOnXC7Nmz8dlnnwEA1q9fb7GdUgpt2rTBI488gm+++abc7cozYcIEAFf7KhMSEkxXY7u6upptd+rUKbRp08YiqTx37lylphu6lq+vL5o2bYqTJ0/i/PnzFuuNr9G1sQDA0KFDzZaXlpbiu+++s9je2B5QmYph586dAcDqFEQ//vgj8vLyrL5vRNdiryURUcU4dWL58MMPQ6/X47nnnrM4zWusZN5I48aNER0dbZpeqKyPP/4Yhw8fRp8+fUwXhxw6dMjqKWVj1c54sc7p06dx5MiRG25XETExMfDz88PKlSvLPQ0OXD2FfvLkSbMKYn5+PqZPn27Wa1kVEydORGFhIebMmWO2/Ouvv7baX2msIBrnEjVauHAhDh06ZLG98aKj33//vcIxjR8/Hi4uLnj99ddx7tw50/KioiI8/fTTAIDJkydXeDyq9TivJRHRDTj1qfDw8HAsWrQIs2bNQrt27TB8+HCEhobi/Pnz2LlzJ4YMGYJFixbdcJz33nsPPXv2xAMPPIANGzagbdu2OHLkCNavX486dergvffeM227bds2PPHEE4iIiEDr1q0RFBSExMRErF+/HgaDAY8++igA4Ndff8WIESNw2223oX379qhXr55pHke9Xm/quawIDw8P3H333fjwww+RnJyMsLAwdO/e3WK7mTNnYubMmejcuTNGjx6N4uJifPPNNxARdOzY8aYmAJ89ezbWrFmDDz74AIcPH0bv3r1x9uxZ/Pe//8WQIUOwadMms+0ffvhhLF26FCNHjsTYsWMRFBSE3bt3Y//+/Va3b926NUJCQhAbGwtPT080bNgQSilMnz7dak8kcHUC+YULF+KJJ55Ahw4dMGbMGHh5eWHjxo04duwYhg0bZqr2Et2IcF5LIqIb03q+o5u54QbzWBrFxcXJXXfdJYGBgeLm5iYNGzaUUaNGyXfffSci1udAvFZSUpJMmTJF6tevLy4uLlK/fn2ZMmWKJCUlmW135MgR+dvf/iadO3eWoKAgcXd3l2bNmsnkyZPlyJEjpu3Onj0rTz/9tPTo0UPq1q0rbm5u0rhxYxk9erTs2bPnRk/JQkJCgmkOxvnz51vdprS0VBYvXizt2rUTDw8PqVevnkybNk0uXLhgmtexrMrMYykicunSJXnwwQelTp064uHhIV27dpU1a9aUO05cXJxERESIj4+P+Pv7y+DBg2Xfvn3lvh+7d++WyMhI8fHxMT1X4/yk13sP161bZ7qfu7u7hIeHy2uvvSZFRUVm2xnnsZw0aZLV1w+AREZGWl1nxHksnfsGzmvJG29OewPnsbTJTf35YjokpdTIoUOHLl23bh1PTZFd2Lp1KyZOnPhtampqb61joeqhlPoQwGkRYdWSyIkopX4CMFlELPuxqMKcuseSiKgasNeSiKgcTCyJiCpBeIU4EVG5HD2xlJKSEsc9l09Op6SkBCLCWdSdH6uWRERWOHpimXHp0iUmlmQ30tPTISKXtY6DqherlkRE1jl6YvnLgQMHPPLy8rSOgwgAsGXLltzLly/v0DoOqhGsWhIRXcOhE0sRSTcYDPuunfOQSAv5+flYu3atEpEvtY6Fqh+rlkRElhw6sQSAS5cuPXf//ffn/Pzzz1qHQrVYfn4+Bg0alKPX678QEcvftSRnxaolEVEZDj2PpZFSapDBYFg9cODA0nvvvde7WbNmMBgMUEppHRo5seLiYqSlpWHz5s2Fy5cvL8zLy9uYmZk5gRfv1C6c15LIOXAeS9twisQSAJRSAQCGBQUFTRSRRiLirnVM5PRK9Hp9RnZ29ub8/PzPROSw1gFRzVNKNQPwA4AwEcnUOh4iqhomlrbhNIklEZFWWLUkcnxMLG3D4XssyTaUUk/+WfUlospjryUREZhY0l/GAWigdRBEjohXiBMRXcXEkojINli1JKJaj4klEZENsGpJRMTEkojIlli1JKJajYklEZGNsGpJRLUdE0siItti1ZKIai0mlkRENsSqJRHVZkwsiYhsj1VLIqqVmFgSEdkYq5ZEVFsxsSQiqh6sWhJRrcPEkoioGrBqSUS1ERNLIqLqw6olEdUqTCyJiKoJq5ZEVNswsSQiql6sWhJRrcHEkoioGrFqSUS1CRNLIqLqx6olEdUKTCyJiKoZq5ZEVFswsSQiqhmsWhKR02NiSURUA1i1JKLagIklEVHNYdWSiJwaE0siohrCqiUROTsmlkRENYtVSyJyWkwsiYhqEKuWROTMmFgSEdU8Vi2JyCkxsSQiqmGsWhKRs2JiSUSkDVYticjpMLEkItIAq5ZE5IyYWBIRaYdVSyJyKkwsiYg0wqolETkbJpZERNpi1ZKInAYTSyIiDbFqSUTOhIklEZH2WLUkIqfAxJKISGOsWhKRs2BiSURkH1i1JCKHx8SSiMgOsGpJRM6AiSURkf1g1ZKIHBoTSyIiO8GqJRE5OiaWRET2hVVLInJYTCyJiOwIq5ZE5MiYWBIR2R9WLYnIITGxJCKyM6xaEpGjYmJJRGSfWLUkIofDxJKIyA6xaklEjoiJJRGR/WLVkogcChNLIiI7xaolETkaJpZERPaNVUsichhMLImI7BirlkTkSJhYEhHZP1YticghMLEkIrJzrFoSkaNgYklE5BhYtSQiu8fEkojIAbBqSUSOgIklEZHjYNWSiOwaE0siIgfBqiUR2TsmlkREjoVVSyKyWy5aB0DaUEp5AahTZpEbgBClVPaf/84VkdSaj4yIrkdEEpVSxqrli0opnVJqlJeXV6+srKxZWsdH5Cj+PA7WLbPIDUADpVTOn//OFZELNR+ZY1MionUMpAEPD4/FSqmpgYGBeQAgIt4A8pRSJQUFBS4ZGRmqpKTES/gBIbI7SqlmAH5QSj3l4+Pzr0aNGgUnJSUVZ2dn+2sdG5Gj8PDwWKzT6UzHwdLSUm+lVK5SqjQ/P5/HwSriqfBaqqCg4D0vL6/CxMRE35SUFN9z587pzp0755WSkuJ73333uXh6en7ELxOR/fmzQtnVx8fHs1OnTu+vXbs2ND4+3kvruIgcTUFBwXuenp6Fp06d8v399999z507p0tJSfH+/fffeRy8CUwsaykR+bWkpOS7jz76yOxLk5qaiiVLlhRnZWW9oFVsRGSdXq8f7uvrm9ilS5eP1q1b571v3z736OhoKKW0Do3I4RiPgx9++CGPgzbExLIWy8jImD1nzpzcgoIC07KXX365UK/Xf8y+EiL7opRSHh4eC9q2bRvy7bff+jChJLp5GRkZs+fOncvjoA0xsazFrq1a8q80IvslIpKbm3vr0aNH13bs2DHnt99+0zokIod3bdWSx8Gbx8SylitbtVywYEER/0ojsl8ikpORkTEmOTl5VteuXXNiY2O1DonI4ZWtWvI4ePOYWNZyxr/WXnrpJbz//vsl/CuNyP4VFBR8nJ2d3f3BBx88M23atIL8/HwA4HlxoiowHgdffPFFHgdtgNMNEZRSHQH84uXl9Vl2dvZ4reMhoopRSnn5+fkt9fX1HXn58uXi7OxsD61jInJEPA7aDhNLAgDo9fpPS0tLnxORJK1jIaLKcXNzm+3i4vJUbm5unRtvTUTW6PX6T/48DiZrHYsjY2JJRERERDbBHksiIiIisgmH/a1wpZQvgKFBQUETADQtLS1lbxFprUSn013Ozs7eUFBQ8JmIcD4YG1FXJ2y8zcvLa4LBYIgsKSnxAy9WoRqk0+nyROTU5cuXlwPYJCI5N7yTBpRSOgB3+Pj4THBzc4soLS31Ab8rVA10Ol0+gNOXLl36FMB6EckEHPRUuFKqt8Fg2NSnTx+ZMGGCT6tWrWAwGDhZMGmqqKgIqamp2LBhQ8Enn3xSXFhYuCIrK+sR/iTYzVFKGXx9fbcGBQV1mjJlimffvn31/v7+0Ov1WodGtYSIIDc3F0eOHMGKFSsyv/vuO8nNze0rIvu0jq0spZSPr69vQr169ZpPmzbNu3fv3jo/Pz/odDw5SbYlIsjLy8Px48fx6aefZu3YsUPl5eUNEZGdDpdYKqVu9fHxidu8ebN3z549tQ6HyKrs7GxERUXlnDhx4sMrV678Xet4HJVSSvn6+iYMHTr01uXLlxt4gCR7sHXrVowePTozOzv7dhE5onU8AKCUcvH19d173333tX7zzTc9WGihmrRr1y4MHjw4OysrK9rh9tIBAQFzXn/9dSaVZNe8vb2xfft2r6Kioof+bNugqrktKCioE5NKsicDBgzAnDlzvH19fZ/QOpYy+jRr1qwZk0rSQs+ePfH66697BwQE/NOh9tRKKUN+fn7fUaNGaR0K0Q35+fkhOjq6CECM1rE4Ki8vrwlTpkzxZFJJ9mbcuHG64uLi0Uopu+jJ8PPzmzxt2jQfJpWklZEjRyI/P7+fo+2tW4eGhhYGBARoHQdRhQwePNjHx8ent9ZxOCqDwRDZt29fuzhwE5XVqFEj+Pr6KgCNtY4FAHQ63R19+vRhVkmaCQwMROPGjYscLbH09/f31zoGogoLCAiAq6trkNZxOKqSkhI/fufJXvn5+ZUA8NM6DgAoKiry5neFtBYQECCOlliq610JGh8fD6UU5s2bV3MREV2HXq+HvZwqc1DX/c47sqioKLubyUIphaioKK3DuCmTJ0+GUgpJSUlVHqOix5I/P5v2chy97nelsLAQzz33HJo3bw43NzcopRAfH6/ZcdMW75NW7DH2ZcuWQSmFZcuWaRqHXq+3my+E07HHg4azadKkCZo0aaJ1GOQk5s2bZzrYEjmbV199FS+++CIaN26M2bNnY+7cudx/OpCkpCQopTB58mStQ7khh50gnYjIka1YsQK5ublah0G1xObNm+Ht7Y2vv/4arq6upuV169bF0aNHERwcrGF0dLNGjBiBHj16oH79+lqHwsSSiEgLjRvbxTUfVEucO3cOQUFBZkklAHh6eqJ169YaRUW24ufnBz8/u2j3dd5T4Tt37kRkZCS8vb0RGBiI8ePH4/fff7e6bWpqKh577DG0aNEC7u7uCA4OxqhRo3Do0CGLbU+cOIEpU6agadOm8PDwQHBwMLp06YInnvhrOjOlFBISEkz/b7xVpIRt7HE6e/Ysxo4di6CgIHh5eSEqKgrff/+9xfa//fYbZs+ejS5duiAoKAgeHh5o2bIlnn76aWRnZ1tsbzxFX1BQgDlz5qBFixZwdXU19dfczHj/+Mc/0LhxYxgMBnTt2hXbtm0DAGRlZWHWrFlo0KABPDw8cPvtt+Onn36q8nthPCWQnJyM5ORks9f42j6hnTt3IiYmBsHBwXB3d0dYWBiee+45i0pR2T6jH374AQMGDIC/v79ZO0NcXBwGDRqEkJAQuLu7IyQkBFFRUfjwww+tv5lkQSmlU0r1U0rd9B+1FXnPRAQff/wxIiIi4OvrC09PT9x66634+OOPzcaKiorC/PnzAQDR0dGmz1PZU4XG1ouMjAzMmjULjRo1gouLi6mnad++fXj00UfRvn17+Pn5wWAwIDw8HC+//DKKioos4rfWLlO2T2r79u3o2bMnvLy8EBQUhEmTJuHSpUtWX4sDBw5g3LhxqF+/Ptzc3BAaGoqZM2eWu/2HH36I9u3bw8PDA40aNcLs2bORn59/w9e8LGOfWWJiIl599VW0bNkSBoMBbdu2RWxsLICrv0Y1Z84c0/6yQ4cO2Lp1q9Xxzpw5g2nTpqFBgwZwc3NDw4YNMW3aNJw9e9bq9ocPH8Zdd90FHx8f+Pn5YfDgwVb32WWtW7cOffv2RUBAADw8PNC+fXu8+uqrKCkpqdRzrwlKKRelVJ8/f6KxyowtHqdPnzbbXxp7acvrsTR+3nNycvD444+jQYMGcHd3R4cOHbB69WqLx6nssaMqKvI5T05Ohk6nQ9++fa2OkZ+fDz8/P7Ro0cKmsV+vx7G81/h///sf7rnnHrRo0QKenp7w8/NDr1698OWXX1qM3bRpUwDA8uXLzY55xtad6z3+999/jyFDhiAwMBAeHh5o3bo15s2bZ/WMifGzcfHiRUydOhV169aFwWBAjx49Ktwm5JQVy927d2PBggUYMmQIZs2ahf379+Ozzz7Drl27sHfvXtxyyy2mbU+dOoWoqCikpKSgf//+GD58OFJTU/Hll19i69at2L59O7p37w7g6l983bp1Q05ODoYMGYKxY8ciOzsbJ06cwFtvvYXXXnsNADB37lwsW7YMycnJmDt3rumxOnXqVKH409PTERERgfr16+PBBx9ESkoKPv/8c0RHR2Pr1q1mzfVr1qzBRx99hOjoaERFRaG0tBS7d+/GwoULkZCQgJ07d1r8hQpcnW/q119/xYABAxAYGIhmzZrd1Hhjx47FwYMHMXToUOTl5WHlypW466678P333+Ohhx5Cfn4+Ro8ejYsXL+Lzzz/HgAEDcPr0afj6/jV3eEXfC39/f8ydOxeLFi0CAPz97383jVH2tVm8eDFmzJiBgIAAxMTEoE6dOti7dy9efPFFxMXFIS4uDm5ubmbP4/vvv8dLL72E6OhoPPjggzhz5gwAYNOmTYiJiYG/vz+GDRuG+vXr4+LFi/jll1+wcuVK3H///RV6bwlNAXzt4+Pzh4uLy9MlJSWfiUjxzQxY3nsmIpgwYQJWrVqFli1bYvz48XBzc8M333yDadOm4ciRI3j11VcBwPRHX0JCAiZNmmRKKK+9yragoAB9+vRBVlYWYmJi4ObmZtqffPDBB9iwYQN69+6NwYMHIzc3F/Hx8XjmmWewd+9ei4PF9WzYsAEbN25ETEwMpk+fjp07d2LFihU4deoUdu3aZbbt+vXrMWbMGOj1egwdOhSNGjXCkSNH8Pbbb2Pr1q3Ys2cPyk7R9vzzz2POnDm45ZZb8MADD8DV1RWff/45jh49WpmX3eTxxx/Hnj17EBMTA71ej9jYWIwfPx4BAQF45513cOjQIQwePBj5+flYtWoVhg4dimPHjpkOlMDVP9h79uyJ1NRUxMTEoF27djh8+DA+/vhjbNy4Ed99951ZInDo0CFEREQgOzsbI0eORFhYGH788UdERESgY8eOVuP8xz/+gQULFqBhw4YYNWoUfH19sXPnTjz11FPYs2cPvvjiiyo9/2oUrpTa7uPjk6zX658sLS1dIyKllR3EuE+8dn9Zkf7KoqIi9O/fH5cvX8bIkSORm5uL2NhYjBkzBlu2bEH//v1N21b12FFRFf2ch4aGolevXoiPj0dKSgoaNGhgNs66deuQmZmJxx57rMZiL88zzzwDNzc39OzZ03RMWb9+PUaPHo0333wTM2fOBHA1d/jb3/6GN954Ax07dsTw4cNNY9zoffzyyy8xbtw4uLm5YezYsahbty62bduG+fPn4+uvv0ZcXBzc3d3N7pORkWH6Y/zee+9Famqq6bi9b98+tG/f/vpPTEQc5gagT0RERLqUIy4uTgAIAPnwww/N1s2fP18AyNSpU82W33HHHeLi4iJff/212fLjx4+Lj4+PhIeHm5a9+eabAkDeeOMNi8e+ePGi2b8jIyPl6stbOcb4J06cKKWlpabl8fHxopSSFi1aSElJiWn577//LgUFBRbjGJ/vp59+ajWuTp06yaVLlyzuV9XxIiIiJDs727Q8NjZWAIi/v7/cfffdUlRUZFq3cOFCASCvv/662ViVeS9EREJDQyU0NNQiVhGRw4cPi4uLi3Tu3NnieS5YsEAAyKuvvmpaVvaz89FHH1mMN3LkSAEgv/76q8W6tLQ0qzGIXH0dgoKC/id28P2xhxuA5g0bNrzy7bffyh133JHl4+OTotfrJwJwsbZ9QEBA8rFjx6y+tjd6z95//30BINOmTTP7/BUUFEhMTIwAkJ9++sm0fO7cuQJA4uLirD5eaGioAJD+/ftLbm6uxfqkpCQpLi42W1ZaWipTp04VALJr1y6zddb2EUuXLhUA4uLiYrZ9cXGxREVFCQD54YcfTMvT0tLE19dXGjZsKMnJyWZjrVq1SgDIo48+alp24sQJcXFxkQYNGsiFCxdMy69cuSKtWrUSABIZGWn1+V9r0qRJAkDCwsIkNTXVtHz37t2m737Pnj3N9guff/65AJBZs2aZjdWnTx8BIEuWLDFbvmTJEgEgffv2NVtufO2u3R8988wzps/E6dOnTcu//vprASCDBg2SnJwc0/LS0lJ5+OGHBYCsXr3atNz42Zo7d+51X4O2bdumA+gi1fNd6dymTZv07du3S9euXTN9fX1P63S60QB01rb39va+dP78+XJjLW9/Wd5zNX7ehw0bZnZM2LZtmwCQAQMGmG1f2WOH8fNT9n0qT2U/5x988IEAkFdeecVirLvuuksAyIkTJ2wau/G7u3TpUotxynuNT506ZbFtVlaWhIeHi5+fn9ln9fTp0wJAJk2aZHGf8h4/MzNT/P39xd3d3ezYVVpaKuPHjxcA8vzzz5uNY/z+zJgxwyzX+PDDDwWAPPTQQ1Yf3ygiIiJd8wNNZW4VTSxbtWpllpSJiOTm5kqdOnXEYDCYPkD79+83HXisefzxxwWAHDx4UET+Sizff//9672uInJziaVer5czZ85YrBsyZIgAkG+//faG41y6dEkAyOTJk63GtW7dukrFdaPx4uPjzZYXFxeLq6urALDYEZw5c8biC1LZ90Lk+onlrFmzyn2tSkpKpE6dOtK1a1fTMuNnp3PnzlbHMyaWv/32m9X15WFiafEdbt6wYcMrxtfnRglmRRLL8t6zDh06iJeXl+Tl5VmsO3DggACQJ554wrSsoomltT8urmffvn0CQObNm2e2/HqJ5X333WcxjnHdm2++aVr2+uuvCwD55JNPrD52ly5dJDg42PRv44Hytddes9j2k08+qVJiuWzZMot1zZo1EwCSkJBgtty4Xyj7GMb9Qdu2bS3226WlpdKmTRsBYNonJicnCwDp0KGDxeNmZWWJv7+/xUF/6NChZmOUlZGRIUopGTVqlGmZvSSWbdu2TTe+DjdKMKsrsUxMTLQ6VmBg4HVfG6Pyjh2VSSwr+znPyMgQd3d3i8/IxYsXxdXVVXr06GHz2KuSWJbntddesziuViWxXLFihQCQ6dOnW2x/5swZcXFxkebNm5stByBeXl6SlZVltryoqEhcXFykS5cu1409IiIi3SlPhUdERFj0Lhn7/rZs2YLffvsN7du3x+7duwEA58+ftzqH17Fjx0z/bd++Pe666y48/fTTeOSRR/DNN99g4MCB6NmzJ1q2bGnT+ENDQ9GoUSOL5b169cKmTZvwyy+/wPhb6SKCpUuXYtmyZTh06BCuXLmC0tK/zpacO3fO6mN069bN6vKqjte5c2ezf+v1etStWxc5OTkWFykYr1pLSUkxLavse3EjxvG2bNli6vUsy9XV1TRmWeW9LmPGjMGaNWvQvXt33HPPPejTpw969eqFunXr3jAWI6XUvbh6Krg2CxQR03mXnj174rvvvvP+7rvvvGfPnv3hgQMH3nFxcfm/4uLi9yo6oLX3LDc3FwcPHkRISAhefvlli/XGnkdrn4Hr8fDwQHh4uNV1hYWFePvttxEbG4tjx44hOzvbmCAAKP+7Y02XLl0sljVs2BDA1dNURsbP+e7du3Hy5EmL++Tn5yMtLQ1paWkIDg7Gr7/+CuDqvuRa1pZVxLXffeDqdzwxMdGi/ce4Xyj73f/5558BAJGRkRb7baUUevfujaNHj+LXX39Fo0aNTM/BuA8sy9vbG506dbLoBdu9eze8vLzw0UcfWX0OBoOh0p+Fa+K8BcD9AGw5x1z90tJSjz/HR58+fbB3716fuLg4nyeffPLTkydP5un1+kdLSkpW2vAxzfj7+5u1LBg1bNgQP/zwg9myqh47KqKyn3M/Pz/ExMRg9erVOHjwoOk7Gxsbi6KiIkycOLHGYr+e1NRUvPzyy/jqq6+QnJyMvLw8s/U3+7jG75a1uWkbNWqE5s2b4/jx48jKyoKPj49pXVhYGLy9vc22d3FxwS233GK2/ymPUyaW5R3sjb1QV65cAQBcvnwZwNX+uU2bNpU7Xk5ODgCgadOm+OGHHzB//nx89dVXpp6cVq1a4fnnn8fdd99do/EDwKxZs/D222+jUaNGGDp0KOrXr2/ql5g/fz4KCgquO9a1qjpe2V5JIxcXF6tXqbm4XP3Ylb2gobLvxY0Yx3vxxRcrtL1Rea/L2LFj4erqikWLFmHJkiV49913TU3Or7/+eoX7Z8k6+atKA1w9FVNh1t6z9PR0iAhSUlJMF+VYU9HPk1HdunXLnZ929OjR2LBhA1q2bGnqZXJ1dUVGRgbeeOONcr871lzve1P2QhPj5/ydd9657ng5OTkIDg427Tus7WPK++zfSHnf/eutK/vdz8zMvO7j16tXD8Bf+73rPYfyxrl8+TKKi4tt+lmoDcq7ytjFxcUs8QKqfuyoiMp+zgFg4sSJWL16NVauXGn64/LTTz+Fq6srxo4dW2Oxl+fy5cu47bbbcObMGURERKBfv37w9/eHXq/HL7/8gnXr1t3041bku3X8+HFkZmaaJZbXe98rcqGbUyaWqampVpdfuHABwF8vmnGn99Zbb+HRRx+t0NgdOnTAl19+iaKiIuzbtw9fffUV3nzzTYwdOxYhISGIiIiosfhTU1PxzjvvoEOHDvjhhx/g6elp2vb8+fPX3YlaOzjezHg3qyrvRUXGu/YLcyPXm9R+5MiRGDlyJDIzM/H999+bGr4HDBiA48ePW1zocS0RqbbKgqNQSjVXSk0D4A4Au3btwv/93/9lHzx4MDM3N/f//ryYp1KX6Fp7z4zvf9euXcudgaAqyvt87N27Fxs2bMCAAQOwadMmlP0FlN27d+ONN96wWQxlGZ/nwYMHK1TJL7vvCA0NNVtn3L/UNONzKO/xjcuN25V9Dtfb/trHUEohLS3tpuO1RkQuAKjcX7E3oJTqrNPpxgPwEBHExcVh9uzZWSdOnLiUnZ39VFUv5qkO1X3sqOznHAAGDRqE4OBgrFq1CgsWLMCpU6ewZ88eDBs2DEFBf/3Krq1i1+muXrxfXGx5LWLZYpDRRx99hDNnzuCFF17As88+a7bu5Zdfxrp16yr0uNdT2e+WrTjldEPfffed2SkoAMjLy8O+fftgMBhMp66NV3tfW9KvCFdXV/To0QPz58/Hm2++CRHBxo0bTeuNB5aqTGORnJxsdYqNb7/9FsBfV5cnJiZCRNCvXz+zL0PZbSvD1uNVRlXeC71eX+7raxzPeArFlnx9fTFw4EC8//77mDx5MlJTU7Fnzx6bP44z27VrFyIiIrIHDx58bs+ePdOzsrIaFxcXf1rZpLI8Pj4+aNOmDY4ePVqhUzfAzX1nT506BQAYMmQIrv1Zver87lT2e2O8YtpaTNX9HS+PcX+2c+dOi/22iFjs94zP4dqr4wEgOzsbv/zyi8Xy7t2749KlSzhx4oTtAq8BIoIdO3bgtttuyxoxYkTSzz//PDUzM7N5SUnJantJKoHqP3ZU5fjg6uqKMWPG4OzZs0hISMCnn34KAJgwYYLZdraK3TjzQtk2DyPjKemyjPuMoUOHWqyz9rhV2T8Z21SsTROUkpKCU6dOoVmzZpUqvlSEUyaWx48ft5in7t///jcuXryIe+65xzTFTLdu3dC9e3d89tln+Pzzzy3GKS0tNc1HCVytSlj7K9mY9RsMBtOywMBAACh37szrKSkpwbPPPmu2k01ISMDmzZvRokUL3HHHHQBgqjh8//33Zqclfv/9dzz99NOVflxbj1cZlX0vgKuvcVpamtX592bMmAEXFxfMnDnTapKekZFh9ctenu3bt1t9HOPnoex7T9f3+++/+1RXQlnWrFmzkJubiwceeMDqac7Tp0+b/dbvzXxnjd+da5Odw4cPY8GCBZUer6KmTJkCHx8fPPvsszh8+LDF+tzcXLM/rsaPHw+9Xo/XX3/dbF+WmZmJF154odrivJ7GjRsjOjraNL1QWR9//DEOHz6MPn36mPrOGzdujN69e+PAgQNYudL8JMBLL71k9Q+JWbNmAQCmTp1qdW7P8+fPV3m6pep07Ngxf3tOKI2q+9hR2c+5kbGX8tNPP8XKlSvh7++PmJiYaom9S5cuUEohNjbW7Fhx4sQJq2csyttnrFq1Cps3b7bYPiAgAEqpSu2fhg0bBj8/PyxdutTsdRMRPPPMMygqKqqWn4h0ylPh/fv3x4wZM7Bp0ya0bt0a+/fvx9atW9GoUSO89NJLZtt+9tlniI6Oxrhx47Bo0SJ07doVHh4eOHPmDH744QdcvHjR9CFZuXIl3n33XURFRaFFixbw9fXFkSNHsHnzZgQHB2Pq1Kmmcfv06YPVq1fj7rvvxuDBg01N/0OGDLlh/B06dEB8fDx69OiBPn364Ny5c4iNjYWrqys++OADU8m9fv36GDVqFL788kvceuut6Nu3Ly5cuICNGzeiT58+SExMrNTrZuvxKqsy7wVw9TX+6aefEBMTg169epnmA+vZsyfat2+Pd999F9OnT0erVq0wePBgNG/eHJmZmUhMTERCQgImT56MxYsXVyi2J554AmfOnEFUVBSaNGkCpRR27dqFH3/8EXfccYdNWiBqidMA7szKyoqvjmSyrIceegi7d+/G8uXL8d1336Ffv34ICQnBhQsXcOzYMezZswerVq0yzQNnnBj92WefxbFjx0y/ZDF9+vQbPla3bt3QrVs3/Pe//8Uff/yBHj164MyZM1i/fj2GDBlidUJpW6hTpw4+++wz3H333ejYsSMGDhyI1q1bIz8/H8nJyUhISMAdd9yBLVu2AABatGiBOXPmYO7cuejQoQPGjBkDFxcXfPnllwgPD8fx48erJc4bee+999CzZ0888MAD2LBhA9q2bYsjR45g/fr1qFOnDt57z/xarnfeeQcRERG47777sHbtWoSFhWHv3r348ccf0atXL4uKz8CBA/HPf/4Tzz//PFq0aIGBAwciNDQUly5dwsmTJ/Htt9/ihRdeQJs2bWryad/IQRHpm5mZGW+PyWRZ1X3sqOzn3KhHjx4ICwvDihUrUFRUhAceeMBizkZbxd6gQQOMHTsWsbGx6Nq1KwYOHIjU1FT873//w8CBAy3msZ04cSIWLlyImTNnIi4uDqGhoThw4AC2bduGkSNHYs2aNWbbe3t747bbbsPOnTsxZcoUhIWFQafTYfz48eX+ipevry8++OAD3HPPPejevTvGjh2LOnXqYPv27fjpp5/QrVs3PPXUUxV6fpUiYvtpEqrrhgpONzR37lxJSEiQXr16iaenp/j7+8u4ceOsTjUhInL58mV57rnnpH379mIwGMTb21vCwsJk/PjxsmbNGtN2u3fvloceekjat28v/v7+YjAYJCwsTGbNmmUxdlFRkcyePVsaN24sLi4u150moCz8Od1HcnKy3H333RIQECAGg0F69+5tMQ+eyNXpNZ544glp0qSJuLu7S1hYmDz//PNSWFhodeqQG02DZMvxrjcdkLWxRCr+XhhjfeCBB6R+/fqi0+msTufw448/yrhx4yQkJERcXV0lODhYunTpIk8//bQcPXrUtN2NpoOIjY2VMWPGSPPmzcXT01P8/PykU6dO8sorr5jN02ftfpxuqOq3ikw3dKMpPD7//HPp16+fBAQEiKurqzRo0ECioqLktddes5h/dtmyZRIeHi7u7u4CwOzze73Ps4hIamqqTJ06VUJCQsTDw0PCw8PlnXfekcTERKvf/+tNN1TZKUuOHTsm06ZNk9DQUHFzc5OAgAAJDw+XWbNmyY8//mix/QcffCBt27YVNzc3adiwoTz55JOSm5tbpemGrE0XU5X9QlJSkkyZMkXq168vLi4uUr9+fZkyZYokJSVZHefgwYMyePBg8fb2Fh8fHxk0aJAcPHjwunF98803EhMTI3Xq1BFXV1epV6+e3H777fL888+b7cPtYbqhyt6qY7qh8j7v1t7fyh47KjPdkFFlP+cif02xBSvTX9k69pycHJk5c6bccsstpumOVq5cWe5r/Msvv0j//v0lICBAfHx8JDIyUrZt21bufuD48eMyePBg8ff3F6WU2fRo19t37Ny5UwYNGiT+/v7i5uYmLVu2lH/+859Wj13X2wfcaB8ocnW6IXV1HMeglOoTERHx5a5du/y1jqW6KKUQGRlZ4Z9OIvv2+eef45FHHlmblpY2QutYHFFgYGDyDz/80LhVq1Zah0JkoV27dhlHjhzpKyL7tY7Fx8fn0smTJwOrenU/kS307Nkzwyl7LImIiIio5jGxJCIiIiKbcLTEUqoyFQiRVkpKSiDVfJGKk+N3nuzWn59Ne7mwht8V0lxJSYnDXRWeUdE56RyVI/W80o2lp6ejqKjIcn4TqhC9Xn/F2b/z5LiuXLmiB2A5+7UGXF1dszMyMoJCQkK0DoVqsfT0dOVoFctjycnJbunp6VrHQVQhmzdvzsrKytqpdRyOKi8vL2H79u0sw5DdOXv2LDIzMwXAGa1jAYDS0tLvd+zYwcoEaeby5ctITk52c6jEUkTyPDw8tl87HxSRPbpy5Qri4uJcAWzQOhZHlZOT8+nSpUtzr/1dYiKtxcbGlrq4uKy2l1aXK1euLPvoo4+yeNaLtLJmzRoYDIZvHCqxBID09PR/Pf7449nWfs6LyF5kZ2ejb9++Oa6urktEJFPreBzY3kuXLv0yadKkPCaXZC+2bt2Kf/3rX9mZmZmvaR1LGTsSExMTZ82alc/kkmrarl278Pjjj2enp6c/71DzWBoppXobDIZN0dHRMnHiRJ+WLVvC09MTSimtQ6NarKioCKmpqVi/fn3Bp59+WlxYWLgiKyvrEXHEL5kdUUoZfH19twYGBnaaMmWKoW/fvi4BAQEWv8lNVF1EBDk5OTh69CiWL1+e+f3330tubm5fEdmndWxlKaV8fH19E+rVq9d8ypQpXpGRkXp/f3/Tr7UR2YqIIDc3F7/99hs++eSTrLi4OJWXlzdERHY6ZGIJAEopXwAxgYGBE5VSTUpLS/ljzaS1Er1efzkzM3N9YWFhrIj8pnVAzkJd/avxNi8vr3sNBkNUSUmJLxxvVgtyYDqdLldETl6+fHkFgM0iYvkD9HZAKaUDcLuPj88ENze3iNLSUl8ArLqQzel0ujwRSbp8+fInADYYz845bGJJRERERPaFf/ETERERkU0wsSQiIiIim2BiSUREREQ2wcSSiIiIiGyCiSURERER2QQTSyIiIiKyCSaWRERERGQTTCyJiIiIyCaYWBIRERGRTTCxJCIiIiKbYGJJRERERDbBxJKIiIiIbIKJJRERERHZBBNLIiIiIrIJJpZEREREZBNMLImIiIjIJphYEhEREZFNMLEkIiIiIptgYklERERENsHEkoiIiIhsgoklEREREdkEE0siIiIisgkmlkRERERkE0wsiYiIiMgmmFgSERERkU0wsSQiIiIim2BiSUREREQ2wcSSiIiIiGyCiSURERER2QQTSyIiIiKyCSaWRERERGQTTCyJiIiIyCaYWBIRERGRTTCxJCIiIiKbYGJJRERERDbBxJKIiIiIbIKJJRERERHZBBNLIiIiIrIJJpZEREREZBNMLImIiIjIJphYEhEREZFNMLEkIiIiIptgYklERERENsHEkoiIiIhsgoklEREREdkEE0siIiIisgkmlkRERERkE0wsiYiIiMgmmFgSERERkU0wsSQiIiIim2BiSUREREQ2wcSSiIiIiGyCiSURERER2QQTSyIiIiKyCSaWRERERGQTTCyJiIiIyCaYWBIRERGRTTCxJCIiIiKbYGJJRERERDbBxJKIiIiIbIKJJRERERHZBBNLIiIiIrIJJpZEREREZBNMLImIiIjIJphYEhEREZFNMLEkIiIiIptgYklERERENsHEkoiIiIhsgoklEREREdkEE0siIiIisgkmlkRERERkE0wsiYiIiMgmmFgSERERkU0wsSQiIiIim2BiSUREREQ2wcSSiIiIiGyCiSURERER2QQTSyIiIiKyCSaWRERERGQTTCyJiIiIyCaYWBIRERGRTTCxJCIiIiKbYGJJRERERDbBxJKIiIiIbIKJJRERERHZBBNLIiIiIrIJJpZEREREZBNMLImIiIjIJphYEhEREZFNMLEkIiIiIpv4f2Od3zM8ALtJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 700x210 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mglearn.plots.plot_grid_search_overview()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this lecture we are going to talk about two such most commonly used **automated optimizations methods** from `scikit-learn`. \n",
    "\n",
    "- Exhaustive grid search: [`sklearn.model_selection.GridSearchCV`](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html)\n",
    "- Randomized search: [`sklearn.model_selection.RandomizedSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)\n",
    "\n",
    "The `CV` stands for cross-validation; these methods have built-in cross-validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Exhaustive grid search: [`sklearn.model_selection.GridSearchCV`](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import set_config\n",
    "\n",
    "set_config(display=\"diagram\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- For `GridSearchCV` we need\n",
    "    - an instantiated model or a pipeline\n",
    "    - a parameter grid: A user specifies a set of values for each hyperparameter. \n",
    "    - other optional arguments \n",
    "\n",
    "The method considers product of the sets and then evaluates each combination one by one.    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "pipe_svm = make_pipeline(StandardScaler(), SVC())\n",
    "\n",
    "param_grid = {\n",
    "    \"svc__gamma\": [0.001, 0.01, 0.1, 1.0, 10, 100],\n",
    "    \"svc__C\": [0.001, 0.01, 0.1, 1.0, 10, 100],\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(\n",
    "    pipe_svm, param_grid, cv=5, n_jobs=-1, return_train_score=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `n_jobs=-1`\n",
    "- Note the `n_jobs=-1` above.\n",
    "- Hyperparameter optimization can be done _in parallel_ for each of the configurations.\n",
    "- This is very useful when scaling up to large numbers of machines in the cloud."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The `__` syntax \n",
    "\n",
    "- Above: we have a nesting of transformers.\n",
    "- We can access the parameters of the \"inner\" objects by using __ to go \"deeper\":\n",
    "- `svc__gamma`: the `gamma` of the `svc` of the pipeline\n",
    "- `svc__C`: the `C` of the `svc` of the pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The `GridSearchCV` object above behaves like a classifier. We can call `fit`, `predict` or `score` on it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 {color: black;background-color: white;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 pre{padding: 0;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-toggleable {background-color: white;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-item {z-index: 1;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-parallel-item:only-child::after {width: 0;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-4fd30623-d1a9-4805-9922-7ebeed6f96f3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[(&#x27;standardscaler&#x27;, StandardScaler()),\n",
       "                                       (&#x27;svc&#x27;, SVC())]),\n",
       "             n_jobs=-1,\n",
       "             param_grid={&#x27;svc__C&#x27;: [0.001, 0.01, 0.1, 1.0, 10, 100],\n",
       "                         &#x27;svc__gamma&#x27;: [0.001, 0.01, 0.1, 1.0, 10, 100]},\n",
       "             return_train_score=True)</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"59a89ed3-470f-4588-99ef-37185ac1b289\" type=\"checkbox\" ><label for=\"59a89ed3-470f-4588-99ef-37185ac1b289\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[(&#x27;standardscaler&#x27;, StandardScaler()),\n",
       "                                       (&#x27;svc&#x27;, SVC())]),\n",
       "             n_jobs=-1,\n",
       "             param_grid={&#x27;svc__C&#x27;: [0.001, 0.01, 0.1, 1.0, 10, 100],\n",
       "                         &#x27;svc__gamma&#x27;: [0.001, 0.01, 0.1, 1.0, 10, 100]},\n",
       "             return_train_score=True)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"1f80b4a6-1c03-4e0c-9d70-803cc2c5d842\" type=\"checkbox\" ><label for=\"1f80b4a6-1c03-4e0c-9d70-803cc2c5d842\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"e8ffc576-609d-49f3-943a-427f84199f9e\" type=\"checkbox\" ><label for=\"e8ffc576-609d-49f3-943a-427f84199f9e\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC()</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[('standardscaler', StandardScaler()),\n",
       "                                       ('svc', SVC())]),\n",
       "             n_jobs=-1,\n",
       "             param_grid={'svc__C': [0.001, 0.01, 0.1, 1.0, 10, 100],\n",
       "                         'svc__gamma': [0.001, 0.01, 0.1, 1.0, 10, 100]},\n",
       "             return_train_score=True)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.fit(X_train, y_train) # all the work is done here\n",
    "grid_search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Fitting the `GridSearchCV` object \n",
    "- Searches for the best hyperparameter values\n",
    "- You can access the best score and the best hyperparameters using `best_score_` and `best_params_` attributes, respectively. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7439609253312309"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'svc__C': 1.0, 'svc__gamma': 0.1}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- It is often helpful to visualize results of all cross-validation experiments. \n",
    "- You can access this information using `cv_results_` attribute of a fitted `GridSearchCV` object.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean_fit_time</th>\n",
       "      <td>0.319606</td>\n",
       "      <td>0.321064</td>\n",
       "      <td>0.284587</td>\n",
       "      <td>0.236678</td>\n",
       "      <td>0.298828</td>\n",
       "      <td>0.276166</td>\n",
       "      <td>0.264388</td>\n",
       "      <td>0.230133</td>\n",
       "      <td>0.256758</td>\n",
       "      <td>0.251678</td>\n",
       "      <td>...</td>\n",
       "      <td>0.304863</td>\n",
       "      <td>0.383871</td>\n",
       "      <td>0.360031</td>\n",
       "      <td>0.528295</td>\n",
       "      <td>0.242858</td>\n",
       "      <td>0.348261</td>\n",
       "      <td>0.666763</td>\n",
       "      <td>0.299643</td>\n",
       "      <td>0.444609</td>\n",
       "      <td>0.372586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std_fit_time</th>\n",
       "      <td>0.023713</td>\n",
       "      <td>0.033357</td>\n",
       "      <td>0.036894</td>\n",
       "      <td>0.036106</td>\n",
       "      <td>0.026777</td>\n",
       "      <td>0.029502</td>\n",
       "      <td>0.038898</td>\n",
       "      <td>0.022421</td>\n",
       "      <td>0.029509</td>\n",
       "      <td>0.044128</td>\n",
       "      <td>...</td>\n",
       "      <td>0.086008</td>\n",
       "      <td>0.143387</td>\n",
       "      <td>0.132233</td>\n",
       "      <td>0.157329</td>\n",
       "      <td>0.04517</td>\n",
       "      <td>0.044605</td>\n",
       "      <td>0.123464</td>\n",
       "      <td>0.044458</td>\n",
       "      <td>0.131219</td>\n",
       "      <td>0.107677</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean_score_time</th>\n",
       "      <td>0.095841</td>\n",
       "      <td>0.106591</td>\n",
       "      <td>0.101623</td>\n",
       "      <td>0.085878</td>\n",
       "      <td>0.106827</td>\n",
       "      <td>0.096781</td>\n",
       "      <td>0.100339</td>\n",
       "      <td>0.082995</td>\n",
       "      <td>0.100254</td>\n",
       "      <td>0.08771</td>\n",
       "      <td>...</td>\n",
       "      <td>0.077875</td>\n",
       "      <td>0.124079</td>\n",
       "      <td>0.098897</td>\n",
       "      <td>0.12621</td>\n",
       "      <td>0.066413</td>\n",
       "      <td>0.067513</td>\n",
       "      <td>0.06585</td>\n",
       "      <td>0.097411</td>\n",
       "      <td>0.130393</td>\n",
       "      <td>0.147969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std_score_time</th>\n",
       "      <td>0.013952</td>\n",
       "      <td>0.016529</td>\n",
       "      <td>0.003895</td>\n",
       "      <td>0.010228</td>\n",
       "      <td>0.015524</td>\n",
       "      <td>0.013255</td>\n",
       "      <td>0.022026</td>\n",
       "      <td>0.009681</td>\n",
       "      <td>0.016962</td>\n",
       "      <td>0.009657</td>\n",
       "      <td>...</td>\n",
       "      <td>0.02544</td>\n",
       "      <td>0.039681</td>\n",
       "      <td>0.034097</td>\n",
       "      <td>0.04328</td>\n",
       "      <td>0.010017</td>\n",
       "      <td>0.019459</td>\n",
       "      <td>0.014379</td>\n",
       "      <td>0.03017</td>\n",
       "      <td>0.039727</td>\n",
       "      <td>0.066615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_svc__C</th>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>...</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_svc__gamma</th>\n",
       "      <td>0.001</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>10</td>\n",
       "      <td>100</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>10</td>\n",
       "      <td>100</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>10</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>params</th>\n",
       "      <td>{'svc__C': 0.001, 'svc__gamma': 0.001}</td>\n",
       "      <td>{'svc__C': 0.001, 'svc__gamma': 0.01}</td>\n",
       "      <td>{'svc__C': 0.001, 'svc__gamma': 0.1}</td>\n",
       "      <td>{'svc__C': 0.001, 'svc__gamma': 1.0}</td>\n",
       "      <td>{'svc__C': 0.001, 'svc__gamma': 10}</td>\n",
       "      <td>{'svc__C': 0.001, 'svc__gamma': 100}</td>\n",
       "      <td>{'svc__C': 0.01, 'svc__gamma': 0.001}</td>\n",
       "      <td>{'svc__C': 0.01, 'svc__gamma': 0.01}</td>\n",
       "      <td>{'svc__C': 0.01, 'svc__gamma': 0.1}</td>\n",
       "      <td>{'svc__C': 0.01, 'svc__gamma': 1.0}</td>\n",
       "      <td>...</td>\n",
       "      <td>{'svc__C': 10, 'svc__gamma': 0.1}</td>\n",
       "      <td>{'svc__C': 10, 'svc__gamma': 1.0}</td>\n",
       "      <td>{'svc__C': 10, 'svc__gamma': 10}</td>\n",
       "      <td>{'svc__C': 10, 'svc__gamma': 100}</td>\n",
       "      <td>{'svc__C': 100, 'svc__gamma': 0.001}</td>\n",
       "      <td>{'svc__C': 100, 'svc__gamma': 0.01}</td>\n",
       "      <td>{'svc__C': 100, 'svc__gamma': 0.1}</td>\n",
       "      <td>{'svc__C': 100, 'svc__gamma': 1.0}</td>\n",
       "      <td>{'svc__C': 100, 'svc__gamma': 10}</td>\n",
       "      <td>{'svc__C': 100, 'svc__gamma': 100}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split0_test_score</th>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>...</td>\n",
       "      <td>0.702786</td>\n",
       "      <td>0.671827</td>\n",
       "      <td>0.517028</td>\n",
       "      <td>0.504644</td>\n",
       "      <td>0.73065</td>\n",
       "      <td>0.73065</td>\n",
       "      <td>0.705882</td>\n",
       "      <td>0.671827</td>\n",
       "      <td>0.517028</td>\n",
       "      <td>0.504644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split1_test_score</th>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>...</td>\n",
       "      <td>0.767802</td>\n",
       "      <td>0.671827</td>\n",
       "      <td>0.510836</td>\n",
       "      <td>0.510836</td>\n",
       "      <td>0.708978</td>\n",
       "      <td>0.758514</td>\n",
       "      <td>0.76161</td>\n",
       "      <td>0.671827</td>\n",
       "      <td>0.510836</td>\n",
       "      <td>0.510836</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split2_test_score</th>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>...</td>\n",
       "      <td>0.693498</td>\n",
       "      <td>0.662539</td>\n",
       "      <td>0.517028</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.662539</td>\n",
       "      <td>0.71517</td>\n",
       "      <td>0.671827</td>\n",
       "      <td>0.662539</td>\n",
       "      <td>0.517028</td>\n",
       "      <td>0.50774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split3_test_score</th>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>...</td>\n",
       "      <td>0.736025</td>\n",
       "      <td>0.667702</td>\n",
       "      <td>0.515528</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.723602</td>\n",
       "      <td>0.720497</td>\n",
       "      <td>0.708075</td>\n",
       "      <td>0.667702</td>\n",
       "      <td>0.515528</td>\n",
       "      <td>0.509317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split4_test_score</th>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>...</td>\n",
       "      <td>0.745342</td>\n",
       "      <td>0.68323</td>\n",
       "      <td>0.515528</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.695652</td>\n",
       "      <td>0.73913</td>\n",
       "      <td>0.71118</td>\n",
       "      <td>0.68323</td>\n",
       "      <td>0.515528</td>\n",
       "      <td>0.509317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean_test_score</th>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>...</td>\n",
       "      <td>0.729091</td>\n",
       "      <td>0.671425</td>\n",
       "      <td>0.51519</td>\n",
       "      <td>0.508371</td>\n",
       "      <td>0.704284</td>\n",
       "      <td>0.732792</td>\n",
       "      <td>0.711715</td>\n",
       "      <td>0.671425</td>\n",
       "      <td>0.51519</td>\n",
       "      <td>0.508371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std_test_score</th>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>...</td>\n",
       "      <td>0.027457</td>\n",
       "      <td>0.006819</td>\n",
       "      <td>0.002278</td>\n",
       "      <td>0.002105</td>\n",
       "      <td>0.024115</td>\n",
       "      <td>0.015284</td>\n",
       "      <td>0.028734</td>\n",
       "      <td>0.006819</td>\n",
       "      <td>0.002278</td>\n",
       "      <td>0.002105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rank_test_score</th>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>15</td>\n",
       "      <td>18</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>11</td>\n",
       "      <td>15</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split0_train_score</th>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>...</td>\n",
       "      <td>0.923256</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.703876</td>\n",
       "      <td>0.80155</td>\n",
       "      <td>0.990698</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split1_train_score</th>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>...</td>\n",
       "      <td>0.91938</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.712403</td>\n",
       "      <td>0.796899</td>\n",
       "      <td>0.987597</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split2_train_score</th>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>...</td>\n",
       "      <td>0.925581</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.726357</td>\n",
       "      <td>0.813953</td>\n",
       "      <td>0.989147</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.999225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split3_train_score</th>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>...</td>\n",
       "      <td>0.923315</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.710302</td>\n",
       "      <td>0.797831</td>\n",
       "      <td>0.984508</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.999225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split4_train_score</th>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>...</td>\n",
       "      <td>0.915569</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.71495</td>\n",
       "      <td>0.793958</td>\n",
       "      <td>0.986057</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.999225</td>\n",
       "      <td>0.999225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean_train_score</th>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>...</td>\n",
       "      <td>0.92142</td>\n",
       "      <td>0.999535</td>\n",
       "      <td>0.999535</td>\n",
       "      <td>0.999535</td>\n",
       "      <td>0.713577</td>\n",
       "      <td>0.800838</td>\n",
       "      <td>0.987601</td>\n",
       "      <td>0.999535</td>\n",
       "      <td>0.999535</td>\n",
       "      <td>0.999535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std_train_score</th>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>...</td>\n",
       "      <td>0.00354</td>\n",
       "      <td>0.00038</td>\n",
       "      <td>0.00038</td>\n",
       "      <td>0.00038</td>\n",
       "      <td>0.007368</td>\n",
       "      <td>0.006992</td>\n",
       "      <td>0.002188</td>\n",
       "      <td>0.00038</td>\n",
       "      <td>0.00038</td>\n",
       "      <td>0.00038</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>22 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                        0   \\\n",
       "mean_fit_time                                     0.319606   \n",
       "std_fit_time                                      0.023713   \n",
       "mean_score_time                                   0.095841   \n",
       "std_score_time                                    0.013952   \n",
       "param_svc__C                                         0.001   \n",
       "param_svc__gamma                                     0.001   \n",
       "params              {'svc__C': 0.001, 'svc__gamma': 0.001}   \n",
       "split0_test_score                                  0.50774   \n",
       "split1_test_score                                  0.50774   \n",
       "split2_test_score                                  0.50774   \n",
       "split3_test_score                                 0.506211   \n",
       "split4_test_score                                 0.509317   \n",
       "mean_test_score                                    0.50775   \n",
       "std_test_score                                    0.000982   \n",
       "rank_test_score                                         21   \n",
       "split0_train_score                                0.507752   \n",
       "split1_train_score                                0.507752   \n",
       "split2_train_score                                0.507752   \n",
       "split3_train_score                                0.508133   \n",
       "split4_train_score                                0.507359   \n",
       "mean_train_score                                   0.50775   \n",
       "std_train_score                                   0.000245   \n",
       "\n",
       "                                                       1   \\\n",
       "mean_fit_time                                    0.321064   \n",
       "std_fit_time                                     0.033357   \n",
       "mean_score_time                                  0.106591   \n",
       "std_score_time                                   0.016529   \n",
       "param_svc__C                                        0.001   \n",
       "param_svc__gamma                                     0.01   \n",
       "params              {'svc__C': 0.001, 'svc__gamma': 0.01}   \n",
       "split0_test_score                                 0.50774   \n",
       "split1_test_score                                 0.50774   \n",
       "split2_test_score                                 0.50774   \n",
       "split3_test_score                                0.506211   \n",
       "split4_test_score                                0.509317   \n",
       "mean_test_score                                   0.50775   \n",
       "std_test_score                                   0.000982   \n",
       "rank_test_score                                        21   \n",
       "split0_train_score                               0.507752   \n",
       "split1_train_score                               0.507752   \n",
       "split2_train_score                               0.507752   \n",
       "split3_train_score                               0.508133   \n",
       "split4_train_score                               0.507359   \n",
       "mean_train_score                                  0.50775   \n",
       "std_train_score                                  0.000245   \n",
       "\n",
       "                                                      2   \\\n",
       "mean_fit_time                                   0.284587   \n",
       "std_fit_time                                    0.036894   \n",
       "mean_score_time                                 0.101623   \n",
       "std_score_time                                  0.003895   \n",
       "param_svc__C                                       0.001   \n",
       "param_svc__gamma                                     0.1   \n",
       "params              {'svc__C': 0.001, 'svc__gamma': 0.1}   \n",
       "split0_test_score                                0.50774   \n",
       "split1_test_score                                0.50774   \n",
       "split2_test_score                                0.50774   \n",
       "split3_test_score                               0.506211   \n",
       "split4_test_score                               0.509317   \n",
       "mean_test_score                                  0.50775   \n",
       "std_test_score                                  0.000982   \n",
       "rank_test_score                                       21   \n",
       "split0_train_score                              0.507752   \n",
       "split1_train_score                              0.507752   \n",
       "split2_train_score                              0.507752   \n",
       "split3_train_score                              0.508133   \n",
       "split4_train_score                              0.507359   \n",
       "mean_train_score                                 0.50775   \n",
       "std_train_score                                 0.000245   \n",
       "\n",
       "                                                      3   \\\n",
       "mean_fit_time                                   0.236678   \n",
       "std_fit_time                                    0.036106   \n",
       "mean_score_time                                 0.085878   \n",
       "std_score_time                                  0.010228   \n",
       "param_svc__C                                       0.001   \n",
       "param_svc__gamma                                     1.0   \n",
       "params              {'svc__C': 0.001, 'svc__gamma': 1.0}   \n",
       "split0_test_score                                0.50774   \n",
       "split1_test_score                                0.50774   \n",
       "split2_test_score                                0.50774   \n",
       "split3_test_score                               0.506211   \n",
       "split4_test_score                               0.509317   \n",
       "mean_test_score                                  0.50775   \n",
       "std_test_score                                  0.000982   \n",
       "rank_test_score                                       21   \n",
       "split0_train_score                              0.507752   \n",
       "split1_train_score                              0.507752   \n",
       "split2_train_score                              0.507752   \n",
       "split3_train_score                              0.508133   \n",
       "split4_train_score                              0.507359   \n",
       "mean_train_score                                 0.50775   \n",
       "std_train_score                                 0.000245   \n",
       "\n",
       "                                                     4   \\\n",
       "mean_fit_time                                  0.298828   \n",
       "std_fit_time                                   0.026777   \n",
       "mean_score_time                                0.106827   \n",
       "std_score_time                                 0.015524   \n",
       "param_svc__C                                      0.001   \n",
       "param_svc__gamma                                     10   \n",
       "params              {'svc__C': 0.001, 'svc__gamma': 10}   \n",
       "split0_test_score                               0.50774   \n",
       "split1_test_score                               0.50774   \n",
       "split2_test_score                               0.50774   \n",
       "split3_test_score                              0.506211   \n",
       "split4_test_score                              0.509317   \n",
       "mean_test_score                                 0.50775   \n",
       "std_test_score                                 0.000982   \n",
       "rank_test_score                                      21   \n",
       "split0_train_score                             0.507752   \n",
       "split1_train_score                             0.507752   \n",
       "split2_train_score                             0.507752   \n",
       "split3_train_score                             0.508133   \n",
       "split4_train_score                             0.507359   \n",
       "mean_train_score                                0.50775   \n",
       "std_train_score                                0.000245   \n",
       "\n",
       "                                                      5   \\\n",
       "mean_fit_time                                   0.276166   \n",
       "std_fit_time                                    0.029502   \n",
       "mean_score_time                                 0.096781   \n",
       "std_score_time                                  0.013255   \n",
       "param_svc__C                                       0.001   \n",
       "param_svc__gamma                                     100   \n",
       "params              {'svc__C': 0.001, 'svc__gamma': 100}   \n",
       "split0_test_score                                0.50774   \n",
       "split1_test_score                                0.50774   \n",
       "split2_test_score                                0.50774   \n",
       "split3_test_score                               0.506211   \n",
       "split4_test_score                               0.509317   \n",
       "mean_test_score                                  0.50775   \n",
       "std_test_score                                  0.000982   \n",
       "rank_test_score                                       21   \n",
       "split0_train_score                              0.507752   \n",
       "split1_train_score                              0.507752   \n",
       "split2_train_score                              0.507752   \n",
       "split3_train_score                              0.508133   \n",
       "split4_train_score                              0.507359   \n",
       "mean_train_score                                 0.50775   \n",
       "std_train_score                                 0.000245   \n",
       "\n",
       "                                                       6   \\\n",
       "mean_fit_time                                    0.264388   \n",
       "std_fit_time                                     0.038898   \n",
       "mean_score_time                                  0.100339   \n",
       "std_score_time                                   0.022026   \n",
       "param_svc__C                                         0.01   \n",
       "param_svc__gamma                                    0.001   \n",
       "params              {'svc__C': 0.01, 'svc__gamma': 0.001}   \n",
       "split0_test_score                                 0.50774   \n",
       "split1_test_score                                 0.50774   \n",
       "split2_test_score                                 0.50774   \n",
       "split3_test_score                                0.506211   \n",
       "split4_test_score                                0.509317   \n",
       "mean_test_score                                   0.50775   \n",
       "std_test_score                                   0.000982   \n",
       "rank_test_score                                        21   \n",
       "split0_train_score                               0.507752   \n",
       "split1_train_score                               0.507752   \n",
       "split2_train_score                               0.507752   \n",
       "split3_train_score                               0.508133   \n",
       "split4_train_score                               0.507359   \n",
       "mean_train_score                                  0.50775   \n",
       "std_train_score                                  0.000245   \n",
       "\n",
       "                                                      7   \\\n",
       "mean_fit_time                                   0.230133   \n",
       "std_fit_time                                    0.022421   \n",
       "mean_score_time                                 0.082995   \n",
       "std_score_time                                  0.009681   \n",
       "param_svc__C                                        0.01   \n",
       "param_svc__gamma                                    0.01   \n",
       "params              {'svc__C': 0.01, 'svc__gamma': 0.01}   \n",
       "split0_test_score                                0.50774   \n",
       "split1_test_score                                0.50774   \n",
       "split2_test_score                                0.50774   \n",
       "split3_test_score                               0.506211   \n",
       "split4_test_score                               0.509317   \n",
       "mean_test_score                                  0.50775   \n",
       "std_test_score                                  0.000982   \n",
       "rank_test_score                                       21   \n",
       "split0_train_score                              0.507752   \n",
       "split1_train_score                              0.507752   \n",
       "split2_train_score                              0.507752   \n",
       "split3_train_score                              0.508133   \n",
       "split4_train_score                              0.507359   \n",
       "mean_train_score                                 0.50775   \n",
       "std_train_score                                 0.000245   \n",
       "\n",
       "                                                     8   \\\n",
       "mean_fit_time                                  0.256758   \n",
       "std_fit_time                                   0.029509   \n",
       "mean_score_time                                0.100254   \n",
       "std_score_time                                 0.016962   \n",
       "param_svc__C                                       0.01   \n",
       "param_svc__gamma                                    0.1   \n",
       "params              {'svc__C': 0.01, 'svc__gamma': 0.1}   \n",
       "split0_test_score                               0.50774   \n",
       "split1_test_score                               0.50774   \n",
       "split2_test_score                               0.50774   \n",
       "split3_test_score                              0.506211   \n",
       "split4_test_score                              0.509317   \n",
       "mean_test_score                                 0.50775   \n",
       "std_test_score                                 0.000982   \n",
       "rank_test_score                                      21   \n",
       "split0_train_score                             0.507752   \n",
       "split1_train_score                             0.507752   \n",
       "split2_train_score                             0.507752   \n",
       "split3_train_score                             0.508133   \n",
       "split4_train_score                             0.507359   \n",
       "mean_train_score                                0.50775   \n",
       "std_train_score                                0.000245   \n",
       "\n",
       "                                                     9   ...  \\\n",
       "mean_fit_time                                  0.251678  ...   \n",
       "std_fit_time                                   0.044128  ...   \n",
       "mean_score_time                                 0.08771  ...   \n",
       "std_score_time                                 0.009657  ...   \n",
       "param_svc__C                                       0.01  ...   \n",
       "param_svc__gamma                                    1.0  ...   \n",
       "params              {'svc__C': 0.01, 'svc__gamma': 1.0}  ...   \n",
       "split0_test_score                               0.50774  ...   \n",
       "split1_test_score                               0.50774  ...   \n",
       "split2_test_score                               0.50774  ...   \n",
       "split3_test_score                              0.506211  ...   \n",
       "split4_test_score                              0.509317  ...   \n",
       "mean_test_score                                 0.50775  ...   \n",
       "std_test_score                                 0.000982  ...   \n",
       "rank_test_score                                      21  ...   \n",
       "split0_train_score                             0.507752  ...   \n",
       "split1_train_score                             0.507752  ...   \n",
       "split2_train_score                             0.507752  ...   \n",
       "split3_train_score                             0.508133  ...   \n",
       "split4_train_score                             0.507359  ...   \n",
       "mean_train_score                                0.50775  ...   \n",
       "std_train_score                                0.000245  ...   \n",
       "\n",
       "                                                   26  \\\n",
       "mean_fit_time                                0.304863   \n",
       "std_fit_time                                 0.086008   \n",
       "mean_score_time                              0.077875   \n",
       "std_score_time                                0.02544   \n",
       "param_svc__C                                       10   \n",
       "param_svc__gamma                                  0.1   \n",
       "params              {'svc__C': 10, 'svc__gamma': 0.1}   \n",
       "split0_test_score                            0.702786   \n",
       "split1_test_score                            0.767802   \n",
       "split2_test_score                            0.693498   \n",
       "split3_test_score                            0.736025   \n",
       "split4_test_score                            0.745342   \n",
       "mean_test_score                              0.729091   \n",
       "std_test_score                               0.027457   \n",
       "rank_test_score                                     3   \n",
       "split0_train_score                           0.923256   \n",
       "split1_train_score                            0.91938   \n",
       "split2_train_score                           0.925581   \n",
       "split3_train_score                           0.923315   \n",
       "split4_train_score                           0.915569   \n",
       "mean_train_score                              0.92142   \n",
       "std_train_score                               0.00354   \n",
       "\n",
       "                                                   27  \\\n",
       "mean_fit_time                                0.383871   \n",
       "std_fit_time                                 0.143387   \n",
       "mean_score_time                              0.124079   \n",
       "std_score_time                               0.039681   \n",
       "param_svc__C                                       10   \n",
       "param_svc__gamma                                  1.0   \n",
       "params              {'svc__C': 10, 'svc__gamma': 1.0}   \n",
       "split0_test_score                            0.671827   \n",
       "split1_test_score                            0.671827   \n",
       "split2_test_score                            0.662539   \n",
       "split3_test_score                            0.667702   \n",
       "split4_test_score                             0.68323   \n",
       "mean_test_score                              0.671425   \n",
       "std_test_score                               0.006819   \n",
       "rank_test_score                                    11   \n",
       "split0_train_score                                1.0   \n",
       "split1_train_score                                1.0   \n",
       "split2_train_score                           0.999225   \n",
       "split3_train_score                           0.999225   \n",
       "split4_train_score                           0.999225   \n",
       "mean_train_score                             0.999535   \n",
       "std_train_score                               0.00038   \n",
       "\n",
       "                                                  28  \\\n",
       "mean_fit_time                               0.360031   \n",
       "std_fit_time                                0.132233   \n",
       "mean_score_time                             0.098897   \n",
       "std_score_time                              0.034097   \n",
       "param_svc__C                                      10   \n",
       "param_svc__gamma                                  10   \n",
       "params              {'svc__C': 10, 'svc__gamma': 10}   \n",
       "split0_test_score                           0.517028   \n",
       "split1_test_score                           0.510836   \n",
       "split2_test_score                           0.517028   \n",
       "split3_test_score                           0.515528   \n",
       "split4_test_score                           0.515528   \n",
       "mean_test_score                              0.51519   \n",
       "std_test_score                              0.002278   \n",
       "rank_test_score                                   15   \n",
       "split0_train_score                               1.0   \n",
       "split1_train_score                               1.0   \n",
       "split2_train_score                          0.999225   \n",
       "split3_train_score                          0.999225   \n",
       "split4_train_score                          0.999225   \n",
       "mean_train_score                            0.999535   \n",
       "std_train_score                              0.00038   \n",
       "\n",
       "                                                   29  \\\n",
       "mean_fit_time                                0.528295   \n",
       "std_fit_time                                 0.157329   \n",
       "mean_score_time                               0.12621   \n",
       "std_score_time                                0.04328   \n",
       "param_svc__C                                       10   \n",
       "param_svc__gamma                                  100   \n",
       "params              {'svc__C': 10, 'svc__gamma': 100}   \n",
       "split0_test_score                            0.504644   \n",
       "split1_test_score                            0.510836   \n",
       "split2_test_score                             0.50774   \n",
       "split3_test_score                            0.509317   \n",
       "split4_test_score                            0.509317   \n",
       "mean_test_score                              0.508371   \n",
       "std_test_score                               0.002105   \n",
       "rank_test_score                                    18   \n",
       "split0_train_score                                1.0   \n",
       "split1_train_score                                1.0   \n",
       "split2_train_score                           0.999225   \n",
       "split3_train_score                           0.999225   \n",
       "split4_train_score                           0.999225   \n",
       "mean_train_score                             0.999535   \n",
       "std_train_score                               0.00038   \n",
       "\n",
       "                                                      30  \\\n",
       "mean_fit_time                                   0.242858   \n",
       "std_fit_time                                     0.04517   \n",
       "mean_score_time                                 0.066413   \n",
       "std_score_time                                  0.010017   \n",
       "param_svc__C                                         100   \n",
       "param_svc__gamma                                   0.001   \n",
       "params              {'svc__C': 100, 'svc__gamma': 0.001}   \n",
       "split0_test_score                                0.73065   \n",
       "split1_test_score                               0.708978   \n",
       "split2_test_score                               0.662539   \n",
       "split3_test_score                               0.723602   \n",
       "split4_test_score                               0.695652   \n",
       "mean_test_score                                 0.704284   \n",
       "std_test_score                                  0.024115   \n",
       "rank_test_score                                        6   \n",
       "split0_train_score                              0.703876   \n",
       "split1_train_score                              0.712403   \n",
       "split2_train_score                              0.726357   \n",
       "split3_train_score                              0.710302   \n",
       "split4_train_score                               0.71495   \n",
       "mean_train_score                                0.713577   \n",
       "std_train_score                                 0.007368   \n",
       "\n",
       "                                                     31  \\\n",
       "mean_fit_time                                  0.348261   \n",
       "std_fit_time                                   0.044605   \n",
       "mean_score_time                                0.067513   \n",
       "std_score_time                                 0.019459   \n",
       "param_svc__C                                        100   \n",
       "param_svc__gamma                                   0.01   \n",
       "params              {'svc__C': 100, 'svc__gamma': 0.01}   \n",
       "split0_test_score                               0.73065   \n",
       "split1_test_score                              0.758514   \n",
       "split2_test_score                               0.71517   \n",
       "split3_test_score                              0.720497   \n",
       "split4_test_score                               0.73913   \n",
       "mean_test_score                                0.732792   \n",
       "std_test_score                                 0.015284   \n",
       "rank_test_score                                       2   \n",
       "split0_train_score                              0.80155   \n",
       "split1_train_score                             0.796899   \n",
       "split2_train_score                             0.813953   \n",
       "split3_train_score                             0.797831   \n",
       "split4_train_score                             0.793958   \n",
       "mean_train_score                               0.800838   \n",
       "std_train_score                                0.006992   \n",
       "\n",
       "                                                    32  \\\n",
       "mean_fit_time                                 0.666763   \n",
       "std_fit_time                                  0.123464   \n",
       "mean_score_time                                0.06585   \n",
       "std_score_time                                0.014379   \n",
       "param_svc__C                                       100   \n",
       "param_svc__gamma                                   0.1   \n",
       "params              {'svc__C': 100, 'svc__gamma': 0.1}   \n",
       "split0_test_score                             0.705882   \n",
       "split1_test_score                              0.76161   \n",
       "split2_test_score                             0.671827   \n",
       "split3_test_score                             0.708075   \n",
       "split4_test_score                              0.71118   \n",
       "mean_test_score                               0.711715   \n",
       "std_test_score                                0.028734   \n",
       "rank_test_score                                      5   \n",
       "split0_train_score                            0.990698   \n",
       "split1_train_score                            0.987597   \n",
       "split2_train_score                            0.989147   \n",
       "split3_train_score                            0.984508   \n",
       "split4_train_score                            0.986057   \n",
       "mean_train_score                              0.987601   \n",
       "std_train_score                               0.002188   \n",
       "\n",
       "                                                    33  \\\n",
       "mean_fit_time                                 0.299643   \n",
       "std_fit_time                                  0.044458   \n",
       "mean_score_time                               0.097411   \n",
       "std_score_time                                 0.03017   \n",
       "param_svc__C                                       100   \n",
       "param_svc__gamma                                   1.0   \n",
       "params              {'svc__C': 100, 'svc__gamma': 1.0}   \n",
       "split0_test_score                             0.671827   \n",
       "split1_test_score                             0.671827   \n",
       "split2_test_score                             0.662539   \n",
       "split3_test_score                             0.667702   \n",
       "split4_test_score                              0.68323   \n",
       "mean_test_score                               0.671425   \n",
       "std_test_score                                0.006819   \n",
       "rank_test_score                                     11   \n",
       "split0_train_score                                 1.0   \n",
       "split1_train_score                                 1.0   \n",
       "split2_train_score                            0.999225   \n",
       "split3_train_score                            0.999225   \n",
       "split4_train_score                            0.999225   \n",
       "mean_train_score                              0.999535   \n",
       "std_train_score                                0.00038   \n",
       "\n",
       "                                                   34  \\\n",
       "mean_fit_time                                0.444609   \n",
       "std_fit_time                                 0.131219   \n",
       "mean_score_time                              0.130393   \n",
       "std_score_time                               0.039727   \n",
       "param_svc__C                                      100   \n",
       "param_svc__gamma                                   10   \n",
       "params              {'svc__C': 100, 'svc__gamma': 10}   \n",
       "split0_test_score                            0.517028   \n",
       "split1_test_score                            0.510836   \n",
       "split2_test_score                            0.517028   \n",
       "split3_test_score                            0.515528   \n",
       "split4_test_score                            0.515528   \n",
       "mean_test_score                               0.51519   \n",
       "std_test_score                               0.002278   \n",
       "rank_test_score                                    15   \n",
       "split0_train_score                                1.0   \n",
       "split1_train_score                                1.0   \n",
       "split2_train_score                           0.999225   \n",
       "split3_train_score                           0.999225   \n",
       "split4_train_score                           0.999225   \n",
       "mean_train_score                             0.999535   \n",
       "std_train_score                               0.00038   \n",
       "\n",
       "                                                    35  \n",
       "mean_fit_time                                 0.372586  \n",
       "std_fit_time                                  0.107677  \n",
       "mean_score_time                               0.147969  \n",
       "std_score_time                                0.066615  \n",
       "param_svc__C                                       100  \n",
       "param_svc__gamma                                   100  \n",
       "params              {'svc__C': 100, 'svc__gamma': 100}  \n",
       "split0_test_score                             0.504644  \n",
       "split1_test_score                             0.510836  \n",
       "split2_test_score                              0.50774  \n",
       "split3_test_score                             0.509317  \n",
       "split4_test_score                             0.509317  \n",
       "mean_test_score                               0.508371  \n",
       "std_test_score                                0.002105  \n",
       "rank_test_score                                     18  \n",
       "split0_train_score                                 1.0  \n",
       "split1_train_score                                 1.0  \n",
       "split2_train_score                            0.999225  \n",
       "split3_train_score                            0.999225  \n",
       "split4_train_score                            0.999225  \n",
       "mean_train_score                              0.999535  \n",
       "std_train_score                                0.00038  \n",
       "\n",
       "[22 rows x 36 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = pd.DataFrame(grid_search.cv_results_)\n",
    "results.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>rank_test_score</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>...</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean_fit_time</th>\n",
       "      <td>0.202746</td>\n",
       "      <td>0.348261</td>\n",
       "      <td>0.304863</td>\n",
       "      <td>0.223569</td>\n",
       "      <td>0.666763</td>\n",
       "      <td>0.242858</td>\n",
       "      <td>0.235208</td>\n",
       "      <td>0.227487</td>\n",
       "      <td>0.240102</td>\n",
       "      <td>0.211701</td>\n",
       "      <td>...</td>\n",
       "      <td>0.256758</td>\n",
       "      <td>0.230133</td>\n",
       "      <td>0.264388</td>\n",
       "      <td>0.276166</td>\n",
       "      <td>0.298828</td>\n",
       "      <td>0.236678</td>\n",
       "      <td>0.284587</td>\n",
       "      <td>0.321064</td>\n",
       "      <td>0.284478</td>\n",
       "      <td>0.340115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std_fit_time</th>\n",
       "      <td>0.027272</td>\n",
       "      <td>0.044605</td>\n",
       "      <td>0.086008</td>\n",
       "      <td>0.034292</td>\n",
       "      <td>0.123464</td>\n",
       "      <td>0.04517</td>\n",
       "      <td>0.041942</td>\n",
       "      <td>0.031567</td>\n",
       "      <td>0.039124</td>\n",
       "      <td>0.032375</td>\n",
       "      <td>...</td>\n",
       "      <td>0.029509</td>\n",
       "      <td>0.022421</td>\n",
       "      <td>0.038898</td>\n",
       "      <td>0.029502</td>\n",
       "      <td>0.026777</td>\n",
       "      <td>0.036106</td>\n",
       "      <td>0.036894</td>\n",
       "      <td>0.033357</td>\n",
       "      <td>0.041944</td>\n",
       "      <td>0.045918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean_score_time</th>\n",
       "      <td>0.061584</td>\n",
       "      <td>0.067513</td>\n",
       "      <td>0.077875</td>\n",
       "      <td>0.067093</td>\n",
       "      <td>0.06585</td>\n",
       "      <td>0.066413</td>\n",
       "      <td>0.08389</td>\n",
       "      <td>0.075283</td>\n",
       "      <td>0.087023</td>\n",
       "      <td>0.069995</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100254</td>\n",
       "      <td>0.082995</td>\n",
       "      <td>0.100339</td>\n",
       "      <td>0.096781</td>\n",
       "      <td>0.106827</td>\n",
       "      <td>0.085878</td>\n",
       "      <td>0.101623</td>\n",
       "      <td>0.106591</td>\n",
       "      <td>0.088089</td>\n",
       "      <td>0.107322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std_score_time</th>\n",
       "      <td>0.008265</td>\n",
       "      <td>0.019459</td>\n",
       "      <td>0.02544</td>\n",
       "      <td>0.010337</td>\n",
       "      <td>0.014379</td>\n",
       "      <td>0.010017</td>\n",
       "      <td>0.007516</td>\n",
       "      <td>0.00862</td>\n",
       "      <td>0.014581</td>\n",
       "      <td>0.012089</td>\n",
       "      <td>...</td>\n",
       "      <td>0.016962</td>\n",
       "      <td>0.009681</td>\n",
       "      <td>0.022026</td>\n",
       "      <td>0.013255</td>\n",
       "      <td>0.015524</td>\n",
       "      <td>0.010228</td>\n",
       "      <td>0.003895</td>\n",
       "      <td>0.016529</td>\n",
       "      <td>0.014612</td>\n",
       "      <td>0.016258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_svc__C</th>\n",
       "      <td>1.0</td>\n",
       "      <td>100</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_svc__gamma</th>\n",
       "      <td>0.1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.001</td>\n",
       "      <td>...</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.001</td>\n",
       "      <td>100</td>\n",
       "      <td>10</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>10</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>params</th>\n",
       "      <td>{'svc__C': 1.0, 'svc__gamma': 0.1}</td>\n",
       "      <td>{'svc__C': 100, 'svc__gamma': 0.01}</td>\n",
       "      <td>{'svc__C': 10, 'svc__gamma': 0.1}</td>\n",
       "      <td>{'svc__C': 10, 'svc__gamma': 0.01}</td>\n",
       "      <td>{'svc__C': 100, 'svc__gamma': 0.1}</td>\n",
       "      <td>{'svc__C': 100, 'svc__gamma': 0.001}</td>\n",
       "      <td>{'svc__C': 0.1, 'svc__gamma': 0.1}</td>\n",
       "      <td>{'svc__C': 1.0, 'svc__gamma': 0.01}</td>\n",
       "      <td>{'svc__C': 0.1, 'svc__gamma': 0.01}</td>\n",
       "      <td>{'svc__C': 10, 'svc__gamma': 0.001}</td>\n",
       "      <td>...</td>\n",
       "      <td>{'svc__C': 0.01, 'svc__gamma': 0.1}</td>\n",
       "      <td>{'svc__C': 0.01, 'svc__gamma': 0.01}</td>\n",
       "      <td>{'svc__C': 0.01, 'svc__gamma': 0.001}</td>\n",
       "      <td>{'svc__C': 0.001, 'svc__gamma': 100}</td>\n",
       "      <td>{'svc__C': 0.001, 'svc__gamma': 10}</td>\n",
       "      <td>{'svc__C': 0.001, 'svc__gamma': 1.0}</td>\n",
       "      <td>{'svc__C': 0.001, 'svc__gamma': 0.1}</td>\n",
       "      <td>{'svc__C': 0.001, 'svc__gamma': 0.01}</td>\n",
       "      <td>{'svc__C': 0.1, 'svc__gamma': 10}</td>\n",
       "      <td>{'svc__C': 0.1, 'svc__gamma': 100}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split0_test_score</th>\n",
       "      <td>0.755418</td>\n",
       "      <td>0.73065</td>\n",
       "      <td>0.702786</td>\n",
       "      <td>0.739938</td>\n",
       "      <td>0.705882</td>\n",
       "      <td>0.73065</td>\n",
       "      <td>0.705882</td>\n",
       "      <td>0.702786</td>\n",
       "      <td>0.693498</td>\n",
       "      <td>0.69969</td>\n",
       "      <td>...</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split1_test_score</th>\n",
       "      <td>0.755418</td>\n",
       "      <td>0.758514</td>\n",
       "      <td>0.767802</td>\n",
       "      <td>0.733746</td>\n",
       "      <td>0.76161</td>\n",
       "      <td>0.708978</td>\n",
       "      <td>0.718266</td>\n",
       "      <td>0.705882</td>\n",
       "      <td>0.702786</td>\n",
       "      <td>0.674923</td>\n",
       "      <td>...</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split2_test_score</th>\n",
       "      <td>0.712074</td>\n",
       "      <td>0.71517</td>\n",
       "      <td>0.693498</td>\n",
       "      <td>0.696594</td>\n",
       "      <td>0.671827</td>\n",
       "      <td>0.662539</td>\n",
       "      <td>0.690402</td>\n",
       "      <td>0.659443</td>\n",
       "      <td>0.653251</td>\n",
       "      <td>0.653251</td>\n",
       "      <td>...</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "      <td>0.50774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split3_test_score</th>\n",
       "      <td>0.754658</td>\n",
       "      <td>0.720497</td>\n",
       "      <td>0.736025</td>\n",
       "      <td>0.720497</td>\n",
       "      <td>0.708075</td>\n",
       "      <td>0.723602</td>\n",
       "      <td>0.692547</td>\n",
       "      <td>0.726708</td>\n",
       "      <td>0.664596</td>\n",
       "      <td>0.680124</td>\n",
       "      <td>...</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "      <td>0.506211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split4_test_score</th>\n",
       "      <td>0.742236</td>\n",
       "      <td>0.73913</td>\n",
       "      <td>0.745342</td>\n",
       "      <td>0.71118</td>\n",
       "      <td>0.71118</td>\n",
       "      <td>0.695652</td>\n",
       "      <td>0.708075</td>\n",
       "      <td>0.692547</td>\n",
       "      <td>0.680124</td>\n",
       "      <td>0.68323</td>\n",
       "      <td>...</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "      <td>0.509317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean_test_score</th>\n",
       "      <td>0.743961</td>\n",
       "      <td>0.732792</td>\n",
       "      <td>0.729091</td>\n",
       "      <td>0.720391</td>\n",
       "      <td>0.711715</td>\n",
       "      <td>0.704284</td>\n",
       "      <td>0.703034</td>\n",
       "      <td>0.697473</td>\n",
       "      <td>0.678851</td>\n",
       "      <td>0.678244</td>\n",
       "      <td>...</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std_test_score</th>\n",
       "      <td>0.016713</td>\n",
       "      <td>0.015284</td>\n",
       "      <td>0.027457</td>\n",
       "      <td>0.015566</td>\n",
       "      <td>0.028734</td>\n",
       "      <td>0.024115</td>\n",
       "      <td>0.010345</td>\n",
       "      <td>0.022019</td>\n",
       "      <td>0.018153</td>\n",
       "      <td>0.014994</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "      <td>0.000982</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split0_train_score</th>\n",
       "      <td>0.827132</td>\n",
       "      <td>0.80155</td>\n",
       "      <td>0.923256</td>\n",
       "      <td>0.751938</td>\n",
       "      <td>0.990698</td>\n",
       "      <td>0.703876</td>\n",
       "      <td>0.730233</td>\n",
       "      <td>0.708527</td>\n",
       "      <td>0.675194</td>\n",
       "      <td>0.67907</td>\n",
       "      <td>...</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split1_train_score</th>\n",
       "      <td>0.829457</td>\n",
       "      <td>0.796899</td>\n",
       "      <td>0.91938</td>\n",
       "      <td>0.757364</td>\n",
       "      <td>0.987597</td>\n",
       "      <td>0.712403</td>\n",
       "      <td>0.724806</td>\n",
       "      <td>0.700775</td>\n",
       "      <td>0.676744</td>\n",
       "      <td>0.682946</td>\n",
       "      <td>...</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split2_train_score</th>\n",
       "      <td>0.83876</td>\n",
       "      <td>0.813953</td>\n",
       "      <td>0.925581</td>\n",
       "      <td>0.760465</td>\n",
       "      <td>0.989147</td>\n",
       "      <td>0.726357</td>\n",
       "      <td>0.737984</td>\n",
       "      <td>0.723256</td>\n",
       "      <td>0.683721</td>\n",
       "      <td>0.694574</td>\n",
       "      <td>...</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "      <td>0.507752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split3_train_score</th>\n",
       "      <td>0.831913</td>\n",
       "      <td>0.797831</td>\n",
       "      <td>0.923315</td>\n",
       "      <td>0.759876</td>\n",
       "      <td>0.984508</td>\n",
       "      <td>0.710302</td>\n",
       "      <td>0.729667</td>\n",
       "      <td>0.701007</td>\n",
       "      <td>0.681642</td>\n",
       "      <td>0.688613</td>\n",
       "      <td>...</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "      <td>0.508133</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>split4_train_score</th>\n",
       "      <td>0.832688</td>\n",
       "      <td>0.793958</td>\n",
       "      <td>0.915569</td>\n",
       "      <td>0.75213</td>\n",
       "      <td>0.986057</td>\n",
       "      <td>0.71495</td>\n",
       "      <td>0.725019</td>\n",
       "      <td>0.718048</td>\n",
       "      <td>0.680868</td>\n",
       "      <td>0.687064</td>\n",
       "      <td>...</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "      <td>0.507359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean_train_score</th>\n",
       "      <td>0.83199</td>\n",
       "      <td>0.800838</td>\n",
       "      <td>0.92142</td>\n",
       "      <td>0.756355</td>\n",
       "      <td>0.987601</td>\n",
       "      <td>0.713577</td>\n",
       "      <td>0.729542</td>\n",
       "      <td>0.710323</td>\n",
       "      <td>0.679634</td>\n",
       "      <td>0.686453</td>\n",
       "      <td>...</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std_train_score</th>\n",
       "      <td>0.003907</td>\n",
       "      <td>0.006992</td>\n",
       "      <td>0.00354</td>\n",
       "      <td>0.003679</td>\n",
       "      <td>0.002188</td>\n",
       "      <td>0.007368</td>\n",
       "      <td>0.004789</td>\n",
       "      <td>0.009034</td>\n",
       "      <td>0.003172</td>\n",
       "      <td>0.00525</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "      <td>0.000245</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>21 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "rank_test_score                                     1   \\\n",
       "mean_fit_time                                 0.202746   \n",
       "std_fit_time                                  0.027272   \n",
       "mean_score_time                               0.061584   \n",
       "std_score_time                                0.008265   \n",
       "param_svc__C                                       1.0   \n",
       "param_svc__gamma                                   0.1   \n",
       "params              {'svc__C': 1.0, 'svc__gamma': 0.1}   \n",
       "split0_test_score                             0.755418   \n",
       "split1_test_score                             0.755418   \n",
       "split2_test_score                             0.712074   \n",
       "split3_test_score                             0.754658   \n",
       "split4_test_score                             0.742236   \n",
       "mean_test_score                               0.743961   \n",
       "std_test_score                                0.016713   \n",
       "split0_train_score                            0.827132   \n",
       "split1_train_score                            0.829457   \n",
       "split2_train_score                             0.83876   \n",
       "split3_train_score                            0.831913   \n",
       "split4_train_score                            0.832688   \n",
       "mean_train_score                               0.83199   \n",
       "std_train_score                               0.003907   \n",
       "\n",
       "rank_test_score                                      2   \\\n",
       "mean_fit_time                                  0.348261   \n",
       "std_fit_time                                   0.044605   \n",
       "mean_score_time                                0.067513   \n",
       "std_score_time                                 0.019459   \n",
       "param_svc__C                                        100   \n",
       "param_svc__gamma                                   0.01   \n",
       "params              {'svc__C': 100, 'svc__gamma': 0.01}   \n",
       "split0_test_score                               0.73065   \n",
       "split1_test_score                              0.758514   \n",
       "split2_test_score                               0.71517   \n",
       "split3_test_score                              0.720497   \n",
       "split4_test_score                               0.73913   \n",
       "mean_test_score                                0.732792   \n",
       "std_test_score                                 0.015284   \n",
       "split0_train_score                              0.80155   \n",
       "split1_train_score                             0.796899   \n",
       "split2_train_score                             0.813953   \n",
       "split3_train_score                             0.797831   \n",
       "split4_train_score                             0.793958   \n",
       "mean_train_score                               0.800838   \n",
       "std_train_score                                0.006992   \n",
       "\n",
       "rank_test_score                                    3   \\\n",
       "mean_fit_time                                0.304863   \n",
       "std_fit_time                                 0.086008   \n",
       "mean_score_time                              0.077875   \n",
       "std_score_time                                0.02544   \n",
       "param_svc__C                                       10   \n",
       "param_svc__gamma                                  0.1   \n",
       "params              {'svc__C': 10, 'svc__gamma': 0.1}   \n",
       "split0_test_score                            0.702786   \n",
       "split1_test_score                            0.767802   \n",
       "split2_test_score                            0.693498   \n",
       "split3_test_score                            0.736025   \n",
       "split4_test_score                            0.745342   \n",
       "mean_test_score                              0.729091   \n",
       "std_test_score                               0.027457   \n",
       "split0_train_score                           0.923256   \n",
       "split1_train_score                            0.91938   \n",
       "split2_train_score                           0.925581   \n",
       "split3_train_score                           0.923315   \n",
       "split4_train_score                           0.915569   \n",
       "mean_train_score                              0.92142   \n",
       "std_train_score                               0.00354   \n",
       "\n",
       "rank_test_score                                     4   \\\n",
       "mean_fit_time                                 0.223569   \n",
       "std_fit_time                                  0.034292   \n",
       "mean_score_time                               0.067093   \n",
       "std_score_time                                0.010337   \n",
       "param_svc__C                                        10   \n",
       "param_svc__gamma                                  0.01   \n",
       "params              {'svc__C': 10, 'svc__gamma': 0.01}   \n",
       "split0_test_score                             0.739938   \n",
       "split1_test_score                             0.733746   \n",
       "split2_test_score                             0.696594   \n",
       "split3_test_score                             0.720497   \n",
       "split4_test_score                              0.71118   \n",
       "mean_test_score                               0.720391   \n",
       "std_test_score                                0.015566   \n",
       "split0_train_score                            0.751938   \n",
       "split1_train_score                            0.757364   \n",
       "split2_train_score                            0.760465   \n",
       "split3_train_score                            0.759876   \n",
       "split4_train_score                             0.75213   \n",
       "mean_train_score                              0.756355   \n",
       "std_train_score                               0.003679   \n",
       "\n",
       "rank_test_score                                     5   \\\n",
       "mean_fit_time                                 0.666763   \n",
       "std_fit_time                                  0.123464   \n",
       "mean_score_time                                0.06585   \n",
       "std_score_time                                0.014379   \n",
       "param_svc__C                                       100   \n",
       "param_svc__gamma                                   0.1   \n",
       "params              {'svc__C': 100, 'svc__gamma': 0.1}   \n",
       "split0_test_score                             0.705882   \n",
       "split1_test_score                              0.76161   \n",
       "split2_test_score                             0.671827   \n",
       "split3_test_score                             0.708075   \n",
       "split4_test_score                              0.71118   \n",
       "mean_test_score                               0.711715   \n",
       "std_test_score                                0.028734   \n",
       "split0_train_score                            0.990698   \n",
       "split1_train_score                            0.987597   \n",
       "split2_train_score                            0.989147   \n",
       "split3_train_score                            0.984508   \n",
       "split4_train_score                            0.986057   \n",
       "mean_train_score                              0.987601   \n",
       "std_train_score                               0.002188   \n",
       "\n",
       "rank_test_score                                       6   \\\n",
       "mean_fit_time                                   0.242858   \n",
       "std_fit_time                                     0.04517   \n",
       "mean_score_time                                 0.066413   \n",
       "std_score_time                                  0.010017   \n",
       "param_svc__C                                         100   \n",
       "param_svc__gamma                                   0.001   \n",
       "params              {'svc__C': 100, 'svc__gamma': 0.001}   \n",
       "split0_test_score                                0.73065   \n",
       "split1_test_score                               0.708978   \n",
       "split2_test_score                               0.662539   \n",
       "split3_test_score                               0.723602   \n",
       "split4_test_score                               0.695652   \n",
       "mean_test_score                                 0.704284   \n",
       "std_test_score                                  0.024115   \n",
       "split0_train_score                              0.703876   \n",
       "split1_train_score                              0.712403   \n",
       "split2_train_score                              0.726357   \n",
       "split3_train_score                              0.710302   \n",
       "split4_train_score                               0.71495   \n",
       "mean_train_score                                0.713577   \n",
       "std_train_score                                 0.007368   \n",
       "\n",
       "rank_test_score                                     7   \\\n",
       "mean_fit_time                                 0.235208   \n",
       "std_fit_time                                  0.041942   \n",
       "mean_score_time                                0.08389   \n",
       "std_score_time                                0.007516   \n",
       "param_svc__C                                       0.1   \n",
       "param_svc__gamma                                   0.1   \n",
       "params              {'svc__C': 0.1, 'svc__gamma': 0.1}   \n",
       "split0_test_score                             0.705882   \n",
       "split1_test_score                             0.718266   \n",
       "split2_test_score                             0.690402   \n",
       "split3_test_score                             0.692547   \n",
       "split4_test_score                             0.708075   \n",
       "mean_test_score                               0.703034   \n",
       "std_test_score                                0.010345   \n",
       "split0_train_score                            0.730233   \n",
       "split1_train_score                            0.724806   \n",
       "split2_train_score                            0.737984   \n",
       "split3_train_score                            0.729667   \n",
       "split4_train_score                            0.725019   \n",
       "mean_train_score                              0.729542   \n",
       "std_train_score                               0.004789   \n",
       "\n",
       "rank_test_score                                      8   \\\n",
       "mean_fit_time                                  0.227487   \n",
       "std_fit_time                                   0.031567   \n",
       "mean_score_time                                0.075283   \n",
       "std_score_time                                  0.00862   \n",
       "param_svc__C                                        1.0   \n",
       "param_svc__gamma                                   0.01   \n",
       "params              {'svc__C': 1.0, 'svc__gamma': 0.01}   \n",
       "split0_test_score                              0.702786   \n",
       "split1_test_score                              0.705882   \n",
       "split2_test_score                              0.659443   \n",
       "split3_test_score                              0.726708   \n",
       "split4_test_score                              0.692547   \n",
       "mean_test_score                                0.697473   \n",
       "std_test_score                                 0.022019   \n",
       "split0_train_score                             0.708527   \n",
       "split1_train_score                             0.700775   \n",
       "split2_train_score                             0.723256   \n",
       "split3_train_score                             0.701007   \n",
       "split4_train_score                             0.718048   \n",
       "mean_train_score                               0.710323   \n",
       "std_train_score                                0.009034   \n",
       "\n",
       "rank_test_score                                      9   \\\n",
       "mean_fit_time                                  0.240102   \n",
       "std_fit_time                                   0.039124   \n",
       "mean_score_time                                0.087023   \n",
       "std_score_time                                 0.014581   \n",
       "param_svc__C                                        0.1   \n",
       "param_svc__gamma                                   0.01   \n",
       "params              {'svc__C': 0.1, 'svc__gamma': 0.01}   \n",
       "split0_test_score                              0.693498   \n",
       "split1_test_score                              0.702786   \n",
       "split2_test_score                              0.653251   \n",
       "split3_test_score                              0.664596   \n",
       "split4_test_score                              0.680124   \n",
       "mean_test_score                                0.678851   \n",
       "std_test_score                                 0.018153   \n",
       "split0_train_score                             0.675194   \n",
       "split1_train_score                             0.676744   \n",
       "split2_train_score                             0.683721   \n",
       "split3_train_score                             0.681642   \n",
       "split4_train_score                             0.680868   \n",
       "mean_train_score                               0.679634   \n",
       "std_train_score                                0.003172   \n",
       "\n",
       "rank_test_score                                      10  ...  \\\n",
       "mean_fit_time                                  0.211701  ...   \n",
       "std_fit_time                                   0.032375  ...   \n",
       "mean_score_time                                0.069995  ...   \n",
       "std_score_time                                 0.012089  ...   \n",
       "param_svc__C                                         10  ...   \n",
       "param_svc__gamma                                  0.001  ...   \n",
       "params              {'svc__C': 10, 'svc__gamma': 0.001}  ...   \n",
       "split0_test_score                               0.69969  ...   \n",
       "split1_test_score                              0.674923  ...   \n",
       "split2_test_score                              0.653251  ...   \n",
       "split3_test_score                              0.680124  ...   \n",
       "split4_test_score                               0.68323  ...   \n",
       "mean_test_score                                0.678244  ...   \n",
       "std_test_score                                 0.014994  ...   \n",
       "split0_train_score                              0.67907  ...   \n",
       "split1_train_score                             0.682946  ...   \n",
       "split2_train_score                             0.694574  ...   \n",
       "split3_train_score                             0.688613  ...   \n",
       "split4_train_score                             0.687064  ...   \n",
       "mean_train_score                               0.686453  ...   \n",
       "std_train_score                                 0.00525  ...   \n",
       "\n",
       "rank_test_score                                      21  \\\n",
       "mean_fit_time                                  0.256758   \n",
       "std_fit_time                                   0.029509   \n",
       "mean_score_time                                0.100254   \n",
       "std_score_time                                 0.016962   \n",
       "param_svc__C                                       0.01   \n",
       "param_svc__gamma                                    0.1   \n",
       "params              {'svc__C': 0.01, 'svc__gamma': 0.1}   \n",
       "split0_test_score                               0.50774   \n",
       "split1_test_score                               0.50774   \n",
       "split2_test_score                               0.50774   \n",
       "split3_test_score                              0.506211   \n",
       "split4_test_score                              0.509317   \n",
       "mean_test_score                                 0.50775   \n",
       "std_test_score                                 0.000982   \n",
       "split0_train_score                             0.507752   \n",
       "split1_train_score                             0.507752   \n",
       "split2_train_score                             0.507752   \n",
       "split3_train_score                             0.508133   \n",
       "split4_train_score                             0.507359   \n",
       "mean_train_score                                0.50775   \n",
       "std_train_score                                0.000245   \n",
       "\n",
       "rank_test_score                                       21  \\\n",
       "mean_fit_time                                   0.230133   \n",
       "std_fit_time                                    0.022421   \n",
       "mean_score_time                                 0.082995   \n",
       "std_score_time                                  0.009681   \n",
       "param_svc__C                                        0.01   \n",
       "param_svc__gamma                                    0.01   \n",
       "params              {'svc__C': 0.01, 'svc__gamma': 0.01}   \n",
       "split0_test_score                                0.50774   \n",
       "split1_test_score                                0.50774   \n",
       "split2_test_score                                0.50774   \n",
       "split3_test_score                               0.506211   \n",
       "split4_test_score                               0.509317   \n",
       "mean_test_score                                  0.50775   \n",
       "std_test_score                                  0.000982   \n",
       "split0_train_score                              0.507752   \n",
       "split1_train_score                              0.507752   \n",
       "split2_train_score                              0.507752   \n",
       "split3_train_score                              0.508133   \n",
       "split4_train_score                              0.507359   \n",
       "mean_train_score                                 0.50775   \n",
       "std_train_score                                 0.000245   \n",
       "\n",
       "rank_test_score                                        21  \\\n",
       "mean_fit_time                                    0.264388   \n",
       "std_fit_time                                     0.038898   \n",
       "mean_score_time                                  0.100339   \n",
       "std_score_time                                   0.022026   \n",
       "param_svc__C                                         0.01   \n",
       "param_svc__gamma                                    0.001   \n",
       "params              {'svc__C': 0.01, 'svc__gamma': 0.001}   \n",
       "split0_test_score                                 0.50774   \n",
       "split1_test_score                                 0.50774   \n",
       "split2_test_score                                 0.50774   \n",
       "split3_test_score                                0.506211   \n",
       "split4_test_score                                0.509317   \n",
       "mean_test_score                                   0.50775   \n",
       "std_test_score                                   0.000982   \n",
       "split0_train_score                               0.507752   \n",
       "split1_train_score                               0.507752   \n",
       "split2_train_score                               0.507752   \n",
       "split3_train_score                               0.508133   \n",
       "split4_train_score                               0.507359   \n",
       "mean_train_score                                  0.50775   \n",
       "std_train_score                                  0.000245   \n",
       "\n",
       "rank_test_score                                       21  \\\n",
       "mean_fit_time                                   0.276166   \n",
       "std_fit_time                                    0.029502   \n",
       "mean_score_time                                 0.096781   \n",
       "std_score_time                                  0.013255   \n",
       "param_svc__C                                       0.001   \n",
       "param_svc__gamma                                     100   \n",
       "params              {'svc__C': 0.001, 'svc__gamma': 100}   \n",
       "split0_test_score                                0.50774   \n",
       "split1_test_score                                0.50774   \n",
       "split2_test_score                                0.50774   \n",
       "split3_test_score                               0.506211   \n",
       "split4_test_score                               0.509317   \n",
       "mean_test_score                                  0.50775   \n",
       "std_test_score                                  0.000982   \n",
       "split0_train_score                              0.507752   \n",
       "split1_train_score                              0.507752   \n",
       "split2_train_score                              0.507752   \n",
       "split3_train_score                              0.508133   \n",
       "split4_train_score                              0.507359   \n",
       "mean_train_score                                 0.50775   \n",
       "std_train_score                                 0.000245   \n",
       "\n",
       "rank_test_score                                      21  \\\n",
       "mean_fit_time                                  0.298828   \n",
       "std_fit_time                                   0.026777   \n",
       "mean_score_time                                0.106827   \n",
       "std_score_time                                 0.015524   \n",
       "param_svc__C                                      0.001   \n",
       "param_svc__gamma                                     10   \n",
       "params              {'svc__C': 0.001, 'svc__gamma': 10}   \n",
       "split0_test_score                               0.50774   \n",
       "split1_test_score                               0.50774   \n",
       "split2_test_score                               0.50774   \n",
       "split3_test_score                              0.506211   \n",
       "split4_test_score                              0.509317   \n",
       "mean_test_score                                 0.50775   \n",
       "std_test_score                                 0.000982   \n",
       "split0_train_score                             0.507752   \n",
       "split1_train_score                             0.507752   \n",
       "split2_train_score                             0.507752   \n",
       "split3_train_score                             0.508133   \n",
       "split4_train_score                             0.507359   \n",
       "mean_train_score                                0.50775   \n",
       "std_train_score                                0.000245   \n",
       "\n",
       "rank_test_score                                       21  \\\n",
       "mean_fit_time                                   0.236678   \n",
       "std_fit_time                                    0.036106   \n",
       "mean_score_time                                 0.085878   \n",
       "std_score_time                                  0.010228   \n",
       "param_svc__C                                       0.001   \n",
       "param_svc__gamma                                     1.0   \n",
       "params              {'svc__C': 0.001, 'svc__gamma': 1.0}   \n",
       "split0_test_score                                0.50774   \n",
       "split1_test_score                                0.50774   \n",
       "split2_test_score                                0.50774   \n",
       "split3_test_score                               0.506211   \n",
       "split4_test_score                               0.509317   \n",
       "mean_test_score                                  0.50775   \n",
       "std_test_score                                  0.000982   \n",
       "split0_train_score                              0.507752   \n",
       "split1_train_score                              0.507752   \n",
       "split2_train_score                              0.507752   \n",
       "split3_train_score                              0.508133   \n",
       "split4_train_score                              0.507359   \n",
       "mean_train_score                                 0.50775   \n",
       "std_train_score                                 0.000245   \n",
       "\n",
       "rank_test_score                                       21  \\\n",
       "mean_fit_time                                   0.284587   \n",
       "std_fit_time                                    0.036894   \n",
       "mean_score_time                                 0.101623   \n",
       "std_score_time                                  0.003895   \n",
       "param_svc__C                                       0.001   \n",
       "param_svc__gamma                                     0.1   \n",
       "params              {'svc__C': 0.001, 'svc__gamma': 0.1}   \n",
       "split0_test_score                                0.50774   \n",
       "split1_test_score                                0.50774   \n",
       "split2_test_score                                0.50774   \n",
       "split3_test_score                               0.506211   \n",
       "split4_test_score                               0.509317   \n",
       "mean_test_score                                  0.50775   \n",
       "std_test_score                                  0.000982   \n",
       "split0_train_score                              0.507752   \n",
       "split1_train_score                              0.507752   \n",
       "split2_train_score                              0.507752   \n",
       "split3_train_score                              0.508133   \n",
       "split4_train_score                              0.507359   \n",
       "mean_train_score                                 0.50775   \n",
       "std_train_score                                 0.000245   \n",
       "\n",
       "rank_test_score                                        21  \\\n",
       "mean_fit_time                                    0.321064   \n",
       "std_fit_time                                     0.033357   \n",
       "mean_score_time                                  0.106591   \n",
       "std_score_time                                   0.016529   \n",
       "param_svc__C                                        0.001   \n",
       "param_svc__gamma                                     0.01   \n",
       "params              {'svc__C': 0.001, 'svc__gamma': 0.01}   \n",
       "split0_test_score                                 0.50774   \n",
       "split1_test_score                                 0.50774   \n",
       "split2_test_score                                 0.50774   \n",
       "split3_test_score                                0.506211   \n",
       "split4_test_score                                0.509317   \n",
       "mean_test_score                                   0.50775   \n",
       "std_test_score                                   0.000982   \n",
       "split0_train_score                               0.507752   \n",
       "split1_train_score                               0.507752   \n",
       "split2_train_score                               0.507752   \n",
       "split3_train_score                               0.508133   \n",
       "split4_train_score                               0.507359   \n",
       "mean_train_score                                  0.50775   \n",
       "std_train_score                                  0.000245   \n",
       "\n",
       "rank_test_score                                    21  \\\n",
       "mean_fit_time                                0.284478   \n",
       "std_fit_time                                 0.041944   \n",
       "mean_score_time                              0.088089   \n",
       "std_score_time                               0.014612   \n",
       "param_svc__C                                      0.1   \n",
       "param_svc__gamma                                   10   \n",
       "params              {'svc__C': 0.1, 'svc__gamma': 10}   \n",
       "split0_test_score                             0.50774   \n",
       "split1_test_score                             0.50774   \n",
       "split2_test_score                             0.50774   \n",
       "split3_test_score                            0.506211   \n",
       "split4_test_score                            0.509317   \n",
       "mean_test_score                               0.50775   \n",
       "std_test_score                               0.000982   \n",
       "split0_train_score                           0.507752   \n",
       "split1_train_score                           0.507752   \n",
       "split2_train_score                           0.507752   \n",
       "split3_train_score                           0.508133   \n",
       "split4_train_score                           0.507359   \n",
       "mean_train_score                              0.50775   \n",
       "std_train_score                              0.000245   \n",
       "\n",
       "rank_test_score                                     21  \n",
       "mean_fit_time                                 0.340115  \n",
       "std_fit_time                                  0.045918  \n",
       "mean_score_time                               0.107322  \n",
       "std_score_time                                0.016258  \n",
       "param_svc__C                                       0.1  \n",
       "param_svc__gamma                                   100  \n",
       "params              {'svc__C': 0.1, 'svc__gamma': 100}  \n",
       "split0_test_score                              0.50774  \n",
       "split1_test_score                              0.50774  \n",
       "split2_test_score                              0.50774  \n",
       "split3_test_score                             0.506211  \n",
       "split4_test_score                             0.509317  \n",
       "mean_test_score                                0.50775  \n",
       "std_test_score                                0.000982  \n",
       "split0_train_score                            0.507752  \n",
       "split1_train_score                            0.507752  \n",
       "split2_train_score                            0.507752  \n",
       "split3_train_score                            0.508133  \n",
       "split4_train_score                            0.507359  \n",
       "mean_train_score                               0.50775  \n",
       "std_train_score                               0.000245  \n",
       "\n",
       "[21 rows x 36 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = pd.DataFrame(grid_search.cv_results_).set_index(\"rank_test_score\").sort_index()\n",
    "results.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's only look at the most relevant rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>rank_test_score</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>...</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "      <th>21</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean_test_score</th>\n",
       "      <td>0.743961</td>\n",
       "      <td>0.732792</td>\n",
       "      <td>0.729091</td>\n",
       "      <td>0.720391</td>\n",
       "      <td>0.711715</td>\n",
       "      <td>0.704284</td>\n",
       "      <td>0.703034</td>\n",
       "      <td>0.697473</td>\n",
       "      <td>0.678851</td>\n",
       "      <td>0.678244</td>\n",
       "      <td>...</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_svc__gamma</th>\n",
       "      <td>0.1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.001</td>\n",
       "      <td>...</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.001</td>\n",
       "      <td>100</td>\n",
       "      <td>10</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>10</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_svc__C</th>\n",
       "      <td>1.0</td>\n",
       "      <td>100</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean_fit_time</th>\n",
       "      <td>0.202746</td>\n",
       "      <td>0.348261</td>\n",
       "      <td>0.304863</td>\n",
       "      <td>0.223569</td>\n",
       "      <td>0.666763</td>\n",
       "      <td>0.242858</td>\n",
       "      <td>0.235208</td>\n",
       "      <td>0.227487</td>\n",
       "      <td>0.240102</td>\n",
       "      <td>0.211701</td>\n",
       "      <td>...</td>\n",
       "      <td>0.256758</td>\n",
       "      <td>0.230133</td>\n",
       "      <td>0.264388</td>\n",
       "      <td>0.276166</td>\n",
       "      <td>0.298828</td>\n",
       "      <td>0.236678</td>\n",
       "      <td>0.284587</td>\n",
       "      <td>0.321064</td>\n",
       "      <td>0.284478</td>\n",
       "      <td>0.340115</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "rank_test_score         1         2         3         4         5         6   \\\n",
       "mean_test_score   0.743961  0.732792  0.729091  0.720391  0.711715  0.704284   \n",
       "param_svc__gamma       0.1      0.01       0.1      0.01       0.1     0.001   \n",
       "param_svc__C           1.0       100        10        10       100       100   \n",
       "mean_fit_time     0.202746  0.348261  0.304863  0.223569  0.666763  0.242858   \n",
       "\n",
       "rank_test_score         7         8         9         10  ...        21  \\\n",
       "mean_test_score   0.703034  0.697473  0.678851  0.678244  ...   0.50775   \n",
       "param_svc__gamma       0.1      0.01      0.01     0.001  ...       0.1   \n",
       "param_svc__C           0.1       1.0       0.1        10  ...      0.01   \n",
       "mean_fit_time     0.235208  0.227487  0.240102  0.211701  ...  0.256758   \n",
       "\n",
       "rank_test_score         21        21        21        21        21        21  \\\n",
       "mean_test_score    0.50775   0.50775   0.50775   0.50775   0.50775   0.50775   \n",
       "param_svc__gamma      0.01     0.001       100        10       1.0       0.1   \n",
       "param_svc__C          0.01      0.01     0.001     0.001     0.001     0.001   \n",
       "mean_fit_time     0.230133  0.264388  0.276166  0.298828  0.236678  0.284587   \n",
       "\n",
       "rank_test_score         21        21        21  \n",
       "mean_test_score    0.50775   0.50775   0.50775  \n",
       "param_svc__gamma      0.01        10       100  \n",
       "param_svc__C         0.001       0.1       0.1  \n",
       "mean_fit_time     0.321064  0.284478  0.340115  \n",
       "\n",
       "[4 rows x 36 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "relevant = [\n",
    "    \"mean_test_score\",\n",
    "    \"param_svc__gamma\",\n",
    "    \"param_svc__C\",\n",
    "    \"mean_fit_time\",\n",
    "    \"rank_test_score\",\n",
    "]\n",
    "results = pd.DataFrame(grid_search.cv_results_)[relevant].set_index(\"rank_test_score\").sort_index()\n",
    "results.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "- Other than searching for best hyperparameter values, `GridSearchCV` also fits a new model on the whole training set with the parameters that yielded the best results. \n",
    "- So we can conveniently call `score` on the test set with a fitted `GridSearchCV` object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7376237623762376"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why `best_score_` and the score above are different? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7439609253312309"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because one is using test data and the other one is using train data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us make a `SVC` pipeline using `best_params_` and see if it matches results from `grid_search.score`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'svc__C': 1.0, 'svc__gamma': 0.1}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8245505269683819"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_svm = make_pipeline(StandardScaler(), SVC(C=1.0, gamma=0.1))  # using grid_search.best_params_\n",
    "best_svm.fit(X_train, y_train)\n",
    "best_svm.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_svm.score(X_train, y_train) == grid_search.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_svm.score(X_test, y_test) == grid_search.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Visualizing the parameter grid as a heatmap "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'svc__gamma': [0.001, 0.01, 0.1, 1.0, 10, 100],\n",
       " 'svc__C': [0.001, 0.01, 0.1, 1.0, 10, 100]}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "param_grid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARkAAAEGCAYAAABRkOFZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAsY0lEQVR4nO3dd3xT9f7H8dc36Qba0l1aNqUKiCBlKA5UUEAUN6I/75XLFVFxX8eVq+Lgqne5UBEFUVy4EFD2kiF7ydayS5vuRXeT7++PxLZp09KSHhrK5/l48GjOOZ9vzjtJ+8kZ5ERprRFCCKOYmjqAEKJ5kyYjhDCUNBkhhKGkyQghDCVNRghhKK+mDnAm+Chf7UeLpo4hRLOVT3aG1jrc1bJzosn40YL+6uqmjiFEs7VMf3u0tmWyuySEMJQ0GSGEoaTJCCEMJU1GCGEoaTJCCENJkxFCGEqajBDCUNJkhBCGkiYjhDCUNBkhhKGkyQghDCVNRghhqHPiA5L1kXBtLx54cwwms4mF05cz+/UfnJb3vKIbL/3wNJbDaQCsnbORz17+FoAnpt9P/+v6kJOWy7ieT0geydPkeTwpU5M3GaXUDGAEkKa17uGYFwLMBjoAR4DbtdbZjmV/B8YCVuBhrfVidzOYTCYemjKWp695mYykLKZsepX187ZwbF+SU92uNft47obXaoxfMnMVc6cs4qlPJrgbRfJInmaXyRN2l2YCQ6vNewZYrrWOA5Y7plFKdQPuALo7xrynlDK7GyC+XxeSEy1YDqdRXlbOqtnruGRkQr3H71qzj/ysk+7GkDySp1lmavImo7VeDWRVmz0S+MRx+xPgxirzv9Jal2itDwOJQD93M4TFhJCelFkxnZGURVhMaI26bhd3Zer2fzP5p2dp3y3W3dVKHslzTmRq8t2lWkRqrVMAtNYpSqkIx/wYYEOVuiTHvBqUUuOAcQB+BNS5MqVqzqv+fVSJ2w5zV4cHKC4opt+w3rw45ynuiX+4Xg+moSSP5GlOmZp8S6aBXDx1uPx2Oq31NK11gtY6wRvfOu80PSmL8NjKLh8WG0JmsvPGVWF+EcUFxQBsWrgds7eZwNBWDYxfP5JH8jSnTJ7aZFKVUtEAjp9pjvlJQNsqdbFAsrsrO7A5kZi4aKI6RODl7cWgUQNZP2+LU03ryOCK2/F9u2AymcjLzHd31ZJH8jT7TJ66uzQP+DPwmuPn3Crzv1BK/Q9oA8QBm9xdmc1qY8pD03l10URMZhOLP17J0b1JjLhvCAA/frCUy28dwIjx12Att1JaVMrk0W9UjH/280foOag7QWGt+OLYVD6d9DWLZqyQPJKnSfJ4WibV1N+FrZT6EhgEhAGpwAvAD8DXQDvgGHCb1jrLUT8R+AtQDjyqtV54qnUEqhAtFxIXwjjL9LdbtdYuT181+ZaM1np0LYtcdgWt9WRgsnGJhBCNyVOPyQghmglpMkIIQ0mTEUIYSpqMEMJQ0mSEEIaSJiOEMJQ0GSGEoaTJCCEMJU1GCGEoaTJCCENJkxFCGEqajBDCUNJkhBCGkiYjhDCUNBkhhKGkyQghDCVNRghhKGkyQghDSZMRQhhKmowQwlDSZIQQhpImI4QwlDQZIYShpMkIIQwlTUYIYShpMkIIQ0mTEUIYqsm/C9tTJFzbiwfeHIPJbGLh9OXMfv0Hp+U9r+jGSz88jeVwGgBr52zks5e/BeCJ6ffT/7o+5KTlMq7nE5JH8jR5Hk/K5LFNRik1AxgBpGmtezjmhQCzgQ7AEeB2rXW2u+symUw8NGUsT1/zMhlJWUzZ9Crr523h2L4kp7pda/bx3A2v1Ri/ZOYq5k5ZxFOfTHA3iuSRPM0ukyfvLs0Ehlab9wywXGsdByx3TLstvl8XkhMtWA6nUV5WzqrZ67hkZEK9x+9as4/8rJONEUXySJ5ml8ljm4zWejWQVW32SOATx+1PgBsbY11hMSGkJ2VWTGckZREWE1qjrtvFXZm6/d9M/ulZ2neLbYxVSx7J0+wzeezuUi0itdYpAFrrFKVURG2FSqlxwDgAPwLqvFOlas7TWjtNJ247zF0dHqC4oJh+w3rz4pynuCf+4QY/gPqQPJKnOWXy2C0Zd2mtp2mtE7TWCd741lmbnpRFeGxllw+LDSEz2XkjqjC/iOKCYgA2LdyO2dtMYGirxg8ueSRPM8t0tjWZVKVUNIDjZ1pj3OmBzYnExEUT1SECL28vBo0ayPp5W5xqWkcGV9yO79sFk8lEXmZ+Y6xe8kieZp3pbNtdmgf8GXjN8XNuY9ypzWpjykPTeXXRRExmE4s/XsnRvUmMuG8IAD9+sJTLbx3AiPHXYC23UlpUyuTRb1SMf/bzR+g5qDtBYa344thUPp30NYtmrJA8kqdJ8nhaJlV9P81TKKW+BAYBYUAq8ALwA/A10A44Btymta5+cLiGQBWi+6urDcsqxLlumf52q9ba5ekrj92S0VqPrmWRdAshziJn2zEZIcRZRpqMEMJQ0mSEEIaSJiOEMJQ0GSGEoaTJCCEMJU1GCGEoaTJCCENJkxFCGEqajBDCUNJkhBCGkiYjhDCUNBkhhKGkyQghDCVNRghhKGkyQghDSZMRQhhKmowQwlDSZIQQhpImI4QwlDQZIYShpMkIIQwlTUYIYShpMkIIQ0mTEUIYSpqMEMJQHvs1tWdawrW9eODNMZjMJhZOX87s139wWt7zim689MPTWA6nAbB2zkY+e/lbAJ6Yfj/9r+tDTlou43o+IXkkT5Pn8aRMHtVklFIzgBFAmta6h4vlCngLGA4UAvdorbe5u16TycRDU8by9DUvk5GUxZRNr7J+3haO7Utyqtu1Zh/P3fBajfFLZq5i7pRFPPXJBHejSB7J0+wyedru0kxgaB3LhwFxjn/jgPcbY6Xx/bqQnGjBcjiN8rJyVs1exyUjE+o9fteafeRnnWyMKJJH8jS7TB7VZLTWq4GsOkpGAp9quw1AsFIq2t31hsWEkJ6UWTGdkZRFWExojbpuF3dl6vZ/M/mnZ2nfLdbd1UoeyXNOZPKo3aV6iAGOV5lOcsxLqV6olBqHfWsHPwLqvFOlas7TWjtNJ247zF0dHqC4oJh+w3rz4pynuCf+4QbGrx/JI3maUyaP2pKpBxdPHdrFPLTW07TWCVrrBG9867zT9KQswmMru3xYbAiZyc4bVIX5RRQXFAOwaeF2zN5mAkNbNTB+/UgeydOcMp1tTSYJaFtlOhZIdvdOD2xOJCYumqgOEXh5ezFo1EDWz9viVNM6MrjidnzfLphMJvIy891dteSRPM0+09m2uzQPmKCU+groD+RqrWvsKjWUzWpjykPTeXXRRExmE4s/XsnRvUmMuG8IAD9+sJTLbx3AiPHXYC23UlpUyuTRb1SMf/bzR+g5qDtBYa344thUPp30NYtmrJA8kqdJ8nhaJlV9P60pKaW+BAYBYUAq8ALgDaC1nuo4hT0F+xmoQmCM1nqL63urFKhCdH91tVGxhTjnLdPfbtVauzx95VFbMlrr0adYroEHz1AcIUQjONuOyQghzjLSZIQQhpImI4QwlDQZIYShpMkIIQwlTUYIYShpMkIIQ0mTEUIYSpqMEMJQ0mSEEIaqs8kopboopQa6mH+ZUqqzcbGEEM3FqbZk3gRcffa7yLFMCCHqdKom00Fr/Wv1mY5PPncwJJEQolk51aew/epY5t+YQYxki/Oh6L2OTR2jQlqucVdEOx2dxhxq6ggeTZeWNnWEGmwemKk2p9qS2ayUurf6TKXUWGCrMZGEEM3JqbZkHgXmKKXuorKpJAA+wE0G5hJCNBN1NhmtdSpwiVLqSuCPL1v7SWvt3rUBhRDnjHpdGU9rvRJYaXAWIUQzJP8ZTwhhKGkyQghDSZMRQhhKmowQwlDSZIQQhpImI4QwlDQZIYShpMkIIQzlUV9T25T6h3blka4jMSnFjyc28dnRVTVqerfuxMNdb8BLmcgpK+ShrVNpGxDOSxfcVVHTxj+Ejw4u4Zvja93Kc1lkZyZeeC1mpfjm8Ham/fZLjZp+Ye2ZeOE1eJnMZJcU8n+rPwVgxdCHKCgvxaZtlGsbt6yY7lYWgD6De3D/63diMptY9Mlqvn5jgdPynpfG88KXD2M5mgHAuvlb+eL1eYTFhPDkB3+ldWQQ2qZZMPNn5r6/tNnlSRjSk/H/vRuz2cTCj1fx9X/mO+e5/HwmffM4liPp9jxzN/P5P+cA8PgH99J/WG9y0vO4r88zbmepyHRtLx54cwwms4mF05cz+/UfnDNd0Y2Xfngay+E0ANbO2chnL38LwBPT76f/dX3ISctlXM8n3MrhcU1GKTUUeAswAx9prV+rtvw84GPgImCi1vo/7q7ThOLx+Jt4bPuHpBXn8lG/h1ibsZcjBWkVNS29/Hg8/ib+tn06qSU5BHu3AOB4YTpjNr5ZcT9zLvsHq9N3u53nhV5DGbP2cyyFeXx31V9ZnvIbB/MzKmpaefsyqfcwxq79gpSiPEJ8A5zu40+rPyW7tMitHBV5TIoH/3s3z478Dxknsnh71fNsWLCDYweSnep2r/+NF25/y2merdzKhxNnk7jzKP4t/Xhn9QtsX7GnxtizPs9b9/D3614lIymLd9a9zIYft3Fs/wnnPOsO8PzNNX9dl8xaw7z3l/Lk9PGnnaFmJhMPTRnL09e8TEZSFlM2vcr6eVs4ti/JqW7Xmn08d8NrNcYvmbmKuVMW8dQnE9zP4vY9NCKllBl4FxgGdANGK6W6VSvLAh4G3G4ufzg/qC1JRRkkF2VRrq0sS93JpeHdnWqGRPVmdfpuUktyAMgpK6hxP31CunCiKJPU4hy38vQMacPRgmyOF+RQpm38lLSHwW3inWqub9uDJSf2k1KUB0BWSaFb66xLfEInUg6lYTmSTnmZlZ+/28TF1/Wu19is1FwSdx4FoOhkMccPpBDaJrh55enbmeSDqVgO2/Os+mYDF1/fp97jd6/dT372Sbcy1MjUrwvJiRYsh9MoLytn1ex1XDIyod7jd63ZR35W42TyqCYD9AMStdaHtNalwFfAyKoFWus0rfVmoKyxVhruG0RacW7FdHpxLuG+gU41bQPCaOXlzzt97mN6v4cZGn1RjfsZHNWLZZYdbueJ9A/EUphXMW0pyiPS3/kaNB1ahhLk48esy+/m+6v+yo3telYs02hmXHoX31/1V0Z1rN8fX11Co1uTnpRVMZ2RnEVom9Y16s7v14X31r3Iy989Rvvz2tR8XO1C6dyzHQe2uHf9Go/L0yaE9KTMyjwnsghzlad/F97f9E9emfsU7c+PcWudpxIWUy1TUhZhMaE16rpd3JWp2//N5J+epX23WEOyeNruUgxwvMp0EtD/dO5IKTUOGAfgGxFYd62LebratFmZiA+M4ZGt0/A1ezO17wT25B7jeKF9F8ZLmRkY1o2piQtPJ+6p82jnRF4mE92Do/nzms/wM3sx+8ox7MhK4sjJLEavmkla8UlCfAOYeen/cTA/ky0Zx04/j4tA1fMk7jzKn7r/jeKCEvpe05Pnv3yYsb0rjy/4tfDlH7Mm8MEzX1KYX3zaWc7aPNuPcHfXR+x5rr2QF755nL/0cO9Yh9uZth3mrg4PUFxQTL9hvXlxzlPcE/9wo2fxtC2Z+vy914vWeprWOkFrneAVVPdF/NJKconwC6qYDvcLIqMkz6kmvTiXjZm/UWwrI7eskJ3Zh+jSMrpi+YCweH7LP0F2qfubmJaiPKICKhtjlH8gacXO92spzGNN6kGKrGVklxaxOf0Y5wVF2h+PozarpJClyfvp2brmu3hDZCRnEx4bUjEd1iaErJQcp5rC/GKKC0oA2LzkV7y8zASGtATA7GXmuc8msPLr9ayb7/61zjwuz4kswmMrtxLCYkLIrJGnqDLP4p2Yvc0EhrZ0e921SU+qlik2hMzkLKcaeyZ7g920cLsjU+NftdHTmkwS0LbKdCxw+kfk6ml/XhJt/cOI9muNlzIzOPJC1qXvdapZk76XnsEdMCsTviZvugW1czowPDiycXaVAHZlJ9OhZQixAcF4KxPXxXZnefJvTjXLU34jIawdZqXwM3txYUgMB/Mz8Dd708LLBwB/szcDIzvxe166W3kObD1Mm04RRLYPw8vbzBW39GPDgu1ONa2rbC127dMRZVLkOfbpH3t3DMcOJPP9u0vcyuGxebYcIqZLFJEdwvHyNjPotgFs+NG5ebWOrHwTi0/ohMmkyMts3OMwTpk2JxITF01Uhwi8vL0YNGog6+dtqZYpuDJT3y6YTCbyMl19b4B7PG13aTMQp5TqCJwA7gDuNHqlVm3jfwfm8r/ef8WkTPyUvJnDBamMjBkAwNwTGzhamMbGzN+Y2f8xNJr5JzZxuCAVAF+TN31D4vj3vu8bKY/mpR2LmH7pnZiV4tsjO0nMT+eOjvbjQF8d3sbB/AxWpx5k/uD7sGnNN0e283teOm1bBPPugNsBMJtMzD+2mzWpB93KY7PaeO/Jz5k85wlMZhNLZq3h6P5khv9lEAALZqzi0hv7MmLslVjLrZQUl/HqmKkAdB8Qx+DRAzm8+zjvrn0RgJkvfcfmJTWuT39W53n30Zn8c/7T9jyf/MzRfSe47q9XA/DTR8u57KZ+jBg32J6nqIxX755SMf6ZTx+k52XnExTWis8S32HWK9+yeObPp53nj0xTHprOq4smYjKbWPzxSo7uTWLEfUMA+PGDpVx+6wBGjL8Ga7mV0qJSJo9+o2L8s58/Qs9B3QkKa8UXx6by6aSvWTTj9K5Vp6rvpzU1pdRw7F+3YgZmaK0nK6XGA2itpyqlooAtQCBgA04C3bTWebXcJS27Rule7/3J8Oz1JRcSP7vIhcRPbZn+dqvW2uXpK0/bkkFrvQBYUG3e1Cq3Ldh3o4QQZwFPOyYjhGhmpMkIIQwlTUYIYShpMkIIQ0mTEUIYSpqMEMJQ0mSEEIaSJiOEMJQ0GSGEoaTJCCEMJU1GCGEoaTJCCENJkxFCGMrjPoVthNY+RdwYs7OpY1TIjQw4ddEZtHuhe1fOa2w27eoCiU0nrzTk1EXnuqtrXyRbMkIIQ0mTEUIYSpqMEMJQ0mSEEIaSJiOEMJQ0GSGEoaTJCCEMJU1GCGEoaTJCCENJkxFCGEqajBDCUNJkhBCGOic+IFkf+9dkMO/V/dismn63xnLVvR2dlq+afphtP1oA+5eZpx0qYNLaKwkI9j7l2NNxcG0qi1/fhbZpet3cnoFjuzotX//x7+xecNyep1yTcTifx38ejn+QzynHno7ewd0Y2+lWTJhYlrqO708srVHTPTCOsR1vxWwyk192kn/sfhOA69tcyeDIgaA1RwuTeef3WZTpcjfznM+9nW7FpEwsTf2F75Jq5ukRFMfYjrfgpczklZ9k4q63ALihzZUMibwEjT3P27995naeviHxTIi7EbMy8VPKRr48WvPL6S8M7syEuJF4KTO5ZQU8uv092gaE83z3uytqov1D+fjQIr5LWuNWHk/K1CRNRik1FHgLMAMfaa1fq7ZcOZYPBwqBe7TW2xzLZgAjgDStdY/GyGOzaua8so9xH/UhKNKPt0dtoPuV4UR2aVlRM2hsRwaNtTePvSvTWP3pUQKCves19nTyLPznTu6aNpDASH+mj15F10FRhHcOrKi5eEwcF4+JA+C3VSlsnHUQ/yCfeo1tKBOKcZ1uZ9Ked8gszeFfFz7FpqxdJBVZKmoCzP7c13kUL+15l4zSbIK87Y8/xCeI66IH8fD2Vyi1lfG3+LFcGp7AyrQNbuW5r/PtvLB7CpmlOfyn15NsytzF8Sp5Wpj9Gd/5dibteY+MEuc8I9pcwYRtkym1lfFk/F+4LLwPK9I2upXnkfibeXL7B6SX5DI14VF+Sd/D0cLUyjxefjwafzNP7/iQtJIcgh15jhemc+/m/1XczzcDn2dtxu7TzuKJmc747pJSygy8CwwDugGjlVLdqpUNA+Ic/8YB71dZNhMY2piZju3KJaxdAKFtA/DyMdFrWBR7VqTVWr99gYXew6NPa2x9JO/OJqRdS1rHtsDsbaL70Fh+W2mptX7PwhN0HxZ7WmPrI65VB1KK00ktyaRcW1mbvpV+IT2dai4PT2BD5g4ySrMByC07WbHMrMz4mLwxYcLX5E1WaY7beSzFGRV51qRvo19ozTzrM3aSUXKKPGYfskpz3cpzXmA7kgszSSnOolxbWZG2nYHh3Z1qBkdexJr0XaSV5ACQUyXPHy4KiSO5KJPU4my38nhapqbYkukHJGqtDwEopb4CRgJ7q9SMBD7VWmtgg1IqWCkVrbVO0VqvVkp1aMxAeanFBEf5VUwHRflx7FfXv3ilRVYOrMngponnN3hsfeWnFhEY6V8x3SrSj+Rdrl/ksqJyDq5LZeizPRs8tr5CfIIrmgdAZmkOXVt1cKpp4x+BlzLzco9H8Df78WPySlalbyKrNJe5J5YxLeEVSm2l7MjZz86c/W7lCfUJqmgeAJkl2bXmeeWCR/A3+/Jj8ipWptnzzDmxnI/6vmzPk72fHW7mCfMNqvhDBUgvyeX8wHZONbEB4XgpM2/0vp8Asy/fJa1hiWWrU81VEb1ZnrrdrSyemKkpDvzGAMerTCc55jW0pk5KqXFKqS1KqS0ns0rrrNXaxfhaaveuSqfDRcEEBHs3eGx9ubjLWu/0t58ttO0Vgn+QT4PH1per4braAzcrM51atuOVve/z4p4p3NZ2GG38Imhh9qdfSE/Gb3mesZufxc/kwxXhfd0L5CJR9cdtViY6t2zLy3veZ9Lud7m97dCKPP1DLmDc5hcYs2kivmb387h+fmrm6doqlr/vnM6TOz/k7g5DiPUPq1jupcxcEtadn9Ma5+JqnpSpKbZkXD7+06ipk9Z6GjANoG2PoDrHBkX5kWMprpjOtRQTGOHrsnZHlV2lho6tr8BIf/JSiyqm81OLaRXu77J276LKXaWGjq2vzNIcwnxaV0yH+gTX2MXILMkmr+wkJbZSSmyl7M1LpEML+/tCakkmeeX2TfENmTuIb9WJn9M3u5fHt0oe39Y185TmkJdTUJFnT26VPMVV8+zkvMCObuVJL8klwje4YjrcN4jMannSS3LJLSug2FZKsa2UX3MO0bllG5KKMgDoH3oev51MItvFLsvZnqkptmSSgLZVpmOB5NOoaTRtewSScbSQrKRCyktt7FhooduVETXqivLLOLQ5i+5XhTd4bEO06R5M1tGTZCcVYC2zsWdREl0HRdWoK84v4+iWDLpeGd3gsQ3xe/5Rov0jiPANxUuZuTS8D5uzdjnVbMr6lW6BXTBhwsfkTdeWHUgqspBekk3XVh3xMdm3/HoGxzsdMD79POEVeS4Lv4hNWb861WzM/JVugZ0r87Sy58koySK+ap6geJKqHAw9HfvzjxMTEEaUXwheysxVEb35JWOPU8269N30DOqESdmPS50f2I6jhZXH7q6K7M2KRtpV8rRMTbElsxmIU0p1BE4AdwB3VquZB0xwHK/pD+RqrVOMCmT2MnHjxPP48N5t2GyafjfFEBXXkvVf2ffYLr7D3u92L0uj68AwfAK8TjnWHSYvE0Of7cmX9/+CzarpdWN7wrsEsvXrwwD0ud1+luvAimQ6XRLhlKe2se6wYePDQ1/zQvcHMWFiedp6jhelcG3UpQAstqwlqSiV7Tl7ebP3s2itWZr6C8cK7S/Z+ozt/PfCZ7BpG4cKklhiWed2nmkHv2ZSjwcxoVieuoHjhRaGOvIs+iNP9l7evujv2Krl+SVzO2/0ehqrI89id/NoG2//9j3/6jUOk1IsTN7EkYJUrm9zMQDzk9dzrDCNTVn7md7vCbTW/JS8kSMF9mbra/KmT0hX/rf/W7dyeGomVX3f+kxQSg0H3sR+CnuG1nqyUmo8gNZ6quMU9hTsZ5EKgTFa6y2OsV8Cg4AwIBV4QWs9va71te0RpB/5ZoBBj6bhcss97ELieXIh8brklbq3+3suWHX1/7ZqrRNcLWuS/yejtV4ALKg2b2qV2xp4sJaxo41NJ4RoTPKxAiGEoaTJCCEMJU1GCGEoaTJCCENJkxFCGEqajBDCUNJkhBCGkiYjhDCUNBkhhKGkyQghDCVNRghhKGkyQghDSZMRQhjqnPhKlEhzCY+3PtzUMSocK89v6ghOCoM9673GYnXvejyNzVIe3NQRakgtC2rqCE5W1bHMs367hBDNjjQZIYShpMkIIQwlTUYIYShpMkIIQ0mTEUIYSpqMEMJQ0mSEEIaSJiOEMJQ0GSGEoaTJCCEMJU1GCGEoaTJCCEOdE5/Cro9FKwp47Pl0rFYYe2cgTz8U4rT8P+9l88X3eQCUl8O+30tJ3d2JkNZmAKxWTb+hx2kTZWb+rBi38/y8soQXJ+Vhs8Ko0f7c/6DzJ5M/mFrA3DlF9nWXQ2JiOVt3RODvr7j91ixKSzVWKwwb7stjT7RyO8+aVcW8NikXqxVuuSOAex90vs8ZU0/y4w+FFXkOJZazZkcU/n6KP92WYc9TDtcM92PCE4Fu59n0cwHvvpiGzQbDRwUx+n7n12v2B1ksn2v/tLvVqjmWWMp3WztTXGjjtScsZKdbUSa4bnQQt4xp7XaenatzmPXKMWxWzaDbw7nhvjZOy3/8MIV18zIBsFk1Jw4WMXXjRbQM9mLaM4fYvjKHwFBvXl9wgdtZ/rBvTQY/vHoAm1Uz4NYYrr63o9PyFdOPsO3HlIpMqYcKeGntIFoEe59ybEMo+3fbG0MpNRR4CzADH2mtX6u2XDmWDwcKgXu01tvqGquUug2YBJwP9NNabzlVjoQL/fSmxe1qXW61as4beJTFs2OIjfai/7BjfP5eFN3ifV3Wz19ykrem5bDs29iKeW9MzWbLr8Xk5dtO2WROdakHq1Vz1eUZzPqiNVHRZkaOyOTtKcHEdXX9nrBsaTEzPirki9khaK0pLNS0aGGirExz281ZvPBiK3pf5FPr+gp13Ru0VqvmuivS+PDzUCKjzYy6Pp1/v9OaLl29XdavXFrMp9NP8vFXYTXy3H1LBn+fFMSFdeQ51aUerFbNn686wr9mxRAe5c0DI48y8e1oOsS5fr1+WXaS72Zk898v2pKZVk5mWjlde/hReNLG+OuP8tK0NrWOhVNf6sFm1Twx5Ff+PjOekCgfnrtlDw/+rwuxcf4u67ctz2bhTAsTZ50PwL5Nefi1MDP1yUP1bjKnutSDzap5dfg6xn90EUGRfrwxaiN3//sCorq4fm73rEzn50+P8sDHCQ0eC/B4t6VbtdYJrpYZtruklDID7wLDgG7AaKVUt2plw4A4x79xwPv1GLsbuBlY3VhZN20vpnMHbzq198bHRzFqZCvmLS6otf6rH/IZdWPlO3lSchkLlhcw9s7GucbHzh1ltO9gpl17L3x8FNff4MfSJcW11s+fW8z1I/0AUErRooX9ZS0vh/JyDcq9PLt2lNG2gxdtHXmGX+/PyjryLJhXxPAb/F3k0ZSXg3Izz/6dxcS096ZNOx+8fRRXXh/IL0trf71Wzs/nquvtr1dohBdde9ifq4CWJtp38SHDUu5WnoO/niSyvS8R7fzw8jEx4LpQti7PrrX+lx8zuXhEaMX0+f0CaRnUuDsVx3blEtYugNC2AXj5mOg9LIrdK9Jrrd+2wELv4VGnNfZUjDwm0w9I1Fof0lqXAl8BI6vVjAQ+1XYbgGClVHRdY7XW+7TWBxoz6AlLOW1jKl/kmGgvTtTyi1dYaGPxykJuua6yqz/2fAav/SMMUyM9mxaLjeg25orpqGgzFovNZW1RkebnVSUMG+ZXMc9q1Qy/NoOEXmlcepkvvXvXvtVQH6kWq1OeyGgzqanWWvLYWLuqmCHDK9/FrVbNzUPTuKx3Khdf6ktPN/NkWMoJj658vcKjvMiwlLmsLS6ysfnnAi4bVnOX0ZJURuLeEs7v5ediZP1lWcoIja7cEgqJ8iE7tdRlbUmRlV/X5NLv2hCXyxtLbmoJwVGVmYKjfMlNK3FZW1pkZf+aDHoOiWzw2PowssnEAMerTCc55tWnpj5j66SUGqeU2qKU2pKe6foP4g+u9hhre7edv7SAS/r6VxyL+XHpSSLCzPS50L1f1NPNs3xpMX36+hDcuvKlNJsVCxaHsX5TODt3lHFgv+s/wPoHcpXHdaBVS0voneBDcLBznu8XRbBiYyS7dpby+4Ezl2f98gK69/EnMNjsNL+owMak+5N54LlwWrQyuxzrjtper20rcuh6UStaBht7ONTl71AttXtWpdPxomBaBHs3eGx9GNlkXOWqHr+2mvqMrZPWeprWOkFrnRAeWvcvUWy0F8dPVG65nEgpp02k61+C2T/kc8eNlVsxv2wqZv6SAjr1Pcyd4y2sXFvE3Q9aGhK1huhoEynJlY3RkmIlMtL1SzV/XjE33OC6wQUGmRhwsQ8/r3L9rlpfkdFmpzypKVYiIlznWTi/iOEjXR+LCAwy0W+AL2tXnf67IkBYtBfpKZWvV7qlnNBaXq+V8/O46gbnrZjyMs2k+5O5emQglw11/6B4SJQ3mSmVjynLUkpwhOuttQ0/ZXHxCGO3YsC+9ZFjqcyUYykhMML1caftVXaVGjq2PoxsMklA2yrTsUByPWvqM7bR9O3lR+LhUg4fK6O0VDN7bj7XX9uiRl1unpXVG4oYObSyyfxzYhjHtnXk0OaOfDE1iisv9WfWu1E1xjZEzwu9OXLEyvFj5ZSWaubPK2bwkJovcl6ejY0bShlybeWyzEwbebn2XaviIs3aNSV07uLeO3WPC705dricJEeeBfOLuHJIzcaWn2dj84YSrrqmcllWprUyT7Fm/doSOnZ27138vJ5+nDhSRsrxMspKNSvn53HJ4Jqv18k8K79uLOKSIZWvl9aa/zxtoV0XH277q/tnlQA6XdASy5ES0o6XUF5qY8NPmfS5OrhGXWF+Ofs25dFncOOsty5tewSSfrSQzKQiykttbF9ooceV4TXqivLLOLg5mx5XRTR4bH0Zuc22GYhTSnUETgB3AHdWq5kHTFBKfQX0B3K11ilKqfR6jG00Xl6Kt/8ZwbDRJ7BaYcwdgXSP92XqJzkAjP9zMABzFhYw5IoAWgQY+9+LvLwUL74cyJ/+LxubFW4b5U/XeG8+n2U/RXzX3QEALFlUzGWX+xJQJU9ampW/PWY/1axtcN31flw92L1dOS8vxcSXgxh3dyY2K9w0KoAu8d7MnmU/2Drqbvsf+LLFxQyslic9zcazj9sfh80G147wZ5CbecxeiodeDOfpPyVhs8Gw2wLp0NWX+Z/nAHD9XcEArF1ykj6XtcC/Sp7dW4pZOiefjvE+jBt+FICxT4bS/8rTv3i52Utxzwvtef0v+7FZ4Ypbw4mNC2DZF2kADL7T/ge8eUk2F1wahF+Ac9Of8mgi+zblk59dzoRLt3PrI7EMuu30/6jtmUzcPDGeafduw2bT9LupDVFxLfnlK/tRiEvusL+H71qWTvzAUHyrZKpt7Oky+hT2cOBN7KehZ2itJyulxgNorac6TmFPAYZiP4U95o9T0q7GOubfBLwDhAM5wA6t9bV15TjVKewzzeO+reAUp7DPNPm2glPztG8rqOsUtqFHn7TWC4AF1eZNrXJbAw/Wd6xj/hxgTuMmFUIYxbPewoQQzY40GSGEoaTJCCEMJU1GCGEoaTJCCENJkxFCGEqajBDCUNJkhBCGkiYjhDCUNBkhhKGkyQghDCVNRghhKEM/he0pHJeOONoIdxUGZDTC/TQWT8sDnpdJ8tStsfK011q7vD7FOdFkGotSakttH2dvCp6WBzwvk+Sp25nII7tLQghDSZMRQhhKmkzDTGvqANV4Wh7wvEySp26G55FjMkIIQ8mWjBDCUNJkhBCGOmebjFJqqFLqgFIqUSn1jIvlSin1tmP5r0qpi041Vil1m1Jqj1LKppRy67Sgm/lmKKXSlFK73cngRrbzlFLrlVIlSqm/GZGh2vrqfLx1PVdnOpdSKkQptVQp9bvjp6FfwtTQDEqpvzuepwNKqTq/BaTetNbn3D/sX7NyEOgE+AA7gW7VaoYDC7F/m+UAYOOpxgLnA/HAKiChKfI5ll0OXATsbqLnLgLoC0wG/nYGXs86H29dz9WZzgX8C3jGcfsZ4HVPyQB0c7yevkBHx+tsdjfDubol0w9I1Fof0lqXAl8BI6vVjAQ+1XYbgGClVHRdY7XW+7TWB5o4H1rr1UBWI+Q4rWxa6zSt9WbAzS+9rp96PN5an6smyDUS+MRx+xPgRg/KMBL4SmtdorU+DCRif73dcq42mRjgeJXpJMe8+tTUZ2xT5jNaU63XHZ6UOVJrnQLg+BlxivozmcGQ5+lcbTLKxbzq5/Jrq6nPWHe5k89oTbVed5yNmZuCIc/TudpkkoC2VaZjgeR61tRnbFPmM1pTrdcdnpQ59Y9dNcfPNA/KYMjzdK42mc1AnFKqo1LKB7gDmFetZh7wJ8eZiQFArmPTsj5jmzKf0c7E429sTfVc1Zblz47bfwbmelCGecAdSilfpVRHIA7Y5PbazsRRdk/8h/2Mw2/Yj6BPdMwbD4x33FbAu47lu6hytsjVWMf8m7C/G5QAqcDiJsr3JZCC/cBrEjD2DD93UY715gE5jtuBBr6WNR5vfZ8rg3/HXOUKBZYDvzt+hnhSBmCi43k6AAxrjAzysQIhhKHO1d0lIcQZIk1GCGEoaTJCCENJkxFCGEqajBDCUNJkhBCGkiYjhDCUV1MHEM2LUuo54C7sH7TLALYCucA47JeGSATu1loXKqVmAkXAeUB7YAz2/4F6MfbLMdzjuM+T2P8z3WAgG3gW++UK2gGPaq3nKaU6ALOAFo4oE7TWvxj8cEU9yJaMaDSOC3XdAvQGbgb+uHDX91rrvlrrC4F92P/X6R9aA1cBjwHzgTeA7sAFSqlejpoWwCqtdR8gH3gFGIL9f1i/5KhJA4ZorS8CRgFvG/EYRcPJloxoTJcCc7XWRQBKqfmO+T2UUq8AwUBLYHGVMfO11loptQtI1VrvcozdA3QAdgClwCJH/S6gRGtd5hjTwTHfG5jiaExWoKsBj0+cBmkyojG5ulQAwEzgRq31TqXUPcCgKstKHD9tVW7/Mf3H72eZrvz8S0Wd1tqmlPqj5jHsnxe7EPsWevFpPwrRqGR3STSmtcD1Sik/pVRL4DrH/FZAilLKG/vxGiMEASlaaxtwN/bLhAoPIFsyotForTcrpeZhv07sUWAL9oO+zwEbHfN2YW86je094Dul1G3ASqDAgHWI0yCfwhaNSinVUmt9UikVAKwGxmmttzV1LtF0ZEtGNLZpSqlugB/wiTQYIVsyQghDyYFfIYShpMkIIQwlTUYIYShpMkIIQ0mTEUIY6v8B8F8NciCtEhMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "scores = np.array(results.mean_test_score).reshape(6, 6)\n",
    "\n",
    "# plot the mean cross-validation scores\n",
    "mglearn.tools.heatmap(\n",
    "    scores,\n",
    "    xlabel=\"gamma\",\n",
    "    xticklabels=param_grid[\"svc__gamma\"],\n",
    "    ylabel=\"C\",\n",
    "    yticklabels=param_grid[\"svc__C\"],\n",
    "    cmap=\"viridis\",\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Note that the range we pick for the parameters play an important role in hyperparameter optimization. \n",
    "- For example, consider the following grid and the corresponding results.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_heatmap(param_grid, pipe, X_train, y_train):\n",
    "    grid_search = GridSearchCV(\n",
    "        pipe, param_grid, cv=5, n_jobs=-1, return_train_score=True\n",
    "    )\n",
    "    grid_search.fit(X_train, y_train)\n",
    "    results = pd.DataFrame(grid_search.cv_results_)\n",
    "    scores = np.array(results.mean_test_score).reshape(6, 6)\n",
    "\n",
    "    # plot the mean cross-validation scores\n",
    "    mglearn.tools.heatmap(\n",
    "        scores,\n",
    "        xlabel=\"gamma\",\n",
    "        xticklabels=param_grid[\"svc__gamma\"],\n",
    "        ylabel=\"C\",\n",
    "        yticklabels=param_grid[\"svc__C\"],\n",
    "        cmap=\"viridis\",\n",
    "    );"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Bad range for hyperparameters "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'svc__C': array([1. , 1.2, 1.4, 1.6, 1.8, 2. ]),\n",
       " 'svc__gamma': array([ 10.        ,  15.84893192,  25.11886432,  39.81071706,\n",
       "         63.09573445, 100.        ])}"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "param_grid2 = {\"svc__C\": np.linspace(1, 2, 6), \"svc__gamma\": np.logspace(1, 2, 6), }\n",
    "param_grid2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQwAAAEGCAYAAACQF6v1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAArQElEQVR4nO2deXhV1bn/P+vkBKIhyBCGkABRgUiQKQ0gjrQgFESxiBdR+ygPXotWtNZeucX6c6pVyxWrpm1KBcEZiwOICo5UFDBMKmqKjQ0JCQlDgpAEQqb398daGU4SwpbkNCf0/TxPnpy911p7fdfaa3/3u/Y5e28jIiiKonjB19oCFEVpO6hhKIriGTUMRVE8o4ahKIpn1DAURfGMv7UFfF+iu4RJfO/w1pZRwzdfnNraEhSlxSniwH4R6VZ/fZszjPje4aSt6dPaMmqY0Gtoa0tQlBbnPVme1dh6nZIoiuIZNQxFUTyjhqEoimfUMBRF8YwahqIonlHDUBTFM2oYiqJ4Rg1DURTPqGEoiuIZNQxFUTyjhqEoimfUMBRF8cxJaRirPyhh4Pk7GTB6J488Wdggfe36w3Qe8C1J47JIGpfFAwsKANiVW87YK3IYdMFOBl+UxRN/PdAiepInDGNx+uMs+eZJps+9vEH6kIsSef3AUlK3zid163yuvXtaTdodi27i5fynWPjFoy2iRfW0TU2hoidod6saY3oDzwA9gSpgoYg8Xi+PAR4HJgGHgetFZGtz6q2sFObM28eaZbHExfgZNTGbS8dHkpjQPiDf+aMieOPZ2IB1fr9h/j3RJA2JoKi4ihETshl34akNyn4ffD4fc1JmMXf8A+zPKSQl7SE2rNxMdnpOQL7t69K5+7KHG5R/Z8laVqSs5s6lt5ywBtXTtjWFkp5gRhgVwB0iMhA4B/i5MSaxXp6JQH/3dyPw5+ZWmratlDPjwzmjbzjt2hmmT4li5ZoST2VjevhJGhIBQFQHH2f1b0dufkWz9CSM7MfujHzyM/dSUV7B2mWfcO6UZM/lt69Lp6iwuFkaVE/b1hRKeoJmGCKSVx0tiEgRkA7E1ss2BXhGLBuBTsaYmObUm5tfQe/Y2sApNsbf6EG/cUspw8dmMenqXL7acbRB+s5d5Xy2/SijkiKaI4fo2C7syymoWd6fU0h0bNcG+RJHDyB123wefHMefRPjmlWn6jm5NIWSnn/LA3SMMfHAcODTekmxwK46yzluXV698jdiIxD6xDYtubHXrBgTuJw0uD2Zm06nQ6SPt94vYerMPHasj69JLy6p4spZeSy4vxsdo8KarO941K/bagwUmbE1k2vib6a0pJSRE4dz32t3cn3Crc2qV/WcPJpCSU/QL3oaYzoArwC/EJFD9ZMbKdLgkBeRhSKSLCLJ3bo2fQDHxfjZlVsbUeTmVdCrR6DJdIwKo0OkbfqksZGUlwv7CyoBKC8Xps3K4+qpUUy9pMNx23c89uUU0i2u9mwQHdeFgt2BF2IPFx2htKQUgLS3txEWHkbHrlHNrlv1nByaQklPUA3DGBOONYvnReTVRrLkAL3rLMcBu5tT54hhEWRklpGZXU5ZmbBsRRGXTogMyJO/t6LGodO2lVJVBV27+BARbvjlHgb2b8ftszs3R0YNOzZlENs/hp7x3fGH+xkz/Tw2rNwckKdzj041nxNG9MPn83GooKhF6lc9bV9TKOkJ5rckBlgEpIvIgmNkWwncYox5CRgFHBSRvGPk9YTfb3jid92ZOCOXykqYeVVHBiW0J3XpdwDMvq4Tr6wqJnXpQfx+OCXCxwupPTHG8PGnR3hueRGDB7YjaZx9pOFvfx3NpLGRTdTYNFWVVaTMWcRDq+/CF+ZjzdMfkvV1DpN/djEAq/7yLhdOO4fJs8dTWVFJ2ZEyHpzxWE35ec/fxpAxgzgtOooXslN55t6XWb34A9UTJD2hqCmU9JhgvVvVGHM+sA7Yjv1aFWAe0AdARFKdqaQAP8Z+rTpTRDY3srkakodGiD4EWFGCy3uyfIuINPgqJmgRhoh8TOPXKOrmEeDnwdKgKErLclL+0lNRlOCghqEoimfUMBRF8YwahqIonlHDUBTFM2oYiqJ4Rg1DURTPqGEoiuIZNQxFUTyjhqEoimfUMBRF8YwahqIonlHDUBTFM2oYiqJ4Rg1DURTPqGEoiuIZNQxFUTyjhqEoimfUMBRF8YwahqIonlHDUBTFM2oYiqJ4Rg1DURTPqGEoiuIZNQxFUTyjhqEoimfUMBRF8cxJaRirPyhh4Pk7GTB6J488Wdggfe36w3Qe8C1J47JIGpfFAwsKANiVW87YK3IYdMFOBl+UxRN/PdAiepInDGNx+uMs+eZJps+9vEH6kIsSef3AUlK3zid163yuvXtaTdodi27i5fynWPjFoy2iRfW0TU2hoidoL2M2xiwGJgN7ReTsRtJPA57Dvs3dD/yfiDzd3HorK4U58/axZlkscTF+Rk3M5tLxkSQmtA/Id/6oCN54NjZgnd9vmH9PNElDIigqrmLEhGzGXXhqg7LfB5/Px5yUWcwd/wD7cwpJSXuIDSs3k52eE5Bv+7p07r7s4Qbl31mylhUpq7lz6S0nrEH1tG1NoaQnmBHGEuDHTaT/HPhaRIYCY4BHjTHtmltp2rZSzowP54y+4bRrZ5g+JYqVa0o8lY3p4SdpSAQAUR18nNW/Hbn5Fc3SkzCyH7sz8snP3EtFeQVrl33CuVOSPZffvi6dosLiZmlQPW1bUyjpCZphiMhHQMP5QJ0sQJQxxgAdXN7mHZ1Abn4FvWNrA6fYGH+jB/3GLaUMH5vFpKtz+WrH0QbpO3eV89n2o4xKimiWnujYLuzLKahZ3p9TSHRs1wb5EkcPIHXbfB58cx59E+OaVafqObk0hZKeoE1JPJACrAR2A1HAdBGpaiyjMeZG4EaAPrFNSxZprHzgctLg9mRuOp0OkT7eer+EqTPz2LE+via9uKSKK2flseD+bnSMCvPeoka1N6YxUGTG1kyuib+Z0pJSRk4czn2v3cn1Cbc2q17Vc/JoCiU9rXnRcwLwGdALGAakGGM6NpZRRBaKSLKIJHfr2vQBHBfjZ1dubUSRm1dBrx6BJtMxKowOkbbpk8ZGUl4u7C+oBKC8XJg2K4+rp0Yx9ZIOJ9i0WvblFNItrvZsEB3XhYLdgYHX4aIjlJaUApD29jbCwsPo2DWq2XWrnpNDUyjpaU3DmAm8KpYMIBM4q7kbHTEsgozMMjKzyykrE5atKOLSCZEBefL3VtQ4dNq2UqqqoGsXHyLCDb/cw8D+7bh9dufmSgFgx6YMYvvH0DO+O/5wP2Omn8eGlZsD8nTu0anmc8KIfvh8Pg4VFLVI/aqn7WsKJT2tOSXJBsYC64wxPYAE4F/N3ajfb3jid92ZOCOXykqYeVVHBiW0J3XpdwDMvq4Tr6wqJnXpQfx+OCXCxwupPTHG8PGnR3hueRGDB7YjaVwWAL/9dTSTxkY2UWPTVFVWkTJnEQ+tvgtfmI81T39I1tc5TP7ZxQCs+su7XDjtHCbPHk9lRSVlR8p4cMZjNeXnPX8bQ8YM4rToKF7ITuWZe19m9eIPVE+Q9ISiplDSY+rPhVoKY8yL2G8/ooE9wD1AOICIpBpjemG/SYkBDPCwiDx3vO0mD42QtDV9gqL5RJjQa2hrS1CUFuc9Wb5FRBp8FRO0CENEZhwnfTcwPlj1K4rS8pyUv/RUFCU4qGEoiuIZNQxFUTyjhqEoimfUMBRF8YwahqIonlHDUBTFM2oYiqJ4Rg1DURTPqGEoiuIZNQxFUTyjhqEoimfUMBRF8YwahqIonlHDUBTFM2oYiqJ4Rg1DURTPqGEoiuIZNQxFUTyjhqEoimfUMBRF8YwahqIonlHDUBTFM2oYiqJ4Rg1DURTPqGEoiuKZk9IwVn9QwsDzdzJg9E4eebKwQfra9YfpPOBbksZlkTQuiwcWFACwK7ecsVfkMOiCnQy+KIsn/nqgRfQkTxjG4vTHWfLNk0yfe3mD9CEXJfL6gaWkbp1P6tb5XHv3tJq0OxbdxMv5T7Hwi0dbRIvqaZuaQkVP0N6taoxZDEwG9orI2cfIMwb4A/YlzftF5KLm1ltZKcyZt481y2KJi/EzamI2l46PJDGhfUC+80dF8MazsQHr/H7D/HuiSRoSQVFxFSMmZDPuwlMblP0++Hw+5qTMYu74B9ifU0hK2kNsWLmZ7PScgHzb16Vz92UPNyj/zpK1rEhZzZ1LbzlhDaqnbWsKJT3BjDCWAD8+VqIxphPwJ+AyERkEXNkSlaZtK+XM+HDO6BtOu3aG6VOiWLmmxFPZmB5+koZEABDVwcdZ/duRm1/RLD0JI/uxOyOf/My9VJRXsHbZJ5w7pcFLsY/J9nXpFBUWN0uD6mnbmkJJT9AMQ0Q+AhrOB2q5GnhVRLJd/r0tUW9ufgW9Y2sDp9gYf6MH/cYtpQwfm8Wkq3P5asfRBuk7d5Xz2fajjEqKaJae6Ngu7MspqFnen1NIdGzXBvkSRw8gddt8HnxzHn0T45pVp+o5uTSFkp6gTUk8MAAIN8asBaKAx0XkmcYyGmNuBG4E6BPbtGSRxsoHLicNbk/mptPpEOnjrfdLmDozjx3r42vSi0uquHJWHgvu70bHqDDvLWpUe2MaA0VmbM3kmvibKS0pZeTE4dz32p1cn3Brs+pVPSePplDS05oXPf3AD4BLgAnA3caYAY1lFJGFIpIsIsndujZ9AMfF+NmVWxtR5OZV0KtHoMl0jAqjQ6Rt+qSxkZSXC/sLKgEoLxemzcrj6qlRTL2kwwk3rpp9OYV0i6s9G0THdaFgd2DgdbjoCKUlpQCkvb2NsPAwOnaNanbdqufk0BRKelrTMHKA1SJSIiL7gY+Aoc3d6IhhEWRklpGZXU5ZmbBsRRGXTogMyJO/t6LGodO2lVJVBV27+BARbvjlHgb2b8ftszs3VwoAOzZlENs/hp7x3fGH+xkz/Tw2rNwckKdzj041nxNG9MPn83GooKhF6lc9bV9TKOlpzSnJCiDFGOMH2gGjgMeau1G/3/DE77ozcUYulZUw86qODEpoT+rS7wCYfV0nXllVTOrSg/j9cEqEjxdSe2KM4eNPj/Dc8iIGD2xH0rgsAH7762gmjY1sosamqaqsImXOIh5afRe+MB9rnv6QrK9zmPyziwFY9Zd3uXDaOUyePZ7KikrKjpTx4Izabpj3/G0MGTOI06KjeCE7lWfufZnViz9QPUHSE4qaQkmPqT8XaimMMS8CY4BoYA9wD/brU0Qk1eX5H2AmUAU8JSJ/ON52k4dGSNqaPkHRfCJM6NXsoEhRQo73ZPkWEWnwVUzQIgwRmeEhz3xgfrA0KIrSspyUv/RUFCU4qGEoiuIZNQxFUTyjhqEoimfUMBRF8YwahqIonlHDUBTFM2oYiqJ4Rg1DURTPqGEoiuKZJg3DGNPPGHNeI+svMMacGTxZiqKEIseLMP4ANHaP7BGXpijKfxDHM4x4Efmi/koR2QzEB0WRoighy/EMo6kHWp7SkkIURQl9jmcYm4wx/11/pTFmFrAlOJIURQlVjvc8jF8ArxljrqHWIJKxT8j6SRB1KYoSgjRpGCKyBzjXGPNDoPplRG+KSPOegaYoSpvE0xO3RORD4MMga1EUJcTRH24piuIZNQxFUTyjhqEoimfUMBRF8YwahqIonlHDUBTFM2oYiqJ4Rg1DURTPnJSGsfqDEgaev5MBo3fyyJOFDdLXrj9M5wHfkjQui6RxWTywoACAXbnljL0ih0EX7GTwRVk88dcDLaInecIwFqc/zpJvnmT63MsbpA+5KJHXDywldet8UrfO59q7p9Wk3bHoJl7Of4qFXzzaIlpUT9vUFCp6gvZuVWPMYmAysFdEzm4i3whgIzBdRJY3t97KSmHOvH2sWRZLXIyfUROzuXR8JIkJ7QPynT8qgjeejQ1Y5/cb5t8TTdKQCIqKqxgxIZtxF57aoOz3wefzMSdlFnPHP8D+nEJS0h5iw8rNZKfnBOTbvi6duy97uEH5d5asZUXKau5cessJa1A9bVtTKOkJZoSxBPhxUxmMMWHAI8Calqo0bVspZ8aHc0bfcNq1M0yfEsXKNSWeysb08JM0xN7RH9XBx1n925GbX9EsPQkj+7E7I5/8zL1UlFewdtknnDulwUuxj8n2dekUFRY3S4PqaduaQklP0AxDRD4CGs4HApkDvALsbal6c/Mr6B1bGzjFxvgbPeg3bill+NgsJl2dy1c7jjZI37mrnM+2H2VUUlOPBDk+0bFd2JdTULO8P6eQ6NiuDfIljh5A6rb5PPjmPPomxjWrTtVzcmkKJT1Bm5IcD2NMLPYW+R8BI46T90bgRoA+sU1LFmmsfOBy0uD2ZG46nQ6RPt56v4SpM/PYsT6+Jr24pIorZ+Wx4P5udIwK89CaprQ3pjFQZMbWTK6Jv5nSklJGThzOfa/dyfUJtzarXtVz8mgKJT2tedHzD8BcEak8XkYRWSgiySKS3K1r0wdwXIyfXbm1EUVuXgW9egSaTMeoMDpE2qZPGhtJebmwv8DKKC8Xps3K4+qpUUy9pMP3bFJD9uUU0i2u9mwQHdeFgt2BgdfhoiOUlpQCkPb2NsLCw+jYNarZdauek0NTKOlpTcNIBl4yxuwEpgF/MsZc3tyNjhgWQUZmGZnZ5ZSVCctWFHHphMiAPPl7K2ocOm1bKVVV0LWLDxHhhl/uYWD/dtw+u3NzpQCwY1MGsf1j6BnfHX+4nzHTz2PDys0BeTr36FTzOWFEP3w+H4cKGnv2suoJtp5Q1BRKelptSiIip1d/NsYsAVaJyOvN3a7fb3jid92ZOCOXykqYeVVHBiW0J3XpdwDMvq4Tr6wqJnXpQfx+OCXCxwupPTHG8PGnR3hueRGDB7YjaVwWAL/9dTSTxkY2UWPTVFVWkTJnEQ+tvgtfmI81T39I1tc5TP7ZxQCs+su7XDjtHCbPHk9lRSVlR8p4cMZjNeXnPX8bQ8YM4rToKF7ITuWZe19m9eITf36R6ml7mkJJj6k/F2opjDEvAmOAaGAPcA8QDiAiqfXyLsEaxnG/Vk0eGiFpa/q0tNwTZkKvoa0tQVFanPdk+RYRafBVTNAiDBGZ8T3yXh8sHYqitBwn5S89FUUJDmoYiqJ4Rg1DURTPqGEoiuIZNQxFUTyjhqEoimfUMBRF8YwahqIonlHDUBTFM2oYiqJ4Rg1DURTPqGEoiuIZNQxFUTyjhqEoimfUMBRF8YwahqIonlHDUBTFM2oYiqJ4Rg1DURTPqGEoiuIZNQxFUTyjhqEoimfUMBRF8YwahqIonlHDUBTFM2oYiqJ45qQ0jNUflDDw/J0MGL2TR54sbJC+dv1hOg/4lqRxWSSNy+KBBQUA7MotZ+wVOQy6YCeDL8riib8eaBE9yROGsTj9cZZ88yTT517eIH3IRYm8fmApqVvnk7p1PtfePa0m7Y5FN/Fy/lMs/OLRFtGietqmplDRE7R3qxpjFgOTgb0icnYj6dcAc91iMXCTiHze3HorK4U58/axZlkscTF+Rk3M5tLxkSQmtA/Id/6oCN54NjZgnd9vmH9PNElDIigqrmLEhGzGXXhqg7LfB5/Px5yUWcwd/wD7cwpJSXuIDSs3k52eE5Bv+7p07r7s4Qbl31mylhUpq7lz6S0nrEH1tG1NoaQnmBHGEuDHTaRnAheJyBDgAWBhS1Satq2UM+PDOaNvOO3aGaZPiWLlmhJPZWN6+EkaEgFAVAcfZ/VvR25+RbP0JIzsx+6MfPIz91JRXsHaZZ9w7pQGL8U+JtvXpVNUWNwsDaqnbWsKJT1BMwwR+QhoOB+oTV8vItUx/0YgriXqzc2voHdsbeAUG+Nv9KDfuKWU4WOzmHR1Ll/tONogfeeucj7bfpRRSRHN0hMd24V9OQU1y/tzComO7dogX+LoAaRum8+Db86jb2KLdIXqOUk0hZKeoE1JviezgLePlWiMuRG4EaBPbNOSRRorH7icNLg9mZtOp0Okj7feL2HqzDx2rI+vSS8uqeLKWXksuL8bHaPCPDeice2NaQwUmbE1k2vib6a0pJSRE4dz32t3cn3Crc2qV/WcPJpCSU+rX/Q0xvwQaxhzj5VHRBaKSLKIJHfr2vQBHBfjZ1dubUSRm1dBrx6BJtMxKowOkbbpk8ZGUl4u7C+oBKC8XJg2K4+rp0Yx9ZIOJ9iqWvblFNItrvZsEB3XhYLdgYHX4aIjlJaUApD29jbCwsPo2DWq2XWrnpNDUyjpaVXDMMYMAZ4CpohIwfHye2HEsAgyMsvIzC6nrExYtqKISydEBuTJ31tR49Bp20qpqoKuXXyICDf8cg8D+7fj9tmdW0IOOzZlENs/hp7x3fGH+xkz/Tw2rNwckKdzj041nxNG9MPn83GooKhF6lc9bV9TKOlptSmJMaYP8CrwUxH5pqW26/cbnvhddybOyKWyEmZe1ZFBCe1JXfodALOv68Qrq4pJXXoQvx9OifDxQmpPjDF8/OkRnltexOCB7UgalwXAb38dzaSxkU3U2DRVlVWkzFnEQ6vvwhfmY83TH5L1dQ6Tf3YxAKv+8i4XTjuHybPHU1lRSdmRMh6c8VhN+XnP38aQMYM4LTqKF7JTeebel1m9+APVEyQ9oagplPSY+nOhlsIY8yIwBogG9gD3AOEAIpJqjHkKuALIckUqROS4l36Th0ZI2po+QdF8IkzoNbS1JShKi/OeLN/S2PEYtAhDRGYcJ/0G4IZg1a8oSsvT6hc9FUVpO6hhKIriGTUMRVE8o4ahKIpn1DAURfGMGoaiKJ5Rw1AUxTNqGIqieEYNQ1EUz6hhKIriGTUMRVE8o4ahKIpn1DAURfFMqDyizzNf7u/GWYtvam0ZNfRlfWtLUJR/GxphKIriGTUMRVE8o4ahKIpn1DAURfGMGoaiKJ5Rw1AUxTNqGIqieEYNQ1EUz6hhKIriGTUMRVE8o4ahKIpn1DAURfGMGoaiKJ5pc3ereuGCM/py18VjCDM+/vb5lyzcsCkgfWSfOP487TJyDh4E4J0dGfzx408B+N0lF/PDfmdQcPgwk//6bIvoSZ4wjJv/MBNfmI+3F73PskdeD0gfclEi978+l/zMvQB8/NqnPPfAcgDuWHQToy75Ad/tPciNQ+5QPf8GPaGoKVT0BM0wjDGLgcnAXhE5u5F0AzwOTAIOA9eLyNbm1uszhnsm/IiZL75K/qEiXpl5Ne//81u+3V8YkG/zrlx+9rcVDcq/+sXXPLf5c35/2YTmSrF6fD7mpMxi7vgH2J9TSEraQ2xYuZns9JyAfNvXpXP3ZQ83KP/OkrWsSFnNnUtvUT3/Bj2hqCmU9ARzSrIE+HET6ROB/u7vRuDPLVHpkF49yTrwHbu+O0h5VRVvfr2Dcf3P9Fx+865cDpaWtoQUABJG9mN3Rj75mXupKK9g7bJPOHdKsufy29elU1RYrHr+TXpCUVMo6QmaYYjIR0BhE1mmAM+IZSPQyRgT09x6e0R1IP9QUc1yflExPaI6NMg3LDaGlbOu5anpl9Mvumtzqz0m0bFd2JdTULO8P6eQ6NiG9SWOHkDqtvk8+OY8+ibGqZ5W0hOKmkJJT2tew4gFdtVZznHr8upnNMbciI1C8J/WucmNmkbWCRKw/FX+Xn74x0UcLi/nojPj+dO0SxmfuuR7ifeKaUSQSKCejK2ZXBN/M6UlpYycOJz7XruT6xNuVT2toCcUNYWSntb8lqTxY7uxlSILRSRZRJLDIiOb3Gh+UTE9O0bVLPeM6sDeopKAPCVlZRwuLwfg79/uxO/z0fmUiO8p3xv7cgrpFld7NoiO60LB7sDA63DREUpL7DQo7e1thIWH0bFrFMFA9bQ9TaGkpzUNIwfoXWc5Dtjd3I1u351PfOfOxJ3WkXCfj0sSE3j/n/8KyBMdeWrN5yExPfAZw4EjLXfdoi47NmUQ2z+GnvHd8Yf7GTP9PDas3ByQp3OPTjWfE0b0w+fzcaigiGCgetqeplDS05pTkpXALcaYl4BRwEERaTAd+b5UinD/Ox+w6KqphPkMyz//ioz9BVw1fAgAL237gh+f1Z8ZSUOprKqitKKC219/q6b8gikTGdm3N51PieCjW27giXUbWP75Vyesp6qyipQ5i3ho9V34wnysefpDsr7OYfLPLgZg1V/e5cJp5zB59ngqKyopO1LGgzMeqyk/7/nbGDJmEKdFR/FCdirP3Psyqxd/oHqCpCcUNYWSHlN/LtRSGGNeBMYA0cAe4B4gHEBEUt3XqinYb1IOAzNFZHPjW6slIra39Pn5L4Oi+UTo+xt9arhy8vGeLN8iIg2+iglahCEiM46TLsDPg1W/oigtj/40XFEUz6hhKIriGTUMRVE8o4ahKIpn1DAURfGMGoaiKJ5Rw1AUxTNqGIqieEYNQ1EUz6hhKIriGTUMRVE8o4ahKIpngna3arAwxuwDslpgU9HA/hbYTksSappUT9OEmh5oOU19RaRb/ZVtzjBaCmPM5sZu321NQk2T6mmaUNMDwdekUxJFUTyjhqEoimf+kw1jYWsLaIRQ06R6mibU9ECQNf3HXsNQFOX7858cYSiK8j1Rw1AUxTsi0qb+gMXAXuDLOuvuBXKBz4AC4Lt66V2Ad4F9QDHwNfAiEFFv23/FvkzpX8D/Yp9yvhTYDqQDv66TdzrwBbADyHbpXwHLXP584Khb/xn2pdOrnbZV9dpzADgCfOnq8wNXYt/dIsA/gfXA0EY07a7eHjDDbaMMqHT1f+LS7nHrxPVBxzoa5gMVrtztrh2FQJHTVQU84dZXuXqrXJ9/BbwGZADlwCFX5rBr6xdOa5lr54fAx0Cp07LZ7c8jbnuHXBuKXR2Zrv+SgX+4NlRi35D3jctXBJS4+o+6/Tvd9V+Va1sV9vcJnwFPu/qOuPoKgT+4vphdp55MILHe/k539aW49Y+5bX7m2ueeb01fYItb/62rf1qdPr8dO1YrqTMWqR2rB12ffe3WdwXWubZ85/J0rqMtz7W9EJhQpy3bnYaPgdGuXKZr93a3/2Z7Pv5a2wBOwDAuBJJoaBi/aiL998DvXEf9BngEeBn7xvjqPH2xgzwXiAE+B/4HeMmlnwrsBOLdzssGurm8bwBjgSjsgZPoNC0GVtepYyxwKYGGcRHWXL5xy/cDs4CBwFVuRydjX179KXB1HU1zsQfM+1iT2Yt9Z+2HwKNuW1uAaa5td7pB9BHwQB0Ny4BX3QDNBE4BOgKrXH99jX2PTALwCTATeBj4ldNW5volB/ujujC33QVY4/29S7vQ6VkLDAHSnJaOQHeXdhuQ6tp9EDjVbetv2NdVnIG9sJcDPAi8hT2A5mIPhhucts0uTzr2xeB1D9a6Znkr9kRyYXUa8IqrbzHW5Ovu78exJ5TX643LZNeecrfcDmgPhAF/d/3/3y4tFjvOXnD7r2Ysur76X9dXTwD73PpI4HmsOae4PI84bbuxJtvead/t6q3bzsuoPRG2A+5y5Ttgx3UvL8dfm5uSyHFe8nyM9CnYAeDHdujlWAOo+6a1p7FOXO7+XsIaT6Qxxo89iMqwznwG9gDfJ/blSy8DV4hIEdaxY90221Hn9Y8i8j727FSXr7Fn2zK3/K7bVrqIvIQ9owBsxL4dTpymvtio5YDLY9zff2MPpEisgRjXfoONJEqxA+cKAGPMD9w2V7p6qtt62C2nYaeuB0Vkh+ub7W471X172OmvwJ5NR7r3zpzitj3N1f8ZzmRF5AuswW4TkUMishd74A9wZboB60XksNvWxcDnIvIvt68OA9dizTISewb/O7AGe3B8DjT6/ksROVRnsY/Tuc4t/8j1z1fU7r8zsNFMH6AH8B5wZvUGjDFhrm+N6wNEpExEjgJzgDedvmp82AP9AbdcdyxOAZa6cfws1sAQkRKsKa11+ZZix/EZri+ed/W97jSMrNfOSKwpLxWRMqwZXo41Ge8+0NoRwwlGGfE0jDB2YkPGxdizV93079z/26gNdZ+v575vAk+57UQDPwX+hDWOfdiQ90aXvzP27BVPrQm94ZazgTuwplWBDW8XUxs+jiEwwjBuWxlu+XFge530tdiB8iunL9xpOooNqf+P2inJNGyIW0Rt6L4UO4hysWfbb7FnoyI3UNZiX1l5Pfagr+6jfS7vHmy4e3U9PQuwg7wcWOLSMrFh/yH39yH2YKjWOQxrQEeBbW7bG10fprj9Un0mP+D2xWfAQ07DU66e3m65BHuAl7vlA05XMfagP+D6oHq6lYGdQrTHmuoup+GPdc7iG7DThUKsgfWndn9vxO7jzUBmnX10GzaaywOK60UdxdgDeiMuynH5X3NpQuBY/K7eOK+sm+b2U/V06IDTVgz8gtqxmF2nrp+7/bgLOFRnW71dvxwGfu712GtzEcYx+DPW8Ydhd9pv6mcwxnTGuvfp2FA30hhzrTHmVGx49nIj2+2OHXC9XLk7jDFniMgB4CZsKL8OO7AFu7N+ISKPYqcU12GnEXnYKUIDxO69OUCMMSYNeyBX1Mv2A+w0ZS4wEuiJPcv2Bf4LONUYE+40fevS/owNxf8LO6i+xg769tiDoBy4GXhLRHbVqau6j3phI4lfuf46r56mQ9iD8zngAmNMD5dnBfaAyMQO8OnYM/WL2AGdBBwVkeHYOXwX7AHYGxv+r8WGzVVO5wXAOdgooC6nYA1gM/ZVm9tdmVtdfRuw5tsdOy52Ys/WPYC5InKXiFS/2zfc/b8PeExEHsNOB94EfuP29yrXJy9iTbEKwBjTC3u96QiwvJ7GucA4oB92fJ7m8l8FdHL9XIIbi5wATtvfgVuoHYtVuMhWRP4oImc6LRF1yu3CjrV+wHVu/x2Xk8IwRGSPiFSKSBX2wuXQeln2YM++mdTO9V8FzsXuyNOx895rsGH/VuAs7NRitYiUu5D5E+xZAxF5Q0RGicho7MAdjD1TvFqtCTu4LneaRjbRhG3YM9ZI7Jz+n3XSIoG7gSkiUoC9hnEImIw9W0djD9TqKcW/XNtexl5fOYI9yz8vIhdjzbQQa2Kjse+33YmNAMKBrmKnWuVuO6Ndv406hvZvsGe6C0Rkt+u/f2DPoHnYM2IlNhrJceuq36G7EygSkWEiMgV7EC11bXzF9XuRW3cKdkqAqyMCO2U5JCJbRWS8iHTFXg/6Ehu5pYlIgYh8ib1eUIQ9uEYCGGOGYg/Yc912RwG/d/3xC+x1o/9yadWvQo9x/R1njHkYGI496O7FjrFTjTEZLm8yNhpc79q2AGvqZ7m0TW7fjK6jYY8xJsZ97kbgyWMPcJrTHoMdx2Cv9yxyY3EH1izqv9j8JSCsetvV5d0++wprzMflpDCMOh0M8BPsIK7LSuxZ5hzsRbEV2AuQ6SKyXUS6Uxvm5mMHzmTsWepHxhLpyv/D1dnd/e+MDW//LiILjDH962i6BHvw/wQ7iI9FV1emPfZMkOqW+wBnA/9PRKrblI09M/cGBmEHzUZsBJLoNE/Ezvn3YQ3nDbcO7MCPBlJF5BoR6SMi8dhIogLwG2NONcb0q+4jV1fdAdid2tdsrsWaZZ4x5mxsZPUlMN7li8AeGNeJSL6r4yNXNtaV62+MudilJWIjhDddH4S7ffElMNQYczq14XeKMaanMaa7MSbMbaMTdgo2FNjh1p+Bvdh8GnYc5Lr6Z2APsH8AiMgFwMWuP/7gtlP9Ju7bRaQP1iAOAH8Tkf8VkTexF67zsBHIYRHpZ4yJw37DEu/KHAbuE5G7se8TznZtPYyNrNJdPSuxkSlYA6p7HWIlcL77fB12HIM1jKtclFDdN2nVY9FxCXY8XOe0zQJWuPF7nuuH4xOMawzB/MOetfOwIXWOa/iz2JD0C7duj0vf7dZ1xV4cK8CeUdJdmXhsSF697UmuXCZ2mtIBe7H0K2xI/z/1dHzt8oqr5zO3U7KwZ/FityNWYs9M66j9Ki0HmOC2Uz2XLQdedNv/idNavf4QNvyur+nP1F7DmI2NMCpcHUew0xODvZ4g7q/C1Z+IHeTfOr3i6tnv6j7k6q7Anp2rv7asDnkrsdcGVrk2l9b524O9qp+JNdr3seb5qeuT6ouk1ddait1+eQ9r2ltc+4qwU4t+dcqVAetcm2+h9ivFKuw0aCP2ANnptFQ67VnYKdTrWAM66volts5+fdz1bT42chxUb39X93lKnTL3Ag+7z8Xu/8XYMfG5+7+ewG9q7qP269tngfZuffVYLXL66o7zbLdc5do1uo62va5fdgMT67TlgOu3D7FRzPtue0WuD77AXZvz8qc/DVcUxTMnxZREUZR/D2oYiqJ4Rg1DURTPqGEoiuIZNQxFUTyjhqEoimfUMBRF8Yz/+FmU/1SMMXdjfy6/C/tjri3Y+0puxP4aMwP4qdg7Spdgfyh2FvYel5nYXyOOBj4VkevdNouBP2LvsTgAzMPe0t0Hex/OSmNMPPbHTJFOyi0isj7IzVU8oBGG0ijGmGTsLfDDgam4e2iAV0VkhIgMxf4yc1adYp2xd4/ejv05+mPYn68PNsYMc3kigbUi8gPsrw1/i/1V5E+wd3yC/dXixSKShL157YlgtFH5/miEoRyL84EVInIEwBjzhlt/tjHmt9h7Njpgnz9RzRsiIsaY7cAeEdnuyn6F/Rn+Z9ifL692+bdj71wtd2Xi3fpw7H0iw7A/nR4QhPYpJ4AahnIszDHWLwEuF5HPjTHXY5/vUc1R97+qzufq5eqxVi619yPU5BORKvegIrARyh7sDWQ+ah/Wo7QyOiVRjsXHwKXGmAhjTAfszVxg73bNc3eRXhOkuk8D8sQ+ruCn2MfNKSGARhhKo4jIJmPMSuzdllnYO2UPYp/N8albt53a50S0JH8CXjHGXIm9y7IkCHUoJ4DeraocE2NMBxEpdk8l+wh7G/TW1taltB4aYShNsdAYk4h9CM5SNQtFIwxFUTyjFz0VRfGMGoaiKJ5Rw1AUxTNqGIqieEYNQ1EUz/x/+6PKiNmz3nUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_heatmap(param_grid2, pipe_svm, X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Different range for hyperparameters yields better results! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQwAAAEGCAYAAACQF6v1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABHfUlEQVR4nO3dd3hUZd7/8fc9M+mkdxIIQiAQpIcmoqiINAV27T4WFhcRRV1YZMVFsaLu87jubkRWRBBFQZGmdCnSe+jNQEhIm/Q6Mykz9++PiYnDJHDIJBvd3/26Li4yc77nnM+UfM99zmTOEVJKFEVRtNC1dABFUX47VMNQFEUz1TAURdFMNQxFUTRTDUNRFM0MLR3gehn8vaV7WEBLx6gV7lnS0hEcBOuqWzqCg7PmgJaO4ER/4df1HEmbraUjOCmlME9KGXrl/b+5huEeFkDHv09o6Ri1psZtaekIDh71zWvpCA5uOTm2pSM48RlrbOkIDmwmU0tHcPKDXJ5a3/1ql0RRFM1Uw1AURTPVMBRF0Uw1DEVRNFMNQ1EUzVTDUBRFM9UwFEXRTDUMRVE0Uw1DURTNVMNQFEUz1TAURdFMNQxFUTT7zX35TIubQmOZ0XUUOiFYmXaYTy/sdKpJCG7H9PiRuOn0FFaWM2HvpwD4Gjx5tcdYYn3DkBJePbaS40WXXcpzfEchX751CZtVcst94Yx+Ksph+rpPMti7xv6lMZtVknnBzL/2JdAqwI0FLyVzdFshfsFuvLW2p0s5frZhazl/eiUXqxUmPOzHjClBDtP/d24hX66wfwu3uhrO/FSJ8WR7ggL1tO+bgm8rHXo9GPSCAxvbupynf3Annu80Bp0QfJ9xgC9StzvV9Apsz3Od7sEgdBRVmZhyeB5tvEN5vdsjtTWtvYL45MImvrm8y+VMCUO7Mem9R9Drdaz/7Ee+fn+tw/Tugzsze+nzZKfmArB7zWGWvLOa0Kggps+fSGC4P9ImWbdwG6vmbnY9z109mfzBeHR6HesXbGHZu6sc89waz+urZpCdkgPArpX7+eKN5QBMW/A0/Uf1oSinmIndp7mUo9kahhCiDbAYiABswMdSyn9cUSOAfwAjARPwhJTyiCvr1SGYeePdPLV/EUZzCV8OnsR241kuluXW1vgaPJl5491M3r+YbEsxQe4+tdNe7DqS3Tk/8efDSzEIPV56N1fiYLNKPn8thekL4wmKcOe135+g1x2BRMV619aMfDKKkU/am0jS1gI2LcqiVYB9vTf/Low7/ieC+S8mu5TjZ1arZMrMXDYuiyI60kD/EWncPcyH+DiP2po/Tw7kz5MDAfhuUxn/+LiIoEB97fQty6MJCdY7LbsxdAimxo3jT0nzybEU80m/KezKO82l8pzamlYGT6bGjePPSQswVhQR4GZ/vS6bchm//4Pa5awc/Fd25J50PZNO8Mz7j/HSPe+Rl1HAv3bMZt+6JNLOZjrUndxznlfu+7vDfdZqKx+/9BXJx1LxauVJ4s7XOLL1lNO815dHx5TECcwY9gZ56QUkHpjD3jWHSDuT7lB3YucZZt3zjtP8mxZtZ3XiBl787NlGZ6jN4vISGlYNTJNSdgEGAM8IIeKvqBkBdKz5NxH4yNWV3hgQzeXyfDJMhVRLKxsyTjAkvIvjSqO6syX7NNmWYgAKKssB8DF40Ce4HSsvH7Y/AGmltNriUp6Lx8sIj/EkrK0nBncd/UeFkPRDYYP1+7/Po/+okNrbcX398PFvur5+IMlCh3ZutI9xw91d8MAYX9ZsLG+wfumqUh4Y69tk679SF/82pJvzyDQXUC2t/GA8xs2hXR1q7ozoxY7ckxgrigAoqnLO2ycolgxzPkZLkcuZ4hLak3nRSPalXKqrrGxfvp+Bo3prmrfAWEzyMfs3w81lFi6fyyQkMtC1PP1iyUzOJjslh+qqarYv281NYxI0z39i5xlKC8pcyvCzZmsYUsqsn0cLUspS4AwQdUXZGGCxtNsHBAghIl1Zb5iXX20jAMixFBPu5fiGj/EJxs/Ni08G/oGvbp7E6KieAER7B1JYWc7rPcaxbPBkXu0+xuURRqGxkqCIuq13YIQ7hcaKemsrzFZO7Cwi4a6geqc3hYzsatpE1TWgqEgDGdn1n1DGZLKxcZuJ349qVXufEDD8wQz6Dkvj48+L653veoR6+JPzi9cr11JMqIefQ00b7xB8DV78q89TLOj3HMMjnX95h0b05Ifsoy7nAQhuHUhuekHt7byMAkJaO//Sd+kXy0d73+DNFdOI6XLlWxvC24bQoUcMZw9dcClPSFQQuen5dXnSCwiJCnaqix/YiXlJf+OttTOJiY92aZ0N+Y8cwxBCtAN6AfuvmBQF/PIAQXrNfVlXzD8R+wgEt1DHN5PTuuq578pLrxh0OuL9WzNx30I8dG4svnkiJ4ouoxc6OvtF8s7JtZwoSufFriP5Q4db+PB840+SU+9lX0R9KeHo1kJie/vV7o40h/ryNBCH7zaXc1NfL4fdkZ1r2tA6wkBOXjV3PZBB51h3bhno1eg89b5eV9zWCx1xflE8f/hjPPRuzOv7LKeK07hssh/3MQg9g0LimZe8vtE5HDLV84Rcef2e5KOXeDR+KpbyCvoO686rXz3HH3rOqJ3u6ePBrCVTmDdjCaZS10ap9b0+TnmOpPBIu8lYyi30G9GL11a+yBNxz7m03vo0+6ckQohWwLfAC1LKK89np+X9gpTyYyllgpQyweDvU88sdYzmEiI8/Wtvh3n6k2MpdarZnfsTZmsVRVUmjuRfopNfBEZLCUZLCSeK7PuGm7NO0dnfpQEPQRHuFGTXjSgKsysJDHOvt3b/2jwGjHbecjSl6EgDlzPqRhQZWdW0Dq9/u7FsVSkPjm3lcF/rCHttWIiBsSNacfCoa78MORXFhP3i9Qr19CevwvFtkmspZn/+eSy2KoqrTBwrvEhsq7rXZUBIHOdLMyisbJphd15GAaHRdaO8kKgg8rOKHGpMpRYs5fbX9eCm4+jd9PgF258rvUHPrCVT2LpsD7vXHHY5T256AaHRde+LkOgg8jMLHGpMpWYs5fbX4sD6pJo8Tb8r2awNQwjhhr1ZLJFSrqinJB1o84vb0UDjjw4Bp4ozaOsTTJRXAAahZ3hUN340nnWo2WY8S++gGPRCh6fOjW4B0aSU5ZJfUYbRXEyMj/0YQv+Q9g4HSxvjhm6tMF6ykHvZQnWljf1r8+h1h/Pw1lRazbmDJfS+o/l2RwD69vQkOaWSlLQqKisly1aXcvddzk24uMTKjn1mxgyvaxjlJhulZbbanzf/aKJrXP3NT6uzJem08Qoh0jMQg9AzNLwHu3NPO9TszD1N94B26IUOD50b8f5tHQ6KDg1vut0RgHOHU4jqEE54TAgGNz1D7u3PvnVJDjWBYXVNLq5Pe3Q6HSX59oY1de4ELp/LZEXixqbJczCZqI6RRLQLw+BmYMgDg9i75pBjnvCAujx9Y2vylNLUmvNTEgEsAM5IKd9voGwN8KwQYinQHyiWUmY1UKuJVdqYc+p7Pur/ODqhY9XlI1woy+G+tn0B+CbtIClluezO+YlvbnkGKSUrLh8mudT+Bnzn1Frm9LoXN52edFMhrxyrr89ppzcI/ueVG/jfCWewWSWD7w0jqqM3W7/KBuD2hyIAOLy5gK6DAvDwdvz04aM/nefsgRLKCqv50+DDjH0umlvvC290HoNB8M+3wxjxUAZWK4x/0I+ucR7M+6wIgEmPBwCwcn05d97qjY933TbFmGvl93+w9/PqanhonC/Db7/6iO9arNLG++dW836vJ9EJHWszD5JSbmRM1AAAVmfsI9WUw/788yzq/yckku8yDpBSbj8vp4fOjb5BHfnbGddep1+yWW18OO1z3l41HZ1ex6bPd5B6JoNRE24DYO2CbQwe15fRT96OtdpKhbmSOU/MBaDrwI4MfXgQF09eZu6e1wFYOHs5BzcddylP4pQFzNnwMjq9jo0Lt5F6Op3RT90JwPf/3swt9w5g9KRhWKutVJoreeuhuk9vZi55nu5DuuIf4suXafNYPPtrNny6tVFZRHNdW1UIcTOwEziB/WNVgJlAWwAp5byappIIDMf+sep4KeWhehZXy7tja6lOAtwwdRLga1MnAb62H+Tyw1JKp49imm2EIaXcRf3HKH5ZI4FnmiuDoihNS/1puKIomqmGoSiKZqphKIqimWoYiqJophqGoiiaqYahKIpmqmEoiqKZahiKomimGoaiKJqphqEoimaqYSiKoplqGIqiaKYahqIomv3mLjMgJdhsv54+V27zuHbRf1CVrP/8nIrSFH49v3mKovzqqYahKIpmqmEoiqKZahiKomimGoaiKJqphqEoimaqYSiKoplqGIqiaKYahqIomqmGoSiKZqphKIqimWoYiqJophqGoiia/ea+rarFoNBYZnQbgV4IVqQeYUHyLqeahOB2zLhxOAahp6jSxPg9CwHwNXgyu+c9dPQNQwKvHF3FscJ0l/Kc2pnP8reSsdkkg+6NZNjEGIfpmxekcfA7+wWCbVZJ9gUT7+4ZhE+AG5/PPMvJ7fn4Brvx1+/6uZTjZ5u2mZg2qwCrTTL+IV+mTwlwmP7+3CKWrigHoNoqOftTFekn2hIUaL+yvNUquWl4Jq0j9axcHOFynv7BnXi+0xh0QvB9xgG+SN3uVNMrsD3PdboHg9BRVGViyuF5tPEO5fVuj9TWtPYK4pMLm/jmsvPrfb0ShnZj0nuPoNfrWP/Zj3z9/lqH6d0Hd2b20ufJTs0FYPeawyx5ZzWhUUFMnz+RwHB/pE2ybuE2Vs3d7Hqeu3oy+YPx6PQ61i/YwrJ3VznmuTWe11fNIDslB4BdK/fzxRvLAZi24Gn6j+pDUU4xE7tPcylHszUMIcSnwGggR0p5Yz3T/YEvsF/N3QD8r5Ryoavr1SF4ufsoJu5dTLa5hKW3TGRb9jkuluXW1vgaPPlr91FM2vcF2eZigtx9aqfN6DaC3TnJTDv0NQahx0vv5lIem1Xy9es/MeXTHgSEe/DefYfpdnsIkbF167xzQlvunNAWgBNb89j6WTo+Afb1DhgXwa2PRLH4L2dcyvEzq1Xy/Mx81i6NIDrSwKCRmYy+y5sundxra6ZODmDq5AAA1m4y8c/5xbXNAiDxkxLiOrpRWmZzOY8OwdS4cfwpaT45lmI+6TeFXXmnuVSeU1vTyuDJ1Lhx/DlpAcaKIgLc7M/dZVMu4/d/ULuclYP/yo7ck65n0gmeef8xXrrnPfIyCvjXjtnsW5dE2tlMh7qTe87zyn1/d7jPWm3l45e+IvlYKl6tPEnc+RpHtp5ymvf68uiYkjiBGcPeIC+9gMQDc9i75hBpZxw3ZCd2nmHWPe84zb9p0XZWJ27gxc+ebXSG2iwuL6Fhi4DhV5n+DHBaStkDGAL8nxDC/Sr1mnQLjCKtvIB0UyHV0sr6jJPcFtHZoWZkdDe2ZJ0h21wMQEGlfWvqY/CgT1AMK9KOAFAtrZRWW1zKc+l4CaFtvQhp44XBXUefkWEc35LXYP2htTkkjAqrvd2xbwA+/k3X1w8mVdChnRvtY9xwdxfcN8aH7zaaGqxftqqM+8fWNbf0zGrWbzEx/mHfJsnTxb8N6eY8Ms0FVEsrPxiPcXNoV4eaOyN6sSP3JMaKIgCKqsqdltMnKJYMcz5GS5HLmeIS2pN50Uj2pVyqq6xsX76fgaN6a5q3wFhM8rFUAMxlFi6fyyQkMtC1PP1iyUzOJjslh+qqarYv281NYxI0z39i5xlKC8pcyvCzZmsYUsodQMHVSgBfIYQAWtXUunz2lzBPv9pGAGC0FBPu5fjmjvEJxs/Ni09veoJltzzF3dE9AIj2DqSwspw3e47l61snMbvHPS6PMIqMFQRG1p1kJyDCgyJjRb21lWYrp3cV0HNYqEvrvJrMbCvRretGC1GRejKz6n/aTSYbm7ebGTeyrmFMfzWft/8ahK6J3jmhHv7kWOper1xLMaEefg41bbxD8DV48a8+T7Gg33MMj3T+5R0a0ZMfso82Sabg1oHkpte9dfMyCghp7fxL36VfLB/tfYM3V0wjpkuU0/TwtiF06BHD2UMXXMoTEhVEbnp+XZ70AkKigp3q4gd2Yl7S33hr7Uxi4qNdWmdDWvKgZyLQBcgETgDPSynrHeMKISYKIQ4JIQ5ZSxreGgKIeu6T0vG2Qaeji38kz+xfwlP7PuepTrcS4xOMXtjvX3bpIPf/OA+ztYoJsYMb89iunlHUlxJObMunfS//2t2R5nDlc3G1PGs3mxiY4FG7O7Jus4nQED29uzfdWcbqfb2uuK0XOuL8opie9ClTkz7h8RuG0sY7pHa6QegZFBLPtpzjTZOpnudDXvHEJR+9xKPxU3l64CxWz9vMq1895zDd08eDWUumMG/GEkylro1S63t5nPIcSeGRdpOZ1Gs6qxPX89rKF11aZ0NasmHcBRwFWgM9gUQhhF99hVLKj6WUCVLKBL2f91UXarSUEOHlX3s73NOfHEupY425hN25yZitVRRVmjicn0qcXzhGSwlGSwknijIA2Jx5ii4BkY1/hEBAuAeFWXUjiqLsCvzD6t/zOrzOcXekOURF6knPtNbezsiyEhmhr7f2m9Xl3D+2Ve3tPQctrN1kolO/yzz2dC7bd1l44tmceufVKqeimDDPutcr1NOfvIoSh5pcSzH7889jsVVRXGXiWOFFYlvVvS4DQuI4X5pBYWXTDLvzMgoIjQ6qvR0SFUR+VpFDjanUgqXc/roe3HQcvZsev2D7c6U36Jm1ZApbl+1h95rDLufJTS8gNLpuRBESHUR+puPg3VRqxlJub0wH1ifV5Gma3cZfasmGMR5YIe2SgRSg8zXmuaaTRZnE+AQR5R2AQegZEXUj241nHWq2Zp+ld1AMeqHDU+9Gt8AoLpblkV9RRra5hHY+9henf2h7LpTm1rcazWK6+ZKTaiYv3Ux1pY3D63LodnuIU525tJqfDhbR/Q7naU0poacHySlVpKRVUVkp+WZ1OaOHOTfh4hIbO/dZuHt43bQ3ZwZx4XBbzh9ow+KPQhlysyeLEl1rcGdL0mnjFUKkZyAGoWdoeA925552qNmZe5ruAe3QCx0eOjfi/ds6HBQdGt50uyMA5w6nENUhnPCYEAxueobc259965IcagLD6ppcXJ/26HQ6SvLtDWvq3AlcPpfJisSNTZPnYDJRHSOJaBeGwc3AkAcGsXfNIcc84QF1efrG1uQppam15MeqacAdwE4hRDgQB1x0daFWaePtE+uYN+BR9ELHyrQkLpTmcl+M/SDRN6mHSCnLY3dOMt8OeRqblKxIO0Jyqf0NOOfEOt7p83vcdHrSywuZdXSVS3n0Bh33z+rIhxOOY7NJBv4+ktYdfdi51D6KGfygfd/36OZcugwKxMPbcWv/6dTT/HSwiLLCKl6+dQ+jptzATfc2ftRjMAg+eCuYux/OxmqFxx/0JT7OnfmL7Vv1Pz5mH+StXl/O0Fu88PFu3m2KVdp4/9xq3u/1JDqhY23mQVLKjYyJGmDPkbGPVFMO+/PPs6j/n5BIvss4QEq5/WNoD50bfYM68rczK5osk81q48Npn/P2quno9Do2fb6D1DMZjJpwGwBrF2xj8Li+jH7ydqzVVirMlcx5Yi4AXQd2ZOjDg7h48jJz97wOwMLZyzm4qfG7SzarjcQpC5iz4WV0eh0bF24j9XQ6o5+6E4Dv/72ZW+4dwOhJw7BWW6k0V/LWQ3Wf3sxc8jzdh3TFP8SXL9PmsXj212z4dGujsogr94WaihDiK+yffoQARuBVwA1ASjlPCNEa+ycpkdh3Zd+RUn5xreV6xbaWse//sVkyN8bTcTtaOoKDCX6pLR3BwR2n7m3pCE58xhpbOoIDm+nqx+Vawg9y+WEppdNHMc02wpBSPnSN6ZnAsOZav6IoTU/9abiiKJqphqEoimaqYSiKoplqGIqiaKYahqIomqmGoSiKZqphKIqimWoYiqJophqGoiiaqYahKIpmqmEoiqKZahiKomj2mztruBAgRPN8w7YxqmT9J59R7Nx01msXKb8ZaoShKIpmqmEoiqKZahiKomimGoaiKJqphqEoimaqYSiKoplqGIqiaKYahqIomqmGoSiKZqphKIqimWoYiqJophqGoiiaqYahKIpmv7lvq2oxKDSWGTeORCcEK9KO8GnyTqeahOB2vNh1BAadnqJKE3/Y8ykAvgZPZvcYQ6xfGFLCK8dWcbzwskt5zuzMZ8Xb55E2yYB7WzP0j+0cpm9dkMqh77MBsFVLjBfLeXP3LfgEuPHly6c5vT2PVkHu/OW7AS7l+NmmbSamzSrAapOMf8iX6VMCHKa/P7eIpSvKAai2Ss7+VEX6ibYEBdq/mWu1Sm4anknrSD0rF0e4nKdvUBzPdhyLXuhYm7Wfr1KdLxTcI6ADz3Ycg0HoKa4q54WkubTxDuWVro/W1kR6BbPw4ga+TXd+va9XwtBuTHrvEfR6Hes/+5Gv31/rML374M7MXvo82am5AOxec5gl76wmNCqI6fMnEhjuj7RJ1i3cxqq5m13Pc1dPJn8wHp1ex/oFW1j27irHPLfG8/qqGWSn2C8qvmvlfr54YzkA0xY8Tf9RfSjKKWZi92ku5Wi2hiGE+BQYDeRIKW9soGYI8AH2izTnSSlvdXW9OgQzu41m4r7PMJpL+GrwU2zPPsvFstzaGl+DJy93G83T+z8n21xMkLtP7bQZN45gd+5PTDu8DIPQ46V3cymPzSpZ/sY5nl7Qi4BwD96//yA33hZCRGyr2prbJ8Rw+4QYAE5uy+XHzy7jE2Bfb/+xkQx+OJolfzntUo6fWa2S52fms3ZpBNGRBgaNzGT0Xd506eReWzN1cgBTJwcAsHaTiX/OL65tFgCJn5QQ19GN0jKby3l0CJ6P+x3Tk/5NbkUx8xJeYE/uKVJNdRdM9jF48kLc75hxdD45FUUEuNmfu8umXP548P3a5Xwz6BV25Z10PZNO8Mz7j/HSPe+Rl1HAv3bMZt+6JNLOZjrUndxznlfu+7vDfdZqKx+/9BXJx1LxauVJ4s7XOLL1lNO815dHx5TECcwY9gZ56QUkHpjD3jWHSDuT7lB3YucZZt3zjtP8mxZtZ3XiBl787NlGZ6jN4vISGrYIGN7QRCFEADAXuEdK2RW4rylWemNgNGnlBWSYCqmWVjZknuC2iM4ONSOjurEl6wzZ5mIACirtW1Mfgwd9gtuxIu0IANXSSmm1xaU8qcdLCGnrRUgbLwzuOnqNDOfE1rwG64+sNdJ7ZHjt7Q59A/EOcK1p/dLBpAo6tHOjfYwb7u6C+8b48N3Ghq8evmxVGfePrWuo6ZnVrN9iYvzDvk2Sp7NfWzJN+WRZCqiWVrbmJDEotKtDzdDw3uzMPUFORREARVVlTsvpHdSRTHM+Rkuhy5niEtqTedFI9qVcqqusbF++n4Gjemuat8BYTPKxVADMZRYun8skJDLQtTz9YslMziY7JYfqqmq2L9vNTWOcLqzeoBM7z1Ba4PycNUazNQwp5Q6g4ColDwMrpJRpNfU5TbHecE9fjDWNAMBoKSHM08+hJqZVCH5uniwYOJ6lgydxd3QPAKK9AymoKOeNnuNYdsvTzO4+xuURRnGOhcAIz9rbAeEeFBsr6q2tNFs5uyuf7sPCXFrn1WRmW4luXTdaiIrUk5lVXW+tyWRj83Yz40bWNYzpr+bz9l+D0DXROyfEw7+2EQDkVhQT4uHvUBPtHYqvwZu/93qafye8wLCIPk7LuT2sF1uMSU2SKbh1ILnpdW/dvIwCQlo7/9J36RfLR3vf4M0V04jpEuU0PbxtCB16xHD20AWX8oREBZGbnl+XJ72AkKhgp7r4gZ2Yl/Q33lo7k5j4aJfW2ZCWPOjZCQgUQmwXQhwWQjzWUKEQYqIQ4pAQ4lB1ccNbw5pqp3skjmfo0gsd8QGtefbAF0zav5iJHYcQ4xOMXujo4h/J15cO8sCOjzBbK/lD7ODGPLZfrtw5oXNEAE5uy+OGXgG1uyPNQdabp/5AazebGJjgUbs7sm6zidAQPb27ezRZnvrWfGVGvdDRyTeal44tYPqx+Tza7k6ivUJqpxuEnptCuvJjzrGmyVTP8yGvCJV89BKPxk/l6YGzWD1vM69+9ZzDdE8fD2YtmcK8GUswlbo2Sq3v5XHKcySFR9pNZlKv6axOXM9rK190aZ0NacmGYQD6AKOAu4BZQohO9RVKKT+WUiZIKRMM/t5XXajRUkK4V90WKtzTj1xLqVPN7pxkzNYqiipNHC64RCe/CIyWEoyWEk4U2fcNN2edpot/a1ceI/7hnhRm171hiowV+IXV/wuXtM5I71Hh9U5rKlGRetIz606bl5FlJTKi/tMMfrO6nPvH1h1r2XPQwtpNJjr1u8xjT+eyfZeFJ551bWCYW1FMmEdA7e1QD3/yK4udag4UnMViq6SkqpzjRRfp0Krudekf3JnzZekU1rOr0hh5GQWERgfV3g6JCiI/q8ihxlRqwVJuHyke3HQcvZsev2D7c6U36Jm1ZApbl+1h95rDLufJTS8gNLpuRBESHUR+puPg3VRqxlJuf58dWJ9Uk6dpdht/qSUbRjqwQUpZLqXMA3YAPVxd6KmiDGJ8gojyCsAg9Axv3Y3t2WcdarZln6F3UAx6ocNT70b3gGhSynLJryjDaC6hnY/9xekf0p6Lpa79QrTt5kteqon8dDPVlTaS1hm58bYQpzpzaTUXDhVy4+2hLq3vWhJ6epCcUkVKWhWVlZJvVpczephzEy4usbFzn4W7h9dNe3NmEBcOt+X8gTYs/iiUITd7sijRtd2ns6WXifIOIcIzCIPQc3tYL/bknXKo2Z17ku7+7dEJHR46N7r4tSXVVPe63B7ei61NtDsCcO5wClEdwgmPCcHgpmfIvf3Zt85x+YFhdRuluD7t0el0lOTbG9bUuRO4fC6TFYkbmybPwWSiOkYS0S4Mg5uBIQ8MYu+aQ455wgPq8vSNrclTSlNryY9VVwOJQggD4A70B/5+9VmuzSptvH1yLR8NeAy90LHq8hEulOVyX4z9INE3qYdIKctjd+5PLL91MlJKVqQdIbmmMcw5uZY5ve/FTacn3VTIrKMrXcqjN+j4/V/jmPdkEjYb9P9dJJEdW7F7qX0UM+hB+77m8R9yiLspCA9vx639Z9NOcuFAIWVFVbw6ZBcjnm3PgHsbP+oxGAQfvBXM3Q9nY7XC4w/6Eh/nzvzFJQD88TH78Z7V68sZeosXPt7Nu02xSRv/PL+C93pORCcE6zMPcKncyN2tBwLwXeZe0kw5HCg4y4J+05BSsjZzP5fK7R9De+jc6BPUiffPLm+6TFYbH077nLdXTUen17Hp8x2knslg1ITbAFi7YBuDx/Vl9JO3Y622UmGuZM4TcwHoOrAjQx8exMWTl5m753UAFs5ezsFNx13KkzhlAXM2vIxOr2Pjwm2knk5n9FN3AvD9vzdzy70DGD1pGNZqK5XmSt56qO5XaeaS5+k+pCv+Ib58mTaPxbO/ZsOnzh9dayGu3BdqKkKIr4AhQAhgBF7F/vEpUsp5NTXTgfGADfhESvnBtZbr3bG1jH3/yWbJ3BgTOu5p6QgOJvlfbOkIDoafGdfSEZy439Pwp1QtwWa61nG5/7wf5PLDUkqnj2KabYQhpXxIQ83fgL81VwZFUZqW+tNwRVE0Uw1DURTNVMNQFEUz1TAURdFMNQxFUTRTDUNRFM1Uw1AURTPVMBRF0Uw1DEVRNFMNQ1EUza7aMIQQsUKIQfXcP1gI0aH5YimK8mt0rRHGB0B935E110xTFOX/I9dqGO2klE7fy5VSHgLaNUsiRVF+ta71bVXPq0zzasogWkkJUjZwjrsWUGFrvtPpNYatvnMCtqBf02uluO5aI4yDQog/XnmnEGIC4Pq5xxRF+U251gjjBWClEOIR6hpEAvYzZP36zoyiKEqzumrDkFIagZuEELcBP1+MaK2UsnHn91IU5TdN0xm3pJTbgG3NnEVRlF859YdbiqJophqGoiiaqYahKIpmqmEoiqKZahiKomimGoaiKJqphqEoimaqYSiKollLXoy52QwKjWVGtxHohWBF6hEWJO9yqkkIbseMG4djEHqKKk2M37MQAF+DJ7N73kNH3zAk8MrRVRwrTHcpz7mduax55wzSCn1/H81tf2zvMP3HT1NI+j4TAJtVknOxjFd23o53gPs1522MTdtMTJ9VgNUGTzzUij9PCXCY/ve5xSxdYb8SudUKZ3+qIu1EG4IC9TX3SQYNz6J1pJ4Vi8NdztMvKI5nO41BL3SszdzPl6nOfyPYM6ADz3a6B73QU1xVzgtHPqKNdyiv3vg/tTWRXsEsvLiR5Zd3upwpYWg3Jr33CHq9jvWf/cjX7691mN59cGdmL32e7NRcAHavOcySd1YTGhXE9PkTCQz3R9ok6xZuY9Xcza7nuasnkz8Yj06vY/2CLSx7d5VjnlvjeX3VDLJT7BcV37VyP1+8Yb9A9bQFT9N/VB+KcoqZ2H2aSzmarWEIIT4FRgM5Usobr1LXF9gHPCCldPkS3DoEL3cfxcS9i8k2l7D0lolsyz7HxbLc2hpfgyd/7T6KSfu+INtcTJC7T+20Gd1GsDsnmWmHvsYg9HjpXfs2qs0qWfXWaZ6c3xf/cE8SH9hL/G1hhMe2qq259Q83cOsfbgDg9LYcdi2+hHeAu6Z5r5fVKvnTzAK+XxpOVKSBwSMzGXWXN106udfW/GmyP3+a7A/A2k0mEueX1DYLgA8/KaFzRzdKymyNzvEzHYLn48bx56SPya0oZl7f59mdd5rUcmNtTSuDJy90/h0vJs0np6KIADf7479syuXJA3+vXc7ym2exM/ek65l0gmfef4yX7nmPvIwC/rVjNvvWJZF2NtOh7uSe87xy398d7rNWW/n4pa9IPpaKVytPEne+xpGtp5zmvb48OqYkTmDGsDfISy8g8cAc9q45RNoZxw3ZiZ1nmHXPO07zb1q0ndWJG3jxs2cbnaE2i8tLaNgiYPjVCoQQeuBdYGNTrbRbYBRp5QWkmwqpllbWZ5zktojODjUjo7uxJesM2eZiAAoqywHwMXjQJyiGFWlHAKiWVkqrLS7luXyiiOA23gS38cbgrqPHyAhObzM2WH9sXRY9RkY2al4tDiVV0KGdgRti3HB3F9w7xofvNzZ89fBvVpVz39i6hpqeWc2GLWaeeLjxTeuXOvu1JcOcT5algGppZavxKINCujrU3BHem505J8ipKAKgqKrMaTm9gzqSYc7HaCl0OVNcQnsyLxrJvpRLdZWV7cv3M3BUb03zFhiLST6WCoC5zMLlc5mERAa6lqdfLJnJ2WSn5FBdVc32Zbu5aYzThdUbdGLnGUoLnJ+zxmi2hiGl3AEUXKNsCvAtkNNU6w3z9KttBABGSzHhXr4ONTE+wfi5efHpTU+w7JanuDu6BwDR3oEUVpbzZs+xfH3rJGb3uMflEUaxsYKAyLpTh/iHe1JsrKi3ttJs5dyuPLrdGX7d82qVmW0lqnXdwDIq0kBmlrXeWpPJxubtZsaO9K6978VXC3jzr4HomuidE+rpT66lqPZ2bkURoR7+DjVtvENo5ebFB72f5t99X2BYRB+n5dwe3pOtxqNNkim4dSC56XVv3byMAkJaO//Sd+kXy0d73+DNFdOI6RLlND28bQgdesRw9tAFl/KERAWRm55flye9gJCoYKe6+IGdmJf0N95aO5OY+GiX1tmQFjvoKYSIwv4V+XkaaicKIQ4JIQ5ZSxreGgLUd7oWecU5ZQw6HV38I3lm/xKe2vc5T3W6lRifYPTCfv+ySwe5/8d5mK1VTIgdrP1BaSQaOKfMme05tOsVgHeAe/0FV5lXqyufi6stc91mMwMSPGp3R9ZtNhEaoqd3dw/XQlyDvOIkQHqhJ843mr8cXcCLRz/msRuGEu0VUjvdIPQMCunK9pxjTbJ+Uc8TIq944pKPXuLR+Kk8PXAWq+dt5tWvnnOY7unjwawlU5g3YwmmUtdGqfW9Pk55jqTwSLvJTOo1ndWJ63lt5YsurbMhLfkpyQfADCll/Zu3X5BSfiylTJBSJuj9vK9aa7SUEOFVt4UK9/Qnx+J4WlKjuYTducmYrVUUVZo4nJ9KnF84RksJRksJJ4oyANiceYouAZHX/8h+wT/cg6Isc+3tYqMFv7D6f+GOra/bHbneebWKitSTkVldezsjq5rICH29td+sLuf+X+yO7DtYwdpNJjr3u8xjT+fy4y4Lf3g2t955tcq1FBPqGVB7O9QjgLyKEseaiiIO5J/DYqukuMrEsaKLdPBtXTu9f3BnzpemU1jZNMPuvIwCQqODam+HRAWRn1XkUGMqtWApt4/2Dm46jt5Nj1+wfTdNb9Aza8kUti7bw+41rp9nKje9gNDouhFFSHQQ+ZmOg3dTqRlLub0xHVifVJPHcWTdFFqyYSQAS4UQl4B7gblCiLGuLvRkUSYxPkFEeQdgEHpGRN3IduNZh5qt2WfpHRSDXujw1LvRLTCKi2V55FeUkW0uoZ2P/cXpH9qeC6Wu/UJE3+hPfpqJgnQT1ZU2jq3LpsttYU515tIqLh4spOvtYdc97/Xo09OD5JRqLqVVUVkpWb66nFHDnJtwcYmNXfssjB5eN+31mYEkH27D2QNtWPxRKLfe7MmniaEu5TlXeplo7xAiPIMwCD23h/dkT94ph5pduafoFnADeqHDQ+dGvF8MaeV1e7F3RPRkSxPtjgCcO5xCVIdwwmNCMLjpGXJvf/atS3KoCQyr2yjF9WmPTqejJN/esKbOncDlc5msSGyaQ3PnDiYT1TGSiHZhGNwMDHlgEHvXHHLMEx5Ql6dvbE2e+s7f7ZoW+1hVSnnDzz8LIRYB30spV7m6XKu08faJdcwb8Ch6oWNlWhIXSnO5L8Z+kOib1EOklOWxOyeZb4c8jU1KVqQdIbnU/gacc2Id7/T5PW46Penlhcw66lokvUHHmJfjWTDxEDabpO+4aCJifdm3LA2AAQ+0BeDUD0Y6DgrG3dtwzXldYTAI3n8riHseNmK1wmMPtiI+zp35i+1b9T8+5gfAmvXl3HGLJz7ezbtNsUob/zi3kr/1+iM6BOuzDnKp3Mg9UQPtOTL2kmbK4UD+ORb0n4aUkrWZ+0kpzwbAQ+dGn6BO/N+Zb5ssk81q48Npn/P2quno9Do2fb6D1DMZjJpwGwBrF2xj8Li+jH7ydqzVVirMlcx5Yi4AXQd2ZOjDg7h48jJz97wOwMLZyzm4yelc2teVJ3HKAuZseBmdXsfGhdtIPZ3O6KfuBOD7f2/mlnsHMHrSMKzVVirNlbz1UN2nNzOXPE/3IV3xD/Hly7R5LJ79NRs+bdw5sMSV+0JNRQjxFTAECAGMwKuAG4CUct4VtYuwN4xrfqzqFdtaxr7vdJrRFvNo7IGWjuBgSuDZaxf9B404/fuWjuDEY4xro8amZjNd/bhcS/hBLj8spXT6KKbZRhhSyoeuo/aJ5sqhKErTUX8ariiKZqphKIqimWoYiqJophqGoiiaqYahKIpmqmEoiqKZahiKomimGoaiKJqphqEoimaqYSiKoplqGIqiaKYahqIomqmGoSiKZr+5ywxIBJXV9Z8hqiWYbA2fTq8l6Oo9SWHL8TBUX7tI+c1QIwxFUTRTDUNRFM1Uw1AURTPVMBRF0Uw1DEVRNFMNQ1EUzVTDUBRFM9UwFEXRTDUMRVE0Uw1DURTNVMNQFEUz1TAURdHsN/flMy0Gh3fg5e53oRM6vrmUxPzzu51q+oXEMLP7XRh0OgorzDy68zMAfN08eLP33XTyC0NKycwj33G0IN2lPMm7ctjw7glsVknv38Vw85MdHabvXpjMibX2ddiskryLpUzfMRwvf/drztsYm7aZmDarAKtNMv4hX6ZPCXCY/v7cIpauKAeg2io5+1MV6SfaEhRo/9Kf1Sq5aXgmrSP1rFwc4XKehMDOTIr9HXohWJ+1j68vb3Gq6e4fy6TYcRiEjuKqcqYfSwRgXNStjIgcgARSyrP4v7NfUiVd/8JbwtBuTHrvEfR6Hes/+5Gv31/rmGdwZ2YvfZ7sVPt1WnevOcySd1YTGhXE9PkTCQz3R9ok6xZuY9Xcza7nuasnkz8Yj06vY/2CLSx7d5VjnlvjeX3VDLJT7BcV37VyP1+8Yb9U8bQFT9N/VB+KcoqZ2H2aSzmarWEIIT4FRgM5Usob65n+CDCj5mYZ8LSU8pir69UheKXHCMbv+gKjuYTltz3J1qxzXCjNq63xdfPg1Z4jeXL3ErLMJQR5eNdOe7n7cHYaL/D8/uW4CR2eBjeX8tisknVvHefRjwfiF+HF/Ad3EHdbBKEd6q7CPmh8LIPGxwJwbns2+z6/iJe/u6Z5r5fVKnl+Zj5rl0YQHWlg0MhMRt/lTZdOdd+6nTo5gKmTAwBYu8nEP+cX1zYLgMRPSojr6EZpma3ROX6mQ/BMx3t56fhH5FUU8a/eU9mXf5I0k7G2xkfvxbMd7+XlE/PIrSjC360VAMHu/oyNuoU/HnqHSlsVL3d5nCFhvdlsdO0C2Tqd4Jn3H+Ole94jL6OAf+2Yzb51SaSdzXSoO7nnPK/c93eH+6zVVj5+6SuSj6Xi1cqTxJ2vcWTrKad5ry+PjimJE5gx7A3y0gtIPDCHvWsOkXbGcUN2YucZZt3zjtP8mxZtZ3XiBl787NlGZ6jN4vISGrYIGH6V6SnArVLK7sAbwMdNsdLuQVGklheSbiqiStpYm36KOyLjHGrubtONzZlnyTKXAFBQYb96to/Bnb4hbVl+KQmAKmmjtKrCpTwZJwoJautDYBsf9G46uo6I4uy27AbrT67L4MYRUY2aV4uDSRV0aOdG+xg33N0F943x4buNDV89fNmqMu4f61N7Oz2zmvVbTIx/uPFN65fi/GLINOeRbcmnWlrZnpPEwOBuDjW3hfdmd95xciuKACiuKqudphc6PHRu6NDhoXcnv7LY9UwJ7cm8aCT7Ui7VVVa2L9/PwFG9Nc1bYCwm+VgqAOYyC5fPZRISGehann6xZCZnk52SQ3VVNduX7eamMU4XVm/QiZ1nKC0ou3ahBs3WMKSUO4CCq0zfI6UsrLm5D4huivWGe/qSba570xjNJYR7Ob6527UKws/Nk8WDH+Pb255kTNvuALTxCaSgwsScPvew8vY/8mbv0XjpXRthlOZY8Ivwqr3tF+5JqdFcb22VuZrk3TnE3xl53fNqlZltJbp13WghKlJPZlb9Q3iTycbm7WbGjaxrGNNfzeftvwaha6J3TrC7P7kVhbW38yqKCPHwd6iJ9gqjlcGL93o8S2LvaQwN7wtAfmUxy9O38fmAV/lq4OuUV5s5UnjO9UytA8lNr3vr5mUUENLa+Ze+S79YPtr7Bm+umEZMlyin6eFtQ+jQI4azhy64lCckKojc9Py6POkFhEQFO9XFD+zEvKS/8dbamcTEN8mvk5Nfy0HPCcD6hiYKISYKIQ4JIQ5Zi8uvuiBRz/lj5BW39UJH18BIntrzFU/uXsLkzoNp1yoIg9ARHxDJVxcPM27rfMzVVUyMG9SIh/OLdV+58oZCAud+NNK2VxBe/u7XPa8reUQDy1y72cTABI/a3ZF1m02Ehujp3d3DpQwO664v4xWvmF7o6OjbhlknPmbm8Xk83HYYUV6htDJ4MTD4Rh7f/zoP73sFT70Ht4f1cT1TPc+HvOKJSz56iUfjp/L0wFmsnreZV796zmG6p48Hs5ZMYd6MJZhKLS7mcb7PKc+RFB5pN5lJvaazOnE9r6180aV1NqTFG4YQ4jbsDWNGQzVSyo+llAlSygS9v09DZQBkm0uJ8KrbQoV7+ZFjLnWq2Wm8gNlaRWGlmUN5aXT2DyfbXEK2uYTjhRkAbMg4Q3xAZOMfHPZRQUl23aigxGjBN8yz3tpT6+t2R653Xq2iIvWkZ1prb2dkWYmMqP8MZt+sLuf+sa1qb+85aGHtJhOd+l3msadz2b7LwhPP5riUJ6+ymFCPuq13iEcA+RUlDjW5FUUcKjhLha2SkupyThRfoL1Pa3oFdCLbUkBxVTlWaWN33nHi/W5wKQ/YRxSh0UF1maKCyM8qcqgxlVqwlNt3Vw9uOo7eTY9fsP250hv0zFoyha3L9rB7zWGX8+SmFxAaXTeiCIkOIj/TcfBuKjVjKbc3pgPrk2ryNM1u4y+1aMMQQnQHPgHGSCnzr1WvxYnCDNq1CiLaOwA3oWNUdFe2Zp13qNmSdY6E4LbohcBTb6B7YBQXSvPIqygn21zCDa3sL87AsBu4UJLrUp6oGwPITy2nML0ca5WNU+sziBsS7lRnKa3i0qF84m6LuO55r0dCTw+SU6pISauislLyzepyRg/zdqorLrGxc5+Fu4fXTXtzZhAXDrfl/IE2LP4olCE3e7IoMcylPOdK0ojyCiHcMwiD0DMkrBf78k861OzNP8mN/u3txyl0bnT2iyHNZCSnoogufjF46Oy7jT0DOjocLG10psMpRHUIJzwmBIObniH39mffuiSHmsCwuo1SXJ/26HQ6SvLtxwmmzp3A5XOZrEjc6HIWgHMHk4nqGElEuzAMbgaGPDCIvWsOOeYJD6jL0ze2Jk8pTa3FPlYVQrQFVgCPSinPX6teK6uUvH50PZ8MegS9EHybepTk0lwevME+VF2acpiLpXnsNCaz5o5J2KRk+aUkfqppDG8cW8//9h2Hm07P5fJCXjq8xqU8OoOOkTO78cWkfUirpOe4toTF+nHo60sAJNzfDoCzW7LocFMo7t6Ga87rCoNB8MFbwdz9cDZWKzz+oC/xce7MX2zfqv/xMfvyV68vZ+gtXvh4N+82xYaND5O/5e1uk9AJHZuy95NqymZU5E0ArM3aw2WTkUMFZ5iX8CISyYasfaSa7Ad/d+Ye48M+f8YqbSSXpbM+a4/rmaw2Ppz2OW+vmo5Or2PT5ztIPZPBqAm32TMt2MbgcX0Z/eTtWKutVJgrmfPEXAC6DuzI0IcHcfHkZebueR2AhbOXc3DTcZfyJE5ZwJwNL6PT69i4cBupp9MZ/dSdAHz/783ccu8ARk8ahrXaSqW5krceqvv0ZuaS5+k+pCv+Ib58mTaPxbO/ZsOnWxuVRVy5L9RUhBBfAUOAEMAIvAq4AUgp5wkhPgF+D6TWzFItpbzmoV/P2CgZ895TzZK5MR7q5PqQsym9FHyipSM4uOf8PS0dwdmowmvX/AfZTA1/StVSfpDLD9f3+9hsIwwp5UPXmP4k8GRzrV9RlKbX4gc9FUX57VANQ1EUzVTDUBRFM9UwFEXRTDUMRVE0Uw1DURTNVMNQFEUz1TAURdFMNQxFUTRTDUNRFM1Uw1AURTPVMBRF0Uw1DEVRNPvtXWZAgrS5dpq6plRlq/9sVS1FL35d24CSCtfOENYcfCurWjrCb9av692lKMqvmmoYiqJophqGoiiaqYahKIpmqmEoiqKZahiKomimGoaiKJqphqEoimaqYSiKoplqGIqiaKYahqIomqmGoSiKZqphKIqi2W/v26oaDA7vwMs970IvBN+kJPHxOecrevcLjeHlHsMwCD2FlSb+58fFAGwdMYXy6kps0ka1zcbvty5wOc+F3UY2v3scaZP0GBfDTRPiHKbvW3Sek+vSAbBV28hPKeWF7aPw8ne/5ryNsXGriamv5GG1Sv7wsB8vTgl0mP5/cwv5ckUZANZqyZmfqsg62Y6gQPs3c61WSf/h6URFGFj9eaTLeQYEd+KFznejF4I16Qf5/NKPTjW9AtvzQtxoDDo9xZXlTD70MW29Q3ij+8O1NVHeQcxP3syytN0uZ0oY1p2n338MnU7HhoXbWPa37xymd7+lC699O43sSzkA7Fp1kCVvrQRg6scTGTCyF0W5JUzsNcPlLAAJd/Vk8gfj0el1rF+whWXvrnLMc2s8r6+aQXZKTZ6V+/nijeUATFvwNP1H9aEop5iJ3ae5lKPZGoYQ4lNgNJAjpbyxnukC+AcwEjABT0gpj7i6Xh2CV3sNZ/zOJWSbSvj2jifZknmeC6V5tTW+bh7M7jWCCTu/JMtcQpCHt8MyHvtxMYWVZlejAGCzSja+fYyH/j0Iv3AvFj68jY5DIgnt4FdbM+CJTgx4ohMAP23P4sAXyXj5u2ua93pZrZLnZuayfllroiMNDBiRzuhhPsTHudfWTJscyLTJ9iby/aZy/vFxUW2zAPjn/GK6dHSnpNTW6Bw/0yGY1mUMzx9eQI6lmE8HPMvO3DNcKs+prWll8GR6lzH86cinGC3FBLr7AJBmyuPxff+sXc6aW2fyY84p1zPpBM/+Yzx/GTmHvPR8/rX3TfZ+f4S0MxkOdSd2neWVcf/rNP/mxTtYM3cTLy582uUs9jw6piROYMawN8hLLyDxwBz2rjlE2pl0xzw7zzDrnnec5t+0aDurEzfw4mfPup7F5SU0bBEw/CrTRwAda/5NBD5qipV2D2pNalkhl8uLqJI21l4+xdDWjlvlu9vcyKaMs2SZSwAoqDA1xarrlXmygMA2PgRG+6B30xE/PJqftmc1WH9qQzrxI6IbNa8WB5Iq6NDOjfYxbri7Cx4Y04rvNpY3WL9sVRkPjPWtvZ2eWc36LSb+8LBvg/Ncj3j/NqSb8sk0F1AtrfyQfYxbwuIdaoZF9mR7zimMlmIACiud8yYEx5JhyifbUuRypri+sWReMJKdkkN1lZUfv97LTXf30Tz/iV1nKS0sczlHbZ5+sWQmZ9fkqWb7st3cNCZBe56dZygtaJo8zdYwpJQ7gIKrlIwBFku7fUCAEMLl8W24lx/ZNY0AINtcQriX45u7nW8w/m6efH7ro6y440nGtu1elxvJp4MfYcUdT/LADb1cjUNpjgW/CK/a275hXpQaLfXWVpmrubjbSOehUdc9r1aZ2dVER9UNLKMiDWRkV9dbazLZ2LjNxO9G+dTeN+2VPOb8NRidrmlOYhTq6UdOTSMAyLEUE+rhOIJq6x2Cn8GLDxMmsnDAs4yI7O20nDsjerA5+1iTZAqJCiQ3Pb/2dm5GAcGtg5zq4gd05KNDc3hrzYvExEc1ybrrzxPkkCcvvYCQqGDnPAM7MS/pb7y1diYx8dHNkqUlj2FEAZd/cTu95j6nTagQYiL2UQiGEP+rLrS+t7FEOtw2CB1dAyN5fMcXeOoNLLttPEcL0rlUVsBD2xaRYykjyMObRYP/hwul+RzKS7u+R+a4cm0hgZ9+zCa6ZzBe/u7XPa/mOPUsUzSwzO83m7ipr2ft7sjazeWEhujp08ODH/c0zS6bqOcBXRlRL3TE+UUx5fB8PHRuzO83mZPFaVw22XczDULPzaFdmPvThibJVN8TIq944pKTLvE/sc9hKa+g7/CezP5mGuO7Tm2a9V87jnOeIyk80m4ylnIL/Ub04rWVL/JE3HNNnqUlPyWp/3e7vjul/FhKmSClTND7+dRXUivbXEKEV90WKsLLjxxzmVPNTuMFzNYqCivNHMxLo7N/OAA5FnttQYWJzZln6R7U+noekxPfcE9Ksut+uUpzzPiG1X/autMb0uk6om7LcD3zahUVaSA9o25EkZFVTevw+rcbX68q44GxrWpv7zlg4ftN5cT2TeWRSUa27TLz2DNGl/LkWIoJ86zbCIR5+pNXUeJUsy//PBZrFcVVJo4WptDRt24wOjAkjnMlGRRWNs2wOy+9gNDoui14aFQQBVmFDjWmUjOW8goADm44it5Nj19w0+ymXSn3ijwh0UHkZzoO3u157KPPA+uTmi1PSzaMdKDNL25HA5muLvREYSbtWgUR7R2Am9Axqk1XtmSdd6jZknmehJC26IXAU2+gR1AUF0rz8NK74WOwb9299G4MCm/PT8W5LuVp3TWQwrQyitLLsVbZOL0hnY63Ou95WUqrSDucR8chkdc97/Xo29OD5JQqUtKqqKyULFtdxui7nJtwcYmVHfvM3DO8btpbLwdz6Ug7kg/GsGReOLfd7MXiD8NdynOmJJ023sFEegViEHqGRvRgZ85ph5oduafpGdAOvdDhoXMjPqCNw0HRptwdATh36AJRsRFEtAvF4Kbn1vsHsvf7ww41geF1TS4uoQM6naAkv7TJMjjkOZhMVMdIItqFYXAzMOSBQexdc+iKPAF1efrGotPpmiVPS+6SrAGeFUIsBfoDxVJK147oAVYpef3oBhYMfhi9ECy/dIzkklwebG/f71168QgXSvPYkX2B7+58CpuUfJOSxE8lubTxCeDDgfcD9mHwd5dPstN4waU8OoOOYS/1YOnTu7HZoMfYGEJj/TjydQoAve+/AYDzWzO5YWAY7t6Ga87rCoNB8I+3Qxj1UBZWq+SJB/3oGufOvz+zH0d46nH7L8Kq9eXceas3Pt7Nu02xShv/d3YNH/T+Azqh4/uMQ6SU5zAuuj8AK9P3k1qey77883w+8HlsSL5LP8jFMvvIxkPnRr/gWN49s6LJMtmsNhJfWMTba/+CTqdj42fbST2dwag/3gHA2vlbGPy7/ox+aijWaiuV5kre/p9/1c7/0ufP0v2WLviH+LLk4r/4/PVv2bBou2t5pixgzoaX0el1bFy4jdTT6Yx+6k4Avv/3Zm65dwCjJw2rzfPWQ3+vnX/mkufpPqQr/iG+fJk2j8Wzv2bDp1sblUVcuS/UVIQQXwFDgBDACLwKuAFIKefVfKyaiP2TFBMwXkp5qP6l1fHsECXbvjupWTI3xn2dklo6goPXwo63dAQHg4/f29IRnPiOduGYVDOQ1b++s5j/IJcfllI6fRTTbCMMKeVD15gugWeaa/2KojQ99afhiqJophqGoiiaqYahKIpmqmEoiqKZahiKomimGoaiKJqphqEoimaqYSiKoplqGIqiaKYahqIomqmGoSiKZqphKIqiWbN9W7W5CCFygdQmWFQIkHfNqv+sX1smlefqfm15oOkyxUgpQ6+88zfXMJqKEOJQfV/fbUm/tkwqz9X92vJA82dSuySKomimGoaiKJr9/9wwPm7pAPX4tWVSea7u15YHmjnT/7fHMBRFuX7/P48wFEW5TqphKIqi2X9NwxBCDBdCnBNCJAsh/lLPdCGE+GfN9ONCiN7XmlcIcZ8Q4pQQwiaEaPRHVS5m+1QIkSOEONnY9buYrbMQYq8QokII8efmyHDF+q76eK/2XDVnBiFEkBBisxDip5r/AxuY96rPZ3NlEEK8VLPOc0KIuxpYpqbHcFVSyt/8P0APXADaA+7AMSD+ipqRwHrsV1wbAOy/1rxAFyAO2A4k/Kez1Uy7BegNnGyh5y0M6Au8Bfz5P/BaXvXxXu25as4MwHvAX2p+/gvwbmOez+bIAMTXrMsDuKEmg76eZV7zMVzr33/LCKMfkCylvCilrASWYr/Y8y81dPHnBueVUp6RUp5rwWzIa1/UulmzSSlzpJQHgf/IxTM0PN5muYi3hgxjgM9qfv4MGFvPrFpe6+bIMAZYKqWskFKmAMk1Wa6k5TFc1X9Lw2jows5aarTM21LZmltLrdcVLZU5XNZcma/m/7AWyNZQBq3r1fIYruq/pWFoubBzQzWaLwrdSK5ka24ttV5X/Joz/9e/jv8tDUPLhZ0bqmmWi0I3Ubbm1lLrdUVLZTb+vOtT839OPTXNna2hDFrXq+UxXNV/S8M4CHQUQtwghHAHHsR+sedfWgM8VnOUfQB1F3/WMm9LZWtuzf3Ym0NLPVdrgMdrfn4cWF1PTXM/nw1lWAM8KITwEELcAHQEDlzH/No19RHmlvqH/ej5eexHiF+uuW8SMKnmZwF8WDP9BL/41KO+eWvuH4e9e1dgv6D0xhbI9hWQhf2gYzow4T/8vEXUrLcEKKr52a8ZX0enx6v1uWrmDMHAFuCnmv+DampbA+uu9V5qzgw19S/XrPMcMOIX93/y83N0tfm1/lN/Gq4oimb/LbskiqL8B6iGoSiKZqphKIqimWoYiqJophqGoiiaqYahKIpmqmEoiqKZoaUDKL9eQohZwCPYv9iUBxwGioGJ2L++nQw8KqU0CSEWAWagMxADjMf+14QDsX8F/YmaZZZh/8OroUAhMBP7167bAi9IKdcIIdoBnwM+NVGelVLuaeaHq2igRhhKvWpOGPR7oBfwO+DnEwitkFL2lVL2AM5g/wvEnwUCtwN/Ar4D/g50BboJIXrW1PgA26WUfYBS4E3gTux/Vft6TU0OcKeUsjfwAPDP5niMyvVTIwylITcDq6WUZgAhxHc1998ohHgTCABaARt/Mc93UkophDgBGKWUJ2rmPQW0A44ClcCGmvoTQIWUsqpmnnY197sBiTVNxgp0aobHpzSCahhKQ+r7yjTAImCslPKYEOIJYMgvplXU/G/7xc8/3/75vVYl676PUFsnpbQJIX6u+RP27+70wD4KtjT6UShNSu2SKA3ZBdwthPAUQrQCRtXc7wtkCSHcsB/faA7+QJaU0gY8iv3Ud8qvgBphKPWSUh4UQqzBfq7IVOAQ9gOes4D9NfedwN5Amtpc4FshxH3ANqC8GdahNIL6tqrSICFEKyllmRDCG9gBTJRSHmnpXErLUSMM5Wo+FkLEA57AZ6pZKGqEoSiKZuqgp6IomqmGoSiKZqphKIqimWoYiqJophqGoiia/T/u6XfM3omtlQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "param_grid3 = {\"svc__C\": np.linspace(1, 2, 6), \"svc__gamma\": np.logspace(-3, 2, 6)}\n",
    "\n",
    "display_heatmap(param_grid3, pipe_svm, X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It seems like we are getting even better cross-validation results with `C` = 2.0 and `gamma` = 0.1 \n",
    "\n",
    "How about exploring different values of `C` close to 2.0? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQwAAAEGCAYAAACQF6v1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABI30lEQVR4nO3dd3hUZd7/8fc9Mymkl0kPRVogUXpAQBRRAQUB22PbdVVYFhHEBZGV/WFdy7rPY1tERJHiouKyNI0ISBGkJUDoAQQC6cmkl8mkTO7fHxMnTGYCRyYRd/d+XRcXOXO+55zPzJl855TMOUJKiaIoiha6qx1AUZR/H6phKIqimWoYiqJophqGoiiaqYahKIpmhqsd4OfyCPSRXhGBVzuGXbhX+dWO4CBYV3+1Izg4bQm+2hGc6M78ul4j2dBwtSM4qaCkUEoZ1vzxf7uG4RURyLV/f/Rqx7B7qsvWqx3BwT2+pqsdwcGoE3df7QhOvMb/ul6jBrP5akdw8p1cdcHV42qXRFEUzVTDUBRFM9UwFEXRTDUMRVE0Uw1DURTNVMNQFEUz1TAURdFMNQxFUTRTDUNRFM1Uw1AURTPVMBRF0Uw1DEVRNPu3+/KZFtcbu/PHHmPRCR3rs1L4NP17p5p+wdfwdI+xGHR6SmurmJryER18jPyl94P2mhifEBad+Y6VF3a5lefwjlI+/UsGDVbJ8P8JY9wfoh3Gf/1RLrvWFwHQYJVkn61m4b5++AUZWPSnc6RuKyUg1IO/fnOdWzl+smmbmVnzirE2SB570J/Z04Mcxr+1oJQvVlcBUG+VnPyxjqyjHQgJ1gNgtUqGjM4hOkrPmuWRbucZGBLHtO7j0QsdSTn7+OzCNqeaPkFdmNZ9HHqhp6yuiqcPfkB7nzBeuPY39pqodqEsObeRVZk73c404NbrmPLmw+j1OjYs+54v30pyGN9rWA9e/GIGeRdsX2Tbtf4AK95YR1hMCLM/mkxwRCCyQfLNkm2sXbDZ/Tyj+jD1ncfQ6XVsWLyFlX9d65jnpnheXjuHvPQCAH5Ys49/vLIKgFmLn2DQmP6UFpQxudcst3K0WcMQQngDOwCvxuWsklK+0KxGAO8CdwBm4FEp5UF3lqtD8EzPcTy1fzEFlnKWDH6SnQVpnK8qsNf4GbyZHT+epw8sId9SRrCnLwAZ5kIe2fN3+3y+Gv4c3+cfdycODVbJ0hcv8NzSOEIiPZl3z3H6jQgmtls7e83Y30cx9vdRABzcUsKGpXn4BdlWzbC7jdz22wgWzj7nVo6fWK2SGXOLSPoiktgoA0PvyGHsKB96dve018ycGsTMqUEAJG0y895HZfZmATD/43LiunlQUen+17J1CGbE3cUzqYsw1ZSxMHEGuwpPcKEq317jZ/Dm6R5382zqRxTUlBLk4QdAptnEpOS37fNZdcM8dpqOuZ9JJ3jyrUd4btybFGYX8/cdL7L3m1QyTuY41B3bfZrn73vb4TFrvZVFz33OmcMXaOfnzfydL3Fw63GnaX9eHh3T509kzshXKMwqZn7y6+xZv5+MtCyHuqM705g37g2n6Tct3c66+d/y7LJpV5zBnsXtObSsBhghpewN9AFGCyGub1ZzO9Ct8d9k4AN3Fxof2J4scxE51SXUSyubcw9zY3hPh5pRUX3Ynn+cfEsZACW1VU7zGRDalWxzEXmWUrfynD1SSURHL8I7eGPw1HH9mFAObClpsX7310UMHhtqH+45MAC/wNbr6ympNXTp5EHnjh54egruG+/LVxtb/nr1yrWV/M8EX/twVk49G7aYeewh/1bJ0yOgA9nVReRaiqmXVrbmH2KoMcGh5paIfuwsOEpBTSkApXWVTvPpF9KN7Ooi8i0tv7ZaxQ3oTM65fPLOm6ivs7J91T4Gj+mnadri/DLOHLZ9M7y60kLmqRyMUe5dEyRuYFdyzuSRl15AfV0921fuYsj4AZqnP7ozjYpi59fsSrRZw5A2P6X0aPzX/J4G44HljbV7gSAhRJQ7yw3zDqCgsREAFFjKCfN2vOBOe18j/h7tWJD4e5ZeP43bo/s6zee2yF5syjviThQAivPqCI3ysg+HRHpSkl/rsram2sqRnWUMHBXi9nJbkpNnJTa6aWshJkpPTq7rC8qYzQ1s3l7NXXc0NYzZLxTx2v8LQddK75ww70BMFzVlU00pYV7N1pePET+PdrzT7wk+THyakZH9neYzIqIPW/MPtUqm0OhgTFnF9uHC7GKM0c6/9D0HduWDPa/wl9Wz6Ngzxml8RAcjXXp35OT+s27lMcaEYMoqasqTVYwxJtSpLn5wdxam/o1Xk+bSMT7WrWW2pE0Pegoh9EKIQ0ABsFlKua9ZSQyQedFwVuNjzeczWQixXwixv67s0hcbEa4ebHbvFb3Q0SMghpkHlzLjwCc83nkE7X2M9vEGoWdYeE+25h295LKulHAZEg5uLaV7P3/77khbcHUbGtFCoKTNZgYP8LLvjnyz2UyYUU+/Xl4u61uLpPn60hPnH8ufDi3m2UOLeOSaW4lt57i+hhoT2F5wuFWW7+r1aH7/njOHzvPb+Jk8MXge6xZu5oXPn3IY7+3rxbwV01k4ZwXmCoubeZwfc8pzMJ2HO01lSt/ZrJu/gZfWPOvWMlvSpg1DSmmVUvYBYoGBQohrm5W4eqc6vaWllIuklAOklAM8An0uucwCSznhF21RhHsHYKopb1ZTxt7C01isdZTVmUktSaebf9PBu8HG7pwqz6G41v3NuJBID4pya+zDxXm1BIV7uqzdm1TM4LFtt3UBti2KrByrfTg710pUpN5l7T/XVfE/E/zsw7tTLCRtMtN9YCaPPGFi+w8WHp1W4HJarUyWMsK8g+zDYV5BFDZbX6aaUpKLTmFpqKWszszh0nN08W86cDwotAenK7IoaYX1BbYtirDYpvVgjAmhKLfUocZcYcFSZVuvKZuOoPfQExBqe630Bj3zVkxn68rd7Fp/wO08pqxiwmKbtiiMsSEU5RQ71JgrqrFU2RpT8obUxjyts9t4sV/ktKqUshTYDoxuNioLaH/RcCxw5UeHgLTyLNr7GIlqF4xB6Lktqjc7C9IcanYWnKB3cCf0QoeXzoOEwPacr2q6bNvIqN5sym2dT6vO1/mRd76Ggswa6msb2JtURP9bgpzqzBX1pCWX0//Wtr0G5oA+XpxJryM9o47aWsk/11UxdqRzEy4rb2DnXgt3jm4a95e5IZw90IHTye1Z/kEYw2/wZun8cLfynKrIJNbHSKR3CAahZ0REH3YXOh5o/sF0nOuCrrGvr/iAjmRcdBD7lsg+bGml3RGAUwfSiekSQURHIwYPPcPvHcTeb1IdaoLDmz6U4vp3RqfTUV5ka1gzF0wk81QOq+dvbJ08KWeI6RZFZKdwDB4Ght8/lD3r9zvmiQhqypPYtTFPRass/2JteZYkDKiTUpYKIdoBtwJ/bVa2HpgmhPgCGASUSSlz3VmuVTbwv2nrebf/4+iE4Ovs/aRXFXBX7EAA1mQlc77KxN7C0/xjyFM0SMn67P2cq7QdlffSeTAwtBtvnFjjTgw7vUHw6Asd+evjJ2mwwk33hhHbzYfvPrO94W99yPYLl7KphOtuCMTbx/HTfv7TZ0hLrqCipJ5pN6Ry74xYht/ndG1WzQwGwTuvhnLnQ3lYrfC7B/yJj/Pko+W2T/XfPxIAwLoNVdx6Yzt8fdr2M8UqG3j31Br+1vf36BBsyE3hfFU+42IGA7A+ew8Z5gKSi06xeNAspJQk5ewjvSoPsK2v/iHd+b+0f7VapgZrA+/P+pTX1s5Gp9ex6dMdXEjLZszEmwFIWryNYXclMnbSCKz1Vmqqa3n90QUAJAzuxq0PDeXcsUwW7H4ZgCUvriJl05UfD2uwNjB/+mJe//bP6PQ6Ni7ZxoUTWYz9w20AfP3hZm6893rGThmJtd5KbXUtrz7YdPZm7ooZ9BqeQKDRn88yFrL8xS/59pMruxataKt7qwohegHLAD22LZkvpZQvCyGmAEgpFzaeVp2PbcvDDDwmpdzf0jwB/LpHSXUR4JapiwBfnroI8OV9J1cdkFI6nYppsy0MKeURwOn0g5Ry4UU/S+DJtsqgKErrUn8ariiKZqphKIqimWoYiqJophqGoiiaqYahKIpmqmEoiqKZahiKomimGoaiKJqphqEoimaqYSiKoplqGIqiaKYahqIomqmGoSiKZv92txmQEqwNLVzj7iqwNHhc7QgOPMSva5UK0TaXT1CuDrWFoSiKZqphKIqimWoYiqJophqGoiiaqYahKIpmqmEoiqKZahiKomimGoaiKJqphqEoimaqYSiKoplqGIqiaKYahqIomqmGoSiKZr+urza2ksHGbszqORad0LEuK4Vl53Y41fQLuYZZPcdgEHpK68z8Yd9HdPQ18lqfB+w10T4hLPrxOz4/v9utPEd3lPD5q+eQDTDsvgjumBzrMP7bj7PY+1UhAFarJPesmXf2DMQvyINPnvuRI9tL8A/14JWvnW5Ve0W+3VrFH583YbXCxIcCmDM9xGH8/y4o4bPVtru519dD2o+15B/rTEiwns6J6fj76dDrwaAXJG/s4HaexJA4pnWbgF7oSMrdx+cXnG9w3TuoC9O6jccg9JTVVfF06gLa+4TxfMJv7TVR7UJZcu5b/pW10+1MA269jilvPoxer2PDsu/58q0kh/G9hvXgxS9mkHfBdmPnXesPsOKNdYTFhDD7o8kERwQiGyTfLNnG2gWb3c8zqg9T33kMnV7HhsVbWPnXtY55born5bVzyEsvAOCHNfv4xyurAJi1+AkGjelPaUEZk3vNcitHmzUMIUR7YDkQCTQAi6SU7zarCQT+AXRozPK/Usol7ixXh+DZhHFMS/6EfEs5y4ZMZUfBSdIrC+w1fgZv5iSM56mUJeRbygj29AXgQlUhD++ab5/PNyP+xLa8E+7EocEqWfHyOWYtSSA4wpNX7j1MnxEhRHf1sdeMnhTL6Em2JnJoazGbl+bgF2T72vzQu8O55TdRfDznR7dy/MRqlUyfa2LjyhhiowwMuj2DO0f6Eh/nZa95Zmowz0wNBuCrTZW8u6iUkGC9ffyWVbEYQ/VO874SOgQz4u5mduqHmGrKWDjgaXabjnPBnG+v8TV483Tc3cw59BEFNaUEefgBkGk28fuUt+zz+efQ5/mh8Jj7mXSCJ996hOfGvUlhdjF/3/Eie79JJeNkjkPdsd2nef6+tx0es9ZbWfTc55w5fIF2ft7M3/kSB7ced5r25+XRMX3+ROaMfIXCrGLmJ7/OnvX7yUjLcqg7ujONeePecJp+09LtrJv/Lc8um3bFGexZ3J5Dy+qBWVLKnsD1wJNCiPhmNU8CJ6SUvYHhwP8JITzdWWhCUCyZVUVkV5dQL61szj3CTeE9HWpGR/dmW95x8i1lAJTUVjnNJ9HYhSxzMXmWUnficO5IBeEdvQlr743BU8fAMWGkbilusT45ycTAsUb7cFxiIL6BrdfXk1MtdOnkQeeOHnh6Cu4f78/6jc7P/ydfrK3g/gn+rbb85noEdCDHXESupZh6aWVrQSpDwxIcam6N6MdO01EKakoBKK2rdJpPv5Bu5FQXkW8pcTtT3IDO5JzLJ++8ifo6K9tX7WPwmH6api3OL+PM4QsAVFdayDyVgzEq2L08A7uScyaPvPQC6uvq2b5yF0PGD9A8/dGdaVQUO79mV6LNGoaUMldKebDx5wogDYhpXgb4CyEE4AcUY2s0VyzMO9DeCADyLWWEeQc41HTwNRLg0Y6FAyexfMiT3BHtvKk/MqoXG3MOuxMFgNL8WkIim3pgcIQnpfk1Lmtrqq0c3VlK/5Ghbi+3Jdl59bSPaWpAMVEGsvNcv+RmcwMbt5m5Z4yf/TEhYPQD2SSOzGDRp2Uup/s5jF6B9kYAYKopw+gV6FAT6xOGv8GHt/s+wYcDnmZkZH+n+YwI78uW/FS38wCERgdjympq6oXZxRijnX/pew7sygd7XuEvq2fRsWfztzZEdDDSpXdHTu4/61YeY0wIpqyipjxZxRhjnN8j8YO7szD1b7yaNJeO8bFO41vDL3IMQwjRCegL7Gs2aj6wHsgB/IH7pZQNLqafDEwG8AwPaD7asdbFY7LZRZ/0QkePwGimJi/GS+fBJ4OncKw0gwyzbaUYhJ4bw3vy/qlNl39yl9F82baQrq8YdnhbMd36+dt3R9qCqzwtxOGrzVUMSWznsDuyc317oiMNFBTWM+r+bHp09eTGwe2uOI/W9dXdP5ZZqQvx1Hvwfv/pnCi7QFa17biPQegZYkzgo7NJLuZ2BZlcvCCyWagzh87z2/iZWKpqSBzZixc+f4rH+8yxj/f29WLeiuksnLMCc4XFzTzOjznlOZjOw52mYqmyMPD2vry05lkejXvKreW60uZnSYQQfsC/gKellOXNRo8CDgHRQB9gvhDCqSNIKRdJKQdIKQcYAnyaj3ZQYCkjwrvpEyrCO5DCmnKnmj2mH7FY6yirM5NafJ5uAVH28UPCunOyPIfiWvc344IjPSnOq7UPl+TXEhTueq8rOamQgWPC3F7mpcRGGcjMbtqiyM6tJzrC9efGyrUVPDDBz+Gx6EhbbbjRwITb/Ug55N4vg6mmjHCvIPtwmFcgRbVlTjXJxSexNNRSXlfFkdJzdPGLto8fFNqD05VZlLjYVbkShdnFhMU2HQg2xoRQlFvqUGOusGCpsm0ppmw6gt5DT0Co7bXSG/TMWzGdrSt3s2v9AbfzmLKKCYtt2qIwxoZQlOO4W2uuqMZSZVsXyRtSG/O0/q5kmzYMIYQHtmaxQkq52kXJY8BqaXMGSAd6uLPME2XZdPA1Et0uGIPQc1tUL3YUpDnUfJ+fRt/gTuiFDi+dB9cGted8pck+flRUbza1wu4IwDXX+ZN/vhpTpoX62gaSk0z0GRHiVGeuqOdUSjl9b3Ee15oS+3hzJr2W9Iw6amslK9dVcOcoX6e6snIrO/ZWM350U8OoMjdQUdlg/3nz92YS4tw65MTJikxifIxEeodgEHpGhPdld+Fxh5pdpmP0CuyMrnF99QzowAVz00HsERF92dpKuyMApw6kE9MlgoiORgweeobfO4i93zjOPzi86UMprn9ndDod5UW2hjVzwUQyT+Wwev7G1smTcoaYblFEdgrH4GFg+P1D2bN+v2OeiKCmPIldG/NUtMryL9aWZ0kEsBhIk1K+1UJZBnALsFMIEQHEAefcWa5VNvDmifW8l/gYeiFYn3WAc5UF3N1+IACrM5M5X2Vid+FpPrvhKaSUrMtK4Wyl7ai8l86DgcauvHZ8jTsx7PQGwcPPd+btScdpsMIN94QT082H7Z/nAjD8QduWzcHNRSQMDcLLx/Hsw4czT3EquYzKknqeuTGF8dM7MOy+iCvOYzAI3nstnNsfzMZqhcceCCAhzouFy0oBmPK7IADWbKjitpt88PVp+kzJN1m553Hb0f76enjwLn9Gj3BuNj9Hg2zgvdOrebPPZHRCsCEnmfNV+dwZPRiAr3L2kGEuILn4JIsHzkJKSVLOPs5X5QG29dU/pDtvnVzlVg6HTNYG3p/1Ka+tnY1Or2PTpzu4kJbNmIk3A5C0eBvD7kpk7KQRWOut1FTX8vqjCwBIGNyNWx8ayrljmSzY/TIAS15cRcqmI27lmT99Ma9/+2d0eh0bl2zjwoksxv7hNgC+/nAzN957PWOnjMRab6W2upZXH2w6ezN3xQx6DU8g0OjPZxkLWf7il3z7ifOpay1E832h1iKEuAHYCRzFdloVYC62U6hIKRcKIaKBpUAUtt3ZN6SU/7jUfH27Rcn49x5rk8xX4okuzn/jcTU9GlBw+aJf0C0n7rzaEZx4jiu82hEcNJjNVzuCk+/kqgNSSqdTMW22hSGl/AHXx7QurskBRrZVBkVRWpf603BFUTRTDUNRFM1Uw1AURTPVMBRF0Uw1DEVRNFMNQ1EUzVTDUBRFM9UwFEXRTDUMRVE0Uw1DURTNVMNQFEUz1TAURdHsP/Kq4b+khkt/v+6/npTq9flPorYwFEXRTDUMRVE0Uw1DURTNVMNQFEUz1TAURdFMNQxFUTRTDUNRFM1Uw1AURTPVMBRF0Uw1DEVRNFMNQ1EUzVTDUBRFM9UwFEXR7D/y26qDjd2Y1XMsOqFjXVYKy8453/+0X8g1zOo5BoPQU1pn5g/7PqKjr5HX+jxgr4n2CWHRj9/x+fndbuU5tqOEL149R0ODZNh9Edw+ub3D+I0fZ7H3K9vd4xusktyzZt7eMwjfIA+WPneaI9tL8A/14KWv+7mV4yffbq3ij8+bsFph4kMBzJnueMf4/11QwmerywHbTZfTfqwl/1hnQoL1dE5Mx99Ph14PBr0geWMHt/MMDIljWvfx6IWOpJx9fHZhm1NNn6AuTOs+Dr3QU1ZXxdMHP6C9TxgvXPsbe01Uu1CWnNvIqsydbmcacOt1THnzYfR6HRuWfc+XbyU5jO81rAcvfjGDvAu29bZr/QFWvLGOsJgQZn80meCIQGSD5Jsl21i7YLP7eUb1Yeo7j6HT69iweAsr/7rWMc9N8by8dg556bZ76/6wZh//eMV2g+pZi59g0Jj+lBaUMbnXLLdytOXd29sDy4FIbDdjXiSlfNdF3XDgHcADKJRS3uTOcnUInk0Yx7TkT8i3lLNsyFR2FJwkvbLpJsV+Bm/mJIznqZQl5FvKCPa03YH8QlUhD++ab5/PNyP+xLa8E+7EocEq+ezls/xxybUER3jy6r2H6D0ilOiuPvaaUZNiGTUpFoDDW4vYvDQH3yAPAIbcHcHNv4nmkzmn3crxE6tVMn2uiY0rY4iNMjDo9gzuHOlLfJyXveaZqcE8MzUYgK82VfLuolJCgpvuKr9lVSzGUL3TvK+EDsGMuLt4JnURppoyFibOYFfhCS5U5dtr/AzePN3jbp5N/YiCmlKCPPwAyDSbmJT8tn0+q26Yx07TMfcz6QRPvvUIz417k8LsYv6+40X2fpNKxskch7pju0/z/H1vOzxmrbey6LnPOXP4Au38vJm/8yUObj3uNO3Py6Nj+vyJzBn5CoVZxcxPfp096/eTkZblUHd0Zxrzxr3hNP2mpdtZN/9bnl027Yoz2LO4PYeW1QOzpJQ9geuBJ4UQ8RcXCCGCgAXAOCllAnCfuwtNCIols6qI7OoS6qWVzblHuCm8p0PN6OjebMs7Tr6lDICS2iqn+SQau5BlLibPUupWnvQjFYR19CasvTcGTx2JY8I4tKWoxfrkpEIGjg2zD3dPDMQ3sPX6enKqhS6dPOjc0QNPT8H94/1Zv9H5+f/ki7UV3D/Bv9WW31yPgA5kVxeRaymmXlrZmn+IocYEh5pbIvqxs+AoBTWlAJTWVTrNp19IN7Kri8i3lLidKW5AZ3LO5ZN33kR9nZXtq/YxeIy2rbvi/DLOHL4AQHWlhcxTORijgt3LM7ArOWfyyEsvoL6unu0rdzFkvNON1Vt0dGcaFcXOr9mVaLOGIaXMlVIebPy5AkgDYpqVPQSsllJmNNYV4KYw70B7IwDIt5QR5h3gUNPB10iARzsWDpzE8iFPckd0X6f5jIzqxcacw+7GoTS/lpDIpk/v4AgvSvNrXdbWVFs5trOE/iND3V5uS7Lz6mkf09SAYqIMZOfVu6w1mxvYuM3MPWP87I8JAaMfyCZxZAaLPi1zOd3PEeYdiOmipmyqKSXMK9Chpr2PET+PdrzT7wk+THyakZH9neYzIqIPW/MPuZ0HIDQ6GFNWsX24MLsYY7TzL33PgV35YM8r/GX1LDr2bP7WhogORrr07sjJ/WfdymOMCcGU1fQhU5hVjDHG+T0SP7g7C1P/xqtJc+kYH+vWMlvyixzDEEJ0AvoC+5qN6g54CCG2A/7Au1LK5S6mnwxMBvAMD2g+2rHWxWNSOg7rhY4egdFMTV6Ml86DTwZP4VhpBhlm20oxCD03hvfk/VObLv/kLqP5slsMCRzZVkzXfv723ZG24CqPaCHPV5urGJLYzmF3ZOf69kRHGigorGfU/dn06OrJjYPbtW5GHEPqhZ44/1hmHvwQL72B9wdM50TZBbKqCwHb+hpqTOCjs9+0yvKFixdENnvhzhw6z2/jZ2KpqiFxZC9e+PwpHu8zxz7e29eLeSums3DOCswVFjfzOD/mlOdgOg93moqlysLA2/vy0ppneTTuKbeW60qbnyURQvgB/wKellKWNxttAPoDY4BRwDwhRPfm85BSLpJSDpBSDjAE+DQf7aDAUkaEd9MnVIR3IIU15U41e0w/YrHWUVZnJrX4PN0Couzjh4R152R5DsW17m/GBUd6UpxXYx8uya8hKNzTZW1ykomBY8JcjmstsVEGMrObtiiyc+uJjnD9ubFybQUPTPBzeCw60lYbbjQw4XY/Ug6598tgspQR5h1kHw7zCnJaX6aaUpKLTmFpqKWszszh0nN08Y+2jx8U2oPTFVmUtML6AtsWRVhs04FgY0wIRbmlDjXmCguWKtt6Tdl0BL2HnoBQ22ulN+iZt2I6W1fuZtf6A27nMWUVExbbtEVhjA2hKKfYocZcUY2lyrYukjekNuZp/V3JNm0YQggPbM1ihZRytYuSLOBbKWWVlLIQ2AH0dmeZJ8qy6eBrJLpdMAah57aoXuwoSHOo+T4/jb7BndALHV46D64Nas/5SpN9/Kio3mxqhd0RgE7X+VNwvhpTpoX62gZSkkz0HhHiVGeuqOd0Sjl9bmm73RGAxD7enEmvJT2jjtpaycp1Fdw5yteprqzcyo691Ywf3dQwqswNVFQ22H/e/L2ZhDjXzU+rUxWZxPoYifQOwSD0jIjow+7C4w41P5iOc13QNfb1FR/QkYyqpr3XWyL7sKWVdkcATh1IJ6ZLBBEdjRg89Ay/dxB7v0l1qAkOb/pQiuvfGZ1OR3mRrWHNXDCRzFM5rJ6/sXXypJwhplsUkZ3CMXgYGH7/UPas3++YJyKoKU9i18Y8Fa2y/Iu15VkSASwG0qSUb7VQtg6YL4QwAJ7AIODtFmo1scoG3jyxnvcSH0MvBOuzDnCusoC72w8EYHVmMuerTOwuPM1nNzyFlJJ1WSmcrbQdlffSeTDQ2JXXjq9xJ4ad3iB46PkuvDPpGNIKQ++JIKabL9s/zwVg+IO2LZvUzUUkDA3Cy8fx7MOimSc5nVxGZUk9s29MZtz0Dgy7L/KK8xgMgvdeC+f2B7OxWuGxBwJIiPNi4bJSAKb8LgiANRuquO0mH3x9mj5T8k1W7nncdrS/vh4evMuf0SOcm83PYZUNvHtqDX/r+3t0CDbkpnC+Kp9xMYMBWJ+9hwxzAclFp1g8aBZSSpJy9pFelQfY1lf/kO78X9q/3MpxsQZrA+/P+pTX1s5Gp9ex6dMdXEjLZszEmwFIWryNYXclMnbSCKz1Vmqqa3n90QUAJAzuxq0PDeXcsUwW7H4ZgCUvriJl0xG38syfvpjXv/0zOr2OjUu2ceFEFmP/cBsAX3+4mRvvvZ6xU0ZirbdSW13Lqw82/RrNXTGDXsMTCDT681nGQpa/+CXffrL1irKI5vtCrUUIcQOwEziK7bQqwFygA4CUcmFj3Wzgscaaj6WU71xqvr7domT8e4+1SeYr8Ycu7p/zb02PB+RfvugXNOL4uKsdwYnXeNPli35BDWbz1Y7g5Du56oCU0ulUTJttYUgpf6DFw3sOdX8D/tZWORRFaT3qT8MVRdFMNQxFUTRTDUNRFM1Uw1AURTPVMBRF0Uw1DEVRNFMNQ1EUzVTDUBRFM9UwFEXRTDUMRVE0u2TDEEJ0FUIMdfH4MCFEl7aLpSjKr9HltjDeAVx9R7a6cZyiKP9FLtcwOkkpnb6XK6XcD3Rqk0SKovxqXe7bqt6XGNe612XTSrR8STkFGuxXElCU1ne5LYwUIcTvmz8ohJgIuH/tMUVR/q1cbgvjaWCNEOJhmhrEAGxXx7qrDXMpivIrdMmGIaXMB4YIIW4Grm18OElKeWXX91IU5d+apituSSm3Ac73r1MU5b+K+sMtRVE0Uw1DURTNVMNQFEUz1TAURdFMNQxFUTRTDUNRFM1Uw1AURTPVMBRF0azNbpV4NV1v7MasnmPRoWNdVgrL03c41fQLuYaZPcZgEHpK68xMSf6IDr5GXuv9gL0m2ieERT9+xxcXdruV59iOEr549RwNDZJh90Vw++T2DuM3fpzF3q9s9/tssEpyz5p5e88gfIM8WPrcaY5sL8E/1IOXvu7nVg778raamfl8IVar5PGHAnh2erDD+P9bUMJnq213IrfWS9J+rCP3WCdCgm03irZaJYNGZxETaWDdp1Fu5xkYEse07uPRCx1JOfv47ILz3wj2CerCtO7j0As9ZXVVPH3wA9r7hPHCtb+x10S1C2XJuY2synT/frcDbr2OKW8+jF6vY8Oy7/nyrSSH8b2G9eDFL2aQd8G23natP8CKN9YRFhPC7I8mExwRiGyQfLNkG2sXbHY/z6g+TH3nMXR6HRsWb2HlX9c65rkpnpfXziEv3XZX+x/W7OMfr6wCYNbiJxg0pj+lBWVM7jXLrRxteff29sByIBLbjZYXSSnfbaE2EdgL3C+lXOXOcnUIno0fx7SUTyiwlLNs8FR2FpwkvarAXuNn8ObZ+PHM2L+EfEsZwZ62O5BnVBXym93z7fNJuvlPbM8/4U4cGqySz14+yx+XXEtwhCev3nuI3iNCie7qY68ZNSmWUZNiATi8tYjNS3PwDfIAYMjdEdz8m2g+mXParRw/sVolT801sWFlNLFRBq6/PYuxI32Jj/O018yaGsysqbYm8vWmKt5dVGpvFgDvfVRGz26elFe4/81YHYIZcXfxTOoiTDVlLEycwa7CE1yoarqptJ/Bm6d73M2zqR9RUFNKkIcfAJlmE5OS37bPZ9UN89hpOuZ+Jp3gybce4blxb1KYXczfd7zI3m9SyTiZ41B3bPdpnr/vbYfHrPVWFj33OWcOX6Cdnzfzd77Ewa3Hnab9eXl0TJ8/kTkjX6Ewq5j5ya+zZ/1+MtKyHOqO7kxj3rg3nKbftHQ76+Z/y7PLpl1xBnsWt+fQsnpglpSyJ3A98KQQIr55kRBCD/wV2NgaC00IiiXLXEROdQn10sqmvCPcGNHToWZUVG+25x8n31IGQEltldN8EkO7kGUuJs9S6lae9CMVhHX0Jqy9NwZPHYljwji0pajF+uSkQgaODbMPd08MxDew9fp6cmoNXTp50LmjB56egvvH+/HVRufn/5OVayu5f4K/fTgrp54NW8w8/pB/i9P8HD0COpBdXUSupZh6aWVr/iGGGhMcam6J6MfOgqMU1JQCUFpX6TSffiHdyK4uIt9S4namuAGdyTmXT955E/V1Vrav2sfgMdq27orzyzhz+AIA1ZUWMk/lYIwKvsxUl8kzsCs5Z/LISy+gvq6e7St3MWS8043VW3R0ZxoVxc6v2ZVos4YhpcyVUh5s/LkCSANiXJROB/4FFLgY97OFeQWSX11mHy6wlBHmFeBQ08HXiL+hHR8MnMSywU9yR3Rfp/ncFtWLTbmH3c5Tml9LSKSXfTg4wovS/FqXtTXVVo7tLKH/yFC3l9uSnLx6YmOaGlBMlIHsvHqXtWZzAxu3mbl7jK/9sVnPF/L6/wtFp2udi5KEeQdiuqgpm2pKCfMKdKhp72PEz6Md7/R7gg8Tn2ZkZH+n+YyI6MPW/EOtkik0OhhTVrF9uDC7GGO08y99z4Fd+WDPK/xl9Sw69nR+a0d0MNKld0dO7j/rVh5jTAimrKYPmcKsYowxzu+R+MHdWZj6N15NmkvH+Fi3ltmSX+QYhhCiE9AX2Nfs8RhsX5MfASReYvrJwGQAz/CAlspstRry6IWOHoHRPJmyGC+dB4uvn8Kx0gwyzLaVYhB6bgzvyYLTmzTM7dKk1B7yyLZiuvbzt++OtAVXeVq6INHXm80MSfS2744kba4izKinf28vvt9d3XYZcQypF3ri/GOZefBDvPQG3h8wnRNlF8iqLgRs62uoMYGPzn7TKssXLl4Q2eyFO3PoPL+Nn4mlqobEkb144fOneLzPHPt4b18v5q2YzsI5KzBXWNzM4/yYU56D6TzcaSqWKgsDb+/LS2ue5dG4p9xarittfpZECOGHbQviaSllebPR7wBzpJTWS81DSrlISjlASjnAEOhzqVIKasqIaNf0CRXuHYipxnGxBZYy9hb+iMVaR1mdmUMl5+nm33TwbkhYd06W51Bc6/5mXHCkJ8V5NfbhkvwagsI9XdYmJ5kYOCbM5bjWEhNlICu7aYsiO7ee6AjXnxtfrq3k/gl+9uHdyRa+3lRF18QLPDwln20/VPPIk/kup9XKZCkjzDvIPhzmFURhs/VlqikluegUloZayurMHC49Rxf/aPv4QaE9OF2RRUkrrC+wbVGExYbYh40xIRTlljrUmCssWKps6zVl0xH0HnoCQm2vld6gZ96K6WxduZtd692/zpQpq5iw2KYtCmNsCEU5xQ415opqLFW2xpS8IbUxT+vsNl6sTRuGEMIDW7NYIaVc7aJkAPCFEOI8cC+wQAgxwZ1lnijLpr2Pkeh2wRiEnpGRvdhZkOZQs6MgjT7BndALHV46DxIC25NeZbKPHxnVu1V2RwA6XedPwflqTJkW6msbSEky0XtEiFOduaKe0ynl9Lml7XZHABL7eHEmvY70jDpqayUr11UydpSvU11ZuZUde6sZN7pp3Kt/DuX8wU6cSenIioUR3HxDO5a/H+FWnlMVmcT6GIn0DsEg9IyI6MPuwuMONT+YjnNd0DX29RUf0JGMiw5i3xLZhy2ttDsCcOpAOjFdIojoaMTgoWf4vYPY+02qQ01weNOHUlz/zuh0OsqLbA1r5oKJZJ7KYfX8Vjksx6mUM8R0iyKyUzgGDwPD7x/KnvX7HfNEBDXlSezamMfV9bvd05ZnSQSwGEiTUr7lqkZKec1F9UuBr6WUa91ZrlU28LcT63lvwGPohOCrrAOcqyzg7vYDAVidmcz5KhN7TKdZMfQppJSsy0rhXKXtk9JL58Gg0K68fnyNOzHs9AbBQ8934Z1Jx5BWGHpPBDHdfNn+eS4Awx+0bdmkbi4iYWgQXj56h+kXzTzJ6eQyKkvqmX1jMuOmd2DYfZFXnMdgELz7mpExD+ZitUoefSCAhDhPPlxmO+7zh9/ZfhHWbqjitpt88PVp241Qq2zg3VNr+Fvf36NDsCE3hfNV+YyLGQzA+uw9ZJgLSC46xeJBs5BSkpSzj/SqPMC2vvqHdOf/0v7VapkarA28P+tTXls7G51ex6ZPd3AhLZsxE28GIGnxNobdlcjYSSOw1lupqa7l9UcXAJAwuBu3PjSUc8cyWbD7ZQCWvLiKlE1O19L+WXnmT1/M69/+GZ1ex8Yl27hwIouxf7gNgK8/3MyN917P2CkjsdZbqa2u5dUHm87ezF0xg17DEwg0+vNZxkKWv/gl335yZdfAEs33hVqLEOIGYCdwFOxXpp0LdACQUi5sVr8UW8O45GlV3+5RMuG9x1o975Wa3Nn9c/6t6dGA3KsdwcGtxydc7QhOvMabLl/0C2owm692BCffyVUHpJROp2LabAtDSvkD2o5B/lT/aFtlURSldag/DVcURTPVMBRF0Uw1DEVRNFMNQ1EUzVTDUBRFM9UwFEXRTDUMRVE0Uw1DURTNVMNQFEUz1TAURdFMNQxFUTRTDUNRFM1Uw1AURbN/u9sMSCmoteovX/hfyirdv5J3a6pr+PWtK6/LlygtUFsYiqJophqGoiiaqYahKIpmqmEoiqKZahiKomimGoaiKJqphqEoimaqYSiKoplqGIqiaKYahqIomqmGoSiKZqphKIqi2b/dl8+0GBLWlWfjx6ATgjWZB1hy1vn+pwNCOjE7/g4MOj0ltVVM2vsJAP4Gb57vNYGu/uFI4MXDazhSmulWnmM7Svji1XM0NEiG3RfB7ZPbO4zf+HEWe7+y3e+zwSrJPWvm7T2D8A3yYOlzpzmyvQT/UA9e+rqfWzl+smmbmVnzirE2SB570J/Z04Mcxr+1oJQvVlcBUG+VnPyxjqyjHQgJtn2RzGqVDBmdQ3SUnjXLr/zG0D8ZFNqdp+PG2W6enZ3CP85vd6rpG9yZGXF3YhB6SuuqmLb/Qzr4GHm518P2muh2IXx8djNfZvzgdqYBt17HlDcfRq/XsWHZ93z5VpLD+F7DevDiFzPIu2Bbb7vWH2DFG+sIiwlh9keTCY4IRDZIvlmyjbULNrufZ1Qfpr7zGDq9jg2Lt7Dyr2sd89wUz8tr55CXbrur/Q9r9vGPV2y3KZ61+AkGjelPaUEZk3vNcitHW969vT2wHIjEdjPmRVLKd5vVPAzMaRysBJ6QUh52Z7k6BM8l3MmUfUvJt5Sz4oYpfJ9/knOVTTfg9Td489y1d/Jk8nLyLGUEe/raxz2bcAe7TT8y++AXGISednoPd+LQYJV89vJZ/rjkWoIjPHn13kP0HhFKdFcfe82oSbGMmhQLwOGtRWxemoNvkG25Q+6O4ObfRPPJnNNu5fiJ1SqZMbeIpC8iiY0yMPSOHMaO8qFnd097zcypQcycGgRA0iYz731UZm8WAPM/LieumwcVle5/M1aHYFaPCTx98GMKLGV8PGgaP5hOcL6qwF7jZ/BmVo8JzEr9hHxLKUEetvWVYS7k0b3v2uez9sY/833BMfcz6QRPvvUIz417k8LsYv6+40X2fpNKxskch7pju0/z/H1vOzxmrbey6LnPOXP4Au38vJm/8yUObj3uNO3Py6Nj+vyJzBn5CoVZxcxPfp096/eTkZblUHd0Zxrzxr3hNP2mpdtZN/9bnl027Yoz2LO4PYeW1QOzpJQ9geuBJ4UQ8c1q0oGbpJS9gFeARe4u9NqgWDLNRWRXl1AvrWzMOcrwiJ4ONbfH9GJr3gnyLGUAlNTaPk19DV70C+nEmswDticgrVTUW9zKk36kgrCO3oS198bgqSNxTBiHthS1WJ+cVMjAsWH24e6JgfgGtl5fT0mtoUsnDzp39MDTU3DfeF++2tjy3cNXrq3kfyY0NdSsnHo2bDHz2EP+rZKnZ2B7ssxF5FQXUy+tbMk7zLAwx7fJbZF9+L7gGPmWUgBK66qc5jMgpCvZ1UX2GnfEDehMzrl88s6bqK+zsn3VPgaP0bZ1V5xfxpnDFwCorrSQeSoHY1Swe3kGdiXnTB556QXU19WzfeUuhox3urF6i47uTKOiuNKtDD9ps4YhpcyVUh5s/LkCSANimtXsllKWNA7uBWLdXW64dwB51WX24XxLGeHejm/ujr6hBHi04+PrH+ezG6YwNqYPALE+wZTUVvFyr7v44oapPH/deLzd3MIoza8lJLLpCgzBEV6U5te6rK2ptnJsZwn9R4a6tcxLycmzEhvdtLUQE6UnJ7feZa3Z3MDm7dXcdUdTw5j9QhGv/b8QdK30zgnzCqSgptQ+XFBTRphXoENNB58w/D3a8ff+k1k8aDqjo5x/eW+J7M13eYdaJVNodDCmrGL7cGF2McZo51/6ngO78sGeV/jL6ll07BnjND6ig5EuvTtycv9Zt/IYY0IwZTV9yBRmFWOMcX6PxA/uzsLUv/Fq0lw6xrv9q+TSL3LQUwjRCegL7LtE2URgQwvTTxZC7BdC7K8vc/50cah18ZhsNqwXOnoGRjMt5VOm7lvO5G7D6eAbil7o6BEQxZcZKTzwwwIs1joe73LjJZd3ObL5wlsKCRzZVkzXfv723ZG24CqPEK4DJW02M3iAl3135JvNZsKMevr1ar1L0LheX44h9UJHD/9YZqcuYebBxTza+Rba+xjt4w1Czw1h8WzNP9o6mVy8HrLZC3fm0Hl+Gz+TJwbPY93Czbzw+VMO4719vZi3YjoL56zAXOHeVqqr1eOU52A6D3eaypS+s1k3fwMvrXnWrWW2pM0bhhDCD/gX8LSUsryFmpuxNYw5rsZLKRdJKQdIKQcYAn1dldjlW8qJbNf0CRXhHYjJUuFUs9v0IxZrHaV1Zg4UnyfOP5J8SzkFlnKOldr2DTfnHqdnYJT2J+tCcKQnxXk19uGS/BqCwj1d1iYnmRg4JszluNYSE6UnK8dqH87OtRIV6fqqWP9cV8X/TPCzD+9OsZC0yUz3gZk88oSJ7T9YeHRagctptSqoKSPcK8g+HO4VSGFNuVPN3qJTWBrqKKszc6gkna5+TevlemMcpyuyKaltnc3uwuxiwmJD7MPGmBCKcksdaswVFixVtvWasukIeg89AaG210pv0DNvxXS2rtzNrvUH3M5jyiomLLZpi8IYG0JRTrFDjbmiGkuVrTElb0htzNM6u40Xa9OGIYTwwNYsVkgpV7dQ0wv4GBgvpWx5516j42XZdPANJbpdEAahZ1T0dXyff9KhZnv+SfqGdEQvdHjrPLguKJZzlSaKairJs5TR0df26TXI2JlzFSZXi9Gs03X+FJyvxpRpob62gZQkE71HhDjVmSvqOZ1STp9b2m53BGBAHy/OpNeRnlFHba3kn+uqGDvSx6murLyBnXst3Dm6adxf5oZw9kAHTie3Z/kHYQy/wZul88PdynOyPItYn1CivIMxCD23RPbmB1OaQ81O0wl6B12DXujw0nmQENje4aDobZF92Jzn1rFyB6cOpBPTJYKIjkYMHnqG3zuIvd+kOtQEhzd9KMX174xOp6O8yNawZi6YSOapHFbP39g6eVLOENMtishO4Rg8DAy/fyh71u93zBMR1JQnsWtjngpaW1ueJRHAYiBNSvlWCzUdgNXAb6WUrXIawCobeOPY13ww8HfohI51WQc5W1nAvR0SAViVkUJ6pYndph/5ctiTSCRrMg5wttL2Bvzr8SRe63MvHjo92eYSnj/sss9ppjcIHnq+C+9MOoa0wtB7Iojp5sv2z3MBGP6g7ZMydXMRCUOD8PJx/LRfNPMkp5PLqCypZ/aNyYyb3oFh9135qUyDQfDOq6Hc+VAeViv87gF/4uM8+Wi57VP9948EALBuQxW33tgOX5+23Qi1ygbePrWOt/pNRC90fJ2TQnpVPhNiBwGwNmsfF6oK2Fd0imXXP41E8lW2rQbAS+dBYkhX3kxzbz1drMHawPuzPuW1tbPR6XVs+nQHF9KyGTPxZgCSFm9j2F2JjJ00Amu9lZrqWl5/dAEACYO7cetDQzl3LJMFu18GYMmLq0jZdMStPPOnL+b1b/+MTq9j45JtXDiRxdg/3AbA1x9u5sZ7r2fslJFY663UVtfy6oNNZ2/mrphBr+EJBBr9+SxjIctf/JJvP9l6RVlE832h1iKEuAHYCRzFdloVYC7QAUBKuVAI8TFwD3ChcXy9lPKSh399ukXL7u9MbJPMV2Jat+1XO4KD3/pnX+0IDkYcu+9qR3Did1fe1Y7goMHc8lmqq+U7ueqAq9/FNtvCkFL+QIuH9+w1k4BJbZVBUZTWpf40XFEUzVTDUBRFM9UwFEXRTDUMRVE0Uw1DURTNVMNQFEUz1TAURdFMNQxFUTRTDUNRFM1Uw1AURTPVMBRF0Uw1DEVRNFMNQ1EUzf4tbzPQRt/IvyI6pwsAXl0e4te1SoX4db0+invUFoaiKJqphqEoimaqYSiKoplqGIqiaKYahqIomqmGoSiKZqphKIqimWoYiqJophqGoiiaqYahKIpmqmEoiqKZahiKomimGoaiKJr9ur7a2EqGhHVlTsIYdEKwJuMAn5zd6VQzILQTs+PvwEOnp6S2iol7PgHA3+DNC70n0NU/HCnhhcNrOFKa6VaeoztK+PzVc8gGGHZfBHdMjnUY/+3HWez9qhAAq1WSe9bMO3sG4hfkwSfP/ciR7SX4h3rwytd93cphX97WKv74vAmrFSY+FMCc6SEO4/93QQmfrbbdzb2+HtJ+rCX/WGdCgvV0TkzH30+HXg8GvSB5Ywe38wwK7c6M7uPRCcHX2cn848J2p5q+wZ15qvs4DEJHaZ2Z6QcW0t4njJeve9heE90uhI/PbuKfmT+4nWnArdcx5c2H0et1bFj2PV++leQwvtewHrz4xQzyLpgA2LX+ACveWEdYTAizP5pMcEQgskHyzZJtrF2w2f08o/ow9Z3H0Ol1bFi8hZV/XeuY56Z4Xl47h7z0AgB+WLOPf7yyCoBZi59g0Jj+lBaUMbnXLLdytFnDEEK0B5YDkdju3r5ISvlusxoBvAvcAZiBR6WUB91Zrg7B3Gvv5A/7lpJfXc5nw6awPf8k5ypN9hp/gzdzr72TqfuWk2cpI8TT1z7u2YQ72FXwI88c+AKD0NNO7+FOHBqskhUvn2PWkgSCIzx55d7D9BkRQnRXH3vN6EmxjJ5kayKHthazeWkOfkG25Q69O5xbfhPFx3N+dCvHT6xWyfS5JjaujCE2ysCg2zO4c6Qv8XFe9ppnpgbzzNRgAL7aVMm7i0oJCdbbx29ZFYsxVO807yuhQzAz7i7+mPoRBZYyPh44nR8KT3C+qsBe42fwZmbcXTyTupj8mlKCPGzrK9Ns4rF979jns2bY/2OH6Zj7mXSCJ996hOfGvUlhdjF/3/Eie79JJeNkjkPdsd2nef6+tx0es9ZbWfTc55w5fIF2ft7M3/kSB7ced5r25+XRMX3+ROaMfIXCrGLmJ7/OnvX7yUjLcqg7ujONeePecJp+09LtrJv/Lc8um3bFGexZ3J5Dy+qBWVLKnsD1wJNCiPhmNbcD3Rr/TQY+cHeh1wbFkllVRLa5hHpp5dvsowyP6Om40JhebMk7QZ6lDIDi2ioAfA1e9A/txJrMA7YnIK1U1FvcynPuSAXhHb0Ja++NwVPHwDFhpG4pbrE+OcnEwLFG+3BcYiC+ga3X15NTLXTp5EHnjh54egruH+/P+o1VLdZ/sbaC+yf4t9rym+sZ2J6s6kJyqoupl1a+yz/MDWEJDjW3RfZlh+kY+TWlAJTWOeftH9KV7Ooi8i2lbmeKG9CZnHP55J03UV9nZfuqfQwe00/TtMX5ZZw5fAGA6koLmadyMEYFu5dnYFdyzuSRl15AfV0921fuYsj4AZqnP7ozjYriSrcy/KTNGoaUMvenrQUpZQWQBsQ0KxsPLJc2e4EgIUSUO8sNbxdgbwQABZYyIto5vuE7+oYS4NGOjwc/zuc3TGFsTB8AYn2CKamt4uXed7Fy2FRe6DXe7S2M0vxaQiI97cPBEZ6U5te4rK2ptnJ0Zyn9R4a6tcxLyc6rp31MUwOKiTKQnVfvstZsbmDjNjP3jPGzPyYEjH4gm8SRGSz6tMzldD9HmFcgBRetL5OljDCvAIea9j5G/A3t+Hv/P7B44FOMjnL+5b01sg/f5R1yOw9AaHQwpqympl6YXYwx2vmXvufArnyw5xX+snoWHXs2f2tDRAcjXXp35OT+s27lMcaEYMoqasqTVYwxxvk9Ej+4OwtT/8arSXPpGB/rNL41/CLHMIQQnYC+wL5mo2KAiw8QZDU+ltts+snYtkDwCHN8Mzkty8Vjza/QZdDpiA+MZvLeJXjpPFh+w2SOlmaiFzp6BETxxrEkjpZm8WzCHTze5UbeP73lss+xJS6vDiZcpYTD24rp1s/fvjvSFlzlaSEOX22uYkhiO4fdkZ3r2xMdaaCgsJ5R92fTo6snNw5ud8V5XK6vZsN6oSMuIIYZBxbhpfdgYeI0jpdlkGm2HfcxCD1DjfEsPLPhinM4ZHLxgshmL9yZQ+f5bfxMLFU1JI7sxQufP8XjfebYx3v7ejFvxXQWzlmBucK9rVRX68cpz8F0Hu40FUuVhYG39+WlNc/yaNxTbi3XlTY/SyKE8AP+BTwtpSxvPtrFJE5vaSnlIinlACnlAEOgr4tJmuRXlxPpHWgfDvcOpMBS4VSzy/Qj1dY6SuvMHCw6T/eASPIt5eRbyjlaats33Jx7nB6Bbm3wEBzpSXFerX24JL+WoHBPl7XJSYUMHBPm1vIuJzbKQGZ20xZFdm490RGuPzdWrq3ggQl+Do9FR9pqw40GJtzuR8oh934ZCmrKCL9ofYV5B1JY4/g2MVnK2Fd0GktDHWV1Zg6XnKOrX9N6ud4Yx+mKbEpqW2ezuzC7mLDYpgPBxpgQinJLHWrMFRYsVbYtxZRNR9B76AkItb1WeoOeeSums3XlbnatP+B2HlNWMWGxTVsUxtgQinIcd2vNFdVYqmzrInlDamOe1t+VbNOGIYTwwNYsVkgpV7soyQLaXzQcC1z50SHgeFk2HXxDiWkXhEHoGR1zHd/nn3So2ZZ/kn4hHdELHd46D64LiiW90kRRTSX51WV09LUdQxhk7OxwsPRKXHOdP/nnqzFlWqivbSA5yUSfESFOdeaKek6llNP3FudxrSmxjzdn0mtJz6ijtlaycl0Fd45ybsJl5VZ27K1m/OimhlFlbqCissH+8+bvzSTEuW5+Wp0sz6J9OyNR3sEYhJ5bI3qzy3TCoWan6QS9gjqhFzq8dB7EB3ZwOCh6a0Tr7Y4AnDqQTkyXCCI6GjF46Bl+7yD2fpPqUBMc3tTk4vp3RqfTUV5ka1gzF0wk81QOq+dvbJ08KWeI6RZFZKdwDB4Ght8/lD3r9zvmiQhqypPYtTFPBa2tLc+SCGAxkCalfKuFsvXANCHEF8AgoExKmdtCrSZW2cDrx7/mg0G/Qyd0rM08yNnKAu7rkAjAPzNSSK80savgR/5545NIKVmdeYAzFbY34BvHk3i977146PRkmUt4/rCrPqed3iB4+PnOvD3pOA1WuOGecGK6+bD9c9vTHP6g7ZPy4OYiEoYG4eXjePbhw5mnOJVcRmVJPc/cmML46R0Ydl/EFecxGATvvRbO7Q9mY7XCYw8EkBDnxcJlpQBM+V0QAGs2VHHbTT74+jR9puSbrNzzuK2f19fDg3f5M3rEpbf4LscqG3jr1Dre6jsJndCRlJNCelU+42OuB2Bd9l4umAvYV3SapYP+iETyVXYy6VX5AHjpPEgM6cbf0txbTxdrsDbw/qxPeW3tbHR6HZs+3cGFtGzGTLwZgKTF2xh2VyJjJ43AWm+lprqW1x9dAEDC4G7c+tBQzh3LZMHulwFY8uIqUjYdcSvP/OmLef3bP6PT69i4ZBsXTmQx9g+3AfD1h5u58d7rGTtlJNZ6K7XVtbz6YNPZm7krZtBreAKBRn8+y1jI8he/5NtPtl5RFtF8X6i1CCFuAHYCR7GdVgWYC3QAkFIubGwq84HR2E6rPial3O9idnY+3aJlt7cntknmKzGj+7arHcHBowEFly/6Bd14bMLVjuDEd0L+1Y7goMFsvtoRnHwnVx2QUjqdimmzLQwp5Q+4PkZxcY0EnmyrDIqitC71p+GKomimGoaiKJqphqEoimaqYSiKoplqGIqiaKYahqIomqmGoSiKZqphKIqimWoYiqJophqGoiiaqYahKIpmqmEoiqJZm31bta0IIUzAhVaYlREobIX5tKZfWyaV59J+bXmg9TJ1lFI6Xc3p365htBYhxH5XX9+9mn5tmVSeS/u15YG2z6R2SRRF0Uw1DEVRNPtvbhiLrnYAF35tmVSeS/u15YE2zvRfewxDUZSf7795C0NRlJ9JNQxFUTT7j2kYQojRQohTQogzQog/uRgvhBDvNY4/IoTod7lphRD3CSGOCyEahBBXfKrKzWyfCCEKhBDu32X4yrL1EELsEULUCCGeaYsMzZZ3yed7qdeqLTMIIUKEEJuFED82/u/yhqmXez3bKoMQ4rnGZZ4SQoxqYZ6ansMlSSn/7f8BeuAs0BnwBA4D8c1q7gA2YLuS+fXAvstNC/QE4oDtwIBfOlvjuBuBfsCxq/S6hQOJwKvAM7/Aurzk873Ua9WWGYA3gT81/vwn4K9X8nq2RQYgvnFZXsA1jRn0LuZ52edwuX//KVsYA4EzUspzUspa4AtsN3q+WEs3fm5xWillmpTy1FXMhpRyB9Dy7d7bOJuUskBKmQLUtVEGBxqeb6vfwFtjhvHAssaflwETXEyqZV23RYbxwBdSyhopZTpwpjFLc1qewyX9pzSMlm7qrKVGy7RXK1tbu1rLdcfVyhwhG+/K1/h/+FXI1lIGrcvV8hwu6T+lYWi5qXNLNZpuCO0Gd7K1tau1XHf8mjP/x6/H/5SGoeWmzi3VtPoNoVsxW1u7Wst1x9XKnP/Trk/j/67uSdnW2VrKoHW5Wp7DJf2nNIwUoJsQ4hohhCfwALYbPV9sPfBI41H262m68bOWaa9WtrbW1s+9LVyt12o98LvGn38HrHNR09avZ0sZ1gMPCCG8hBDXAN2A5J8xvXatfYT5av3DdvT8NLYjxH9ufGwKMKXxZwG83zj+KBed9XA1bePjd2Hr3jVAPrDxKmT7HMjFdtAxC5j4C79ukY3LLQdKG38OaMP16PR8tb5WbZwhFNgC/Nj4f0hjbTTwzeXeS22ZobH+z43LPAXcftHjH//0Gl1qeq3/1J+GK4qi2X/KLomiKL8A1TAURdFMNQxFUTRTDUNRFM1Uw1AURTPVMBRF0Uw1DEVRNDNc7QDKr5cQYh7wMLYvNhUCB4AyYDK2r2+fAX4rpTQLIZYC1UAPoCPwGLa/JhyM7SvojzbOsxLbH17dCpQAc7F97boD8LSUcr0QohPwKeDbGGWalHJ3Gz9dRQO1haG41HjBoHuAvsDdwE8XEFotpUyUUvYG0rD9BeJPgoERwB+Br4C3gQTgOiFEn8YaX2C7lLI/UAH8BbgN21/VvtxYUwDcJqXsB9wPvNcWz1H5+dQWhtKSG4B1UspqACHEV42PXyuE+AsQBPgBGy+a5isppRRCHAXypZRHG6c9DnQCDgG1wLeN9UeBGillXeM0nRof9wDmNzYZK9C9DZ6fcgVUw1Ba4uor0wBLgQlSysNCiEeB4ReNq2n8v+Gin38a/um9Viebvo9gr5NSNgghfqr5I7bv7vTGthVsueJnobQqtUuitOQH4E4hhLcQwg8Y0/i4P5ArhPDAdnyjLQQCuVLKBuC32C59p/wKqC0MxSUpZYoQYj22a0VeAPZjO+A5D9jX+NhRbA2ktS0A/iWEuA/YBlS1wTKUK6C+raq0SAjhJ6WsFEL4ADuAyVLKg1c7l3L1qC0M5VIWCSHiAW9gmWoWitrCUBRFM3XQU1EUzVTDUBRFM9UwFEXRTDUMRVE0Uw1DURTN/j+tiVb5l6KLlwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "param_grid4 = {\"svc__C\": np.linspace(2, 3, 6), \"svc__gamma\": np.logspace(-3, 2, 6)}\n",
    "\n",
    "display_heatmap(param_grid4, pipe_svm, X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "That's good! We are finding some more options for `C` where the accuracy is 0.75.\n",
    "\n",
    "The tricky part is we do not know in advance what range of hyperparameters might work the best for the given problem, model, and the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### True/False \n",
    "\n",
    "- If you get optimal results at the edges of your parameter grid, it might be a good idea to adjust the range of values in your parameter grid.   \n",
    "- Grid search is guaranteed to find best hyperparameters values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "***Note***\n",
    "`GridSearchCV` allows the `param_grid` to be a list of dictionaries. Sometimes some hyperparameters are applicable only for certain models. \n",
    "For example, in the context of `SVC`, `C` and `gamma` are applicable when the kernel is `rbf` whereas only `C` is applicable for `kernel=\"linear\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Problems with exhaustive grid search \n",
    "\n",
    "- Required number of models to evaluate grows **exponentially with the dimensionality** of the configuration space. \n",
    "- Example: Suppose you have\n",
    "    - 5 hyperparameters \n",
    "    - 10 different values for each hyperparameter\n",
    "    - You'll be evaluating $10^5=100,000$ models! That is you'll be calling `cross_validate` 100,000 times!\n",
    "- Exhaustive search may become **infeasible fairly quickly**.\n",
    "- Other options?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Randomized hyperparameter search\n",
    "\n",
    "- Randomized hyperparameter optimization \n",
    "    - [`sklearn.model_selection.RandomizedSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html)\n",
    "- Samples configurations at **random until certain budget** (e.g., time) is exhausted "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Grid size: 36\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'svc__gamma': [0.001, 0.01, 0.1, 1.0, 10, 100],\n",
       " 'svc__C': [0.001, 0.01, 0.1, 1.0, 10, 100]}"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "\n",
    "param_grid = {\n",
    "    \"svc__gamma\": [0.001, 0.01, 0.1, 1.0, 10, 100],\n",
    "    \"svc__C\": [0.001, 0.01, 0.1, 1.0, 10, 100],\n",
    "}\n",
    "\n",
    "print(\"Grid size:\", np.prod([len(v) for v in param_grid.values()]))\n",
    "param_grid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_search = RandomizedSearchCV(\n",
    "    pipe_svm, param_distributions=param_grid, n_jobs=-1, n_iter=10, cv=5, random_state=123\n",
    ")\n",
    "random_search.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>rank_test_score</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>6</th>\n",
       "      <th>6</th>\n",
       "      <th>6</th>\n",
       "      <th>6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean_test_score</th>\n",
       "      <td>0.732792</td>\n",
       "      <td>0.711715</td>\n",
       "      <td>0.678851</td>\n",
       "      <td>0.652824</td>\n",
       "      <td>0.508371</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "      <td>0.50775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_svc__gamma</th>\n",
       "      <td>0.01</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.001</td>\n",
       "      <td>100</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.1</td>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_svc__C</th>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean_fit_time</th>\n",
       "      <td>0.361828</td>\n",
       "      <td>0.765091</td>\n",
       "      <td>0.348906</td>\n",
       "      <td>0.205961</td>\n",
       "      <td>0.425573</td>\n",
       "      <td>0.331889</td>\n",
       "      <td>0.36394</td>\n",
       "      <td>0.35472</td>\n",
       "      <td>0.58357</td>\n",
       "      <td>0.239231</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "rank_test_score          1         2         3         4         5         6  \\\n",
       "mean_test_score   0.732792  0.711715  0.678851  0.652824  0.508371   0.50775   \n",
       "param_svc__gamma      0.01       0.1      0.01     0.001       100     0.001   \n",
       "param_svc__C           100       100       0.1       1.0       1.0      0.01   \n",
       "mean_fit_time     0.361828  0.765091  0.348906  0.205961  0.425573  0.331889   \n",
       "\n",
       "rank_test_score         6        6        6         6  \n",
       "mean_test_score   0.50775  0.50775  0.50775   0.50775  \n",
       "param_svc__gamma      0.1      100      100     0.001  \n",
       "param_svc__C         0.01     0.01    0.001       0.1  \n",
       "mean_fit_time     0.36394  0.35472  0.58357  0.239231  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(random_search.cv_results_)[relevant].set_index(\"rank_test_score\").sort_index().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `n_iter`\n",
    "\n",
    "- Note the `n_iter`, we didn't need this for `GridSearchCV`.\n",
    "- Larger `n_iter` will take longer but it'll do more searching.\n",
    "  - Remember you still need to multiply by number of folds! \n",
    "  - Thus, number of models to train will be `n_iter * cv`\n",
    "- I have also set `random_state` but you don't have to do it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Range of `C`\n",
    "\n",
    "- Note the exponential range for `C`. This is quite common.\n",
    "- There is no point trying $C=\\{1,2,3\\ldots,100\\}$ because $C=1,2,3$ are too similar to each other.\n",
    "- Often we're trying to find an order of magnitude, e.g. $C=\\{0.01,0.1,1,10,100\\}$. \n",
    "- We can also write that as $C=\\{10^{-2},10^{-1},10^0,10^1,10^2\\}$. \n",
    "- Or, in other words, $C$ values to try are $10^n$ for $n=-2,-1,0,1,2$ which is basically what we have above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "(Optional) Another thing we can do is give probability distributions to draw from:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import expon, lognorm, loguniform, randint, uniform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_dist = {\n",
    "    \"svc__C\": uniform(0.1, 1e4),  # loguniform(1e-3, 1e3),\n",
    "    \"svc__gamma\": loguniform(1e-5, 1e3),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "random_search = RandomizedSearchCV(\n",
    "    pipe_svm, param_dist, n_iter=100, verbose=1, n_jobs=-1, random_state=123\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 100 candidates, totalling 500 fits\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 {color: black;background-color: white;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 pre{padding: 0;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-toggleable {background-color: white;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-estimator:hover {background-color: #d4ebff;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-item {z-index: 1;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-parallel::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-parallel-item:only-child::after {width: 0;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;position: relative;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-240d3b4f-6061-4b44-8451-5b1b8cb9a149\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>RandomizedSearchCV(estimator=Pipeline(steps=[(&#x27;standardscaler&#x27;,\n",
       "                                              StandardScaler()),\n",
       "                                             (&#x27;svc&#x27;, SVC())]),\n",
       "                   n_iter=100, n_jobs=-1,\n",
       "                   param_distributions={&#x27;svc__C&#x27;: &lt;scipy.stats._distn_infrastructure.rv_frozen object at 0x7f51b0072ef0&gt;,\n",
       "                                        &#x27;svc__gamma&#x27;: &lt;scipy.stats._distn_infrastructure.rv_frozen object at 0x7f51affe9ae0&gt;},\n",
       "                   random_state=123, verbose=1)</pre><b>Please rerun this cell to show the HTML repr or trust the notebook.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"f6e7d808-1de3-4303-99b1-c30f6ae9fa45\" type=\"checkbox\" ><label for=\"f6e7d808-1de3-4303-99b1-c30f6ae9fa45\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomizedSearchCV</label><div class=\"sk-toggleable__content\"><pre>RandomizedSearchCV(estimator=Pipeline(steps=[(&#x27;standardscaler&#x27;,\n",
       "                                              StandardScaler()),\n",
       "                                             (&#x27;svc&#x27;, SVC())]),\n",
       "                   n_iter=100, n_jobs=-1,\n",
       "                   param_distributions={&#x27;svc__C&#x27;: &lt;scipy.stats._distn_infrastructure.rv_frozen object at 0x7f51b0072ef0&gt;,\n",
       "                                        &#x27;svc__gamma&#x27;: &lt;scipy.stats._distn_infrastructure.rv_frozen object at 0x7f51affe9ae0&gt;},\n",
       "                   random_state=123, verbose=1)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"c049dee8-7530-4b52-8c80-04745d15532a\" type=\"checkbox\" ><label for=\"c049dee8-7530-4b52-8c80-04745d15532a\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"15d0088c-3b58-4313-a28b-6b5d68a47d68\" type=\"checkbox\" ><label for=\"15d0088c-3b58-4313-a28b-6b5d68a47d68\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC()</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "RandomizedSearchCV(estimator=Pipeline(steps=[('standardscaler',\n",
       "                                              StandardScaler()),\n",
       "                                             ('svc', SVC())]),\n",
       "                   n_iter=100, n_jobs=-1,\n",
       "                   param_distributions={'svc__C': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f51b0072ef0>,\n",
       "                                        'svc__gamma': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f51affe9ae0>},\n",
       "                   random_state=123, verbose=1)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7383804780493433"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>rank_test_score</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>82</th>\n",
       "      <th>82</th>\n",
       "      <th>82</th>\n",
       "      <th>82</th>\n",
       "      <th>82</th>\n",
       "      <th>96</th>\n",
       "      <th>96</th>\n",
       "      <th>96</th>\n",
       "      <th>96</th>\n",
       "      <th>96</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean_test_score</th>\n",
       "      <td>0.73838</td>\n",
       "      <td>0.7359</td>\n",
       "      <td>0.735277</td>\n",
       "      <td>0.733415</td>\n",
       "      <td>0.731556</td>\n",
       "      <td>0.729716</td>\n",
       "      <td>0.729068</td>\n",
       "      <td>0.728454</td>\n",
       "      <td>0.72845</td>\n",
       "      <td>0.72845</td>\n",
       "      <td>...</td>\n",
       "      <td>0.508371</td>\n",
       "      <td>0.508371</td>\n",
       "      <td>0.508371</td>\n",
       "      <td>0.508371</td>\n",
       "      <td>0.508371</td>\n",
       "      <td>0.50713</td>\n",
       "      <td>0.50713</td>\n",
       "      <td>0.50713</td>\n",
       "      <td>0.50713</td>\n",
       "      <td>0.50713</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_svc__gamma</th>\n",
       "      <td>0.00271</td>\n",
       "      <td>0.001946</td>\n",
       "      <td>0.00283</td>\n",
       "      <td>0.003148</td>\n",
       "      <td>0.003834</td>\n",
       "      <td>0.015524</td>\n",
       "      <td>0.00067</td>\n",
       "      <td>0.005593</td>\n",
       "      <td>0.00089</td>\n",
       "      <td>0.000458</td>\n",
       "      <td>...</td>\n",
       "      <td>33.017913</td>\n",
       "      <td>51.754341</td>\n",
       "      <td>126.722678</td>\n",
       "      <td>84.511472</td>\n",
       "      <td>120.952433</td>\n",
       "      <td>766.440217</td>\n",
       "      <td>739.685456</td>\n",
       "      <td>918.053216</td>\n",
       "      <td>263.398099</td>\n",
       "      <td>806.79529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_svc__C</th>\n",
       "      <td>3427.738338</td>\n",
       "      <td>6964.791856</td>\n",
       "      <td>2865.466167</td>\n",
       "      <td>4258.402903</td>\n",
       "      <td>7224.533826</td>\n",
       "      <td>1511.374523</td>\n",
       "      <td>3617.986556</td>\n",
       "      <td>2408.658977</td>\n",
       "      <td>7636.928414</td>\n",
       "      <td>9053.515757</td>\n",
       "      <td>...</td>\n",
       "      <td>5826.810879</td>\n",
       "      <td>7087.073954</td>\n",
       "      <td>6648.824488</td>\n",
       "      <td>6007.085678</td>\n",
       "      <td>957.225166</td>\n",
       "      <td>4830.442643</td>\n",
       "      <td>9019.213727</td>\n",
       "      <td>7049.688305</td>\n",
       "      <td>3370.763834</td>\n",
       "      <td>26.980646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean_fit_time</th>\n",
       "      <td>1.800691</td>\n",
       "      <td>2.129358</td>\n",
       "      <td>1.229454</td>\n",
       "      <td>2.363242</td>\n",
       "      <td>4.471639</td>\n",
       "      <td>3.34881</td>\n",
       "      <td>0.882294</td>\n",
       "      <td>2.175671</td>\n",
       "      <td>1.967938</td>\n",
       "      <td>0.846148</td>\n",
       "      <td>...</td>\n",
       "      <td>0.356378</td>\n",
       "      <td>0.358163</td>\n",
       "      <td>0.354666</td>\n",
       "      <td>0.345583</td>\n",
       "      <td>0.454863</td>\n",
       "      <td>0.313766</td>\n",
       "      <td>0.286004</td>\n",
       "      <td>0.282581</td>\n",
       "      <td>0.290457</td>\n",
       "      <td>0.2878</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 100 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "rank_test_score            1            2            3            4   \\\n",
       "mean_test_score       0.73838       0.7359     0.735277     0.733415   \n",
       "param_svc__gamma      0.00271     0.001946      0.00283     0.003148   \n",
       "param_svc__C      3427.738338  6964.791856  2865.466167  4258.402903   \n",
       "mean_fit_time        1.800691     2.129358     1.229454     2.363242   \n",
       "\n",
       "rank_test_score            5            6            7            8   \\\n",
       "mean_test_score      0.731556     0.729716     0.729068     0.728454   \n",
       "param_svc__gamma     0.003834     0.015524      0.00067     0.005593   \n",
       "param_svc__C      7224.533826  1511.374523  3617.986556  2408.658977   \n",
       "mean_fit_time        4.471639      3.34881     0.882294     2.175671   \n",
       "\n",
       "rank_test_score            9            9   ...           82           82  \\\n",
       "mean_test_score       0.72845      0.72845  ...     0.508371     0.508371   \n",
       "param_svc__gamma      0.00089     0.000458  ...    33.017913    51.754341   \n",
       "param_svc__C      7636.928414  9053.515757  ...  5826.810879  7087.073954   \n",
       "mean_fit_time        1.967938     0.846148  ...     0.356378     0.358163   \n",
       "\n",
       "rank_test_score            82           82          82           96  \\\n",
       "mean_test_score      0.508371     0.508371    0.508371      0.50713   \n",
       "param_svc__gamma   126.722678    84.511472  120.952433   766.440217   \n",
       "param_svc__C      6648.824488  6007.085678  957.225166  4830.442643   \n",
       "mean_fit_time        0.354666     0.345583    0.454863     0.313766   \n",
       "\n",
       "rank_test_score            96           96           96         96  \n",
       "mean_test_score       0.50713      0.50713      0.50713    0.50713  \n",
       "param_svc__gamma   739.685456   918.053216   263.398099  806.79529  \n",
       "param_svc__C      9019.213727  7049.688305  3370.763834  26.980646  \n",
       "mean_fit_time        0.286004     0.282581     0.290457     0.2878  \n",
       "\n",
       "[4 rows x 100 columns]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(random_search.cv_results_)[relevant].set_index(\"rank_test_score\").sort_index().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- This is a bit fancy. What's nice is that you can have it concentrate more on certain values by setting the distribution. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "### Advantages of `RandomizedSearchCV`\n",
    "\n",
    "- Faster compared to `GridSearchCV`.\n",
    "- Adding parameters that do not influence the performance does not affect efficiency.\n",
    "- Works better when some parameters are more important than others. \n",
    "- In general, I recommend using `RandomizedSearchCV` rather than `GridSearchCV`.    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Advantages of `RandomizedSearchCV`\n",
    "\n",
    "![](img/randomsearch_bergstra.png)\n",
    "    \n",
    "Source: [Bergstra and Bengio, Random Search for Hyper-Parameter Optimization, JMLR 2012](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- The yellow on the left shows how your scores are going to change when you vary the unimportant hyperparameter.\n",
    "- The green on the top shows how your scores are  going to change when you vary the important hyperparameter.\n",
    "- You don't know in advance which hyperparameters are important for your problem.\n",
    "- In the left figure, 6 of the 9 searches are useless because they are only varying the unimportant parameter.\n",
    "- In the right figure, all 9 searches are useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Fancier methods (optional)\n",
    "- Both `GridSearchCV` and `RandomizedSearchCV` do each trial independently.\n",
    "- What if you could learn from your experience, e.g. learn that `max_depth=3` is bad?\n",
    "  - That could save time because you wouldn't try combinations involving `max_depth=3` in the future.\n",
    "- We can do this with `scikit-optimize`, which is a completely different package from `scikit-learn`\n",
    "- It uses a technique called \"model-based optimization\" and we'll specifically use \"Bayesian optimization\".\n",
    "  - In short, it uses machine learning to predict what hyperparameters will be good.\n",
    "  - Machine learning on machine learning!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- This is an active research area and there are sophisticated packages for this.\n",
    "\n",
    "Here are some examples \n",
    "- [hyperopt-sklearn](https://github.com/hyperopt/hyperopt-sklearn)\n",
    "- [auto-sklearn](https://github.com/automl/auto-sklearn)\n",
    "- [SigOptSearchCV](https://sigopt.com/docs/overview/scikit_learn)\n",
    "- [TPOT](https://github.com/rhiever/tpot)\n",
    "- [hyperopt](https://github.com/hyperopt/hyperopt)\n",
    "- [hyperband](https://github.com/zygmuntz/hyperband)\n",
    "- [SMAC](http://www.cs.ubc.ca/labs/beta/Projects/SMAC/)\n",
    "- [MOE](https://github.com/Yelp/MOE)\n",
    "- [pybo](https://github.com/mwhoffman/pybo)\n",
    "- [spearmint](https://github.com/HIPS/Spearmint)\n",
    "- [BayesOpt](https://github.com/rmcantin/bayesopt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Questions for class discussion (hyperparameter optimization)\n",
    "\n",
    "- Suppose you have 10 hyperparameters, each with 4 possible values. If you run `GridSearchCV` with this parameter grid, how many cross-validation experiments it would carry out? \n",
    "- `GridSearchCV` exhaustively searches the grid and so it's guaranteed to give you the optimal hyperparameters for the given problem. \n",
    "- It is possible to get different hyperparameters in different runs of `RandomizedSearchCV`.\n",
    "- Suppose you have 10 hyperparameters and each takes 4 values. If you run `RandomizedSearchCV` with this parameter grid, how many cross-validation experiments it would carry out? \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Optimization bias/Overfitting of the validation set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Overfitting of the validation error \n",
    "\n",
    "- Why do we need to evaluate the model on the test set in the end?\n",
    "- Why not just use cross-validation on the whole dataset? \n",
    "- While carrying out hyperparameter optimization, we usually try over many possibilities.  \n",
    "- If our dataset is small and if your validation set is hit too many times, we suffer from **optimization bias** or **overfitting the validation set**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Optimization bias of parameter learning\n",
    "- Overfitting of the training error\n",
    "- An example: \n",
    "    - During training, we could search over tons of different decision trees.\n",
    "    - So we can get \"lucky\" and find one with low training error by chance.\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Optimization bias of hyper-parameter learning\n",
    "\n",
    "- Overfitting of the validation error\n",
    "- An example: \n",
    "    - Here, we might optimize the validation error over 1000 values of `max_depth`.\n",
    "    - One of the 1000 trees might have low validation error by chance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Example 1: Optimization bias (optional)\n",
    "\n",
    "Consider a multiple-choice (a,b,c,d) \"test\" with **10 questions**:\n",
    "- If you choose answers randomly, expected grade is 25% (no bias).\n",
    "- If you fill out two tests randomly and pick the best, expected grade is 33%.\n",
    "    - Optimization bias of ~8%.\n",
    "- If you take the best among 10 random tests, expected grade is ~47%.\n",
    "- If you take the best among 100, expected grade is ~62%.\n",
    "- If you take the best among 1000, expected grade is ~73%.\n",
    "- If you take the best among 10000, expected grade is ~82%.\n",
    "    - You have so many \"chances\" that you expect to do well.\n",
    "    \n",
    "**But on new questions the \"random choice\" accuracy is still 25%.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The expected grade among the best of 1 tests is : 0.25\n",
      "The expected grade among the best of 2 tests is : 0.33\n",
      "The expected grade among the best of 10 tests is : 0.47\n",
      "The expected grade among the best of 100 tests is : 0.62\n",
      "The expected grade among the best of 1000 tests is : 0.73\n",
      "The expected grade among the best of 10000 tests is : 0.83\n"
     ]
    }
   ],
   "source": [
    "# (optional) Code attribution: Rodolfo Lourenzutti\n",
    "number_tests = [1, 2, 10, 100, 1000, 10000]\n",
    "for ntests in number_tests:\n",
    "    y = np.zeros(10000)\n",
    "    for i in range(10000):\n",
    "        y[i] = np.max(np.random.binomial(10.0, 0.25, ntests))\n",
    "    print(\n",
    "        \"The expected grade among the best of %d tests is : %0.2f\"\n",
    "        % (ntests, np.mean(y) / 10.0)\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Example 2: Optimization bias (optional)\n",
    "\n",
    "- If we instead used a **100-question** test then:\n",
    "    - Expected grade from best over 1 randomly-filled test is 25%.\n",
    "    - Expected grade from best over 2 randomly-filled test is ~27%.\n",
    "    - Expected grade from best over 10 randomly-filled test is ~32%.\n",
    "    - Expected grade from best over 100 randomly-filled test is ~36%.\n",
    "    - Expected grade from best over 1000 randomly-filled test is ~40%.\n",
    "    - Expected grade from best over 10000 randomly-filled test is ~43%.\n",
    "\n",
    "- The optimization bias **grows with the number of things we try**.\n",
    "    - “Complexity” of the set of models we search over.\n",
    "- But, optimization bias **shrinks quickly with the number of examples**.\n",
    "    - But it’s still non-zero and growing if you over-use your validation set!    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The expected grade among the best of 1 tests is : 0.25\n",
      "The expected grade among the best of 2 tests is : 0.28\n",
      "The expected grade among the best of 10 tests is : 0.32\n",
      "The expected grade among the best of 100 tests is : 0.36\n",
      "The expected grade among the best of 1000 tests is : 0.40\n",
      "The expected grade among the best of 10000 tests is : 0.43\n"
     ]
    }
   ],
   "source": [
    "# (optional) Code attribution: Rodolfo Lourenzutti\n",
    "number_tests = [1, 2, 10, 100, 1000, 10000]\n",
    "for ntests in number_tests:\n",
    "    y = np.zeros(10000)\n",
    "    for i in range(10000):\n",
    "        y[i] = np.max(np.random.binomial(100.0, 0.25, ntests))\n",
    "    print(\n",
    "        \"The expected grade among the best of %d tests is : %0.2f\"\n",
    "        % (ntests, np.mean(y) / 100.0)\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Optimization bias on the Spotify dataset\n",
    "\n",
    "Notice the unrealistic setting `test_size=0.99`. We do that here for demonstration only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_tiny, X_test_big, y_train_tiny, y_test_big = train_test_split(\n",
    "    X_spotify, y_spotify, test_size=0.99, random_state=42\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20, 13)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_tiny.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>acousticness</th>\n",
       "      <th>danceability</th>\n",
       "      <th>duration_ms</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>key</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>mode</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>time_signature</th>\n",
       "      <th>valence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>0.055100</td>\n",
       "      <td>0.547</td>\n",
       "      <td>251093</td>\n",
       "      <td>0.643</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0.2670</td>\n",
       "      <td>-8.904</td>\n",
       "      <td>1</td>\n",
       "      <td>0.2270</td>\n",
       "      <td>143.064</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.1870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1687</th>\n",
       "      <td>0.000353</td>\n",
       "      <td>0.420</td>\n",
       "      <td>210240</td>\n",
       "      <td>0.929</td>\n",
       "      <td>0.000747</td>\n",
       "      <td>7</td>\n",
       "      <td>0.1220</td>\n",
       "      <td>-3.899</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1210</td>\n",
       "      <td>127.204</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.3180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>871</th>\n",
       "      <td>0.314000</td>\n",
       "      <td>0.430</td>\n",
       "      <td>193427</td>\n",
       "      <td>0.734</td>\n",
       "      <td>0.000286</td>\n",
       "      <td>9</td>\n",
       "      <td>0.0808</td>\n",
       "      <td>-10.043</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1020</td>\n",
       "      <td>133.992</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0537</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1123</th>\n",
       "      <td>0.082100</td>\n",
       "      <td>0.725</td>\n",
       "      <td>246653</td>\n",
       "      <td>0.711</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>10</td>\n",
       "      <td>0.0931</td>\n",
       "      <td>-4.544</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0335</td>\n",
       "      <td>93.003</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.4760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1396</th>\n",
       "      <td>0.286000</td>\n",
       "      <td>0.616</td>\n",
       "      <td>236960</td>\n",
       "      <td>0.387</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>9</td>\n",
       "      <td>0.2770</td>\n",
       "      <td>-6.079</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0335</td>\n",
       "      <td>81.856</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.4700</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      acousticness  danceability  duration_ms  energy  instrumentalness  key  \\\n",
       "130       0.055100         0.547       251093   0.643          0.000000    1   \n",
       "1687      0.000353         0.420       210240   0.929          0.000747    7   \n",
       "871       0.314000         0.430       193427   0.734          0.000286    9   \n",
       "1123      0.082100         0.725       246653   0.711          0.000000   10   \n",
       "1396      0.286000         0.616       236960   0.387          0.000000    9   \n",
       "\n",
       "      liveness  loudness  mode  speechiness    tempo  time_signature  valence  \n",
       "130     0.2670    -8.904     1       0.2270  143.064             4.0   0.1870  \n",
       "1687    0.1220    -3.899     0       0.1210  127.204             4.0   0.3180  \n",
       "871     0.0808   -10.043     0       0.1020  133.992             4.0   0.0537  \n",
       "1123    0.0931    -4.544     1       0.0335   93.003             4.0   0.4760  \n",
       "1396    0.2770    -6.079     0       0.0335   81.856             4.0   0.4700  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_tiny.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = make_pipeline(StandardScaler(), SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Grid size: 900\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'svc__gamma': array([1.e-20, 1.e-19, 1.e-18, 1.e-17, 1.e-16, 1.e-15, 1.e-14, 1.e-13,\n",
       "        1.e-12, 1.e-11, 1.e-10, 1.e-09, 1.e-08, 1.e-07, 1.e-06, 1.e-05,\n",
       "        1.e-04, 1.e-03, 1.e-02, 1.e-01, 1.e+00, 1.e+01, 1.e+02, 1.e+03,\n",
       "        1.e+04, 1.e+05, 1.e+06, 1.e+07, 1.e+08, 1.e+09]),\n",
       " 'svc__C': array([1.e-20, 1.e-19, 1.e-18, 1.e-17, 1.e-16, 1.e-15, 1.e-14, 1.e-13,\n",
       "        1.e-12, 1.e-11, 1.e-10, 1.e-09, 1.e-08, 1.e-07, 1.e-06, 1.e-05,\n",
       "        1.e-04, 1.e-03, 1.e-02, 1.e-01, 1.e+00, 1.e+01, 1.e+02, 1.e+03,\n",
       "        1.e+04, 1.e+05, 1.e+06, 1.e+07, 1.e+08, 1.e+09])}"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "param_grid = {\n",
    "    \"svc__gamma\": 10.0 ** np.arange(-20, 10),\n",
    "    \"svc__C\": 10.0 ** np.arange(-20, 10),\n",
    "}\n",
    "\n",
    "print(\"Grid size:\", np.prod([len(v) for v in param_grid.values()]))\n",
    "param_grid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_search = RandomizedSearchCV(\n",
    "    pipe, param_distributions=param_grid, n_jobs=-1, n_iter=900, cv=5, random_state=123\n",
    ")\n",
    "random_search.fit(X_train_tiny, y_train_tiny);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>rank_test_score</th>\n",
       "      <th>1</th>\n",
       "      <th>1</th>\n",
       "      <th>3</th>\n",
       "      <th>3</th>\n",
       "      <th>3</th>\n",
       "      <th>3</th>\n",
       "      <th>3</th>\n",
       "      <th>3</th>\n",
       "      <th>3</th>\n",
       "      <th>3</th>\n",
       "      <th>...</th>\n",
       "      <th>20</th>\n",
       "      <th>20</th>\n",
       "      <th>20</th>\n",
       "      <th>20</th>\n",
       "      <th>20</th>\n",
       "      <th>20</th>\n",
       "      <th>20</th>\n",
       "      <th>20</th>\n",
       "      <th>20</th>\n",
       "      <th>900</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean_test_score</th>\n",
       "      <td>0.8</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.75</td>\n",
       "      <td>...</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_svc__gamma</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100000.0</td>\n",
       "      <td>1000000000.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param_svc__C</th>\n",
       "      <td>1000000000.0</td>\n",
       "      <td>100000000.0</td>\n",
       "      <td>1000000000.0</td>\n",
       "      <td>10000.0</td>\n",
       "      <td>1000000.0</td>\n",
       "      <td>10000000.0</td>\n",
       "      <td>100000.0</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>10000000.0</td>\n",
       "      <td>100000000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1000000000.0</td>\n",
       "      <td>1000000000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean_fit_time</th>\n",
       "      <td>0.01006</td>\n",
       "      <td>0.009334</td>\n",
       "      <td>0.010692</td>\n",
       "      <td>0.018828</td>\n",
       "      <td>0.011188</td>\n",
       "      <td>0.00936</td>\n",
       "      <td>0.009404</td>\n",
       "      <td>0.012803</td>\n",
       "      <td>0.010117</td>\n",
       "      <td>0.009571</td>\n",
       "      <td>...</td>\n",
       "      <td>0.010388</td>\n",
       "      <td>0.010264</td>\n",
       "      <td>0.009756</td>\n",
       "      <td>0.010275</td>\n",
       "      <td>0.012348</td>\n",
       "      <td>0.009535</td>\n",
       "      <td>0.010359</td>\n",
       "      <td>0.013648</td>\n",
       "      <td>0.00904</td>\n",
       "      <td>0.010052</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 900 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "rank_test_score            1            1             3         3    \\\n",
       "mean_test_score            0.8          0.8          0.75      0.75   \n",
       "param_svc__gamma           0.0          0.0         0.001     0.001   \n",
       "param_svc__C      1000000000.0  100000000.0  1000000000.0   10000.0   \n",
       "mean_fit_time          0.01006     0.009334      0.010692  0.018828   \n",
       "\n",
       "rank_test_score         3           3         3         3           3    \\\n",
       "mean_test_score        0.75        0.75      0.75      0.75        0.75   \n",
       "param_svc__gamma      0.001         0.0     0.001     0.001       0.001   \n",
       "param_svc__C      1000000.0  10000000.0  100000.0    1000.0  10000000.0   \n",
       "mean_fit_time      0.011188     0.00936  0.009404  0.012803    0.010117   \n",
       "\n",
       "rank_test_score           3    ...       20        20        20        20   \\\n",
       "mean_test_score          0.75  ...      0.65      0.65      0.65      0.65   \n",
       "param_svc__gamma        0.001  ...       0.0       0.0       0.0       0.0   \n",
       "param_svc__C      100000000.0  ...       0.0       0.0       0.0       0.0   \n",
       "mean_fit_time        0.009571  ...  0.010388  0.010264  0.009756  0.010275   \n",
       "\n",
       "rank_test_score        20        20        20        20            20   \\\n",
       "mean_test_score       0.65      0.65      0.65      0.65          0.65   \n",
       "param_svc__gamma       0.0       0.0       0.0  100000.0  1000000000.0   \n",
       "param_svc__C           0.0       0.0       0.0       0.0  1000000000.0   \n",
       "mean_fit_time     0.012348  0.009535  0.010359  0.013648       0.00904   \n",
       "\n",
       "rank_test_score            900  \n",
       "mean_test_score           0.55  \n",
       "param_svc__gamma           0.0  \n",
       "param_svc__C      1000000000.0  \n",
       "mean_fit_time         0.010052  \n",
       "\n",
       "[4 rows x 900 columns]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(random_search.cv_results_)[relevant].set_index(\"rank_test_score\").sort_index().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the results: one might claim that we found a model that performs with 0.8 accuracy on our dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Do we really **believe that 0.80** is a good estimate of our test data?\n",
    "- Do we really believe that `gamma`=0.0 and C=1_000_000_000 are the best hyperparameters? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let's find out the **test score** with this best model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6163366336633663"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- The results above are **overly optimistic**. \n",
    "    - because our training data is very small and so our validation splits in cross validation would be small. \n",
    "    - because of the small dataset and the fact that we **hit the small validation set 900 times** and it's possible that we got lucky on the validation set! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- As we suspected, the best cross-validation score is not a good estimate of our test data; it is overly optimistic. \n",
    "- We can trust this test score because the test set is of good size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1997, 13)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test_big.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Overfitting of the validation data\n",
    "\n",
    "The following plot demonstrates what happens during overfitting of the validation data.  \n",
    "<center>\n",
    "<img src='./img/optimization-bias.png' width=\"600\">\n",
    "</center>\n",
    "\n",
    "\n",
    "[Source](https://amueller.github.io/COMS4995-s20/slides/aml-03-supervised-learning/#20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Thus, not only can we not trust the cv scores, we also cannot trust cv's ability to choose the best hyperparameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Why do we need a test set? \n",
    "- This is why we need a test set.\n",
    "- The frustrating part is that if our dataset is small then our test set is also small 😔. \n",
    "- But we don't have a lot of better alternatives, unfortunately, if we have a **small dataset**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### When test score is much lower than CV score\n",
    "- What to do if your test score is much lower than your cross-validation score:\n",
    "    - Try simpler models and **use the test set a couple of times**; it's not the end of the world.\n",
    "    - **Communicate** this clearly when you report the results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Large datasets solve many of these problems\n",
    "- With infinite amounts of training data, overfitting would not be a problem and you could have your test score = your train score.\n",
    "    - Overfitting happens because you only see a bit of data and you learn patterns that are overly specific to your sample.\n",
    "    - If you saw \"all\" the data, then the notion of \"overly specific\" would not apply.\n",
    "- So, more data will make your test score better and robust. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Would you trust the model?  \n",
    "\n",
    "- You have a dataset and you give me half of it. I build a model using all the data you have given me and I tell you that the model accuracy is 0.99. Would it classify the rest of the data with similar accuracy? \n",
    "\n",
    "1. Probably \n",
    "2. Probably not "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Would you trust the model?  \n",
    "\n",
    "- You have a dataset and you give me half of it. I build a model using 80% of the data given to me and report the accuracy of 0.95 on the remaining 20% of the data. Would it classify the rest of the data with similar accuracy? \n",
    "\n",
    "1. Probably \n",
    "2. Probably not "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Would you trust the model?  \n",
    "\n",
    "- You have a dataset and you give me 1/10th of it. The dataset given to me is rather small and so I split it into 96% train and 4% validation split. I carry out hyperparameter optimization using a single 4% validation split and report validation accuracy of 0.97. Would it classify the rest of the data with similar accuracy? \n",
    "\n",
    "1. Probably \n",
    "2. Probably not "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final comments and summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Automated hyperparameter optimization\n",
    "\n",
    "- Advantages \n",
    "    - reduce human effort\n",
    "    - less prone to error and improve reproducibility\n",
    "    - data-driven approaches may be effective\n",
    "- Disadvantages\n",
    "    - may be hard to incorporate intuition\n",
    "    - be careful about overfitting on the validation set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often, especially on typical datasets, we get back `scikit-learn`'s default hyperparameter values. This means that the defaults are well chosen by `scikit-learn` developers!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The problem of finding the best values for the important hyperparameters is tricky because \n",
    "    - You may have a lot of them (e.g. deep learning). \n",
    "    - You may have multiple hyperparameters which may interact with each other in unexpected ways.    \n",
    "- The **best settings depend on the specific data/problem**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional readings and resources\n",
    "\n",
    "- [Preventing \"overfitting\" of cross-validation data](http://www.robotics.stanford.edu/~ang/papers/cv-final.pdf) by Andrew Ng"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330]",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}