main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bf9ced51",
   "metadata": {},
   "source": [
    "Recomendamos leer el archivo PDF para una versión con mejor maquetación que la presentada aquí dadas las limitaciones de los cuadernos de Jupyter. Se pueden descargar los archivos usando el siguiente repositorio: https://github.com/Rumoa/Optional_tensorflow\n",
    "\n",
    "# Abstract\n",
    "\n",
    "En este documento utilizamos dos tipos de algoritmos de Machine Learning: redes neuronales _feedforward_ y árboles de clasificación para clasificar el conjunto de datos _Chess (King-Rook vs. King) Data Set_.  Dado el desbalanceo que existe entre las clases de la columna a clasificar, también utilizamos el algoritmo _SMOTE_ para homogeneizar el conjunto.\n",
    "\n",
    "Todas las pruebas se realizan con TensorFlow con el fin de implementar técnicas vistas en clase. Se estudian distintas configuraciones de redes neuronales como el número de capas y profundidad de las mismas, y el uso de técnicas como _Dropout_ para evitar el sobreentrenamiento. Los árboles de clasificación son implementados con una nueva _API_ de Tensorflow: _TensorFlowDecisionTrees_. Comparamos el rendimiento de los árboles de clasificación de esta librería con los de _Scikit-Learn_ y las redes neuronales anteriores.\n",
    "\n",
    "# Análisis exploratorio de datos\n",
    "\n",
    "El conjunto de datos con el que trabajaremos es una recopilación de situaciones de ajedrez en las cuales el rey y la torre blanca intentan dar mate al rey negro en un determinado número de jugadas. El conjunto se puede obtener en https://archive.ics.uci.edu/ml/datasets/Chess+%28King-Rook+vs.+King%29 .\n",
    "\n",
    "El objetivo es determinar, a partir de las posiciones del rey y la torre negra, el número de movimientos necesario para dar mate al rey negro. Es posible que la configuración de piezas no dé lugar a un mate y acabe en tablas. Además, todos los datos están generados asumiendo un estilo de juego óptimo usando el estimador de teoría de juegos _Minimax_.\n",
    "\n",
    "Nuestro conjunto de datos está formado por 7 columnas, de las cuales las 6 primeras serán las variables predictoras y la última la variable a predecir:\n",
    "\n",
    "1. Columna del rey blanco (wkc)\n",
    "\n",
    "2. Fila del rey blanco (wkr)\n",
    "3. Columna de la torre blanca (wrc)\n",
    "4. Fila de la torre blanca (wrr)\n",
    "5. Columna del rey negro (bkc)\n",
    "6. Fila del rey negor (bkr)\n",
    "7. Número de movimientos óptimos para que ganen las blancas. Varían del 0 al 16 más la posibilidad de empate.\n",
    "\n",
    "Nuestro trabajo será construir modelos que sean capaces de predecir el número de movimientos óptimos para ganar a partir de las posiciones de las tres piezas.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "47b23122",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from IPython.display import set_matplotlib_formats\n",
    "set_matplotlib_formats('png', 'pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "539602b9",
   "metadata": {},
   "source": [
    "Importamos los datos usando pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "334aa626",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"krkopt.data\", header=None)\n",
    "data.columns = [\"wkc\", \"wkr\", \"wrc\", \"wrr\", \"bkc\", \"bkr\", \"opt rank\" ]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13fd4385",
   "metadata": {},
   "source": [
    "Veamos la estructura del conjunto:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "edaad76a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wkc</th>\n",
       "      <th>wkr</th>\n",
       "      <th>wrc</th>\n",
       "      <th>wrr</th>\n",
       "      <th>bkc</th>\n",
       "      <th>bkr</th>\n",
       "      <th>opt rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>b</td>\n",
       "      <td>3</td>\n",
       "      <td>c</td>\n",
       "      <td>2</td>\n",
       "      <td>draw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>2</td>\n",
       "      <td>draw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>1</td>\n",
       "      <td>d</td>\n",
       "      <td>1</td>\n",
       "      <td>draw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>1</td>\n",
       "      <td>d</td>\n",
       "      <td>2</td>\n",
       "      <td>draw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>2</td>\n",
       "      <td>c</td>\n",
       "      <td>1</td>\n",
       "      <td>draw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28051</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>g</td>\n",
       "      <td>7</td>\n",
       "      <td>e</td>\n",
       "      <td>5</td>\n",
       "      <td>sixteen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28052</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>g</td>\n",
       "      <td>7</td>\n",
       "      <td>e</td>\n",
       "      <td>6</td>\n",
       "      <td>sixteen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28053</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>g</td>\n",
       "      <td>7</td>\n",
       "      <td>e</td>\n",
       "      <td>7</td>\n",
       "      <td>sixteen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28054</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>g</td>\n",
       "      <td>7</td>\n",
       "      <td>f</td>\n",
       "      <td>5</td>\n",
       "      <td>sixteen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28055</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>g</td>\n",
       "      <td>7</td>\n",
       "      <td>g</td>\n",
       "      <td>5</td>\n",
       "      <td>sixteen</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28056 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      wkc  wkr wrc  wrr bkc  bkr opt rank\n",
       "0       a    1   b    3   c    2     draw\n",
       "1       a    1   c    1   c    2     draw\n",
       "2       a    1   c    1   d    1     draw\n",
       "3       a    1   c    1   d    2     draw\n",
       "4       a    1   c    2   c    1     draw\n",
       "...    ..  ...  ..  ...  ..  ...      ...\n",
       "28051   b    1   g    7   e    5  sixteen\n",
       "28052   b    1   g    7   e    6  sixteen\n",
       "28053   b    1   g    7   e    7  sixteen\n",
       "28054   b    1   g    7   f    5  sixteen\n",
       "28055   b    1   g    7   g    5  sixteen\n",
       "\n",
       "[28056 rows x 7 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63164a80",
   "metadata": {},
   "source": [
    "Existe un total de 28056 casos para 7 columnas. \n",
    "\n",
    "Comprobamos si existen valores perdidos:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a8072366",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "wkc         0\n",
       "wkr         0\n",
       "wrc         0\n",
       "wrr         0\n",
       "bkc         0\n",
       "bkr         0\n",
       "opt rank    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(data.isnull())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cd9216a",
   "metadata": {},
   "source": [
    "No existen valores omitidos, por lo que podemos usar todos los casos sin ningún problema.\n",
    "\n",
    "Estamos interesados en predecir los valores de _otp rank_, nos preguntamos cuál es la distribución de los casos para esta columna:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7f471729",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='opt rank', ylabel='count'>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjEwIDAgb2JqCjw8IC9Bbm5vdHMgWyBdIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDQwMi4zMzM1OTc5NTYzIDI4OS41MjQ0OTU1Nzc3IF0gL1BhcmVudCAyIDAgUgovUmVzb3VyY2VzIDggMCBSIC9UeXBlIC9QYWdlID4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTEgMCBSID4+CnN0cmVhbQp4nLVawW4cOQ6911fouHuYikiKInWcYGcD7G12Dex5kGk7zjjOrOOJF/v1+6Rud6mq1QmmgTYQIPXMlvgk8pGsNoWP05sfKdx9CTF8xL+XQOFdePO33df797t/vnsb3n+ZIvBPU4o8i4gWx+ND/8heZuWUSgYe148fpulxwvr4zDssfTdNKrPEIqYhp9m92X2axH2mLfzQw+zY8HXZZZEVjN1up/+E0y1UzSyI4DEw2ZxSeNqFf4fHEOdCmciTUcaDuiUWyzFPcc4gmcUswvruvGXYWIKjzpRPOFqcYzqhuKBUeI6qRRvD4xIruDH8OfxpjjExu5p681w95WwCiqkws2YudKA4NAwbw2kyme2UIW4rnzLs0DwTsabGe1mhQy/k51RwKyV6OdyMJVwNg6BEIvwyke8JnrEMG8tpwqnLKUOKNvMpxR72OQljIcDdGh16IUdj0aju3AKOouZIKRs4smcEZim4qMbxjGXYWE4TEW58QJJ19gHJBbY8R5MUqcLLIj18IcssiLeSkdb1gcULq7SbZJMsCYvbnuUZy7CxBEuGIA1YpoSdT1l2sONznpM18ssiHXwhS01WnKpcVO0o2TUbUyOZWgryIVzHhmFjCI7JR5JDWUaa08FIQy+RG/HjEgt4Ib+aUvjJsQZddkkOdfTGz8EHTMqe39gwbAzBDyIxEBxyGilOB3uG30a6p31cpIcv5IhcQlrFGu3V9SQlZbIaqcktqkTmQ904Yxk2lmDpOhIdjnEkOh1cCMIWc2r3uCzSw5eyRJQVjyh71XczQpUrOlXtZIiB5/JKcmgYNoYTvJaR5jD5SHM6uOhcALI16sdFevhijlK4mO+D0IiTctbSOgBBTDKy/ZXk0DJsLMGSeaQ5aBhGmtPBJOgHMlOSii+rrPBLszKiNyuKcGveR4Vista0tCgOCS3Kh7QcW4aNJXgmGukOOoWR7nQwJZ5NUYpKxZdVVvilPKko8o2aUGZUdMMnKFWeHhOaOPVX+Rlbho0leGoZ6Q9UeKQ/HUytkXP42/gfV1nhF/JEuGUSRKA3XQERiAnXjsDdUCmgqweeZyzDxhI8zYcKVGSoQAtMBZSRHe06l0V6+NKOQFHItVb7dke1N0Uo1ua1UIGwGsr+oSMYW4aNJViWPNIgIRppUAej3EKznbhl57LKCr+0v8sIBzRt1ro2NIvYncQbT1RGq0+H/m5sGTaWEzzXkQoJtGSgQh3MiM6M+Hdt/I+rrPBLe3UTEkF61VvSXJvuiJ688kSpSsUsvfbqY8uwsQRPbDRQIUllpEIdzEZzscjU6B8X6eGLJy6rN6SlzYr4iTXP6m1iAsEvGWxeR66hZdhYgqXySIMQ7iMN6uDa7kHkUusDulVW+KU8qQ7B7PVW1KQ2G9SS00QYHYq+Bu3YMGwMwRJnP1AgVJ2RAnWwc50CIlrlh36RHj5yfPMj79953AWq7z1A7KW9ogAyZXzi9A1Fh2Lt1/2ntxOFl+ntTXjzdwoUw81tK6eGtsugB2H18MP5XyGxPEoyZsgNLmmOEfrCEWC4+dTewdz8Ov3l16dfXv4abj6Gn26m5vWEVpdPve3Qa3hrUISC/icVToLTxhYlYZSGMPXe/m/39HntbSmgeeJth17DW7SIZEkxqHLkScosSSNGR4WMdd5+ftytnSU0n+nU2x6+hrtEUrMFI4KTpOov+j6UrvqWq/f3+WVzuGi5ETan/nbwVfzFaJxrdivmKZwvhkeOlq1IXJ3v84en3faEaz0ZeNzBV/FYUelUmUsyjKWCdgV6ZBm6lHuPbz//8bRxGJ3dIN16+CoOZxxxrj05ZN1rSCjUE/qtskq42/uv2xMuPMq4Hr6Kw46UFtRvNJtRJjT1mCKyIkp85fCX+/+u/UW1G+VcD1/DXwgDojZlzQYRRsuDIqciFN3Syt/d193jxuM6mQ087uCreIzZsDBaWkcrBI8TIhpNfULHsMq63f3dh+eNx8lHWdfDV/EYUVDQp2r05KlWDRQ7oyK01rXH+60QM3qZQdb18FUcRn+EDEtSf3xKtVdj7M6uqyB+PgkJ11HS9fBV/EVZRofsHmNG4cBM4lRbN9sUjt3DaRRLTKO86+GruFznw/qOs77ZzBip5ohZDVGCzn9d63YPW2kTllHi9fA1XJb6vpuIq5sKl3FA5EnrIad1ubt/et6dnHOiUe718FWcrgOqMfTNJfoE7cCOOGS2YtuKN3A6x1H+9fBVnEaZRlPhKHWRWnDkUjJaDUy866p3O/DZfJSDPXwVn6FJrmiFMSWLQKLwS4x0ivZ4W/h6n1cbI7IwFWFUYg2lvgmv/2kd6u/P4emXx9/WTMffqKZBpz3miW6itNNAscPE0Ya9uls8t039xqf2IbraZkHH23Ad6MmrZcwYATHKqyjt96IYz29XZyO0DnGz3wJ/d0OtX6FCxyNOMrcN+RsbckRkYEZYH+OCfne7YrNi2EWZK2V/c/Kt7VKGUBvUZL3fAn9vQwxhc2SqM43JfsPUbbiOaky8P2AID5RmOxxkqV9EQBdiu/olSN9//uPx2Dtw+Mf+jwTasLz+E4FxCJ75ln761/jr/k9nv+7HJ/7Unw2s7LuVvrnDz9P/AXu4vQgKZW5kc3RyZWFtCmVuZG9iagoxMSAwIG9iagoxOTU2CmVuZG9iagoxNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMwNCA+PgpzdHJlYW0KeJw9kjuSwzAMQ3udghfIjPiT5PNkJ5X3/u0+MslWgEmJACgvdZmypjwgaSYJ/9Hh4WI75XfYns3MwLVELxPLKc+hK8TcRfmymY26sjrFqsMwnVv0qJyLhk2TmucqSxm3C57DtYnnln3EDzc0qAd1jUvCDd3VaFkKzXB1/zu9R9l3NTwXm1Tq1BePF1EV5vkhT6KH6UrifDwoIVx7MEYWEuRT0UCOs1yt8l5C9g63GrLCQWpJ57MnPNh1ek8ubhfNEA9kuVT4TlHs7dAzvuxKCT0StuFY7n07mrHpGps47H7vRtbKjK5oIX7IVyfrJWDcUyZFEmROtlhui9We7qEopnOGcxkg6tmKhlLmYlerfww7bywv2SzIlMwLMkanTZ44eMh+jZr0eZXneP0BbPNzOwplbmRzdHJlYW0KZW5kb2JqCjE3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjMwID4+CnN0cmVhbQp4nDVRSW7DMAy86xXzgQDiLr/HQU/t/68d0glgYGhLnM0RGxsReInBz0HkxlvWjJr4m8ld8bs8FR4Jt4InUQRehnvZCS5vGJf9OMx88F5aOZMaTzIgF9n08ETIYJdA6MDsGtRhm2kn+oaEz45INRtZTl9L0EurEChP2X6nC0q0rerP7bMutO1rTzjZ7aknlU8gnluyApeNV0wWYxn0ROUuxfRBqrOFnoTyonwOsvmoIRJdopyBJwYHo0A7sOe2n4lXhaB1dZ+2jaEaKR1P/zY0NUki5BMlnNnSuFv4/p57/fwDplRTnwplbmRzdHJlYW0KZW5kb2JqCjE4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjI3ID4+CnN0cmVhbQp4nDVPO7IDIQzrOYUukBmMbWDPs5lUL/dvn2SyDRL+SPL0REcmXubICKzZ8bYWGYgZ+BZT8a897cOE6j24hwjl4kKYYSScNeu4m6fjxb9d5TPWwbsNvmKWFwS2MJP1lcWZy3bBWBoncU6yG2PXRGxjXevpFNYRTCgDIZ3tMCXIHBUpfbKjjDk6TuSJ52KqxS6/72F9waYxosIcVwVP0GRQlj3vJqAdF/Tf1Y3fSTSLXgIykWBhnSTmzllO+NVrR8dRiyIxJ6QZ5DIR0pyuYgqhCcU6OwoqFQWX6nPK3T7/aF1bTQplbmRzdHJlYW0KZW5kb2JqCjE5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ1ID4+CnN0cmVhbQp4nEVQu41DMQzrPQUXCGD9LHued0iV2789SkZwhSFaP5JaEpiIwEsMsZRv4kdGQT0LvxeF4jPEzxeFQc6EpECc9RkQmXiG2kZu6HZwzrzDM4w5AhfFWnCm05n2XNjknAcnEM5tlPGMQrpJVBVxVJ9xTPGqss+N14GltWyz05HsIY2ES0klJpd+Uyr/tClbKujaRROwSOSBk0004Sw/Q5JizKCUUfcwtY70cbKRR3XQydmcOS2Z2e6n7Ux8D1gmmVHlKZ3nMj4nqfNcTn3usx3R5KKlVfuc/d6RlvIitduh1elXJVGZjdWnkLg8/4yf8f4DjqBZPgplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTMzID4+CnN0cmVhbQp4nE2PQRLDMAgD736FnoCxAfOedHpK/n8tkDbuBe2MgJGGMAg8YgzrMCW8evvhVaRLcDaO+SUZRTwIagvcF1QFR2OKnfjY3aHspeLpFE2L6xFz07SkdDdRKm29ncj4wH2f3h9VtiSdgh5b6oQu0STyRQJz2FQwz+rGS0uPp+3Z3h9mPjPXCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA5MCA+PgpzdHJlYW0KeJxNjUESwCAIA++8Ik9QRND/dHrS/1+r1A69wE4CiRZFgvQ1aksw7rgyFWtQKZiUl8BVMFwL2u6iyv4ySUydhtN7twODsvFxg9JJ+/ZxegCr/XoG3Q/SHCJYCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMzggPj4Kc3RyZWFtCnicRVJLcsUwCNvnFFwgM+Zn4/O8Tlfp/beVcDrdPPQMCAkyPWVIptw2lmSE5BzypVdkiNWQn0aORMQQ3ymhwK7yubyWxFzIbolK8aEdP5elNzLNrtCqt0enNotGNSsj5yBDhHpW6MzuUdtkw+t2Iek6UxaHcCz/QwWylHXKKZQEbUHf2CPobxY8EdwGs+Zys7lMbvW/7lsLntc6W7FtB0AJlnPeYAYAxMMJ2gDE3NreFikoH1W6iknCrfJcJztQttCqdLw3gBkHGDlgw5KtDtdobwDDPg/0okbF9hWgqCwg/s7ZZsHeMclIsCfmBk49cTrFkXBJOMYCQIqt4hS68R3Y4i8Xroia8Al1OmVNvMKe2uLHQpMI71JxAvAiG25dHUW1bE/nCbQ/KpIzYqQexNEJkdSSzhEUlwb10Br7uIkZr43E5p6+3T/COZ/r+xcWuIPgCmVuZHN0cmVhbQplbmRvYmoKMjMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjMgPj4Kc3RyZWFtCnicRZC5dQQxDENzVYESeIA66hk/R7P9pwtpvN5A+niEeIg9CcNyXcWF0Q0/3rbMNLyOMtyN9WXG+KixQE7QBxgiE1ejSfXtijNU6eHVYq6jolwvOiISzJLjq0AjfDqyx0Nb25l+Oq9/7CHvE/8qKuduYQEuqu5A+VIf8dSP2VHqmqGPKitrHmravwi7IpS2fVxOZZy6ewe0wmcrV/t9A6jnOoAKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDY4ID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiFtCNEGUglgQpWYmZhBJOAMilwYAybQV5QplbmRzdHJlYW0KZW5kb2JqCjI1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggODEgPj4Kc3RyZWFtCnicPcy7FYAwCAXQPlO8EUJ8gOzjsdL9W8FEG7h81QMdIRnUDW4dh7SsS3eTfep6tYmkyIDSU2pcGk6MqGl9qX1q4Lsb5kvViT/Nz+cDh8cZawplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDUgPj4Kc3RyZWFtCnicMzK3UDBQsDQBEoYWJgrmZgYKKYZclhBWLhdMLAfMAtGWcAoingYAn30MtQplbmRzdHJlYW0KZW5kb2JqCjI3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTYxID4+CnN0cmVhbQp4nEWQSxLDIAxD95xCR/BHBnyedLpK77+tIU2zgKexQAZ3JwSptQUT0QUvbUu6Cz5bCc7GeOg2bjUS5AR1gFak42iUUn25xWmVdPFoNnMrC60THWYOepSjGaAQOhXe7aLkcqbuzvlHcPVf9Uex7pzNxMBk5Q6EZvUp7nybHVFd3WR/0mNu1mt/FfaqsLSspeWE285dM6AE7qkc7f0FqXM6hAplbmRzdHJlYW0KZW5kb2JqCjI4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjE0ID4+CnN0cmVhbQp4nD1QuxFDMQjrPQUL5M587TfPy6XL/m0knKRCNkISlJpMyZSHOsqSrClPHT5LYoe8h+VuZDYlKkUvk7Al99AK8X2J5hT33dWWs0M0l2g5fgszKqobHdNLNppwKhO6oNzDM/oNbXQDVocesVsg0KRg17YgcscPGAzBmROLIgxKTQb/rXL3UtzvPRxvooiUdPCu+eX0y88tvE49jkS6vfmKa3GmOgpEcEZq8op0YcWyyEOk1QQ1PQNrtQCu3nr5N2hHdBmA7BOJ4zSlHEP/1rjH6wOHilL0CmVuZHN0cmVhbQplbmRvYmoKMjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MCA+PgpzdHJlYW0KeJxFjLsNwDAIRHumYAR+JmafKJWzfxsgStxwT7p7uDoSMlPeYYaHBJ4MLIZT8QaZo2A1uEZSjZ3so7BuX3WB5npTq/X3BypPdnZxPc3LGfQKZW5kc3RyZWFtCmVuZG9iagozMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIzNiA+PgpzdHJlYW0KeJxNUEtuRCEM23OKXOBJJCEBzkPVVef+27HDVO0qhhh/SA/pslUe61NidYns8qVNl8oyeRWo5U/b/1EMAm7/0MhBtLeMnWLmEtbFwiQ85TQjGyfXLB+PO08bZoXGxI3jnS4ZYJ8WATVblc2BOW06N0C6kBq3qrPeZFAMIupCzQeTLpyn0ZeIOZ6oYEp3JrWQG1w+1aEDcVq9Crlji5NvxBxZocBh0Exx1l8B1qjJslnIIEmGIc59o3uUCo2oynkrFcIPk6ER9YbVoAaVuYWiqeWS/B3aAjAFtox16QxKgaoAwd8qp32/ASSNXVMKZW5kc3RyZWFtCmVuZG9iagozMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE1NyA+PgpzdHJlYW0KeJxFkLkRQzEIRHNVQQkSsAjqscfRd/+pF/lKtG8ALYevJVOqHyciptzXaPQweQ6fTSVWLNgmtpMachsWQUoxmHhOMaujt6GZh9TruKiquHVmldNpy8rFf/NoVzOTPcI16ifwTej4nzy0qehboK8LlH1AtTidSVAxfa9igaOcdn8inBjgPhlHmSkjcWJuCuz3GQBmvle4xuMF3QE3eQplbmRzdHJlYW0KZW5kb2JqCjMyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzMyID4+CnN0cmVhbQp4nC1SOY4kMQzL/Qp+YADr8vGeHkzU+/90SVUFBapsyzzkcsNEJX4skNtRa+LXRmagwvCvq8yF70jbyDqIa8hFXMmWwmdELOQxxDzEgu/b+Bke+azMybMHxi/Z9xlW7KkJy0LGizO0wyqOwyrIsWDrIqp7eFOkw6kk2OOL/z7FcxeCFr4jaMAv+eerI3i+pEXaPWbbtFsPlmlHlRSWg+1pzsvkS+ssV8fj+SDZ3hU7QmpXgKIwd8Z5Lo4ybWVEa2Fng6TGxfbm2I+lBF3oxmWkOAL5mSrCA0qazGyiIP7I6SGnMhCmrulKJ7dRFXfqyVyzubydSTJb90WKzRTO68KZ9XeYMqvNO3mWE6VORfgZe7YEDZ3j6tlrmYVGtznBKyV8NnZ6cvK9mlkPyalISBXTugpOo8gUS9iW+JqKmtLUy/Dfl/cZf/8BM+J8AQplbmRzdHJlYW0KZW5kb2JqCjMzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTcgPj4Kc3RyZWFtCnicMza0UDCAwxRDLgAalALsCmVuZHN0cmVhbQplbmRvYmoKMzQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMzEgPj4Kc3RyZWFtCnicRY/LDQQhDEPvVOES8hk+qYfVntj+r+swmkFC+EEiO/EwCKzz8jbQxfDRosM3/jbVq2OVLB+6elJWD+mQh7zyFVBpMFHEhVlMHUNhzpjKyJYytxvhtk2DrGyVVK2DdjwGD7anZasIfqltYeos8QzCVV64xw0/kEutd71Vvn9CUzCXCmVuZHN0cmVhbQplbmRvYmoKMzUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMzggPj4Kc3RyZWFtCnicNVI5rt1ADOt9Cl0ggHbNnOcFqX7u34aUXwpDtFaKmo4WlWn5ZSFVLZMuv+1JbYkb8vfJCokTklcl2qUMkVD5PIVUv2fLvL7WnBEgS5UKk5OSxyUL/gyX3i4c52NrP48jdz16YFWMhBIByxQTo2tZOrvDmo38PKYBP+IRcq5YtxxjFUgNunHaFe9D83nIGiBmmJaKCl1WiRZ+QfGgR61991hUWCDR7RxJcIyNUJGAdoHaSAw5sxa7qC/6WZSYCXTtiyLuosASScycYl06+g8+dCyovzbjy6+OSvpIK2tM2nejSWnMIpOul0VvN299PbhA8y7Kf17NIEFT1ihpfNCqnWMomhllhXccmgw0xxyHzBM8hzMSlPR9KH5fSya6KJE/Dg2hf18eo4ycBm8Bc9GftooDF/HZYa8cYIXSxZrkfUAqE3pg+v/X+Hn+/AMctoBUCmVuZHN0cmVhbQplbmRvYmoKMzYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDggPj4Kc3RyZWFtCnicLVE5kgNBCMvnFXpCc9PvscuR9//pCsoBg4ZDIDotcVDGTxCWK97yyFW04e+ZGMF3waHfynUbFjkQFUjSGFRNqF28Hr0HdhxmAvOkNSyDGesDP2MKN3pxeEzG2e11GTUEe9drT2ZQMisXccnEBVN12MiZw0+mjAvtXM8NyLkR1mUYpJuVxoyEI00hUkih6iapM0GQBKOrUaONHMV+6csjnWFVI2oM+1xL29dzE84aNDsWqzw5pUdXnMvJxQsrB/28zcBFVBqrPBAScL/bQ/2c7OQ33tK5s8X0+F5zsrwwFVjx5rUbkE21+Dcv4vg94+v5/AOopVsWCmVuZHN0cmVhbQplbmRvYmoKMzcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNzEgPj4Kc3RyZWFtCnicTZBNDkIhEIP3nKIXMKHzA4/zaFzp/bd28PnigvRLIUOnwwMdR+JGR4bO6HiwyTEOvAsyJl6N85+M6ySOCeoVbcG6tDvuzSwxJywTI2BrlNybRxT44ZgLQYLs8sMXGESka5hvNZ91k35+u9Nd1KV199MjCpzIjlAMG3AF2NM9DtwSzu+aJr9UKRmbOJQPVBeRstkJhailYpdTVWiM4lY974te7fkBwfY7+wplbmRzdHJlYW0KZW5kb2JqCjM4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzIgPj4Kc3RyZWFtCnicNYyxEcAwCAN7ptAINlhg75NLRfZvQ3xOAy8dD5eiwVoNuoIjcHWp/NEjXbkpRZdjzoLhcapfSDFGPagj497HT7lfcBYSfQplbmRzdHJlYW0KZW5kb2JqCjM5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggODggPj4Kc3RyZWFtCnicNYy7EcAwCEN7T8EIBouP98mlSvZvg+3QgKR394KDOkHyuBspnC5u2Vd6G4+TniYAsfRMQ+3fYEXVi1oULV9uY9BiKr4/+iQglnXyXjj0kBLeH8UXHXsKZW5kc3RyZWFtCmVuZG9iago0MCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDg3ID4+CnN0cmVhbQp4nDVNuRHAMAjrmYIRzKPY7JNL5ezfBuy4QTp9IJQba+QBguGdbyH4pi8ZhHUITyq7JTpsoYazCpKJ4Vc2eFWuiva1konsbKYx2KBl+tHOt0nPB6XeG5gKZW5kc3RyZWFtCmVuZG9iago0MSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDc0ID4+CnN0cmVhbQp4nD2MwQ2AMAwD/50iIzSJTTIQ4gX7f2kK7cc+nWTTKF3gFWlChJzayElPW+6ehIODFJCwX23o1b4qS3uqIGoy/jZ8d9cLdxwXTgplbmRzdHJlYW0KZW5kb2JqCjQyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjEwID4+CnN0cmVhbQp4nDVQyw1DMQi7ZwoWqBQCgWSeVr11/2tt0DthEf9CWMiUCHmpyc4p6Us+OkwPti6/sSILrXUl7MqaIJ4r76GZsrHR2OJgcBomXoAWN2DoaY0aNXThgqYulUKBxSXwmXx1e+i+Txl4ahlydgQRQ8lgCWq6Fk1YtDyfkE4B4v9+w+4t5KGS88qeG/kbnO3wO7Nu4SdqdiLRchUy1LM0xxgIE0UePHlFpnDis9Z31TQS1GYLTpYBrk4/jA4AYCJeWYDsrkQ5S9KOpZ9vvMf3D0AAU7QKZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iago8PCAvQmFzZUZvbnQgL0RlamFWdVNhbnMgL0NoYXJQcm9jcyAxNSAwIFIKL0VuY29kaW5nIDw8Ci9EaWZmZXJlbmNlcyBbIDMyIC9zcGFjZSA0OCAvemVybyAvb25lIC90d28gL3RocmVlIC9mb3VyIDk3IC9hIDk5IC9jIC9kIC9lIC9mIC9nIC9oIC9pCjEwNyAvayAvbCAxMTAgL24gL28gL3AgMTE0IC9yIC9zIC90IC91IC92IC93IC94IDEyMiAveiBdCi9UeXBlIC9FbmNvZGluZyA+PgovRmlyc3RDaGFyIDAgL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRm9udERlc2NyaXB0b3IgMTMgMCBSCi9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdIC9MYXN0Q2hhciAyNTUgL05hbWUgL0RlamFWdVNhbnMKL1N1YnR5cGUgL1R5cGUzIC9UeXBlIC9Gb250IC9XaWR0aHMgMTIgMCBSID4+CmVuZG9iagoxMyAwIG9iago8PCAvQXNjZW50IDkyOSAvQ2FwSGVpZ2h0IDAgL0Rlc2NlbnQgLTIzNiAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRm9udE5hbWUgL0RlamFWdVNhbnMgL0l0YWxpY0FuZ2xlIDAKL01heFdpZHRoIDEzNDIgL1N0ZW1WIDAgL1R5cGUgL0ZvbnREZXNjcmlwdG9yIC9YSGVpZ2h0IDAgPj4KZW5kb2JqCjEyIDAgb2JqClsgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAKNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCAzMTggNDAxIDQ2MCA4MzggNjM2Cjk1MCA3ODAgMjc1IDM5MCAzOTAgNTAwIDgzOCAzMTggMzYxIDMxOCAzMzcgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNgo2MzYgNjM2IDMzNyAzMzcgODM4IDgzOCA4MzggNTMxIDEwMDAgNjg0IDY4NiA2OTggNzcwIDYzMiA1NzUgNzc1IDc1MiAyOTUKMjk1IDY1NiA1NTcgODYzIDc0OCA3ODcgNjAzIDc4NyA2OTUgNjM1IDYxMSA3MzIgNjg0IDk4OSA2ODUgNjExIDY4NSAzOTAgMzM3CjM5MCA4MzggNTAwIDUwMCA2MTMgNjM1IDU1MCA2MzUgNjE1IDM1MiA2MzUgNjM0IDI3OCAyNzggNTc5IDI3OCA5NzQgNjM0IDYxMgo2MzUgNjM1IDQxMSA1MjEgMzkyIDYzNCA1OTIgODE4IDU5MiA1OTIgNTI1IDYzNiAzMzcgNjM2IDgzOCA2MDAgNjM2IDYwMCAzMTgKMzUyIDUxOCAxMDAwIDUwMCA1MDAgNTAwIDEzNDIgNjM1IDQwMCAxMDcwIDYwMCA2ODUgNjAwIDYwMCAzMTggMzE4IDUxOCA1MTgKNTkwIDUwMCAxMDAwIDUwMCAxMDAwIDUyMSA0MDAgMTAyMyA2MDAgNTI1IDYxMSAzMTggNDAxIDYzNiA2MzYgNjM2IDYzNiAzMzcKNTAwIDUwMCAxMDAwIDQ3MSA2MTIgODM4IDM2MSAxMDAwIDUwMCA1MDAgODM4IDQwMSA0MDEgNTAwIDYzNiA2MzYgMzE4IDUwMAo0MDEgNDcxIDYxMiA5NjkgOTY5IDk2OSA1MzEgNjg0IDY4NCA2ODQgNjg0IDY4NCA2ODQgOTc0IDY5OCA2MzIgNjMyIDYzMiA2MzIKMjk1IDI5NSAyOTUgMjk1IDc3NSA3NDggNzg3IDc4NyA3ODcgNzg3IDc4NyA4MzggNzg3IDczMiA3MzIgNzMyIDczMiA2MTEgNjA1CjYzMCA2MTMgNjEzIDYxMyA2MTMgNjEzIDYxMyA5ODIgNTUwIDYxNSA2MTUgNjE1IDYxNSAyNzggMjc4IDI3OCAyNzggNjEyIDYzNAo2MTIgNjEyIDYxMiA2MTIgNjEyIDgzOCA2MTIgNjM0IDYzNCA2MzQgNjM0IDU5MiA2MzUgNTkyIF0KZW5kb2JqCjE1IDAgb2JqCjw8IC9hIDE2IDAgUiAvYyAxNyAwIFIgL2QgMTggMCBSIC9lIDE5IDAgUiAvZiAyMCAwIFIgL2ZvdXIgMjEgMCBSCi9nIDIyIDAgUiAvaCAyMyAwIFIgL2kgMjQgMCBSIC9rIDI1IDAgUiAvbCAyNiAwIFIgL24gMjcgMCBSIC9vIDI4IDAgUgovb25lIDI5IDAgUiAvcCAzMCAwIFIgL3IgMzEgMCBSIC9zIDMyIDAgUiAvc3BhY2UgMzMgMCBSIC90IDM0IDAgUgovdGhyZWUgMzUgMCBSIC90d28gMzYgMCBSIC91IDM3IDAgUiAvdiAzOCAwIFIgL3cgMzkgMCBSIC94IDQwIDAgUiAveiA0MSAwIFIKL3plcm8gNDIgMCBSID4+CmVuZG9iagozIDAgb2JqCjw8IC9GMSAxNCAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgPj4KZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMCAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjQzIDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMTA2MDcxNzI4MzcrMDInMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My4zLjQsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My4zLjQpID4+CmVuZG9iagp4cmVmCjAgNDQKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMTExMzcgMDAwMDAgbiAKMDAwMDAxMDk0MyAwMDAwMCBuIAowMDAwMDEwOTc1IDAwMDAwIG4gCjAwMDAwMTEwNzQgMDAwMDAgbiAKMDAwMDAxMTA5NSAwMDAwMCBuIAowMDAwMDExMTE2IDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDQwNyAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDI0MzggMDAwMDAgbiAKMDAwMDAwOTU4MCAwMDAwMCBuIAowMDAwMDA5MzgwIDAwMDAwIG4gCjAwMDAwMDg5NjAgMDAwMDAgbiAKMDAwMDAxMDYzMyAwMDAwMCBuIAowMDAwMDAyNDU5IDAwMDAwIG4gCjAwMDAwMDI4MzYgMDAwMDAgbiAKMDAwMDAwMzEzOSAwMDAwMCBuIAowMDAwMDAzNDM5IDAwMDAwIG4gCjAwMDAwMDM3NTcgMDAwMDAgbiAKMDAwMDAwMzk2MyAwMDAwMCBuIAowMDAwMDA0MTI1IDAwMDAwIG4gCjAwMDAwMDQ1MzYgMDAwMDAgbiAKMDAwMDAwNDc3MiAwMDAwMCBuIAowMDAwMDA0OTEyIDAwMDAwIG4gCjAwMDAwMDUwNjUgMDAwMDAgbiAKMDAwMDAwNTE4MiAwMDAwMCBuIAowMDAwMDA1NDE2IDAwMDAwIG4gCjAwMDAwMDU3MDMgMDAwMDAgbiAKMDAwMDAwNTg1NSAwMDAwMCBuIAowMDAwMDA2MTY0IDAwMDAwIG4gCjAwMDAwMDYzOTQgMDAwMDAgbiAKMDAwMDAwNjc5OSAwMDAwMCBuIAowMDAwMDA2ODg4IDAwMDAwIG4gCjAwMDAwMDcwOTIgMDAwMDAgbiAKMDAwMDAwNzUwMyAwMDAwMCBuIAowMDAwMDA3ODI0IDAwMDAwIG4gCjAwMDAwMDgwNjggMDAwMDAgbiAKMDAwMDAwODIxMiAwMDAwMCBuIAowMDAwMDA4MzcyIDAwMDAwIG4gCjAwMDAwMDg1MzEgMDAwMDAgbiAKMDAwMDAwODY3NyAwMDAwMCBuIAowMDAwMDExMTk3IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gNDMgMCBSIC9Sb290IDEgMCBSIC9TaXplIDQ0ID4+CnN0YXJ0eHJlZgoxMTM1NAolJUVPRgo=\n",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEiCAYAAADTSFSPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAtUElEQVR4nO3debgcVbX38e9KICQBwmDCIAGCEgWCIDcxhOnKnECARMYggchgmAVBveAAoqL4XhSZEeEyCIq5gIAoIgRQ8aIYBJkRVIQAQkR5Ra+gwHr/WKvfUzR9Tld3nT5D8vs8Tz+nu7prn93VVbX2VLvM3REREWnXkP7OgIiIDG4KJCIiUokCiYiIVKJAIiIilSiQiIhIJQokIiJSyVL9nYFOGT16tI8bN66/syEiMmiMHj2am2+++WZ3n9bKeottIBk3bhwLFizo72yIiAwqZja61XXUtCUiIpUokIiISCUKJCIiUokCiYiIVKJAIiIilSiQiIhIJQokIiJSiQKJiIhUokAiIiKVLLZXtouIdNKTX/tjW+uNO3a1Xs5J/1ONREREKlEgERGRShRIRESkEgUSERGpRIFEREQqUSAREZFKFEhERKQSBRIREalEgURERCpRIBERkUoUSEREpBIFEhERqUSBREREKlEgERGRShRIRESkEt2PRESWGD+7fFHb625xwJhezMniRTUSERGpRIFEREQqUSAREZFKFEhERKSSjgcSMxtqZvea2Y35emUzu8XMHs+/KxU+e6KZPWFmj5nZ1MLyiWb2QL53lplZp/MtIiLl9EWN5BjgkcLrE4D57j4emJ+vMbMNgFnABGAacJ6ZDc11zgfmAuPzMa0P8i0iIiV0NJCY2VhgOnBRYfEM4LJ8fhkws7D8Knd/1d1/DzwBTDaz1YFR7n6XuztweWEdERHpZ52ukXwN+ATwRmHZqu7+HED+XSWXrwE8Xfjcwly2Rj6vXy4iIgNAxwKJme0CvODu95RdpcEy72F5o/8518wWmNmCRYvav/BIRETK62SNZAtgNzN7ErgK2NbMrgCez+Yq8u8L+fmFwJqF9ccCz+bysQ2Wv4W7X+juk9x90pgxugpVRKQvdCyQuPuJ7j7W3ccRnei3ufts4AZgTn5sDnB9Pr8BmGVmy5jZOkSn+t3Z/PWymU3J0VoHFNYREZF+1h9zbZ0GzDOzg4GngL0A3P0hM5sHPAy8Bhzp7q/nOocDlwIjgJvyISIiA0CfBBJ3vwO4I5+/CGzXzedOBU5tsHwBsGHncigiIu3Sle0iIlKJAomIiFSiQCIiIpUokIiISCUKJCIiUokCiYiIVKJAIiIilSiQiIhIJQokIiJSiQKJiIhUokAiIiKVKJCIiEglCiQiIlJJf0wjLyJS2revaf9up/vuoRvc9QXVSEREpBIFEhERqUSBREREKlEgERGRShRIRESkEgUSERGpRIFEREQqUSAREZFKFEhERKQSBRIREalEgURERCpRIBERkUoUSEREpBIFEhERqUSBREREKlEgERGRShRIRESkEgUSERGpRIFEREQqUSAREZFKFEhERKQSBRIREamkY4HEzIab2d1m9msze8jMTsnlK5vZLWb2eP5dqbDOiWb2hJk9ZmZTC8snmtkD+d5ZZmadyreIiLSmkzWSV4Ft3X1j4L3ANDObApwAzHf38cD8fI2ZbQDMAiYA04DzzGxopnU+MBcYn49pHcy3iIi0oGOBxMPf8uXS+XBgBnBZLr8MmJnPZwBXufur7v574AlgspmtDoxy97vc3YHLC+uIiEg/62gfiZkNNbP7gBeAW9z9F8Cq7v4cQP5dJT++BvB0YfWFuWyNfF6/XEREBoCOBhJ3f93d3wuMJWoXG/bw8Ub9Ht7D8rcmYDbXzBaY2YJFixa1nF8REWldn4zacveXgDuIvo3ns7mK/PtCfmwhsGZhtbHAs7l8bIPljf7Phe4+yd0njRkzpje/goiIdKOTo7bGmNmK+XwEsD3wKHADMCc/Nge4Pp/fAMwys2XMbB2iU/3ubP562cym5GitAwrriIhIP1uqg2mvDlyWI6+GAPPc/UYzuwuYZ2YHA08BewG4+0NmNg94GHgNONLdX8+0DgcuBUYAN+VDREQGgI4FEne/H9ikwfIXge26WedU4NQGyxcAPfWviIhIP9GV7SIiUokCiYiIVNLJPhIRWYJ95LtPN/9QN876wJrNPyQDhmokIiJSyWJfI1l0/hVtrTfm8Nm9nBMRkcWTaiQiIlKJAomIiFSiQCIiIpUokIiISCUKJCIiUokCiYiIVFIqkJjZ/DLLRERkydPjdSRmNhwYCYw2s5XousnUKODtHc6biIgMAs0uSDwUOJYIGvfQFUj+CpzbuWyJiMhg0WMgcfczgTPN7Gh3P7uP8iQiIoNIqSlS3P1sM9scGFdcx90v71C+RERkkCgVSMzsm8A7gfuA2l0LHVAgERFZwpWdtHESsIG7eyczIyIig0/Z60geBFbrZEZERGRwKlsjGQ08bGZ3A6/WFrr7bh3JlYiIDBplA8lnO5kJEREZvMqO2vpxpzMiIiKDU9lRWy8To7QAhgFLA39391GdypiIiAwOZWskyxdfm9lMYHInMiQiIoNLW7P/uvt1wLa9mxURERmMyjZt7V54OYS4rkTXlIiISOlRW7sWnr8GPAnM6PXciIjIoFO2j+TATmdEREQGp7I3thprZt81sxfM7Hkzu8bMxnY6cyIiMvCV7Wy/BLiBuC/JGsD3cpmIiCzhygaSMe5+ibu/lo9LgTEdzJeIiAwSZQPJn8xstpkNzcds4MVOZkxERAaHsoHkIGBv4I/Ac8CegDrgRUSk9PDfzwNz3P0vAGa2MnA6EWBERKRNz59xf9vrrvrRjXoxJ+0rWyPZqBZEANz9z8AmncmSiIgMJmUDyRAzW6n2ImskZWszIiKyGCsbDL4C/I+ZXU1MjbI3cGrHciUiIoNGqRqJu18O7AE8DywCdnf3b/a0jpmtaWa3m9kjZvaQmR2Ty1c2s1vM7PH8W6zpnGhmT5jZY2Y2tbB8opk9kO+dZWbWzpcVEZHeV3r2X3d/2N3Pcfez3f3hEqu8Bhzv7usDU4AjzWwD4ARgvruPB+bna/K9WcAEYBpwnpkNzbTOB+YC4/MxrWy+RUSks9qaRr4Md3/O3X+Vz18GHiGuip8BXJYfuwyYmc9nAFe5+6vu/nvgCWCyma0OjHL3u9zdgcsL64iISD/rWCApMrNxxCivXwCruvtzEMEGWCU/tgbwdGG1hblsjXxev1xERAaAjo+8MrPlgGuAY939rz10bzR6w3tY3uh/zSWawFhrrbVaz6zIEm73a37e9rrX7jGlF3Mig0lHayRmtjQRRK5092tz8fPZXEX+fSGXLwTWLKw+Fng2l49tsPwt3P1Cd5/k7pPGjNFUYCIifaFjgSRHVl0MPOLuXy28dQMwJ5/PAa4vLJ9lZsuY2TpEp/rd2fz1splNyTQPKKwjIiL9rJNNW1sA+wMPmNl9ueyTwGnAPDM7GHgK2AvA3R8ys3nAw8SIryPd/fVc73DgUmAEcFM+RERkAOhYIHH3O2ncvwGwXTfrnEqDCx3dfQGwYe/lTkREekufjNoSEZHFlwKJiIhUokAiIiKVKJCIiEglCiQiIlKJAomIiFSiQCIiIpUokIiISCW6Xa7IYmC3q29sa70b9tyll3MiSyLVSEREpBIFEhERqUSBREREKlEgERGRShRIRESkEgUSERGpRIFEREQqUSAREZFKFEhERKQSBRIREalEgURERCrRXFsi/WSXq69se90b99yvF3MiUo1qJCIiUokCiYiIVKJAIiIilSiQiIhIJQokIiJSiQKJiIhUokAiIiKVKJCIiEglCiQiIlKJAomIiFSiQCIiIpUokIiISCUKJCIiUokCiYiIVKJAIiIilXQskJjZf5nZC2b2YGHZymZ2i5k9nn9XKrx3opk9YWaPmdnUwvKJZvZAvneWmVmn8iwiIq3rZI3kUmBa3bITgPnuPh6Yn68xsw2AWcCEXOc8Mxua65wPzAXG56M+TRER6UcdCyTu/hPgz3WLZwCX5fPLgJmF5Ve5+6vu/nvgCWCyma0OjHL3u9zdgcsL64iIyADQ130kq7r7cwD5d5VcvgbwdOFzC3PZGvm8frmIiAwQA6WzvVG/h/ewvHEiZnPNbIGZLVi0aFGvZU5ERLrX14Hk+WyuIv++kMsXAmsWPjcWeDaXj22wvCF3v9DdJ7n7pDFjxvRqxkVEpLG+DiQ3AHPy+Rzg+sLyWWa2jJmtQ3Sq353NXy+b2ZQcrXVAYR0RERkAlupUwmb2bWBrYLSZLQROBk4D5pnZwcBTwF4A7v6Qmc0DHgZeA45099czqcOJEWAjgJvyISIiA0THAom779vNW9t18/lTgVMbLF8AbNiLWRMRkV40UDrbRURkkFIgERGRShRIRESkEgUSERGpRIFEREQqUSAREZFKFEhERKQSBRIREalEgURERCpRIBERkUo6NkWKyOJo+rXntb3u93c/ohdzIjJwqEYiIiKVKJCIiEglCiQiIlKJ+khkibDzdZ9pe90fzPx8L+ZEZPGjGomIiFSiQCIiIpUokIiISCUKJCIiUok622XAOvC709pe95IP/LAXcyIiPVGNREREKlEgERGRShRIRESkEvWRiIgsJl44+9a2113l6O3bXlc1EhERqUSBREREKlEgERGRShRIRESkEgUSERGpRKO2BrGbL9657XWnHvyDXsyJiCzJVCMREZFKFEhERKQSNW1Jr/vKt6e2td7x+97cyzkRkb6gGomIiFSiQCIiIpUokIiISCUKJCIiUsmg6Ww3s2nAmcBQ4CJ3P62fs9S2+8/fre11Nzr8hl7MSZdvXtpeBznA/h9SJ7nIkmxQ1EjMbChwLrATsAGwr5lt0L+5EhERGDw1ksnAE+7+OwAzuwqYATzcVxn443knt73uakec0os5EREZWAZFjQRYA3i68HphLhMRkX5m7t7feWjKzPYCprr7Ifl6f2Cyux9d97m5wNx8+W7gsSZJjwb+1EvZ7K20lKe+T0t56vu0lKe+T6tMOn8CcPdprSQ8WJq2FgJrFl6PBZ6t/5C7XwhcWDZRM1vg7pOqZ6/30lKe+j4t5anv01Ke+j6t3sxTvcHStPVLYLyZrWNmw4BZQGeGL4mISEsGRY3E3V8zs6OAm4nhv//l7g/1c7ZERIRBEkgA3P0HQG/fRKN0M1gfpqU89X1aylPfp6U89X1avZmnNxkUne0iIjJwDZY+EhERGaAUSEREpBIFkmRmNhDTWhz1xvbprW08GH6rwZDHJcFAOUcMxP1BgSR5dhaZ2fpmNqpicsOLL9r54c1sjJmt2G4GOrGzdZemmZXej8zMCtt6SivrdpNGpe1USGcHM1u33XSaMbMROXS91e01HNgsn69nZpu2+f/f8tsNpIDeU1qt/o/eSqd+3cK+soKZLdtumnVpjTazEW2uu3Sr/7u7NFtZ3vCzS3pnu5kNcfc38vnqwMXAKe7+i+J7LaR3MLAt8BRwt7t/N5f//x2gRBofAaYCfwGedvcTW8lDXVp7AusBjwD3uPuTbaZT3IE/BIwgRv2d7+6vtZHedODTwI7u/nKbeWp7O9W+j5kt7e7/MrNbgSvc/dJ28tLkfx0PbAn8Dfiyuz9Ydt8yszWAXYEdgAnAtFZ/w7rfblNgaeDnbf5uuwP/An7j7o/Vp9+uujwekHl8yt1vaeV/1BdUgFeBP7j7n3spbx8jJo/9CzEL+Q9bPLaLaR0H7AvcC9zl7pe0sO4xwPbAncAd7v6LXvhuhwJrAw8B33f3l8rup0t8jaQQRMa5+3PAlcDXzWxsG0Fkb+B4Ypjdq8D7zeyI/D9ld7RZxISUs4G/A+9pJQ91aR0KnEycwOYAh5jZ+9pJq7CzHQscQMx9dhhdU9K0kq+dgC8AR7n7y+2UrKpup8LvsXr+/Q7xm72pJFa1xG1mWwI7A6cDDwBXm9nG7v6GxazWzfL5DPAG8V3vrgWRVmo1hd/uI8BXiH3hXjMb1+J3+SDwReBg4Cgz26OWftXtVMjjdOBEYGNgVp64S/+PQjofB04FPg6cZWYb9kLeNgO2AT4GXAecY2bTW/n+hbTeB/wbcDRwE7BPHq9l1t0883EVMAw43My2auOrFdPcFjgIeIWoAZ9kZivnftp8X3P3JfoBGLAd8L/A14ipWA4EPgcMbyGd1Ygd7MB8vQKwBxFUlm8hnRnAJOAY4gLMpXP5Ji1+r6HApcDEfL0ecfI+tsK2Gg58I5//B3Bj/p+RLaYzAXiGKNHVlg1pMY09gPe1up2ATYEtCvn4HXA58GtiNul/I2abXqHVPDX4X9OAW4FjCsuOJUp8E1tIZxiwJ3AK8CVglVy+UgtpTCZKmeQ2m1/8fmTrRA/r70ME2xG5HxxJ3B9o9yrbqO5/fJgoyL29sP2+ARzfYjqTgBvz+enESX9IbR9pM2/bAD8DTq7bB38DfKCFdIyonb4AfCqXjSBqFzc1Oz6BXYCXgF3y9XgiGF0CbNvmd/sQUbNZP19PAf4zt93byqShGkmYD1wGrAXMI6p344BVy6RhZgcRpbT/JUr967j7/wW+B7yTrlJvGcsTJ58d3H2qR7PLIcDBZdtSzWwyMBJ4EdjfzIa7+6OZ7i619t0S6dSXsoYAq5jZtcSJaQ93f50oOW5dIr19zOwTHrMS7AC8x8xOgagZlin5mNmBWbp8HbiF1rfTKsDvzWyVzMfWwCeBa4hguw9Rav8SsFyz/DSxgJgEbzMzWwXA3b8GXEHUepep38ZmNqqWfzPbycxOBea6+9XAd4mTzlFmNgP4rJm9rdE/bvDbPQv8yMy+AkwnmsjeMLN9M1/NaszrAnsB73H3V4CriZPodDNr605tDfI4lGjq2Sxf30n8LpOyNlUmzeHAIuAxM/syUVjYx6N1Yauytd8GefsJ8XtONrO1sknoGuCzwMlmtmx3tZLi8jzf3EkUMGeZ2Wh3/0emfxbw71bo86tP091vBO7L/4u7Pw58n2i63rvMOaJBPu8nCmVz8vUviN93JHC8aiTNI/HuwEXA+uTBRfRvnA68BlxZIo2dgdOAcfn6ZGIn2Zho2/4ZWYLsIY0Dc8fYKl9/nmg3XZ8oPd4PTCj5nVYGzs7vshVxUpyb7+1GlM6WLZGOFZ7/O3EiWSm307OFvB5AlLDXLpHmZKLUf1S+Xh+4Azi9TD7y9SeBy/P5V4F7ymwn3lz63hi4iyjdDcllS+XvthoRQHr8zZp8z32ADxBBahhxUJ4GrFr8nRqsNxK4njiga9vqOOJkem1+5j2Z1uPAbiV+u5FEf8MYohZyJzA035tNzGO3ag/fZU26anufzO39rny9OlGL6Hb9kvvXe4AV8vkHgb8C78vXyxOl9dVLpLkp8InM1zzgp2SJGjgU+DklanF1edufOD73ytcX5H4yrvY5emhxqEtrv9xP98zXp1M4P+TvNLKbdacTrRXvzNe3AzcV3h/XxnfbHFgvn7+bqOnUjk0janaljoO2DpTB+uCtJ6UReVB+jggo36PrBDkJGN9DWkOIKv4DxAlsbC6fQLTx3k6UFN7bJE8zgLuJk/9/AR/JdD+defo2WeUs8f1qVdNDiE5wiOr3VcRJ5G5g4xa32bHAj4ngdhMwMQ/Kp/KA+hVNglxuk9oBPZEIkh/N1+/JdMeUzM87gfOBNfL16ZmPbrdT3cFzBFGVP56o0exEnGxHEieeLSvuY8fk9ppNNJttQZzE/5u4y2eP3zP3h1uBrwNzctlSuf41hc+t1s0+XQyYxwE/Ikq6G+fjV0Q/xzlEc163v12ufx3RbHIIMCr37buADfIzQytur2Pzdzgf+EouO4xo+tmi0Xfs4XjenTjulsnnF2TeTwYebLafdpO32/MYegKYSdSavkE0863VQlrHEcfgnPwNZufyrxG1iW73C6LJ/Haiz+cndAXZW4lO+na2+/HAbcB5eewsRzTrPg98ouX0quwEg+lRdzKZTZQ0ts/X6+UP/Wfg98CmJdJbMf8unz/IV+veHwWMaJLGDKKUtHq+3j13rKOAYblsqZLfbyuiqeGLRCn4R8BJ+d4Q4hbFo1vcThOAW/L5OUQpr1YKexdRS1mjSXrvIgLikWQpnAjSL9HVRjyswXpTgEmF3+tTZImLqIl8p+7zTbcTEQAX0BWE5hLBfvs8QRwJrNPm/jWEaAq9Ol9/mqhdLJOvVyaaTxuW8HhzANiUKKmeCSyXy4YSfVK31H++m/Qm5//fkQietxOBZB2i+ehosnTbzfo7Aj/O5z+lq2AyhOiruY0oQffYt9Ikj9sAt+Xzq4kmv1oN8WjgSaJQ1az/ZuXC84vJGi7RXHMIcTy9q8W8LU9XzfdEopBZ+y2H5G/TtJZUSOvCfP5xYs7AZQrf9Utki0aDddcG5uXzk3IfWKbw/vdoIaDlOpsAP8znFxCFlNr5ZiJRAFqpld+2rR1gMD+IgPETok/jZ8AJhfcOBL5F85PjYURzw6lEMFiBqPJ/uYV8jMwTxkvAiYXlM4kT71G5wzb9MYkS64rE1PoLier94UQzxuQ2t9MmRDPfGUSJ6AdkYCSa81ZssI7R4ASX2+hs4sRdq5mcSwTRFRp8fk+inXtt4uQ8Pr/bmUQT4Kjc+Tdr4fuMIErXU4kawmHE4IOf5/6wLSWDdjfpr5p/ryEKA9cVttdBwNu7+y3pCs7rEwFzeO4bdxAFnpH5/hC6GUxAFIZqNdKpwG/patJcjmiCnE/cIK7R+sPoqmW8nxhMMYcIvjfTdRJdJ/+W6oRtss22Ik7SRwI/LPyPWon7LftYLt+SPG6J5sOzyGOIrqa/huv2kJf62s0oIhBfBlxLDrwhmvJ6bCFokNayub9+P/eL2kn7IOpaCBqs+3ZiAMJFFIII0Xy6XJvfbRJxPH6KOK5r322H/LtMmXTflGbVnWGgP3hzCXt9oplnKBHdbyE6ML9Q+EyPI5CI0vGdeeDeTFdJbTRR/f18iTwdRoyoOiV3pkfJ0V75/i703G5dPOh3JILjROKkexLwUaIU+gbwZVofETU9D56Vierzo4X3PpzLVmyw3nJ1n/sc0fQ0nAgmZxHB99g8ONZskMboPOA2yf9/BVFjMKKG9K383Z6jEIBLfq+5RLPC9cSolKOIZo+DaLFUV5fuBkTTybL5/RbRFVgOIJryGhZO6Aoi2+e++Ic8wEcSzWK3EqXqbvdLombwCaIUWQte/03UGmrpj8zf5HtEYKk/uaxL1GKvzHx8gOh0vb3wmeOIJre2Rz9lOtvkMbA2UUO8u/DekXR19HYXeN9NDD8/mujT2i6308WZ7q+Aw1vIT/EcMYW4jfeI/O0WARsWfssHGu233aS1OVEIWokoIN5HVyvIh4gmrbW7WXddooC6DHFMLyCb2nN//TXl+o2KaY7Ovyvk/nEvXUHkMKLW+pbjutQ2rLJDDKYH0fG9K9ERN5Oori9FlNyfBD5bv+EbpPFeojNwfaL28qNMYyhxElmJJk0jRHvrg5nWWXlwnpQHbalhjoWD/ptEMDuKaLP9eu6g2+bnpgLvbnE7HZDpbJ2vdyHamS8n2lXvJUbu1K+3G3BxPp+TO/5OxEnhVqKEN5moUfyI7jvFlydKp/9NtN0eQZTGPlD4zFTihL1hi99tONHcUWtim02cbN/StFYyvdpJehgxqGGz/J5nECec04mTWo/5JILmo8BGRLA7h+jYHkacdO+k+0BUHCwwgegvencuuz63f61zfQQwqod8nE50dB+Wry8mCgN75351b6vbvMH/GJppnZavP07UTk+gK9A3/B9EYaL2XdYjbqX9scL7hxJ9VH8nWhuaDiqpS/8Yon/rS0RBaiOiZva7/E2a9gcW0qr1QZxL1Gp2IM41jxLH0wM9HAMfI2of3ybOD3tknn6W+9ZDZfNRSPNoohD2FaIJ/cPE+eeyzOuvq/y2be8QA/1Bliry+fuJk1OtiWA2ObafKO2dRA+ljPzcAbkTnE6UUm4tvHcYUUUv0wz1ydrOnyeKw3LnmEKWCEqmUzvoD8nXqxEl+d8QHWZl22+H1L0+lBg2fHi+Hknc2viUzOtbqvXA24hgsSERqL9FDLmsvX8R2Sabr3u8Poc4ufwN+GThN7oI2LvwmSpt80OIps0HKh08WSvM5ycA1xdezyBqFG8pWADvqP1u+XoPsm8lX08lBkZ8hggQK3Tz/4ulzXG57S/K36pWer2GCNxNa6VEAWV/ImDsQtQI9yaaFc+gQQGixe1VOx7Xym2/BxHctiEKL6fS/cm1+F1rpeh3EIWyz9R9dldg3RbzNgWYn88vJFouagWFSURQKVsTmUhXH8TXyf48Ioi+kyiIvr2bdKaS5xbieL4on69GBKOdG+1TTb7bPkTz7Qq5vU4hjusJRKA8ihy91fZvW2XlgfoggsjZeQLaMw+mYl/ITKIN+UyitPGOJukdQ5RG7ssD7AxitMOKRM3kfgonlSZpzSRKisWT0O15ELdyAWTxoD+gsPxwotmsx+/UIL0JdLW/ziKa6bYquW6tFjGPqBmdRY7KyveHEE0mtY7jZp2naxNNPY8SwWtkbuergJm9sH/U0is1Gq6bNNYkSnFfpasGcBFwZMn9cwpdI6/WypPGzoXPnEvUCHarbcO6NIonrqNyP/5qnhguIWoStWByJTmqsOR324040W9HnLxOocXSfYM0t8pjpjaAYts8Rt/W6Dv1kM5BRAfx0XkMjCOaiFoaaURd81zu/18gRk3eRFfQ277Zd6cwqIYoHK7Lm/sgamltV3+MN8jHjkTT7zF5TNXWLXV+6SZ/RxMB6ENEoaIWiEuNlCzzGDR3SGzRs0Tn9zuJfoPlgHXNbDV3/6O7X2dmfyUCwbnu/rvuEjKzbYhgNIWoDq5A7MgfIg7Q14EPuvvDJfN2B1HC2c/M7iBKZMsBL3lc6FWKuz8BPGFmLwFfMLP/S8z/83ZiLPjfelrfzDYhAsVZOY3LkcCzZnaBu1+VF25daGZHuPvtTfLyspnNJ/obTiECyg/MbCHRhLgNMXpraH7em6T3B+APOSXHd4i5na4iru1pa06huvT/18wubZaP7pjZaKKPZhuib+KjecHhXUQQ7GndpTymPXnGzB40s2uIk/58YGszexcxAGAC8D9EUL/B66brqeU9LwbciDjhb08MJHiGqJ0cZmbnuft+rXw/d7/BzP5JFJheI/bvv7eSRl6wV9y+/yBKw5ea2deJmpYTJe0Xi9+phzTnEq0JnyYKKyu6++fNbFfgTjP7p8cFn83ythwwx8y+RRzbbyeC8DTixD2h8P92I2qH3aU1AjjOzH5AnG8mAf+HGB23BbC5u79iZocRNYOZxDQk5IWHe5vZlcTvtyJRKN2TOK9s43HR6DHAFmZ2QLNzhJmtDLzf3b+b22URMRL1i8Cf3X3b/NzHgFFm9tn6fasdi10gqe3AZvYG8aNCHKTTgAPzBPKcu99WMskHiPb5f5rZWkSV9Eozu5340a9293+VzZ/HRGjnEdX6WhPOIe7+p7Jp1KX3PTP7F7HzvgbsVyKIGNGpPd1iosp3EDv9nsCOZra8u19qZssAp5vZlh5X3/ZkHlE7OpfoMD6caOudQRxUB3pc7d/Kd/uVxaSTtwGvuftlrazfJO12g8iRxGCE3wJPuPsJFrNFH0vUEMeZ2WnAX+r/R+6br1lMnGhEAPkiEZQuIZpg9ycCwkeJQsthZjai0fa3mNDxHKIp5LcZuPckfs+XiY7av7bzPT0mI7wnny9qZd1iEDGz/Yna2+NETfnnRKFsG+KkujYwo0HgaWQt4mS8I3GC/JKZDXP3Jyzmn2o631X+n7+ZmRP76W/JiUPzd5tpZmcQ/aZziGt5Gm7DTOsfZnYnUXB9gmgiesPMriauy7jAzB4gmsb3qx0DZrYzETiWzv/1R2J49rDcRq8BR5jZq0TteXbJguYrwDQzO5nYB3bM7fYwsMDM/o0YrPBBYP/eCCLAYtu0tR9xxe5GRBXzM/n4IVF9batKR5wUjyZKD3fTwwWLJdMbScUmg0JaY8p8L2KKkFpTzJfye1xbeH820UZc63DttnO2m/QnEgfULKJze1S727uQ5oa02Obdof1qFlGjHEuMJvtW3fvjyeaqJvvQPbntryAGHzwCHFf4zAgiWN1HkwtIiY7TZ4FZ+XoI0fzzH1W3ey9sr48Qndf7Ec2AhxfeW46oWTS9voMYpFEL1o8BNxfeO5KoMZXJT/3orGszvVoT4wrESfbLRJ9Xt81JdWlNI0a6vUxe0Jrfb32is/wYCn0QRKB/JveX7XKf+jVdQ+PH5b52JlHzatqxXpef2cRlAPX9bp8iOvGvpWJ/11v+f3/uaB3cgT8HfDyfDyNKd7UmhBtocww8Mbb/H3mAVxq90o/bZjwxfPYSoqS/HzHC6iOFzxySO/AKbf6PjSl02C8Ojzwx7E4EyrnkiL18r9SEmkQJ9Nt5UtwDuK/wmzzDmycE/CjlZzSYTtSOi8GkpQJAL22jdxSer5D70NDi9srjscf9qu6kWJzyZyIxfPnofO+DRItBqyMTd6CrY/3jFEZB0WQmigZp7Qz8JJ/vSNQAa/1aW9Fg+pTcDl8jRl3+IveLjxGF39qUJRvVfssW8/PvRM12PWLk4zcL743Lvz1eKN3Wb9/XO1tfPIgq83UUIjlRXTyOEld395DuyplupZpIfz/oGvFVG5m1U36vYwqfqXQiImoR3V45PZgexBDkjxLNdX/lzSP2PkxcANd0oAQxRPzreRK5k655kzbI3+DfK+RxJ+Laij37YfsYUft8hMJ1VBlI7iJnHc5lh1ByGhrePOXPBfl8JjEqcD4xHLZpgY7oh6kF/YNy3c0L73+SqBGcTDQz9XhBct02/zU5gKCQv38Qw2wfpDDSi6gF1S62PJMYplzcXp8mgtqJRGGvaYGXrpFlS+fvcBs5+IboJ7uOGLq/N9Gn29JM3aX3gb7e6fpox16RGEr4BWJ0yHSiJFNqSGyTtFu+6nOgPXjziK99ctlEonQ0u7/zN5AexHDou+maVuXLuS+tRQSYHuerapDeUUR7de0q4vcT7fS1k2aVYc070OJovV7aRrVrWd5FjG78bL7elRiRWJukcHaeKJsWMOhhyp98f3WaFHbyxLoa0VRbG/o/mejIPqXus/sSzYGlh8HSdTuEb9Qtfz/RjPSuwrLabA3rEE2jmxH9Jl+lcP0YMUrxSlocpUVeUEsUbPYtLF+WuFbktlb205b3gb7e6frqQY5eyg34I3q5TXBxeOSBfj/RfjqDGGHV1lxTi+ODN0+rMpqokXyBKPlfTgwwaPXCsFUzjVuImuGjwPT+/q4VtlGxGWponigfz2NvFNGHMZ9oUr6vzPail6f8Ifoit6SrpL4pccuHg+o+V6oZiejw/0Q+34AogNUHpuLcafWzNVxK3seFGBV2AW9uDWh6gSzN76szkagB1aZj6ZW+2O4ei/2tdi3uvWHeZCTTksrMphHThfwdONjjHh2ScgjoYcQJ7TFipM9aRDPIv7y929UuS4woXAl4xt1/WXLU0oBlZkcRNf+biWCyK3C2u5+Rw1xXA170BiPAcuj0GHd/yMx2JJpFf0xs80OJTuxXiRFq/0lMjdPjaCOru0Vsjh47AjjD3edZ3Ib3BiIAnNvid51MBIPz3P0cM1ufmL14gbt/rMHnlyf6K14mRmP9lBjRdb3HqMudif6eu9z93DL7Qg7tvYcYzfhCjih9g2i6+yyxnTYngsrxHT//9WXpRY+B+SBGcvXrCJ+B+uCt06rsRzTXdKSteTA+iH6BnxCB8U6iWWddoiP81BLrj6dzU/5sQNeFttOJAFVrzt2S6BNZgXIXQ7Z9OwQaz9ZwMVkbJTrqexzxl5/rs/vqtLSd+3sn1EOPwfCgl6ZVWRwfxPUWu9N15XStP2IXYpBLmdsX9NaUP5uQIxCJGshDmac9ctnO5H1B8nWp2SRo83YIhfXX5q2zNRxENI82nJG5QRp9dl+dVh+L3QWJIh0ynGg62NvdH+nvzAwwTxI3ZXvW3bcCMLPjiM7u7bzcVfEXEO37x+UV6pcTF8weTvQHlL2FbJkLbYcRt8S+lqglNErHvNA05u6/MbPvEcFgTzO7xt0X5FXpu5rZOd7DBbfe/WwN/yL6KZvyWjQxO5QIQjPc/Rkze5moxf2LqC1fRQwC6DMKJCIleMVpVRZz9xDzx71hZlsTfUj7EaX+UlOrePUpf1Yhbnx2i5ltS1zot9DdXwIuMrNXiHu2D3f3C8zslh7ytmzt/5nZh4kr80cSw3PJtNc2s0XEoIC9egoidd+z0mwNOSXLTkTz4T9z6pWxxMSpJxGFna97G313VSz2ne0i0nlZA9gtHy8C/+nuD7SZ1jTePOVP0xqgmY0nJoVcSDQjXUxc+3O5u5+VnzmEmO3iM92d+HPushnufrCZzSFmsvgM0ay5ItGEtx7RRLY50UfS8gAVM9sQeCUDaKvr1gaAPE00/f2BCCRPE9c4PdVqmlUpkIhIr7GY7BNvYf65btIZk+mUnufLzE4nrqL/D3c/38x2IkZ93e7uZ+ZnRnn3c2e9jWh2OpYIhl8hRlZ9J9+/iJhFeVq+Hu4tTLTaW8xsONG5/1t3/7OZzSaauqa5+z/7Oj+gpi0R6UVVA0ghnZYmikzFfpY/u/t3zOwF4Dwze9Hdr+guiKR/ErWgk4iZiZ8nmtZq5gLfNLPlsunr1TbyWFkGr1+a2RAzO5gIfPv2VxABBRIRWUzU9bOcmn+HEwHiZyXW79XbIfSBATMARE1bIrLYafdCWzNbm7iu5Vyib+SPxISKfyUutDzU3R/sSKbbMFAuZFUgEZHFUo7k8naaycxsItFf8mlimpxhxIWN7TS5LfYUSEREGjCzjYlhup929/P7Oz8DmQKJiEg3cpjuP9z9t/2dl4FMgURERCoZ0t8ZEBGRwU2BREREKlEgERGRShRIRESkEgUSERGpRIFEpJeZ2bFmNrJDaW9tZjd2Im2RdimQiPS+Y4n7V/TIzIZ2PisinadAItKEmR1nZg/m49hcNs7MHjWzy8zsfjO72sxGmtlHiBljbzez2xuk9aSZnWRmdwJ7mdmHzeyXZvZrM7umVpMxs0vN7Cwz+x8z+13eDKk+rfeZ2b1m9o7ObgGRnimQiPQg51w6kLjd6xTgw2a2Sb79buBCd9+ImNTviLyJ0rPANu6+TTfJvuLuW7r7VcC17v4+d98YeIS4gVLN6sCWxL3PT6vL1+bEtOkz3P13vfFdRdqlQCLSsy2B77r73/MeFNcCW+V7T7t7bXryK/KzZXyn8HxDM/upmT1A3J52QuG969z9DXd/GFi1sHx94EJg1/64G55IPQUSkZ5ZD+/Vzy9Udr6h4r3CLyXuSf4e4h4YwwvvFW+cVMzHc8ArwCaIDAAKJCI9+wkwM/s/lgU+QNzkCGAtM9ssn+8L3JnPXwaWL5n+8sBzeYva/Uqu8xIwHfiimW1dch2RjlEgEemBu/+KqDXcDfwCuMjd7823HwHmmNn9wMpAbarxC4GbGnW2N/CZTPcW4NEW8vU8sCtwrpltWnY9kU7Q7L8ibTCzccCN7r5hf+dFpL+pRiIiIpWoRiIiIpWoRiIiIpUokIiISCUKJCIiUokCiYiIVKJAIiIilSiQiIhIJf8PnMyGdvSfMGMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.xticks(rotation=45)\n",
    "sns.countplot(x='opt rank',\n",
    "             data=data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9cafc348",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class=draw, n=2796 (9.966%)\n",
      "Class=zero, n=27 (0.096%)\n",
      "Class=one, n=78 (0.278%)\n",
      "Class=two, n=246 (0.877%)\n",
      "Class=three, n=81 (0.289%)\n",
      "Class=four, n=198 (0.706%)\n",
      "Class=five, n=471 (1.679%)\n",
      "Class=six, n=592 (2.110%)\n",
      "Class=seven, n=683 (2.434%)\n",
      "Class=eight, n=1433 (5.108%)\n",
      "Class=nine, n=1712 (6.102%)\n",
      "Class=ten, n=1985 (7.075%)\n",
      "Class=eleven, n=2854 (10.173%)\n",
      "Class=twelve, n=3597 (12.821%)\n",
      "Class=thirteen, n=4194 (14.949%)\n",
      "Class=fourteen, n=4553 (16.228%)\n",
      "Class=fifteen, n=2166 (7.720%)\n",
      "Class=sixteen, n=390 (1.390%)\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "counter = Counter(data['opt rank'])\n",
    "for k,v in counter.items():\n",
    "\tper = v / len(data['opt rank']) * 100\n",
    "\tprint('Class=%s, n=%d (%.3f%%)' % (k, v, per))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb6dd3f2",
   "metadata": {},
   "source": [
    "Tal y como vemos, se trata de un problema de clasificación desbalanceada. Existe una gran tendencia en la distribución a necesitar un número de entre diez y catorce pasos para acabar la partida. Destacamos, de igual manera, la importante cantidad de veces que acaba en tablas. Hay muy poca representanción de partidas que puedan acabar en el rango de uno a siete movimientos.\n",
    "\n",
    "Este tipo de conjuntos de datos son problemáticos ya que el gran desbalanceo existente entre las clases impide que los modelos construidos consigan buenas métricas en la clasificación. Por ello, usaremos, además de los datos _raw_, el algoritmo SMOTE, que realiza un sobremuestreo de los datos. Su funcionamiento es parecido a la interpolación, pero mucho más complejo. Esta técnica funciona mejor en problemas binarios con predictores continuos, donde la interpolación es más natural. Sin embargo, como también acepta datos categóricas, vemos interesante su uso y ver qué resultados ofrece."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dac494ea",
   "metadata": {},
   "source": [
    "Continuamos con la distribución de las demás clases en cada variable predictora:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "01b3c061",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjEwIDAgb2JqCjw8IC9Bbm5vdHMgWyBdIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDg1Ni4zOTM3NSA4NTUuMjY4NzUgXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMSAwIFIgPj4Kc3RyZWFtCnicxVzLkhy3Ebz3V/TRPhjE+3GUwrYifJPNCJ/F18o0Vw6vbOv3nYWeGVShH5gZ7VCMIGMn2YPsagCFTKB6zfx5evONmZ9+nvX8GX9/mc383fzmjx//94/3H//63bfz+58nDfx5yiEqV1wK+PSFfcohKBszfvqCK/mnH6fppwlt4wvfodmnaQpFxfqfPnplPH39ebJZn5pq6BeGZmdVOaGXBjgInk/Tv+eNxq3VKtkwu5hVnl8+zn+ff5q1MsVHnbJ3Bh98cPjB4bJJq+hsrp/c/PK0f+XcXTlNMSiTV6EZXZRzJnexMTiYrHyJwXjgrREB1/i+n2+JMGc0ZJMPdN9BB1esCbEgQpOcrp/SEuHOlXN35TQZMIHF2C5E/ByMk/AXDkeTVCkZLRPeWhH4HTFaW7wxKfpEd550yNm64CbxPzXGnSvn7krEmIIKvhuLGKEGN7WMNjFEG5xMxI/aFEsxtlYEfkeMiQ856zJ+8EVbitGbWP+cYty5cu6unCbrcLHTsetHDAGFW4pdPzI4m6AcRouveGtF4JcY33xjl6zyhKn/mQYacgslAiBTTsqG1WPmKCbY6Q6mb5FDfpm+fTu/+bOZjZ7ffpoyvlSWJ+iLCmF++2H63Q+/n99+nv/0dqpck/FWeb+KU8DHLMaVUy5jLO86lpJUtGY18Tk8YCn0hE1H817S2GBV1q6ffAI+prFoOy5XNpYPZxZ5R2jK5rI8Ixdwd3W+0hd++Wd3Y5sZHWuDiX3S27opj0VEX3rc+fol4tE7LMFapTEmTeY0DN0JHiOrxk55FUM3eBtdyZXKar3LlooqxugsgmLoiC3hOxhwBjdXlsD8Plt0SaXiMZE5G0MHbNE55NOUUyqmuMoWD9hKUBErmzWCraEjtmJo7Qg+hFBMZcv7bLh9hSSis+g3hu6w6fPUSBi0KQaLJaXkpeOM3ufLxmCWByQuzsfQEV/WmMA+56yzy3Hh4yNFfE0jxf0B6+dsPPrK1V7zCmOs5s23z1VB1Vn9r//+9J9zE3b+yyK6amqUkmt7Wm2Jn+lvmwLqeUdA4fIbRBi/urVy0LauMS0p39SE/3R5VLbmlpCVC2ZZUFw+fbdLLSdFukjHRZVeJKTzXumNjHN+8vz2G8pvs7WwEpG0cG0RfD0d6Sg5r4UkHjaW25WQZLBYhVkr26vzDUE+QEp6jX5ZS0nvMR7WUpLBSWtVoo+u4q0Vgd8T5AO0pMfMKWstGXDXeq0lGRx1VF5bCDYKsrUi8HuCfICYxExWZi0msSQouxaTDA5IxRkyrFAwrBWBXycmXTIqrtWkgI+1ET0nn1aC0sjFBfm83lo/ajl8zOOtQzI9CZ7GYzueTGpmJSkFPOBJ6Hdve03pusUSysKuNaWAj3kwSnBLznfx+M1FEgsvEtFybS8qX+Sdbed45zHhrpKVjszJNbqyEQXyckhZJgomBu9QYbzGk7REUDqTt0Ai2pGWjBAmUWNlzUYSNnhIiAFukfkw9IqNO+qyEUY8lIC1BumJEzJ4RBgx0rHagLe4XHYEJiPMAXpUa+MkYYOHhBjy4EIiCjnYHY3ZCGEAsLbi/rIgZPAOoQ54kIvMxOCPwSftLPX5jsxklCXWtcl7SdngMSXWcGPwbDMp92uVpi2FHs0rSM29ybYlkkgPbiit5x2lhctvkWvici43d1sfyk3v4TBOmb2Xmy/XyM1osTSs7j/789Pn999QfputhU25uUXwFbctHXKIRSp2Ir4IY+Y0FgEnAmRwKhl6xRRDaYO1IvB7gnyA3IzwHKSAXZZBwsTkgmUiyyAbDNNOekXXfQXWisDvCfIBcjNmUodFR9mTSeOmMvktESSDE4w8FpCga0+2VgR+T5APkJvJ0BYHprPsyeSi0gl6R/Ykg6vzJ4dAPcZaEfg9QYaU263786zzNCcL7fsSsAS5c+XcXYkgsV6FQgpMBhlJPNpsup5sMP4i3mIXvLUi8LuCNAlDhSYbPriQ3fIHQTobSqa5Zk9Bbl85d1ciyIhcmbvJhyBp0NFo6HqywUgytPFct3BYIxy+K+24NujC0hnoHUSY/Klv/CnrbF44dxciQEiARA9DzkdkEFU1osysDBaGnbWybeRvCNEvz5+y4lP3cZqy9crFbgrRQod8UPVzt9Jd4KyN0jqUCrdGOHydjYveKht9n8AEfGxHoitYe1a2p9uxjxg0Ofg+hwh4wBOTMtr3drHbGY8l0WZjP40FPOApHsvMyl59kDzoaaVdWM0kDh/zIEWqEhY/xng+djzBqkg6s1tgODzgCWipf2qfOhasvhZJpM/wHB6wpKR0fcCC50ny0P4u/vbCR8CDMyjtoaZCb+Z/7HhgdV1ZaQ8BD3igGt2a54dNf4AUqPS5KzuX/dIP0E1pGzGHy1UuO9KuzTUuuxEFrEQJ34leMDF4h0rjGSwHUkHTbqDXudC6XelC788YYSSPnA2MlSBs8M7OBvLusgcSQiKlRIKy2HbwsEcYNdIIRl9xgpDBI0La5tM+B2T6fDrGMQcRkiz1xkadJWGDh4QBAxnLgLMlYzHY3rhohAn3VyDzsZBzQgaPCOGusWxDviaf7XJ0ZA8iTIGO0/GlJAkbPCQMRpFcj6akfNoLO4iQZjYd2wUjCBk8Iswa8wEuhNS5Xzy9CyNLH3JBGj4d5f06T783tbecKGz0lp193rGzuPwWTywu555+t/Whp09ozy4biZ2lf7nqBOlyouUSBlB/oHUBeQkS5Idy503sy/EZA9elHZd2vqKZtxg0de1ikWUPb1wtO4uMg5i2kM3R1a2K1gCDbw/uASa+YGFb3DcLzpCzWbw6i46jGerRl1JFM2uCw7eH94jKI5NUXIw3jw+KNuYq/3l8DM1Yg5Gta9blbXD89gAfYN2Nxx0tppsHCCltF4vOA2RopiIxTSmXt9DQ24N7gGU3CQKmyFn2vFT2kDWXvcfQnFU6WzreBsfvCPD17bqlBS3LiYasae2SkETvcRT9hIV2KczhbXD8jvTy2mbdWoynJOcZVVchYdRaMREeQzHLsMyXUGrYrQ2O3x7eoVG3JHuinDFUvYep73yU84ij2ZFbWQJoLTT0OpOeHJZy32UpBno8mgNrps8LN4ZI2TxkNbRvEHyfJhh6TAF3nUX73eGqgXiI5A7lTGXocftQZwi2CrPG0R2sYgwgFYZusnB0wBEcopUUvqNAx+XFdXMKhg4oMpnOKB9V6GoOaW908dx8pDH0mMNqqrUTDLFjoA0b3SsWjg4YyOHa6nAbR+o4YlSh9MKBowOOiNFt5IPKO2WTRp2qJkt1lxfj3R1vbwnQS40am77HFZPOIYsNCyYp0VrKeqIO7gLunZ/T1vFSokXlklhzkfDMttm+UBmHZ2pI03Muhg4qCo2jXYCgI8aWdjtOu7HVY+SoRWANHHHVk+ViQnBW+x2TfeGiceazNtZzMoYO2PAvBJuzPqeow6jqFCmSipmsEc+RoSO2bPCd5DyWF5127HUbg2Q1fXZG1IEydMBGtRb0noILWALyjrdubBhQBQ9CiyfJ0BEb8pbBgmWLzuZKX23xOBZbbWl37zWqMvnE3bKDoihz02Z2V1/tSUX95qWNg5bHFZm0d1f3C2GbHDPTV52PtwN7kczO5/XsztmZP7tDVh+wZac3mv+atZgRw4USOQ+OeIuuB2ssOoZaU/2BL77Gd2lD4HdE+ABT7ZKnxQ2sIsISITpplRERNhQzte6EOTptY20I/I4IH1GFqS3ERO+rvcUtrXw1Qy2WSF9i1LYWml7aEPgdET7AWHtbsFr3xhp9r+LKWHOUXh2Bki1L5K0Njt8R4QPctQ8Jk7Z31z7R2zk2y3koUMwEjMbahbyJBt8T3+uba4+lPa3MNYSQyitzzVG+S8Db2Nw9uCHPvLa7puMdl2qxCY+vSozeXTOU3ljSeE6eNglYGwK/I75De41cCZnV22vQKr2y1wy1kMwGmtnWcubWhsCvLGfGUM1RPpdngR4bFqgjSsJa+sfu7I92KHyoB/ichKEDEtpAXVmv7mTaG7ecxMp5y9BjEq/pOabOfHXH0hgryzGsTH8MHZC4fD4lbRzdoTSeCNaV2A1gjg44Yli2L0Qg3Ym0LzCzK7vN0QFJscsehoikO44OFm525bg5ekwS6PXdFUn35mUIml7s7Sw3Rwckvsn0RvJuU+VTVXk8nUIL1/2uf1NxU6m2kt2B72Yl5YfGu7HQS94FAzBYwcPgUfmzQa5HHi1UAOTMjiFmhJnebHUlyFpkBg8JE22DJQ2ZDhc+LGFH8le5eB1kzTyDR4QWekQXjG7jQ7E75pF1HHwgPiGhyq5r8IjQYaRjeNnsorFuWDPvSqGXL22URfoMHhIW2pFI6O8Uw6VcYeBYqTo8vI5l3Rn1W6ZLFHdvurnu6uu9nywEZ7Z1t+2rKrvzucxHONd3171I2I6lxWnp+VRanJZeTrbZTbJT8C3nutH8b1LWzYJj5dssOoF6iGeTSq135m0w/I4IH1vTzSNstds8Qoa6Wkiqa/blbXD8jggfW9DNz1pa4TaLkKMBucmbtETI2uD4HRE+tppbnCZdqrZ5hAyFSo0h2dTVcgv8jggfW8rNI2wl2zxChnIHx9vYdHY3RPjQOm4eYavX5hEylL5GvzzAzaKMW+D3ZJoHlnHzZaKVa/N1gqF0TmO9r/V+vA2O3xHftTXcYkm71GqLNa2hpIkcchDdE2+D4zfXcPOszNBj5V8ruAcnxKxQm3MwdMARo4qCoDsiZhXanIChAwIq2x4cEbPqbD5tGDo4SbcG4R4eEbPCbE7B0AEFVI8fHBGzsmzOwdABB9VqHx4Rs4JsPqYZesxQq7QHR8SsGJtzMHTA4TC+rzoiTtVFrk6I3/UvQG9q01aJOvCqrDD70KvywkykEodhIQteL+iwKjvRb8LC7NWOJOigKNtYGlkUv6Bj8Ki+lqowIiwjPCa04rAom34hF61YJUrCBg8JEy3CgX6VjzF5WJRtMT2dh62VETJ4RIj5gz7G44/Olz0rzgipYq4YOGFJ2OAhIdU5FGQ4g2XfD4uyIX8g55CrZRU4g0eETpe6LxGTpgOjUVE26VfawQ8yQgYPCQN1t/PQ21Gvi7JXL0Vo+qUUJSdZyc/gESGUBLrbwv7Hku1qs2FYBf5rvf9OFtmyraIIfNMPd1df755lwTjz/rttX1EBDplwfjGq8/6XTPr99H8aMcdECmVuZHN0cmVhbQplbmRvYmoKMTEgMCBvYmoKMzg5NgplbmRvYmoKMTYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDQgPj4Kc3RyZWFtCnicPZI7ksMwDEN7nYIXyIz4k+TzZCeV9/7tPjLJVoBJiQAoL3WZsqY8IGkmCf/R4eFiO+V32J7NzMC1RC8TyynPoSvE3EX5spmNurI6xarDMJ1b9Kici4ZNk5rnKksZtwuew7WJ55Z9xA83NKgHdY1Lwg3d1WhZCs1wdf87vUfZdzU8F5tU6tQXjxdRFeb5IU+ih+lK4nw8KCFcezBGFhLkU9FAjrNcrfJeQvYOtxqywkFqSeezJzzYdXpPLm4XzRAPZLlU+E5R7O3QM77sSgk9ErbhWO59O5qx6RqbOOx+70bWyoyuaCF+yFcn6yVg3FMmRRJkTrZYbovVnu6hKKZzhnMZIOrZioZS5mJXq38MO28sL9ksyJTMCzJGp02eOHjIfo2a9HmV53j9AWzzczsKZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIzNyA+PgpzdHJlYW0KeJxFUUlyBCEMu/cr9IGpwivwnk7NqfP/aywzSU4WYGsxaYGBLXiJIdbAzIEvuXxN6DR8NzLb8DrZHnBPuC7cl8uCZ8KWwFdUl3e9L13ZSH13h6p+ZmR7s0jNkJWVOvVCNCbYIRE9IzLJVixzg6QprVLlvihbgC7qlbZOO42SoCMU4W+UI+HpFUp2TWwaq9Q6oKEIy7YuiDqZJKJ2YXFq8ZYhIp91YzXH+ItOInbH4/6sMOtRJJLSZwfdcSajTZZdAzm5eaqwVio5iD5e0caE6nSqgWO817b0E2ngufZf4Qc+ff+PGPq53j/G7lwiCmVuZHN0cmVhbQplbmRvYmoKMTggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzAgPj4Kc3RyZWFtCnicNVFJbsMwDLzrFfOBAOIuv8dBT+3/rx3SCWBgaEuczREbGxF4icHPQeTGW9aMmvibyV3xuzwVHgm3gidRBF6Ge9kJLm8Yl/04zHzwXlo5kxpPMiAX2fTwRMhgl0DowOwa1GGbaSf6hoTPjkg1G1lOX0vQS6sQKE/ZfqcLSrSt6s/tsy607WtPONntqSeVTyCeW7ICl41XTBZjGfRE5S7F9EGqs4WehPKifA6y+aghEl2inIEnBgejQDuw57afiVeFoHV1n7aNoRopHU//NjQ1SSLkEyWc2dK4W/j+nnv9/AOmVFOfCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMjcgPj4Kc3RyZWFtCnicNU87sgMhDOs5hS6QGYxtYM+zmVQv92+fZLINEv5I8vRERyZe5sgIrNnxthYZiBn4FlPxrz3tw4TqPbiHCOXiQphhJJw167ibp+PFv13lM9bBuw2+YpYXBLYwk/WVxZnLdsFYGidxTrIbY9dEbGNd6+kU1hFMKAMhne0wJcgcFSl9sqOMOTpO5InnYqrFLr/vYX3BpjGiwhxXBU/QZFCWPe8moB0X9N/Vjd9JNIteAjKRYGGdJObOWU741WtHx1GLIjEnpBnkMhHSnK5iCqEJxTo7CioVBZfqc8rdPv9oXVtNCmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDUgPj4Kc3RyZWFtCnicRVC7jUMxDOs9BRcIYP0se553SJXbvz1KRnCFIVo/kloSmIjASwyxlG/iR0ZBPQu/F4XiM8TPF4VBzoSkQJz1GRCZeIbaRm7odnDOvMMzjDkCF8VacKbTmfZc2OScBycQzm2U8YxCuklUFXFUn3FM8aqyz43XgaW1bLPTkewhjYRLSSUml35TKv+0KVsq6NpFE7BI5IGTTTThLD9DkmLMoJRR9zC1jvRxspFHddDJ2Zw5LZnZ7qftTHwPWCaZUeUpnecyPiep81xOfe6zHdHkoqVV+5z93pGW8iK126HV6VclUZmN1aeQuDz/jJ/x/gOOoFk+CmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzOTIgPj4Kc3RyZWFtCnicPVJLbgUxCNvPKbhApfBNcp6p3u7df1ubzFSqCi8DtjGUlwypJT/qkogzTH71cl3iUfK9bGpn5iHuLjam+FhyX7qG2HLRmmKxTxzJL8i0VFihVt2jQ/GFKBMPAC3ggQXhvhz/8ReowdewhXLDe2QCYErUbkDGQ9EZSFlBEWH7kRXopFCvbOHvKCBX1KyFoXRiiA2WACm+qw2JmKjZoIeElZKqHdLxjKTwW8FdiWFQW1vbBHhm0BDZ3pGNETPt0RlxWRFrPz3po1EytVEZD01nfPHdMlLz0RXopNLI3cpDZ89CJ2Ak5kmY53Aj4Z7bQQsx9HGvlk9s95gpVpHwBTvKAQO9/d6Sjc974CyMXNvsTCfw0WmnHBOtvh5i/YM/bEubXMcrh0UUqLwoCH7XQRNxfFjF92SjRHe0AdYjE9VoJRAMEsLO7TDyeMZ52d4VtOb0RGijRB7UjhE9KLLF5ZwVsKf8rM2xHJ4PJntvtI+UzMyohBXUdnqots9jHdR3nvv6/AEuAKEZCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMzMgPj4Kc3RyZWFtCnicTY9BEsMwCAPvfoWegLEB8550ekr+fy2QNu4F7YyAkYYwCDxiDOswJbx6++FVpEtwNo75JRlFPAhqC9wXVAVHY4qd+Njdoeyl4ukUTYvrEXPTtKR0N1Eqbb2dyPjAfZ/eH1W2JJ2CHlvqhC7RJPJFAnPYVDDP6sZLS4+n7dneH2Y+M9cKZW5kc3RyZWFtCmVuZG9iagoyMyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0NyA+PgpzdHJlYW0KeJxNUbttRDEM698UXOAA62t5ngtSXfZvQ8kIkMIgoS8ppyUW9sZLDOEHWw++5JFVQ38ePzHsMyw9yeTUP+a5yVQUvhWqm5hQF2Lh/WgEvBZ0LyIrygffj2UMc8734KMQl2AmNGCsb0kmF9W8M2TCiaGOw0GbVBh3TRQsrhXNM8jtVjeyOrMgbHglE+LGAEQE2ReQzWCjjLGVkMVyHqgKkgVaYNfpG1GLgiuU1gl0otbEuszgq+f2djdDL/LgqLp4fQzrS7DC6KV7LHyuQh/M9Ew7d0kjvfCmExFmDwVSmZ2RlTo9Yn23QP+fZSv4+8nP8/0LFShcKgplbmRzdHJlYW0KZW5kb2JqCjI0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggOTAgPj4Kc3RyZWFtCnicTY1BEsAgCAPvvCJPUETQ/3R60v9fq9QOvcBOAokWRYL0NWpLMO64MhVrUCmYlJfAVTBcC9ruosr+MklMnYbTe7cDg7LxcYPSSfv2cXoAq/16Bt0P0hwiWAplbmRzdHJlYW0KZW5kb2JqCjI1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzM4ID4+CnN0cmVhbQp4nEVSS3LFMAjb5xRcIDPmZ+PzvE5X6f23lXA63Tz0DAgJMj1lSKbcNpZkhOQc8qVXZIjVkJ9GjkTEEN8pocCu8rm8lsRcyG6JSvGhHT+XpTcyza7QqrdHpzaLRjUrI+cgQ4R6VujM7lHbZMPrdiHpOlMWh3As/0MFspR1yimUBG1B39gj6G8WPBHcBrPmcrO5TG71v+5bC57XOluxbQdACZZz3mAGAMTDCdoAxNza3hYpKB9VuopJwq3yXCc7ULbQqnS8N4AZBxg5YMOSrQ7XaG8Awz4P9KJGxfYVoKgsIP7O2WbB3jHJSLAn5gZOPXE6xZFwSTjGAkCKreIUuvEd2OIvF66ImvAJdTplTbzCntrix0KTCO9ScQLwIhtuXR1FtWxP5wm0PyqSM2KkHsTRCZHUks4RFJcG9dAa+7iJGa+NxOaevt0/wjmf6/sXFriD4AplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTYzID4+CnN0cmVhbQp4nEWQuXUEMQxDc1WBEniAOuoZP0ez/acLabzeQPp4hHiIPQnDcl3FhdENP962zDS8jjLcjfVlxviosUBO0AcYIhNXo0n17YozVOnh1WKuo6JcLzoiEsyS46tAI3w6ssdDW9uZfjqvf+wh7xP/KirnbmEBLqruQPlSH/HUj9lR6pqhjyorax5q2r8IuyKUtn1cTmWcunsHtMJnK1f7fQOo5zqACmVuZHN0cmVhbQplbmRvYmoKMjcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MSA+PgpzdHJlYW0KeJw9zLsVgDAIBdA+U7wRQnyA7OOx0v1bwUQbuHzVAx0hGdQNbh2HtKxLd5N96nq1iaTIgNJTalwaToyoaX2pfWrguxvmS9WJP83P5wOHxxlrCmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjEgPj4Kc3RyZWFtCnicRZBLEsMgDEP3nEJH8EcGfJ50ukrvv60hTbOAp7FABncnBKm1BRPRBS9tS7oLPlsJzsZ46DZuNRLkBHWAVqTjaJRSfbnFaZV08Wg2cysLrRMdZg56lKMZoBA6Fd7touRypu7O+Udw9V/1R7HunM3EwGTlDoRm9SnufJsdUV3dZH/SY27Wa38V9qqwtKyl5YTbzl0zoATuqRzt/QWpczqECmVuZHN0cmVhbQplbmRvYmoKMjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTQgPj4Kc3RyZWFtCnicPVC7EUMxCOs9BQvkznztN8/Lpcv+bSScpEI2QhKUmkzJlIc6ypKsKU8dPktih7yH5W5kNiUqRS+TsCX30ArxfYnmFPfd1ZazQzSXaDl+CzMqqhsd00s2mnAqE7qg3MMz+g1tdANWhx6xWyDQpGDXtiByxw8YDMGZE4siDEpNBv+tcvdS3O89HG+iiJR08K755fTLzy28Tj2ORLq9+YprcaY6CkRwRmryinRhxbLIQ6TVBDU9A2u1AK7eevk3aEd0GYDsE4njNKUcQ//WuMfrA4eKUvQKZW5kc3RyZWFtCmVuZG9iagozMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDgwID4+CnN0cmVhbQp4nEWMuw3AMAhEe6ZgBH4mZp8olbN/GyBK3HBPunu4OhIyU95hhocEngwshlPxBpmjYDW4RlKNneyjsG5fdYHmelOr9fcHKk92dnE9zcsZ9AplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTU3ID4+CnN0cmVhbQp4nEWQuRFDMQhEc1VBCRKwCOqxx9F3/6kX+Uq0bwAth68lU6ofJyKm3Ndo9DB5Dp9NJVYs2Ca2kxpyGxZBSjGYeE4xq6O3oZmH1Ou4qKq4dWaV02nLysV/82hXM5M9wjXqJ/BN6PifPLSp6FugrwuUfUC1OJ1JUDF9r2KBo5x2fyKcGOA+GUeZKSNxYm4K7PcZAGa+V7jG4wXdATd5CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA2OCA+PgpzdHJlYW0KeJwzMzZTMFCwMAISpqaGCuZGlgophlxAPoiVywUTywGzzCzMgSwjC5CWHC5DC2MwbWJspGBmYgZkWSAxILrSAHL4EpEKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMxNyA+PgpzdHJlYW0KeJw1UktyQzEI279TcIHOmL99nnSyau6/rYQnK7AtQEIuL1nSS37UJdulw+RXH/clsUI+j+2azFLF9xazFM8tr0fPEbctCgRREz34MicVItTP1Og6eGGXPgOvEE4pFngHkwAGr+FfeJROg8A7GzLeEZORGhAkwZpLi01IlD1J/Cvl9aSVNHR+Jitz+XtyqRRqo8kIFSBYudgHpCspHiQTPYlIsnK9N1aI3pBXksdnJSYZEN0msU20wOPclbSEmZhCBeZYgNV0s7r6HExY47CE8SphFtWDTZ41qYRmtI5jZMN498JMiYWGwxJQm32VCaqXj9PcCSOmR0127cKyWzbvIUSj+TMslMHHKCQBh05jJArSsIARgTm9sIq95gs5FsCIZZ2aLAxtaCW7eo6FwNCcs6Vhxtee1/P+B0Vbe6MKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzMSA+PgpzdHJlYW0KeJxFj8sNBCEMQ+9U4RLyGT6ph9We2P6v6zCaQUL4QSI78TAIrPPyNtDF8NGiwzf+NtWrY5UsH7p6UlYP6ZCHvPIVUGkwUcSFWUwdQ2HOmMrIljK3G+G2TYOsbJVUrYN2PAYPtqdlqwh+qW1h6izxDMJVXrjHDT+QS613vVW+f0JTMJcKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMzOCA+PgpzdHJlYW0KeJw1Ujmu3UAM630KXSCAds2c5wWpfu7fhpRfCkO0VoqajhaVafllIVUtky6/7UltiRvy98kKiROSVyXapQyRUPk8hVS/Z8u8vtacESBLlQqTk5LHJQv+DJfeLhznY2s/jyN3PXpgVYyEEgHLFBOja1k6u8Oajfw8pgE/4hFyrli3HGMVSA26cdoV70PzecgaIGaYlooKXVaJFn5B8aBHrX33WFRYINHtHElwjI1QkYB2gdpIDDmzFruoL/pZlJgJdO2LIu6iwBJJzJxiXTr6Dz50LKi/NuPLr45K+kgra0zad6NJacwik66XRW83b309uEDzLsp/Xs0gQVPWKGl80KqdYyiaGWWFdxyaDDTHHIfMEzyHMxKU9H0ofl9LJrookT8ODaF/Xx6jjJwGbwFz0Z+2igMX8dlhrxxghdLFmuR9QCoTemD6/9f4ef78Axy2gFQKZW5kc3RyZWFtCmVuZG9iagozNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0OCA+PgpzdHJlYW0KeJwtUTmSA0EIy+cVekJz0++xy5H3/+kKygGDhkMgOi1xUMZPEJYr3vLIVbTh75kYwXfBod/KdRsWORAVSNIYVE2oXbwevQd2HGYC86Q1LIMZ6wM/Ywo3enF4TMbZ7XUZNQR712tPZlAyKxdxycQFU3XYyJnDT6aMC+1czw3IuRHWZRikm5XGjIQjTSFSSKHqJqkzQZAEo6tRo40cxX7pyyOdYVUjagz7XEvb13MTzho0OxarPDmlR1ecy8nFCysH/bzNwEVUGqs8EBJwv9tD/Zzs5Dfe0rmzxfT4XnOyvDAVWPHmtRuQTbX4Ny/i+D3j6/n8A6ilWxYKZW5kc3RyZWFtCmVuZG9iagozNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE3MSA+PgpzdHJlYW0KeJxNkE0OQiEQg/ecohcwofMDj/NoXOn9t3bw+eKC9EshQ6fDAx1H4kZHhs7oeLDJMQ68CzImXo3zn4zrJI4J6hVtwbq0O+7NLDEnLBMjYGuU3JtHFPjhmAtBguzywxcYRKRrmG81n3WTfn67013UpXX30yMKnMiOUAwbcAXY0z0O3BLO75omv1QpGZs4lA9UF5Gy2QmFqKVil1NVaIziVj3vi17t+QHB9jv7CmVuZHN0cmVhbQplbmRvYmoKMzggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4OCA+PgpzdHJlYW0KeJw1jLsRwDAIQ3tPwQgGi4/3yaVK9m+D7dCApHf3goM6QfK4GymcLm7ZV3obj5OeJgCx9ExD7d9gRdWLWhQtX25j0GIqvj/6JCCWdfJeOPSQEt4fxRcdewplbmRzdHJlYW0KZW5kb2JqCjM5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjEwID4+CnN0cmVhbQp4nDVQyw1DMQi7ZwoWqBQCgWSeVr11/2tt0DthEf9CWMiUCHmpyc4p6Us+OkwPti6/sSILrXUl7MqaIJ4r76GZsrHR2OJgcBomXoAWN2DoaY0aNXThgqYulUKBxSXwmXx1e+i+Txl4ahlydgQRQ8lgCWq6Fk1YtDyfkE4B4v9+w+4t5KGS88qeG/kbnO3wO7Nu4SdqdiLRchUy1LM0xxgIE0UePHlFpnDis9Z31TQS1GYLTpYBrk4/jA4AYCJeWYDsrkQ5S9KOpZ9vvMf3D0AAU7QKZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iago8PCAvQmFzZUZvbnQgL0RlamFWdVNhbnMgL0NoYXJQcm9jcyAxNSAwIFIKL0VuY29kaW5nIDw8Ci9EaWZmZXJlbmNlcyBbIDQ4IC96ZXJvIC9vbmUgL3R3byAvdGhyZWUgL2ZvdXIgL2ZpdmUgL3NpeCAvc2V2ZW4gL2VpZ2h0IDk3IC9hIC9iIC9jIC9kCi9lIC9mIC9nIC9oIDEwNyAvayAxMTAgL24gL28gMTE0IC9yIDExNiAvdCAvdSAxMTkgL3cgXQovVHlwZSAvRW5jb2RpbmcgPj4KL0ZpcnN0Q2hhciAwIC9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnREZXNjcmlwdG9yIDEzIDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9EZWphVnVTYW5zCi9TdWJ0eXBlIC9UeXBlMyAvVHlwZSAvRm9udCAvV2lkdGhzIDEyIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0FzY2VudCA5MjkgL0NhcEhlaWdodCAwIC9EZXNjZW50IC0yMzYgL0ZsYWdzIDMyCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnROYW1lIC9EZWphVnVTYW5zIC9JdGFsaWNBbmdsZSAwCi9NYXhXaWR0aCAxMzQyIC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCAwID4+CmVuZG9iagoxMiAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxNSAwIG9iago8PCAvYSAxNiAwIFIgL2IgMTcgMCBSIC9jIDE4IDAgUiAvZCAxOSAwIFIgL2UgMjAgMCBSIC9laWdodCAyMSAwIFIKL2YgMjIgMCBSIC9maXZlIDIzIDAgUiAvZm91ciAyNCAwIFIgL2cgMjUgMCBSIC9oIDI2IDAgUiAvayAyNyAwIFIgL24gMjggMCBSCi9vIDI5IDAgUiAvb25lIDMwIDAgUiAvciAzMSAwIFIgL3NldmVuIDMyIDAgUiAvc2l4IDMzIDAgUiAvdCAzNCAwIFIKL3RocmVlIDM1IDAgUiAvdHdvIDM2IDAgUiAvdSAzNyAwIFIgL3cgMzggMCBSIC96ZXJvIDM5IDAgUiA+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTQgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8ID4+CmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTAgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iago0MCAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjEwNjA3MTczMzA3KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuMy40LCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuMy40KSA+PgplbmRvYmoKeHJlZgowIDQxCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDEzMTYwIDAwMDAwIG4gCjAwMDAwMTI5NjYgMDAwMDAgbiAKMDAwMDAxMjk5OCAwMDAwMCBuIAowMDAwMDEzMDk3IDAwMDAwIG4gCjAwMDAwMTMxMTggMDAwMDAgbiAKMDAwMDAxMzEzOSAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzOTcgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDA0MzY4IDAwMDAwIG4gCjAwMDAwMTE2MjQgMDAwMDAgbiAKMDAwMDAxMTQyNCAwMDAwMCBuIAowMDAwMDExMDA2IDAwMDAwIG4gCjAwMDAwMTI2NzcgMDAwMDAgbiAKMDAwMDAwNDM4OSAwMDAwMCBuIAowMDAwMDA0NzY2IDAwMDAwIG4gCjAwMDAwMDUwNzYgMDAwMDAgbiAKMDAwMDAwNTM3OSAwMDAwMCBuIAowMDAwMDA1Njc5IDAwMDAwIG4gCjAwMDAwMDU5OTcgMDAwMDAgbiAKMDAwMDAwNjQ2MiAwMDAwMCBuIAowMDAwMDA2NjY4IDAwMDAwIG4gCjAwMDAwMDY5ODggMDAwMDAgbiAKMDAwMDAwNzE1MCAwMDAwMCBuIAowMDAwMDA3NTYxIDAwMDAwIG4gCjAwMDAwMDc3OTcgMDAwMDAgbiAKMDAwMDAwNzk1MCAwMDAwMCBuIAowMDAwMDA4MTg0IDAwMDAwIG4gCjAwMDAwMDg0NzEgMDAwMDAgbiAKMDAwMDAwODYyMyAwMDAwMCBuIAowMDAwMDA4ODUzIDAwMDAwIG4gCjAwMDAwMDg5OTMgMDAwMDAgbiAKMDAwMDAwOTM4MyAwMDAwMCBuIAowMDAwMDA5NTg3IDAwMDAwIG4gCjAwMDAwMDk5OTggMDAwMDAgbiAKMDAwMDAxMDMxOSAwMDAwMCBuIAowMDAwMDEwNTYzIDAwMDAwIG4gCjAwMDAwMTA3MjMgMDAwMDAgbiAKMDAwMDAxMzIyMCAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDQwIDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSA0MSA+PgpzdGFydHhyZWYKMTMzNzcKJSVFT0YK\n",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAANYCAYAAADZn0yoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABkxUlEQVR4nO39fdhdZX3nf78/BgbxgQrlgolJKNRGK9ARhkxKh7Yq6JDah2BH2niMkrtlfmkZrNqf0wo6U2l75/7ZqQ8ttjATnwg+0dyoQ3TEGqOCOggGi0CIDFEoRFISn7EdMyZ+f3/s89LNlYuQXNfae18P79dx7GOv9d3r4VwSz2t/9lrrXKkqJEmSJEnT97hRN0CSJEmS5goDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSApRkvSSX5qVG3Q9L8YJ8jSZoOA5YkSZIkdcSAJUnSFCQ5bNRtkDS32K/MDQYsjUyS30ryob757Uk29M0/kOS0Cev8fKs/t82fkmRTkm8keSjJa4Z2AJJmlY76nEpycZJ7gHuG1XZJs8/B9jmT9StJVia5Lcl3knw5yYoRHIKmyIClUboB+IUkj0uyEDgcOAsgyU8CTwJuH184ybnA+4B/W1WfTPJk4OPAR4GnAj8FbB7uIUiaRabV5/Rt5zzgZ4GTh9RuSbPTofQ559H6lSTLgauBPwCeAvwicN8wG67p8TSkRqaqvpLkYeA04OnA3wKnJflp4OeAT1fVD5IAnA/8LvCCqrqjbeJXgH+oqje2+e8BNw/xECTNIh30OeP+n6r6xvBaLmk2OsQ+54f9SpILgXdU1aa2qa8OvfGaFgOWRu0G4Dn0zj7dAHwLeDa9jueGvuVeCVw94YvOEuDLw2ikpDljOn3OuAcG2kJJc8nB9jn9/coS4CPDaZ4GwUsENWrjHc8vtOkb6HU8z+aRHc/5wHlJXtlXewB42lBaKWmumE6fM64G20RJc8jB9jn9/Yrfb2a5VPl3QqOT5OnArcBDVfVTSY6id53xYcDRVbUvSQFLgf8DfAp4Q1Vd0e7B+l/AnwFXAv8MOLmqvExQ0qSm0+e09QtYWlXbR9F+SbPLofQ54/1KuwfrY8C/BT4JLASeXFVfGsUx6NB5BksjVVX/C/gu8Ok2/x3gK8Bnq2rfhGXvB84BXp3k31fVw8DzgV8F/oHeyDvPHWLzJc0y0+lzht1WSbPfofQ5fevcAvwW8Gbg2/TOdP3EUBqsTngGS5IkSZI64hksSZIkSeqIAUuSJEmSOmLAkiRJkqSOGLAkSZIkqSPz7kHDxx57bJ144omjboY0b9x6661fq6qxUbdjFOxvpOGaz/0N2OdIw/Zofc68C1gnnngiW7ZsGXUzpHkjyd+Pug2jYn8jDdd87m/APkcatkfrc7xEUJIkSZI6YsCSJEmSpI4YsCRJkiSpIwYsSZIkSeqIAUuSJEmSOmLAkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5IkSZI6YsCSJEmSpI4YsCRJkiSpIwYsSZIkSerIYaNugKTJ3fCLzx51Ew7Js2+8YdRNmDXO+IOrR92EQ3Lrn18w6iZIkma5bWs/MZDtPvO1Zw9ku9MxsDNYSd6RZFeSO/tqf57kS0luT/LBJE/p++zSJNuT3J3k3L76GUnuaJ9dniStfkSSv2n1m5OcOKhjkSRJkqSDMcgzWFcBfwX0/1S7Cbi0qvYm+TPgUuDVSU4GVgGnAE8FPp7k6VW1D7gSWAN8DvgIsAK4HrgQ+GZV/VSSVcCfAb85wOORJEmS5oy1L3nRQLb72ndfO5DtzhYDC1hVdePEs0pV9bG+2c8B4/9VVwLXVNUe4N4k24HlSe4DjqqqmwCSXA2cRy9grQQua+tfC/xVklRVDeSAJEmSpAH6q1d9aCDbfdkbf3Ug29XkRnkP1m8Df9OmF9ELXON2tNr32/TE+vg6DwC0M2LfBn4c+NrEHSVZQ+8sGCeccEJ3RyBJkqQ5a1D3Q3vf8tw2klEEk7wW2Au8Z7w0yWJ1gPqB1tm/WLWuqpZV1bKxsbFDba4kSZIkHZShB6wkq4FfAf5d3+V8O4AlfYstBh5s9cWT1B+xTpLDgB8DvjG4lkuSJEnSgQ01YCVZAbwa+LWq+qe+jzYCq9rIgCcBS4Fbqmon8HCSM9vogRcA1/Wts7pNvwj4hPdfSZIkSRqlQQ7T/j7gJuAZSXYkuZDeqIJPBjYluS3JfwWoqq3ABuAu4KPAxW0EQYCLgLcB24Ev0xvgAuDtwI+3ATH+b+CSQR2LJEnSY0ny+CS3JPlikq1J/rjVL0vy1fbd57YkL+hb55AeUyNp5hvkKIIvnqT89gMsvxZYO0l9C3DqJPXvAedPp42SJEkd2gOcXVXfTXI48Jkk4z8Mv7mq3tC/8BQfUyNphhvJIBeSJElzTfV8t80e3l4Hun3hh4+pqap76V2tszzJQtpjatrtD+OPqZE0CxiwJEmSOpJkQZLbgF3Apqq6uX30siS3J3lHkqNb7YePnGnGH0eziEd/TM3E/a1JsiXJlt27d3d5KJKmyIAlSZLUkaraV1Wn0Rv5eHmSU+ld7vc04DRgJ/DGtvhUHlMzcX8+ikaaYQxYkiRJHauqbwGfAlZU1UMteP0AeCuwvC02lcfUSJrhBjbIhSRJ0nySZAz4flV9K8mRwPOAP0uysD16BuCFwJ1teiPw3iRvojfIxfhjavYleTjJmcDN9B5T85ahHow0y1122WUj265nsCTNCe2+hl1J7uyr/XmSL7X7Hj6Y5Cl9nx3S0MjtOX1/0+o3JzlxmMcnaVZYCHwyye3A5+ndg/Vh4L+0fuV24LnA78OUH1MjaYbzDJakueIqes/au7qvtgm4tKr2Jvkz4FLg1VMcGvlC4JtV9VNJVgF/BvzmUI5M0qxQVbcDp09Sf+kB1jmkx9RImvkMWJLmhKq6ceJZpar6WN/s54AXtekfDo0M3NseWL48yX20oZEBkowPjXx9W+eytv61wF8lSRtCWZJmlDP+4OrHXmgKbv3zCwayXWku8RJBSfPFb/OjS2ymMjTyD9epqr3At4Efn7gTh0yWJGl+M2BJmvOSvBbYC7xnvDTJYo81NPJBDZvskMmSJM1vXiIoaU5Lshr4FeCcvsv5pjI08vg6O5IcBvwY8I0BNl2SZoX7/+RnBrLdE/7ojoFsVxo0z2BJmrOSrABeDfxaVf1T30cbgVVtZMCT+NHQyDuBh5Oc2UYPvAC4rm+d1W36RcAnvP9KkiRN5BksSXNCkvcBzwGOTbIDeB29UQOPADa10dY/V1W/W1Vbk4wPjbyX/YdGvgo4kt49W+P3bb0deFcbEOMb9EYhlCRJegQDlqQ5oapePEn57QdY/pCGRq6q7wHnT6eNkiRp7jNgSZIkadY46y1nDWS7n/29zw5ku5p/vAdLkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5IkSZI6YsCSJEmSpI4YsCRJkiSpIwYsSZIkSeqIAUuSJEmSOmLAkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5IkSZI6YsCSJEmSpI4YsCRJkiSpIwYsSZIkSeqIAUuSJEmSOmLAkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5IkSZI6YsCSJEmSpI4YsCRJkiSpIwYsSZIkSeqIAUuSJKkDSR6f5JYkX0yyNckft/oxSTYluae9H923zqVJtie5O8m5ffUzktzRPrs8SUZxTJIOnQFLkiSpG3uAs6vqWcBpwIokZwKXAJuraimwuc2T5GRgFXAKsAK4IsmCtq0rgTXA0vZaMcTjkDQNAwtYSd6RZFeSO/tqnf2Ck+SIJH/T6jcnOXFQxyJJkvRYque7bfbw9ipgJbC+1dcD57XplcA1VbWnqu4FtgPLkywEjqqqm6qqgKv71pE0ww3yDNZV7P9rS5e/4FwIfLOqfgp4M/BnAzsSSZKkg5BkQZLbgF3Apqq6GTi+qnYCtPfj2uKLgAf6Vt/Raova9MT6ZPtbk2RLki27d+/u9FgkTc3AAlZV3Qh8Y0K5y19w+rd1LXCO1ydLkqRRqqp9VXUasJjed5lTD7D4ZN9b6gD1yfa3rqqWVdWysbGxQ26vpO4N+x6sLn/B+eE6VbUX+Dbw45Pt1F93JEnSMFXVt4BP0bvy5qH2ozHtfVdbbAewpG+1xcCDrb54krqkWWCmDHIxlV9w/HVHkiTNGEnGkjylTR8JPA/4ErARWN0WWw1c16Y3AqvafeUn0bsV4pb2I/TDSc5sV+dc0LeOpBnusCHv76EkC6tqZwe/4IyvsyPJYcCPsf8liZIkScOyEFjf7iN/HLChqj6c5CZgQ5ILgfuB8wGqamuSDcBdwF7g4qra17Z1Eb372Y8Erm8vSbPAsAPW+C84r2f/X3Dem+RNwFP50S84+5I83IY4vZneLzhvmbCtm4AXAZ9o92lJkiQNXVXdDpw+Sf3rwDmPss5aYO0k9S3Age7fkjRDDSxgJXkf8Bzg2CQ7gNfRC1Zd/YLzduBdSbbTO3O1alDHIkmSJEkHY2ABq6pe/CgfdfILTlV9jxbQJEmSJGkmmCmDXEiSJEnSrGfAkiRJkqSOGLAkSZIkqSMGLElzQpJ3JNmV5M6+2jFJNiW5p70f3ffZpUm2J7k7ybl99TOS3NE+u7w9g4b2nJq/afWbk5w41AOUJEmzggFL0lxxFbBiQu0SYHNVLQU2t3mSnExv5NFT2jpXtOfWAFwJrKH3uIilfdu8EPhmVf0U8GbgzwZ2JJIkadYyYEmaE6rqRvZ/2PhKYH2bXg+c11e/pqr2VNW9wHZgeXsA+lFVdVN7rt7VE9YZ39a1wDnjZ7ckSZLGGbAkzWXHV9VOgPZ+XKsvAh7oW25Hqy1q0xPrj1inqvYC3wZ+fOIOk6xJsiXJlt27d3d4KJIkaTYwYEmajyY781QHqB9onUcWqtZV1bKqWjY2NjaNJkqSpNnIgCVpLnuoXfZHe9/V6juAJX3LLQYebPXFk9QfsU6Sw4AfY/9LEiVJ0jxnwJI0l20EVrfp1cB1ffVVbWTAk+gNZnFLu4zw4SRntvurLpiwzvi2XgR8ot2nJUmS9EOHjboBktSFJO8DngMcm2QH8Drg9cCGJBcC9wPnA1TV1iQbgLuAvcDFVbWvbeoieiMSHglc314AbwfelWQ7vTNXq4ZwWJIkaZYxYEmaE6rqxY/y0TmPsvxaYO0k9S3AqZPUv0cLaJIkSY/GSwQlSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5IkSZI6YsCSJEmSpI4YsCRJkiSpIwYsSZIkSeqIAUuSJEmSOmLAkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5IkSZI6ctioGyBNx1lvOWvUTTgkn/29z466CZIkSRogz2BJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEkdSLIkySeTbEuyNckrWv2yJF9Nclt7vaBvnUuTbE9yd5Jz++pnJLmjfXZ5kozimCQdOodplyRJ6sZe4FVV9YUkTwZuTbKpffbmqnpD/8JJTgZWAacATwU+nuTpVbUPuBJYA3wO+AiwArh+SMchaRo8gyVJktSBqtpZVV9o0w8D24BFB1hlJXBNVe2pqnuB7cDyJAuBo6rqpqoq4GrgvMG2XlJXDFiSJEkdS3IicDpwcyu9LMntSd6R5OhWWwQ80LfajlZb1KYn1ifbz5okW5Js2b17d5eHIGmKDFiSJEkdSvIk4P3AK6vqO/Qu93sacBqwE3jj+KKTrF4HqO9frFpXVcuqatnY2Nh0my6pAwYsSZKkjiQ5nF64ek9VfQCgqh6qqn1V9QPgrcDytvgOYEnf6ouBB1t98SR1SbOAAUuSJKkDbaS/twPbqupNffWFfYu9ELizTW8EViU5IslJwFLglqraCTyc5My2zQuA64ZyEJKmzVEEJUmSunEW8FLgjiS3tdprgBcnOY3eZX73Ab8DUFVbk2wA7qI3AuHFbQRBgIuAq4Aj6Y0e6AiC0ixhwJIkSepAVX2Gye+f+sgB1lkLrJ2kvgU4tbvWSRoWA5YkqTP3/8nPjLoJh+SEP7pj1E2QJM0x3oMlSZIkSR0ZScBK8vtJtia5M8n7kjw+yTFJNiW5p70f3bf8pUm2J7k7ybl99TOS3NE+u7zdCCpJkiRJIzH0gJVkEfByYFlVnQosAFYBlwCbq2opsLnNk+Tk9vkpwArgiiQL2uauBNbQG3VnaftckiRJkkZiVJcIHgYcmeQw4An0nu2wEljfPl8PnNemVwLXVNWeqroX2A4sb0OeHlVVN1VVAVf3rSNJkiRJQzf0gFVVXwXeANxP72nm366qjwHHt+c+0N6Pa6ssAh7o28SOVlvUpifWJUmSJGkkRnGJ4NH0zkqdBDwVeGKSlxxolUlqdYD6ZPtck2RLki27d+8+1CZLkiRJ0kEZxSWCzwPurardVfV94APAvwYeGn/SeXvf1ZbfASzpW38xvUsKd7TpifX9VNW6qlpWVcvGxsY6PRhJkiRJGjeKgHU/cGaSJ7RR/84BtgEbgdVtmdXAdW16I7AqyRFJTqI3mMUt7TLCh5Oc2bZzQd86kiRJkjR0Q3/QcFXdnORa4AvAXuDvgHXAk4ANSS6kF8LOb8tvTbIBuKstf3FV7Wubuwi4CjgSuL69JEmSJGkkhh6wAKrqdcDrJpT30DubNdnya4G1k9S3AKd23kBJkiRJmoJRDdMuSZIkSXOOAUvSnJfk95NsTXJnkvcleXySY5JsSnJPez+6b/lLk2xPcneSc/vqZyS5o312ebv/U5Ik6YcMWJLmtCSLgJcDy6rqVGABsAq4BNhcVUuBzW2eJCe3z08BVgBXJFnQNnclsIbeYDtL2+eSJEk/ZMCSNB8cBhyZ5DDgCfQe6bASWN8+Xw+c16ZXAtdU1Z6quhfYDixvj484qqpuqqoCru5bR5IkCTBgSZrjquqrwBvojU66E/h2VX0MOL497oH2flxbZRHwQN8mdrTaojY9sf4IPthckqT5zYAlaU5r91atBE4Cngo8MclLDrTKJLU6QP2RBR9sLknSvGbAkjTXPQ+4t6p2V9X3gQ8A/xp4qF32R3vf1ZbfASzpW38xvUsKd7TpiXVJkqQfMmBJmuvuB85M8oQ26t85wDZgI7C6LbMauK5NbwRWJTkiyUn0BrO4pV1G+HCSM9t2LuhbR5IkCRjRg4YlaViq6uYk1wJfAPYCfwesA54EbEhyIb0Qdn5bfmuSDcBdbfmLq2pf29xFwFXAkcD17SVJkvRDBixJc15VvQ543YTyHnpnsyZbfi2wdpL6FuDUzhsoSZLmDC8RlCRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5IkSZI6YsCSJEmSpI4YsCRJkiSpIwYsSZIkSeqIAUuSJEmSOmLAkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5IkqQNJliT5ZJJtSbYmeUWrH5NkU5J72vvRfetcmmR7kruTnNtXPyPJHe2zy5NkFMck6dAZsCRJkrqxF3hVVT0TOBO4OMnJwCXA5qpaCmxu87TPVgGnACuAK5IsaNu6ElgDLG2vFcM8EElTd1ABK8nmg6lJ0nTZ30iaCabSF1XVzqr6Qpt+GNgGLAJWAuvbYuuB89r0SuCaqtpTVfcC24HlSRYCR1XVTVVVwNV960ia4Q470IdJHg88ATi2nc4ePz19FPDUAbdN0jxifyNpJuiqL0pyInA6cDNwfFXthF4IS3JcW2wR8Lm+1Xa02vfb9MT6ZPtZQ+9MFyeccMLBNk/SAB0wYAG/A7ySXodyKz/qZL4D/PXgmiVpHrK/kTQTTLsvSvIk4P3AK6vqOwe4fWqyD+oA9f2LVeuAdQDLli2bdBlJw3XAgFVVfwn8ZZLfq6q3DKlNkuYh+xtJM8F0+6Ikh9MLV++pqg+08kNJFrazVwuBXa2+A1jSt/pi4MFWXzxJXdIs8FhnsACoqrck+dfAif3rVNXVA2qXpHnK/kbSTDCVvqiN9Pd2YFtVvanvo43AauD17f26vvp7k7yJ3hmzpcAtVbUvycNJzqR3ieEFgD88SbPEQQWsJO8CngbcBuxr5fGbLiWpM/Y3kmaCKfZFZwEvBe5IclurvYZesNqQ5ELgfuB8gKrammQDcBe9EQgvrqrxfV0EXAUcCVzfXpJmgYMKWMAy4OQ2ko0kDZL9jaSZ4JD7oqr6DJPfPwVwzqOssxZYO0l9C3Dqwe5b0sxxsM/BuhP454NsiCQ19jeSZgL7IklTcrBnsI4F7kpyC7BnvFhVvzaQVkmaz+xvJM0E9kWSpuRgA9Zlg2yEJPW5bNQNkCTsiyRN0cGOInjDoBsiSWB/I2lmsC+SNFUHO4rgw/zoAXf/DDgc+MeqOmpQDZM0P9nfSJoJ7IskTdXBnsF6cv98kvOA5YNokKT5zf5G0kxgXyRpqg52FMFHqKr/DpzdbVMkaX/2N5JmAvsiSQfrYC8R/PW+2cfRezaEz6iR1Dn7G0kzgX2RpKk62FEEf7Vvei9wH7Cy89ZIkv2NpJnBvkjSlBzsPVi/NeiGSBLY30iaGeyLJE3VQd2DlWRxkg8m2ZXkoSTvT7J40I2TNP/Y30iaCeyLJE3VwQ5y8U5gI/BUYBHwoVabkiRPSXJtki8l2Zbk55Ick2RTknva+9F9y1+aZHuSu5Oc21c/I8kd7bPLk2SqbZI0Y3Ta30jSFNkXSZqSgw1YY1X1zqra215XAWPT2O9fAh+tqp8GngVsAy4BNlfVUmBzmyfJycAq4BRgBXBFkgVtO1cCa4Cl7bViGm2SNDN03d9I0lTYF0makoMNWF9L8pIkC9rrJcDXp7LDJEcBvwi8HaCq/k9VfYvejaPr22LrgfPa9ErgmqraU1X3AtuB5UkWAkdV1U1VVcDVfetImr06628kaRrsiyRNycEGrN8GfgP4B2An8CJgqjd//iSwG3hnkr9L8rYkTwSOr6qdAO39uLb8IuCBvvV3tNqiNj2xLml267K/kaSpsi+SNCUHG7D+FFhdVWNVdRy9TueyKe7zMOBfAldW1enAP9IuB3wUk91XVQeo77+BZE2SLUm27N69+1DbK2m4uuxvJGmq7IskTcnBBqx/UVXfHJ+pqm8Ap09xnzuAHVV1c5u/ll7geqhd9kd739W3/JK+9RcDD7b64knq+6mqdVW1rKqWjY15+bQ0w3XZ30jSVNkXSZqSgw1Yj5swqt8xHPxDih+hqv4BeCDJM1rpHOAueiP1rG611cB1bXojsCrJEUlOojeYxS3tMsKHk5zZRg+8oG8dSbNXZ/2NJE2DfZGkKTnYjuKNwP9Mci29y/B+A1g7jf3+HvCeJP8M+Aq9a5ofB2xIciFwP3A+QFVtTbKBXgjbC1xcVfvadi4CrgKOBK5vL0mzW9f9jSRNhX2RpCk5qIBVVVcn2QKcTe/ep1+vqrumutOqug1YNslH5zzK8muZpFOrqi3AqVNth6SZp+v+BnrP3gPeRq+/KHr3UtwN/A1wInAf8BvjlwMluRS4ENgHvLyq/rbVz+BHP+p8BHhFG8VU0hwziL5I0vxw0Ke6W6dixyJp4AbQ34w/e+9F7cz5E4DX0Hv23uuTXEJvsJ1XT3j23lOBjyd5ejtzPv7svc/RC1gr8My5NGf53UfSVBzsPViSNCv57D1JkjRMBixJc91Qn73nYyEkSZrfDFiS5rqhPnvPx0JIkjS/GbAkzXVDf/aeJEmavwxYkuY0n70nSZKGyQfmSZoPfPaeJEkaCgOWpDnPZ+9JkqRh8RJBSZIkSeqIAUuSJEmSOmLAkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjriMO3zwP1/8jOjbsIhOeGP7hh1EyRJkqQp8QyWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJUkeSvCPJriR39tUuS/LVJLe11wv6Prs0yfYkdyc5t69+RpI72meXJ8mwj0XS1BiwJEmSunMVsGKS+pur6rT2+ghAkpOBVcApbZ0rkixoy18JrAGWttdk25Q0AxmwJEmSOlJVNwLfOMjFVwLXVNWeqroX2A4sT7IQOKqqbqqqAq4GzhtIgyV1zoAlSZI0eC9Lcnu7hPDoVlsEPNC3zI5WW9SmJ9b3k2RNki1JtuzevXsQ7ZZ0iAxYkiRJg3Ul8DTgNGAn8MZWn+y+qjpAff9i1bqqWlZVy8bGxjpoqqTpMmBJkiQNUFU9VFX7quoHwFuB5e2jHcCSvkUXAw+2+uJJ6pJmAQOWJEnSALV7qsa9EBgfYXAjsCrJEUlOojeYxS1VtRN4OMmZbfTAC4DrhtpoSVN22KgbIEnSbHDWW84adRMO2Wd/77OjbsK8k+R9wHOAY5PsAF4HPCfJafQu87sP+B2AqtqaZANwF7AXuLiq9rVNXURvRMIjgevbS9IsYMCSJEnqSFW9eJLy2w+w/Fpg7ST1LcCpHTZN0pB4iaAkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1ZGQBK8mCJH+X5MNt/pgkm5Lc096P7lv20iTbk9yd5Ny++hlJ7mifXZ4kozgWSZIkSYLRnsF6BbCtb/4SYHNVLQU2t3mSnAysAk4BVgBXJFnQ1rkSWAMsba8Vw2m6JEmSJO1vJAEryWLgl4G39ZVXAuvb9HrgvL76NVW1p6ruBbYDy5MsBI6qqpuqqoCr+9aRJEmSpKEb1RmsvwD+EPhBX+34qtoJ0N6Pa/VFwAN9y+1otUVtemJ9P0nWJNmSZMvu3bs7OQBJkiRJmmjoASvJrwC7qurWg11lklodoL5/sWpdVS2rqmVjY2MHuVtJkiRJOjSjOIN1FvBrSe4DrgHOTvJu4KF22R/tfVdbfgewpG/9xcCDrb54krokPYKD6kiSpGEZesCqqkuranFVnUhv8IpPVNVLgI3A6rbYauC6Nr0RWJXkiCQn0RvM4pZ2GeHDSc5sX3Qu6FtHkvo5qI4kSRqKmfQcrNcDz09yD/D8Nk9VbQU2AHcBHwUurqp9bZ2L6A2UsR34MnD9sBstaWZzUB1JkjRMh41y51X1KeBTbfrrwDmPstxaYO0k9S3AqYNroaQ54C/oDarz5L7aIwbVSdI/qM7n+pYbHzzn+xzCoDr0znRxwgkndNB8SZI0m8ykM1iS1CkH1ZEkScM20jNYkjRg44PqvAB4PHBU/6A67eyVg+pIkqTOeAZL0pzloDqSJGnYPIMlaT56PbAhyYXA/cD50BtUJ8n4oDp72X9QnauAI+kNqOOgOpIkaT8GLEnzgoPqSJKkYfASQUmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5IkqSNJ3pFkV5I7+2rHJNmU5J72fnTfZ5cm2Z7k7iTn9tXPSHJH++zy9gw+SbOAAUuSJKk7VwErJtQuATZX1VJgc5snycn0HoJ+SlvniiQL2jpXAmvoPfB86STblDRDGbAkSZI6UlU3At+YUF4JrG/T64Hz+urXVNWeqroX2A4sT7IQOKqqbqqqAq7uW0fSDGfAkiRJGqzjq2onQHs/rtUXAQ/0Lbej1Ra16Yn1/SRZk2RLki27d+/uvOGSDp0BS5IkaTQmu6+qDlDfv1i1rqqWVdWysbGxThsnaWoMWJIkSYP1ULvsj/a+q9V3AEv6llsMPNjqiyepS5oFDFiSJEmDtRFY3aZXA9f11VclOSLJSfQGs7ilXUb4cJIz2+iBF/StI2mGO2zUDZAkSZorkrwPeA5wbJIdwOuA1wMbklwI3A+cD1BVW5NsAO4C9gIXV9W+tqmL6I1IeCRwfXtJmgUMWJIkSR2pqhc/ykfnPMrya4G1k9S3AKd22DRJQ+IlgpIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1ZOgBK8mSJJ9Msi3J1iSvaPVjkmxKck97P7pvnUuTbE9yd5Jz++pnJLmjfXZ5kgz7eCRJkiRp3CjOYO0FXlVVzwTOBC5OcjJwCbC5qpYCm9s87bNVwCnACuCKJAvatq4E1gBL22vFMA9EkiRJkvoNPWBV1c6q+kKbfhjYBiwCVgLr22LrgfPa9ErgmqraU1X3AtuB5UkWAkdV1U1VVcDVfetIkiRJ0tCN9B6sJCcCpwM3A8dX1U7ohTDguLbYIuCBvtV2tNqiNj2xPtl+1iTZkmTL7t27Oz0GSTOblyVLkqRhGlnASvIk4P3AK6vqOwdadJJaHaC+f7FqXVUtq6plY2Njh95YSbOZlyVLkqShGUnASnI4vXD1nqr6QCs/1C77o73vavUdwJK+1RcDD7b64knqkvRDXpYsSZKGaRSjCAZ4O7Ctqt7U99FGYHWbXg1c11dfleSIJCfR+9X4lnYZ4cNJzmzbvKBvHUnazzAuS/aSZEmS5rfDRrDPs4CXAnckua3VXgO8HtiQ5ELgfuB8gKrammQDcBe9S30urqp9bb2LgKuAI4Hr22tKzviDq6e66kjc+ucXjLoJ0qwy8bLkA9w+Na3LkqtqHbAOYNmyZZNetixJkuauoQesqvoMk39RATjnUdZZC6ydpL4FOLW71kmaiw50WXJV7fSyZEmS1JWRjiIoSYPmZcmSZook97WRSG9LsqXVDnlEU0kzmwFL0lw3flny2e1LzW1JXkDvsuTnJ7kHeH6bp6q2AuOXJX+U/S9Lfhu9gS++zDQuS5Y0bz23qk6rqmVtfiojmkqawUZxD5YkDY2XJUua4VYCz2nT64FPAa+mb0RT4N4k24HlwE0jaKOkQ+AZLEmSpOEo4GNJbk2yptUOdURTSTOcZ7AkSZKG46yqejDJccCmJF86wLIHNXJpC2prAE444YRuWilpWgxYkiSJG37x2aNuwiF59o03jLoJh6yqHmzvu5J8kN4lf4c6ounEbfpoCGmG8RJBSZKkAUvyxCRPHp8G/g1wJ4c4oulwWy1pKjyDJUmSNHjHAx9sDzk/DHhvVX00yeeBDUkuBO4HzofeiKZJxkc03csjRzSVNIMZsCRJkgasqr4CPGuS+tc5xBFNJc1sXiIoSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdWTWB6wkK5LcnWR7kktG3R5Jc5t9jqRhsb+RZqdZHbCSLAD+Gvgl4GTgxUlOHm2rJM1V9jmShsX+Rpq9ZnXAApYD26vqK1X1f4BrgJUjbpOkucs+R9Kw2N9Is1SqatRtmLIkLwJWVNW/b/MvBX62ql42Ybk1wJo2+wzg7iE281jga0Pc3zB5bLPTsI/tJ6pqbIj7G5iD6XPsbwbGY5u9hnl886q/afUu+pxh/jca9r93j2127m+2HNukfc5h02/PSGWS2n6JsarWAesG35z9JdlSVctGse9B89hmp7l8bEPwmH2O/c1geGyz11w/vgEa2necYf43Gva/B49tdu5vth/bbL9EcAewpG9+MfDgiNoiae6zz5E0LPY30iw12wPW54GlSU5K8s+AVcDGEbdJ0txlnyNpWOxvpFlqVl8iWFV7k7wM+FtgAfCOqto64mZNNJJLhYbEY5ud5vKxDdQs6HPm8n9bj232muvHNxBD7m+G+d9o2P8ePLbZub9ZfWyzepALSZIkSZpJZvslgpIkSZI0YxiwJEmSJKkjBiwdsiQnJrlz1O3Q9CS5LMl/HHU7NH1J3pFk11z8/2WSJUk+mWRbkq1JXjHqNnUlyeOT3JLki+3Y/njUbepakgVJ/i7Jh0fdFj3SfPpbPhf/3iV5eesX3zPqtnRprvy7NGBJ0ux3FbBi1I0YkL3Aq6rqmcCZwMVJTh5xm7qyBzi7qp4FnAasSHLmaJvUuVcA20bdCGkO+g/AC6rq3426IdqfAWtAkvz3JLe2XyXXPPYas85hSdYnuT3JtUmeMOoGdSXJBe24vpjkXaNuT5eSvDbJ3Uk+Djxj1O1RN6rqRuAbo27HIFTVzqr6Qpt+mN6X9UWjbVU3que7bfbw9pozI08lWQz8MvC2UbdFj2qof8uH+fd12H/vkryknZG+Lcl/S7JggPv6r8BPAhuT/P6g9tO3v/+c5EtJNiV53xDOBi5I8tb2HfpjSY4c1I4G9X3dgDU4v11VZwDLgJcn+fFRN6hjzwDWVdW/AL5D75eUWS/JKcBr+dGvynPpcqQz6D1H5XTg14F/NdoWSYcmyYn0/v3ePOKmdKZdQncbsAvYVFVz5tiAvwD+EPjBiNuhRze0v+XD/Ps67L93SZ4J/CZwVlWdBuwDBnZmqap+l95Dp59bVW8e1H4AkiwD/i0/+t9y2SD31ywF/rqqTgG+1fY/KAP5vm7AGpyXJ/ki8Dl6T2JfOuL2dO2Bqvpsm3438POjbEyHzgauraqvAVTVXDor8AvAB6vqn6rqO/jASs0iSZ4EvB94Zfv3OydU1b72hWwxsDzJqSNuUieS/Aqwq6puHXVbdEDD/Fs+zL+vw/57dw5wBvD59oPJOfTOMM0FPw9cV1X/u11F8KEh7PPeqrqtTd8KnDjAfQ3k+/qsftDwTJXkOcDzgJ+rqn9K8ing8aNs0wBMvIxlrlzWEubOsUxmLh+b5qgkh9MLV++pqg+Muj2DUFXfan8rVgCz/gZv4Czg15K8gN7fv6OSvLuqXjLidumRhvm3fNh/X4e5rwDrq+rSIe5zWDKCfe7pm94HDOQSwUF+X/cM1mD8GPDN9h/rp+ndmD3XnJDk59r0i4HPjLIxHdoM/Mb4KeIkx4y4PV26EXhhkiOTPBn41VE3SHosSQK8HdhWVW8adXu6lGQsyVPa9JH0/tB/aaSN6khVXVpVi6vqRHqXan3CcDUjDfNv+TD/vg77791m4EVJjoPesSX5iQHvc1g+A/xqG/X0SfTuq5wrBvZ93YA1GB+ld+Po7cCf0jvtONdsA1a3YzwGuHLE7elEVW0F1gI3tFPGc+YLXRso4G+A2+idDfj0SBukziR5H3AT8IwkO5JcOOo2degs4KXA2e3m8dvaWZG5YCHwydaPfp7ePVgOZ65hGtrf8mH+fR3237uqugv4T8DH2v+Wm+j9/3vWq6rP07vE8ovAB4AtwLdH2qjuDOz7eqq8YkiSJEnS/pI8qaq+20aZvBFYMz66qybnPViSJEmSHs269vzBx9O718xw9Rg8gyVJkiRJHfEeLEmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwNOMl+e6o2yBp/rDPkSRNhwFLkqRDkGTBqNsgaW6yf5kbDFgauSR/mOTlbfrNST7Rps9J8u6+5Y5NclOSX+5b744kX0zy+tG0XtJsM5U+J8lzknwyyXuBO0bUdEmz0GP1OUm+m+RPktwM/FySC5Lc3r7fvGukjdeUGLA0E9wI/EKbXgY8KcnhwM/Tnr6e5HjgfwB/VFX/I8kvAecBP1tVzwL+y9BbLWm2OuQ+py27HHhtVZ085PZKmt0eq895InBnVf0s8E3gtcDZ7fvNK0bQXk2TAUszwa3AGUmeDOwBbqLXAf0CvY7ncGAz8IdVtamt8zzgnVX1TwBV9Y2ht1rSbDWVPgfglqq6d9iNlTTrPVafsw94f1v2bODaqvoa+P1mtjJgaeSq6vvAfcBvAf+TXmfzXOBpwDZgL73O6dy+1QL4lGxJh2yKfQ7APw6vlZLmioPoc75XVfva4n6/mQMMWJopbgT+Y3v/NPC7wG1VVfQ6mt8GfjrJJW35jwG/neQJAEmOGX6TJc1ih9rnSNJ0HKjP6bcZ+I0kPw5+v5mtDFiaKT4NLARuqqqHgO+1GgDtl51VwHOT/Ieq+iiwEdiS5DZ6nZYkHaxD6nNG00RJc8gB+5xxVbUVWAvckOSLwJuG2kp1IvsHZ0mSJEnSVHgGS5IkSZI6YsCSJEmSpI4YsCRJkiSpIwYsSZIkSeqIAUuSJEmSOmLAkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5I0byS5L8nzJqk/J8mOUbRJ0txmvzP/GLAkSZIkqSMGLM1KSQ47mJokSdJs53ec2cWApZFL8ltJPtQ3vz3Jhr75B5KclqSSXJzkHuCe8VPrSV6d5B+Ad46i/ZJmnX+V5K4k30zyziSPn7hAkpe3ZRa3+ZVJbkvynSRfTrJi+M2WNIsdUr/jd5zZzTSsmeAG4M1JHgccDxwOnAWQ5CeBJwG3t2XPA34W+N/t/Z8DxwA/gT8YSDo4/w44F/hH4EPAfwI+Pv5hkv8MvBB4dlXtTrIcuBp4EbAZWAg8ediNljSrHWq/81P4HWfW8j+WRq6qvgI8DJwGPBv4W+CrSX66zX+6qn7QFv9/quobVfW/2/wPgNdV1Z6+miQdyF9V1QNV9Q1gLfDiVk+SN9H7EvTcqtrd6hcC76iqTVX1g6r6alV9aQTtljR7HWq/A37HmbU8g6WZ4gbgOcBPtelv0QtXP9fmxz0wYb3dVfW9IbRP0tzR34/8PfDUNv0UYA3wm1X17b5llgAfGU7TJM1Rh9rvgN9xZi3PYGmmGA9Yv9Cmb6AXsJ7NIwNWTVhv4rwkPZYlfdMnAA+26W8CvwK8M8lZfcs8ADxtSG2TNDcdar8DfseZtQxYmiluAJ4LHFlVO4BPAyuAHwf+bpQNkzTnXNxuIj8GeA3wN+MfVNWn6N0r8cEkP9vKbwd+K8k5SR6XZFG7hFmSDtah9juaxQxYmhGq6n8B36UXrKiq7wBfAT5bVftG2TZJc857gY/R62O+Avx/+z+sqk3AbwEbk5xRVbe0+TcD36b3g9BPDLXFkma7Q+p3ht88dSlVnn2UJEmSpC54BkuSJEmSOmLAkiRJkqSOGLAkSZI6lGRBkr9L8uE2f0ySTUnuae9H9y17aZLtSe5Ocm5f/Ywkd7TPLk+SURyLpENnwJIkSerWK4BtffOXAJuraimwuc2T5GRgFXAKvZFzr0iyoK1zJb3nIy1trxXDabqk6Zp3Dxo+9thj68QTTxx1M6R549Zbb/1aVY2Nuh2jYH8jDddM6G+SLAZ+GVgL/N+tvJLesx4B1gOfAl7d6tdU1R7g3iTbgeVJ7gOOqqqb2javBs4Drj/Qvu1zpOF6tD5n3gWsE088kS1btoy6GdK8keTvR92GUbG/kYZrhvQ3fwH8IfDkvtrxVbUToKp2Jjmu1RcBn+tbbkerfb9NT6zvJ8kaeme6OOGEE+xzpCF6tD7HSwQlSZI6kORXgF1VdevBrjJJrQ5Q379Yta6qllXVsrGxeXmxgDTjGLAkzXlJ7ms3i9+WZEuredO5pK6dBfxau8TvGuDsJO8GHkqyEKC972rL7wCW9K2/GHiw1RdPUpc0CxiwJM0Xz62q06pqWZv3pnNJnaqqS6tqcVWdSK8f+URVvQTYCKxui60GrmvTG4FVSY5IchK9fuWWdjnhw0nObD/kXNC3jqQZzoAlab5aSe9mc9r7eX31a6pqT1XdC4zfdL6QdtN5VRVwdd86knQgrween+Qe4PltnqraCmwA7gI+ClxcVfvaOhcBb6PXB32ZxxjgQtLMMe8GuZA0LxXwsSQF/LeqWscAbzqXpKr6FL3RAqmqrwPnPMpya+mNODixvgU4dXAtlDQoBixJ88FZVfVgC1GbknzpAMtO66bziSN6SZKk+cVLBCXNeVX1YHvfBXwQWM6Abjp3RC9JkuY3A5akOS3JE5M8eXwa+DfAnXjTuSRJGgAvEZQ01x0PfLCNqH4Y8N6q+miSzwMbklwI3A+cD72bzpOM33S+l/1vOr8KOJLeDefedC5Jkh7BgCVpTquqrwDPmqTuTeeSJKlzXiIoSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkccpl2duOEXnz2Q7T77xhv2q/3Vqz40kH297I2/OpDtStLBWvuSFw1ku69997UD2a6kbvkdZ24YesBK8njgRuCItv9rq+p1SS4D/i9gd1v0NVX1kbbOpcCFwD7g5VX1t61+Bj966OdHgFdUVQ3vaDQf+IVHkiRJB2sUZ7D2AGdX1XeTHA58Jsn17bM3V9Ub+hdOcjKwCjgFeCrw8SRPr6p9wJXAGuBz9ALWCuB6JEmSJGkEhh6w2hmm77bZw9vrQGedVgLXVNUe4N4k24HlSe4DjqqqmwCSXA2chwELgLPectZAtvvZ3/vsQLarnm1rPzGQ7T7ztWcPZLuSJM1lw7wFQnPHSAa5SLIgyW3ALmBTVd3cPnpZktuTvCPJ0a22CHigb/UdrbaoTU+sT7a/NUm2JNmye/fuyRaRJEmSpGkbySAX7fK+05I8BfhgklPpXe73p/TOZv0p8Ebgt4FMtokD1Cfb3zpgHcCyZcu8R0tqLrvsslm1XUmSpJlupKMIVtW3knwKWNF/71WStwIfbrM7gCV9qy0GHmz1xZPUZ6T7/+RnBrLdE/7ojoFsV5IkSdKhG/olgknG2pkrkhwJPA/4UpKFfYu9ELizTW8EViU5IslJwFLglqraCTyc5MwkAS4ArhvWcUiSJEnSRKM4g7UQWJ9kAb2At6GqPpzkXUlOo3eZ333A7wBU1dYkG4C7gL3Axe0SQ4CL+NEw7dfjABeSJEmSRmgUowjeDpw+Sf2lB1hnLbB2kvoW4NROGyhJkiRJUzSSUQQlSZIkaS4a6SAXo3bGH1w9kO3e+ucXDGS7kiRJkmY2z2BJkiRJUkcMWJIkSZLUkXl9iaAkSdJc5G0Q0uh4BkuSJEmSOmLAkiRJkqSOGLAkSZIkqSPegyVJ0gH81as+NJDtvuyNvzqQ7UqSRsuAJUmSJM1Da1/yooFs97XvvnYg250tvERQkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5IkSZI6YsCSJEmSpI4YsCRJkjqQ5PFJbknyxSRbk/xxq1+W5KtJbmuvF/Stc2mS7UnuTnJuX/2MJHe0zy5PklEck6RD53OwJEmSurEHOLuqvpvkcOAzSa5vn725qt7Qv3CSk4FVwCnAU4GPJ3l6Ve0DrgTWAJ8DPgKsAK5H0oznGSxJkqQOVM932+zh7VUHWGUlcE1V7amqe4HtwPIkC4GjquqmqirgauC8ATZdUoc8gyVJktSRJAuAW4GfAv66qm5O8kvAy5JcAGwBXlVV3wQW0TtDNW5Hq32/TU+sT7a/NfTOdHHCCSd0fDQz01lvOWsg2/3s7312INvV/OMZLEmSpI5U1b6qOg1YTO9s1Kn0Lvd7GnAasBN4Y1t8svuq6gD1yfa3rqqWVdWysbGxabZeUhcMWJIkSR2rqm8BnwJWVNVDLXj9AHgrsLwttgNY0rfaYuDBVl88SV3SLGDAkiRJ6kCSsSRPadNHAs8DvtTuqRr3QuDONr0RWJXkiCQnAUuBW6pqJ/BwkjPb6IEXANcN6zgkTY/3YEmSJHVjIbC+3Yf1OGBDVX04ybuSnEbvMr/7gN8BqKqtSTYAdwF7gYvbCIIAFwFXAUfSGz3QEQSlWcKAJUmS1IGquh04fZL6Sw+wzlpg7ST1LcCpnTZQ0lAYsCRJs8oNv/jsgWz32TfeMJDtSpLmF+/BkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwJEmSJKkjDtMuSZIkaaC2rf3EQLb7zNeePZDtTodnsCRJkiSpIwYsSZIkSeqIAUuSJEmSOmLAkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjgw9YCV5fJJbknwxydYkf9zqxyTZlOSe9n503zqXJtme5O4k5/bVz0hyR/vs8iQZ9vFIkiRJ0rhRnMHaA5xdVc8CTgNWJDkTuATYXFVLgc1tniQnA6uAU4AVwBVJFrRtXQmsAZa214ohHockSZIkPcLQA1b1fLfNHt5eBawE1rf6euC8Nr0SuKaq9lTVvcB2YHmShcBRVXVTVRVwdd86kiRJkjR0I7kHK8mCJLcBu4BNVXUzcHxV7QRo78e1xRcBD/StvqPVFrXpifXJ9rcmyZYkW3bv3t3psUiSJEnSuJEErKraV1WnAYvpnY069QCLT3ZfVR2gPtn+1lXVsqpaNjY2dsjtlSRJkqSDMdJRBKvqW8Cn6N079VC77I/2vqsttgNY0rfaYuDBVl88SV2SJEmSRuKwYe8wyRjw/ar6VpIjgecBfwZsBFYDr2/v17VVNgLvTfIm4Kn0BrO4par2JXm4DZBxM3AB8JbhHo0kSdL8dv+f/MxAtnvCH90xkO1KgzaKM1gLgU8muR34PL17sD5ML1g9P8k9wPPbPFW1FdgA3AV8FLi4qva1bV0EvI3ewBdfBq4f5oFImh3afZ9/l+TDbd7HQkiSpIEY+hmsqrodOH2S+teBcx5lnbXA2knqW4AD3b8lSQCvALYBR7X58cdCvD7JJW3+1RMeC/FU4ONJnt5+1Bl/LMTngI/Qu7TZH3UkSdIjjPQeLEkatCSLgV+md7Z7nI+FkCRJA2HAkjTX/QXwh8AP+mo+FkKSJA2EAUvSnJXkV4BdVXXrwa4ySc3HQkiSpIM29HuwJGmIzgJ+LckLgMcDRyV5N+2xEFW108dCSJKkLnkGS9KcVVWXVtXiqjqR3uAVn6iql/Cjx0LA/o+FWJXkiCQn8aPHQuwEHk5yZhs98IK+dSRJkn7IM1iS5qPXAxuSXAjcD5wPvcdCJBl/LMRe9n8sxFXAkfRGD3QEQUmSZqjLLrtsZNs1YEmaF6rqU8Cn2rSPhZAkSQPhJYKSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSVIHkjw+yS1Jvphka5I/bvVjkmxKck97P7pvnUuTbE9yd5Jz++pnJLmjfXZ5ewafpFnAgCVJktSNPcDZVfUs4DRgRZIzgUuAzVW1FNjc5klyMr2HoJ8CrACuSLKgbetKYA29B54vbZ9LmgUMWJIkSR2onu+22cPbq4CVwPpWXw+c16ZXAtdU1Z6quhfYDixPshA4qqpuqqoCru5bR9IMZ8CSJEnqSJIFSW4DdgGbqupm4Piq2gnQ3o9riy8CHuhbfUerLWrTE+uT7W9Nki1JtuzevbvTY5E0NQYsSZKkjlTVvqo6DVhM72zUqQdYfLL7quoA9cn2t66qllXVsrGxsUNur6TuGbAkSZI6VlXfAj5F796ph9plf7T3XW2xHcCSvtUWAw+2+uJJ6pJmAQOWJElSB5KMJXlKmz4SeB7wJWAjsLotthq4rk1vBFYlOSLJSfQGs7ilXUb4cJIz2+iBF/StI2mGO2zUDZAkSZojFgLr20iAjwM2VNWHk9wEbEhyIXA/cD5AVW1NsgG4C9gLXFxV+9q2LgKuAo4Erm8vSbOAAUuSJKkDVXU7cPok9a8D5zzKOmuBtZPUtwAHun9L0gzlJYKSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdGXrASrIkySeTbEuyNckrWv2yJF9Nclt7vaBvnUuTbE9yd5Jz++pnJLmjfXZ5kgz7eCRJkiRp3GEj2Ode4FVV9YUkTwZuTbKpffbmqnpD/8JJTgZWAacATwU+nuTpVbUPuBJYA3wO+AiwArh+SMchSQLOestZA9nuZ3/vswPZriRJgzT0M1hVtbOqvtCmHwa2AYsOsMpK4Jqq2lNV9wLbgeVJFgJHVdVNVVXA1cB5g229JEmSJD26kd6DleRE4HTg5lZ6WZLbk7wjydGttgh4oG+1Ha22qE1PrE+2nzVJtiTZsnv37i4PQZIkSZJ+aGQBK8mTgPcDr6yq79C73O9pwGnATuCN44tOsnodoL5/sWpdVS2rqmVjY2PTbbokSZIkTWokASvJ4fTC1Xuq6gMAVfVQVe2rqh8AbwWWt8V3AEv6Vl8MPNjqiyepS5IkSdJIjGIUwQBvB7ZV1Zv66gv7FnshcGeb3gisSnJEkpOApcAtVbUTeDjJmW2bFwDXDeUgJEmSJGkSoxhF8CzgpcAdSW5rtdcAL05yGr3L/O4DfgegqrYm2QDcRW8EwovbCIIAFwFXAUfSGz3QEQQlSZIkjczQA1ZVfYbJ75/6yAHWWQusnaS+BTi1u9ZJkiRJ0tSNdBRBSZIkSZpLDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSR1IsiTJJ5NsS7I1ySta/bIkX01yW3u9oG+dS5NsT3J3knP76mckuaN9dnmSjOKYJB26w0bdAEmSpDliL/CqqvpCkicDtybZ1D57c1W9oX/hJCcDq4BTgKcCH0/y9KraB1wJrAE+B3wEWAFcP6TjkDQNnsGSJEnqQFXtrKovtOmHgW3AogOsshK4pqr2VNW9wHZgeZKFwFFVdVNVFXA1cN5gWy+pKwYsSZKkjiU5ETgduLmVXpbk9iTvSHJ0qy0CHuhbbUerLWrTE+uT7WdNki1JtuzevbvLQ5A0RQYsSZKkDiV5EvB+4JVV9R16l/s9DTgN2Am8cXzRSVavA9T3L1atq6plVbVsbGxsuk2X1AEDliRJUkeSHE4vXL2nqj4AUFUPVdW+qvoB8FZgeVt8B7Ckb/XFwIOtvniSuqRZwIAlSZLUgTbS39uBbVX1pr76wr7FXgjc2aY3AquSHJHkJGApcEtV7QQeTnJm2+YFwHVDOQhJ0+YogpIkSd04C3gpcEeS21rtNcCLk5xG7zK/+4DfAaiqrUk2AHfRG4Hw4jaCIMBFwFXAkfRGD3QEQWmWMGBJkiR1oKo+w+T3T33kAOusBdZOUt8CnNpd6yQNi5cISpIkSVJHDFiSJEmS1BEDliRJkiR1xIAlaU5L8vgktyT5YpKtSf641Y9JsinJPe396L51Lk2yPcndSc7tq5+R5I722eVtdC9JkqQfMmBJmuv2AGdX1bPoPeRzRZIzgUuAzVW1FNjc5klyMrAKOAVYAVyRZEHb1pXAGnpDKS9tn0uSJP2QAUvSnFY9322zh7dXASuB9a2+HjivTa8ErqmqPVV1L7AdWN6eY3NUVd1UVQVc3beOJEkSYMCSNA8kWdCeSbML2FRVNwPHt4d50t6Pa4svAh7oW31Hqy1q0xPrE/e1JsmWJFt2797d+bFIkqSZzYAlac6rqn1VdRqwmN7ZqAM9W2ay+6rqAPWJ+1pXVcuqatnY2NiU2itJkmYvA5akeaOqvgV8it69Uw+1y/5o77vaYjuAJX2rLQYebPXFk9QlSZJ+yIAlaU5LMpbkKW36SOB5wJeAjcDqtthq4Lo2vRFYleSIJCfRG8zilnYZ4cNJzmyjB17Qt44kSRIAh426AZI0YAuB9W0kwMcBG6rqw0luAjYkuRC4HzgfoKq2JtkA3AXsBS6uqn1tWxcBVwFHAte3lyRJ0g8ZsCTNaVV1O3D6JPWvA+c8yjprgbWT1LcAB7p/S5IkzXMGLEmS5qltaz8xkO0+87VnD2S7kjQbeA+WJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRHzQsSXPQ/X/yMwPZ7gl/dMdAtitJ0lzhGSxJkiRJ6sjQA1aSJUk+mWRbkq1JXtHqxyTZlOSe9n503zqXJtme5O4k5/bVz0hyR/vs8iQZ9vFIkiRJ0rhRnMHaC7yqqp4JnAlcnORk4BJgc1UtBTa3edpnq4BTgBXAFUkWtG1dCawBlrbXimEeiCRJkiT1G3rAqqqdVfWFNv0wsA1YBKwE1rfF1gPntemVwDVVtaeq7gW2A8uTLASOqqqbqqqAq/vWkSRJkqShG+k9WElOBE4HbgaOr6qd0AthwHFtsUXAA32r7Wi1RW16Yn2y/axJsiXJlt27d3d6DJIkSZI0bmQBK8mTgPcDr6yq7xxo0UlqdYD6/sWqdVW1rKqWjY2NHXpjJUmSJOkgTGuY9iSbq+qcx6pNst7h9MLVe6rqA638UJKFVbWzXf63q9V3AEv6Vl8MPNjqiyepS5qDptrfSJoZLrvsslm1XfscSVM1pTNYSR6f5Bjg2CRHtxEAj2mX/D31MdYN8HZgW1W9qe+jjcDqNr0auK6vvirJEUlOojeYxS3tMsKHk5zZtnlB3zqS5ojp9DeSdKjscyRN11TPYP0O8Ep6Hc2t/Ohyve8Af/0Y654FvBS4I8ltrfYa4PXAhiQXAvcD5wNU1dYkG4C76I1AeHFV7WvrXQRcBRwJXN9ekuaW6fQ3knSo7HMkTcuUAlZV/SXwl0l+r6recojrfobJ758CmPS0e1WtBdZOUt8CnHoo+5c0u0ynv5GkQ2WfI2m6pnUPVlW9Jcm/Bk7s31ZVXT3NdknSI9jfSBom+xxJUzXdQS7eBTwNuA0Yv2xv/JlUktQZ+xtJw2SfI2mqphWwgGXAye1Bv5I0SPY3kobJPkfSlEz3OVh3Av+8i4ZI0mOwv5E0TPY5kqZkumewjgXuSnILsGe8WFW/Ns3tStJE9jeShsk+R9KUTDdgXdZFIyTpIFw26gZImlcuO9QVkiyhd4/WPwd+AKyrqr9sz9X6G3oDZtwH/EZVfbOtcylwIb37vF5eVX/b6mfwo0fRfAR4hZcrSrPDdEcRvKGrhkjSgdjfSBqmKfY5e4FXVdUXkjwZuDXJJuD/A2yuqtcnuQS4BHh1kpOBVcAp9J679fEkT2/P+7wSWAN8jl7AWoHP+5RmhWndg5Xk4STfaa/vJdmX5DtdNU6SxtnfSBqmqfQ5VbWzqr7Qph8GtgGLgJXA+rbYeuC8Nr0SuKaq9lTVvcB2YHmShcBRVXVTO2t1dd86kma46Z7BenL/fJLzgOXT2aYkTcb+RtIwTbfPSXIicDpwM3B8Ve1s292Z5Li22CJ6Z6jG7Wi177fpifXJ9rOG3pkuTjjhhINtnqQBmu4ogo9QVf8dOLvLbUrSZOxvJA3TofQ5SZ4EvB94ZVUd6KxXJtvVAeqTtWtdVS2rqmVjY2MH0zxJAzbdBw3/et/s4+g9M8IbMCV1zv5G0jBNtc9Jcji9cPWeqvpAKz+UZGE7e7UQ2NXqO4AlfasvBh5s9cWT1CXNAtMdRfBX+6b30hsZZ+U0tylJk7G/kTRMh9znJAnwdmBbVb2p76ONwGrg9e39ur76e5O8id4gF0uBW6pqX7sH7Ex6lxheALxl2kckaSimew/Wb3XVEEk6EPsbScM0xT7nLOClwB1Jbmu119ALVhuSXAjcD5zf9rE1yQbgLnoh7uI2giDARfxomPbrcQRBadaY7iWCi+n9onIWvdPmn6H3nIYdB1xRkg6R/Y2kYZpKn1NVn2Hy+6cAznmUddYCayepbwFOPcRmS5oBpjvIxTvpnd5+Kr3RbT7UapLUNfsbScNknyNpSqYbsMaq6p1Vtbe9rgIcwkbSINjfSBom+xxJUzLdgPW1JC9JsqC9XgJ8vYuGSdIE9jeShsk+R9KUTDdg/TbwG8A/ADuBFwHeiC5pEOxvJA2TfY6kKZnuMO1/Cqyuqm8CJDkGeAO9TkmSumR/I2mY7HMkTcl0A9a/GO94AKrqG0lOn+Y2JWkys7q/OeMPrh7Idm/98wsGsl1Js7vPkTQ6071E8HFJjh6fab/uTDe0SdJk7G8kDZN9jqQpmW5H8Ubgfya5lt4zIn6DSZ7lIEkdsL+RNEz2OZKmZFoBq6quTrIFOJveg/V+varu6qRlktTH/kbSMNnnSJqqaZ/qbp2NHY6kgbO/kTRM9jmSpmK692BJkiRJkhoDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSR0YSsJK8I8muJHf21S5L8tUkt7XXC/o+uzTJ9iR3Jzm3r35GkjvaZ5cnybCPRZIkSZLGjeoM1lXAiknqb66q09rrIwBJTgZWAae0da5IsqAtfyWwBljaXpNtU5IkSZKGYiQBq6puBL5xkIuvBK6pqj1VdS+wHVieZCFwVFXdVFUFXA2cN5AGS5IkSdJBmGn3YL0sye3tEsKjW20R8EDfMjtabVGbnljfT5I1SbYk2bJ79+5BtFuSJEmSZlTAuhJ4GnAasBN4Y6tPdl9VHaC+f7FqXVUtq6plY2NjHTRV0myRZEmSTybZlmRrkle0+jFJNiW5p70f3beO931KkqQpmTEBq6oeqqp9VfUD4K3A8vbRDmBJ36KLgQdbffEkdUnqtxd4VVU9EzgTuLjd23kJsLmqlgKb27z3fUqSpGmZMQGr3VM17oXA+AiDG4FVSY5IchK9LzW3VNVO4OEkZ7ZfkS8ArhtqoyXNeFW1s6q+0KYfBrbRu5x4JbC+LbaeH93D6X2fkiRpyg4bxU6TvA94DnBskh3A64DnJDmN3mV+9wG/A1BVW5NsAO6i90v0xVW1r23qInojEh4JXN9ekjSpJCcCpwM3A8e3H2qoqp1JjmuLLQI+17fa+P2d3+cg7vtMsobeWS5OOOGEjo9AkiTNdCMJWFX14knKbz/A8muBtZPUtwCndtg0SXNUkicB7wdeWVXfOcDtU9O677Oq1gHrAJYtWzbpfaGSJGnumjGXCErSoCQ5nF64ek9VfaCVHxq/NLm972p17/uUJElTZsCSNKe1ezTfDmyrqjf1fbQRWN2mV/Ojezi971OSJE2ZAUvSXHcW8FLg7CS3tdcLgNcDz09yD/D8Nk9VbQXG7/v8KPvf9/k2egNffBnv+5Q0QXuW564kd/bVLkvy1Ql90PhnPhZCmmNGcg+WJA1LVX2Gye+fAjjnUdbxvk9JU3UV8Ff0Rhrt9+aqekN/YcJjIZ4KfDzJ09uPOuOPhfgc8BF6j4XwRx1pFvAMliRJUkeq6kbgGwe5uI+FkOYgA5YkSdLgvSzJ7e0SwqNbbRHwQN8y449/WMRBPBYCeo+GSLIlyZbdu3cPot2SDpEBS5IkabCuBJ4GnAbsBN7Y6tN6LAT0Hg1RVcuqatnY2FgHTZU0XQYsSZKkAaqqh6pqX1X9AHgrsLx95GMhpDnIgCVJkjRA48/ca14IjI8w6GMhpDnIUQQlSZI6kuR9wHOAY5PsAF4HPCfJafQu87sP+B3oPRYiyfhjIfay/2MhrgKOpDd6oCMISrOEAUuSJKkjVfXiScpvP8DyPhZCmmO8RFCSJEmSOmLAkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5IkSZI6YsCSJEmSpI4YsCRJkiSpIwYsSZIkSeqIAUuSJEmSOmLAkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjhiwJEmSJKkjBixJkiRJ6ogBS5IkSZI6YsCSJEmSpI4YsCRJkiSpIwYsSZIkSeqIAUuSJEmSOmLAkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFLkiRJkjpiwJIkSZKkjowkYCV5R5JdSe7sqx2TZFOSe9r70X2fXZpke5K7k5zbVz8jyR3ts8uTZNjHIkmSJEnjRnUG6ypgxYTaJcDmqloKbG7zJDkZWAWc0ta5IsmCts6VwBpgaXtN3KYkSZIkDc1IAlZV3Qh8Y0J5JbC+Ta8HzuurX1NVe6rqXmA7sDzJQuCoqrqpqgq4um8dSZIkSRq6mXQP1vFVtROgvR/X6ouAB/qW29Fqi9r0xPp+kqxJsiXJlt27d3fecEmSJEmCmRWwHs1k91XVAer7F6vWVdWyqlo2NjbWaeMkSZIkadxMClgPtcv+aO+7Wn0HsKRvucXAg62+eJK6JEmSJI3ETApYG4HVbXo1cF1ffVWSI5KcRG8wi1vaZYQPJzmzjR54Qd86kiRJkjR0oxqm/X3ATcAzkuxIciHweuD5Se4Bnt/mqaqtwAbgLuCjwMVVta9t6iLgbfQGvvgycP1QD0SSJKmPj6KRdNgodlpVL36Uj855lOXXAmsnqW8BTu2waZIkSdNxFfBX9EY3Hjf+KJrXJ7mkzb96wqNongp8PMnT2w/J44+i+RzwEXqPovGHZGkWmEmXCEqSJM1qPopGkgFLkiRpsHwUjTSPGLAkSZJGw0fRSHOQAUuSJGmwfBSNNI8YsCRJkgbLR9FI88hIRhGUJEmai9qjaJ4DHJtkB/A6eo+e2dAeS3M/cD70HkWTZPxRNHvZ/1E0VwFH0hs90BEEpVnCgCVJktQRH0UjyUsEJUmSJKkjBixJkiRJ6ogBS5IkSZI6YsCSJEmSpI4YsCRJkiSpIwYsSZIkSeqIAUuSJEmSOmLAkiRJkqSOGLAkSZIkqSMGLEmSJEnqiAFL0pyW5B1JdiW5s692TJJNSe5p70f3fXZpku1J7k5ybl/9jCR3tM8uT5JhH4skSZr5DFiS5rqrgBUTapcAm6tqKbC5zZPkZGAVcEpb54okC9o6VwJrgKXtNXGbkiRJBixJc1tV3Qh8Y0J5JbC+Ta8HzuurX1NVe6rqXmA7sDzJQuCoqrqpqgq4um8dSZKkHzJgSZqPjq+qnQDt/bhWXwQ80LfcjlZb1KYn1veTZE2SLUm27N69u/OGS5Kkmc2AJUk/Mtl9VXWA+v7FqnVVtayqlo2NjXXaOEmSNPMZsCTNRw+1y/5o77tafQewpG+5xcCDrb54krokSdIjGLAkzUcbgdVtejVwXV99VZIjkpxEbzCLW9plhA8nObONHnhB3zqSJEk/dNioGyBJg5TkfcBzgGOT7ABeB7we2JDkQuB+4HyAqtqaZANwF7AXuLiq9rVNXURvRMIjgevbS5Ik6REMWJLmtKp68aN8dM6jLL8WWDtJfQtwaodNkyRJc5CXCEqSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktSRGRewktyX5I4ktyXZ0mrHJNmU5J72fnTf8pcm2Z7k7iTnjq7lkiRJkua7GRewmudW1WlVtazNXwJsrqqlwOY2T5KTgVXAKcAK4IokC0bRYEmSJEmaqQFropXA+ja9Hjivr35NVe2pqnuB7cDy4TdPkiRJkmZmwCrgY0luTbKm1Y6vqp0A7f24Vl8EPNC37o5We4Qka5JsSbJl9+7dA2y6JEmSpPlsJgass6rqXwK/BFyc5BcPsGwmqdV+hap1VbWsqpaNjY111U5JkqSD5n3m0vww4wJWVT3Y3ncBH6R3yd9DSRYCtPddbfEdwJK+1RcDDw6vtZIkSYfE+8ylOW5GBawkT0zy5PFp4N8AdwIbgdVtsdXAdW16I7AqyRFJTgKWArcMt9WSJElT5n3m0hxz2KgbMMHxwAeTQK9t762qjyb5PLAhyYXA/cD5AFW1NckG4C5gL3BxVe0bTdMlSZIOaPw+8wL+W1WtY8J95kn67zP/XN+6j3qfObAG4IQTThhk2yUdpBkVsKrqK8CzJql/HTjnUdZZC6wdcNMkSZKm66yqerCFqE1JvnSAZQ/6PnNgHcCyZcv2+1zS8M2oSwQlSZLmKu8zl+YHA5YkSdKAeZ+5NH/MqEsEJUmS5ijvM5fmCQOWJEnSgHmfuTR/eImgJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHTFgSZIkSVJHDFiSJEmS1BEDliRJkiR1xIAlSZIkSR0xYEmSJElSRwxYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHVk1gesJCuS3J1ke5JLRt0eSXObfY6kYbG/kWanWR2wkiwA/hr4JeBk4MVJTh5tqyTNVfY5kobF/kaavWZ1wAKWA9ur6itV9X+Aa4CVI26TpLnLPkfSsNjfSLNUqmrUbZiyJC8CVlTVv2/zLwV+tqpeNmG5NcCaNvsM4O4p7O5Y4GvTaO5M3ddc35/HNvr9/URVjXXdmFE4mD5nFvY3w97fXD62Ye/PY9vfvOpvWn229Tn+u3V/M21f09nfpH3OYdNvz0hlktp+ibGq1gHrprWjZEtVLZvONmbivub6/jy22bu/Geox+5zZ1t8Me39z+diGvT+Pbc7zO84s299cPrZh72+2H9tsv0RwB7Ckb34x8OCI2iJp7rPPkTQs9jfSLDXbA9bngaVJTkryz4BVwMYRt0nS3GWfI2lY7G+kWWpWXyJYVXuTvAz4W2AB8I6q2jqg3U3r9PsM3tdc35/HNnv3N+MMsc+Zy/9t5/KxDXt/Htsc5necWbm/uXxsw97frD62WT3IhSRJkiTNJLP9EkFJkiRJmjEMWJIkSZLUEQPWY0jyjiS7ktw5hH0tSfLJJNuSbE3yigHv7/FJbknyxba/Px7k/to+FyT5uyQfHsK+7ktyR5LbkmwZ8L6ekuTaJF9q//1+boD7ekY7pvHXd5K8coD7+/327+POJO9L8vhB7Uujk+SyJP9x1O3oUpKXt/8/vmfUbelSkhOH8TdpFObysc1Ec/U7zii+37T9+h1n+vuaE99xDFiP7SpgxZD2tRd4VVU9EzgTuDjJyQPc3x7g7Kp6FnAasCLJmQPcH8ArgG0D3ke/51bVaUN4lsJfAh+tqp8GnsUAj7Gq7m7HdBpwBvBPwAcHsa8ki4CXA8uq6lR6N1qvGsS+pAH4D8ALqurfjboh0gx1FXPzO84ovt+A33Gmba58xzFgPYaquhH4xpD2tbOqvtCmH6b3D3jRAPdXVfXdNnt4ew1s1JMki4FfBt42qH2MQpKjgF8E3g5QVf+nqr41pN2fA3y5qv5+gPs4DDgyyWHAE/A5LAOV5IIkt7dfXt814H29NsndST4OPGOQ+2r7e0n7Vfm2JP8tyYIB7uu/Aj8JbEzy+4PaT9/+/nP7dXdT+xV00GcDFyR5a/vl9WNJjhzUjpL89yS3tn2tGdR++hyWZH37/8G1SZ4whH3OS3P1O86wv9+A33EGZNZ+xzFgzVBJTgROB24e8H4WJLkN2AVsqqpB7u8vgD8EfjDAffQr4GPti8EgvxT8JLAbeGe7NOBtSZ44wP31WwW8b1Abr6qvAm8A7gd2At+uqo8Nan/zXZJTgNfyo19eB3kJzRn0/v2cDvw68K8Gta+2v2cCvwmc1X6Z3AcM7MxSVf0uvT+Uz62qNw9qPwBJlgH/lh/9bznoX5MBlgJ/XVWnAN9q+x+U366qM+gd18uT/PgA9wW9sL+uqv4F8B16ZyI1hwzjO86Qv9+A33EGYdZ+xzFgzUBJngS8H3hlVX1nkPuqqn3ty85iYHmSUwexnyS/AuyqqlsHsf1HcVZV/Uvgl+hdivCLA9rPYcC/BK6sqtOBfwQuGdC+fii9B0/+GvD/H+A+jgZWAicBTwWemOQlg9qfOBu4tqq+BlBVg/xl+ReAD1bVP7V+ZtAPMD2H3uUen29fes6h94d7Lvh54Lqq+t/tl/kPDWGf91bVbW36VuDEAe7r5Um+CHwOWEIv3A3SA1X12Tb9bnr/+2qOGNZ3nGF9vwG/4wzCbP+OY8CaYZIcTq/jeU9VfWBY+22nez/F4K7FPgv4tST3AdcAZyd594D2BUBVPdjed9G7fnf5gHa1A9jR9+vYtfQ6o0H7JeALVfXQAPfxPHpf5HZX1feBDwD/eoD7m+/CgC9jmWCY+wqwfvza+qp6RlVdNsT9D1JGsM89fdP76H0J6lyS59DrB36unVX9O2DQA91M/HfpAzvniFF8xxnC9xvwO84gzOrvOAasGSRJ6F3juq2q3jSE/Y0leUqbPpLeP7QvDWJfVXVpVS2uqhPpnfL9RFUN7ExIkicmefL4NPBvgIGMklRV/wA8kGT8HpZzgLsGsa8JXswAT5039wNnJnlC+/d5DsO9gXe+2Qz8xvglWEmOGeC+bgRemOTI9v+VXx3gvqB3bC9Kchz0ji3JTwx4n8PyGeBX0xu57En07sOYK34M+GZV/VOSn6Y3OMGgndA3StmL6f3vq1lumN9xhvn9BvyOMyCz+juOAesxJHkfcBPwjCQ7klw4wN2dBbyU3i8f48NTvmCA+1sIfDLJ7cDn6V2jPPChRYfkeOAz7bKWW4D/UVUfHeD+fg94T/vf8jTg/zfAfdFu+n4+vV9bBqb9YnUt8AXgDnp9xrpB7nM+q6qtwFrghvZvd2BfQtrN5n8D3EbvF+VPD2pfbX93Af+J3j0DtwOb6PVBs15VfZ7eJZZfpPf/yS3At0faqO58lN6gE7cDf0rvMsFB2wasbvs8BrhyCPucl+bwd5y5/P0G/I7TiUF+x0mVZ94lSZqOJE+qqu+2LwY3AmvGR0yTJM0vA7lmW5KkeWZdes/0eTy9e80MV5I0T3kGS5IkSZI64j1YkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkuadJCcm2e+5MUnuS3LsKNokae6yz5lfDFialZIsONC8JEnSbJcev6/PMv4H04yR5A+TvLxNvznJJ9r0OUneneS7Sf4kyc3Az02cH2XbJc1KhyVZn+T2JNe2Z1gBkOTIJB9N8n+1+Qvacl9M8q7RNVnSLHZQfU4727UtyRX0HoK7ZHRN1lQYsDST3Aj8QpteBjwpyeHAzwOfBp4I3FlVP1tVn5lkXpIOxTOAdVX1L4DvAP+h1Z8EfAh4b1W9NckpwGuBs6vqWcArRtJaSbPdQfU5fcteXVWnV9XfD7+pmg4DlmaSW4EzkjwZ2APcRC9o/QK9gLUPeH/f8hPnJelQPFBVn23T76b3Yw7AdcA7q+rqNn82cG1VfQ2gqr4x3GZKmiMOts8B+Puq+txQW6fOGLA0Y1TV94H7gN8C/ie9UPVc4GnANuB7VbWvb5WJ85J0KOpR5j8L/FKStPlMsqwkHaqD7XMA/nE4TdIgGLA009wI/Mf2/mngd4HbqsovN5K6dkKS8fs3XwyMX2r8R8DXgSva/GbgN5L8OECSY4baSklzxcH2OZrlDFiaaT4NLARuqqqHgO+1miR1bRuwOsntwDHAlX2fvRJ4fJL/UlVbgbXADUm+CLxp6C2VNBccVJ8zioapW/HEgCRJkiR1wzNYkiRJktQRA5YkSZIkdcSAJUmSJEkdMWBJkiRJUkcMWJIkSZLUEQOWJEmSJHXEgCVJkiRJHfl/AUGdYhOjciewAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x864 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "f, ax = plt.subplots(2,3, figsize=(12,12) )\n",
    "sns.countplot(ax= ax[0,0], x='wkc',\n",
    "             data=data)\n",
    "sns.countplot(ax= ax[0,1], x='wkr',\n",
    "             data=data)\n",
    "sns.countplot(ax= ax[0,2], x='wrc',\n",
    "             data=data)\n",
    "sns.countplot(ax= ax[1,0], x='wrr',\n",
    "             data=data)\n",
    "sns.countplot(ax= ax[1,1], x='bkc',\n",
    "             data=data)\n",
    "sns.countplot(ax= ax[1,2], x='bkr',\n",
    "             data=data)\n",
    "\n",
    "\n",
    "ax[0,0].set_title(\"wkc\")\n",
    "ax[0,1].set_title(\"wkr\")\n",
    "ax[0,2].set_title(\"wrc\")\n",
    "ax[1,0].set_title(\"wrr\")\n",
    "ax[1,1].set_title(\"bkc\")\n",
    "ax[1,2].set_title(\"bkr\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig('foo.pdf', transparent=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11e8852f",
   "metadata": {},
   "source": [
    "Existe una mayor homogeneidad en las variables predictoras correspondiente a la posición de la torres. \n",
    "Es curioso que el rey blanco suele estar en la columna _d_ en la primera fila, mientras que el rey negro presenta una distribución mucho más variada.\n",
    "\n",
    "Todavía no hemos realizado ningúna conversión de los tipos ya que preferimos aplazarlo hasta la aplicación de determinados algoritmos, que nos exijan la codificación de las columnas en un deterimnado tipo."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b544dc88",
   "metadata": {},
   "source": [
    "## Importancia de variables\n",
    "\n",
    "Podemos utilizar algoritmos basados en árboles de decisión como _RandomForest_ para estimar qué variables predictoras importan más a la hora de determinar la variable a predecir.\n",
    "\n",
    "Para ello utilizaremos el paquete _Scikit-learn_. En primer lugar codificaremos numéricamente las variables categóricas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "0f815d35",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_aux = data.copy()\n",
    "data_aux[['wkc', 'wrc', 'bkc']] = data_aux[['wkc', 'wrc', 'bkc']].astype('category')\n",
    "\n",
    "data_aux['wkc'] = data_aux['wkc'].cat.codes\n",
    "data_aux['wrc'] = data_aux['wrc'].cat.codes\n",
    "data_aux['bkc'] = data_aux['bkc'].cat.codes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2deaccf3",
   "metadata": {},
   "source": [
    "Entrenamos un modelo de _RandomForest_:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9dc1bc71",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(random_state=0)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data_aux.drop(\"opt rank\", axis=1), data_aux['opt rank']\n",
    "                                                    ,test_size=0.2)\n",
    "feature_names = [f'feature {i}' for i in range(data_aux.drop('opt rank', axis=1).shape[1])]\n",
    "\n",
    "forest = RandomForestClassifier(random_state=0)\n",
    "forest.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fac08f5e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjEwIDAgb2JqCjw8IC9Bbm5vdHMgWyBdIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDQyNC4zNjI1IDI4MC4wMzQzNzUgXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMSAwIFIgPj4Kc3RyZWFtCniczVfLciM1FN33V2gJC5T70ms5qWFSUDULwFWsQ8ZjEhIDTkKKv+fIbqelHhubFUlV/Di+ukdHOrpXze5uuHjHbvXoyN3h/8Wxu3IX75d/3d4sf7y6dDePAwF/GEzMa5SAL/fTF8nkSU1TAEr911+HYT0gN0ZcIe1qGAJ5rr+p+pi3UcjLyXNNNYH3Eygh+URlh+6HdyBYPg9/ui9Ta0yeagrMtb5vlu5nt3aIEw4xR0EMeYvjXxoISUPOokGK26yOR7pZ5DDE4E1JKHfSSvSEBDNtE4rF4hLVItApQ4tuxf3g/m95LNFHYjLp9DH2gXO0kDuBDZyzVxGmUuEpSQu/FYkZnkp47d0pjKmGJEU6iQ0sQl4pWOSqccrS4W9EpFj2xSTNfCqp+CCZZ0ZtYCH2RpZiNWWTpcPfiEhlBLOGmVnVyMeSy8ysDcwpoHhl7Ol9m6SF34rExF5G2raUYjdyLGlm1gbmCLVGJVrVOGXp8FeRF+9k1xlWKN/oDj6jP9RiDmRAmUoihVM3hQaFnv0Mhkv0gZfhcuEuPrBjcovPaBbsvmGkyuotE84SV63Ji1s8bHvO4tPw1ctvN1+7xZ37djFs6Qc2iClz5Q16Li2b+RjGQuxDofqhY970zELqqVY17Q9PA5/LLRR8FlG4TlVd2XmpJd/MZEs0j/232XK38NnkMfpSVMYFh9GNlGfiNzPxKuhP1LeyhxY9l10lo7PvBhVPNhP+y3y/NUcfEmpNv+otfDZ1Ll7V8rjqVbh+yT8TfvDKgkbWY4eZ0QHy2AyKz/W9UuCGRIc5IqKSWeGGY8JOcWBbKaWI7lvE9kzhMBNTrKaNlBqqBjzFxSReYkrKIVseyfiILDYEB0NdaMkm8CSZZm8hZUuakuzJjilLCDbLmluyCTxJllAWLOECxrHEkUyOKBNGsFoJ1pA14ElbwIOYFKVc4l6YHBEm+JDx0u7YhJ1kstqdMhdLwfbu0FdZx0qkT9uiiuUrZXZSPi6v1+7T8mazvH5cutu1u33443lz+/T3Pud/apNjq6lthn3YtZmhfwyZWgucrqIovF3DKUgYC453XYyfmpaQxIdQytZ+E4qLqKdsUtIuvqnjXNQHpKJZfd8uonHgPA6Zqi/XeYSQKPZVWURwCDVL3A2ZKiYjGc4aLlhdIRWcrMg4WDIOmAodK5YGAvKsADLqWsJhtHFI07rFfd+07X49D1W2g09YSHngIe3hyEMaos9+zGtimxz/kpmqnvE6wltVq1fvSvUu4zq3tVfU/cBq1g/L66dnWA0e/X3zdL2+WT6658fb9cp9fP/dq2GHfwDOTxPcCmVuZHN0cmVhbQplbmRvYmoKMTEgMCBvYmoKOTkyCmVuZG9iagoxNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJw9kEsSwyAMQ/ecQkfA+H+edLpK7r+tDZ1ssBiE9MB9YiKjFieCr8SHBqXDJPBsFYR7MNkRcoTkBE2GsoMkcQ0NBqXCpmOZ78mmddJKrLzRftl3NGaddIotRYd2If/n9SLco+Aa6xk8D2AxyNpKpeyZMFplpq7yqOi1H9PhPQ9Eq8Xl9Qau8NpHN6koKkvq/kR3NNj+kbf7Ht8fmWU4JAplbmRzdHJlYW0KZW5kb2JqCjE3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzQgPj4Kc3RyZWFtCnicMzU3VTBQsLQAEqaG5grmRpYKKYZcQD6IlcsFE8sBs8xMzIAsQ0tklomxIZBlYmGGxDI2sYDKIlgGQBpsTQ7M9ByuNAADcRiTCmVuZHN0cmVhbQplbmRvYmoKMTggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA0OSA+PgpzdHJlYW0KeJwzsjRVMFCwtAAShpbmCuZGlgophlxAPoiVywUTywGzDIA0WGkOTEUOVxoApUQM5AplbmRzdHJlYW0KZW5kb2JqCjE5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggODcgPj4Kc3RyZWFtCnicPY67EcAwCEN7pmAE8wmGfXKpnP3bgD9p0EM6TrgJNgzP0e3CzoE3Qe5FL7Aub4AKIYskGfn2zsWiVpnFr6ZF6oQ0SZw3UehOi0rnA+P0Dng+unUdegplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzA0ID4+CnN0cmVhbQp4nD2SO5LDMAxDe52CF8iM+JPk82Qnlff+7T4yyVaASYkAKC91mbKmPCBpJgn/0eHhYjvld9iezczAtUQvE8spz6ErxNxF+bKZjbqyOsWqwzCdW/SonIuGTZOa5ypLGbcLnsO1ieeWfcQPNzSoB3WNS8IN3dVoWQrNcHX/O71H2Xc1PBebVOrUF48XURXm+SFPoofpSuJ8PCghXHswRhYS5FPRQI6zXK3yXkL2DrcassJBaknnsyc82HV6Ty5uF80QD2S5VPhOUezt0DO+7EoJPRK24VjufTuasekamzjsfu9G1sqMrmghfshXJ+slYNxTJkUSZE62WG6L1Z7uoSimc4ZzGSDq2YqGUuZiV6t/DDtvLC/ZLMiUzAsyRqdNnjh4yH6NmvR5led4/QFs83M7CmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzcgPj4Kc3RyZWFtCnicRVFJcgQhDLv3K/SBqcIr8J5Ozanz/2ssM0lOFmBrMWmBgS14iSHWwMyBL7l8Teg0fDcy2/A62R5wT7gu3JfLgmfClsBXVJd3vS9d2Uh9d4eqfmZke7NIzZCVlTr1QjQm2CERPSMyyVYsc4OkKa1S5b4oW4Au6pW2TjuNkqAjFOFvlCPh6RVKdk1sGqvUOqChCMu2Log6mSSidmFxavGWISKfdWM1x/iLTiJ2x+P+rDDrUSSS0mcH3XEmo02WXQM5uXmqsFYqOYg+XtHGhOp0qoFjvNe29BNp4Ln2X+EHPn3/jxj6ud4/xu5cIgplbmRzdHJlYW0KZW5kb2JqCjIyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjMwID4+CnN0cmVhbQp4nDVRSW7DMAy86xXzgQDiLr/HQU/t/68d0glgYGhLnM0RGxsReInBz0HkxlvWjJr4m8ld8bs8FR4Jt4InUQRehnvZCS5vGJf9OMx88F5aOZMaTzIgF9n08ETIYJdA6MDsGtRhm2kn+oaEz45INRtZTl9L0EurEChP2X6nC0q0rerP7bMutO1rTzjZ7aknlU8gnluyApeNV0wWYxn0ROUuxfRBqrOFnoTyonwOsvmoIRJdopyBJwYHo0A7sOe2n4lXhaB1dZ+2jaEaKR1P/zY0NUki5BMlnNnSuFv4/p57/fwDplRTnwplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjI3ID4+CnN0cmVhbQp4nDVPO7IDIQzrOYUukBmMbWDPs5lUL/dvn2SyDRL+SPL0REcmXubICKzZ8bYWGYgZ+BZT8a897cOE6j24hwjl4kKYYSScNeu4m6fjxb9d5TPWwbsNvmKWFwS2MJP1lcWZy3bBWBoncU6yG2PXRGxjXevpFNYRTCgDIZ3tMCXIHBUpfbKjjDk6TuSJ52KqxS6/72F9waYxosIcVwVP0GRQlj3vJqAdF/Tf1Y3fSTSLXgIykWBhnSTmzllO+NVrR8dRiyIxJ6QZ5DIR0pyuYgqhCcU6OwoqFQWX6nPK3T7/aF1bTQplbmRzdHJlYW0KZW5kb2JqCjI0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ1ID4+CnN0cmVhbQp4nEVQu41DMQzrPQUXCGD9LHued0iV2789SkZwhSFaP5JaEpiIwEsMsZRv4kdGQT0LvxeF4jPEzxeFQc6EpECc9RkQmXiG2kZu6HZwzrzDM4w5AhfFWnCm05n2XNjknAcnEM5tlPGMQrpJVBVxVJ9xTPGqss+N14GltWyz05HsIY2ES0klJpd+Uyr/tClbKujaRROwSOSBk0004Sw/Q5JizKCUUfcwtY70cbKRR3XQydmcOS2Z2e6n7Ux8D1gmmVHlKZ3nMj4nqfNcTn3usx3R5KKlVfuc/d6RlvIitduh1elXJVGZjdWnkLg8/4yf8f4DjqBZPgplbmRzdHJlYW0KZW5kb2JqCjI1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ3ID4+CnN0cmVhbQp4nE1Ru21EMQzr3xRc4ADra3meC1Jd9m9DyQiQwiChLymnJRb2xksM4QdbD77kkVVDfx4/MewzLD3J5NQ/5rnJVBS+FaqbmFAXYuH9aAS8FnQvIivKB9+PZQxzzvfgoxCXYCY0YKxvSSYX1bwzZMKJoY7DQZtUGHdNFCyuFc0zyO1WN7I6syBseCUT4sYARATZF5DNYKOMsZWQxXIeqAqSBVpg1+kbUYuCK5TWCXSi1sS6zOCr5/Z2N0Mv8uCounh9DOtLsMLopXssfK5CH8z0TDt3SSO98KYTEWYPBVKZnZGVOj1ifbdA/59lK/j7yc/z/QsVKFwqCmVuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMzggPj4Kc3RyZWFtCnicRVJLcsUwCNvnFFwgM+Zn4/O8Tlfp/beVcDrdPPQMCAkyPWVIptw2lmSE5BzypVdkiNWQn0aORMQQ3ymhwK7yubyWxFzIbolK8aEdP5elNzLNrtCqt0enNotGNSsj5yBDhHpW6MzuUdtkw+t2Iek6UxaHcCz/QwWylHXKKZQEbUHf2CPobxY8EdwGs+Zys7lMbvW/7lsLntc6W7FtB0AJlnPeYAYAxMMJ2gDE3NreFikoH1W6iknCrfJcJztQttCqdLw3gBkHGDlgw5KtDtdobwDDPg/0okbF9hWgqCwg/s7ZZsHeMclIsCfmBk49cTrFkXBJOMYCQIqt4hS68R3Y4i8Xroia8Al1OmVNvMKe2uLHQpMI71JxAvAiG25dHUW1bE/nCbQ/KpIzYqQexNEJkdSSzhEUlwb10Br7uIkZr43E5p6+3T/COZ/r+xcWuIPgCmVuZHN0cmVhbQplbmRvYmoKMjcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA2OCA+PgpzdHJlYW0KeJwzMrdQMFCwNAEShhYmCuZmBgophlxAvqmJuUIuF0gMxMoBswyAtCWcgohbQjRBlIJYEKVmJmYQSTgDIpcGAMm0FeUKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDgxID4+CnN0cmVhbQp4nD3MuxWAMAgF0D5TvBFCfIDs47HS/VvBRBu4fNUDHSEZ1A1uHYe0rEt3k33qerWJpMiA0lNqXBpOjKhpfal9auC7G+ZL1Yk/zc/nA4fHGWsKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1NSA+PgpzdHJlYW0KeJxFkUuSAyAIRPeegiOA/OQ8mZpVcv/tNJhMNnaXqP2ESiOmEiznFHkw/cjyzWS26bUcq52NAooiFMzkKvRYgdWdKeLMtUS19bEyctzpHYPiDeeunFSyuFHGOqo6FTim58r6qu78uCzKviOHMgVs1jkONnDltmGME6PNVneH+0SQp5Opo+J2kGz4g5PGvsrVFbhONvvqJRgHgn6hCUzyTaB1hkDj5il6cgn28XG780Cwt7wJpGwI5MgQjA5Bu06uf3Hr/N7/OsOd59oMV4538TtMa7vjLzHJirmARe4U1PM9F63rDB3vyZljctN9Q+dcsMvdQabP/B/r9w9QimaICmVuZHN0cmVhbQplbmRvYmoKMzAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjEgPj4Kc3RyZWFtCnicRZBLEsMgDEP3nEJH8EcGfJ50ukrvv60hTbOAp7FABncnBKm1BRPRBS9tS7oLPlsJzsZ46DZuNRLkBHWAVqTjaJRSfbnFaZV08Wg2cysLrRMdZg56lKMZoBA6Fd7touRypu7O+Udw9V/1R7HunM3EwGTlDoRm9SnufJsdUV3dZH/SY27Wa38V9qqwtKyl5YTbzl0zoATuqRzt/QWpczqECmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTQgPj4Kc3RyZWFtCnicPVC7EUMxCOs9BQvkznztN8/Lpcv+bSScpEI2QhKUmkzJlIc6ypKsKU8dPktih7yH5W5kNiUqRS+TsCX30ArxfYnmFPfd1ZazQzSXaDl+CzMqqhsd00s2mnAqE7qg3MMz+g1tdANWhx6xWyDQpGDXtiByxw8YDMGZE4siDEpNBv+tcvdS3O89HG+iiJR08K755fTLzy28Tj2ORLq9+YprcaY6CkRwRmryinRhxbLIQ6TVBDU9A2u1AK7eevk3aEd0GYDsE4njNKUcQ//WuMfrA4eKUvQKZW5kc3RyZWFtCmVuZG9iagozMiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDgwID4+CnN0cmVhbQp4nEWMuw3AMAhEe6ZgBH4mZp8olbN/GyBK3HBPunu4OhIyU95hhocEngwshlPxBpmjYDW4RlKNneyjsG5fdYHmelOr9fcHKk92dnE9zcsZ9AplbmRzdHJlYW0KZW5kb2JqCjMzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjM2ID4+CnN0cmVhbQp4nE1QS25EIQzbc4pc4EkkIQHOQ9VV5/7bscNU7SqGGH9ID+myVR7rU2J1iezypU2XyjJ5FajlT9v/UQwCbv/QyEG0t4ydYuYS1sXCJDzlNCMbJ9csH487TxtmhcbEjeOdLhlgnxYBNVuVzYE5bTo3QLqQGreqs95kUAwi6kLNB5MunKfRl4g5nqhgSncmtZAbXD7VoQNxWr0KuWOLk2/EHFmhwGHQTHHWXwHWqMmyWcggSYYhzn2je5QKjajKeSsVwg+ToRH1htWgBpW5haKp5ZL8HdoCMAW2jHXpDEqBqgDB3yqnfb8BJI1dUwplbmRzdHJlYW0KZW5kb2JqCjM0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDkgPj4Kc3RyZWFtCnicMza0UDBQMDQwB5JGhkCWkYlCiiEXSADEzOWCCeaAWQZAGqI4B64mhysNAMboDSYKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE1NyA+PgpzdHJlYW0KeJxFkLkRQzEIRHNVQQkSsAjqscfRd/+pF/lKtG8ALYevJVOqHyciptzXaPQweQ6fTSVWLNgmtpMachsWQUoxmHhOMaujt6GZh9TruKiquHVmldNpy8rFf/NoVzOTPcI16ifwTej4nzy0qehboK8LlH1AtTidSVAxfa9igaOcdn8inBjgPhlHmSkjcWJuCuz3GQBmvle4xuMF3QE3eQplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzMyID4+CnN0cmVhbQp4nC1SOY4kMQzL/Qp+YADr8vGeHkzU+/90SVUFBapsyzzkcsNEJX4skNtRa+LXRmagwvCvq8yF70jbyDqIa8hFXMmWwmdELOQxxDzEgu/b+Bke+azMybMHxi/Z9xlW7KkJy0LGizO0wyqOwyrIsWDrIqp7eFOkw6kk2OOL/z7FcxeCFr4jaMAv+eerI3i+pEXaPWbbtFsPlmlHlRSWg+1pzsvkS+ssV8fj+SDZ3hU7QmpXgKIwd8Z5Lo4ybWVEa2Fng6TGxfbm2I+lBF3oxmWkOAL5mSrCA0qazGyiIP7I6SGnMhCmrulKJ7dRFXfqyVyzubydSTJb90WKzRTO68KZ9XeYMqvNO3mWE6VORfgZe7YEDZ3j6tlrmYVGtznBKyV8NnZ6cvK9mlkPyalISBXTugpOo8gUS9iW+JqKmtLUy/Dfl/cZf/8BM+J8AQplbmRzdHJlYW0KZW5kb2JqCjM3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTcgPj4Kc3RyZWFtCnicMza0UDCAwxRDLgAalALsCmVuZHN0cmVhbQplbmRvYmoKMzggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMzEgPj4Kc3RyZWFtCnicRY/LDQQhDEPvVOES8hk+qYfVntj+r+swmkFC+EEiO/EwCKzz8jbQxfDRosM3/jbVq2OVLB+6elJWD+mQh7zyFVBpMFHEhVlMHUNhzpjKyJYytxvhtk2DrGyVVK2DdjwGD7anZasIfqltYeos8QzCVV64xw0/kEutd71Vvn9CUzCXCmVuZHN0cmVhbQplbmRvYmoKMzkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMzggPj4Kc3RyZWFtCnicNVI5rt1ADOt9Cl0ggHbNnOcFqX7u34aUXwpDtFaKmo4WlWn5ZSFVLZMuv+1JbYkb8vfJCokTklcl2qUMkVD5PIVUv2fLvL7WnBEgS5UKk5OSxyUL/gyX3i4c52NrP48jdz16YFWMhBIByxQTo2tZOrvDmo38PKYBP+IRcq5YtxxjFUgNunHaFe9D83nIGiBmmJaKCl1WiRZ+QfGgR61991hUWCDR7RxJcIyNUJGAdoHaSAw5sxa7qC/6WZSYCXTtiyLuosASScycYl06+g8+dCyovzbjy6+OSvpIK2tM2nejSWnMIpOul0VvN299PbhA8y7Kf17NIEFT1ihpfNCqnWMomhllhXccmgw0xxyHzBM8hzMSlPR9KH5fSya6KJE/Dg2hf18eo4ycBm8Bc9GftooDF/HZYa8cYIXSxZrkfUAqE3pg+v/X+Hn+/AMctoBUCmVuZHN0cmVhbQplbmRvYmoKNDAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDggPj4Kc3RyZWFtCnicLVE5kgNBCMvnFXpCc9PvscuR9//pCsoBg4ZDIDotcVDGTxCWK97yyFW04e+ZGMF3waHfynUbFjkQFUjSGFRNqF28Hr0HdhxmAvOkNSyDGesDP2MKN3pxeEzG2e11GTUEe9drT2ZQMisXccnEBVN12MiZw0+mjAvtXM8NyLkR1mUYpJuVxoyEI00hUkih6iapM0GQBKOrUaONHMV+6csjnWFVI2oM+1xL29dzE84aNDsWqzw5pUdXnMvJxQsrB/28zcBFVBqrPBAScL/bQ/2c7OQ33tK5s8X0+F5zsrwwFVjx5rUbkE21+Dcv4vg94+v5/AOopVsWCmVuZHN0cmVhbQplbmRvYmoKNDEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNzEgPj4Kc3RyZWFtCnicTZBNDkIhEIP3nKIXMKHzA4/zaFzp/bd28PnigvRLIUOnwwMdR+JGR4bO6HiwyTEOvAsyJl6N85+M6ySOCeoVbcG6tDvuzSwxJywTI2BrlNybRxT44ZgLQYLs8sMXGESka5hvNZ91k35+u9Nd1KV199MjCpzIjlAMG3AF2NM9DtwSzu+aJr9UKRmbOJQPVBeRstkJhailYpdTVWiM4lY974te7fkBwfY7+wplbmRzdHJlYW0KZW5kb2JqCjQyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggODggPj4Kc3RyZWFtCnicNYy7EcAwCEN7T8EIBouP98mlSvZvg+3QgKR394KDOkHyuBspnC5u2Vd6G4+TniYAsfRMQ+3fYEXVi1oULV9uY9BiKr4/+iQglnXyXjj0kBLeH8UXHXsKZW5kc3RyZWFtCmVuZG9iago0MyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzOCA+PgpzdHJlYW0KeJw9j0EOAzEIA+95hT8QKXZCWN6zVU/b/19Lmt1e0AiMMRZCQ2+oag6bgg3Hi6VLqNbwKYqJSg7ImWAOpaTSHWeRemI4GNwetBvO4rHp+hG7klZ90OZGuiVogkfsU2nclnETxAM1Beop6lyjvBC5n6lX2DSS3bSykms4pt+956nr/9NV3l9f3y6MCmVuZHN0cmVhbQplbmRvYmoKNDQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTAgPj4Kc3RyZWFtCnicNVDLDUMxCLtnChaoFAKBZJ5WvXX/a23QO2ER/0JYyJQIeanJzinpSz46TA+2Lr+xIgutdSXsypognivvoZmysdHY4mBwGiZegBY3YOhpjRo1dOGCpi6VQoHFJfCZfHV76L5PGXhqGXJ2BBFDyWAJaroWTVi0PJ+QTgHi/37D7i3koZLzyp4b+Ruc7fA7s27hJ2p2ItFyFTLUszTHGAgTRR48eUWmcOKz1nfVNBLUZgtOlgGuTj+MDgBgIl5ZgOyuRDlL0o6ln2+8x/cPQABTtAplbmRzdHJlYW0KZW5kb2JqCjE0IDAgb2JqCjw8IC9CYXNlRm9udCAvRGVqYVZ1U2FucyAvQ2hhclByb2NzIDE1IDAgUgovRW5jb2RpbmcgPDwKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDQ2IC9wZXJpb2QgNDggL3plcm8gL29uZSAvdHdvIC90aHJlZSA1MyAvZml2ZSA2OCAvRCA3MCAvRiA3MwovSSA3NyAvTSA5NyAvYSAvYiAvYyAvZCAvZSAxMDMgL2cgMTA1IC9pIDEwNyAvayAxMDkgL20gL24gL28gL3AgMTE0IC9yIC9zCi90IC91IDExOSAvdyAxMjEgL3kgXQovVHlwZSAvRW5jb2RpbmcgPj4KL0ZpcnN0Q2hhciAwIC9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnREZXNjcmlwdG9yIDEzIDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9EZWphVnVTYW5zCi9TdWJ0eXBlIC9UeXBlMyAvVHlwZSAvRm9udCAvV2lkdGhzIDEyIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0FzY2VudCA5MjkgL0NhcEhlaWdodCAwIC9EZXNjZW50IC0yMzYgL0ZsYWdzIDMyCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnROYW1lIC9EZWphVnVTYW5zIC9JdGFsaWNBbmdsZSAwCi9NYXhXaWR0aCAxMzQyIC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCAwID4+CmVuZG9iagoxMiAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxNSAwIG9iago8PCAvRCAxNiAwIFIgL0YgMTcgMCBSIC9JIDE4IDAgUiAvTSAxOSAwIFIgL2EgMjAgMCBSIC9iIDIxIDAgUiAvYyAyMiAwIFIKL2QgMjMgMCBSIC9lIDI0IDAgUiAvZml2ZSAyNSAwIFIgL2cgMjYgMCBSIC9pIDI3IDAgUiAvayAyOCAwIFIgL20gMjkgMCBSCi9uIDMwIDAgUiAvbyAzMSAwIFIgL29uZSAzMiAwIFIgL3AgMzMgMCBSIC9wZXJpb2QgMzQgMCBSIC9yIDM1IDAgUgovcyAzNiAwIFIgL3NwYWNlIDM3IDAgUiAvdCAzOCAwIFIgL3RocmVlIDM5IDAgUiAvdHdvIDQwIDAgUiAvdSA0MSAwIFIKL3cgNDIgMCBSIC95IDQzIDAgUiAvemVybyA0NCAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE0IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL0NBIDAgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMSA+PgovQTIgPDwgL0NBIDEgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCA+PgplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDEwIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKNDUgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIxMDYwNzE3NDEzOCswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjMuNCkgPj4KZW5kb2JqCnhyZWYKMCA0NgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAxMTAwNCAwMDAwMCBuIAowMDAwMDEwODEwIDAwMDAwIG4gCjAwMDAwMTA4NDIgMDAwMDAgbiAKMDAwMDAxMDk0MSAwMDAwMCBuIAowMDAwMDEwOTYyIDAwMDAwIG4gCjAwMDAwMTA5ODMgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzk3IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMTQ2NCAwMDAwMCBuIAowMDAwMDA5NDIyIDAwMDAwIG4gCjAwMDAwMDkyMjIgMDAwMDAgbiAKMDAwMDAwODc2NCAwMDAwMCBuIAowMDAwMDEwNDc1IDAwMDAwIG4gCjAwMDAwMDE0ODQgMDAwMDAgbiAKMDAwMDAwMTcxNyAwMDAwMCBuIAowMDAwMDAxODYzIDAwMDAwIG4gCjAwMDAwMDE5ODQgMDAwMDAgbiAKMDAwMDAwMjE0MyAwMDAwMCBuIAowMDAwMDAyNTIwIDAwMDAwIG4gCjAwMDAwMDI4MzAgMDAwMDAgbiAKMDAwMDAwMzEzMyAwMDAwMCBuIAowMDAwMDAzNDMzIDAwMDAwIG4gCjAwMDAwMDM3NTEgMDAwMDAgbiAKMDAwMDAwNDA3MSAwMDAwMCBuIAowMDAwMDA0NDgyIDAwMDAwIG4gCjAwMDAwMDQ2MjIgMDAwMDAgbiAKMDAwMDAwNDc3NSAwMDAwMCBuIAowMDAwMDA1MTAzIDAwMDAwIG4gCjAwMDAwMDUzMzcgMDAwMDAgbiAKMDAwMDAwNTYyNCAwMDAwMCBuIAowMDAwMDA1Nzc2IDAwMDAwIG4gCjAwMDAwMDYwODUgMDAwMDAgbiAKMDAwMDAwNjIwNiAwMDAwMCBuIAowMDAwMDA2NDM2IDAwMDAwIG4gCjAwMDAwMDY4NDEgMDAwMDAgbiAKMDAwMDAwNjkzMCAwMDAwMCBuIAowMDAwMDA3MTM0IDAwMDAwIG4gCjAwMDAwMDc1NDUgMDAwMDAgbiAKMDAwMDAwNzg2NiAwMDAwMCBuIAowMDAwMDA4MTEwIDAwMDAwIG4gCjAwMDAwMDgyNzAgMDAwMDAgbiAKMDAwMDAwODQ4MSAwMDAwMCBuIAowMDAwMDExMDY0IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gNDUgMCBSIC9Sb290IDEgMCBSIC9TaXplIDQ2ID4+CnN0YXJ0eHJlZgoxMTIyMQolJUVPRgo=\n",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAf1UlEQVR4nO3de7wdVX338c+XIBeRGDCBQAIGEKHgAy2Ei0JLoKAEhNiKilhRLsaolIenRY13UIto66U+QiMiyEVFQcBQoqDSIHLRBOWuaAwgIWDCHUEDCd/+MXNg53Auc04yZ0/2/r5fr/M6e9bMmv2bvZPzm7VmzRrZJiIiomnWancAERERfUmCioiIRkqCioiIRkqCioiIRkqCioiIRkqCioiIRkqCiq4k6cOSzmx3HN0kn3kMlXIfVAyVpLuBTYEVLcWvtL14Ffd5rO0fr1p0ax5JJwGvsP1P7Y5lTSXJwBJggu3lZdnawGJgnG2VZXOBPYFnAAO/Ay4Evmh7WbnNSeT7aIS0oGK4DrH9kpafYSen1aH8Y7TGWVPjbqhHgaktywcBj/Sx3XG2NwQ2A/4VOByYI0m1RxhDkgQVq42kl0r6uqT7Jd0n6dOSRpXrtpF0laSHJD0o6ZuSxpTrzgO2BC6T9CdJH5A0RdKiXvu/W9L+5euTJF0k6XxJjwPvHOj9+4j1JEnnl68nSbKkoyTdK+kRSTMk7SbpFkmPSvpKS913SrpW0v+X9Jik30j6+5b1m0uaLelhSQskvavX+7bGPQP4MPCW8thvLrc7StKvJT0haaGkd7fsY4qkRZL+VdKS8niPalm/vqTPS7qnjO9nktYv1+0p6brymG6WNKXXcS0s3/MuSW/r57P7hqRP946nZfmD5ef/hKQ7ez6bfj7zd0j6Q/lv4iO9juGc8rv4dflvYqV/D304DziyZflI4Nz+Nrb9pO25wKHAq4GDB9l/jLAkqFidzgGWA68A/gZ4LXBsuU7AZ4DNgb8CtgBOArD9duAPPN8q+1zF95sGXASMAb45yPtXsQewLfAW4EvAR4D9gR2BN0vap9e2C4GxwCeAiyVtXK77NrCoPNbDgFNaE1ivuL8OnAJ8pzz2ncttlgCvB0YDRwFflLRLyz7GAy8FJgDHAKdJ2qhc9x/ArsBrgI2BDwDPSpoAXA58uiw/EfiepHGSNgC+DEwtWxevAW4awmcHgKTtgOOA3cr9vA64e4AqewPbAX8PfFzSX5XlnwAmAVsDBwBVutsuBf5O0pjy5Odvge8PVsn2H4D55fbRIElQMVyXlmfhj0q6VNKmFN0rJ5RnpkuAL1J0n2B7ge0f2V5meynwBWCf/ndfyfW2L7X9LMUf8n7fv6JP2f6L7SuBJ4Fv215i+z7gGoqk12MJ8CXbz9j+DnAncLCkLSj+6H6w3NdNwJnA2/uK2/af+wrE9uW2f+/C1cCVrPwH9Bngk+X7zwH+BGwnaS3gaOD/2r7P9grb15XXV/4JmGN7TvneP6L4w3xQuc9ngVdJWt/2/bZvH8Jn12MFsC6wg6QX2b7b9u8H2P5k23+2fTNwM9CToN8MnGL7EduLKJLnYP4CXEZxgnE4MLssq2IxRdKOBkmCiuF6g+0x5c8bgJcDLwLu70lcwFeBTQAkbSLpgrLr53HgfIrWx6q4t+X1gO9f0R9bXv+5j+WXtCzf55VHGN1D0WLaHHjY9hO91k3oJ+4+SZoq6Yaym/BRiiTS+nk91DMYoPRUGd9YYD2gr6TwcuBNLScWj1Ik081sP0nxh30GxWd4uaTtB4uzN9sLgBMoWsdLyu988wGqPNDHMUDxObZ+ToN+ZqVzKbr2Buze68ME4OEhbB8jIAkqVpd7gWXA2JbENdr2juX6z1CMmtrJ9miKs/nWi9K9h5M+Cby4Z6G8ljSu1zatdQZ7/9VtgrTSRfUtKc7CFwMbS9qw17r7+on7BcuS1gW+R9FVt6ntMcAcVv68+vMgRathmz7W3Quc1/L5jLG9ge1TAWxfYfsAisEDvwG+1s97rPTdUHQ3Pn8w9rds702REA18tkLcvd0PTGxZ3qJivWso4t8U+FmVCmWrd9eybjRIElSsFrbvp+iG+ryk0ZLWUjEwoqcbb0OKbqhHy2sh7++1iz9SXG/o8VtgPUkHS3oR8FGKrqPhvv/qtglwvKQXSXoTxXW1ObbvBa4DPiNpPUk7UVwj+uYA+/ojMKnsngNYh+JYlwLLJU2luJ42qLK78yzgC+VgjVGSXl0mvfOBQyS9rixfrxzgMFHSppIOLa9FLaP4rlb08zY3AQdJ2ljSeIoWE1Bcg5K0X/l+f6Foefa3n4F8F/iQpI3Kfy/HVTx+A4cAh/Zq4b6ApBeX/z6+D/yC4iQgGiQJKlanIyn+uN5BMbz3IoqzWYCTgV2Axygu1F/cq+5ngI+WXU8n2n4MeC/F9Zv7KM7aBxvFNdD7r24/pxhQ8SDwb8Bhth8q172V4gL/YuAS4BPl9Z7+XFj+fkjSL8vuweMp/kg/AhxBcT2lqhOBW4F5FN1WnwXWKpPnNIpRg0spWlTvp/g7sBbFkOvFZZ19KD7/vpxHcb3oboqTgu+0rFsXOJXic3mAIpF/eAix9/gkxfd9F/Bjiu9yWZWKtm8f5PrZVyQ9QXFi8CWK1uqBZXKPBsmNuhFDJOmdFDcV793uWLqFpPcAh9uuq0UcDZQWVEQ0jqTNJO1VdtVuR9G6u6TdccXIyl3sEdFE61CMwtyKYoaIC4DT2xlQjLx08UVERCOliy8iIhqpo7r4xo4d60mTJrU7jIiIGIIbb7zxQdu973PsrAQ1adIk5s+f3+4wIiJiCCTd01d5uvgiIqKRkqAiIqKRak1Qkg4snwezQNLMPtZPU/G8nZskzZe0d9W6ERHR2WpLUOXknqdRPAJhB+CtknbotdlPgJ1t/zXFIwLOHELdiIjoYHW2oHYHFtheaPtpihvtprVuYPtPLRM6bsDzszoPWjciIjpbnQlqAis/w2URKz8TBwBJ/yDpNxQTiB49lLpl/ell9+D8pUuXrpbAIyKi/epMUH09u+YF01bYvsT29sAbgE8NpW5Z/wzbk21PHjfuBcPoIyJiDVVnglrEyg8Zm0gxlX+fbP8U2EbS2KHWjYiIzlNngpoHbCtpK0nrAIfT65k2kl7R81RSSbtQTBD5UJW6ERHR2WqbScL2cknHAVcAo4CzbN8uaUa5fhbwRuBISc9QPHnzLeWgiT7r1hVrRBVTpkwBYO7cuW2NI6Jb1DrVke059HqMcpmYel5/luJpn5XqRkRE98hMEhER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBER0UhJUBHRrylTpjBlypR2hxFdKgkqIiIaae12BxAxkibNvHzYdR9Y+NAq7ePuUw8e9ntHdKO0oCIiopFqTVCSDpR0p6QFkmb2sf5tkm4pf66TtHPLursl3SrpJknz64wzIiKap7YuPkmjgNOAA4BFwDxJs23f0bLZXcA+th+RNBU4A9ijZf2+th+sK8aIiGiuOltQuwMLbC+0/TRwATCtdQPb19l+pFy8AZhYYzwREbEGqTNBTQDubVleVJb15xjgBy3LBq6UdKOk6f1VkjRd0nxJ85cuXbpKAUdERHPUOYpPfZS5zw2lfSkS1N4txXvZXixpE+BHkn5j+6cv2KF9BkXXIJMnT+5z/xERseapswW1CNiiZXkisLj3RpJ2As4Eptl+qKfc9uLy9xLgEoouw4iI6BJ1Jqh5wLaStpK0DnA4MLt1A0lbAhcDb7f925byDSRt2PMaeC1wW42xRkREw9TWxWd7uaTjgCuAUcBZtm+XNKNcPwv4OPAy4HRJAMttTwY2BS4py9YGvmX7h3XFGhERzTNogirvQTqbIkk8Mtj2rWzPAeb0KpvV8vpY4Ng+6i0Edu5dHhER3aNKF9/hwOYU9zFdIOl1Kps2ERERdRk0QdleYPsjwCuBbwFnAX+QdLKkjesOMCIiulOlQRLlSLvPA/8OfA84DHgcuKq+0CIioptVuQZ1I/Ao8HVgpu1l5aqfS9qrxtgiIqKLVRnF96Zy0MJzJG1l+y7b/1hTXBER0eWqdPFdVLEsIiJitem3BSVpe2BH4KWSWltKo4H16g4sIqIdeh5xP3fu3LbGEQN38W0HvB4YAxzSUv4E8K4aY4qIiOg/Qdn+PvB9Sa+2ff0IxhQRq1Eecx9rqoG6+D5g+3PAEZLe2nu97eNrjSwiIrraQF18vy5/53HrEREx4gbq4rusfGz7q2y/fwRjioiIGHiYue0VwK4jFEtERMRzqtyo+ytJs4ELgSd7Cm1fXFtUERHR9aokqI2Bh4D9WspM8aDBiIiIWgyaoGwfNRKBREREtKoyWezZFC2mldg+upaIIhpq/BGntjuEiK5SpYvvv1terwf8A7C4nnAiIiIKVbr4vte6LOnbwI9riygiIoKKDyzsZVtgy9UdSEREtMeUKVOemyS3Sapcg3qC4hqUyt8PAB+sOa6IiOhyVbr4NhyJQCIiIlpVGSRB+TyovSlaUNfYvrTOoCIiIga9BiXpdGAGcCtwGzBD0ml1BxYREd2tSgtqH4oJYw0g6RyKZBUREVGbKqP47mTlUXtbALfUE05EREShSgvqZcCvJf2iXN4NuL6cQBbbh9YVXEREdK8qCerjw925pAOB/wRGAWfaPrXX+rfx/JD1PwHvsX1zlboREdHZqgwzvxpA0ujW7W0/PFC98mGHpwEHAIuAeZJm276jZbO7gH1sPyJpKnAGsEfFuhFRs8w/GO1U5Ubd6cCngD8Dz/L8DbtbD1J1d2CB7YXlfi4ApgHPJRnb17VsfwMwsWrdiIjobFW6+N4P7Gj7wSHuewJwb8vyImCPAbY/BvjBMOtGRESHqZKgfg88NYx9q4+yFzy2A0DSvhQJau9h1J0OTAfYcstMERgR0SmqJKgPAddJ+jmwrKfQ9vGD1FtEMSS9x0T6eEyHpJ2AM4Gpth8aSt0yjjMorl0xefLkPpNYRESseaokqK8CV1HcnPvsEPY9D9hW0lbAfcDhwBGtG0jakuLR8W+3/duh1I2IiM5WJUEtt/0vQ92x7eWSjgOuoBgqfpbt2yXNKNfPohjC/jLgdEk97zW5v7pDjSEiutekmZcPq94DCx9apfp3n3rwsOrFC1VJUP9TXue5jJW7+AYcZl5uMweY06tsVsvrY4Fjq9aNiIjuUSVB9XStfailrMow84iIiGGrcqPuViMRSERERKt+E5Sk/WxfVT4L6gVsX1xfWBER0e0GakHtQzF675A+1pli9F1EREQt+k1Qtj9R/j5q5MKJiIgoVHkeVERExIhLgoqIiEZKgoqIiEaqch8Ukl4DTGLl50GdW1NMERERlZ4HdR6wDXATsKIsNpAEFRERtanSgpoM7GA7M4VHRMSIqXIN6jZgfN2BREREtKrSghoL3CHpF6w8WeyhtUUVERFD1mkzuFdJUCfV8s4REREDqDJZ7NUjEUhERESrgSaL/ZntvSU9QTFq77lVgG2Prj26iIjoWgPNxbd3+XvDkQsnIiKikJkkIiKikZKgIiKikZKgIiKikSolKEkvl7R/+Xp9SbkuFRERtRo0QUl6F3AR8NWyaCJwaY0xRUREVGpBvQ/YC3gcwPbvgE3qDCoiIqJKglpm++meBUlrs/J9UREREatdlQR1taQPA+tLOgC4ELis3rAiIqLbVUlQM4GlwK3Au4E5wEfrDCoiIqLKXHzPAl8DviZpY2Bing0VEZ1q/BGntjuEKFUZxTdX0ugyOd0EnC3pC7VHFhERXa1KF99LbT8O/CNwtu1dgf2r7FzSgZLulLRA0sw+1m8v6XpJyySd2Gvd3ZJulXSTpPlV3i8iIjpHledBrS1pM+DNwEeq7ljSKOA04ABgETBP0mzbd7Rs9jBwPPCGfnazr+0Hq75nRER0jiotqE8CVwALbM+TtDXwuwr1di/rLCyHqV8ATGvdwPYS2/OAZ4YYd0REdLhBE5TtC23vZPu95fJC22+ssO8JwL0ty4vKsqoMXCnpRknT+9tI0nRJ8yXNX7p06RB2HxERTTZoF5+k9YBjgB2B9XrKbR89WNU+yoYy+m8v24slbQL8SNJvbP/0BTu0zwDOAJg8eXJGF0ZEdIgqXXznAeOB1wFXU8zF90SFeouALVqWJwKLqwZme3H5ewlwCUWXYUREdIkqCeoVtj8GPGn7HOBg4P9UqDcP2FbSVpLWAQ4HZlcJStIGPTOmS9oAeC1wW5W6ERHRGaqM4usZwPCopFcBDwCTBqtke7mk4ygGWIwCzrJ9u6QZ5fpZksYD84HRwLOSTgB2AMYCl0jqifFbtn84lAOLiIg1W5UEdYakjYCPUbSAXgJ8vMrObc+hmBqptWxWy+sHKLr+ensc2LnKe0RERGeqMtXRmeXLq4Gt6w0nIiKiUGWqo00lfV3SD8rlHSQdU39oERHRzaoMkvgGxXWkzcvl3wIn1BRPREQEUC1BjbX9XeBZKAY/ACtqjSoiIrpelQT1pKSXUd5kK2lP4LFao4qIiK5XZRTfv1CM3ttG0rXAOOCwWqOKiIgR09RnYA2YoMoZyfcpf7ajmL7oTtuZ3DUiImo1YBef7RXANNvLbd9u+7Ykp4iIGAlVuviulfQV4DvAkz2Ftn9ZW1QREdH1qiSo15S/P9lSZmC/1R9OREREocpMEvuORCARERGtqswkcYqkMS3LG0n6dK1RRURE16tyH9RU24/2LNh+BDiotogiIiKolqBGSVq3Z0HS+sC6A2wfERGxyqoMkjgf+ImksykGRxwNnFNrVBER0fWqDJL4nKRbgP0pbtT9lO0rao8sIiK6WpUWFMCvgeW2fyzpxZI2tP1EnYFFRER3qzKK713ARcBXy6IJwKU1xhQREVFpkMT7gL0oHsOO7d8Bm9QZVERERJUEtcz20z0LktamfPRGREREXaokqKslfRhYX9IBwIXAZfWGFRER3a5KgpoJLAVuBd4NzAE+WmdQERERVYaZPwt8rfyJiIgYEf0mKEm3MsC1Jts71RJRREQEA7egXl/+fl/5+7zy99uAp2qLKCIiggESlO17ACTtZXuvllUzJV3Lys+HioiIWK2qDJLYQNLePQuSXgNsUF9IERER1RLUMcBpku6WdBdwOsWEsYOSdKCkOyUtkDSzj/XbS7pe0jJJJw6lbkREdLYqo/huBHaWNBqQ7ceq7FjSKOA04ABgETBP0mzbd7Rs9jBwPPCGYdSNiIgOVqUFBYDtx6smp9LuwALbC8uZKC4ApvXa5xLb84Bnhlo3IiI6W+UENQwTgHtblheVZau1rqTpkuZLmr906dJhBRoREc1TZ4JSH2VV5/CrXNf2GbYn2548bty4ysFFRESzVXoeVDlyb1Lr9rbPHaTaImCLluWJwOKKca1K3ajZlClTAJg7d25b44iIzjZogpJ0HrANcBOwoiw2MFiCmgdsK2kr4D7gcOCIinGtSt2IiOgAVVpQk4EdbA/pERu2l0s6DrgCGAWcZft2STPK9bMkjQfmA6OBZyWdUL7X433VHcr7R0TEmq1KgroNGA/cP9Sd255DMft5a9msltcPUHTfVaobERHdo0qCGgvcIekXwLKeQtuH1hZVRER0vSoJ6qS6g4iIiOitykwSV49EIBEREa0GvQ9K0p6S5kn6k6SnJa2Q9PhIBBcREd2ryo26XwHeCvwOWB84tiyLiIioTaUbdW0vkDTK9grgbEnX1RxXRER0uSoJ6ilJ6wA3SfocxXDzPA+qlFkVIiLqUaWL7+3ldscBT1JMQfTGOoOKiIioMorvHknrA5vZPnkEYoqIiKg0iu8Qinn4flgu/7Wk2TXHFRERXa5KF99JFA8QfBTA9k0UM5tHRETUpkqCWj7EJ+lGRESsskqTxUo6AhglaVvgeCDDzDvApJmXD6veAwsfWqX6AHefevCw60ZEd6jSgvpnYEeKiWK/DTwOnFBjTBEREZVG8T0FfKT8iYiIGBH9JqjBRurlcRsREVGngVpQrwbupejW+zmgEYkoIiKCgRPUeOAAiolijwAuB76dR69HRMRI6HeQhO0Vtn9o+x3AnsACYK6kfx6x6CIiomsNOEhC0rrAwRStqEnAl4GL6w8rIiK63UCDJM4BXgX8ADjZ9m0jFlUbtOueoNwPFBHRt4FaUG+nmL38lcDx0nNjJATY9uiaY4uIiC7Wb4KyXeUm3oiIiFokCUVERCMlQUVERCMlQUVERCMlQUVERCPVmqAkHSjpTkkLJM3sY70kfblcf4ukXVrW3S3pVkk3SZpfZ5wREdE8VZ4HNSySRgGnUUyXtAiYJ2m27TtaNpsKbFv+7AH8V/m7x762H6wrxhie8Uec2u4QIqIL1NmC2h1YYHuh7aeBC4BpvbaZBpzrwg3AGEmb1RhTRESsIepMUBMoZkPvsagsq7qNgSsl3Shpen9vImm6pPmS5i9dunQ1hB0REU1QZ4Lq6/EcHsI2e9nehaIb8H2S/q6vN7F9hu3JtiePGzdu+NFGRESj1JmgFgFbtCxPBBZX3cZ2z+8lwCUUXYYREdEl6kxQ84BtJW0laR3gcKD3U3pnA0eWo/n2BB6zfb+kDSRtCCBpA+C1QEdPVhsRESurbRSf7eWSjgOuAEYBZ9m+XdKMcv0sYA5wEMWzpp4CjiqrbwpcUk5QuzbwLds/rCvWVZERbRER9agtQQHYnkORhFrLZrW8NvC+PuotBHauM7aIiGi2zCQRERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNlAQVERGNVGuCknSgpDslLZA0s4/1kvTlcv0tknapWjciIjpbbQlK0ijgNGAqsAPwVkk79NpsKrBt+TMd+K8h1I2IiA5WZwtqd2CB7YW2nwYuAKb12mYacK4LNwBjJG1WsW5ERHSwtWvc9wTg3pblRcAeFbaZULEuAJKmU7S+AP4k6c5ViHm4xgIPDqeiPruaIxk5wz5m6M7jzjGvUXLMQ7AajvnlfRXWmaDUR5krblOlblFonwGcMbTQVi9J821PbmcMI60bjxm687hzzN2hicdcZ4JaBGzRsjwRWFxxm3Uq1I2IiA5W5zWoecC2kraStA5wODC71zazgSPL0Xx7Ao/Zvr9i3YiI6GC1taBsL5d0HHAFMAo4y/btkmaU62cBc4CDgAXAU8BRA9WtK9bVoK1djG3SjccM3XncOebu0Lhjlt3npZ2IiIi2ykwSERHRSElQERHRSElQERHRSElQEfEcSWtJek2744j6laOntxh8y/ZJgopKyj9ct7U7jqiX7WeBz7c7jqifixFyl7Y7joEkQQ2TpFMkjWlZ3kjSp9sYUq3KP1w3S9qy3bGMJEnn9PE9n9XGkEbClZLeKKmvGV06Vpd+1zdI2q3dQfQnw8yHSdKvbP9Nr7Jf2t6lvzprOklXAbsBvwCe7Cm3fWjbgqpZP9/zC8o6iaQngA2A5cBfKKYes+3RbQ2sZl36Xd8BvBK4h+L/dM93vVNbAyvVOdVRpxslaV3bywAkrQ+s2+aY6nZyuwNog7UkbWT7EQBJG9PB/28krQUcaPvadsfSBl31XZemtjuAgXT6h1+n84GfSDqbYiLbo4Fz2xtS7V5s+wetBeXMIFe3KZ6R8HngOkkXUXzPbwb+rb0h1cf2s5L+A3h1u2Npg76+61PaG1Lt9rf99dYCSacCjXhIbLr4VoGkA4H9KZrFV9q+os0h1UrSdcBHbV9VLn8QmGK70Wdhw1W2JvYEHgX2o/ief2L7jnbGVTdJJwO3ABe7y/5AlA9G7abv+gfA+ba/WS6fDqxr+5j2RlZIghomSVP7ak2Ucwx2JEljgf8G3g8cCGwPHG77mbYGViNJ19vuqtZEeQ3qxcAKYFlZ3A3XoI7pqzVhuxGtiTqUlyZmA2dRdPc9bPuEtgbVIqP4hu9jkvbrWShbEx391F/bDwKHAqcBmwOHdXJyKnXjiLZLgfcAO9vesPzp6ORUOkzS23oWytbEuDbGUxtJG5fX2NYHjgU+CDwOfLIsb4S0oIapm1oT5Rl16z+UdShGeJkOP7PuxtZEeeK1N/C3wNbAr4BrbP9nWwOrWdNbE6uTpLt4/uGwrb8BsL11m0JbSRLUKpC0CfBj4Ebg6E7vr5d0HnAN8FPbv2l3PCOh5Zivsf3rdsczUiSNorilYF9gBvBn29u3N6p69GoxbAh8H/gZ8HEA2w+3I66RUCbl91KckJji3/os239ua2ClJKgh6vLWRNedWXfpMf+E4j6o6yn+YP3M9pL2RlWfNaU1UQdJ36Xo2vtmWfRWYIztN7cvquclQQ1TN7YmoLvOrHt02zFL+iKwK0WX5rXAT4Hrm3JWXZemtybqIOlm2zsPVtYuSVDDlDPrzj+zhu485h6SXkLxlOsTgfG2O/pG9Ka3Juog6RsUSfiGcnkP4B2239vWwEq5UXeYbF8l6WpWPrN+FdCxCYri3phdKY7zMeDRchh2x55h0oXHLOk4ihOvXSmmwDmLIjl3uu16tRz+R9LNbYumRpJupWglvgg4UtIfyuWXA4259ysJapj6OLPerdPPrG3/P1jpzPpsYDwdPMVTNx4zxdDjLwA32l7e7mBG0K8k7dmrNdGpUz69vt0BVJEENXw5s+6CM+tuPGbb/97uGEbSmtKaWJ1s39PuGKpIghqmnFl3zZl1Nx5zt1kjWhPdKIMkhqmPM+ufUgySuKqtgUVEdIi0oIYvZ9YRETVKCyoiIhopk8VGREQjJUFFREQjJUFFREQjJUFFREQj/S8XoDSdNPtODAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "importances = forest.feature_importances_\n",
    "std = np.std([\n",
    "    tree.feature_importances_ for tree in forest.estimators_], axis=0)\n",
    "\n",
    "forest_importances = pd.Series(importances, index=feature_names)\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "forest_importances.plot.bar(yerr=std, ax=ax)\n",
    "ax.set_title(\"Feature importances using MDI\")\n",
    "ax.set_ylabel(\"Mean decrease in impurity\")\n",
    "ax.set_xticks([0,1,2,3,4,5]) # values\n",
    "labels = list(data_aux.columns[:-1])\n",
    "ax.set_xticklabels(labels) # labels\n",
    "positions = (1, 2, 3,4, 5)\n",
    "\n",
    "\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a73c4f3e",
   "metadata": {},
   "source": [
    "Las variables más importantes a la hora de determinar el número de pasos son las coordenadas de la torre junto a las coordenadas del rey negro. Parece que la posición del rey blanco no es tan importante como el papel de la torre a la hora de ir _acorralando_ al rey negro."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ac7a41d",
   "metadata": {},
   "source": [
    "Podemos dar por finalizado el análisis exploratorio de datos, ya que no necesitamos hacer ningún test estadístico o análisis continuo de los datos debido a la naturaleza categórica de los mismos. Además, la distribución de clases por columna no sigue ninguna distribución normal, atendiendo a las figuras obtenidas."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb67e878",
   "metadata": {},
   "source": [
    "# Creación de modelos con redes neuronales con TensorFlow\n",
    "\n",
    "De acuerdo a lo aprendido en la asignatura, utilizaremos TensorFlow en su versión 2.5 para crear distintos modelos de redes neuronales para abordar este problema de clasificación. Los datos que usaremos son los originales y los balanceados usando SMOTE. \n",
    "\n",
    "Para cada conjunto crearemos distintos modelos de perceptrón multicapa en los cuales variaremos la profundidad del mismo (número de capas) así como la cantidad de neuronas por capa. \n",
    "\n",
    "Haremos uso de técnicas como dropout y callbacks. La técnica del dropout consiste en desactivar ciertas neuronas por capa de forma aleatoria en cada época con el fin de evitar que la red neuronal sobreaprenda. Con esta idea, se debe poder conseguir redes neuronales de mayor tamaño que permitan aprender más características del conjunto de datos aún sin sobreaprender. \n",
    "\n",
    "Los callbacks son utilidades que incluye Tensorflow que facilitan el entrenamiento de los modelos. Podemos guardar en disco el modelo que mejor resultado da en función de una métrica de control. En nuestro caso, estamos interesados en la función _EarlyStopping_ que permite acabar el entrenamiento de forma automática si, tras un número determinado a elegir de épocas, la métrica que elijamos no ha mejorado. Esta ventaja es fundamental ya que nos ahorra tener que incluir el número de épocas dentro de los parámetros tuneables durante el entrenamiento.\n",
    "\n",
    "Para realizar el entrenamiento, usamos Linux junto a una tarjeta gráfica Nvidia 1070 max-q que nos ayuda a acelerar el trabajo de forma drástica. El entorno de Linux permite la configuración de las librerías CUDA de forma sencilla.\n",
    "\n",
    "De entre los modelos vistos en clase, nos hemos decantado por el perceptrón multicapa ya que es el que más fácilmente se adapta a este problema. Por un lado, no podemos usar las redes convolucionales, joya de la corona del deep learning actualmente, ya que se usan en reconomiento de imágenes. No sabemos cómo utilizar las capas convolucionales en este tipo de datos. Muchas funcionalidades de TensorFlow no son usadas, como _DataAugmentation_, que permite variar el conjunto de entrenamiento para que la red neuronal tenga más variedad durante el mismo. Por otro lado, los mapas autogenerativos así como las redes neuronales con funciones de base radial no tienen suficiente documentación en internet para este tipo de problemas. Las redes neuronales recurrentes no se suelen usar en estas situaciones."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01dacad6",
   "metadata": {},
   "source": [
    "## Datos _raw_ con _smote_ y _dropout_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "224177e6",
   "metadata": {},
   "source": [
    "Importamos los datos y separamos en variables predictoras: X y variable a predecir: y."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3c3cef72",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wkc</th>\n",
       "      <th>wkr</th>\n",
       "      <th>wrc</th>\n",
       "      <th>wrr</th>\n",
       "      <th>bkc</th>\n",
       "      <th>bkr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>b</td>\n",
       "      <td>3</td>\n",
       "      <td>c</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>1</td>\n",
       "      <td>d</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>1</td>\n",
       "      <td>d</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>2</td>\n",
       "      <td>c</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28051</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>g</td>\n",
       "      <td>7</td>\n",
       "      <td>e</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28052</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>g</td>\n",
       "      <td>7</td>\n",
       "      <td>e</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28053</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>g</td>\n",
       "      <td>7</td>\n",
       "      <td>e</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28054</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>g</td>\n",
       "      <td>7</td>\n",
       "      <td>f</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28055</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>g</td>\n",
       "      <td>7</td>\n",
       "      <td>g</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28056 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      wkc  wkr wrc  wrr bkc  bkr\n",
       "0       a    1   b    3   c    2\n",
       "1       a    1   c    1   c    2\n",
       "2       a    1   c    1   d    1\n",
       "3       a    1   c    1   d    2\n",
       "4       a    1   c    2   c    1\n",
       "...    ..  ...  ..  ...  ..  ...\n",
       "28051   b    1   g    7   e    5\n",
       "28052   b    1   g    7   e    6\n",
       "28053   b    1   g    7   e    7\n",
       "28054   b    1   g    7   f    5\n",
       "28055   b    1   g    7   g    5\n",
       "\n",
       "[28056 rows x 6 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "data = pd.read_csv(\"krkopt.data\", header=None)\n",
    "data.columns = [\"wkc\", \"wkr\", \"wrc\", \"wrr\", \"bkc\", \"bkr\", \"opt rank\" ]\n",
    "X = data.iloc[:, 0:6]\n",
    "y = data['opt rank']\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e557a94",
   "metadata": {},
   "source": [
    "Codificamos los valores \"a\", \"b\", \"c\" de las variables categóricas a enteros para que puedan ser utilizadas por las redes neuronales."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "03dc0557",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wkc</th>\n",
       "      <th>wkr</th>\n",
       "      <th>wrc</th>\n",
       "      <th>wrr</th>\n",
       "      <th>bkc</th>\n",
       "      <th>bkr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28051</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28052</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28053</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28054</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28055</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28056 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       wkc  wkr  wrc  wrr  bkc  bkr\n",
       "0        0    1    1    3    2    2\n",
       "1        0    1    2    1    2    2\n",
       "2        0    1    2    1    3    1\n",
       "3        0    1    2    1    3    2\n",
       "4        0    1    2    2    2    1\n",
       "...    ...  ...  ...  ...  ...  ...\n",
       "28051    1    1    6    7    4    5\n",
       "28052    1    1    6    7    4    6\n",
       "28053    1    1    6    7    4    7\n",
       "28054    1    1    6    7    5    5\n",
       "28055    1    1    6    7    6    5\n",
       "\n",
       "[28056 rows x 6 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[\"wkc\"]=X[\"wkc\"].astype('category')\n",
    "X[\"wrc\"]=X[\"wrc\"].astype('category')\n",
    "X[\"bkc\"]=X[\"bkc\"].astype('category')\n",
    "X[\"wkc\"]=X[\"wkc\"].cat.codes\n",
    "X[\"wrc\"]=X[\"wrc\"].cat.codes\n",
    "X[\"bkc\"]=X[\"bkc\"].cat.codes\n",
    "y = y.astype('category')\n",
    "y = y.cat.codes\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a25dd995",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder, LabelEncoder\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tensorflow import keras\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.utils import to_categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89a855c9",
   "metadata": {},
   "source": [
    "Creamos los conjuntos de entrenamiento y test con una proporción 80-20 usando herramientas de _sklearn_ y utilizamos oversampling con _smote_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6bbbfa24",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,\n",
    "                                                   y, test_size=0.2,\n",
    "                                                   random_state = 1)\n",
    "\n",
    "from imblearn.over_sampling import SMOTE\n",
    "oversample = SMOTE()\n",
    "\n",
    "\n",
    "X_smote, y_smote = oversample.fit_resample(X, y)\n",
    "\n",
    "X_train_smote, X_test_smote, y_train_smote, y_test_smote = train_test_split(X_smote,\n",
    "                                                   y_smote, test_size=0.2,\n",
    "                                                   random_state = 1)\n",
    "\n",
    "\n",
    "\n",
    "y_train = to_categorical(y_train)\n",
    "y_test  = to_categorical(y_test)\n",
    "y_train_smote = to_categorical(y_train_smote)\n",
    "y_test_smote = to_categorical(y_test_smote)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee32551c",
   "metadata": {},
   "source": [
    "Como vemos, _smote_ permite utilizar datos categóricos aunque no sea lo más recomendable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "64473d3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wkc</th>\n",
       "      <th>wkr</th>\n",
       "      <th>wrc</th>\n",
       "      <th>wrr</th>\n",
       "      <th>bkc</th>\n",
       "      <th>bkr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81949</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81950</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81951</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81952</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81953</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>81954 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       wkc  wkr  wrc  wrr  bkc  bkr\n",
       "0        0    1    1    3    2    2\n",
       "1        0    1    2    1    2    2\n",
       "2        0    1    2    1    3    1\n",
       "3        0    1    2    1    3    2\n",
       "4        0    1    2    2    2    1\n",
       "...    ...  ...  ...  ...  ...  ...\n",
       "81949    2    3    0    1    2    1\n",
       "81950    2    2    0    7    0    2\n",
       "81951    2    1    0    8    0    1\n",
       "81952    2    1    0    8    0    1\n",
       "81953    2    3    6    1    2    1\n",
       "\n",
       "[81954 rows x 6 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_smote"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15bb784e",
   "metadata": {},
   "source": [
    "Estandarizamos los datos de entrada usando _zscore_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "dd0689d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler().fit(X_train)\n",
    "X_train = scaler.transform(X_train)\n",
    "X_test = scaler.transform(X_test)\n",
    "\n",
    "scaler = StandardScaler().fit(X_train_smote)\n",
    "X_train_smote = scaler.transform(X_train_smote)\n",
    "X_test_smote = scaler.transform(X_test_smote)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24f43e60",
   "metadata": {},
   "source": [
    "### Definición de funciones\n",
    "\n",
    "Definimos tres funciones que nos serán útiles para automatizar el proceso:\n",
    "\n",
    "- _make_my_model_multi_ permite crear un perceptrón multicapa proporcionándole la estructura de la forma $[n_1, n_2, ..., n_i,..., n_N]$ donde $n_i$ es el número de neuronas de la capa oculta $i$. Además de otros parámetros como la forma de entrada, salida y la función de activación que queramos usar.\n",
    "- _compile_fit_multiclass_ entrena un modelo de entrada con el conjunto de entrenamiento usando un conjunto de validación 80-20 y produce predicciones con un conjunto test. Utilizamos la herramienta _ModelCheckpoint_ para guardar el mejor modelo durante el entrenamiento y _EarlyStopping_ para parar el entrenamiento si el valor de _loss_ en el conjunto de validación no mejora en 10 épocas. Así evitamos el sobreaprendizaje.\n",
    "\n",
    "- _compute_metrics_multiclass_ calcula las métricas precision, recall, F1 y Kappa a partir de las predicciones y los valores exactos de test.\n",
    "\n",
    "- _make_my_model_multi_dropout_ es una modificación que permite crear un modelo introduciendo capas internas de dropout. La estructura de la red se introduce de la forma $[n_1, n_2, ..., n_i,..., n_N]$ donde $n_i$ es el número de neuronas de la capa $i$ si escribimos un número entero o una capa dropout con un valor  de desactivación $n_i$ si introducimos un elemento de tipo carácter. Por ejemplo, [50, \"0.2\", 50, \"0.2\"] creará unas capas ocultas de la forma: capa con 50 neuronas, dropout 0.2, capa con 50 neuronas y dropout de 0.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bdbb956a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "\n",
    "\n",
    "checkpoint_filepath = '/tmp/checkpoint'\n",
    "from sklearn.metrics import confusion_matrix, precision_score, \\\n",
    "f1_score, cohen_kappa_score, recall_score\n",
    "\n",
    "def make_my_model_multi( units_per_layer, input_s, output_s, activation_='relu'):\n",
    "    model = Sequential()\n",
    "    depth = len(units_per_layer)\n",
    "    model.add(Dense(units_per_layer[0], activation=activation_, input_shape=(input_s,)))\n",
    "    for i in range(1, depth):\n",
    "        model.add(Dense(units_per_layer[i], activation=activation_))\n",
    "    model.add(Dense(output_s, activation = 'softmax'))   \n",
    "    \n",
    "    return model\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "def compile_fit_multiclass(modelo, X_train, X_test, y_train, batch, epochs, verbose=0):\n",
    "    modelo.compile(loss='categorical_crossentropy',\n",
    "             optimizer='adam',\n",
    "             metrics=['accuracy'])\n",
    "    \n",
    "    early_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, verbose=True)\n",
    "\n",
    "    model_checkpoint = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_filepath,\n",
    "                                                        save_weights_only=True,\n",
    "                                                        monitor='val_loss',\n",
    "                                                        mode='min',\n",
    "                                                        save_best_only=True,\n",
    "                                                        verbose=False)\n",
    "\n",
    "    modelo.fit(X_train, y_train, epochs=epochs, batch_size=batch, verbose=verbose, validation_split=0.2, callbacks = [early_stopping, model_checkpoint])\n",
    "    model.load_weights(checkpoint_filepath)\n",
    "    predictions = modelo.predict(X_test)\n",
    "    return predictions\n",
    "\n",
    "def compute_metrics_multiclass(y_test, y_pred):\n",
    "    results=[]\n",
    "    results.append(precision_score(y_test, np.round(y_pred), average=\"micro\"))\n",
    "    results.append(recall_score(y_test, np.round(y_pred), average=\"micro\"))\n",
    "    results.append(f1_score(y_test, np.round(y_pred), average=\"micro\"))\n",
    "    results.append(cohen_kappa_score(y_test, np.round(y_pred)))\n",
    "    return results\n",
    "\n",
    "\n",
    "from tensorflow.keras.layers import Dropout\n",
    "\n",
    "def make_my_model_multi_dropout( units_per_layer, input_s, output_s, activation_='relu'):\n",
    "    model = Sequential()\n",
    "    depth = len(units_per_layer)\n",
    "    model.add(Dense(units_per_layer[0], activation=activation_, input_shape=(input_s,)))\n",
    "    for i in range(1, depth):\n",
    "        if isinstance(units_per_layer[i], str):\n",
    "            a = units_per_layer[i]\n",
    "            dropout_r = float(a)\n",
    "            model.add(Dropout(dropout_r))\n",
    "        else:\n",
    "            model.add(Dense(units_per_layer[i], activation=activation_))\n",
    "    model.add(Dense(output_s, activation = 'softmax'))   \n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adfb7954",
   "metadata": {},
   "source": [
    "Comprobamos que los datos tienen las dimensiones correctas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "bcdc9dae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((22444, 6), (65563, 6), (22444, 18), (65563, 18), (5612, 6), (5612, 18))"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape, X_train_smote.shape, y_train.shape, y_train_smote.shape, X_test.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bc81264",
   "metadata": {},
   "source": [
    "### Pruebas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c5a0dad",
   "metadata": {},
   "source": [
    "#### Datos _raw_\n",
    "Creamos un total de treinta experimentos en las que crearemos redes neuronales de la misma cantidad de neuronas por capas con distinto número de neuronas y distinto número de capas. El número de neuronas será: [50, 100, 150, 200, 250] y el tamaño variará de entre una y seis capas ocultas. Guardamos las predicciones junto a las métricas y la matriz de confusión. Escribimos en disco el objeto mediante _joblib_ para su posterior análisis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "294ccb2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = []\n",
    "seed = 1\n",
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "6589d531",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\n",
      "[0.5563079116179616, 0.5563079116179616, 0.5563079116179616, 0.5014778486804247]\n",
      "[50, 50]\n",
      "Epoch 00278: early stopping\n",
      "[0.6776550249465432, 0.6776550249465432, 0.6776550249465432, 0.6394872591777784]\n",
      "[50, 50, 50]\n",
      "Epoch 00189: early stopping\n",
      "[0.7004632929436921, 0.7004632929436921, 0.7004632929436921, 0.66527050745813]\n",
      "[50, 50, 50, 50]\n",
      "Epoch 00135: early stopping\n",
      "[0.7033143264433357, 0.7033143264433357, 0.7033143264433357, 0.6683596780441357]\n",
      "[50, 50, 50, 50, 50]\n",
      "Epoch 00083: early stopping\n",
      "[0.7140057020669993, 0.7140057020669993, 0.7140057020669993, 0.6802859773047576]\n",
      "[50, 50, 50, 50, 50, 50]\n",
      "Epoch 00101: early stopping\n",
      "[0.7325374198146828, 0.7325374198146828, 0.7325374198146829, 0.7014741703016625]\n",
      "[100]\n",
      "[0.5841054882394868, 0.5841054882394868, 0.5841054882394868, 0.532926380750063]\n",
      "[100, 100]\n",
      "Epoch 00226: early stopping\n",
      "[0.7145402708481825, 0.7145402708481825, 0.7145402708481825, 0.6810448690167336]\n",
      "[100, 100, 100]\n",
      "Epoch 00145: early stopping\n",
      "[0.7808267997148967, 0.7808267997148967, 0.7808267997148967, 0.7552060507593273]\n",
      "[100, 100, 100, 100]\n",
      "Epoch 00117: early stopping\n",
      "[0.7770848182466144, 0.7770848182466144, 0.7770848182466144, 0.7511752665391203]\n",
      "[100, 100, 100, 100, 100]\n",
      "Epoch 00083: early stopping\n",
      "[0.7854597291518175, 0.7854597291518175, 0.7854597291518175, 0.7603159232769513]\n",
      "[100, 100, 100, 100, 100, 100]\n",
      "Epoch 00089: early stopping\n",
      "[0.7861724875267284, 0.7861724875267284, 0.7861724875267283, 0.7611696219845376]\n",
      "[150]\n",
      "[0.6051318602993585, 0.6051318602993585, 0.6051318602993585, 0.5571977645455048]\n",
      "[150, 150]\n",
      "Epoch 00196: early stopping\n",
      "[0.7462580185317177, 0.7462580185317177, 0.7462580185317177, 0.7163812931399909]\n",
      "[150, 150, 150]\n",
      "Epoch 00117: early stopping\n",
      "[0.7813613684960798, 0.7813613684960798, 0.7813613684960798, 0.7561423259186348]\n",
      "[150, 150, 150, 150]\n",
      "Epoch 00087: early stopping\n",
      "[0.7957947255880257, 0.7957947255880257, 0.7957947255880257, 0.7718799039500605]\n",
      "[150, 150, 150, 150, 150]\n",
      "Epoch 00094: early stopping\n",
      "[0.8209194583036351, 0.8209194583036351, 0.8209194583036351, 0.799991822447679]\n",
      "[150, 150, 150, 150, 150, 150]\n",
      "Epoch 00083: early stopping\n",
      "[0.8184248039914469, 0.8184248039914469, 0.8184248039914469, 0.7973288336155344]\n",
      "[200]\n",
      "[0.6286528866714184, 0.6286528866714184, 0.6286528866714184, 0.5841443006982328]\n",
      "[200, 200]\n",
      "Epoch 00160: early stopping\n",
      "[0.7530292230933714, 0.7530292230933714, 0.7530292230933714, 0.7243038312259522]\n",
      "[200, 200, 200]\n",
      "Epoch 00094: early stopping\n",
      "[0.7794012829650748, 0.7794012829650748, 0.7794012829650748, 0.7533991727673046]\n",
      "[200, 200, 200, 200]\n",
      "Epoch 00072: early stopping\n",
      "[0.8029223093371347, 0.8029223093371347, 0.8029223093371347, 0.7797459386027457]\n",
      "[200, 200, 200, 200, 200]\n",
      "Epoch 00064: early stopping\n",
      "[0.8048823948681397, 0.8048823948681397, 0.8048823948681397, 0.7819079964441699]\n",
      "[200, 200, 200, 200, 200, 200]\n",
      "Epoch 00061: early stopping\n",
      "[0.8191375623663578, 0.8191375623663578, 0.8191375623663577, 0.7979856944628764]\n",
      "[250]\n",
      "[0.6416607270135424, 0.6416607270135424, 0.6416607270135424, 0.5990285359827496]\n",
      "[250, 250]\n",
      "Epoch 00145: early stopping\n",
      "[0.7528510334996437, 0.7528510334996437, 0.7528510334996438, 0.7238185938600421]\n",
      "[250, 250, 250]\n",
      "Epoch 00065: early stopping\n",
      "[0.7694226657163221, 0.7694226657163221, 0.7694226657163221, 0.7424900002897106]\n",
      "[250, 250, 250, 250]\n",
      "Epoch 00072: early stopping\n",
      "[0.8161083392729864, 0.8161083392729864, 0.8161083392729864, 0.7947878390730547]\n",
      "[250, 250, 250, 250, 250]\n",
      "Epoch 00060: early stopping\n",
      "[0.8251960085531005, 0.8251960085531005, 0.8251960085531005, 0.8045311760051372]\n",
      "[250, 250, 250, 250, 250, 250]\n",
      "Epoch 00057: early stopping\n",
      "[0.8218104062722738, 0.8218104062722738, 0.8218104062722738, 0.80115129573033]\n"
     ]
    }
   ],
   "source": [
    "size_config = [50, 100, 150, 200, 250]\n",
    "for size in size_config:\n",
    "    layer_config = [[size], [size]*2, [size]*3, [size]*4, [size]*5, [size]*6]\n",
    "    for layers in layer_config:\n",
    "        np.random.seed(seed)\n",
    "        tf.random.set_seed(seed)\n",
    "        print(layers)\n",
    "        model = make_my_model_multi(layers, 6, 18, activation_='relu' )\n",
    "        preds = compile_fit_multiclass(model, X_train, X_test, y_train, 256, 300, verbose=0)\n",
    "        metrics = compute_metrics_multiclass(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "        confusion = confusion_matrix(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "        aux = { \"layer config\" : layers,\n",
    "               #\"Model\": model,\n",
    "               \"Predictions\" : preds,\n",
    "               \"Metrics\" : metrics,\n",
    "               \"Confusion\" : confusion\n",
    "\n",
    "        }\n",
    "        print(metrics)\n",
    "        results.append(aux)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "949befee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['results_1_joblib']"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import joblib\n",
    " \n",
    "joblib.dump(results, 'results_1_joblib')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82917edc",
   "metadata": {},
   "source": [
    "#### Datos con _smote_\n",
    "\n",
    "Repetimos el mismo esquema con los datos con _smote_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "ce64b3f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_smote = []\n",
    "seed = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "0a9840dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\n",
      "[0.2735210263720599, 0.2735210263720599, 0.2735210263720599, 0.21056283287164057]\n",
      "[50, 50]\n",
      "Epoch 00195: early stopping\n",
      "[0.3016749821810406, 0.3016749821810406, 0.3016749821810406, 0.23858740268705947]\n",
      "[50, 50, 50]\n",
      "Epoch 00149: early stopping\n",
      "[0.319672131147541, 0.319672131147541, 0.319672131147541, 0.2573189896196104]\n",
      "[50, 50, 50, 50]\n",
      "Epoch 00150: early stopping\n",
      "[0.3257305773342837, 0.3257305773342837, 0.3257305773342837, 0.26329646382574623]\n",
      "[50, 50, 50, 50, 50]\n",
      "Epoch 00105: early stopping\n",
      "[0.32216678545972915, 0.32216678545972915, 0.32216678545972915, 0.26004197522733685]\n",
      "[50, 50, 50, 50, 50, 50]\n",
      "Epoch 00170: early stopping\n",
      "[0.3310762651461155, 0.3310762651461155, 0.3310762651461155, 0.2689915712533639]\n",
      "[100]\n",
      "[0.29971489665003564, 0.29971489665003564, 0.29971489665003564, 0.23722592685369615]\n",
      "[100, 100]\n",
      "Epoch 00174: early stopping\n",
      "[0.32323592302209553, 0.32323592302209553, 0.32323592302209553, 0.26046775820865786]\n",
      "[100, 100, 100]\n",
      "Epoch 00133: early stopping\n",
      "[0.3341054882394868, 0.3341054882394868, 0.3341054882394868, 0.27209407806805874]\n",
      "[100, 100, 100, 100]\n",
      "Epoch 00065: early stopping\n",
      "[0.34052031361368496, 0.34052031361368496, 0.34052031361368496, 0.2793803604511679]\n",
      "[100, 100, 100, 100, 100]\n",
      "Epoch 00065: early stopping\n",
      "[0.33856022808268, 0.33856022808268, 0.33856022808268, 0.2775702647152132]\n",
      "[100, 100, 100, 100, 100, 100]\n",
      "Epoch 00078: early stopping\n",
      "[0.3447968638631504, 0.3447968638631504, 0.3447968638631504, 0.2820060064899931]\n",
      "[150]\n",
      "[0.30024946543121883, 0.30024946543121883, 0.30024946543121883, 0.23828899508645218]\n",
      "[150, 150]\n",
      "Epoch 00145: early stopping\n",
      "[0.32555238774055595, 0.32555238774055595, 0.32555238774055595, 0.26301755677369276]\n",
      "[150, 150, 150]\n",
      "Epoch 00095: early stopping\n",
      "[0.323592302209551, 0.323592302209551, 0.323592302209551, 0.26068327259513024]\n",
      "[150, 150, 150, 150]\n",
      "Epoch 00065: early stopping\n",
      "[0.3469351389878831, 0.3469351389878831, 0.3469351389878831, 0.28626249332475373]\n",
      "[150, 150, 150, 150, 150]\n",
      "Epoch 00076: early stopping\n",
      "[0.3551318602993585, 0.3551318602993585, 0.3551318602993585, 0.29535007953313264]\n",
      "[150, 150, 150, 150, 150, 150]\n",
      "Epoch 00089: early stopping\n",
      "[0.3544191019244476, 0.3544191019244476, 0.3544191019244476, 0.2932453533302928]\n",
      "[200]\n",
      "[0.31272273699215963, 0.31272273699215963, 0.31272273699215963, 0.2508736929301466]\n",
      "[200, 200]\n",
      "Epoch 00155: early stopping\n",
      "[0.3362437633642195, 0.3362437633642195, 0.3362437633642195, 0.2741266338548447]\n",
      "[200, 200, 200]\n",
      "Epoch 00072: early stopping\n",
      "[0.33481824661439774, 0.33481824661439774, 0.33481824661439774, 0.2737471663026608]\n",
      "[200, 200, 200, 200]\n",
      "Epoch 00059: early stopping\n",
      "[0.34337134711332856, 0.34337134711332856, 0.34337134711332856, 0.28165430613441933]\n",
      "[200, 200, 200, 200, 200]\n",
      "Epoch 00065: early stopping\n",
      "[0.34746970776906627, 0.34746970776906627, 0.34746970776906627, 0.2864768326195607]\n",
      "[200, 200, 200, 200, 200, 200]\n",
      "Epoch 00078: early stopping\n",
      "[0.35655737704918034, 0.35655737704918034, 0.3565573770491803, 0.2961375972429202]\n",
      "[250]\n",
      "[0.32020669992872414, 0.32020669992872414, 0.32020669992872414, 0.2589827289284935]\n",
      "[250, 250]\n",
      "Epoch 00116: early stopping\n",
      "[0.3271560940841055, 0.3271560940841055, 0.3271560940841055, 0.26503000969397217]\n",
      "[250, 250, 250]\n",
      "Epoch 00084: early stopping\n",
      "[0.3394511760513186, 0.3394511760513186, 0.3394511760513186, 0.2769489283726123]\n",
      "[250, 250, 250, 250]\n",
      "Epoch 00068: early stopping\n",
      "[0.35477548111190305, 0.35477548111190305, 0.3547754811119031, 0.29475622872925544]\n",
      "[250, 250, 250, 250, 250]\n",
      "Epoch 00045: early stopping\n",
      "[0.34978617248752675, 0.34978617248752675, 0.34978617248752675, 0.2885785810340571]\n",
      "[250, 250, 250, 250, 250, 250]\n",
      "Epoch 00057: early stopping\n",
      "[0.3617248752672844, 0.3617248752672844, 0.36172487526728436, 0.3008943441645957]\n"
     ]
    }
   ],
   "source": [
    "size_config = [50, 100, 150, 200, 250]\n",
    "for size in size_config:\n",
    "    layer_config = [[size], [size]*2, [size]*3, [size]*4, [size]*5, [size]*6]\n",
    "    for layers in layer_config:\n",
    "        np.random.seed(seed)\n",
    "        tf.random.set_seed(seed)\n",
    "        print(layers)\n",
    "        model = make_my_model_multi(layers, 6, 18, activation_='relu' )\n",
    "        preds = compile_fit_multiclass(model, X_train_smote, X_test, y_train_smote, 256, 300, verbose=0)\n",
    "        metrics = compute_metrics_multiclass(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "        confusion = confusion_matrix(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "        aux = { \"layer config\" : layers,\n",
    "               #\"Model\": model,\n",
    "               \"Predictions\" : preds,\n",
    "               \"Metrics\" : metrics,\n",
    "               \"Confusion\" : confusion\n",
    "\n",
    "        }\n",
    "        print(metrics)\n",
    "        results_smote.append(aux)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "963d21c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['results_smote_joblib']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(results_smote, 'results_smote_joblib')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "037dc8dd",
   "metadata": {},
   "source": [
    "#### Datos raw con dropout\n",
    "A la hora de realizar experimentos con dropout, creamos el mismo esquema que en casos anteriores intercalando capas dropout de 3 posibles valores: 0.1, 0.2 y 0.3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "91ce76c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_dropout = []\n",
    "seed = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "961283ef",
   "metadata": {},
   "source": [
    "Mostramos la configuración de los experimentos. En total, realizaremos noventa casos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "c1a3c64e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50, '0.1']\n",
      "[50, '0.1', 50, '0.1']\n",
      "[50, '0.1', 50, '0.1', 50, '0.1']\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "[50, '0.2']\n",
      "[50, '0.2', 50, '0.2']\n",
      "[50, '0.2', 50, '0.2', 50, '0.2']\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "[50, '0.3']\n",
      "[50, '0.3', 50, '0.3']\n",
      "[50, '0.3', 50, '0.3', 50, '0.3']\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "[100, '0.1']\n",
      "[100, '0.1', 100, '0.1']\n",
      "[100, '0.1', 100, '0.1', 100, '0.1']\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "[100, '0.2']\n",
      "[100, '0.2', 100, '0.2']\n",
      "[100, '0.2', 100, '0.2', 100, '0.2']\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "[100, '0.3']\n",
      "[100, '0.3', 100, '0.3']\n",
      "[100, '0.3', 100, '0.3', 100, '0.3']\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "[150, '0.1']\n",
      "[150, '0.1', 150, '0.1']\n",
      "[150, '0.1', 150, '0.1', 150, '0.1']\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "[150, '0.2']\n",
      "[150, '0.2', 150, '0.2']\n",
      "[150, '0.2', 150, '0.2', 150, '0.2']\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "[150, '0.3']\n",
      "[150, '0.3', 150, '0.3']\n",
      "[150, '0.3', 150, '0.3', 150, '0.3']\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "[200, '0.1']\n",
      "[200, '0.1', 200, '0.1']\n",
      "[200, '0.1', 200, '0.1', 200, '0.1']\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "[200, '0.2']\n",
      "[200, '0.2', 200, '0.2']\n",
      "[200, '0.2', 200, '0.2', 200, '0.2']\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "[200, '0.3']\n",
      "[200, '0.3', 200, '0.3']\n",
      "[200, '0.3', 200, '0.3', 200, '0.3']\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "[250, '0.1']\n",
      "[250, '0.1', 250, '0.1']\n",
      "[250, '0.1', 250, '0.1', 250, '0.1']\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "[250, '0.2']\n",
      "[250, '0.2', 250, '0.2']\n",
      "[250, '0.2', 250, '0.2', 250, '0.2']\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "[250, '0.3']\n",
      "[250, '0.3', 250, '0.3']\n",
      "[250, '0.3', 250, '0.3', 250, '0.3']\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n"
     ]
    }
   ],
   "source": [
    "size_config = [50, 100, 150, 200, 250]\n",
    "dropout_rate = [\"0.1\", \"0.2\", \"0.3\"]\n",
    "\n",
    "for size in size_config:\n",
    "    for size_d in (dropout_rate):\n",
    "        layer_config_dense = [[size], [size]*2, [size]*3, [size]*4, [size]*5, [size]*6]\n",
    "        layer_config_dropout = [[size_d], [size_d]*2, [size_d]*3, [size_d]*4, [size_d]*5, [size_d]*6]\n",
    "        for layers_dense, layers_dropout in zip(layer_config_dense, layer_config_dropout):\n",
    "            final_design = [None]*(len(layers_dense)+len(layers_dropout))\n",
    "            final_design[::2] = layers_dense\n",
    "            final_design[1::2] = layers_dropout\n",
    "            print(final_design)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "fdeef16c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50, '0.1']\n",
      "[0.5306486101211689, 0.5306486101211689, 0.5306486101211689, 0.47299466324494865]\n",
      "[50, '0.1', 50, '0.1']\n",
      "[0.6582323592302209, 0.6582323592302209, 0.6582323592302209, 0.6168927965574147]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00274: early stopping\n",
      "[0.7109764789736279, 0.7109764789736279, 0.7109764789736278, 0.677058966107785]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00187: early stopping\n",
      "[0.7066999287241625, 0.7066999287241625, 0.7066999287241625, 0.6719743342215174]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00152: early stopping\n",
      "[0.6903064861012117, 0.6903064861012117, 0.6903064861012117, 0.6533659428663383]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00176: early stopping\n",
      "[0.6915538132573058, 0.6915538132573058, 0.6915538132573058, 0.6548578250956504]\n",
      "[50, '0.2']\n",
      "Epoch 00253: early stopping\n",
      "[0.5172843905915895, 0.5172843905915895, 0.5172843905915895, 0.45668070032612573]\n",
      "[50, '0.2', 50, '0.2']\n",
      "[0.607448325017819, 0.607448325017819, 0.607448325017819, 0.5594133149504249]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00155: early stopping\n",
      "[0.6181397006414825, 0.6181397006414825, 0.6181397006414825, 0.5718332632332058]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00221: early stopping\n",
      "[0.642551674982181, 0.642551674982181, 0.642551674982181, 0.5987753202495634]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00135: early stopping\n",
      "[0.5915894511760513, 0.5915894511760513, 0.5915894511760513, 0.5418785789667531]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00100: early stopping\n",
      "[0.577512473271561, 0.577512473271561, 0.577512473271561, 0.5258937398977566]\n",
      "[50, '0.3']\n",
      "Epoch 00249: early stopping\n",
      "[0.5110477548111191, 0.5110477548111191, 0.5110477548111191, 0.44936967418461227]\n",
      "[50, '0.3', 50, '0.3']\n",
      "Epoch 00214: early stopping\n",
      "[0.5673556664290805, 0.5673556664290805, 0.5673556664290805, 0.5134787826514988]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00235: early stopping\n",
      "[0.5616535994297933, 0.5616535994297933, 0.5616535994297933, 0.5066628382810214]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00124: early stopping\n",
      "[0.5342124019957234, 0.5342124019957234, 0.5342124019957234, 0.4751719679727483]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00127: early stopping\n",
      "[0.5263720598717035, 0.5263720598717035, 0.5263720598717035, 0.4667612314609778]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00108: early stopping\n",
      "[0.5226300784034212, 0.5226300784034212, 0.5226300784034212, 0.46306875130107517]\n",
      "[100, '0.1']\n",
      "Epoch 00249: early stopping\n",
      "[0.5461511047754811, 0.5461511047754811, 0.5461511047754811, 0.49062978889937336]\n",
      "[100, '0.1', 100, '0.1']\n",
      "[0.7521382751247327, 0.7521382751247327, 0.7521382751247327, 0.7227156116282598]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1']\n",
      "[0.8257305773342837, 0.8257305773342837, 0.8257305773342837, 0.8054057526695507]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00165: early stopping\n",
      "[0.7993585174625801, 0.7993585174625801, 0.7993585174625801, 0.775965204492387]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00221: early stopping\n",
      "[0.8182466143977192, 0.8182466143977192, 0.8182466143977192, 0.7970575500307149]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00199: early stopping\n",
      "[0.8155737704918032, 0.8155737704918032, 0.8155737704918032, 0.7940122031851752]\n",
      "[100, '0.2']\n",
      "Epoch 00249: early stopping\n",
      "[0.5386671418389166, 0.5386671418389166, 0.5386671418389166, 0.48189218191606287]\n",
      "[100, '0.2', 100, '0.2']\n",
      "[0.7248752672843906, 0.7248752672843906, 0.7248752672843906, 0.6919934078576849]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00238: early stopping\n",
      "[0.7674625801853172, 0.7674625801853172, 0.7674625801853173, 0.740127140644157]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00191: early stopping\n",
      "[0.7610477548111191, 0.7610477548111191, 0.7610477548111191, 0.7329919066201204]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00158: early stopping\n",
      "[0.7273699215965788, 0.7273699215965788, 0.7273699215965788, 0.6954064061016627]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00089: early stopping\n",
      "[0.6945830363506771, 0.6945830363506771, 0.6945830363506771, 0.6587686427382717]\n",
      "[100, '0.3']\n",
      "Epoch 00214: early stopping\n",
      "[0.5340342124019958, 0.5340342124019958, 0.5340342124019958, 0.47652760627391544]\n",
      "[100, '0.3', 100, '0.3']\n",
      "Epoch 00252: early stopping\n",
      "[0.6764076977904491, 0.6764076977904491, 0.6764076977904491, 0.637487713970418]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00195: early stopping\n",
      "[0.7033143264433357, 0.7033143264433357, 0.7033143264433357, 0.6684001878880355]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00157: early stopping\n",
      "[0.6847826086956522, 0.6847826086956522, 0.6847826086956522, 0.6471773664755471]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00136: early stopping\n",
      "[0.6781895937277262, 0.6781895937277262, 0.6781895937277262, 0.6394933393602311]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00124: early stopping\n",
      "[0.6370277975766215, 0.6370277975766215, 0.6370277975766215, 0.5935140862849402]\n",
      "[150, '0.1']\n",
      "Epoch 00272: early stopping\n",
      "[0.5620099786172488, 0.5620099786172488, 0.5620099786172488, 0.508341442519633]\n",
      "[150, '0.1', 150, '0.1']\n",
      "Epoch 00266: early stopping\n",
      "[0.7749465431218817, 0.7749465431218817, 0.7749465431218816, 0.7484450378452736]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00200: early stopping\n",
      "[0.8355310049893087, 0.8355310049893087, 0.8355310049893087, 0.8162182870942731]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00197: early stopping\n",
      "[0.8597647897362795, 0.8597647897362795, 0.8597647897362795, 0.8433538331608326]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00110: early stopping\n",
      "[0.8241268709907341, 0.8241268709907341, 0.8241268709907341, 0.8035528526761304]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00230: early stopping\n",
      "[0.8624376336421953, 0.8624376336421953, 0.8624376336421952, 0.846293252489121]\n",
      "[150, '0.2']\n",
      "[0.5563079116179616, 0.5563079116179616, 0.5563079116179616, 0.5015463823385387]\n",
      "[150, '0.2', 150, '0.2']\n",
      "[0.7615823235923022, 0.7615823235923022, 0.7615823235923023, 0.7335657779063354]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00204: early stopping\n",
      "[0.8095153243050606, 0.8095153243050606, 0.8095153243050606, 0.7871495134285849]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00264: early stopping\n",
      "[0.8342836778332146, 0.8342836778332146, 0.8342836778332146, 0.8149513387287599]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00148: early stopping\n",
      "[0.8022095509622238, 0.8022095509622238, 0.8022095509622238, 0.778780229359274]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00098: early stopping\n",
      "[0.7696008553100498, 0.7696008553100498, 0.7696008553100498, 0.7424088375196287]\n",
      "[150, '0.3']\n",
      "Epoch 00249: early stopping\n",
      "[0.5411617961511048, 0.5411617961511048, 0.5411617961511048, 0.4847149956069786]\n",
      "[150, '0.3', 150, '0.3']\n",
      "[0.7403777619387027, 0.7403777619387027, 0.7403777619387029, 0.7093364064692692]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00248: early stopping\n",
      "[0.785816108339273, 0.785816108339273, 0.785816108339273, 0.7605921518542224]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00140: early stopping\n",
      "[0.7521382751247327, 0.7521382751247327, 0.7521382751247327, 0.722902071589828]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00187: early stopping\n",
      "[0.7599786172487527, 0.7599786172487527, 0.7599786172487527, 0.731518863771794]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00147: early stopping\n",
      "[0.7337847469707769, 0.7337847469707769, 0.733784746970777, 0.7018631579115805]\n",
      "[200, '0.1']\n",
      "[0.5771560940841055, 0.5771560940841055, 0.5771560940841055, 0.5250361092757381]\n",
      "[200, '0.1', 200, '0.1']\n",
      "Epoch 00227: early stopping\n",
      "[0.7902708481824662, 0.7902708481824662, 0.7902708481824661, 0.765865311916265]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00246: early stopping\n",
      "[0.8709907341411262, 0.8709907341411262, 0.8709907341411262, 0.8558632550430805]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00150: early stopping\n",
      "[0.8661796151104776, 0.8661796151104776, 0.8661796151104776, 0.8504775774300332]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00171: early stopping\n",
      "[0.8736635780470421, 0.8736635780470421, 0.8736635780470421, 0.858903886573265]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00143: early stopping\n",
      "[0.8647540983606558, 0.8647540983606558, 0.8647540983606559, 0.8489033371956115]\n",
      "[200, '0.2']\n",
      "[0.5718104062722738, 0.5718104062722738, 0.5718104062722738, 0.5193521921585287]\n",
      "[200, '0.2', 200, '0.2']\n",
      "[0.7906272273699216, 0.7906272273699216, 0.7906272273699216, 0.7660980388359336]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2']\n",
      "[0.8549536707056308, 0.8549536707056308, 0.8549536707056308, 0.8381041473725426]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00131: early stopping\n",
      "[0.8232359230220955, 0.8232359230220955, 0.8232359230220955, 0.8025074171854785]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00219: early stopping\n",
      "[0.8558446186742694, 0.8558446186742694, 0.8558446186742694, 0.8391218900502941]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00152: early stopping\n",
      "[0.8330363506771205, 0.8330363506771205, 0.8330363506771205, 0.8132935419648581]\n",
      "[200, '0.3']\n",
      "Epoch 00253: early stopping\n",
      "[0.5481111903064861, 0.5481111903064861, 0.5481111903064861, 0.4926703109101981]\n",
      "[200, '0.3', 200, '0.3']\n",
      "Epoch 00268: early stopping\n",
      "[0.7617605131860299, 0.7617605131860299, 0.7617605131860298, 0.7338285623633978]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00245: early stopping\n",
      "[0.8200285103349965, 0.8200285103349965, 0.8200285103349965, 0.7989672711956026]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00211: early stopping\n",
      "[0.8134354953670706, 0.8134354953670706, 0.8134354953670706, 0.7917231289447606]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00133: early stopping\n",
      "[0.770848182466144, 0.770848182466144, 0.770848182466144, 0.7438576324656899]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00159: early stopping\n",
      "[0.7790449037776194, 0.7790449037776194, 0.7790449037776194, 0.7530774131803226]\n",
      "[250, '0.1']\n",
      "[0.5851746258018532, 0.5851746258018532, 0.5851746258018532, 0.5348236817948813]\n",
      "[250, '0.1', 250, '0.1']\n",
      "Epoch 00251: early stopping\n",
      "[0.8057733428367784, 0.8057733428367784, 0.8057733428367784, 0.7828834924115987]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00150: early stopping\n",
      "[0.8490734141126158, 0.8490734141126158, 0.8490734141126158, 0.8314265685216257]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00113: early stopping\n",
      "[0.8610121168923734, 0.8610121168923734, 0.8610121168923734, 0.8447712258954628]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00109: early stopping\n",
      "[0.8740199572344975, 0.8740199572344975, 0.8740199572344975, 0.8592490994133825]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00133: early stopping\n",
      "[0.8713471133285816, 0.8713471133285816, 0.8713471133285816, 0.8562902115842986]\n",
      "[250, '0.2']\n",
      "Epoch 00249: early stopping\n",
      "[0.5712758374910906, 0.5712758374910906, 0.5712758374910906, 0.5190729423874725]\n",
      "[250, '0.2', 250, '0.2']\n",
      "[0.8034568781183179, 0.8034568781183179, 0.8034568781183179, 0.7804932079709793]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00192: early stopping\n",
      "[0.8526372059871703, 0.8526372059871703, 0.8526372059871702, 0.8354667548029857]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00129: early stopping\n",
      "[0.8483606557377049, 0.8483606557377049, 0.8483606557377049, 0.8308142882391145]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00179: early stopping\n",
      "[0.8635067712045617, 0.8635067712045617, 0.8635067712045617, 0.8475984137698004]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00136: early stopping\n",
      "[0.8390947968638631, 0.8390947968638631, 0.8390947968638631, 0.8203757068695277]\n",
      "[250, '0.3']\n",
      "[0.5659301496792587, 0.5659301496792587, 0.5659301496792587, 0.5130749928933578]\n",
      "[250, '0.3', 250, '0.3']\n",
      "Epoch 00266: early stopping\n",
      "[0.7783321454027085, 0.7783321454027085, 0.7783321454027085, 0.7522834553134814]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3']\n",
      "[0.8531717747683535, 0.8531717747683535, 0.8531717747683535, 0.8360999258915827]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00262: early stopping\n",
      "[0.8465787598004276, 0.8465787598004276, 0.8465787598004277, 0.8288494408217892]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00249: early stopping\n",
      "[0.8406985032074127, 0.8406985032074127, 0.8406985032074127, 0.8221270376207139]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00210: early stopping\n",
      "[0.8127227369921597, 0.8127227369921597, 0.8127227369921597, 0.7909136938135756]\n"
     ]
    }
   ],
   "source": [
    "size_config = [50, 100, 150, 200, 250]\n",
    "dropout_rate = [\"0.1\", \"0.2\", \"0.3\"]\n",
    "\n",
    "for size in size_config:\n",
    "    for size_d in (dropout_rate):\n",
    "        layer_config_dense = [[size], [size]*2, [size]*3, [size]*4, [size]*5, [size]*6]\n",
    "        layer_config_dropout = [[size_d], [size_d]*2, [size_d]*3, [size_d]*4, [size_d]*5, [size_d]*6]\n",
    "        for layers_dense, layers_dropout in zip(layer_config_dense, layer_config_dropout):\n",
    "            final_design = [None]*(len(layers_dense)+len(layers_dropout))\n",
    "            final_design[::2] = layers_dense\n",
    "            final_design[1::2] = layers_dropout\n",
    "            np.random.seed(seed)\n",
    "            tf.random.set_seed(seed)\n",
    "            print(final_design)\n",
    "            model = make_my_model_multi_dropout(final_design, 6, 18, activation_='relu' )\n",
    "            preds = compile_fit_multiclass(model, X_train, X_test, y_train, 256, 300, verbose=0)\n",
    "            metrics = compute_metrics_multiclass(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "            confusion = confusion_matrix(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "            aux = { \"layer config\" : final_design,\n",
    "                   #\"Model\": model,\n",
    "                   \"Predictions\" : preds,\n",
    "                   \"Metrics\" : metrics,\n",
    "                   \"Confusion\" : confusion\n",
    "\n",
    "            }\n",
    "            print(metrics)\n",
    "            results_dropout.append(aux)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "7c9177e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['results_dropout']"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(results_dropout, 'results_dropout')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6f7a4dd",
   "metadata": {},
   "source": [
    "#### _Dropout_ usando _smote_\n",
    "Repetimos el procedimiento con estos datos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "0ca1315a",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_dropout_smote = []\n",
    "seed = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "a25ad520",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50, '0.1']\n",
      "[0.26300784034212404, 0.26300784034212404, 0.26300784034212404, 0.20085519448743994]\n",
      "[50, '0.1', 50, '0.1']\n",
      "Epoch 00265: early stopping\n",
      "[0.2998930862437634, 0.2998930862437634, 0.2998930862437634, 0.23713346452369544]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00199: early stopping\n",
      "[0.3205630791161796, 0.3205630791161796, 0.3205630791161796, 0.258903190675154]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00162: early stopping\n",
      "[0.30648610121168923, 0.30648610121168923, 0.30648610121168923, 0.2430873107926549]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00142: early stopping\n",
      "[0.3075552387740556, 0.3075552387740556, 0.3075552387740556, 0.24394819172656956]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00099: early stopping\n",
      "[0.28367783321454026, 0.28367783321454026, 0.28367783321454026, 0.22122127360587496]\n",
      "[50, '0.2']\n",
      "Epoch 00271: early stopping\n",
      "[0.2494654312188168, 0.2494654312188168, 0.2494654312188168, 0.1875769065873798]\n",
      "[50, '0.2', 50, '0.2']\n",
      "Epoch 00175: early stopping\n",
      "[0.28421240199572345, 0.28421240199572345, 0.28421240199572345, 0.22168654599358906]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00163: early stopping\n",
      "[0.27619387027797576, 0.27619387027797576, 0.27619387027797576, 0.2133303494383716]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00111: early stopping\n",
      "[0.2729864575908767, 0.2729864575908767, 0.2729864575908767, 0.21067733393453925]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00222: early stopping\n",
      "[0.2920527441197434, 0.2920527441197434, 0.2920527441197434, 0.2270944345574314]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00159: early stopping\n",
      "[0.2719173200285103, 0.2719173200285103, 0.2719173200285103, 0.20919106407082955]\n",
      "[50, '0.3']\n",
      "Epoch 00216: early stopping\n",
      "[0.2459016393442623, 0.2459016393442623, 0.2459016393442623, 0.18529233154432556]\n",
      "[50, '0.3', 50, '0.3']\n",
      "Epoch 00154: early stopping\n",
      "[0.2656806842480399, 0.2656806842480399, 0.2656806842480399, 0.20272738533056844]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00121: early stopping\n",
      "[0.26069137562366357, 0.26069137562366357, 0.26069137562366357, 0.19940871426469609]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00122: early stopping\n",
      "[0.24376336421952957, 0.24376336421952957, 0.24376336421952957, 0.17933947264543937]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00177: early stopping\n",
      "[0.2275481111903065, 0.2275481111903065, 0.2275481111903065, 0.1614652947632501]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00160: early stopping\n",
      "[0.21596578759800428, 0.21596578759800428, 0.21596578759800428, 0.1511632883484848]\n",
      "[100, '0.1']\n",
      "[0.2594440484675695, 0.2594440484675695, 0.2594440484675695, 0.1955059345370388]\n",
      "[100, '0.1', 100, '0.1']\n",
      "[0.3447968638631504, 0.3447968638631504, 0.3447968638631504, 0.28340547320298026]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00219: early stopping\n",
      "[0.372416250890948, 0.372416250890948, 0.372416250890948, 0.31370181177426715]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00185: early stopping\n",
      "[0.35673556664290806, 0.35673556664290806, 0.35673556664290806, 0.2965402996283344]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00164: early stopping\n",
      "[0.36689237348538845, 0.36689237348538845, 0.3668923734853885, 0.3079995227352187]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00100: early stopping\n",
      "[0.35263720598717035, 0.35263720598717035, 0.35263720598717035, 0.29222509720780787]\n",
      "[100, '0.2']\n",
      "Epoch 00290: early stopping\n",
      "[0.25659301496792586, 0.25659301496792586, 0.25659301496792586, 0.19235916152006827]\n",
      "[100, '0.2', 100, '0.2']\n",
      "Epoch 00251: early stopping\n",
      "[0.3143264433357092, 0.3143264433357092, 0.3143264433357092, 0.25120042985546787]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00129: early stopping\n",
      "[0.3282252316464718, 0.3282252316464718, 0.3282252316464718, 0.26708859834267673]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00105: early stopping\n",
      "[0.315751960085531, 0.315751960085531, 0.315751960085531, 0.2545872547346073]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00191: early stopping\n",
      "[0.3321454027084818, 0.3321454027084818, 0.3321454027084818, 0.27051519260022494]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00215: early stopping\n",
      "[0.3225231646471846, 0.3225231646471846, 0.3225231646471846, 0.2600452708848978]\n",
      "[100, '0.3']\n",
      "[0.2553456878118318, 0.2553456878118318, 0.2553456878118318, 0.19117397326811747]\n",
      "[100, '0.3', 100, '0.3']\n",
      "Epoch 00267: early stopping\n",
      "[0.3036350677120456, 0.3036350677120456, 0.3036350677120456, 0.24018452699497383]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00131: early stopping\n",
      "[0.32430506058446185, 0.32430506058446185, 0.32430506058446185, 0.2625623102105059]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00178: early stopping\n",
      "[0.3014967925873129, 0.3014967925873129, 0.3014967925873129, 0.23783108719859625]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00093: early stopping\n",
      "[0.2965074839629366, 0.2965074839629366, 0.2965074839629366, 0.23485587148830267]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00111: early stopping\n",
      "[0.33018531717747684, 0.33018531717747684, 0.33018531717747684, 0.26997768943588174]\n",
      "[150, '0.1']\n",
      "[0.27049180327868855, 0.27049180327868855, 0.27049180327868855, 0.2070689926269983]\n",
      "[150, '0.1', 150, '0.1']\n",
      "Epoch 00236: early stopping\n",
      "[0.36617961511047753, 0.36617961511047753, 0.36617961511047753, 0.3065969957363909]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00179: early stopping\n",
      "[0.3544191019244476, 0.3544191019244476, 0.3544191019244476, 0.29432310566920317]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00151: early stopping\n",
      "[0.38025659301496795, 0.38025659301496795, 0.38025659301496795, 0.3217359080760902]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00199: early stopping\n",
      "[0.38934426229508196, 0.38934426229508196, 0.38934426229508196, 0.3320630520189505]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00179: early stopping\n",
      "[0.39468995010691377, 0.39468995010691377, 0.39468995010691377, 0.33666606146261224]\n",
      "[150, '0.2']\n",
      "[0.2557020669992872, 0.2557020669992872, 0.2557020669992872, 0.19120997021688535]\n",
      "[150, '0.2', 150, '0.2']\n",
      "Epoch 00201: early stopping\n",
      "[0.3499643620812545, 0.3499643620812545, 0.3499643620812545, 0.28962387662704503]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00227: early stopping\n",
      "[0.3643977191732003, 0.3643977191732003, 0.3643977191732003, 0.3047718177375881]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00263: early stopping\n",
      "[0.3832858161083393, 0.3832858161083393, 0.3832858161083393, 0.3251691504162134]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00151: early stopping\n",
      "[0.3891660727013542, 0.3891660727013542, 0.3891660727013542, 0.3318250780448826]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00069: early stopping\n",
      "[0.35548823948681396, 0.35548823948681396, 0.3554882394868139, 0.29658654328800893]\n",
      "[150, '0.3']\n",
      "Epoch 00300: early stopping\n",
      "[0.2540983606557377, 0.2540983606557377, 0.2540983606557377, 0.18919799124540182]\n",
      "[150, '0.3', 150, '0.3']\n",
      "Epoch 00230: early stopping\n",
      "[0.3396293656450463, 0.3396293656450463, 0.3396293656450463, 0.2786682676628811]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00197: early stopping\n",
      "[0.3677833214540271, 0.3677833214540271, 0.36778332145402703, 0.3088770225577919]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00140: early stopping\n",
      "[0.3462223806129722, 0.3462223806129722, 0.3462223806129722, 0.28653161153670004]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00129: early stopping\n",
      "[0.3579828937990021, 0.3579828937990021, 0.35798289379900206, 0.29920706564380195]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00137: early stopping\n",
      "[0.3549536707056308, 0.3549536707056308, 0.35495367070563083, 0.2956493499086582]\n",
      "[200, '0.1']\n",
      "[0.2758374910905203, 0.2758374910905203, 0.2758374910905203, 0.21182638080300087]\n",
      "[200, '0.1', 200, '0.1']\n",
      "Epoch 00203: early stopping\n",
      "[0.36065573770491804, 0.36065573770491804, 0.3606557377049181, 0.30122411585270636]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00187: early stopping\n",
      "[0.37829650748396293, 0.37829650748396293, 0.37829650748396293, 0.319156284144949]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00136: early stopping\n",
      "[0.39522451888809695, 0.39522451888809695, 0.395224518888097, 0.3373918663980583]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00082: early stopping\n",
      "[0.40021382751247325, 0.40021382751247325, 0.40021382751247325, 0.34222347641272977]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00109: early stopping\n",
      "[0.38649322879543835, 0.38649322879543835, 0.38649322879543835, 0.32731143396239426]\n",
      "[200, '0.2']\n",
      "[0.268888096935139, 0.268888096935139, 0.268888096935139, 0.20499839988310808]\n",
      "[200, '0.2', 200, '0.2']\n",
      "Epoch 00167: early stopping\n",
      "[0.35655737704918034, 0.35655737704918034, 0.3565573770491803, 0.2967274796130971]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00232: early stopping\n",
      "[0.40324305060584464, 0.40324305060584464, 0.40324305060584464, 0.3467882041439525]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00161: early stopping\n",
      "[0.3955808980755524, 0.3955808980755524, 0.3955808980755524, 0.3376063898548457]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00241: early stopping\n",
      "[0.3907697790449038, 0.3907697790449038, 0.3907697790449038, 0.3324561672984019]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00141: early stopping\n",
      "[0.3679615110477548, 0.3679615110477548, 0.36796151104775476, 0.3083622483336965]\n",
      "[200, '0.3']\n",
      "Epoch 00153: early stopping\n",
      "[0.26924447612259444, 0.26924447612259444, 0.26924447612259444, 0.2058282571547384]\n",
      "[200, '0.3', 200, '0.3']\n",
      "Epoch 00175: early stopping\n",
      "[0.3556664290805417, 0.3556664290805417, 0.3556664290805417, 0.2950894640652668]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00232: early stopping\n",
      "[0.39326443335709194, 0.39326443335709194, 0.39326443335709194, 0.33580720228153926]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00232: early stopping\n",
      "[0.37954383464005703, 0.37954383464005703, 0.37954383464005703, 0.3216607699137475]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00140: early stopping\n",
      "[0.3638631503920171, 0.3638631503920171, 0.3638631503920171, 0.30498056948130214]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00129: early stopping\n",
      "[0.36689237348538845, 0.36689237348538845, 0.3668923734853885, 0.3069702792714195]\n",
      "[250, '0.1']\n",
      "Epoch 00285: early stopping\n",
      "[0.2753029223093371, 0.2753029223093371, 0.2753029223093371, 0.21098176024808302]\n",
      "[250, '0.1', 250, '0.1']\n",
      "Epoch 00271: early stopping\n",
      "[0.36350677120456165, 0.36350677120456165, 0.36350677120456165, 0.3031787683148758]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00117: early stopping\n",
      "[0.39861012116892375, 0.39861012116892375, 0.39861012116892375, 0.3411751823555649]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00116: early stopping\n",
      "[0.3923734853884533, 0.3923734853884533, 0.3923734853884533, 0.3343374374058219]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00126: early stopping\n",
      "[0.3907697790449038, 0.3907697790449038, 0.3907697790449038, 0.33151015653713056]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00116: early stopping\n",
      "[0.4000356379187455, 0.4000356379187455, 0.4000356379187456, 0.34147287960435413]\n",
      "[250, '0.2']\n",
      "Epoch 00244: early stopping\n",
      "[0.2662152530292231, 0.2662152530292231, 0.2662152530292231, 0.20255805666422488]\n",
      "[250, '0.2', 250, '0.2']\n",
      "Epoch 00252: early stopping\n",
      "[0.3699215965787598, 0.3699215965787598, 0.3699215965787598, 0.312080858448489]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00234: early stopping\n",
      "[0.3995010691375624, 0.3995010691375624, 0.3995010691375624, 0.3421950134241337]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00159: early stopping\n",
      "[0.3923734853884533, 0.3923734853884533, 0.3923734853884533, 0.3339574861445036]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00117: early stopping\n",
      "[0.3914825374198147, 0.3914825374198147, 0.3914825374198147, 0.33271842857445766]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00132: early stopping\n",
      "[0.3995010691375624, 0.3995010691375624, 0.3995010691375624, 0.34119409276086266]\n",
      "[250, '0.3']\n",
      "Epoch 00285: early stopping\n",
      "[0.2690662865288667, 0.2690662865288667, 0.2690662865288667, 0.2052302756253448]\n",
      "[250, '0.3', 250, '0.3']\n",
      "Epoch 00242: early stopping\n",
      "[0.3602993585174626, 0.3602993585174626, 0.3602993585174626, 0.3011976641288431]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00216: early stopping\n",
      "[0.39611546685673554, 0.39611546685673554, 0.39611546685673554, 0.3388907927897745]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00160: early stopping\n",
      "[0.3923734853884533, 0.3923734853884533, 0.3923734853884533, 0.33487742605489634]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00174: early stopping\n",
      "[0.3766928011404134, 0.3766928011404134, 0.3766928011404134, 0.317371421227289]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00160: early stopping\n",
      "[0.387384176764077, 0.387384176764077, 0.387384176764077, 0.3296291053670306]\n"
     ]
    }
   ],
   "source": [
    "size_config = [50, 100, 150, 200, 250]\n",
    "dropout_rate = [\"0.1\", \"0.2\", \"0.3\"]\n",
    "\n",
    "for size in size_config:\n",
    "    for size_d in (dropout_rate):\n",
    "        layer_config_dense = [[size], [size]*2, [size]*3, [size]*4, [size]*5, [size]*6]\n",
    "        layer_config_dropout = [[size_d], [size_d]*2, [size_d]*3, [size_d]*4, [size_d]*5, [size_d]*6]\n",
    "        for layers_dense, layers_dropout in zip(layer_config_dense, layer_config_dropout):\n",
    "            final_design = [None]*(len(layers_dense)+len(layers_dropout))\n",
    "            final_design[::2] = layers_dense\n",
    "            final_design[1::2] = layers_dropout\n",
    "            np.random.seed(seed)\n",
    "            tf.random.set_seed(seed)\n",
    "            print(final_design)\n",
    "            model = make_my_model_multi_dropout(final_design, 6, 18, activation_='relu' )\n",
    "            preds = compile_fit_multiclass(model, X_train_smote, X_test, y_train_smote, 256, 300, verbose=0)\n",
    "            metrics = compute_metrics_multiclass(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "            confusion = confusion_matrix(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "            aux = { \"layer config\" : final_design,\n",
    "                   #\"Model\": model,\n",
    "                   \"Predictions\" : preds,\n",
    "                   \"Metrics\" : metrics,\n",
    "                   \"Confusion\" : confusion\n",
    "\n",
    "            }\n",
    "            print(metrics)\n",
    "            results_dropout_smote.append(aux)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "1d0d1938",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['results_dropout_smote']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import joblib\n",
    "joblib.dump(results_dropout_smote, 'results_dropout_smote')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc7f8f39",
   "metadata": {},
   "source": [
    "## Datos codificados con _one_hot_ con _smote_ y _dropout_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f8f707c",
   "metadata": {},
   "source": [
    "Repetimos los mismos pasos que en el apartado anterior la diferencia de que codificamos los datos de las variables predictoras usando _dummy variables_ o codificación _one-hot_. Así, compararemos el rendimiento de los modelos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5f0a397d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "data = pd.read_csv(\"krkopt.data\", header=None)\n",
    "data.columns = [\"wkc\", \"wkr\", \"wrc\", \"wrr\", \"bkc\", \"bkr\", \"opt rank\" ]\n",
    "X = data.iloc[:, 0:6]\n",
    "y = data['opt rank']\n",
    "X[\"wkc\"]=X[\"wkc\"].astype('category')\n",
    "X[\"wrc\"]=X[\"wrc\"].astype('category')\n",
    "X[\"bkc\"]=X[\"bkc\"].astype('category')\n",
    "X[\"wkc\"]=X[\"wkc\"].cat.codes\n",
    "X[\"wrc\"]=X[\"wrc\"].cat.codes\n",
    "X[\"bkc\"]=X[\"bkc\"].cat.codes\n",
    "y = y.astype('category')\n",
    "y = y.cat.codes\n",
    "\n",
    "from sklearn.preprocessing import OneHotEncoder, LabelEncoder\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "cat_cols = list(X.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0343e1cf",
   "metadata": {},
   "source": [
    "Codificamos usando _get_dummies_ de pandas. Veremos que ahora tenemos 40 variables predictoras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "00c2f862",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wkc_0</th>\n",
       "      <th>wkc_1</th>\n",
       "      <th>wkc_2</th>\n",
       "      <th>wkc_3</th>\n",
       "      <th>wkr_1</th>\n",
       "      <th>wkr_2</th>\n",
       "      <th>wkr_3</th>\n",
       "      <th>wkr_4</th>\n",
       "      <th>wrc_0</th>\n",
       "      <th>wrc_1</th>\n",
       "      <th>...</th>\n",
       "      <th>bkc_6</th>\n",
       "      <th>bkc_7</th>\n",
       "      <th>bkr_1</th>\n",
       "      <th>bkr_2</th>\n",
       "      <th>bkr_3</th>\n",
       "      <th>bkr_4</th>\n",
       "      <th>bkr_5</th>\n",
       "      <th>bkr_6</th>\n",
       "      <th>bkr_7</th>\n",
       "      <th>bkr_8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28051</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28052</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28053</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28054</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28055</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28056 rows × 40 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       wkc_0  wkc_1  wkc_2  wkc_3  wkr_1  wkr_2  wkr_3  wkr_4  wrc_0  wrc_1  \\\n",
       "0          1      0      0      0      1      0      0      0      0      1   \n",
       "1          1      0      0      0      1      0      0      0      0      0   \n",
       "2          1      0      0      0      1      0      0      0      0      0   \n",
       "3          1      0      0      0      1      0      0      0      0      0   \n",
       "4          1      0      0      0      1      0      0      0      0      0   \n",
       "...      ...    ...    ...    ...    ...    ...    ...    ...    ...    ...   \n",
       "28051      0      1      0      0      1      0      0      0      0      0   \n",
       "28052      0      1      0      0      1      0      0      0      0      0   \n",
       "28053      0      1      0      0      1      0      0      0      0      0   \n",
       "28054      0      1      0      0      1      0      0      0      0      0   \n",
       "28055      0      1      0      0      1      0      0      0      0      0   \n",
       "\n",
       "       ...  bkc_6  bkc_7  bkr_1  bkr_2  bkr_3  bkr_4  bkr_5  bkr_6  bkr_7  \\\n",
       "0      ...      0      0      0      1      0      0      0      0      0   \n",
       "1      ...      0      0      0      1      0      0      0      0      0   \n",
       "2      ...      0      0      1      0      0      0      0      0      0   \n",
       "3      ...      0      0      0      1      0      0      0      0      0   \n",
       "4      ...      0      0      1      0      0      0      0      0      0   \n",
       "...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...   \n",
       "28051  ...      0      0      0      0      0      0      1      0      0   \n",
       "28052  ...      0      0      0      0      0      0      0      1      0   \n",
       "28053  ...      0      0      0      0      0      0      0      0      1   \n",
       "28054  ...      0      0      0      0      0      0      1      0      0   \n",
       "28055  ...      1      0      0      0      0      0      1      0      0   \n",
       "\n",
       "       bkr_8  \n",
       "0          0  \n",
       "1          0  \n",
       "2          0  \n",
       "3          0  \n",
       "4          0  \n",
       "...      ...  \n",
       "28051      0  \n",
       "28052      0  \n",
       "28053      0  \n",
       "28054      0  \n",
       "28055      0  \n",
       "\n",
       "[28056 rows x 40 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = pd.get_dummies(X,columns=cat_cols)\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "2a7a87b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,\n",
    "                                                   y, test_size=0.2,\n",
    "                                                   random_state = 1)\n",
    "\n",
    "from imblearn.over_sampling import SMOTE\n",
    "oversample = SMOTE()\n",
    "\n",
    "\n",
    "X_smote, y_smote = oversample.fit_resample(X, y)\n",
    "\n",
    "X_train_smote, X_test_smote, y_train_smote, y_test_smote = train_test_split(X_smote,\n",
    "                                                   y_smote, test_size=0.2,\n",
    "                                                   random_state = 1)\n",
    "\n",
    "\n",
    "\n",
    "y_train = to_categorical(y_train)\n",
    "y_test  = to_categorical(y_test)\n",
    "y_train_smote = to_categorical(y_train_smote)\n",
    "y_test_smote = to_categorical(y_test_smote)\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler().fit(X_train)\n",
    "X_train = scaler.transform(X_train)\n",
    "X_test = scaler.transform(X_test)\n",
    "\n",
    "scaler = StandardScaler().fit(X_train_smote)\n",
    "X_train_smote = scaler.transform(X_train_smote)\n",
    "X_test_smote = scaler.transform(X_test_smote)\n",
    "\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "\n",
    "\n",
    "checkpoint_filepath = '/tmp/checkpoint'\n",
    "\n",
    "from sklearn.metrics import confusion_matrix, precision_score, \\\n",
    "f1_score, cohen_kappa_score, recall_score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25244856",
   "metadata": {},
   "source": [
    "### Pruebas\n",
    "Todos los casos son los mismos que en los datos con la anterior codificación. Guardaremos en archivos todos los objetos de cada conjunto de experimentos.\n",
    "\n",
    "#### Datos  _one_hot_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "e50e14e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = []\n",
    "seed = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "9133c415",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\n",
      "[0.7132929436920884, 0.7132929436920884, 0.7132929436920883, 0.6795940399725571]\n",
      "[50, 50]\n",
      "Epoch 00118: early stopping\n",
      "[0.7359230220955096, 0.7359230220955096, 0.7359230220955095, 0.7051213677686434]\n",
      "[50, 50, 50]\n",
      "Epoch 00075: early stopping\n",
      "[0.7464362081254454, 0.7464362081254454, 0.7464362081254454, 0.7169690264820239]\n",
      "[50, 50, 50, 50]\n",
      "Epoch 00059: early stopping\n",
      "[0.7293300071275838, 0.7293300071275838, 0.7293300071275838, 0.6976471559645523]\n",
      "[50, 50, 50, 50, 50]\n",
      "Epoch 00053: early stopping\n",
      "[0.7323592302209551, 0.7323592302209551, 0.7323592302209551, 0.7011602714040706]\n",
      "[50, 50, 50, 50, 50, 50]\n",
      "Epoch 00054: early stopping\n",
      "[0.7245188880969351, 0.7245188880969351, 0.7245188880969351, 0.6923109986216298]\n",
      "[100]\n",
      "Epoch 00176: early stopping\n",
      "[0.7405559515324305, 0.7405559515324305, 0.7405559515324305, 0.7101055066873646]\n",
      "[100, 100]\n",
      "Epoch 00053: early stopping\n",
      "[0.7624732715609408, 0.7624732715609408, 0.7624732715609408, 0.7348246606533331]\n",
      "[100, 100, 100]\n",
      "Epoch 00042: early stopping\n",
      "[0.7859942979330007, 0.7859942979330007, 0.7859942979330007, 0.761075013060205]\n",
      "[100, 100, 100, 100]\n",
      "Epoch 00039: early stopping\n",
      "[0.7845687811831789, 0.7845687811831789, 0.7845687811831789, 0.7594270506445013]\n",
      "[100, 100, 100, 100, 100]\n",
      "Epoch 00034: early stopping\n",
      "[0.7802922309337135, 0.7802922309337135, 0.7802922309337134, 0.7544084918576515]\n",
      "[100, 100, 100, 100, 100, 100]\n",
      "Epoch 00029: early stopping\n",
      "[0.7590876692801141, 0.7590876692801141, 0.7590876692801141, 0.7307269833855377]\n",
      "[150]\n",
      "Epoch 00150: early stopping\n",
      "[0.7587312900926586, 0.7587312900926586, 0.7587312900926585, 0.7304863274796372]\n",
      "[150, 150]\n",
      "Epoch 00059: early stopping\n",
      "[0.8080898075552387, 0.8080898075552387, 0.8080898075552387, 0.7856176438592615]\n",
      "[150, 150, 150]\n",
      "Epoch 00038: early stopping\n",
      "[0.8071988595866001, 0.8071988595866001, 0.8071988595866, 0.7847174514580395]\n",
      "[150, 150, 150, 150]\n",
      "Epoch 00030: early stopping\n",
      "[0.7995367070563079, 0.7995367070563079, 0.799536707056308, 0.7760993871183968]\n",
      "[150, 150, 150, 150, 150]\n",
      "Epoch 00024: early stopping\n",
      "[0.785816108339273, 0.785816108339273, 0.785816108339273, 0.760816977877812]\n",
      "[150, 150, 150, 150, 150, 150]\n",
      "Epoch 00026: early stopping\n",
      "[0.8013186029935851, 0.8013186029935851, 0.8013186029935851, 0.7778395069749614]\n",
      "[200]\n",
      "Epoch 00108: early stopping\n",
      "[0.7633642195295794, 0.7633642195295794, 0.7633642195295794, 0.7355328449643446]\n",
      "[200, 200]\n",
      "Epoch 00041: early stopping\n",
      "[0.8031004989308624, 0.8031004989308624, 0.8031004989308624, 0.7800639075355223]\n",
      "[200, 200, 200]\n",
      "Epoch 00032: early stopping\n",
      "[0.8096935138987883, 0.8096935138987883, 0.8096935138987883, 0.787236216263784]\n",
      "[200, 200, 200, 200]\n",
      "Epoch 00031: early stopping\n",
      "[0.8308980755523877, 0.8308980755523877, 0.8308980755523878, 0.8111400326514505]\n",
      "[200, 200, 200, 200, 200]\n",
      "Epoch 00028: early stopping\n",
      "[0.8145046329294369, 0.8145046329294369, 0.8145046329294369, 0.7932395201038133]\n",
      "[200, 200, 200, 200, 200, 200]\n",
      "Epoch 00020: early stopping\n",
      "[0.7856379187455452, 0.7856379187455452, 0.7856379187455452, 0.7609006763147126]\n",
      "[250]\n",
      "Epoch 00093: early stopping\n",
      "[0.7756593014967926, 0.7756593014967926, 0.7756593014967926, 0.74921524924313]\n",
      "[250, 250]\n",
      "Epoch 00039: early stopping\n",
      "[0.8241268709907341, 0.8241268709907341, 0.8241268709907341, 0.8035364436115411]\n",
      "[250, 250, 250]\n",
      "Epoch 00030: early stopping\n",
      "[0.8301853171774768, 0.8301853171774768, 0.8301853171774768, 0.8104398918945435]\n",
      "[250, 250, 250, 250]\n",
      "Epoch 00023: early stopping\n",
      "[0.8171774768353528, 0.8171774768353528, 0.8171774768353528, 0.7955487752872001]\n",
      "[250, 250, 250, 250, 250]\n",
      "Epoch 00022: early stopping\n",
      "[0.8038132573057734, 0.8038132573057734, 0.8038132573057732, 0.7810034973873443]\n",
      "[250, 250, 250, 250, 250, 250]\n",
      "Epoch 00022: early stopping\n",
      "[0.8084461867426942, 0.8084461867426942, 0.8084461867426942, 0.7861858456085945]\n"
     ]
    }
   ],
   "source": [
    "size_config = [50, 100, 150, 200, 250]\n",
    "for size in size_config:\n",
    "    layer_config = [[size], [size]*2, [size]*3, [size]*4, [size]*5, [size]*6]\n",
    "    for layers in layer_config:\n",
    "        np.random.seed(seed)\n",
    "        tf.random.set_seed(seed)\n",
    "        print(layers)\n",
    "        model = make_my_model_multi(layers, 40, 18, activation_='relu' )\n",
    "        preds = compile_fit_multiclass(model, X_train, X_test, y_train, 256, 300, verbose=0)\n",
    "        metrics = compute_metrics_multiclass(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "        confusion = confusion_matrix(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "        aux = { \"layer config\" : layers,\n",
    "               #\"Model\": model,\n",
    "               \"Predictions\" : preds,\n",
    "               \"Metrics\" : metrics,\n",
    "               \"Confusion\" : confusion\n",
    "\n",
    "        }\n",
    "        print(metrics)\n",
    "        results.append(aux)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "c36a46dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['results_1_onehot_joblib']"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import joblib\n",
    " \n",
    "joblib.dump(results, 'results_1_onehot_joblib')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2930d4d",
   "metadata": {},
   "source": [
    "### Datos _one_hot_ con _SMOTE_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "796d830a",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_smote = []\n",
    "seed = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "16966118",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\n",
      "[0.09853884533143265, 0.09853884533143265, 0.09853884533143265, 0.05990480607973814]\n",
      "[50, 50]\n",
      "Epoch 00270: early stopping\n",
      "[0.06931575196008553, 0.06931575196008553, 0.06931575196008553, 0.037006904026981036]\n",
      "[50, 50, 50]\n",
      "Epoch 00155: early stopping\n",
      "[0.06967213114754098, 0.06967213114754098, 0.06967213114754098, 0.03441175321042145]\n",
      "[50, 50, 50, 50]\n",
      "Epoch 00153: early stopping\n",
      "[0.07448325017818959, 0.07448325017818959, 0.07448325017818959, 0.04118489587487906]\n",
      "[50, 50, 50, 50, 50]\n",
      "Epoch 00120: early stopping\n",
      "[0.06789023521026372, 0.06789023521026372, 0.06789023521026372, 0.028956419894087593]\n",
      "[50, 50, 50, 50, 50, 50]\n",
      "Epoch 00200: early stopping\n",
      "[0.05434782608695652, 0.05434782608695652, 0.05434782608695652, 0.018471319233337335]\n",
      "[100]\n",
      "[0.11101211689237349, 0.11101211689237349, 0.11101211689237349, 0.07134215115338971]\n",
      "[100, 100]\n",
      "Epoch 00285: early stopping\n",
      "[0.1097647897362794, 0.1097647897362794, 0.1097647897362794, 0.07580671664360639]\n",
      "[100, 100, 100]\n",
      "Epoch 00226: early stopping\n",
      "[0.11279401282965075, 0.11279401282965075, 0.11279401282965075, 0.0702367187003482]\n",
      "[100, 100, 100, 100]\n",
      "Epoch 00157: early stopping\n",
      "[0.09764789736279401, 0.09764789736279401, 0.09764789736279401, 0.058277308802495265]\n",
      "[100, 100, 100, 100, 100]\n",
      "Epoch 00162: early stopping\n",
      "[0.07947255880256593, 0.07947255880256593, 0.07947255880256593, 0.04168589497606212]\n",
      "[100, 100, 100, 100, 100, 100]\n",
      "Epoch 00154: early stopping\n",
      "[0.07662152530292231, 0.07662152530292231, 0.07662152530292231, 0.041859363311582354]\n",
      "[150]\n",
      "[0.11119030648610122, 0.11119030648610122, 0.11119030648610122, 0.07057217306767738]\n",
      "[150, 150]\n",
      "Epoch 00192: early stopping\n",
      "[0.12918745545260157, 0.12918745545260157, 0.12918745545260157, 0.09148213445074216]\n",
      "[150, 150, 150]\n",
      "Epoch 00166: early stopping\n",
      "[0.13809693513898788, 0.13809693513898788, 0.13809693513898788, 0.0914795402998081]\n",
      "[150, 150, 150, 150]\n",
      "Epoch 00140: early stopping\n",
      "[0.10816108339272987, 0.10816108339272987, 0.10816108339272985, 0.06738482220517195]\n",
      "[150, 150, 150, 150, 150]\n",
      "Epoch 00109: early stopping\n",
      "[0.06699928724162509, 0.06699928724162509, 0.06699928724162509, 0.03164283938023682]\n",
      "[150, 150, 150, 150, 150, 150]\n",
      "Epoch 00093: early stopping\n",
      "[0.06254454739843193, 0.06254454739843193, 0.06254454739843193, 0.02622167248184981]\n",
      "[200]\n",
      "Epoch 00192: early stopping\n",
      "[0.09461867426942266, 0.09461867426942266, 0.09461867426942266, 0.0597149180355222]\n",
      "[200, 200]\n",
      "Epoch 00254: early stopping\n",
      "[0.1876336421952958, 0.1876336421952958, 0.1876336421952958, 0.14429603051691353]\n",
      "[200, 200, 200]\n",
      "Epoch 00120: early stopping\n",
      "[0.15217391304347827, 0.15217391304347827, 0.15217391304347827, 0.10779708923931663]\n",
      "[200, 200, 200, 200]\n",
      "Epoch 00129: early stopping\n",
      "[0.1003207412687099, 0.1003207412687099, 0.1003207412687099, 0.06150141024915745]\n",
      "[200, 200, 200, 200, 200]\n",
      "Epoch 00081: early stopping\n",
      "[0.08357091945830364, 0.08357091945830364, 0.08357091945830364, 0.047671217979086355]\n",
      "[200, 200, 200, 200, 200, 200]\n",
      "Epoch 00082: early stopping\n",
      "[0.0792943692088382, 0.0792943692088382, 0.0792943692088382, 0.0425269634720844]\n",
      "[250]\n",
      "Epoch 00254: early stopping\n",
      "[0.10655737704918032, 0.10655737704918032, 0.10655737704918032, 0.07031624765828526]\n",
      "[250, 250]\n",
      "Epoch 00174: early stopping\n",
      "[0.14593727726300784, 0.14593727726300784, 0.14593727726300784, 0.10398057068418642]\n",
      "[250, 250, 250]\n",
      "Epoch 00106: early stopping\n",
      "[0.17783321454027085, 0.17783321454027085, 0.17783321454027085, 0.13176963959600485]\n",
      "[250, 250, 250, 250]\n",
      "Epoch 00087: early stopping\n",
      "[0.1172487526728439, 0.1172487526728439, 0.1172487526728439, 0.07717883510268653]\n",
      "[250, 250, 250, 250, 250]\n",
      "Epoch 00081: early stopping\n",
      "[0.08802565930149679, 0.08802565930149679, 0.08802565930149679, 0.05372309817465015]\n",
      "[250, 250, 250, 250, 250, 250]\n",
      "Epoch 00058: early stopping\n",
      "[0.08214540270848182, 0.08214540270848182, 0.08214540270848182, 0.047641839366908134]\n"
     ]
    }
   ],
   "source": [
    "size_config = [50, 100, 150, 200, 250]\n",
    "for size in size_config:\n",
    "    layer_config = [[size], [size]*2, [size]*3, [size]*4, [size]*5, [size]*6]\n",
    "    for layers in layer_config:\n",
    "        np.random.seed(seed)\n",
    "        tf.random.set_seed(seed)\n",
    "        print(layers)\n",
    "        model = make_my_model_multi(layers, 40, 18, activation_='relu' )\n",
    "        preds = compile_fit_multiclass(model, X_train_smote, X_test, y_train_smote, 256, 300, verbose=0)\n",
    "        metrics = compute_metrics_multiclass(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "        confusion = confusion_matrix(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "        aux = { \"layer config\" : layers,\n",
    "               #\"Model\": model,\n",
    "               \"Predictions\" : preds,\n",
    "               \"Metrics\" : metrics,\n",
    "               \"Confusion\" : confusion\n",
    "\n",
    "        }\n",
    "        print(metrics)\n",
    "        results_smote.append(aux)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "db6a676d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['results_smote_onehot_joblib']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(results_smote, 'results_smote_onehot_joblib')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df502e8a",
   "metadata": {},
   "source": [
    "### Datos _one_hot_ con _dropout_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "ed8f5094",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_dropout = []\n",
    "seed = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "71cf813a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50, '0.1']\n",
      "[0.6911974340698503, 0.6911974340698503, 0.6911974340698503, 0.6544424729231852]\n",
      "[50, '0.1', 50, '0.1']\n",
      "Epoch 00249: early stopping\n",
      "[0.7549893086243763, 0.7549893086243763, 0.7549893086243763, 0.7260805176671765]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00244: early stopping\n",
      "[0.7685317177476836, 0.7685317177476836, 0.7685317177476836, 0.7412666342935919]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00204: early stopping\n",
      "[0.7628296507483963, 0.7628296507483963, 0.7628296507483963, 0.7349489199206065]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00204: early stopping\n",
      "[0.755167498218104, 0.755167498218104, 0.755167498218104, 0.726566737550184]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00138: early stopping\n",
      "[0.7446543121881682, 0.7446543121881682, 0.7446543121881682, 0.7145851936293055]\n",
      "[50, '0.2']\n",
      "[0.6797933000712758, 0.6797933000712758, 0.6797933000712758, 0.6415147506179972]\n",
      "[50, '0.2', 50, '0.2']\n",
      "Epoch 00194: early stopping\n",
      "[0.7211332858161084, 0.7211332858161084, 0.7211332858161085, 0.6880538576802332]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00202: early stopping\n",
      "[0.7189950106913756, 0.7189950106913756, 0.7189950106913756, 0.6857954888177956]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00177: early stopping\n",
      "[0.7049180327868853, 0.7049180327868853, 0.7049180327868853, 0.6696354479193032]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00184: early stopping\n",
      "[0.7040270848182466, 0.7040270848182466, 0.7040270848182466, 0.6688250607918782]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00225: early stopping\n",
      "[0.6983250178189594, 0.6983250178189594, 0.6983250178189594, 0.6625351015506182]\n",
      "[50, '0.3']\n",
      "Epoch 00290: early stopping\n",
      "[0.6584105488239487, 0.6584105488239487, 0.6584105488239487, 0.6172402972361986]\n",
      "[50, '0.3', 50, '0.3']\n",
      "Epoch 00182: early stopping\n",
      "[0.6837134711332858, 0.6837134711332858, 0.6837134711332858, 0.6456665282835925]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00152: early stopping\n",
      "[0.66928011404134, 0.66928011404134, 0.66928011404134, 0.6293258262824244]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00182: early stopping\n",
      "[0.655559515324305, 0.655559515324305, 0.655559515324305, 0.6142799530678658]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00192: early stopping\n",
      "[0.6407697790449037, 0.6407697790449037, 0.6407697790449037, 0.5974540868981456]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00216: early stopping\n",
      "[0.6076265146115467, 0.6076265146115467, 0.6076265146115467, 0.5617307756281984]\n",
      "[100, '0.1']\n",
      "Epoch 00267: early stopping\n",
      "[0.7537419814682823, 0.7537419814682823, 0.7537419814682823, 0.7247488644953375]\n",
      "[100, '0.1', 100, '0.1']\n",
      "Epoch 00193: early stopping\n",
      "[0.8221667854597291, 0.8221667854597291, 0.8221667854597291, 0.8014009505335618]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00182: early stopping\n",
      "[0.8504989308624377, 0.8504989308624377, 0.8504989308624377, 0.8330632580919843]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00146: early stopping\n",
      "[0.8480042765502495, 0.8480042765502495, 0.8480042765502495, 0.8302515913483358]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00148: early stopping\n",
      "[0.8478260869565217, 0.8478260869565217, 0.8478260869565218, 0.8300096411794345]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00150: early stopping\n",
      "[0.8460441910192444, 0.8460441910192444, 0.8460441910192444, 0.8281644104059738]\n",
      "[100, '0.2']\n",
      "[0.7535637918745546, 0.7535637918745546, 0.7535637918745546, 0.7243112074616954]\n",
      "[100, '0.2', 100, '0.2']\n",
      "Epoch 00200: early stopping\n",
      "[0.8059515324305061, 0.8059515324305061, 0.8059515324305061, 0.7831509826329692]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00258: early stopping\n",
      "[0.8403421240199572, 0.8403421240199572, 0.8403421240199572, 0.8216186407532786]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00189: early stopping\n",
      "[0.8145046329294369, 0.8145046329294369, 0.8145046329294369, 0.7927907643878691]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00129: early stopping\n",
      "[0.792943692088382, 0.792943692088382, 0.792943692088382, 0.7687186237795841]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00236: early stopping\n",
      "[0.8111190306486101, 0.8111190306486101, 0.8111190306486101, 0.7890115939135013]\n",
      "[100, '0.3']\n",
      "[0.7442979330007128, 0.7442979330007128, 0.7442979330007128, 0.7138441190567824]\n",
      "[100, '0.3', 100, '0.3']\n",
      "Epoch 00208: early stopping\n",
      "[0.7888453314326443, 0.7888453314326443, 0.7888453314326443, 0.7640197609347319]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00221: early stopping\n",
      "[0.8061297220242338, 0.8061297220242338, 0.8061297220242338, 0.7833711703582897]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00153: early stopping\n",
      "[0.7676407697790449, 0.7676407697790449, 0.7676407697790449, 0.7403424377519607]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00156: early stopping\n",
      "[0.7667498218104063, 0.7667498218104063, 0.7667498218104063, 0.7393739781195857]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00176: early stopping\n",
      "[0.7414468995010691, 0.7414468995010691, 0.7414468995010691, 0.7111252105230017]\n",
      "[150, '0.1']\n",
      "[0.7820741268709908, 0.7820741268709908, 0.7820741268709906, 0.7564243977333958]\n",
      "[150, '0.1', 150, '0.1']\n",
      "Epoch 00129: early stopping\n",
      "[0.855488239486814, 0.855488239486814, 0.855488239486814, 0.8385669050046001]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00079: early stopping\n",
      "[0.8590520313613685, 0.8590520313613685, 0.8590520313613685, 0.8426740003139367]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00109: early stopping\n",
      "[0.8781183178902352, 0.8781183178902352, 0.8781183178902352, 0.8639675844777932]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00094: early stopping\n",
      "[0.8700997861724875, 0.8700997861724875, 0.8700997861724875, 0.8549745938221411]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00093: early stopping\n",
      "[0.8695652173913043, 0.8695652173913043, 0.8695652173913043, 0.8543679160066875]\n",
      "[150, '0.2']\n",
      "Epoch 00248: early stopping\n",
      "[0.7751247327156094, 0.7751247327156094, 0.7751247327156094, 0.7486044176314802]\n",
      "[150, '0.2', 150, '0.2']\n",
      "Epoch 00220: early stopping\n",
      "[0.860655737704918, 0.860655737704918, 0.860655737704918, 0.844376829162256]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00231: early stopping\n",
      "[0.8761582323592302, 0.8761582323592302, 0.8761582323592302, 0.8616730939921385]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00199: early stopping\n",
      "[0.8679615110477548, 0.8679615110477548, 0.8679615110477547, 0.852580315610961]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00136: early stopping\n",
      "[0.8601211689237348, 0.8601211689237348, 0.8601211689237348, 0.8437704300336455]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00139: early stopping\n",
      "[0.8535281539558089, 0.8535281539558089, 0.8535281539558089, 0.8364105315970465]\n",
      "[150, '0.3']\n",
      "Epoch 00253: early stopping\n",
      "[0.7713827512473271, 0.7713827512473271, 0.7713827512473271, 0.744263007539312]\n",
      "[150, '0.3', 150, '0.3']\n",
      "Epoch 00214: early stopping\n",
      "[0.8458660014255167, 0.8458660014255167, 0.8458660014255168, 0.8278767855942375]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00220: early stopping\n",
      "[0.840520313613685, 0.840520313613685, 0.840520313613685, 0.8219800613018564]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00173: early stopping\n",
      "[0.8300071275837491, 0.8300071275837491, 0.8300071275837491, 0.8101729751256426]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00171: early stopping\n",
      "[0.8218104062722738, 0.8218104062722738, 0.8218104062722738, 0.8010098078549712]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00130: early stopping\n",
      "[0.7952601568068425, 0.7952601568068425, 0.7952601568068425, 0.7712983060607658]\n",
      "[200, '0.1']\n",
      "Epoch 00228: early stopping\n",
      "[0.7961511047754811, 0.7961511047754811, 0.796151104775481, 0.7720837157452116]\n",
      "[200, '0.1', 200, '0.1']\n",
      "Epoch 00097: early stopping\n",
      "[0.8569137562366358, 0.8569137562366358, 0.8569137562366358, 0.8402549944374698]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00098: early stopping\n",
      "[0.8884533143264434, 0.8884533143264434, 0.8884533143264434, 0.8754311944881142]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00065: early stopping\n",
      "[0.8700997861724875, 0.8700997861724875, 0.8700997861724875, 0.854908986256218]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00077: early stopping\n",
      "[0.8845331432644333, 0.8845331432644333, 0.8845331432644333, 0.8710306582427136]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00068: early stopping\n",
      "[0.8734853884533144, 0.8734853884533144, 0.8734853884533144, 0.8588150087272403]\n",
      "[200, '0.2']\n",
      "Epoch 00292: early stopping\n",
      "[0.7993585174625801, 0.7993585174625801, 0.7993585174625801, 0.7756947986927558]\n",
      "[200, '0.2', 200, '0.2']\n",
      "Epoch 00173: early stopping\n",
      "[0.8761582323592302, 0.8761582323592302, 0.8761582323592302, 0.8616832826121679]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00178: early stopping\n",
      "[0.8929080541696365, 0.8929080541696365, 0.8929080541696365, 0.8804258424474269]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00122: early stopping\n",
      "[0.8889878831076266, 0.8889878831076266, 0.8889878831076266, 0.876025663923951]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00113: early stopping\n",
      "[0.8752672843905915, 0.8752672843905915, 0.8752672843905915, 0.86073847814212]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00162: early stopping\n",
      "[0.8863150392017106, 0.8863150392017106, 0.8863150392017106, 0.8731436557602646]\n",
      "[200, '0.3']\n",
      "[0.7986457590876693, 0.7986457590876693, 0.7986457590876693, 0.7748684094073761]\n",
      "[200, '0.3', 200, '0.3']\n",
      "Epoch 00230: early stopping\n",
      "[0.8674269422665716, 0.8674269422665716, 0.8674269422665717, 0.8519922476261953]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00214: early stopping\n",
      "[0.8788310762651461, 0.8788310762651461, 0.8788310762651462, 0.8646983800933996]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00172: early stopping\n",
      "[0.8717034925160371, 0.8717034925160371, 0.8717034925160371, 0.8566902616204991]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00223: early stopping\n",
      "[0.8738417676407698, 0.8738417676407698, 0.8738417676407699, 0.8591722739548441]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00165: early stopping\n",
      "[0.8581610833927299, 0.8581610833927299, 0.8581610833927299, 0.8415628528427186]\n",
      "[250, '0.1']\n",
      "Epoch 00175: early stopping\n",
      "[0.7995367070563079, 0.7995367070563079, 0.799536707056308, 0.7759209362036418]\n",
      "[250, '0.1', 250, '0.1']\n",
      "Epoch 00089: early stopping\n",
      "[0.8727726300784034, 0.8727726300784034, 0.8727726300784034, 0.8579218239707193]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00071: early stopping\n",
      "[0.8864932287954383, 0.8864932287954383, 0.8864932287954383, 0.873233938730633]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00054: early stopping\n",
      "[0.8807911617961511, 0.8807911617961511, 0.8807911617961511, 0.866858669993142]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00069: early stopping\n",
      "[0.8875623663578047, 0.8875623663578047, 0.8875623663578046, 0.8744596060520364]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00087: early stopping\n",
      "[0.894511760513186, 0.894511760513186, 0.894511760513186, 0.8821913607432853]\n",
      "[250, '0.2']\n",
      "Epoch 00177: early stopping\n",
      "[0.7922309337134711, 0.7922309337134711, 0.7922309337134711, 0.7676712411254928]\n",
      "[250, '0.2', 250, '0.2']\n",
      "Epoch 00183: early stopping\n",
      "[0.8918389166072701, 0.8918389166072701, 0.8918389166072701, 0.8792620364600945]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00093: early stopping\n",
      "[0.8877405559515325, 0.8877405559515325, 0.8877405559515325, 0.8746325738298955]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00096: early stopping\n",
      "[0.8872059871703493, 0.8872059871703493, 0.8872059871703494, 0.874061027009303]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00090: early stopping\n",
      "[0.8800784034212402, 0.8800784034212402, 0.8800784034212402, 0.8661096664833664]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00124: early stopping\n",
      "[0.8866714183891661, 0.8866714183891661, 0.8866714183891661, 0.8735009741103202]\n",
      "[250, '0.3']\n",
      "Epoch 00273: early stopping\n",
      "[0.8107626514611547, 0.8107626514611547, 0.8107626514611547, 0.7883434831162922]\n",
      "[250, '0.3', 250, '0.3']\n",
      "Epoch 00182: early stopping\n",
      "[0.8809693513898789, 0.8809693513898789, 0.8809693513898789, 0.8671143715434837]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00133: early stopping\n",
      "[0.8847113328581611, 0.8847113328581611, 0.884711332858161, 0.8712432971205801]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00164: early stopping\n",
      "[0.8850677120456165, 0.8850677120456165, 0.8850677120456164, 0.871636611045392]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00189: early stopping\n",
      "[0.8875623663578047, 0.8875623663578047, 0.8875623663578046, 0.874455680487535]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00142: early stopping\n",
      "[0.8711689237348539, 0.8711689237348539, 0.871168923734854, 0.856223700873793]\n"
     ]
    }
   ],
   "source": [
    "size_config = [50, 100, 150, 200, 250]\n",
    "dropout_rate = [\"0.1\", \"0.2\", \"0.3\"]\n",
    "\n",
    "for size in size_config:\n",
    "    for size_d in (dropout_rate):\n",
    "        layer_config_dense = [[size], [size]*2, [size]*3, [size]*4, [size]*5, [size]*6]\n",
    "        layer_config_dropout = [[size_d], [size_d]*2, [size_d]*3, [size_d]*4, [size_d]*5, [size_d]*6]\n",
    "        for layers_dense, layers_dropout in zip(layer_config_dense, layer_config_dropout):\n",
    "            final_design = [None]*(len(layers_dense)+len(layers_dropout))\n",
    "            final_design[::2] = layers_dense\n",
    "            final_design[1::2] = layers_dropout\n",
    "            np.random.seed(seed)\n",
    "            tf.random.set_seed(seed)\n",
    "            print(final_design)\n",
    "            model = make_my_model_multi_dropout(final_design, 40, 18, activation_='relu' )\n",
    "            preds = compile_fit_multiclass(model, X_train, X_test, y_train, 256, 300, verbose=0)\n",
    "            metrics = compute_metrics_multiclass(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "            confusion = confusion_matrix(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "            aux = { \"layer config\" : final_design,\n",
    "                   #\"Model\": model,\n",
    "                   \"Predictions\" : preds,\n",
    "                   \"Metrics\" : metrics,\n",
    "                   \"Confusion\" : confusion\n",
    "\n",
    "            }\n",
    "            print(metrics)\n",
    "            results_dropout.append(aux)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "96434a94",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['results_dropout_one_hot']"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(results_dropout, 'results_dropout_one_hot')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32ec72fe",
   "metadata": {},
   "source": [
    "#### Datos _one_hot_ _SMOTE dropout_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "0db5570f",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_dropout_smote = []\n",
    "seed = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "a844d985",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50, '0.1']\n",
      "[0.17925873129009265, 0.17925873129009265, 0.17925873129009265, 0.12232832383335368]\n",
      "[50, '0.1', 50, '0.1']\n",
      "[0.1261582323592302, 0.1261582323592302, 0.1261582323592302, 0.08089709192596017]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00248: early stopping\n",
      "[0.0650392017106201, 0.0650392017106201, 0.0650392017106201, 0.03040751666739172]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00189: early stopping\n",
      "[0.042230933713471135, 0.042230933713471135, 0.042230933713471135, 0.01270048583862049]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00194: early stopping\n",
      "[0.038132573057733425, 0.038132573057733425, 0.038132573057733425, 0.005452686722108413]\n",
      "[50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1', 50, '0.1']\n",
      "Epoch 00232: early stopping\n",
      "[0.0345687811831789, 0.0345687811831789, 0.0345687811831789, 0.008312049551169154]\n",
      "[50, '0.2']\n",
      "[0.1626870990734141, 0.1626870990734141, 0.1626870990734141, 0.10466572690499598]\n",
      "[50, '0.2', 50, '0.2']\n",
      "Epoch 00255: early stopping\n",
      "[0.13863150392017107, 0.13863150392017107, 0.13863150392017107, 0.085656992318495]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00128: early stopping\n",
      "[0.05737704918032787, 0.05737704918032787, 0.05737704918032787, 0.02477694026158095]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00191: early stopping\n",
      "[0.03189593727726301, 0.03189593727726301, 0.03189593727726301, 0.0039150488723641574]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00126: early stopping\n",
      "[0.033856022808267994, 0.033856022808267994, 0.033856022808267994, 0.009683362769514314]\n",
      "[50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2', 50, '0.2']\n",
      "Epoch 00153: early stopping\n",
      "[0.043478260869565216, 0.043478260869565216, 0.043478260869565216, 0.021719267754409355]\n",
      "[50, '0.3']\n",
      "[0.17587312900926586, 0.17587312900926586, 0.17587312900926586, 0.12135664118870049]\n",
      "[50, '0.3', 50, '0.3']\n",
      "Epoch 00250: early stopping\n",
      "[0.101568068424804, 0.101568068424804, 0.101568068424804, 0.05585754164033441]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00186: early stopping\n",
      "[0.049002138275124736, 0.049002138275124736, 0.04900213827512473, 0.025606993164014047]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00152: early stopping\n",
      "[0.023521026372059873, 0.023521026372059873, 0.023521026372059876, -0.0008839246776410903]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00122: early stopping\n",
      "[0.019600855310049892, 0.019600855310049892, 0.019600855310049892, 0.0035310058483916107]\n",
      "[50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3', 50, '0.3']\n",
      "Epoch 00110: early stopping\n",
      "[0.02690662865288667, 0.02690662865288667, 0.02690662865288667, 0.009679007414853946]\n",
      "[100, '0.1']\n",
      "[0.20224518888096935, 0.20224518888096935, 0.20224518888096935, 0.14544292498918487]\n",
      "[100, '0.1', 100, '0.1']\n",
      "Epoch 00240: early stopping\n",
      "[0.24679258731290094, 0.24679258731290094, 0.24679258731290094, 0.1934243712900997]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00198: early stopping\n",
      "[0.09230220955096223, 0.09230220955096223, 0.09230220955096224, 0.05140908127877386]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "[0.07305773342836779, 0.07305773342836779, 0.07305773342836779, 0.03229868220347709]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00239: early stopping\n",
      "[0.05238774055595153, 0.05238774055595153, 0.05238774055595153, 0.01593217893389176]\n",
      "[100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1', 100, '0.1']\n",
      "Epoch 00220: early stopping\n",
      "[0.05327868852459016, 0.05327868852459016, 0.05327868852459016, 0.018773343435891765]\n",
      "[100, '0.2']\n",
      "[0.2241625089094797, 0.2241625089094797, 0.2241625089094797, 0.16380451668419682]\n",
      "[100, '0.2', 100, '0.2']\n",
      "[0.20990734141126158, 0.20990734141126158, 0.20990734141126158, 0.1564546087020312]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2']\n",
      "[0.0935495367070563, 0.0935495367070563, 0.0935495367070563, 0.049653103710961655]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00172: early stopping\n",
      "[0.039379900213827514, 0.039379900213827514, 0.039379900213827514, 0.010444041860423137]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "[0.06147540983606557, 0.06147540983606557, 0.06147540983606557, 0.016954035230797748]\n",
      "[100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2', 100, '0.2']\n",
      "Epoch 00139: early stopping\n",
      "[0.05915894511760513, 0.05915894511760513, 0.05915894511760513, 0.018481805134154428]\n",
      "[100, '0.3']\n",
      "[0.20420527441197434, 0.20420527441197434, 0.20420527441197434, 0.146919703577035]\n",
      "[100, '0.3', 100, '0.3']\n",
      "[0.17230933713471133, 0.17230933713471133, 0.17230933713471133, 0.11901675774047382]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00193: early stopping\n",
      "[0.042943692088382036, 0.042943692088382036, 0.042943692088382036, 0.01206205191427645]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00164: early stopping\n",
      "[0.03991446899501069, 0.03991446899501069, 0.03991446899501069, 0.00613134448216468]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00189: early stopping\n",
      "[0.04240912330719886, 0.04240912330719886, 0.04240912330719886, 0.00991607580343623]\n",
      "[100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3', 100, '0.3']\n",
      "Epoch 00179: early stopping\n",
      "[0.03563791874554526, 0.03563791874554526, 0.03563791874554526, 0.012061464890697371]\n",
      "[150, '0.1']\n",
      "[0.1970776906628653, 0.1970776906628653, 0.1970776906628653, 0.14580931580526268]\n",
      "[150, '0.1', 150, '0.1']\n",
      "[0.3145046329294369, 0.3145046329294369, 0.3145046329294369, 0.26160453102930736]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00217: early stopping\n",
      "[0.17836778332145403, 0.17836778332145403, 0.17836778332145403, 0.13225478504378496]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "[0.09390591589451176, 0.09390591589451176, 0.09390591589451176, 0.057542665344226474]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "[0.0718104062722737, 0.0718104062722737, 0.0718104062722737, 0.033419143155595354]\n",
      "[150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1', 150, '0.1']\n",
      "Epoch 00233: early stopping\n",
      "[0.0616535994297933, 0.0616535994297933, 0.0616535994297933, 0.02521055524335547]\n",
      "[150, '0.2']\n",
      "[0.22042052744119744, 0.22042052744119744, 0.22042052744119744, 0.1651157706698071]\n",
      "[150, '0.2', 150, '0.2']\n",
      "Epoch 00264: early stopping\n",
      "[0.23431931575196008, 0.23431931575196008, 0.23431931575196008, 0.1814566324589243]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00181: early stopping\n",
      "[0.12883107626514612, 0.12883107626514612, 0.12883107626514612, 0.08123675934587382]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00227: early stopping\n",
      "[0.08125445473984319, 0.08125445473984319, 0.08125445473984319, 0.042625704486619176]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00284: early stopping\n",
      "[0.05666429080541696, 0.05666429080541696, 0.05666429080541696, 0.022088509992709504]\n",
      "[150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2', 150, '0.2']\n",
      "Epoch 00206: early stopping\n",
      "[0.04686386315039202, 0.04686386315039202, 0.04686386315039202, 0.013243538873656369]\n",
      "[150, '0.3']\n",
      "[0.2166785459729152, 0.2166785459729152, 0.2166785459729152, 0.16013192583393254]\n",
      "[150, '0.3', 150, '0.3']\n",
      "Epoch 00269: early stopping\n",
      "[0.20153243050605846, 0.20153243050605846, 0.20153243050605846, 0.1489163218797086]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00233: early stopping\n",
      "[0.09141126158232359, 0.09141126158232359, 0.09141126158232359, 0.05088555218453383]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00235: early stopping\n",
      "[0.048467569493941556, 0.048467569493941556, 0.048467569493941556, 0.01460299355630068]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00239: early stopping\n",
      "[0.04436920883820385, 0.04436920883820385, 0.044369208838203854, 0.0130637778773115]\n",
      "[150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3', 150, '0.3']\n",
      "Epoch 00229: early stopping\n",
      "[0.04526015680684248, 0.04526015680684248, 0.045260156806842484, 0.007616860340662668]\n",
      "[200, '0.1']\n",
      "[0.2533856022808268, 0.2533856022808268, 0.2533856022808268, 0.197391515941576]\n",
      "[200, '0.1', 200, '0.1']\n",
      "[0.3708125445473984, 0.3708125445473984, 0.3708125445473984, 0.3210530887142903]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00276: early stopping\n",
      "[0.22202423378474698, 0.22202423378474698, 0.22202423378474698, 0.17342244981580968]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00187: early stopping\n",
      "[0.11546685673556664, 0.11546685673556664, 0.11546685673556664, 0.07693413882061018]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00295: early stopping\n",
      "[0.09248039914468995, 0.09248039914468995, 0.09248039914468995, 0.054437648776313186]\n",
      "[200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1', 200, '0.1']\n",
      "Epoch 00223: early stopping\n",
      "[0.07056307911617961, 0.07056307911617961, 0.07056307911617961, 0.03598036748298006]\n",
      "[200, '0.2']\n",
      "[0.2569493941553813, 0.2569493941553813, 0.2569493941553813, 0.1990989622048731]\n",
      "[200, '0.2', 200, '0.2']\n",
      "Epoch 00300: early stopping\n",
      "[0.28813257305773343, 0.28813257305773343, 0.28813257305773343, 0.23566194103112692]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00297: early stopping\n",
      "[0.216143977191732, 0.216143977191732, 0.216143977191732, 0.16476544862583697]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00206: early stopping\n",
      "[0.09016393442622951, 0.09016393442622951, 0.09016393442622953, 0.051738070452585716]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "[0.08107626514611546, 0.08107626514611546, 0.08107626514611546, 0.04782415240770388]\n",
      "[200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2', 200, '0.2']\n",
      "Epoch 00194: early stopping\n",
      "[0.06290092658588739, 0.06290092658588739, 0.06290092658588739, 0.02886584321690111]\n",
      "[200, '0.3']\n",
      "[0.2753029223093371, 0.2753029223093371, 0.2753029223093371, 0.2168872081718204]\n",
      "[200, '0.3', 200, '0.3']\n",
      "[0.231111903064861, 0.231111903064861, 0.231111903064861, 0.17615806450636173]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00246: early stopping\n",
      "[0.10727013542409124, 0.10727013542409124, 0.10727013542409124, 0.06523663598466534]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "[0.061831789023521024, 0.061831789023521024, 0.061831789023521024, 0.03089862212446237]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00245: early stopping\n",
      "[0.043300071275837494, 0.043300071275837494, 0.043300071275837494, 0.011053024973720404]\n",
      "[200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3', 200, '0.3']\n",
      "Epoch 00280: early stopping\n",
      "[0.040805416963649324, 0.040805416963649324, 0.040805416963649324, 0.011774770560165515]\n",
      "[250, '0.1']\n",
      "[0.26300784034212404, 0.26300784034212404, 0.26300784034212404, 0.20673535816629096]\n",
      "[250, '0.1', 250, '0.1']\n",
      "Epoch 00241: early stopping\n",
      "[0.3558446186742694, 0.3558446186742694, 0.35584461867426936, 0.3066293157389701]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00169: early stopping\n",
      "[0.24358517462580184, 0.24358517462580184, 0.24358517462580184, 0.1961415507087051]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00167: early stopping\n",
      "[0.1443335709194583, 0.1443335709194583, 0.1443335709194583, 0.10284190805322913]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00163: early stopping\n",
      "[0.08267997148966501, 0.08267997148966501, 0.08267997148966501, 0.04849906766953527]\n",
      "[250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1', 250, '0.1']\n",
      "Epoch 00245: early stopping\n",
      "[0.08856022808267996, 0.08856022808267996, 0.08856022808267996, 0.05350068877068037]\n",
      "[250, '0.2']\n",
      "[0.2966856735566643, 0.2966856735566643, 0.2966856735566643, 0.23811703510767068]\n",
      "[250, '0.2', 250, '0.2']\n",
      "[0.3670705630791162, 0.3670705630791162, 0.3670705630791161, 0.3162973225364475]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2']\n",
      "[0.17979330007127584, 0.17979330007127584, 0.17979330007127584, 0.1300558613010846]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00276: early stopping\n",
      "[0.12081254454739843, 0.12081254454739843, 0.12081254454739844, 0.08267604380040405]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "Epoch 00188: early stopping\n",
      "[0.07323592302209551, 0.07323592302209551, 0.07323592302209551, 0.039733431057557]\n",
      "[250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2', 250, '0.2']\n",
      "[0.07377049180327869, 0.07377049180327869, 0.07377049180327869, 0.041056259095956116]\n",
      "[250, '0.3']\n",
      "[0.3132573057733428, 0.3132573057733428, 0.3132573057733428, 0.25290627435956714]\n",
      "[250, '0.3', 250, '0.3']\n",
      "Epoch 00284: early stopping\n",
      "[0.2425160370634355, 0.2425160370634355, 0.2425160370634355, 0.19115487345844973]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00270: early stopping\n",
      "[0.10762651461154668, 0.10762651461154668, 0.10762651461154668, 0.0654504653495317]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00211: early stopping\n",
      "[0.0778688524590164, 0.0778688524590164, 0.0778688524590164, 0.04145904731879446]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "Epoch 00232: early stopping\n",
      "[0.05167498218104063, 0.05167498218104063, 0.05167498218104063, 0.019842027765843095]\n",
      "[250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3', 250, '0.3']\n",
      "[0.05434782608695652, 0.05434782608695652, 0.05434782608695652, 0.026805608899612032]\n"
     ]
    }
   ],
   "source": [
    "size_config = [50, 100, 150, 200, 250]\n",
    "dropout_rate = [\"0.1\", \"0.2\", \"0.3\"]\n",
    "\n",
    "for size in size_config:\n",
    "    for size_d in (dropout_rate):\n",
    "        layer_config_dense = [[size], [size]*2, [size]*3, [size]*4, [size]*5, [size]*6]\n",
    "        layer_config_dropout = [[size_d], [size_d]*2, [size_d]*3, [size_d]*4, [size_d]*5, [size_d]*6]\n",
    "        for layers_dense, layers_dropout in zip(layer_config_dense, layer_config_dropout):\n",
    "            final_design = [None]*(len(layers_dense)+len(layers_dropout))\n",
    "            final_design[::2] = layers_dense\n",
    "            final_design[1::2] = layers_dropout\n",
    "            np.random.seed(seed)\n",
    "            tf.random.set_seed(seed)\n",
    "            print(final_design)\n",
    "            model = make_my_model_multi_dropout(final_design, 40, 18, activation_='relu' )\n",
    "            preds = compile_fit_multiclass(model, X_train_smote, X_test, y_train_smote, 256, 300, verbose=0)\n",
    "            metrics = compute_metrics_multiclass(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "            confusion = confusion_matrix(np.argmax(preds, axis = 1), np.argmax(y_test, axis = 1))\n",
    "            aux = { \"layer config\" : final_design,\n",
    "                   #\"Model\": model,\n",
    "                   \"Predictions\" : preds,\n",
    "                   \"Metrics\" : metrics,\n",
    "                   \"Confusion\" : confusion\n",
    "\n",
    "            }\n",
    "            print(metrics)\n",
    "            results_dropout_smote.append(aux)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "2d22f56c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['results_dropout_smote_one_hot']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import joblib\n",
    "joblib.dump(results_dropout_smote, 'results_dropout_smote_one_hot')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30e5f50a",
   "metadata": {},
   "source": [
    "## Análisis de los resultados\n",
    "Una vez que hemos producido los experimentos, leemos los conjuntos de datos y creamos un dataset con todas las configuraciones y las métricas obtenidas. Para ello hacemos uso de la función _ReadAndCreate_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "bcb64c55",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "70a3c79e",
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_config = []\n",
    "Precision = []\n",
    "Recall = []\n",
    "F1 = []\n",
    "Cohen_kappa = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "da27f3f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def ReadAndCreate(file):\n",
    "    results_1 = joblib.load(file)\n",
    "    layer_config = []\n",
    "    Precision = []\n",
    "    Recall = []\n",
    "    F1 = []\n",
    "    Cohen_kappa = []\n",
    "    for i in range(len(results_1)):\n",
    "        aux = results_1[i]\n",
    "        layer_config.append(aux['layer config'])\n",
    "        Precision.append(aux['Metrics'][0])\n",
    "        Recall.append(aux['Metrics'][1])\n",
    "        F1.append(aux['Metrics'][2])\n",
    "        Cohen_kappa.append(aux['Metrics'][3])\n",
    "        \n",
    "    data = {'Hidden layers' : layer_config,\n",
    "            'Precision' : Precision,\n",
    "           'Recall' : Recall,\n",
    "            'F1' : F1,\n",
    "           'Cohen kappa' : Cohen_kappa}\n",
    "    data = pd.DataFrame(data)\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2efbdb45",
   "metadata": {},
   "source": [
    "### Resultados con datos raw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "4ed209f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>[250, 250, 250, 250, 250]</td>\n",
       "      <td>0.825196</td>\n",
       "      <td>0.825196</td>\n",
       "      <td>0.825196</td>\n",
       "      <td>0.804531</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>[250, 250, 250, 250, 250, 250]</td>\n",
       "      <td>0.821810</td>\n",
       "      <td>0.821810</td>\n",
       "      <td>0.821810</td>\n",
       "      <td>0.801151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>[150, 150, 150, 150, 150]</td>\n",
       "      <td>0.820919</td>\n",
       "      <td>0.820919</td>\n",
       "      <td>0.820919</td>\n",
       "      <td>0.799992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>[200, 200, 200, 200, 200, 200]</td>\n",
       "      <td>0.819138</td>\n",
       "      <td>0.819138</td>\n",
       "      <td>0.819138</td>\n",
       "      <td>0.797986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>[150, 150, 150, 150, 150, 150]</td>\n",
       "      <td>0.818425</td>\n",
       "      <td>0.818425</td>\n",
       "      <td>0.818425</td>\n",
       "      <td>0.797329</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>[250, 250, 250, 250]</td>\n",
       "      <td>0.816108</td>\n",
       "      <td>0.816108</td>\n",
       "      <td>0.816108</td>\n",
       "      <td>0.794788</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Hidden layers  Precision    Recall        F1  Cohen kappa\n",
       "28       [250, 250, 250, 250, 250]   0.825196  0.825196  0.825196     0.804531\n",
       "29  [250, 250, 250, 250, 250, 250]   0.821810  0.821810  0.821810     0.801151\n",
       "16       [150, 150, 150, 150, 150]   0.820919  0.820919  0.820919     0.799992\n",
       "23  [200, 200, 200, 200, 200, 200]   0.819138  0.819138  0.819138     0.797986\n",
       "17  [150, 150, 150, 150, 150, 150]   0.818425  0.818425  0.818425     0.797329\n",
       "27            [250, 250, 250, 250]   0.816108  0.816108  0.816108     0.794788"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data = ReadAndCreate(\"results_1_joblib\")\n",
    "results_data.sort_values(\"Precision\", ascending=False).head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f0c69fed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[50]</td>\n",
       "      <td>0.556308</td>\n",
       "      <td>0.556308</td>\n",
       "      <td>0.556308</td>\n",
       "      <td>0.501478</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[100]</td>\n",
       "      <td>0.584105</td>\n",
       "      <td>0.584105</td>\n",
       "      <td>0.584105</td>\n",
       "      <td>0.532926</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>[150]</td>\n",
       "      <td>0.605132</td>\n",
       "      <td>0.605132</td>\n",
       "      <td>0.605132</td>\n",
       "      <td>0.557198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>[200]</td>\n",
       "      <td>0.628653</td>\n",
       "      <td>0.628653</td>\n",
       "      <td>0.628653</td>\n",
       "      <td>0.584144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>[250]</td>\n",
       "      <td>0.641661</td>\n",
       "      <td>0.641661</td>\n",
       "      <td>0.641661</td>\n",
       "      <td>0.599029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[50, 50]</td>\n",
       "      <td>0.677655</td>\n",
       "      <td>0.677655</td>\n",
       "      <td>0.677655</td>\n",
       "      <td>0.639487</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Hidden layers  Precision    Recall        F1  Cohen kappa\n",
       "0           [50]   0.556308  0.556308  0.556308     0.501478\n",
       "6          [100]   0.584105  0.584105  0.584105     0.532926\n",
       "12         [150]   0.605132  0.605132  0.605132     0.557198\n",
       "18         [200]   0.628653  0.628653  0.628653     0.584144\n",
       "24         [250]   0.641661  0.641661  0.641661     0.599029\n",
       "1       [50, 50]   0.677655  0.677655  0.677655     0.639487"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data.sort_values(\"Precision\", ascending=True).head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7263e691",
   "metadata": {},
   "source": [
    "Estudiando los datos con mayor puntuación en Precision descubrimos que los modelos con mayor número de neuronas y capas obtienen buenas puntuaciones, sin llegar al 83%. Inicialmente podríamos esperar que mayor número de neuronas está directamente relacionado con puntuación en metricas, pero vemos que no es así. Configuraciones de capas distintas con números de neuronas por encima de 150 dan lugar a resultados similares. Probablemente, la diferencia en puntuación se deba a la naturaleza aleatoria de las redes neuronales a la hora de inicializar los pesos así como en el algoritmo de minimización. Todos los modelos con características parecidas dan lugar a resultados muy similares.\n",
    "\n",
    "Por otro lado, en los peores resultados obtenemos lo esperado. Los modelos de una capa con pocas neuronas clasifican muy mal y mejoran su puntuación en orden creciente de neuronas."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53b09f2c",
   "metadata": {},
   "source": [
    "### Resultados datos raw one-hot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "53b5cda7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>[200, 200, 200, 200]</td>\n",
       "      <td>0.830898</td>\n",
       "      <td>0.830898</td>\n",
       "      <td>0.830898</td>\n",
       "      <td>0.811140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>[250, 250, 250]</td>\n",
       "      <td>0.830185</td>\n",
       "      <td>0.830185</td>\n",
       "      <td>0.830185</td>\n",
       "      <td>0.810440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>[250, 250]</td>\n",
       "      <td>0.824127</td>\n",
       "      <td>0.824127</td>\n",
       "      <td>0.824127</td>\n",
       "      <td>0.803536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>[250, 250, 250, 250]</td>\n",
       "      <td>0.817177</td>\n",
       "      <td>0.817177</td>\n",
       "      <td>0.817177</td>\n",
       "      <td>0.795549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>[200, 200, 200, 200, 200]</td>\n",
       "      <td>0.814505</td>\n",
       "      <td>0.814505</td>\n",
       "      <td>0.814505</td>\n",
       "      <td>0.793240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>[200, 200, 200]</td>\n",
       "      <td>0.809694</td>\n",
       "      <td>0.809694</td>\n",
       "      <td>0.809694</td>\n",
       "      <td>0.787236</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Hidden layers  Precision    Recall        F1  Cohen kappa\n",
       "21       [200, 200, 200, 200]   0.830898  0.830898  0.830898     0.811140\n",
       "26            [250, 250, 250]   0.830185  0.830185  0.830185     0.810440\n",
       "25                 [250, 250]   0.824127  0.824127  0.824127     0.803536\n",
       "27       [250, 250, 250, 250]   0.817177  0.817177  0.817177     0.795549\n",
       "22  [200, 200, 200, 200, 200]   0.814505  0.814505  0.814505     0.793240\n",
       "20            [200, 200, 200]   0.809694  0.809694  0.809694     0.787236"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data = ReadAndCreate(\"results_1_onehot_joblib\")\n",
    "results_data.sort_values(\"Precision\", ascending=False).head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "719e25f1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[50]</td>\n",
       "      <td>0.713293</td>\n",
       "      <td>0.713293</td>\n",
       "      <td>0.713293</td>\n",
       "      <td>0.679594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[50, 50, 50, 50, 50, 50]</td>\n",
       "      <td>0.724519</td>\n",
       "      <td>0.724519</td>\n",
       "      <td>0.724519</td>\n",
       "      <td>0.692311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[50, 50, 50, 50]</td>\n",
       "      <td>0.729330</td>\n",
       "      <td>0.729330</td>\n",
       "      <td>0.729330</td>\n",
       "      <td>0.697647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[50, 50, 50, 50, 50]</td>\n",
       "      <td>0.732359</td>\n",
       "      <td>0.732359</td>\n",
       "      <td>0.732359</td>\n",
       "      <td>0.701160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[50, 50]</td>\n",
       "      <td>0.735923</td>\n",
       "      <td>0.735923</td>\n",
       "      <td>0.735923</td>\n",
       "      <td>0.705121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[100]</td>\n",
       "      <td>0.740556</td>\n",
       "      <td>0.740556</td>\n",
       "      <td>0.740556</td>\n",
       "      <td>0.710106</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Hidden layers  Precision    Recall        F1  Cohen kappa\n",
       "0                      [50]   0.713293  0.713293  0.713293     0.679594\n",
       "5  [50, 50, 50, 50, 50, 50]   0.724519  0.724519  0.724519     0.692311\n",
       "3          [50, 50, 50, 50]   0.729330  0.729330  0.729330     0.697647\n",
       "4      [50, 50, 50, 50, 50]   0.732359  0.732359  0.732359     0.701160\n",
       "1                  [50, 50]   0.735923  0.735923  0.735923     0.705121\n",
       "6                     [100]   0.740556  0.740556  0.740556     0.710106"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data.sort_values(\"Precision\", ascending=True).head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bb81f25",
   "metadata": {},
   "source": [
    "Estudiamos los modelos con mayor puntuación. El uso de usar la codificación _one-hot_ de las variables dan lugar a prácticamente los mismos resultados que sin usar esta transformación. Además, los modelos que consiguen las mayores puntuaciones también son muy parecidos.\n",
    "\n",
    "Sin embargo, en los modelos que obtienen los peores resultados, éstos son muchos mejores que los obtenidos por los peores modelos sin usar _one-hot_. Las peores métricas obtenidas está acotadas inferiormente, en el caso de una sola capa oculta de 50 neuronas, en el 71%. Esto nos puede indicar que, mientras que usar _one-hot_ es beneficioso en este tipo de modelos, sea difícil mejorar los resultados obtenidos en los mejores casos."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9c32869",
   "metadata": {},
   "source": [
    "### Resultados datos raw smote"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "0318b87f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>[250, 250, 250, 250, 250, 250]</td>\n",
       "      <td>0.361725</td>\n",
       "      <td>0.361725</td>\n",
       "      <td>0.361725</td>\n",
       "      <td>0.300894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>[200, 200, 200, 200, 200, 200]</td>\n",
       "      <td>0.356557</td>\n",
       "      <td>0.356557</td>\n",
       "      <td>0.356557</td>\n",
       "      <td>0.296138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>[150, 150, 150, 150, 150]</td>\n",
       "      <td>0.355132</td>\n",
       "      <td>0.355132</td>\n",
       "      <td>0.355132</td>\n",
       "      <td>0.295350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>[250, 250, 250, 250]</td>\n",
       "      <td>0.354775</td>\n",
       "      <td>0.354775</td>\n",
       "      <td>0.354775</td>\n",
       "      <td>0.294756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>[150, 150, 150, 150, 150, 150]</td>\n",
       "      <td>0.354419</td>\n",
       "      <td>0.354419</td>\n",
       "      <td>0.354419</td>\n",
       "      <td>0.293245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>[250, 250, 250, 250, 250]</td>\n",
       "      <td>0.349786</td>\n",
       "      <td>0.349786</td>\n",
       "      <td>0.349786</td>\n",
       "      <td>0.288579</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Hidden layers  Precision    Recall        F1  Cohen kappa\n",
       "29  [250, 250, 250, 250, 250, 250]   0.361725  0.361725  0.361725     0.300894\n",
       "23  [200, 200, 200, 200, 200, 200]   0.356557  0.356557  0.356557     0.296138\n",
       "16       [150, 150, 150, 150, 150]   0.355132  0.355132  0.355132     0.295350\n",
       "27            [250, 250, 250, 250]   0.354775  0.354775  0.354775     0.294756\n",
       "17  [150, 150, 150, 150, 150, 150]   0.354419  0.354419  0.354419     0.293245\n",
       "28       [250, 250, 250, 250, 250]   0.349786  0.349786  0.349786     0.288579"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data = ReadAndCreate(\"results_smote_joblib\")\n",
    "results_data.sort_values(\"Precision\", ascending=False).head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "3c596ea9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[50]</td>\n",
       "      <td>0.273521</td>\n",
       "      <td>0.273521</td>\n",
       "      <td>0.273521</td>\n",
       "      <td>0.210563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[100]</td>\n",
       "      <td>0.299715</td>\n",
       "      <td>0.299715</td>\n",
       "      <td>0.299715</td>\n",
       "      <td>0.237226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>[150]</td>\n",
       "      <td>0.300249</td>\n",
       "      <td>0.300249</td>\n",
       "      <td>0.300249</td>\n",
       "      <td>0.238289</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[50, 50]</td>\n",
       "      <td>0.301675</td>\n",
       "      <td>0.301675</td>\n",
       "      <td>0.301675</td>\n",
       "      <td>0.238587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>[200]</td>\n",
       "      <td>0.312723</td>\n",
       "      <td>0.312723</td>\n",
       "      <td>0.312723</td>\n",
       "      <td>0.250874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[50, 50, 50]</td>\n",
       "      <td>0.319672</td>\n",
       "      <td>0.319672</td>\n",
       "      <td>0.319672</td>\n",
       "      <td>0.257319</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Hidden layers  Precision    Recall        F1  Cohen kappa\n",
       "0           [50]   0.273521  0.273521  0.273521     0.210563\n",
       "6          [100]   0.299715  0.299715  0.299715     0.237226\n",
       "12         [150]   0.300249  0.300249  0.300249     0.238289\n",
       "1       [50, 50]   0.301675  0.301675  0.301675     0.238587\n",
       "18         [200]   0.312723  0.312723  0.312723     0.250874\n",
       "2   [50, 50, 50]   0.319672  0.319672  0.319672     0.257319"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data.sort_values(\"Precision\", ascending=True).head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9afa97c",
   "metadata": {},
   "source": [
    "En un intento de mejorar el desbalanceo existente en las clases a clasificar, utilizamos el algoritmo SMOTE para mejorar el dataset. El motivo por el que usamos un algoritmo de sobremuestreo en vez de _tirar a la baja_ con undersampling es por el beneficio que tienen las redes neuronales al aumentar el tamaño del dataset. Además, en pruebas realizadas con árboles de decisión en otras asignaturas, descubrimos que, a pesar de romper el problema de desbalanceo, obtenemos peores resultados al disponer de menos casos.\n",
    "\n",
    "No ha mejorado la clasificación. Los resultados son muy malos. Las mejores redes obtienen métricas alrededor del 35%, mientras que las mejoras obtienen un 27%. Al haber tan poca diferencia entre los mejores y peores resultados, podemos estar seguros que no se debe a un número insuficiente de neuronas y capas.\n",
    "\n",
    "En los problemas en los que las clases están muy desbalanceadas, SMOTE no funciona bien. Esta es una de esas ocasiones. Además, este algoritmo funciona mejor en problemas de clasificación binarios, donde fue concebido.\n",
    "\n",
    "Como hemos mencionado antes, SMOTE está diseñado para problemas de predictores continuos, donde es más natural la interpolación. Sin embargo, como vimos que no daba ningún error al ejecutar la librería, vimos interesante ver el desempeño de la misma en problemas categóricos."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "995f8fd9",
   "metadata": {},
   "source": [
    "### Resultados datos smote one-hot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "6ee20adf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>[200, 200]</td>\n",
       "      <td>0.187634</td>\n",
       "      <td>0.187634</td>\n",
       "      <td>0.187634</td>\n",
       "      <td>0.144296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>[250, 250, 250]</td>\n",
       "      <td>0.177833</td>\n",
       "      <td>0.177833</td>\n",
       "      <td>0.177833</td>\n",
       "      <td>0.131770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>[200, 200, 200]</td>\n",
       "      <td>0.152174</td>\n",
       "      <td>0.152174</td>\n",
       "      <td>0.152174</td>\n",
       "      <td>0.107797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>[250, 250]</td>\n",
       "      <td>0.145937</td>\n",
       "      <td>0.145937</td>\n",
       "      <td>0.145937</td>\n",
       "      <td>0.103981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>[150, 150, 150]</td>\n",
       "      <td>0.138097</td>\n",
       "      <td>0.138097</td>\n",
       "      <td>0.138097</td>\n",
       "      <td>0.091480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>[150, 150]</td>\n",
       "      <td>0.129187</td>\n",
       "      <td>0.129187</td>\n",
       "      <td>0.129187</td>\n",
       "      <td>0.091482</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Hidden layers  Precision    Recall        F1  Cohen kappa\n",
       "19       [200, 200]   0.187634  0.187634  0.187634     0.144296\n",
       "26  [250, 250, 250]   0.177833  0.177833  0.177833     0.131770\n",
       "20  [200, 200, 200]   0.152174  0.152174  0.152174     0.107797\n",
       "25       [250, 250]   0.145937  0.145937  0.145937     0.103981\n",
       "14  [150, 150, 150]   0.138097  0.138097  0.138097     0.091480\n",
       "13       [150, 150]   0.129187  0.129187  0.129187     0.091482"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data = ReadAndCreate(\"results_smote_onehot_joblib\")\n",
    "results_data.sort_values(\"Precision\", ascending=False).head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "7396a931",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[50, 50, 50, 50, 50, 50]</td>\n",
       "      <td>0.054348</td>\n",
       "      <td>0.054348</td>\n",
       "      <td>0.054348</td>\n",
       "      <td>0.018471</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>[150, 150, 150, 150, 150, 150]</td>\n",
       "      <td>0.062545</td>\n",
       "      <td>0.062545</td>\n",
       "      <td>0.062545</td>\n",
       "      <td>0.026222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>[150, 150, 150, 150, 150]</td>\n",
       "      <td>0.066999</td>\n",
       "      <td>0.066999</td>\n",
       "      <td>0.066999</td>\n",
       "      <td>0.031643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[50, 50, 50, 50, 50]</td>\n",
       "      <td>0.067890</td>\n",
       "      <td>0.067890</td>\n",
       "      <td>0.067890</td>\n",
       "      <td>0.028956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[50, 50]</td>\n",
       "      <td>0.069316</td>\n",
       "      <td>0.069316</td>\n",
       "      <td>0.069316</td>\n",
       "      <td>0.037007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[50, 50, 50]</td>\n",
       "      <td>0.069672</td>\n",
       "      <td>0.069672</td>\n",
       "      <td>0.069672</td>\n",
       "      <td>0.034412</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Hidden layers  Precision    Recall        F1  Cohen kappa\n",
       "5         [50, 50, 50, 50, 50, 50]   0.054348  0.054348  0.054348     0.018471\n",
       "17  [150, 150, 150, 150, 150, 150]   0.062545  0.062545  0.062545     0.026222\n",
       "16       [150, 150, 150, 150, 150]   0.066999  0.066999  0.066999     0.031643\n",
       "4             [50, 50, 50, 50, 50]   0.067890  0.067890  0.067890     0.028956\n",
       "1                         [50, 50]   0.069316  0.069316  0.069316     0.037007\n",
       "2                     [50, 50, 50]   0.069672  0.069672  0.069672     0.034412"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data.sort_values(\"Precision\", ascending=True).head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26d3d378",
   "metadata": {},
   "source": [
    "La codificación one-hot ha empeorado los resultados con SMOTE. Las métricas están acotadas entre un 18% y un 5%."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "159fc4b6",
   "metadata": {},
   "source": [
    "### Resultados datos raw con dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "c429daf2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>[250, 0.1, 250, 0.1, 250, 0.1, 250, 0.1, 250, ...</td>\n",
       "      <td>0.874020</td>\n",
       "      <td>0.874020</td>\n",
       "      <td>0.874020</td>\n",
       "      <td>0.859249</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>[200, 0.1, 200, 0.1, 200, 0.1, 200, 0.1, 200, ...</td>\n",
       "      <td>0.873664</td>\n",
       "      <td>0.873664</td>\n",
       "      <td>0.873664</td>\n",
       "      <td>0.858904</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>[250, 0.1, 250, 0.1, 250, 0.1, 250, 0.1, 250, ...</td>\n",
       "      <td>0.871347</td>\n",
       "      <td>0.871347</td>\n",
       "      <td>0.871347</td>\n",
       "      <td>0.856290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>[200, 0.1, 200, 0.1, 200, 0.1]</td>\n",
       "      <td>0.870991</td>\n",
       "      <td>0.870991</td>\n",
       "      <td>0.870991</td>\n",
       "      <td>0.855863</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>[200, 0.1, 200, 0.1, 200, 0.1, 200, 0.1]</td>\n",
       "      <td>0.866180</td>\n",
       "      <td>0.866180</td>\n",
       "      <td>0.866180</td>\n",
       "      <td>0.850478</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>[200, 0.1, 200, 0.1, 200, 0.1, 200, 0.1, 200, ...</td>\n",
       "      <td>0.864754</td>\n",
       "      <td>0.864754</td>\n",
       "      <td>0.864754</td>\n",
       "      <td>0.848903</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Hidden layers  Precision    Recall  \\\n",
       "94  [250, 0.1, 250, 0.1, 250, 0.1, 250, 0.1, 250, ...   0.874020  0.874020   \n",
       "76  [200, 0.1, 200, 0.1, 200, 0.1, 200, 0.1, 200, ...   0.873664  0.873664   \n",
       "95  [250, 0.1, 250, 0.1, 250, 0.1, 250, 0.1, 250, ...   0.871347  0.871347   \n",
       "74                     [200, 0.1, 200, 0.1, 200, 0.1]   0.870991  0.870991   \n",
       "75           [200, 0.1, 200, 0.1, 200, 0.1, 200, 0.1]   0.866180  0.866180   \n",
       "77  [200, 0.1, 200, 0.1, 200, 0.1, 200, 0.1, 200, ...   0.864754  0.864754   \n",
       "\n",
       "          F1  Cohen kappa  \n",
       "94  0.874020     0.859249  \n",
       "76  0.873664     0.858904  \n",
       "95  0.871347     0.856290  \n",
       "74  0.870991     0.855863  \n",
       "75  0.866180     0.850478  \n",
       "77  0.864754     0.848903  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data = ReadAndCreate(\"results_dropout\")\n",
    "results_data.sort_values(\"Precision\", ascending=False).head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "aca02e84",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>[50, 0.3]</td>\n",
       "      <td>0.511048</td>\n",
       "      <td>0.511048</td>\n",
       "      <td>0.511048</td>\n",
       "      <td>0.449370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>[50, 0.2]</td>\n",
       "      <td>0.517284</td>\n",
       "      <td>0.517284</td>\n",
       "      <td>0.517284</td>\n",
       "      <td>0.456681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>[50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, ...</td>\n",
       "      <td>0.522630</td>\n",
       "      <td>0.522630</td>\n",
       "      <td>0.522630</td>\n",
       "      <td>0.463069</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>[50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]</td>\n",
       "      <td>0.526372</td>\n",
       "      <td>0.526372</td>\n",
       "      <td>0.526372</td>\n",
       "      <td>0.466761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[50, 0.1]</td>\n",
       "      <td>0.527263</td>\n",
       "      <td>0.527263</td>\n",
       "      <td>0.527263</td>\n",
       "      <td>0.469118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>[50, 0.1]</td>\n",
       "      <td>0.530649</td>\n",
       "      <td>0.530649</td>\n",
       "      <td>0.530649</td>\n",
       "      <td>0.472995</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Hidden layers  Precision    Recall  \\\n",
       "30                                          [50, 0.3]   0.511048  0.511048   \n",
       "24                                          [50, 0.2]   0.517284  0.517284   \n",
       "35  [50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, ...   0.522630  0.522630   \n",
       "34      [50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]   0.526372  0.526372   \n",
       "0                                           [50, 0.1]   0.527263  0.527263   \n",
       "18                                          [50, 0.1]   0.530649  0.530649   \n",
       "\n",
       "          F1  Cohen kappa  \n",
       "30  0.511048     0.449370  \n",
       "24  0.517284     0.456681  \n",
       "35  0.522630     0.463069  \n",
       "34  0.526372     0.466761  \n",
       "0   0.527263     0.469118  \n",
       "18  0.530649     0.472995  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data.sort_values(\"Precision\", ascending=True).head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3aebb60f",
   "metadata": {},
   "source": [
    "Los mejores resultados mejoran levemente los obtenidos sin dropout. Al igual que en estos últimos, las modelos más grandes obtienen mejores resultados. Notemos que todos los que aparecen con mejores métricas tienen un porcentaje de desactivación muy pequeño.\n",
    "\n",
    "En el polo opuesto, los modelos que obtuvieron peor resultado fueron los modelos con el mayor porcentaje de desactivación en dropout: 30% y menor número de neuronas. Esto es lógico, ya que no parece ser un dataset lo suficientemente grande como para que se produzca un sobreaprendizaje durante el entrenamiento.\n",
    "\n",
    "Recordamos que el número de épocas en el entrenamiento viene controlado por tensorflow mediante los _Callbacks_, como comentamos anteriormente, por lo que si no mejora la _accuracy_ del conjunto de validación en 10 épocas, finaliza el entrenamiento. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3098953",
   "metadata": {},
   "source": [
    "### Resultados datos raw one-hot con dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "89662e43",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>[250, 0.1, 250, 0.1, 250, 0.1, 250, 0.1, 250, ...</td>\n",
       "      <td>0.894512</td>\n",
       "      <td>0.894512</td>\n",
       "      <td>0.894512</td>\n",
       "      <td>0.882191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>[200, 0.2, 200, 0.2, 200, 0.2]</td>\n",
       "      <td>0.892908</td>\n",
       "      <td>0.892908</td>\n",
       "      <td>0.892908</td>\n",
       "      <td>0.880426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>[250, 0.2, 250, 0.2]</td>\n",
       "      <td>0.891839</td>\n",
       "      <td>0.891839</td>\n",
       "      <td>0.891839</td>\n",
       "      <td>0.879262</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>[200, 0.2, 200, 0.2, 200, 0.2, 200, 0.2]</td>\n",
       "      <td>0.888988</td>\n",
       "      <td>0.888988</td>\n",
       "      <td>0.888988</td>\n",
       "      <td>0.876026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>[200, 0.1, 200, 0.1, 200, 0.1]</td>\n",
       "      <td>0.888453</td>\n",
       "      <td>0.888453</td>\n",
       "      <td>0.888453</td>\n",
       "      <td>0.875431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>[250, 0.2, 250, 0.2, 250, 0.2]</td>\n",
       "      <td>0.887741</td>\n",
       "      <td>0.887741</td>\n",
       "      <td>0.887741</td>\n",
       "      <td>0.874633</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Hidden layers  Precision    Recall  \\\n",
       "77  [250, 0.1, 250, 0.1, 250, 0.1, 250, 0.1, 250, ...   0.894512  0.894512   \n",
       "62                     [200, 0.2, 200, 0.2, 200, 0.2]   0.892908  0.892908   \n",
       "79                               [250, 0.2, 250, 0.2]   0.891839  0.891839   \n",
       "63           [200, 0.2, 200, 0.2, 200, 0.2, 200, 0.2]   0.888988  0.888988   \n",
       "56                     [200, 0.1, 200, 0.1, 200, 0.1]   0.888453  0.888453   \n",
       "80                     [250, 0.2, 250, 0.2, 250, 0.2]   0.887741  0.887741   \n",
       "\n",
       "          F1  Cohen kappa  \n",
       "77  0.894512     0.882191  \n",
       "62  0.892908     0.880426  \n",
       "79  0.891839     0.879262  \n",
       "63  0.888988     0.876026  \n",
       "56  0.888453     0.875431  \n",
       "80  0.887741     0.874633  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data = ReadAndCreate(\"results_dropout_one_hot\")\n",
    "results_data.sort_values(\"Precision\", ascending=False).head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "ce879045",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>[50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, ...</td>\n",
       "      <td>0.607627</td>\n",
       "      <td>0.607627</td>\n",
       "      <td>0.607627</td>\n",
       "      <td>0.561731</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>[50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]</td>\n",
       "      <td>0.640770</td>\n",
       "      <td>0.640770</td>\n",
       "      <td>0.640770</td>\n",
       "      <td>0.597454</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>[50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]</td>\n",
       "      <td>0.655560</td>\n",
       "      <td>0.655560</td>\n",
       "      <td>0.655560</td>\n",
       "      <td>0.614280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>[50, 0.3]</td>\n",
       "      <td>0.658411</td>\n",
       "      <td>0.658411</td>\n",
       "      <td>0.658411</td>\n",
       "      <td>0.617240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>[50, 0.3, 50, 0.3, 50, 0.3]</td>\n",
       "      <td>0.669280</td>\n",
       "      <td>0.669280</td>\n",
       "      <td>0.669280</td>\n",
       "      <td>0.629326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[50, 0.2]</td>\n",
       "      <td>0.679793</td>\n",
       "      <td>0.679793</td>\n",
       "      <td>0.679793</td>\n",
       "      <td>0.641515</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Hidden layers  Precision    Recall  \\\n",
       "17  [50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, ...   0.607627  0.607627   \n",
       "16      [50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]   0.640770  0.640770   \n",
       "15               [50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]   0.655560  0.655560   \n",
       "12                                          [50, 0.3]   0.658411  0.658411   \n",
       "14                        [50, 0.3, 50, 0.3, 50, 0.3]   0.669280  0.669280   \n",
       "6                                           [50, 0.2]   0.679793  0.679793   \n",
       "\n",
       "          F1  Cohen kappa  \n",
       "17  0.607627     0.561731  \n",
       "16  0.640770     0.597454  \n",
       "15  0.655560     0.614280  \n",
       "12  0.658411     0.617240  \n",
       "14  0.669280     0.629326  \n",
       "6   0.679793     0.641515  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data.sort_values(\"Precision\", ascending=True).head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca7c0a6b",
   "metadata": {},
   "source": [
    "Los resultados son esperables vistos los casos anteriores. Dropout mejora también el caso en el que usamos _one-hot_ encoding. Estos son los mejores resultados que obtenemos."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42bfd91d",
   "metadata": {},
   "source": [
    "### Resultados datos raw smote con dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "8a21bb44",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>[200, 0.2, 200, 0.2, 200, 0.2]</td>\n",
       "      <td>0.403243</td>\n",
       "      <td>0.403243</td>\n",
       "      <td>0.403243</td>\n",
       "      <td>0.346788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>[200, 0.1, 200, 0.1, 200, 0.1, 200, 0.1, 200, ...</td>\n",
       "      <td>0.400214</td>\n",
       "      <td>0.400214</td>\n",
       "      <td>0.400214</td>\n",
       "      <td>0.342223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>[250, 0.1, 250, 0.1, 250, 0.1, 250, 0.1, 250, ...</td>\n",
       "      <td>0.400036</td>\n",
       "      <td>0.400036</td>\n",
       "      <td>0.400036</td>\n",
       "      <td>0.341473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>[250, 0.2, 250, 0.2, 250, 0.2, 250, 0.2, 250, ...</td>\n",
       "      <td>0.399501</td>\n",
       "      <td>0.399501</td>\n",
       "      <td>0.399501</td>\n",
       "      <td>0.341194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>[250, 0.2, 250, 0.2, 250, 0.2]</td>\n",
       "      <td>0.399501</td>\n",
       "      <td>0.399501</td>\n",
       "      <td>0.399501</td>\n",
       "      <td>0.342195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>[250, 0.1, 250, 0.1, 250, 0.1]</td>\n",
       "      <td>0.398610</td>\n",
       "      <td>0.398610</td>\n",
       "      <td>0.398610</td>\n",
       "      <td>0.341175</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Hidden layers  Precision    Recall  \\\n",
       "62                     [200, 0.2, 200, 0.2, 200, 0.2]   0.403243  0.403243   \n",
       "58  [200, 0.1, 200, 0.1, 200, 0.1, 200, 0.1, 200, ...   0.400214  0.400214   \n",
       "77  [250, 0.1, 250, 0.1, 250, 0.1, 250, 0.1, 250, ...   0.400036  0.400036   \n",
       "83  [250, 0.2, 250, 0.2, 250, 0.2, 250, 0.2, 250, ...   0.399501  0.399501   \n",
       "80                     [250, 0.2, 250, 0.2, 250, 0.2]   0.399501  0.399501   \n",
       "74                     [250, 0.1, 250, 0.1, 250, 0.1]   0.398610  0.398610   \n",
       "\n",
       "          F1  Cohen kappa  \n",
       "62  0.403243     0.346788  \n",
       "58  0.400214     0.342223  \n",
       "77  0.400036     0.341473  \n",
       "83  0.399501     0.341194  \n",
       "80  0.399501     0.342195  \n",
       "74  0.398610     0.341175  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data = ReadAndCreate(\"results_dropout_smote\")\n",
    "results_data.sort_values(\"Precision\", ascending=False).head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "fb2d636d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>[50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, ...</td>\n",
       "      <td>0.215966</td>\n",
       "      <td>0.215966</td>\n",
       "      <td>0.215966</td>\n",
       "      <td>0.151163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>[50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]</td>\n",
       "      <td>0.227548</td>\n",
       "      <td>0.227548</td>\n",
       "      <td>0.227548</td>\n",
       "      <td>0.161465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>[50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]</td>\n",
       "      <td>0.243763</td>\n",
       "      <td>0.243763</td>\n",
       "      <td>0.243763</td>\n",
       "      <td>0.179339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>[50, 0.3]</td>\n",
       "      <td>0.245902</td>\n",
       "      <td>0.245902</td>\n",
       "      <td>0.245902</td>\n",
       "      <td>0.185292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[50, 0.2]</td>\n",
       "      <td>0.249465</td>\n",
       "      <td>0.249465</td>\n",
       "      <td>0.249465</td>\n",
       "      <td>0.187577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>[150, 0.3]</td>\n",
       "      <td>0.254098</td>\n",
       "      <td>0.254098</td>\n",
       "      <td>0.254098</td>\n",
       "      <td>0.189198</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Hidden layers  Precision    Recall  \\\n",
       "17  [50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, ...   0.215966  0.215966   \n",
       "16      [50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]   0.227548  0.227548   \n",
       "15               [50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]   0.243763  0.243763   \n",
       "12                                          [50, 0.3]   0.245902  0.245902   \n",
       "6                                           [50, 0.2]   0.249465  0.249465   \n",
       "48                                         [150, 0.3]   0.254098  0.254098   \n",
       "\n",
       "          F1  Cohen kappa  \n",
       "17  0.215966     0.151163  \n",
       "16  0.227548     0.161465  \n",
       "15  0.243763     0.179339  \n",
       "12  0.245902     0.185292  \n",
       "6   0.249465     0.187577  \n",
       "48  0.254098     0.189198  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data.sort_values(\"Precision\", ascending=True).head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7b589dd",
   "metadata": {},
   "source": [
    "Se obtienen mejores resultados que sin incluir dropout, pero siguen siendo malos."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "061be1e2",
   "metadata": {},
   "source": [
    "### Resultados datos smote one-hot con dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "4e13762d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>[200, 0.1, 200, 0.1]</td>\n",
       "      <td>0.370813</td>\n",
       "      <td>0.370813</td>\n",
       "      <td>0.370813</td>\n",
       "      <td>0.321053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>[250, 0.2, 250, 0.2]</td>\n",
       "      <td>0.367071</td>\n",
       "      <td>0.367071</td>\n",
       "      <td>0.367071</td>\n",
       "      <td>0.316297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>[250, 0.1, 250, 0.1]</td>\n",
       "      <td>0.355845</td>\n",
       "      <td>0.355845</td>\n",
       "      <td>0.355845</td>\n",
       "      <td>0.306629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>[150, 0.1, 150, 0.1]</td>\n",
       "      <td>0.314505</td>\n",
       "      <td>0.314505</td>\n",
       "      <td>0.314505</td>\n",
       "      <td>0.261605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>[250, 0.3]</td>\n",
       "      <td>0.313257</td>\n",
       "      <td>0.313257</td>\n",
       "      <td>0.313257</td>\n",
       "      <td>0.252906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>[250, 0.2]</td>\n",
       "      <td>0.296686</td>\n",
       "      <td>0.296686</td>\n",
       "      <td>0.296686</td>\n",
       "      <td>0.238117</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Hidden layers  Precision    Recall        F1  Cohen kappa\n",
       "55  [200, 0.1, 200, 0.1]   0.370813  0.370813  0.370813     0.321053\n",
       "79  [250, 0.2, 250, 0.2]   0.367071  0.367071  0.367071     0.316297\n",
       "73  [250, 0.1, 250, 0.1]   0.355845  0.355845  0.355845     0.306629\n",
       "37  [150, 0.1, 150, 0.1]   0.314505  0.314505  0.314505     0.261605\n",
       "84            [250, 0.3]   0.313257  0.313257  0.313257     0.252906\n",
       "78            [250, 0.2]   0.296686  0.296686  0.296686     0.238117"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data = ReadAndCreate(\"results_dropout_smote_one_hot\")\n",
    "results_data.sort_values(\"Precision\", ascending=False).head(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "3316602b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hidden layers</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>[50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]</td>\n",
       "      <td>0.019601</td>\n",
       "      <td>0.019601</td>\n",
       "      <td>0.019601</td>\n",
       "      <td>0.003531</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>[50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]</td>\n",
       "      <td>0.023521</td>\n",
       "      <td>0.023521</td>\n",
       "      <td>0.023521</td>\n",
       "      <td>-0.000884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>[50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, ...</td>\n",
       "      <td>0.026907</td>\n",
       "      <td>0.026907</td>\n",
       "      <td>0.026907</td>\n",
       "      <td>0.009679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>[50, 0.2, 50, 0.2, 50, 0.2, 50, 0.2]</td>\n",
       "      <td>0.031896</td>\n",
       "      <td>0.031896</td>\n",
       "      <td>0.031896</td>\n",
       "      <td>0.003915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>[50, 0.2, 50, 0.2, 50, 0.2, 50, 0.2, 50, 0.2]</td>\n",
       "      <td>0.033856</td>\n",
       "      <td>0.033856</td>\n",
       "      <td>0.033856</td>\n",
       "      <td>0.009683</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[50, 0.1, 50, 0.1, 50, 0.1, 50, 0.1, 50, 0.1, ...</td>\n",
       "      <td>0.034569</td>\n",
       "      <td>0.034569</td>\n",
       "      <td>0.034569</td>\n",
       "      <td>0.008312</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Hidden layers  Precision    Recall  \\\n",
       "16      [50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]   0.019601  0.019601   \n",
       "15               [50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3]   0.023521  0.023521   \n",
       "17  [50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, 50, 0.3, ...   0.026907  0.026907   \n",
       "9                [50, 0.2, 50, 0.2, 50, 0.2, 50, 0.2]   0.031896  0.031896   \n",
       "10      [50, 0.2, 50, 0.2, 50, 0.2, 50, 0.2, 50, 0.2]   0.033856  0.033856   \n",
       "5   [50, 0.1, 50, 0.1, 50, 0.1, 50, 0.1, 50, 0.1, ...   0.034569  0.034569   \n",
       "\n",
       "          F1  Cohen kappa  \n",
       "16  0.019601     0.003531  \n",
       "15  0.023521    -0.000884  \n",
       "17  0.026907     0.009679  \n",
       "9   0.031896     0.003915  \n",
       "10  0.033856     0.009683  \n",
       "5   0.034569     0.008312  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_data.sort_values(\"Precision\", ascending=True).head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "471ac3a5",
   "metadata": {},
   "source": [
    "Se repite el comportamiento, usar dropout con un porcentaje de desactivación pequeño mejora los resultados, pero éstos eran muy malos per se."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8f29b78",
   "metadata": {},
   "source": [
    "## Conclusiones"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71d11fb6",
   "metadata": {},
   "source": [
    "Hemos construido modelos de redes neuronales del tipo perceptrón multicapa con distintas configuraciones de capas ocultas y neuronas. Los datos utilizados han sido los originales, la codificación de las variables predictoras mediante _one-hot_ y sus equivalentes tras usar _SMOTE_. También se ha implementado _callbacks_ para parar el entrenamiento si el valor de _accuracy_ en el set de validación no mejora tras 10 épocas, evitando el sobreaprendizaje y retirando el número de épocas como parámetro a estudiar y modificar. A todos estos modelos, se hicieron experimentos introduciendo capas intermedias con _dropout_ con tres posibles valores de desactivación de neuronas: 10, 20 y 30%. \n",
    "\n",
    "Los resultados son los siguientes. El uso de _SMOTE_ no aporta ningún beneficio. Las métricas obtenidas son mucho peores que usando datos sin _SMOTE_. La razón puede deberse al gran desbalanceo entre clases que existe en la variable a predecir y la naturaleza categórica de las predictoras.\n",
    "\n",
    "Introducir _dropout_ mejora el resultado en todos los casos, incluso en los peores.\n",
    "\n",
    "Creemos haber llegado a resultados próximos a los máximos posibles usando perceptrones multicapa ya que, en los casos con mejor puntuación, no existe una relación directa entre mayor número de neuronas y rendimiento. Se dan casos en los que configuraciones con menos capas obtienen mejores resultados que equivalentes con más. Sin embargo, está claro que un número grande neuronas es beneficioso para el modelo.\n",
    "\n",
    "Todas las métricas se comportan de forma parecida: si un modelo obtiene mejor resultado que otro, todas sus métricas son mejores que las del otro."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "849b1e33",
   "metadata": {},
   "source": [
    "# Creación de árboles de clasificación con TensorFlowDecisionTrees\n",
    "\n",
    "Con el fin de investigar más y aprendiendo sobre TensorFlow, descubrimos que, recientemente (Mayo de 2021), se ha incluido en la librería una API en la que se  implementan tres algoritmos muy famosos de árboles de decisión: Random Forest, Gradient Boosted Trees y CART.\n",
    "\n",
    "Estos algoritmos, los cuales hemos visto en el máster, se ejecutan en C++ con la librería, también de Google, Yggdrasil Decision Forests. En realidad, al igual que la API en Python de TensorFlow, estamos utilizando un wrapper en el cual creamos los modelos fácilmente en Python y posteriormente serán entrenados en C++. De momento, esta librería no hace uso de la tarjeta gráfica.\n",
    "\n",
    "La API se llama TensorFlow Decision Forest y puede ser instalada con `pip3 install tensorflow_decision_forests --upgrade`. \n",
    "\n",
    "Probaremos los tres algoritmos y modificaremos ciertos parámetros para el entrenamiento."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fc7a8dc",
   "metadata": {},
   "source": [
    "Al ser una nueva librería, nos parece interesante ir comentando cómo construir los modelos.\n",
    "\n",
    "La lectura de datos se realiza de la misma forma que en casos anteriores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b8e2d387",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import tensorflow_decision_forests as tfdf\n",
    "from wurlitzer import sys_pipes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e0d89519",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"krkopt.data\", header=None)\n",
    "data.columns = [\"wkc\", \"wkr\", \"wrc\", \"wrr\", \"bkc\", \"bkr\", \"opt rank\" ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fc61d4bc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wkc</th>\n",
       "      <th>wkr</th>\n",
       "      <th>wrc</th>\n",
       "      <th>wrr</th>\n",
       "      <th>bkc</th>\n",
       "      <th>bkr</th>\n",
       "      <th>opt rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>b</td>\n",
       "      <td>3</td>\n",
       "      <td>c</td>\n",
       "      <td>2</td>\n",
       "      <td>draw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>2</td>\n",
       "      <td>draw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>1</td>\n",
       "      <td>d</td>\n",
       "      <td>1</td>\n",
       "      <td>draw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>1</td>\n",
       "      <td>d</td>\n",
       "      <td>2</td>\n",
       "      <td>draw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>c</td>\n",
       "      <td>2</td>\n",
       "      <td>c</td>\n",
       "      <td>1</td>\n",
       "      <td>draw</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  wkc  wkr wrc  wrr bkc  bkr opt rank\n",
       "0   a    1   b    3   c    2     draw\n",
       "1   a    1   c    1   c    2     draw\n",
       "2   a    1   c    1   d    1     draw\n",
       "3   a    1   c    1   d    2     draw\n",
       "4   a    1   c    2   c    1     draw"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e2276d6",
   "metadata": {},
   "source": [
    "Es necesario destacar cuál va a ser la columna que contenga la variable a predecir para usar _tfdf_. En nuestro caso, _opt rank_.\n",
    "\n",
    "Mostramos también las clases de esta variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e95659ae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label classes: ['draw', 'zero', 'one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine', 'ten', 'eleven', 'twelve', 'thirteen', 'fourteen', 'fifteen', 'sixteen']\n"
     ]
    }
   ],
   "source": [
    "label = \"opt rank\"\n",
    "\n",
    "classes = data[label].unique().tolist()\n",
    "print(f\"Label classes: {classes}\")\n",
    "\n",
    "#convert to integer category\n",
    "\n",
    "data[label] = data[label].map(classes.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f9d066d",
   "metadata": {},
   "source": [
    "Creamos el conjunto de entrenamiento y test con una proporción 80-20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7da4ee12",
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 1\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e6738e74",
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_dataset(dataset, test_ratio=0.2):\n",
    "    test_indices = np.random.rand(len(dataset)) < test_ratio\n",
    "    return dataset[~test_indices], dataset[test_indices]\n",
    "\n",
    "train, test = split_dataset(data)\n",
    "test_aux = test.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d222ab4e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22450 examples in training, 5606 examples for testing.\n"
     ]
    }
   ],
   "source": [
    "print(\"{} examples in training, {} examples for testing.\".format(\n",
    "    len(train), len(test)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b47874dd",
   "metadata": {},
   "source": [
    "Encontramos las primeras peculariedades. Es necesario utilizar la función _pd_dataframe_to_tf_dataset_ para transformar nuestro conjunto de datos a un objeto que pueda ser procesado por la librería _yggrdrasil_. Más adelante comentaremos un _bug_ que hemos encontrado en esta función que nos impide realizar ciertas pruebas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f6b31b99",
   "metadata": {},
   "outputs": [],
   "source": [
    "train = tfdf.keras.pd_dataframe_to_tf_dataset(train, label=label)\n",
    "test = tfdf.keras.pd_dataframe_to_tf_dataset(test, label=label)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2d91b74",
   "metadata": {},
   "source": [
    "## Creación de Random Forest con datos _raw_\n",
    "\n",
    "Creamos un modelo de Random Forest usando la función _RandomForestModel_. Es necesario compilar el modelo con la métrica que nosotros queramos. Inicialmente intentamos usar Precision, para poder comparar con las redes neuronales, pero obtuvimos error ya que no parece completamente compatible esta métrica aún. Por ello, usamos _accuracy_ y más tarde usamos precision.\n",
    "\n",
    "Además, en este primer modelo, mostramos un log usando _sys_pipes_ que permite ver los entresijos y detalles del entrenamiento. Como vemos, la propia librería detecta las categorías de las variables predictoras.\n",
    "\n",
    "En el caso de los árboles de decisión, no se utilizan épocas ya que los distintos árboles que componen el bosque de árboles se entrenan con todo el conjunto de entrenamiento. Además, el propio algoritmo tiene una medida de estimación de validación, por lo que tampoco hay que usar conjunto de validación."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9b84fe84",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-06-10 17:12:50.174954: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)\n",
      "2021-06-10 17:12:50.195009: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199995000 Hz\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "351/351 [==============================] - 3s 592us/step\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO kernel.cc:746] Start Yggdrasil model training\n",
      "[INFO kernel.cc:747] Collect training examples\n",
      "[INFO kernel.cc:392] Number of batches: 351\n",
      "[INFO kernel.cc:393] Number of examples: 22450\n",
      "[INFO kernel.cc:769] Dataset:\n",
      "Number of records: 22450\n",
      "Number of columns: 7\n",
      "\n",
      "Number of columns by type:\n",
      "\tCATEGORICAL: 4 (57.1429%)\n",
      "\tNUMERICAL: 3 (42.8571%)\n",
      "\n",
      "Columns:\n",
      "\n",
      "CATEGORICAL: 4 (57.1429%)\n",
      "\t0: \"bkc\" CATEGORICAL has-dict vocab-size:9 zero-ood-items most-frequent:\"h\" 3859 (17.1893%)\n",
      "\t2: \"wkc\" CATEGORICAL has-dict vocab-size:5 zero-ood-items most-frequent:\"d\" 9677 (43.1047%)\n",
      "\t4: \"wrc\" CATEGORICAL has-dict vocab-size:9 zero-ood-items most-frequent:\"f\" 2930 (13.0512%)\n",
      "\t6: \"__LABEL\" CATEGORICAL integerized vocab-size:19 no-ood-item\n",
      "\n",
      "NUMERICAL: 3 (42.8571%)\n",
      "\t1: \"bkr\" NUMERICAL mean:4.45768 min:1 max:8 sd:2.24698\n",
      "\t3: \"wkr\" NUMERICAL mean:1.84909 min:1 max:4 sd:0.924757\n",
      "\t5: \"wrr\" NUMERICAL mean:4.51042 min:1 max:8 sd:2.27881\n",
      "\n",
      "Terminology:\n",
      "\tnas: Number of non-available (i.e. missing) values.\n",
      "\tood: Out of dictionary.\n",
      "\tmanually-defined: Attribute which type is manually defined by the user i.e. the type was not automatically inferred.\n",
      "\ttokenized: The attribute value is obtained through tokenization.\n",
      "\thas-dict: The attribute is attached to a string dictionary e.g. a categorical attribute stored as a string.\n",
      "\tvocab-size: Number of unique values.\n",
      "\n",
      "[INFO kernel.cc:772] Configure learner\n",
      "[INFO kernel.cc:797] Training config:\n",
      "learner: \"RANDOM_FOREST\"\n",
      "features: \"bkc\"\n",
      "features: \"bkr\"\n",
      "features: \"wkc\"\n",
      "features: \"wkr\"\n",
      "features: \"wrc\"\n",
      "features: \"wrr\"\n",
      "label: \"__LABEL\"\n",
      "task: CLASSIFICATION\n",
      "[yggdrasil_decision_forests.model.random_forest.proto.random_forest_config] {\n",
      "  num_trees: 300\n",
      "  decision_tree {\n",
      "    max_depth: 16\n",
      "    min_examples: 5\n",
      "    in_split_min_examples_check: true\n",
      "    missing_value_policy: GLOBAL_IMPUTATION\n",
      "    allow_na_conditions: false\n",
      "    categorical_set_greedy_forward {\n",
      "      sampling: 0.1\n",
      "      max_num_items: -1\n",
      "      min_item_frequency: 1\n",
      "    }\n",
      "    growing_strategy_local {\n",
      "    }\n",
      "    categorical {\n",
      "      cart {\n",
      "      }\n",
      "    }\n",
      "    num_candidate_attributes_ratio: -1\n",
      "    axis_aligned_split {\n",
      "    }\n",
      "  }\n",
      "  winner_take_all_inference: true\n",
      "  compute_oob_performances: true\n",
      "  compute_oob_variable_importances: false\n",
      "  adapt_bootstrap_size_ratio_for_maximum_training_duration: false\n",
      "}\n",
      "\n",
      "[INFO kernel.cc:800] Deployment config:\n",
      "\n",
      "[INFO kernel.cc:837] Train model\n",
      "[INFO random_forest.cc:303] Training random forest on 22450 example(s) and 6 feature(s).\n",
      "[INFO random_forest.cc:578] Training of tree  1/300 (tree index:1) done accuracy:0.663478 logloss:12.1295\n",
      "[INFO random_forest.cc:578] Training of tree  11/300 (tree index:13) done accuracy:0.691763 logloss:5.05263\n",
      "[INFO random_forest.cc:578] Training of tree  21/300 (tree index:21) done accuracy:0.730233 logloss:2.65216\n",
      "[INFO random_forest.cc:578] Training of tree  31/300 (tree index:30) done accuracy:0.742316 logloss:1.95862\n",
      "[INFO random_forest.cc:578] Training of tree  41/300 (tree index:40) done accuracy:0.750379 logloss:1.58524\n",
      "[INFO random_forest.cc:578] Training of tree  51/300 (tree index:50) done accuracy:0.753497 logloss:1.37915\n",
      "[INFO random_forest.cc:578] Training of tree  61/300 (tree index:60) done accuracy:0.755768 logloss:1.26819\n",
      "[INFO random_forest.cc:578] Training of tree  71/300 (tree index:70) done accuracy:0.758708 logloss:1.17384\n",
      "[INFO random_forest.cc:578] Training of tree  81/300 (tree index:80) done accuracy:0.75902 logloss:1.11116\n",
      "[INFO random_forest.cc:578] Training of tree  91/300 (tree index:90) done accuracy:0.760713 logloss:1.0598\n",
      "[INFO random_forest.cc:578] Training of tree  101/300 (tree index:100) done accuracy:0.761336 logloss:1.02395\n",
      "[INFO random_forest.cc:578] Training of tree  111/300 (tree index:110) done accuracy:0.764276 logloss:0.992012\n",
      "[INFO random_forest.cc:578] Training of tree  121/300 (tree index:121) done accuracy:0.763964 logloss:0.961497\n",
      "[INFO random_forest.cc:578] Training of tree  131/300 (tree index:131) done accuracy:0.763563 logloss:0.944803\n",
      "[INFO random_forest.cc:578] Training of tree  141/300 (tree index:139) done accuracy:0.764009 logloss:0.928943\n",
      "[INFO random_forest.cc:578] Training of tree  151/300 (tree index:150) done accuracy:0.764365 logloss:0.913852\n",
      "[INFO random_forest.cc:578] Training of tree  161/300 (tree index:160) done accuracy:0.764811 logloss:0.89613\n",
      "[INFO random_forest.cc:578] Training of tree  171/300 (tree index:170) done accuracy:0.765479 logloss:0.881226\n",
      "[INFO random_forest.cc:578] Training of tree  181/300 (tree index:180) done accuracy:0.764944 logloss:0.866638\n",
      "[INFO random_forest.cc:578] Training of tree  191/300 (tree index:191) done accuracy:0.765345 logloss:0.863178\n",
      "[INFO random_forest.cc:578] Training of tree  201/300 (tree index:200) done accuracy:0.764855 logloss:0.848468\n",
      "[INFO random_forest.cc:578] Training of tree  211/300 (tree index:212) done accuracy:0.765345 logloss:0.843659\n",
      "[INFO random_forest.cc:578] Training of tree  221/300 (tree index:218) done accuracy:0.766414 logloss:0.837226\n",
      "[INFO random_forest.cc:578] Training of tree  231/300 (tree index:230) done accuracy:0.76588 logloss:0.834492\n",
      "[INFO random_forest.cc:578] Training of tree  241/300 (tree index:240) done accuracy:0.765924 logloss:0.828863\n",
      "[INFO random_forest.cc:578] Training of tree  251/300 (tree index:250) done accuracy:0.766503 logloss:0.822741\n",
      "[INFO random_forest.cc:578] Training of tree  261/300 (tree index:262) done accuracy:0.766102 logloss:0.819977\n",
      "[INFO random_forest.cc:578] Training of tree  271/300 (tree index:270) done accuracy:0.766503 logloss:0.815393\n",
      "[INFO random_forest.cc:578] Training of tree  281/300 (tree index:281) done accuracy:0.767261 logloss:0.809227\n",
      "[INFO random_forest.cc:578] Training of tree  291/300 (tree index:290) done accuracy:0.767305 logloss:0.804649\n",
      "[INFO random_forest.cc:578] Training of tree  300/300 (tree index:297) done accuracy:0.765702 logloss:0.803169\n",
      "[INFO random_forest.cc:645] Final OOB metrics: accuracy:0.765702 logloss:0.803169\n",
      "[INFO kernel.cc:856] Export model in log directory: /tmp/tmpusz_27m4\n",
      "[INFO kernel.cc:864] Save model in resources\n",
      "[INFO kernel.cc:929] Loading model from path\n",
      "[INFO decision_forest.cc:590] Model loaded with 300 root(s), 1374910 node(s), and 6 input feature(s).\n",
      "[INFO abstract_model.cc:876] Engine \"RandomForestGeneric\" built\n",
      "[INFO kernel.cc:797] Use fast generic engine\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.metrics import Precision\n",
    "model_rf= tfdf.keras.RandomForestModel()\n",
    "\n",
    "model_rf.compile(\n",
    "                metrics=[\"accuracy\"])\n",
    "\n",
    "with sys_pipes():\n",
    "    model_rf.fit(x=train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d223419",
   "metadata": {},
   "source": [
    "Se han creado trescientos árboles. El log nos muestra algunos de estos. Vemos que en general se obtiene un 76% de _accuracy_. Podemos obtener un resumen más reducido usando _summary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f4d66df0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"random_forest_model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Total params: 1\n",
      "Trainable params: 0\n",
      "Non-trainable params: 1\n",
      "_________________________________________________________________\n",
      "Type: \"RANDOM_FOREST\"\n",
      "Task: CLASSIFICATION\n",
      "Label: \"__LABEL\"\n",
      "\n",
      "Input Features (6):\n",
      "\tbkc\n",
      "\tbkr\n",
      "\twkc\n",
      "\twkr\n",
      "\twrc\n",
      "\twrr\n",
      "\n",
      "No weights\n",
      "\n",
      "Variable Importance: NUM_NODES:\n",
      "    1. \"wrr\" 216342.000000 ################\n",
      "    2. \"wrc\" 165397.000000 ###########\n",
      "    3. \"bkr\" 119478.000000 #######\n",
      "    4. \"bkc\" 112101.000000 #######\n",
      "    5. \"wkc\" 51255.000000 ##\n",
      "    6. \"wkr\" 22732.000000 \n",
      "\n",
      "Variable Importance: NUM_AS_ROOT:\n",
      "    1. \"bkr\" 205.000000 ################\n",
      "    2. \"wkr\" 78.000000 #####\n",
      "    3. \"wkc\" 12.000000 \n",
      "    4. \"bkc\"  5.000000 \n",
      "\n",
      "Variable Importance: SUM_SCORE:\n",
      "    1. \"wrr\" 2938632.366295 ################\n",
      "    2. \"wrc\" 2833222.359850 ##############\n",
      "    3. \"bkr\" 2734362.605383 #############\n",
      "    4. \"bkc\" 2527649.034721 ###########\n",
      "    5. \"wkr\" 1702105.100839 ###\n",
      "    6. \"wkc\" 1365919.120243 \n",
      "\n",
      "Variable Importance: MEAN_MIN_DEPTH:\n",
      "    1. \"__LABEL\" 12.360672 ################\n",
      "    2.     \"wrr\"  7.176756 #########\n",
      "    3.     \"wrc\"  5.558875 ######\n",
      "    4.     \"wkc\"  4.410273 #####\n",
      "    5.     \"wkr\"  2.251511 ##\n",
      "    6.     \"bkc\"  2.117759 ##\n",
      "    7.     \"bkr\"  0.481818 \n",
      "\n",
      "\n",
      "\n",
      "Winner take all: true\n",
      "Out-of-bag evaluation: accuracy:0.765702 logloss:0.803169\n",
      "Number of trees: 300\n",
      "Total number of nodes: 1374910\n",
      "\n",
      "Number of nodes by tree:\n",
      "Count: 300 Average: 4583.03 StdDev: 81.4853\n",
      "Min: 4331 Max: 4813 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 4331, 4355)  1   0.33%   0.33%\n",
      "[ 4355, 4379)  0   0.00%   0.33%\n",
      "[ 4379, 4403)  1   0.33%   0.67%\n",
      "[ 4403, 4427)  2   0.67%   1.33% #\n",
      "[ 4427, 4451)  6   2.00%   3.33% ##\n",
      "[ 4451, 4475) 14   4.67%   8.00% ####\n",
      "[ 4475, 4500) 25   8.33%  16.33% ######\n",
      "[ 4500, 4524) 27   9.00%  25.33% #######\n",
      "[ 4524, 4548) 24   8.00%  33.33% ######\n",
      "[ 4548, 4572) 40  13.33%  46.67% ##########\n",
      "[ 4572, 4596) 38  12.67%  59.33% ##########\n",
      "[ 4596, 4620) 29   9.67%  69.00% #######\n",
      "[ 4620, 4644) 30  10.00%  79.00% ########\n",
      "[ 4644, 4669) 18   6.00%  85.00% #####\n",
      "[ 4669, 4693) 15   5.00%  90.00% ####\n",
      "[ 4693, 4717) 12   4.00%  94.00% ###\n",
      "[ 4717, 4741)  8   2.67%  96.67% ##\n",
      "[ 4741, 4765)  4   1.33%  98.00% #\n",
      "[ 4765, 4789)  1   0.33%  98.33%\n",
      "[ 4789, 4813]  5   1.67% 100.00% #\n",
      "\n",
      "Depth by leafs:\n",
      "Count: 687605 Average: 12.3605 StdDev: 1.81321\n",
      "Min: 5 Max: 15 Ignored: 0\n",
      "----------------------------------------------\n",
      "[  5,  6)     10   0.00%   0.00%\n",
      "[  6,  7)    215   0.03%   0.03%\n",
      "[  7,  8)   2159   0.31%   0.35%\n",
      "[  8,  9)  10417   1.51%   1.86% #\n",
      "[  9, 10)  31503   4.58%   6.44% ##\n",
      "[ 10, 11)  68800  10.01%  16.45% #####\n",
      "[ 11, 12) 108362  15.76%  32.21% ########\n",
      "[ 12, 13) 129972  18.90%  51.11% ##########\n",
      "[ 13, 14) 130129  18.92%  70.04% ##########\n",
      "[ 14, 15) 106086  15.43%  85.46% ########\n",
      "[ 15, 15]  99952  14.54% 100.00% ########\n",
      "\n",
      "Number of training obs by leaf:\n",
      "Count: 687605 Average: 9.79487 StdDev: 10.2582\n",
      "Min: 5 Max: 257 Ignored: 0\n",
      "----------------------------------------------\n",
      "[   5,  17) 623807  90.72%  90.72% ##########\n",
      "[  17,  30)  40317   5.86%  96.59% #\n",
      "[  30,  42)  11791   1.71%  98.30%\n",
      "[  42,  55)   5563   0.81%  99.11%\n",
      "[  55,  68)   2509   0.36%  99.47%\n",
      "[  68,  80)   1172   0.17%  99.64%\n",
      "[  80,  93)    775   0.11%  99.76%\n",
      "[  93, 106)    492   0.07%  99.83%\n",
      "[ 106, 118)    269   0.04%  99.87%\n",
      "[ 118, 131)    202   0.03%  99.90%\n",
      "[ 131, 144)    173   0.03%  99.92%\n",
      "[ 144, 156)    111   0.02%  99.94%\n",
      "[ 156, 169)     73   0.01%  99.95%\n",
      "[ 169, 182)     67   0.01%  99.96%\n",
      "[ 182, 194)     65   0.01%  99.97%\n",
      "[ 194, 207)    113   0.02%  99.98%\n",
      "[ 207, 220)     62   0.01%  99.99%\n",
      "[ 220, 232)     41   0.01% 100.00%\n",
      "[ 232, 245)      2   0.00% 100.00%\n",
      "[ 245, 257]      1   0.00% 100.00%\n",
      "\n",
      "Attribute in nodes:\n",
      "\t216342 : wrr [NUMERICAL]\n",
      "\t165397 : wrc [CATEGORICAL]\n",
      "\t119478 : bkr [NUMERICAL]\n",
      "\t112101 : bkc [CATEGORICAL]\n",
      "\t51255 : wkc [CATEGORICAL]\n",
      "\t22732 : wkr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 0:\n",
      "\t205 : bkr [NUMERICAL]\n",
      "\t78 : wkr [NUMERICAL]\n",
      "\t12 : wkc [CATEGORICAL]\n",
      "\t5 : bkc [CATEGORICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 1:\n",
      "\t328 : bkr [NUMERICAL]\n",
      "\t225 : wkr [NUMERICAL]\n",
      "\t176 : bkc [CATEGORICAL]\n",
      "\t162 : wkc [CATEGORICAL]\n",
      "\t9 : wrr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 2:\n",
      "\t630 : bkc [CATEGORICAL]\n",
      "\t508 : wkr [NUMERICAL]\n",
      "\t455 : bkr [NUMERICAL]\n",
      "\t399 : wkc [CATEGORICAL]\n",
      "\t96 : wrr [NUMERICAL]\n",
      "\t12 : wrc [CATEGORICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 3:\n",
      "\t1211 : bkc [CATEGORICAL]\n",
      "\t982 : wkr [NUMERICAL]\n",
      "\t875 : wkc [CATEGORICAL]\n",
      "\t779 : bkr [NUMERICAL]\n",
      "\t411 : wrr [NUMERICAL]\n",
      "\t242 : wrc [CATEGORICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 5:\n",
      "\t4253 : wrc [CATEGORICAL]\n",
      "\t3484 : bkc [CATEGORICAL]\n",
      "\t3272 : bkr [NUMERICAL]\n",
      "\t2809 : wrr [NUMERICAL]\n",
      "\t2720 : wkc [CATEGORICAL]\n",
      "\t2352 : wkr [NUMERICAL]\n",
      "\n",
      "Condition type in nodes:\n",
      "\t358552 : HigherCondition\n",
      "\t328753 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 0:\n",
      "\t283 : HigherCondition\n",
      "\t17 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 1:\n",
      "\t562 : HigherCondition\n",
      "\t338 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 2:\n",
      "\t1059 : HigherCondition\n",
      "\t1041 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 3:\n",
      "\t2328 : ContainsBitmapCondition\n",
      "\t2172 : HigherCondition\n",
      "Condition type in nodes with depth <= 5:\n",
      "\t10457 : ContainsBitmapCondition\n",
      "\t8433 : HigherCondition\n",
      "\n",
      "Training OOB:\n",
      "\ttrees: 1, Out-of-bag evaluation: accuracy:0.663478 logloss:12.1295\n",
      "\ttrees: 11, Out-of-bag evaluation: accuracy:0.691763 logloss:5.05263\n",
      "\ttrees: 21, Out-of-bag evaluation: accuracy:0.730233 logloss:2.65216\n",
      "\ttrees: 31, Out-of-bag evaluation: accuracy:0.742316 logloss:1.95862\n",
      "\ttrees: 41, Out-of-bag evaluation: accuracy:0.750379 logloss:1.58524\n",
      "\ttrees: 51, Out-of-bag evaluation: accuracy:0.753497 logloss:1.37915\n",
      "\ttrees: 61, Out-of-bag evaluation: accuracy:0.755768 logloss:1.26819\n",
      "\ttrees: 71, Out-of-bag evaluation: accuracy:0.758708 logloss:1.17384\n",
      "\ttrees: 81, Out-of-bag evaluation: accuracy:0.75902 logloss:1.11116\n",
      "\ttrees: 91, Out-of-bag evaluation: accuracy:0.760713 logloss:1.0598\n",
      "\ttrees: 101, Out-of-bag evaluation: accuracy:0.761336 logloss:1.02395\n",
      "\ttrees: 111, Out-of-bag evaluation: accuracy:0.764276 logloss:0.992012\n",
      "\ttrees: 121, Out-of-bag evaluation: accuracy:0.763964 logloss:0.961497\n",
      "\ttrees: 131, Out-of-bag evaluation: accuracy:0.763563 logloss:0.944803\n",
      "\ttrees: 141, Out-of-bag evaluation: accuracy:0.764009 logloss:0.928943\n",
      "\ttrees: 151, Out-of-bag evaluation: accuracy:0.764365 logloss:0.913852\n",
      "\ttrees: 161, Out-of-bag evaluation: accuracy:0.764811 logloss:0.89613\n",
      "\ttrees: 171, Out-of-bag evaluation: accuracy:0.765479 logloss:0.881226\n",
      "\ttrees: 181, Out-of-bag evaluation: accuracy:0.764944 logloss:0.866638\n",
      "\ttrees: 191, Out-of-bag evaluation: accuracy:0.765345 logloss:0.863178\n",
      "\ttrees: 201, Out-of-bag evaluation: accuracy:0.764855 logloss:0.848468\n",
      "\ttrees: 211, Out-of-bag evaluation: accuracy:0.765345 logloss:0.843659\n",
      "\ttrees: 221, Out-of-bag evaluation: accuracy:0.766414 logloss:0.837226\n",
      "\ttrees: 231, Out-of-bag evaluation: accuracy:0.76588 logloss:0.834492\n",
      "\ttrees: 241, Out-of-bag evaluation: accuracy:0.765924 logloss:0.828863\n",
      "\ttrees: 251, Out-of-bag evaluation: accuracy:0.766503 logloss:0.822741\n",
      "\ttrees: 261, Out-of-bag evaluation: accuracy:0.766102 logloss:0.819977\n",
      "\ttrees: 271, Out-of-bag evaluation: accuracy:0.766503 logloss:0.815393\n",
      "\ttrees: 281, Out-of-bag evaluation: accuracy:0.767261 logloss:0.809227\n",
      "\ttrees: 291, Out-of-bag evaluation: accuracy:0.767305 logloss:0.804649\n",
      "\ttrees: 300, Out-of-bag evaluation: accuracy:0.765702 logloss:0.803169\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model_rf.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc258091",
   "metadata": {},
   "source": [
    "También recibimos información de las variables más importantes, que coinciden con el resultado de EDA (lógico ya que usa randomForest). También podemos ver el número de nodos y de hojas.\n",
    "\n",
    "Procedemos a probar el árbol con el conjunto test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b68b1609",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "88/88 [==============================] - 1s 6ms/step - loss: 0.0000e+00 - accuracy: 0.7683\n",
      "\n",
      "loss: 0.0000\n",
      "accuracy: 0.7683\n"
     ]
    }
   ],
   "source": [
    "evaluation = model_rf.evaluate(test, return_dict=True)\n",
    "print()\n",
    "\n",
    "for name, value in evaluation.items():\n",
    "    print(f\"{name}: {value:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e1b8fef",
   "metadata": {},
   "source": [
    "Obtenemos las predicciones para poder utilizar distintas métricas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "d94c8a88",
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = model_rf.predict(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "6112ccfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test_aux = test_aux[\"opt rank\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bcbd0ed8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix, precision_score, \\\n",
    "f1_score, cohen_kappa_score, recall_score\n",
    "\n",
    "def compute_metrics_multiclass(y_test, y_pred):\n",
    "    results=[]\n",
    "    results.append(precision_score(y_test, np.round(y_pred), average=\"micro\"))\n",
    "    results.append(recall_score(y_test, np.round(y_pred), average=\"micro\"))\n",
    "    results.append(f1_score(y_test, np.round(y_pred), average=\"micro\"))\n",
    "    results.append(cohen_kappa_score(y_test, np.round(y_pred)))\n",
    "    return results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e48c91a",
   "metadata": {},
   "source": [
    "Utilizamos la función argmax para poder obtener la clase predicha en cada caso."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "3c6757ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.7682839814484481, 0.7682839814484481, 0.768283981448448, 0.7408866826454885]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = np.argmax(preds, axis=1)\n",
    "\n",
    "metrics_rf = compute_metrics_multiclass(y_test_aux, preds)\n",
    "metrics_rf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "818284ef",
   "metadata": {},
   "source": [
    "Los resultados oscilan el 76% en todas las métricas. No son mejores que los obtenidos por las redes neuronales usando datos por defecto."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37e18596",
   "metadata": {},
   "source": [
    "Podemos realizar una visualización del árbol promedio de los 300, aunque, al ser multiclase, no nos es fácil interpretarlo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "31843468",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<script src=\"https://d3js.org/d3.v6.min.js\"></script>\n",
       "<div id=\"tree_plot_2fb9602ae33a4717960e562f4259ce2b\"></div>\n",
       "<script>\n",
       "/*\n",
       " * Copyright 2021 Google LLC.\n",
       " * Licensed under the Apache License, Version 2.0 (the \"License\");\n",
       " * you may not use this file except in compliance with the License.\n",
       " * You may obtain a copy of the License at\n",
       " *\n",
       " *     https://www.apache.org/licenses/LICENSE-2.0\n",
       " *\n",
       " * Unless required by applicable law or agreed to in writing, software\n",
       " * distributed under the License is distributed on an \"AS IS\" BASIS,\n",
       " * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
       " * See the License for the specific language governing permissions and\n",
       " * limitations under the License.\n",
       " */\n",
       "\n",
       "/**\n",
       " *  Plotting of decision trees generated by TF-DF.\n",
       " *\n",
       " *  A tree is a recursive structure of node objects.\n",
       " *  A node contains one or more of the following components:\n",
       " *\n",
       " *    - A value: Representing the output of the node. If the node is not a leaf,\n",
       " *      the value is only present for analysis i.e. it is not used for\n",
       " *      predictions.\n",
       " *\n",
       " *    - A condition : For non-leaf nodes, the condition (also known as split)\n",
       " *      defines a binary test to branch to the positive or negative child.\n",
       " *\n",
       " *    - An explanation: Generally a plot showing the relation between the label\n",
       " *      and the condition to give insights about the effect of the condition.\n",
       " *\n",
       " *    - Two children : For non-leaf nodes, the children nodes. The first\n",
       " *      children (i.e. \"node.children[0]\") is the negative children (drawn in\n",
       " *      red). The second children is the positive one (drawn in green).\n",
       " *\n",
       " */\n",
       "\n",
       "/**\n",
       " * Generate the default options object.\n",
       " * @return {!options} Default configuration options.\n",
       " */\n",
       "function build_default_options() {\n",
       "  // Display configuration.\n",
       "  // Note: Dimensions are expressed in pixels.\n",
       "  return {\n",
       "    // Margin around the entire plot.\n",
       "    margin: 10,\n",
       "\n",
       "    // Size of a tree node.\n",
       "    node_x_size: 160,\n",
       "    node_y_size: 12 * 2 + 4,\n",
       "\n",
       "    // Space between tree nodes.\n",
       "    node_x_offset: 160 + 20,\n",
       "    node_y_offset: 12 * 2 + 4 + 5,\n",
       "\n",
       "    // Text size.\n",
       "    font_size: 10,\n",
       "\n",
       "    // Rounding effect of the edges.\n",
       "    // This value is the distance (in pixel) of the Bezier control anchor from\n",
       "    // the source point.\n",
       "    edge_rounding: 20,\n",
       "\n",
       "    // Padding inside nodes.\n",
       "    node_padding: 2,\n",
       "\n",
       "    // Show a bb box around the plot. For debug only.\n",
       "    show_plot_bounding_box: false,\n",
       "  };\n",
       "}\n",
       "\n",
       "/**\n",
       " * Plots a single decision tree into a DOM element.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!tree} raw_tree Recursive tree structure.\n",
       " * @param {string} canvas_id Id of the output dom element.\n",
       " */\n",
       "function display_tree(options, raw_tree, canvas_id) {\n",
       "  // Get default options.\n",
       "  const default_options = build_default_options();\n",
       "  Object.keys(default_options).forEach(function(key) {\n",
       "    if (!(key in options)) {\n",
       "      options[key] = default_options[key];\n",
       "    }\n",
       "  });\n",
       "\n",
       "  console.log(options);\n",
       "\n",
       "  // Determine the node placement.\n",
       "  const tree_struct = d3.tree().nodeSize(\n",
       "      [options.node_y_offset, options.node_x_offset])(d3.hierarchy(raw_tree));\n",
       "\n",
       "  // Boundaries of the node placement.\n",
       "  let x_min = Infinity;\n",
       "  let x_max = -x_min;\n",
       "  let y_min = Infinity;\n",
       "  let y_max = -x_min;\n",
       "\n",
       "  tree_struct.each(d => {\n",
       "    if (d.x > x_max) x_max = d.x;\n",
       "    if (d.x < x_min) x_min = d.x;\n",
       "    if (d.y > y_max) y_max = d.y;\n",
       "    if (d.y < y_min) y_min = d.y;\n",
       "  });\n",
       "\n",
       "  // Size of the plot.\n",
       "  const width = y_max - y_min + options.node_x_size + options.margin * 2;\n",
       "  const height = x_max - x_min + options.node_y_size + options.margin * 2 +\n",
       "      options.node_y_offset - options.node_y_size;\n",
       "\n",
       "  const plot = d3.select(canvas_id);\n",
       "\n",
       "  // Tool tip\n",
       "  options.tooltip = plot.append('div')\n",
       "                        .attr('width', 100)\n",
       "                        .attr('height', 100)\n",
       "                        .style('padding', '4px')\n",
       "                        .style('background', '#fff')\n",
       "                        .style('box-shadow', '4px 4px 0px rgba(0,0,0,0.1)')\n",
       "                        .style('border', '1px solid black')\n",
       "                        .style('font-family', 'sans-serif')\n",
       "                        .style('font-size', options.font_size)\n",
       "                        .style('position', 'absolute')\n",
       "                        .style('z-index', '10')\n",
       "                        .attr('pointer-events', 'none')\n",
       "                        .style('display', 'none');\n",
       "\n",
       "  // Create canvas\n",
       "  const svg = plot.append('svg').attr('width', width).attr('height', height);\n",
       "  const graph =\n",
       "      svg.style('overflow', 'visible')\n",
       "          .append('g')\n",
       "          .attr('font-family', 'sans-serif')\n",
       "          .attr('font-size', options.font_size)\n",
       "          .attr(\n",
       "              'transform',\n",
       "              () => `translate(${options.margin},${\n",
       "                  - x_min + options.node_y_offset / 2 + options.margin})`);\n",
       "\n",
       "  // Plot bounding box.\n",
       "  if (options.show_plot_bounding_box) {\n",
       "    svg.append('rect')\n",
       "        .attr('width', width)\n",
       "        .attr('height', height)\n",
       "        .attr('fill', 'none')\n",
       "        .attr('stroke-width', 1.0)\n",
       "        .attr('stroke', 'black');\n",
       "  }\n",
       "\n",
       "  // Draw the edges.\n",
       "  display_edges(options, graph, tree_struct);\n",
       "\n",
       "  // Draw the nodes.\n",
       "  display_nodes(options, graph, tree_struct);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Draw the nodes of the tree.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!graph} graph D3 search handle containing the graph.\n",
       " * @param {!tree_struct} tree_struct Structure of the tree (node placement,\n",
       " *     data, etc.).\n",
       " */\n",
       "function display_nodes(options, graph, tree_struct) {\n",
       "  const nodes = graph.append('g')\n",
       "                    .selectAll('g')\n",
       "                    .data(tree_struct.descendants())\n",
       "                    .join('g')\n",
       "                    .attr('transform', d => `translate(${d.y},${d.x})`);\n",
       "\n",
       "  nodes.append('rect')\n",
       "      .attr('x', 0.5)\n",
       "      .attr('y', 0.5)\n",
       "      .attr('width', options.node_x_size)\n",
       "      .attr('height', options.node_y_size)\n",
       "      .attr('stroke', 'lightgrey')\n",
       "      .attr('stroke-width', 1)\n",
       "      .attr('fill', 'white')\n",
       "      .attr('y', -options.node_y_size / 2);\n",
       "\n",
       "  // Brackets on the right of condition nodes without children.\n",
       "  non_leaf_node_without_children =\n",
       "      nodes.filter(node => node.data.condition != null && node.children == null)\n",
       "          .append('g')\n",
       "          .attr('transform', `translate(${options.node_x_size},0)`);\n",
       "\n",
       "  non_leaf_node_without_children.append('path')\n",
       "      .attr('d', 'M0,0 C 10,0 0,10 10,10')\n",
       "      .attr('fill', 'none')\n",
       "      .attr('stroke-width', 1.0)\n",
       "      .attr('stroke', '#F00');\n",
       "\n",
       "  non_leaf_node_without_children.append('path')\n",
       "      .attr('d', 'M0,0 C 10,0 0,-10 10,-10')\n",
       "      .attr('fill', 'none')\n",
       "      .attr('stroke-width', 1.0)\n",
       "      .attr('stroke', '#0F0');\n",
       "\n",
       "  const node_content = nodes.append('g').attr(\n",
       "      'transform',\n",
       "      `translate(0,${options.node_padding - options.node_y_size / 2})`);\n",
       "\n",
       "  node_content.append(node => create_node_element(options, node));\n",
       "}\n",
       "\n",
       "/**\n",
       " * Creates the D3 content for a single node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!node} node Node to draw.\n",
       " * @return {!d3} D3 content.\n",
       " */\n",
       "function create_node_element(options, node) {\n",
       "  // Output accumulator.\n",
       "  let output = {\n",
       "    // Content to draw.\n",
       "    content: d3.create('svg:g'),\n",
       "    // Vertical offset to the next element to draw.\n",
       "    vertical_offset: 0\n",
       "  };\n",
       "\n",
       "  // Conditions.\n",
       "  if (node.data.condition != null) {\n",
       "    display_condition(options, node.data.condition, output);\n",
       "  }\n",
       "\n",
       "  // Values.\n",
       "  if (node.data.value != null) {\n",
       "    display_value(options, node.data.value, output);\n",
       "  }\n",
       "\n",
       "  // Explanations.\n",
       "  if (node.data.explanation != null) {\n",
       "    display_explanation(options, node.data.explanation, output);\n",
       "  }\n",
       "\n",
       "  return output.content.node();\n",
       "}\n",
       "\n",
       "\n",
       "/**\n",
       " * Adds a single line of text inside of a node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {string} text Text to display.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_node_text(options, text, output) {\n",
       "  output.content.append('text')\n",
       "      .attr('x', options.node_padding)\n",
       "      .attr('y', output.vertical_offset)\n",
       "      .attr('alignment-baseline', 'hanging')\n",
       "      .text(text);\n",
       "  output.vertical_offset += 10;\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds a single line of text inside of a node with a tooltip.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {string} text Text to display.\n",
       " * @param {string} tooltip Text in the Tooltip.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_node_text_with_tooltip(options, text, tooltip, output) {\n",
       "  const item = output.content.append('text')\n",
       "                   .attr('x', options.node_padding)\n",
       "                   .attr('alignment-baseline', 'hanging')\n",
       "                   .text(text);\n",
       "\n",
       "  add_tooltip(options, item, () => tooltip);\n",
       "  output.vertical_offset += 10;\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds a tooltip to a dom element.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!dom} target Dom element to equip with a tooltip.\n",
       " * @param {!func} get_content Generates the html content of the tooltip.\n",
       " */\n",
       "function add_tooltip(options, target, get_content) {\n",
       "  function show(d) {\n",
       "    options.tooltip.style('display', 'block');\n",
       "    options.tooltip.html(get_content());\n",
       "  }\n",
       "\n",
       "  function hide(d) {\n",
       "    options.tooltip.style('display', 'none');\n",
       "  }\n",
       "\n",
       "  function move(d) {\n",
       "    options.tooltip.style('display', 'block');\n",
       "    options.tooltip.style('left', (d.pageX + 5) + 'px');\n",
       "    options.tooltip.style('top', d.pageY + 'px');\n",
       "  }\n",
       "\n",
       "  target.on('mouseover', show);\n",
       "  target.on('mouseout', hide);\n",
       "  target.on('mousemove', move);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds a condition inside of a node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!condition} condition Condition to display.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_condition(options, condition, output) {\n",
       "  threshold_format = d3.format('r');\n",
       "\n",
       "  if (condition.type === 'IS_MISSING') {\n",
       "    display_node_text(options, `${condition.attribute} is missing`, output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'IS_TRUE') {\n",
       "    display_node_text(options, `${condition.attribute} is true`, output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'NUMERICAL_IS_HIGHER_THAN') {\n",
       "    format = d3.format('r');\n",
       "    display_node_text(\n",
       "        options,\n",
       "        `${condition.attribute} >= ${threshold_format(condition.threshold)}`,\n",
       "        output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'CATEGORICAL_IS_IN') {\n",
       "    display_node_text_with_tooltip(\n",
       "        options, `${condition.attribute} in [...]`,\n",
       "        `${condition.attribute} in [${condition.mask}]`, output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'CATEGORICAL_SET_CONTAINS') {\n",
       "    display_node_text_with_tooltip(\n",
       "        options, `${condition.attribute} intersect [...]`,\n",
       "        `${condition.attribute} intersect [${condition.mask}]`, output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (condition.type === 'NUMERICAL_SPARSE_OBLIQUE') {\n",
       "    display_node_text_with_tooltip(\n",
       "        options, `Sparse oblique split...`,\n",
       "        `[${condition.attributes}]*[${condition.weights}]>=${\n",
       "            threshold_format(condition.threshold)}`,\n",
       "        output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  display_node_text(\n",
       "      options, `Non supported condition ${condition.type}`, output);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds a value inside of a node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!value} value Value to display.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_value(options, value, output) {\n",
       "  if (value.type === 'PROBABILITY') {\n",
       "    const left_margin = 0;\n",
       "    const right_margin = 50;\n",
       "    const plot_width = options.node_x_size - options.node_padding * 2 -\n",
       "        left_margin - right_margin;\n",
       "\n",
       "    let cusum = Array.from(d3.cumsum(value.distribution));\n",
       "    cusum.unshift(0);\n",
       "    const distribution_plot = output.content.append('g').attr(\n",
       "        'transform', `translate(0,${output.vertical_offset + 0.5})`);\n",
       "\n",
       "    distribution_plot.selectAll('rect')\n",
       "        .data(value.distribution)\n",
       "        .join('rect')\n",
       "        .attr('height', 10)\n",
       "        .attr(\n",
       "            'x',\n",
       "            (d, i) =>\n",
       "                (cusum[i] * plot_width + left_margin + options.node_padding))\n",
       "        .attr('width', (d, i) => d * plot_width)\n",
       "        .style('fill', (d, i) => d3.schemeSet1[i]);\n",
       "\n",
       "    const num_examples =\n",
       "        output.content.append('g')\n",
       "            .attr('transform', `translate(0,${output.vertical_offset})`)\n",
       "            .append('text')\n",
       "            .attr('x', options.node_x_size - options.node_padding)\n",
       "            .attr('alignment-baseline', 'hanging')\n",
       "            .attr('text-anchor', 'end')\n",
       "            .text(`(${value.num_examples})`);\n",
       "\n",
       "    const distribution_details = d3.create('ul');\n",
       "    distribution_details.selectAll('li')\n",
       "        .data(value.distribution)\n",
       "        .join('li')\n",
       "        .append('span')\n",
       "        .text(\n",
       "            (d, i) =>\n",
       "                'class ' + i + ': ' + d3.format('.3%')(value.distribution[i]));\n",
       "\n",
       "    add_tooltip(options, distribution_plot, () => distribution_details.html());\n",
       "    add_tooltip(options, num_examples, () => 'Number of examples');\n",
       "\n",
       "    output.vertical_offset += 10;\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  if (value.type === 'REGRESSION') {\n",
       "    display_node_text(\n",
       "        options,\n",
       "        'value: ' + d3.format('r')(value.value) + ` (${value.num_examples})`,\n",
       "        output);\n",
       "    return;\n",
       "  }\n",
       "\n",
       "  display_node_text(options, `Non supported value ${value.type}`, output);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Adds an explanation inside of a node.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!explanation} explanation Explanation to display.\n",
       " * @param {!output} output Output display accumulator.\n",
       " */\n",
       "function display_explanation(options, explanation, output) {\n",
       "  // Margin before the explanation.\n",
       "  output.vertical_offset += 10;\n",
       "\n",
       "  display_node_text(\n",
       "      options, `Non supported explanation ${explanation.type}`, output);\n",
       "}\n",
       "\n",
       "\n",
       "/**\n",
       " * Draw the edges of the tree.\n",
       " * @param {!options} options Dictionary of configurations.\n",
       " * @param {!graph} graph D3 search handle containing the graph.\n",
       " * @param {!tree_struct} tree_struct Structure of the tree (node placement,\n",
       " *     data, etc.).\n",
       " */\n",
       "function display_edges(options, graph, tree_struct) {\n",
       "  // Draw an edge between a parent and a child node with a bezier.\n",
       "  function draw_single_edge(d) {\n",
       "    return 'M' + (d.source.y + options.node_x_size) + ',' + d.source.x + ' C' +\n",
       "        (d.source.y + options.node_x_size + options.edge_rounding) + ',' +\n",
       "        d.source.x + ' ' + (d.target.y - options.edge_rounding) + ',' +\n",
       "        d.target.x + ' ' + d.target.y + ',' + d.target.x;\n",
       "  }\n",
       "\n",
       "  graph.append('g')\n",
       "      .attr('fill', 'none')\n",
       "      .attr('stroke-width', 1.2)\n",
       "      .selectAll('path')\n",
       "      .data(tree_struct.links())\n",
       "      .join('path')\n",
       "      .attr('d', draw_single_edge)\n",
       "      .attr(\n",
       "          'stroke', d => (d.target === d.source.children[0]) ? '#0F0' : '#F00');\n",
       "}\n",
       "\n",
       "display_tree({}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.09706013363028954, 0.0005790645879732739, 0.0026280623608017817, 0.010200445434298442, 0.0032516703786191537, 0.0072160356347438755, 0.017060133630289534, 0.01977728285077951, 0.024409799554565702, 0.05309576837416481, 0.06293986636971047, 0.07060133630289532, 0.10311804008908686, 0.12650334075723832, 0.15287305122494432, 0.158173719376392, 0.07643652561247216, 0.014075723830734967], \"num_examples\": 22450.0}, \"condition\": {\"type\": \"NUMERICAL_IS_HIGHER_THAN\", \"attribute\": \"bkr\", \"threshold\": 2.5}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.10473432235947751, 0.0, 0.00035463088835037534, 0.0, 0.0, 0.00035463088835037534, 0.002127785330102252, 0.003960044919912524, 0.01365328920148945, 0.016490336308292452, 0.04426975589573852, 0.06135114368461493, 0.11070394231337549, 0.14758555470181453, 0.18783616052958213, 0.19510609374076482, 0.09456823689343342, 0.016904072344701225], \"num_examples\": 16919.0}, \"condition\": {\"type\": \"CATEGORICAL_IS_IN\", \"attribute\": \"wkc\", \"mask\": [\"b\", \"a\"]}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.10992217898832685, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.002188715953307393, 0.002188715953307393, 0.010700389105058366, 0.025535019455252918, 0.04547665369649805, 0.08073929961089495, 0.16609922178988326, 0.25583657587548636, 0.23176070038910507, 0.06955252918287938], \"num_examples\": 4112.0}, \"condition\": {\"type\": \"NUMERICAL_IS_HIGHER_THAN\", \"attribute\": \"bkr\", \"threshold\": 6.5}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.09959758551307847, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.006036217303822937, 0.002012072434607646, 0.02112676056338028, 0.04024144869215292, 0.07344064386317907, 0.11569416498993963, 0.23038229376257546, 0.3591549295774648, 0.04627766599597585, 0.006036217303822937], \"num_examples\": 994.0}, \"condition\": {\"type\": \"CATEGORICAL_IS_IN\", \"attribute\": \"wrc\", \"mask\": [\"f\", \"h\", \"e\", \"a\", \"d\"]}}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.11321359846055164, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0009621552277100705, 0.0022450288646568314, 0.0073765234124438745, 0.02084669660038486, 0.036561898652982684, 0.06959589480436178, 0.14560615779345734, 0.22289929441949968, 0.290891597177678, 0.08980115458627325], \"num_examples\": 3118.0}, \"condition\": {\"type\": \"CATEGORICAL_IS_IN\", \"attribute\": \"bkc\", \"mask\": [\"h\", \"a\", \"b\"]}}]}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.10306863434059499, 0.0, 0.00046849379245724994, 0.0, 0.0, 0.00046849379245724994, 0.0028109627547434997, 0.005231514015772624, 0.01733427032091825, 0.02108222066057625, 0.05504802061372687, 0.07285078472710237, 0.13164675568048723, 0.16904817677832434, 0.1948153353634731, 0.17560708987272586, 0.05051924728664012, 0.0], \"num_examples\": 12807.0}, \"condition\": {\"type\": \"NUMERICAL_IS_HIGHER_THAN\", \"attribute\": \"wkr\", \"threshold\": 1.5}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.09838343015913109, 0.0, 0.0007577671129072998, 0.0, 0.0, 0.0007577671129072998, 0.004041424602172266, 0.007956554685526649, 0.023238191462490527, 0.025890376357666077, 0.08133367011871685, 0.10015155342258146, 0.17504420308158625, 0.20762818893660015, 0.20661783278605708, 0.06592573882293508, 0.0022733013387218996, 0.0], \"num_examples\": 7918.0}, \"condition\": {\"type\": \"CATEGORICAL_IS_IN\", \"attribute\": \"bkc\", \"mask\": [\"g\", \"f\", \"e\", \"d\", \"c\"]}}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.1106565759869094, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0008181632235631009, 0.0008181632235631009, 0.007772550623849458, 0.013295152382900388, 0.012476989159337287, 0.028635712824708528, 0.06136224176723256, 0.10656575986909389, 0.17570055226017592, 0.3532419717733688, 0.1286561669052976, 0.0], \"num_examples\": 4889.0}, \"condition\": {\"type\": \"CATEGORICAL_IS_IN\", \"attribute\": \"bkc\", \"mask\": [\"h\", \"a\"]}}]}]}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.0735852467908154, 0.002350388718134153, 0.009582354004700777, 0.041403001265593924, 0.01319833664798409, 0.028204664617609836, 0.06273729886096546, 0.06816127282589043, 0.0573133248960405, 0.16506960766588322, 0.12005062375700597, 0.09889712529379859, 0.0799132164165612, 0.06201410233230881, 0.04592297956969806, 0.0451997830410414, 0.02097269933104321, 0.005423973964924968], \"num_examples\": 5531.0}, \"condition\": {\"type\": \"CATEGORICAL_IS_IN\", \"attribute\": \"bkc\", \"mask\": [\"h\", \"g\", \"f\", \"e\", \"d\", \"c\"]}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.07390042420183077, 0.0011163206072784104, 0.0017861129716454567, 0.01518196025898638, 0.0015628488501897744, 0.008484036615315918, 0.03415941058271936, 0.05648582272828757, 0.059834784550122797, 0.18508595668676045, 0.14333556597454788, 0.11654387139986604, 0.08617995088189329, 0.07300736771600803, 0.05514623799955347, 0.055592766242464835, 0.02589863808885912, 0.006697923643670462], \"num_examples\": 4479.0}, \"condition\": {\"type\": \"CATEGORICAL_IS_IN\", \"attribute\": \"wkc\", \"mask\": [\"b\", \"a\"]}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.08094327597195666, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.009560229445506692, 0.015296367112810707, 0.052262587635436585, 0.11281070745697896, 0.11536010197578075, 0.1504142766093053, 0.1338432122370937, 0.11854684512428298, 0.12619502868068833, 0.06564690885914595, 0.019120458891013385], \"num_examples\": 1569.0}, \"condition\": {\"type\": \"NUMERICAL_IS_HIGHER_THAN\", \"attribute\": \"wrr\", \"threshold\": 3.5}}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.07010309278350516, 0.001718213058419244, 0.0027491408934707906, 0.02336769759450172, 0.0024054982817869417, 0.013058419243986255, 0.05257731958762887, 0.08178694158075601, 0.0838487972508591, 0.256701030927835, 0.15979381443298968, 0.11718213058419244, 0.05154639175257732, 0.04020618556701031, 0.020962199312714775, 0.01752577319587629, 0.004467353951890034, 0.0], \"num_examples\": 2910.0}, \"condition\": {\"type\": \"NUMERICAL_IS_HIGHER_THAN\", \"attribute\": \"wrr\", \"threshold\": 3.5}}]}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.07224334600760456, 0.0076045627376425855, 0.04277566539923954, 0.15304182509505704, 0.06273764258555133, 0.11216730038022814, 0.1844106463878327, 0.11787072243346007, 0.04657794676806084, 0.07984790874524715, 0.02091254752851711, 0.02376425855513308, 0.053231939163498096, 0.015209125475285171, 0.006653992395437262, 0.0009505703422053232, 0.0, 0.0], \"num_examples\": 1052.0}, \"condition\": {\"type\": \"CATEGORICAL_IS_IN\", \"attribute\": \"wrc\", \"mask\": [\"a\", \"c\", \"b\"]}, \"children\": [{\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.19241192411924118, 0.02168021680216802, 0.03523035230352303, 0.04607046070460705, 0.11653116531165311, 0.04878048780487805, 0.12195121951219512, 0.10840108401084012, 0.07317073170731707, 0.04065040650406504, 0.05962059620596206, 0.05420054200542006, 0.04878048780487805, 0.024390243902439025, 0.005420054200542005, 0.0027100271002710027, 0.0, 0.0], \"num_examples\": 369.0}, \"condition\": {\"type\": \"CATEGORICAL_IS_IN\", \"attribute\": \"wkc\", \"mask\": [\"d\"]}}, {\"value\": {\"type\": \"PROBABILITY\", \"distribution\": [0.007320644216691069, 0.0, 0.04685212298682284, 0.2108345534407028, 0.03367496339677892, 0.14641288433382138, 0.21815519765739386, 0.12298682284040996, 0.032210834553440704, 0.10102489019033675, 0.0, 0.007320644216691069, 0.055636896046852125, 0.010248901903367497, 0.007320644216691069, 0.0, 0.0, 0.0], \"num_examples\": 683.0}, \"condition\": {\"type\": \"CATEGORICAL_IS_IN\", \"attribute\": \"wkc\", \"mask\": [\"c\"]}}]}]}]}, \"#tree_plot_2fb9602ae33a4717960e562f4259ce2b\")\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfdf.model_plotter.plot_model_in_colab(model_rf, tree_idx=0, max_depth=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4a717d5",
   "metadata": {},
   "source": [
    "Podemos ver cómo ha evolucionado el entrenamiento del modelo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "1f9273ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def make_figure(model):\n",
    "    logs = model.make_inspector().training_logs()\n",
    "\n",
    "    plt.figure(figsize=(12, 4))\n",
    "\n",
    "    plt.subplot(1, 2, 1)\n",
    "    plt.plot([log.num_trees for log in logs], [log.evaluation.accuracy for log in logs])\n",
    "    plt.xlabel(\"Number of trees\")\n",
    "    plt.ylabel(\"Accuracy (out-of-bag)\")\n",
    "\n",
    "    plt.subplot(1, 2, 2)\n",
    "    plt.plot([log.num_trees for log in logs], [log.evaluation.loss for log in logs])\n",
    "    plt.xlabel(\"Number of trees\")\n",
    "    plt.ylabel(\"Logloss (out-of-bag)\")\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0b1c9e1a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtcAAAEGCAYAAACuBLlKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABCIUlEQVR4nO3deXxc5XX/8c8ZaUa7LNmWF2wLG2NjlgB2HLIQlkAWKFvWFhKSNEsJbUhI2qaBX5v912ahSeGXjZKEkIVACJACCQ0kFEJCE7DBBmyMjTFgyTuWZMvapTm/P+6VPAgtI2tG9470fb9e85p7n7lz79HYvj565nnOY+6OiIiIiIiMXyLqAEREREREJgsl1yIiIiIiOaLkWkREREQkR5Rci4iIiIjkiJJrEREREZEcKY46gFyaOXOmL1y4MOowRETG7NFHH33R3euijmMi6Z4tIoVqpHv2pEquFy5cyOrVq6MOQ0RkzMzshahjmGi6Z4tIoRrpnq1hISIiIiIiOaLkWkREREQkR5Rci4iIiIjkiJJrEREREZEcUXItIiIiIpIjSq5FRERERHJEybWIiIiISI5MqjrXIiJjkU472/d1sGVPGy80tdPTmybtHj6gL+145jZw4oJpnLKkjmTR1OybMLPrgXOB3e5+XNh2FXAe0A08C3zA3Vtyfe197T1c/9BzvPHo2bxi/rRcn15EJCeUXIvIpNfR3cdzL7bx7J4D4aONZ3cfYMuLB+jsSY/5fDMrU5x/wjzevmIexx5WjZnlIerYugH4FvDjjLbfAle6e6+ZfRW4Evh0ri/c3ZfmmvueYUZlSsm1iMSWkmsRGda+9h7WNDSzZmsLaxpa6OjuZc60Mg6bVsqcaaXMnVbKnGllzJ1WyszKEooS2SeZPX1pdu7rZFtLB9tbOtjW3MH2fR00Ngf7O/Z1kipOMKMixYyKEqZXpJhemWJGRSrYDtvNYG9bN3sPdNHU1s3etm6aDnSH213sbeumpb1n4LpmML+2jMV1lbx28QyOqKtgcV0lC2dUUJpMkEgYCTMSRvgcbBcljJ4+54GNu7n9sW385M/Pc/1Dz3HU7CrevmIeb10+j9nVpfn4Y4gVd3/QzBYOars3Y/fPwDvzce2a8iQAzW09oxwpIhIdJdciE6wv7SSM2PV29qWdTbtaeWxrkEw/trWZLXvaAEgYLJ1dRXVZkicaW7hnfSfdvS/t8S1KGLOrSqitSOEOaXfcoa9/mEU6GF6Rdqe7N82LB7pI+0tjmFGRYl5tGUtmVXHq0jp6+5ymtm5ePNDFs3sOsOr5bprbu1/2vn5mML38YPJ91JwqplekmFVVOpBEL5pZQWmy6JA+o1Sx8eZj5/DmY+fQ0t7Nr57Ywe2PNfLl/36ar/7maU4+ciZvXzGPC06YR2IMv2hMMh8Efp6PEyeLElSVFtPc3p2P04uI5ISSa5E8aOvq5YW97bywNxjL+8LedrY2tfHC3na2t3RQliwKe37LMnqAg+e5YU+wmdHU1k1TWxd7B3piw+cDQY9sd2+aWdWlzK4qYVZ1CbOrS5lVVcrs6hJmVZdSWXLwn/iBrl527e9k1/5Odu/vYndrJ7v2d7Frfyc793WyYcd+2rr7AJhekWJFfQ3vWDGf5fU1HD+/5iXncnea23vYsa+Dnfs62bGvkx37gt7mlvael/b6Jg5uFyUMMyhOGHOqSzmspox5tWXBc01ZVklvX9rZ19Ez8Lk4DPRm15SnxtR7Ph415Skufs3hXPyaw3nuxTZ++Vgjt6/ZxncfeJa3njhvQmKIGzP7Z6AXuHGEYy4BLgGor68f8zVqy1NKrkUk1pRci4xBZ08fe1q7ePFAV/jc/ZL9Xa2dNDS18+KBl/7nX1uepH5GBa88vJYLTjyMju40O/cHyegfn3mR3a2dw/bGDlaeKmJGZYrpFSWUFCVYt20fv9vXSUdP38uOrUgVURMmI+3dL3+9LFk0kIi/85XzWV5fy/L6Guqnl4/Ys25mA73Dxx42sWNfixIHr33krAm99LAWzazg7998FJ9441L2HOiK3bcSE8HM3k8w0fFMdx/2b7O7XwdcB7By5cos/9YfVFuepLldw0JEJL6UXEus9PSl+fGfXuDBTXuoLC2mujTJtLIk1WWZ28HztLIk08tTVJcVH1Iy09OXpqktSI6b2oLhBi3tPQPPLe3dNGc8N7d109rVO+S5asqTzKwsoa6yhDOXzaZ+RjkLZ1Rw+Ixy6meUU12aHDGW3r40ew50sWNf0Iu8vaUD4CVji/vHGw/Vu+vuYc90F7v3d7K7tSvspe6iub2b2vIUs/t7tqtLBnq3K0sO7bOToSUSNiXGXQ9mZmcRTGA8zd3b83mtGvVci0jMKbmW2Hh4y14+e8d6Nu5qZcmsSvrc2d/Ry/6OHrr7hq/oUJwwaiteOtFtZmUwAa62PElb9+De5qDHualt+P+gq0qLqSlPUlseDDVYOLOC2vIUdVUlzKzsfy6hrqqEGRUlpIrHV5atuCgRDgcpO6T3mxlVpUmqSpMcOatyXLGIjMTMbgJOB2aaWSPwOYLqICXAb8Nf1v7s7pfm4/q15Um2vHggH6cWEckJJdcSuT2tXXz57g3cvmYb82rKuO69r+RNx8we6FF1d7p60+zr6GF/Rw/7O3vY19FDS3tPOCb54HjkvQe6WLdtH3vbumntPNjLXJYsoq4qSIYXzazgVQunD0qQgyS6pjxJTVmS4ilaw1hkNO5+0RDNP5io69dWpGhRtRARiTEl1xKZvrTz0z+/wL/fu5HOnj4++obFXPaGJZSlXjrswcwoTRZRmiwa01fu3b1pWjq6qUgVU1Giv+oik0FteYrWrl66e9Pj/sZIRCQflHFIJB7b2sxn/msd67fv5/VHzuQLFxzL4rrcDmdIFSeYVTX1xr+KTGa1Ya3rlo5u/fsWkVhSci0Txt3Zvq+T//e7Z/j56gZmV5fwrXcv55xXzNWkOhHJSk15CoCW9h4l1yISS3lNrsMZ5NcARcD33f0rg17/FPCejFiOBurcvcnMaoDvA8cBDnzQ3f+Uz3hlfNyD+sONzR00NLXT0NxOQ1NH+NxOY3MHXb1pihPGJacewcfPXPKS2skiIqOpDZPr5hEmJIuIRClvmY2ZFQHfBt4ENAKrzOxOd3+q/xh3vwq4Kjz+POCT7t4UvnwN8Bt3f6eZpYDyfMUqhy6ddla/0Mwda7dxz/qdL6vvXF1azILp5SyZVcUbjprFgunlnHzkTFW0EJFDMrAEumpdi0hM5bPb8CRgs7tvATCzm4ELgKeGOf4i4Kbw2GrgVOCvAdy9G1A3RUy4O+u37+fOx7dz1+Pb2bGvk9JkgjOPns3yBTXMry1nfm0ZC6aXM61s5PrOIiJjMb0i7LlWrWsRial8JtfzgIaM/Ubg1UMdaGblwFnAZWHTEcAe4IdmdgLwKHC5u7cN8d5xLaUr2Xt2zwHuXBsk1FtebKM4YZy2tI4rzl7GG4+erYocIpJ3A8NClFyLSEzlMxsaaobacEvdngc8lDEkpBhYAXzM3R82s2uAK4DPvOyE41xKd6rr7Uvz/N429nX0hHWkewfqSe/LqCn9wt52nt7Zihm8etF0PnzKEZx93Bxqw14kEZGJUJYqoqQ4QYuGhYhITOUzuW4EFmTszwe2D3PshYRDQjLe2+juD4f7txIk15JDf3p2L5+5Yx2bdw+92llZsmhg6fEZFSX8yzlHc+7xhzFnmmboi0h0astTmtAoIrGVz+R6FbDEzBYB2wgS6HcPPsjMpgGnARf3t7n7TjNrMLOj3H0jcCbDj9WWMdq9v5N/vXsDd6zdzoLpZXzl7a9gbk1ZkEiXFlNdlqS6NKkFGkQklmrKk5rQKCKxlbfk2t17zewy4B6CUnzXu/t6M7s0fP3a8NC3AfcOMZ76Y8CNYaWQLcAH8hXrVNHbl+bHf3qB//jtJrp603z8zCX83emLKU0Wjf5mEZGYqC1P0aIx1yISU3mdgebudwN3D2q7dtD+DcANQ7x3LbAyf9FNLaufb+Jf/msdT+9s5bSldXzh/GNZOLMi6rBERMZsekWKDTv3Rx2GiMiQVN5hknvxQBdf+e+nufXRRg6bVsq1F6/gLcfO0YqIIlKwasqTmtAoIrGl5HqS6uju46ZHtnL17zbR3t3Hpact5uNnHkl5Sn/kIlLY+oeFpNNOIqGOAhGJF2Vak8y+9h5+/Kfn+eH/Pk9TWzevWzyDL15wLEfOqoo6NBGRnKgpT5J2aO3sZVq5FqoSkXhRcj1J7NjXwQ/+8Bw3PbKVtu4+zlg2i789fTGvWjg96tBERHIqcyEZJdciEjdKrgvc5t0HuO7BZ/nlmm2kHc4/4TA+ctoRLJtTHXVoIiJ5kbkE+kI0MVtE4kXJdYF6vKGF7zywmXuf2kVJcYJ3n1TPh085ggXTy6MOTUQkr2rC3motgS4icaTkusDs6+jhy3dv4OZVDUwrS/KxNxzJ+1+3kBmVJVGHJiIyIQaGhbSpYoiIxI+S6wLym3U7+Mwd69l7oIuPnHoEHztzCZUl+iMUkaklc8y1iEjcKDMrALv2d/LZO9Zxz/pdHDO3mh/+9as4bt60qMMSEYlEVWkxCUO1rkUklpRcx1g67fx8dQP/dvcGunvTXHH2Mj70+kUkixJRhyYiEplEwqgpT6nnWkRiScl1TG3Zc4Arb3+Sh59r4jVHTOfLbz+eRVquXEQEgNrypJJrEYklJdcx05d2rv39s1xz3zOUFCf46jtewV+uXKDlykVEMtSWpzShUURiScl1zPzng89y1T0bOfu4OXzh/GOZVV0adUgiIrFTU56isbk96jBERF5Gg3djZFtLB9+8bzNvOXY23734lUqsRUSGUVue1IRGEYklJdcx8qW7nsJxPnvesVGHIiIyJDO73sx2m9m6jLbpZvZbM3smfK7Ndxy1FZrQKCLxpOQ6Jh7YuJvfrN/Jx85YwryasqjDEREZzg3AWYPargDuc/clwH3hfl7Vlqfo6k3T0d2X70uJiIyJkusY6Ozp4/N3rueImRV8+JRFUYcjIjIsd38QaBrUfAHwo3D7R8Bb8x1HbbgEepN6r0UkZpRcx8D3HtzC83vb+cIFx1JSXBR1OCIiYzXb3XcAhM+zhjvQzC4xs9VmtnrPnj2HfMGagSXQlVyLSLwouY5YQ1M737p/M+e8Yi6nLKmLOhwRkbxy9+vcfaW7r6yrO/R7Xn/PtSY1ikjcKLmO2BfueoqihPEv5x4ddSgiIodql5nNBQifd+f7grUVYc+1hoWISMwouY7QfRt28bsNu7j8zCXMnaZJjCJSsO4E3h9uvx+4I98XrBnouVZyLSLxouQ6Ip09fXz+rvUcOauSD5ysSYwiUhjM7CbgT8BRZtZoZh8CvgK8ycyeAd4U7udVbTjmukmrNIpIzGiFxoh894FnaWjq4Gd/82pSxfodR0QKg7tfNMxLZ05kHMmiBFUlxRoWIiKxo6wuAi/sbeO7v3+W8084jNctnhl1OCIiBammIqlhISISO0quJ5i78/k715NMGP98jiYxiogcqtryFM2qFiIiMaPkeoL99qld3L9xD59801JmV5dGHY6ISMGqKU+p51pEYkfJ9QTq6O7jC3c9xVGzq3j/6xZGHY6ISEGrLU+q51pEYmfUCY1mlgBOAA4DOoD17r4r34FNRt/7wxa2tXTw80teQ7JIv9eIiIxHbXlKKzSKSOwMm1yb2WLg08AbgWeAPUApsNTM2oH/BH7k7umJCLTQ9fSl+emfX+D0o+p49REzog5HRKTg1ZanaO3qpacvrQ4LEYmNke5G/xf4KbDY3d/i7he7+zvd/XjgfGAa8N6RTm5mZ5nZRjPbbGZXDPH6p8xsbfhYZ2Z9ZjY94/UiM1tjZr86tB8vPu7bsJvdrV2859WHRx2KiMikUFuhJdBFJH6G7bkeoZYp7r4buHqkE5tZEfBtggUFGoFVZnanuz+VcZ6rgKvC488DPunuTRmnuRzYAFSP+pPE3M8e2cqc6lLecFRd1KGIiEwKNeFCMi3t3dRVlUQcjYhIIJsx128fonkf8GSYZA/nJGCzu28Jz3MzcAHw1DDHXwTclHHd+cA5wL8Cfz9anHHW0NTOH57Zw8fPWEKxvroUEcmJ2nAJdE1qFJE4yWaFxg8BrwXuD/dPB/5MMPb6i+7+k2HeNw9oyNhvBF491IFmVg6cBVyW0Xw18E9A1UjBmdklwCUA9fX1Ix0amZse2YoBF560IOpQREQws1LgXOAUDk5WXwf82t3XRxnbWPQvga5VGkUkTrLpRk0DR7v7O9z9HcAxQBdBovzpEd5nQ7T5MMeeBzzUPyTEzM4Fdrv7o6MF5+7XuftKd19ZVxe/IRc9fWluWd3IGctmMXdaWdThiMgUZ2afBx4i6DR5mGBy+i1AL/AVM/utmR0fXYTZq60Ik2tVDBGRGMmm53rhoNJ7u4Gl7t5kZiN9F9cIZHbVzge2D3PshWQMCQFOBs43s78gqFBSbWY/dfeLs4g3Vn771C5ePNDFu18dz151EZlyVrn754d57RtmNgsoiBuWhoWISBxlk1z/IazW8Ytw/x3Ag2ZWAbSM8L5VwBIzWwRsI0ig3z34IDObBpwGDCTO7n4lcGX4+unAPxZiYg3ws4e3Mq+mjNOWzoo6FBER3P3Xo7y+m6ATJfbKkkWkihNapVFEYiWb5PqjBAn1yQRDPX4M3ObuDrxhuDe5e6+ZXQbcAxQB17v7ejO7NHz92vDQtwH3unvbof8Y8fT8i238cfOL/P2bllKUGGqUjIhINMzsLl4+VG8fsBr4T3fvnPioxsbMwlUalVyLSHyMmlyHSfSt4WNM3P1u4O5BbdcO2r8BuGGEczwAPDDWa8fBTau2UpQw/upVmsgoIrGzBajj4JC8vwJ2AUuB7zHKOgZxUVue0rAQEYmVbErxvQb4JnA0kCLohW5z94KvPZ1P3b1pbl3dyJnLZjG7ujTqcEREBlvu7qdm7N9lZg+6+6lmVlAVQzShUUTiJJtqId8iqEH9DFAGfJgg2ZYR3LN+J3vbujWRUUTiqs7MBm5Q4fbMcLdgstXaCg0LEZF4yWbMNe6+2cyK3L0P+KGZ/W+e4yp4P3t4K/Nryzh1SfzKA4qIAP8A/NHMniWYT7MI+LtwsvqPIo1sDGrKU1r+XERiJZvkut3MUsBaM/sasAOoyG9YhW3LngP8actePvWWo0hoIqOIxJC7321mS4BlBMn10xmTGK+OLLAxqi1P0tLRg7tjpvutiEQvm2Eh7w2PuwxoI6hd/Y58BlXobnpkK8UJ410r50cdiojISJYARwHHA39pZu+LOJ4xqy1P0Zd29nf2Rh2KiAiQXbWQF8Ke64XA7cBGd9cAt2F09vRx66ONvOmY2cyq0kRGEYknM/sccDrBqrt3A2cDfyQot1owasIl0Fvau5lWlow4GhGRLHquzewc4Fng/xFMbtxsZmfnO7BCdc/6nTS392gio4jE3TuBM4Gd7v4B4ASgJNqQxm56RZBQN6liiIjERDZjrr8OvMHdNwOY2WLg18B/5zOwQnXjw1upn17OyYtnjn6wiEh0Otw9bWa9ZlZNsCrjEVEHNVYHe641qVFE4iGbMde7+xPr0BYKZGncibZ5dyuPPNfERSfVayKjiMTdajOrIVgw5lHgMeCRSCM6BLVhcq1yfCISF8P2XJvZ28PN9WZ2N3ALwVK57wJWTUBsBednDzeQLNJERhGJP3f/u3DzWjP7DVDt7k9EGdOhqC0PhoVolUYRiYuRhoWcl7G9Czgt3N4D1OYtogLV2dPHbY818uZj5zCzsuCGLYrIFBR2oryeoOPkj0DBJdfVpUkSFkxoFBGJg2GT63CCi2Tp7id3sK+jh/ecpImMIhJ/ZvYd4EjgprDpI2b2Rnf/6DjO+UmCVXwdeBL4QEbt7LxIJIya8pQmNIpIbGS1QmM/M3vM3VfkK5hCdsvqBhbNrOC1i2dEHYqISDZOA45zdwcwsx8RJMSHxMzmAR8HjnH3DjO7BbgQuCEHsY6opjypCY0iEhvZTGjMpFl6w9iwo5XXLZ6hFcJEpFBsBDK/alvA+IeFFANlZlYMlAPbx3m+rNSWpzShUURiY9jk2swuD59Pzmj+dd4jKkD7O3vY19HDgunlUYciIjIiM7vLzO4EZgAbzOwBM3sA2ADUHep53X0b8O/AVmAHsM/d7x3i+peY2WozW71nz55DvdxL1JYnNaFRRGJjpGEhHwCuAb4JrABw93+ZiKAKzbbmDgAW1Cq5FpHY+/d8nNTMaoELgEVAC/ALM7vY3X+aeZy7XwdcB7By5UrPxbVrylOs374/F6cSERm3kZLrDWb2PFBnZplfFRrg7n58XiMrIA1N7QDMry2LOBIRkZG5++8Ht5nZue7+q3Ge+o3Ac+6+Jzzn7cDrgJ+O+K4cCHquNSxEROJhpGohF5nZHOAe4PyJC6nwNIY910quRaRAfREYb3K9FXiNmZUDHQRLq68eb2DZqK1I0dmTpqO7j7JU0URcUkRkWCNOaHT3ne5+AsH4uarwsd3dX5iI4ApFY3MH5akiplekog5FRORQjHsmtrs/DNxKsNLjkwT/v1w33vNmQ6s0ikicjFotxMxOA54Bvg18B9hkZqfmO7BC0tDczvzaMlUKEZHYM7P7wuevZjR/JBfndvfPufsydz/O3d/r7l25OO9oDq7SqORaRKKXTZ3rbwBvdveNAGa2lGDRgVfmM7BC0tjcwXxNZhSRwjA37DQ538xuJui17jWz/onrj0Ua3SGoCXuuVetaROIgm+Q62Z9YA7j7JjNL5jGmgtPY3M5JC7UivIgUhM8CVwDzCTpPMjlwxoRHNE4aFiIicZJNcr3azH4A/CTcfw/waP5CKiz72nto7exVz7WIFAR3vxW41cw+4+5fijqeXKit6B8Wop5rEYleNsn13wIfJVjW1oAHCcZeC8F4a1ClEBEpLO7+JTM7H+ifQ/NADsrxRaKmLOy5blPPtYhEb9TkOpyQ8g3gG2Y219135D+swtFfhk+rM4pIITGzLwMnATeGTZeb2cnufmWEYR2SVHGCypJiDQsRkVjIpuc6068JV2uUQKN6rkWkMJ0DnOjuaQAz+xGwBii45BqgpjypCY0iEgujluIbRLXmBmls7qCypJhpZZrjKSIFpyZje1pUQeRCbXlKPdciEgtj7bn+Xl6iKGCNqnEtIoXpy8AaM7ufoOPkVAq01xqCnmtNaBSROMhmEZn+KiG4+3cGt011DU2qcS0ihcfdbwJeA9wePl7r7jdHG9Whm16R0oRGEYmFbIaFHJu5Y2ZFZLmAjJmdZWYbzWyzmV0xxOufMrO14WOdmfWZ2XQzW2Bm95vZBjNbb2aXZ/fjTCx3H+i5FhEpNO6+w93vBGa7+86o4xkPDQsRkbgYNrk2syvNrBU43sz2h49WYDdwx2gnDpPwbwNnA8cAF5nZMZnHuPtV7n6iu59I8HXk7929CegF/sHdjyboWfno4PfGQUt7D23dfaoUIiKF7tKoAxivmvIkrZ299Palow5FRKa4YZNrd/+yu1cBV7l7dfiocvcZWZZqOgnY7O5b3L0buBm4YITjLyJYVr2/N+WxcLsV2ADMy/JnmjCqcS0ik0TBTxrpX6WxpUPjrkUkWtlMaPxvMzt1cKO7PzjK++YBDRn7jcCrhzrQzMqBs4DLhnhtIbAceHiY914CXAJQX18/Ski51V/jWsm1iBQaM1vk7s+Fu+cN0VZQasqDik0t7d3MrCyJOBoRmcqySa4/lbFdStAj/ShwxijvG6onxIc59jzgoXBIyMETmFUCtwGfcPf9Q73R3a8DrgNYuXLlcOfPi4M1rjUsREQKzm2E6xa4e2PYditZzqmJm/6ea1UMEZGoZbNC43mZ+2a2APhaFuduBBZk7M8Htg9z7IWEQ0IyrpMkuPnf6O63Z3G9CdfQ1EF1qWpci0jhMLNlBBPVp5nZ2zNeqiboQClI0yuC5LpJFUNEJGJjrXMNQdJ8XBbHrQKWmNkiYBtBAv3uwQeZ2TTgNODijDYDfgBscPdvHEKMEyKoFKJeaxEpKEcB5xIsIJPZedIK/E0UAeVC5rAQEZEojZpcm9k3OTicIwGcCDw+2vvcvdfMLgPuAYqA6919vZldGr5+bXjo24B73b0t4+0nA+8FnjSztWHb/3H3u0f9iSZQY3MHR9RVRB2GiEjW3P0O4A4ze627/ynqeHJFw0JEJC6y6blenbHdC9zk7g9lc/IwGb57UNu1g/ZvAG4Y1PZHYj57Pahx3cGpS+uiDkVE5FBcYmYv66l29w9GEcx4laeKSBUlVOtaRCKXzZjrH5lZClgaNm3Mb0iFYW9bNx09faoUIiKF6lcZ26UE3yIONy8m9syMmvIkLW3quRaRaGUzLOR04EfA8wS9yQvM7P1ZlOKb1PrL8C3QmGsRKUDuflvmvpndBPwuonByYnpFiib1XItIxLIZFvJ14M3uvhHAzJYSVPYoyHJNudLQFJbhm66eaxGZFJYAE7tYQI7VlCc1oVFEIpdNcp3sT6wB3H1TWCZvSju4gIx6rkWk8JhZK8FkdQufdwKfjjSocaotT/HM7gNRhyEiU1xWExrN7AfAT8L99xAsIjOlNTa3U1uepLLkUKoZiohEy92roo4h12rKU+q5FpHIZZMZ/i3wUeDjBD0cDwLfyWdQhaChuUO91iJS0MzsfODUcPcBd//VSMfHXW15kpb2HtydYLkEEZGJl021kC7gG+FDQo3N7SybM+k6fkRkijCzrwCvAm4Mmy43s5Pd/coIwxqX2vIUvWmntauX6tIpP3pRRCKSGO4FM7vLzM4bany1mR1hZl80s4Kshzpe7s429VyLSGH7C+BN7n69u18PnAWcE3FM41IbLoHerCXQRSRCwybXBMvgngI8bWarzOxuM/sfM3sO+E/g0fCGPOXsae2iqzetGtciUuhqMranjfdkZlZjZrea2dNmtsHMXjvec45FbbgEulZpFJEoDTssxN13Av8E/JOZLQTmAh3AJndvn5jw4qlBNa5FpPB9GVhjZvcTzKc5FRjvkJBrgN+4+zvDxccm9CZZM7AEunquRSQ6WZW6cPfnCRaREYLx1oB6rkWkYLn7TWb2AMG4awM+HXaqHBIzqyZI0P86PH83MKFZbn/PtSqGiEiURhoWIsPor3E9T8m1iBSY8JtIANx9h7vf6e539CfWFph/CKc+AtgD/NDM1pjZ982sYojrX2Jmq81s9Z49ew71xxhSbX/PtZZAF5EIKbk+BI3N7cysTFGeUo1rESk4V5nZbWb2PjM71sxmmVm9mZ1hZl8CHgKOPoTzFgMrgO+6+3KgDbhi8EHufp27r3T3lXV1deP6QQarLkuSMPVci0i0Rs0Ozexc4G53T09APAWhsbmDeRpvLSIFyN3fZWbHECwI9kGC+TTtwAbgbuBf3b3zEE7dCDS6+8Ph/q0MkVznU1HCmFaWpEnJtYhEKJuu1wuBa8zsNuCH7r4hzzHFXkNTO8fOG/fEehGRSLj7U8A/5/icO82swcyOcveNwJnAU7m8RjZqy1OqFiIikRp1WIi7XwwsB54lGEv3p3DM3JRcQSWddra1dKhSiIjIy30MuNHMngBOBP5togOoKU9qWIiIRCqrMdfuvh+4DbiZ4CvEtwGPmdnH8hhbLO1u7aKnz1UpRERkEHdfG46nPt7d3+ruzRMdQ215ShMaRSRSoybX4SqNvwT+B0gCJ7n72cAJwD/mOb7YaVAZPhGR2KopT6nnWkQilc2Y63cB/+HuD2Y2unv7VFz+vL/G9YLpGhYiIoXLzE4G1rp7m5ldTFDp4xp3fyHi0MZleoUmNIpItLIZFvI54JH+HTMr66+T6u735Smu2GpoCmtc16jnWkQK2neBdjM7gWA13heAH0cb0vjVlKfo7EnT2dMXdSgiMkVlk1z/Asgsw9cXtk1Jjc3t1FWVUJosijoUEZHx6HV3By4g6LG+Bij4ieq1WgJdRCKWTXJdHC5jCwwsaZvKX0jx1tjcwQKNtxaRwtdqZlcCFwO/NrMignk1Ba1/CXRNahSRqGSTXO8xs/P7d8zsAuDF/IUUbw3N7cxXGT4RKXx/BXQBHwqXPp8HXBVtSONXE/Zca1KjiEQlmwmNlxLULf0WYEAD8L68RhVTvX1pdrR0ct7x6rkWkYLXSjAcpM/MlgLLgJsijmncaivCnmstJCMiERk1uXb3Z4HXmFklYO7emv+w4mlXaxe9aVelEBGZDB4ETjGzWuA+YDVBb/Z7Io1qnKaHPdeqGCIiUcmm5xozOwc4Fig1MwDc/Yt5jCuWGppU41pEJg0LS6p+CPimu3/NzNZGHdR4DQwLaVNyLSLRyGYRmWsJejM+RjAs5F3A4XmOK5Yam4MyfBpzLSKTgJnZawl6qn8dthV8GaRUcYKKVJGGhYhIZLKZ0Pg6d38f0OzuXwBeCyzIb1jx1NjcjhkcVlMadSgiIuP1CeBK4Jfuvt7MjgDujzak3KitSLG3rSvqMERkispmWEhn+NxuZocBe4FF+QspvhqaOphdVUpJccF37ojIFOfuvwd+b2ZVZlbp7luAj0cdVy4smVXJU9v3Rx2GiExR2fRc32VmNQQlmh4DnifLGeVmdpaZbTSzzWZ2xRCvf8rM1oaPdWbWZ2bTs3lvFBqb2zXeWkQmBTN7hZmtAdYBT5nZo2Z2bNRx5cKK+lqe2X2AfR0aGiIiE2/E5NrMEsB97t7i7rcRjLVe5u6fHe3E4YIE3wbOBo4BLjKzYzKPcfer3P1Edz+R4OvJ37t7UzbvjUJjc4cqhYjIZPGfwN+7++HuXg/8A/C9iGPKieX1tQA83tASbSAiMiWNmFy7exr4esZ+l7vvy/LcJwGb3X1LuKrjzQTL7A7nIg72iI/1vXnX05dmx74O9VyLyGRR4e4DY6zd/QGgIrpwcueEBdMwg8e2NkcdiohMQdkMC7nXzN5h/TX4sjePYMGZfo1h28uYWTlwFnDbIbz3EjNbbWar9+zZM8YQs7dzXydpVxk+EZk0tpjZZ8xsYfj4F+C5qIPKharSJEfNrmLN1paoQxGRKSib5PrvgV8AXWa238xazSybmSJDJeM+zLHnAQ+5e9NY3+vu17n7SndfWVdXl0VYh6ahOahxvUBl+ERkcvggUAfcDvwy3P5ApBHl0PL6GtZsbSadHu6/HRGR/MhmhcaqQzx3Iy8t2Tcf2D7MsRfy0kmSY3nvhGhsUo1rEZk83L2ZSVIdZCjL62u56ZEGtrzYxpGzKqMOR0SmkFGTazM7dah2d39wlLeuApaY2SJgG0EC/e4hzj8NOA24eKzvnUiNze0kDOaqxrWIFDAzu4vhv0XE3c+fwHDyZkV9DRCMu1ZyLSITKZs615/K2C4lmGz4KHDGSG9y914zuwy4h2DVr+vDhQouDV+/Njz0bcC97t422nuz/JnyorG5g7nTykgWZTOSRkQktv496gAmwhEzK6kuLWbN1mb+cuWUXPdMRCKSzbCQ8zL3zWwB8LVsTu7udwN3D2q7dtD+DcAN2bw3Sg3N7czTZEYRKXDh4jGTXiJhnFhfq0mNIjLhDqUbthE4LteBxF1js8rwicjkYWZPmtkTgx5/MLP/MLMZUceXCyvqa9i4q5XWTi0mIyITJ5sx19/k4Pi8BHAi8HgeY4qd7t40O/d3qlKIiEwm/w30AT8L9y8kqNS0j+DbxPOGflvhWF5fizs80biPk4+cGXU4IjJFZDPmenXGdi9wk7s/lKd4Yml7SweuGtciMrmc7O4nZ+w/aWYPufvJZnbxsO8qICcuqAHgsRealVyLyITJJrm+Feh09z4IljU3s3J3b89vaPHR2KwyfCIy6VSa2avd/WEAMzsJ6C+r0RtdWLkzrSzJkbMqWaNl0EVkAmUz5vo+ILPLtgz4XX7CiafG/gVkpqvnWkQmjQ8D3zez58zseeD7wIfNrAL4cqSR5dCKcDEZdy0mIyITI5vkutTdD/TvhNtTqgu3obmdooQxp1o1rkVkcnD3Ve7+CoJ5NCe6+/FhW5u733Ko5w2/3VxjZr/KWbDjsKK+lub2Hp7fO2W+bBWRiGWTXLeZ2Yr+HTN7JdCRv5DiJ6hxXUqxalyLyCRhZtPM7BsE307+zsy+Hi7qNV6XAxtycJ6cWF5fCwTjrkVEJkI22eIngF+EJZr+APwcuCyvUcVMY3OHKoWIyGRzPdAK/GX42A/8cDwnNLP5wDkEQ0xiYcmsSqpKinlsq5JrEZkY2Swis8rMlgFHEZRpetrdp1TR0Iamdk5bWhd1GCIiubTY3d+Rsf8FM1s7znNeDfwTUDXcAWZ2CXAJQH19/TgvN7pEwjhhQY0WkxGRCTNqz7WZfRSocPd17v4kwQzzv8t/aPHQ2dPH7tYuVQoRkcmmw8xe379jZiczjiF/ZnYusNvdHx3pOHe/zt1XuvvKurqJ6bRYUV/D0zv309Y1KYqgiEjMZTMs5G/cvaV/x92bgb/JW0Qx8/zeNgAWzlRyLSKTyqXAt83s+bBayLeAj4zjfCcD54fnuhk4w8x+Ou4oc2B5fS3pcDEZEZF8yya5TpiZ9e+YWRGQyl9I8bJxZysAR80Z9ltOEZGC4+6Pu/sJwPHA8e6+HDhjHOe70t3nu/tCgtUe/8fdY7EYzcBiMhp3LSITIJvk+h7gFjM708zOAG4CfpPfsOJj065WihPGETMrRz9YRKTAuPt+d98f7v59pMHkSW1FiiNmVmjctYhMiGxWaPw0weSTvyWY0Hgv8L18BhUnG3ceYNHMClLFKsMnIpOejX7I6Nz9AeCBXJwrV5bX1/LAxt24OxlfxoqI5NyoGaO7p939Wnd/ZzizfD3wzfyHFg+bdrWyVENCRGRqmLTLGC6vr2FvWzcNTVNqmQYRiUBW3bFmdqKZfTWcqPIl4Om8RhUT7d29bG1q56jZSq5FZHIws1Yz2z/EoxU4LOr48mVF/2IyGnctInk27LAQM1tKMCnlImAvweIx5u5vmKDYIvfMrmDV96VKrkVkknD3KXlDO2pOFeWpItZsbeaty+dFHY6ITGIjjbl+GvgDcJ67bwYws09OSFQxsXGXKoWIiEwGRQnjhPk1PKZJjSKSZyMNC3kHsBO438y+Z2ZnkqPJLoVi085WSooT1E9XjWsRkUK34vAaNuzYT0d3X9ShiMgkNmxy7e6/dPe/ApYRzPr+JDDbzL5rZm+eoPgitXFXK0tmV1KUmFK/U4iITErLF9TSm3ae3KbFZEQkf7KpFtLm7je6+7nAfGAtcEW+A4uDTbtaNd5aRGSSWF5fA2hSo4jk15iKN7t7k7v/p7sf8ipehaKlvZtd+7tUKUREZJKYUVnC4TPKWaPkWkTySCujDGNTf6UQTWYUEZk0VtTX8tjWFtwnbUlvEYmYkuthbOqvFKKeaxGRSWN5fQ17WrvY1qLFZEQkP5RcD2PTrlaqSoqZO6006lBERCRHDi4m0xJtICIyaSm5HsbGncGy52aqFCIiMlkcNaeK0mRC465FJG+UXA/B3VUpRERkEkoWJThei8mISB4puR7CngNdNLf3cNTsyqhDERGRHFtRX8tT2/fR2aPFZEQk95RcD2HTTlUKERGZrJbX19DT56zfrsVkRCT38ppcm9lZZrbRzDab2ZALz5jZ6Wa21szWm9nvM9o/GbatM7ObzGzCZhZuVKUQEZFJa2AxmRdaIo1DRCanvCXXZlYEfBs4GzgGuMjMjhl0TA3wHeB8dz8WeFfYPg/4OLDS3Y8DioAL8xXrYJt2tjKzMsWMypKJuqSIiEyQWVWlzK8tY02DJjWKSO7ls+f6JGCzu29x927gZuCCQce8G7jd3bcCuPvujNeKgTIzKwbKge15jPUlNmoyo4jIpPbKw2v532f3sqe1K+pQRGSSyWdyPQ9oyNhvDNsyLQVqzewBM3vUzN4H4O7bgH8HtgI7gH3ufu9QFzGzS8xstZmt3rNnz7iDTqedZ5Rci4hMan97+mI6uvu4/OY19KW1WqOI5E4+k+uhCkQPvoMVA68EzgHeAnzGzJaaWS1BL/ci4DCgwswuHuoi7n6du69095V1dXXjDnpbSwdt3X1KrkVEJrFlc6r5v289jv99di9X/25T1OGIyCRSnMdzNwILMvbn8/KhHY3Ai+7eBrSZ2YPACeFrz7n7HgAzux14HfDTPMYLZCx7Pkdl+EREJrN3rVzA6ueb+eb/bGbF4bW84ahZUYckIpNAPnuuVwFLzGyRmaUIJiTeOeiYO4BTzKzYzMqBVwMbCIaDvMbMyi1YIvHMsD3v+iuFLFHPtYjIpPeFC47l6LnVfPLna2lsbo86HBGZBPKWXLt7L3AZcA9BYnyLu683s0vN7NLwmA3Ab4AngEeA77v7Ond/GLgVeAx4MozzunzFmmnTzlYOm1ZKdWlyIi4nIiIRKk0W8Z33rKCvz/noz9bQ3ZuOOiQRKXB5rXPt7ne7+1J3X+zu/xq2Xevu12Ycc5W7H+Pux7n71Rntn3P3ZWH7e919QqZ0b9x1QIvHiIhMIYtmVnDVu47n8YYW/u3uCfmSVEQmMa3QmKG3L82zuw9o8RgRkSnmrOPm8uHXL+KG/32eux6fsMqvIjIJKbnO8Pzedrr70qoUIiIyBX367GW88vBarrjtCTbvPhB1OCJSoJRcZzhYKUTJtYjIWJjZAjO738w2mNl6M7s86pjGKlmU4FvvXk5Jsoi/u/FR2rt7ow5JRAqQkusMG3e2YgZHzlIZPhGRMeoF/sHdjwZeA3zUzI6JOKYxmzutjGsuPJFndh/gX365DnctMCMiY6PkOsOmXa0snFFBabIo6lBERAqKu+9w98fC7VaCKlGDV+UtCKcsqePyM5dw+5pt3LyqYfQ3iIhkyOciMgVn465Wls5Wr7WIyHiY2UJgOfDwEK9dAlwCUF9fP7GBjcHHzljCoy8087k71uMOF520gGDZBRGRkannOtTZ08fzL7apUoiIyDiYWSVwG/AJd98/+HV3v87dV7r7yrq6uokPMEtFCeP/XbickxZN5//88kn+5serefHAhFSEFZECp+Q69OyeA6Qd1bgWETlEZpYkSKxvdPfbo45nvGorUvz4gyfx2XOP4cFnXuSsqx/k/qd3Rx2WiMSckuvQQKUQ9VyLiIyZBWMmfgBscPdvRB1PriQSxgdfv4g7LzuZmZUlfOCGVXzmv9bR0d0XdWgiElNKrkMbdx4gWWQsnFkRdSgiIoXoZOC9wBlmtjZ8/EXUQeXKsjnV/NdHT+ZvTlnET/78Aud+8w+s27Yv6rBEJIaUXIc27WplcV0lySJ9JCIiY+Xuf3R3c/fj3f3E8HF31HHlUmmyiH8+5xhu/PCraevq463ffojvPLCZvrTK9YnIQcokQxt3tmplRhERGdXJR87kN584hbccO4ev/WYjF133Z9ZsbVZNbBEBlFwD0NrZw7aWDq3MKCIiWakpT/Gtdy/nG395Aht27Odt3/lfzrr6D/zgj8/R1NYddXgiEiHVuQae2X0AQD3XIiKSNTPj7Svm86ZjZvOrJ3Zw86oGvvSrp/jqfz/Nm46dzV+tXMDrj5xJIqH62CJTiZJrYNNOVQoREZFDU1Wa5KKT6rnopHqe3rmfn69q4JdrtvHrJ3Ywr6aMd62cz7tWLmBeTVnUoYrIBFByTbAyY1myiPm1uvGJiMihWzanms+ddyxXnL2Me9fv4pbVDVz9u2e45r5nOH5+DSfOn8bx82s4YUENR8ysUK+2yCSk5JqgUsjS2ZW6yYmISE6UFBdx3gmHcd4Jh9HQ1M6tjzby5y17+cWjjfzoTy8AUFVSzCvmT+OEBTWcED7PqS7VMusiBU7JNUGN6zccFd9leEVEpHAtmF7OJ9+0FIC+tPPsngOsbWjh8YYWnmjcx/ce3EJvWM5vZmUJy+ZUsWxOFUfNqeLoudUcOauS0mRRlD+CiIzBlE+u9x7o4sUDXaoUIiIieVeUMJbOrmLp7Cr+cuUCADp7+tiwYz+PN7Swbvt+Nu5s5Sd/foGu3jQACYNFMytYNreaZbOrWDqnigW15cyrKaO6rFg93SIxM+WT6027VClERESiU5osYnl9Lcvrawfa+tLO83vbeHpHK0/v3M/TO1t5orGFXz+x4yXvrUgVMa+2jMNqgse8/kdtGbOrSqmrKqEspV5vkYmk5HpXWClEPdciIhITRQljcV0li+sqOef4uQPtB7p62bz7ANtbOtjW3MG2lg62t3SwfV8HTzTuG7LGdlVJMXXVJcyqKmFWmHDPqiphVnUJ0ytKmFaWpKYsybSyJNVlSYo0/0hkXKZ8cr1xVyvTypLMqiqJOhQREZERVZYUc+KCGk5cUDPk6x3dfWzfFyTeu1u72N3aye79XewJt59obGF3axft3X3DXqOqtJhpYbLd/6guTVJVWkxVaZLqsuC5qrR4oP3g68UUF2l9OpnapnxyvWlnK0fNrtKYNRERKXhlqaKBHu+RHOjqZff+Tprbu9nX0cO+jh5a2g8+7+9v6+jhmd0HaO3sobWzd8SkfCCGZBGVYaJdVZqkqqR4IPEuSxZRkiyipDgRPoooSR7cLk0mSPW3Fw+1naAkWUSqKEGyyPR/t8TSlE6u3Z2Nu1q54MTDog5FRERkwlSWFFM5SgI+lJ6+NAc6e9kfJtv7O3rYH+4f6OyltbOXA13Ba62dvbR29dLa2cPO/Z20dvbQ1ZumqydNZ28f7uP7GcwgVXQw6U4VHUy8DybiB5P20oFEPnjO3E8VJyjJPFf/Y4hz9rf3H1OcUJIvLzWlk+vgH3uvVmYUERHJQrIoQW1FitqK1LjO4+709DldvX1Bwt2bpqunj86eNN19wXbwHLzW3dd3cLs3TVdvX/jc/3p6YL8747XOnjT7Onro7Ane09mTprOnb+A8uTCQ5BclwCBhRsLAwmcInvvbkwNJe2aSHiTw/cl7UcIoMiORCM5TlHHOokR4vvCYYD94PrjNQJuREUv/eTDMyIjz4HsSmdfIOH8iI6aB6w6KIZF46bkZ2O6PI9zOOPfAdnhtS2TsZ1yrkH6BmdLJ9cZw2XNVChEREZk4Zkaq2EgVJ4jqf+B02g8m631BQn4wQc9M2PuGbO8evN+XHuiNT7uTdscd0h78MuEOfe70hL80ZJ5nX0dPuB38UtHX56T94Hn6t/vS4XnSTp876fB5vN8CFAIzwoQ7M6EPfsno/8Wl/5eDgV8aMpL9/l8yBlL0jLZbPvJapo/zF8ZMUzq5Lk8V88ajZ6tSiIiIyBSTSBhlqaKwVGEy6nDGxcPEO0i4gyS+Lx0k3x6+7gQJOv0JPwcTde9P3sOEPR22pzPOG1zj5e3ptA+0Bb9IHDx3Osz6+7cHnmHgOgO/QAza70sf/IVi6Gv2nzNsC3+JSafJiMUHrhf+6AOfl8NAQ64r5Ezp5PqkRdM5adH0qMMQEREROWRmRnGRTe2kLkZUL0dEREREJEfymlyb2VlmttHMNpvZFcMcc7qZrTWz9Wb2+4z2GjO71cyeNrMNZvbafMYqIiIiIjJeefsGwcyKgG8DbwIagVVmdqe7P5VxTA3wHeAsd99qZrMyTnEN8Bt3f6eZpYDyfMUqIiIiIpIL+ey5PgnY7O5b3L0buBm4YNAx7wZud/etAO6+G8DMqoFTgR+E7d3u3pLHWEVERERExi2fyfU8oCFjvzFsy7QUqDWzB8zsUTN7X9h+BLAH+KGZrTGz75tZRR5jFREREREZt3wm10PVNRlcibEYeCVwDvAW4DNmtjRsXwF8192XA23AcGO2LzGz1Wa2es+ePTkLXkRERERkrPKZXDcCCzL25wPbhzjmN+7e5u4vAg8CJ4Ttje7+cHjcrQTJ9su4+3XuvtLdV9bV1eX0BxARERERGYt8JtergCVmtiickHghcOegY+4ATjGzYjMrB14NbHD3nUCDmR0VHncm8BQiIiIiIjFmnsc1M83sL4CrgSLgenf/VzO7FMDdrw2P+RTwASANfN/drw7bTwS+D6SALcAH3L15lOvtAV4YQ4gzgRfHcHycFGrsinviFWrsUy3uw919Sn39dgj3bJh6fy+iVqhxQ+HGrrgnVs7v2XlNruPOzFa7+8qo4zgUhRq74p54hRq74pahFOrnq7gnXqHGrrgnVj7i1gqNIiIiIiI5ouRaRERERCRHpnpyfV3UAYxDocauuCdeocauuGUohfr5Ku6JV6ixK+6JlfO4p/SYaxERERGRXJrqPdciIiIiIjmj5FpEREREJEembHJtZmeZ2UYz22xmQy6tHhdm9ryZPWlma81sddg23cx+a2bPhM+1UccJYGbXm9luM1uX0TZsrGZ2ZfhnsNHM3hJN1MPG/Xkz2xZ+7mvDuu39r8Ul7gVmdr+ZbTCz9WZ2edge6898hLhj/ZmbWamZPWJmj4dxfyFsj/XnPRnonp0fumdPLN2zI4l94u/b7j7lHgSL2jwLHEGwSM3jwDFRxzVCvM8DMwe1fQ24Ity+Avhq1HGGsZxKsFT9utFiBY4JP/sSYFH4Z1IUo7g/D/zjEMfGKe65wIpwuwrYFMYX6898hLhj/ZkDBlSG20ngYeA1cf+8C/2he3ZeY9U9e2Lj1j174mOf8Pv2VO25PgnY7O5b3L0buBm4IOKYxuoC4Efh9o+At0YXykHu/iDQNKh5uFgvAG529y53fw7YTPBnM+GGiXs4cYp7h7s/Fm63AhuAecT8Mx8h7uHEJW539wPhbjJ8ODH/vCcB3bPzRPfsiaV79sSL4r49VZPreUBDxn4jI/8liZoD95rZo2Z2Sdg22913QPCXHpgVWXSjGy7WQvhzuMzMngi/guz/yiiWcZvZQmA5wW/lBfOZD4obYv6Zm1mRma0FdgO/dfeC+rwLVKF9jrpnRyfW949MumdPnIm+b0/V5NqGaItzTcKT3X0FcDbwUTM7NeqAciTufw7fBRYDJwI7gK+H7bGL28wqgduAT7j7/pEOHaItstiHiDv2n7m797n7icB84CQzO26Ew2MTd4ErtM9R9+xoxP7+0U/37Ik10fftqZpcNwILMvbnA9sjimVU7r49fN4N/JLg64ldZjYXIHzeHV2Eoxou1lj/Obj7rvAfZBr4Hge/FopV3GaWJLjZ3ejut4fNsf/Mh4q7UD5zAHdvAR4AzqIAPu8CV1Cfo+7Z0SiU+4fu2dGZqPv2VE2uVwFLzGyRmaWAC4E7I45pSGZWYWZV/dvAm4F1BPG+Pzzs/cAd0USYleFivRO40MxKzGwRsAR4JIL4htT/jy70NoLPHWIUt5kZ8ANgg7t/I+OlWH/mw8Ud98/czOrMrCbcLgPeCDxNzD/vSUD37IlVkH+f437/AN2zJyreTJHct8cy+3EyPYC/IJjt+izwz1HHM0KcRxDMWn0cWN8fKzADuA94JnyeHnWsYVw3EXw11EPw29+HRooV+Ofwz2AjcHbM4v4J8CTwRPiPbW4M4349wddVTwBrw8dfxP0zHyHuWH/mwPHAmjC+dcBnw/ZYf96T4aF7dt7i1T17YuPWPXviY5/w+7aWPxcRERERyZGpOixERERERCTnlFyLiIiIiOSIkmsRERERkRxRci0iIiIikiNKrkVEREREckTJtcSGmbmZfT1j/x/N7PM5OvcNZvbOXJxrlOu8y8w2mNn9g9oXmtm78319EZGJonu2yNCUXEucdAFvN7OZUQeSycyKxnD4h4C/c/c3DGpfCAx5ozaz4kMMTUQkSrpniwxBybXESS9wHfDJwS8M7sUwswPh8+lm9nszu8XMNpnZV8zsPWb2iJk9aWaLM07zRjP7Q3jcueH7i8zsKjNbZWZPmNlHMs57v5n9jKBA/uB4LgrPv87Mvhq2fZag0P61ZnbVoLd8BTjFzNaa2SfN7K/N7Bdmdhdwb7iq2/VhHGvM7IJR4ptrZg+G51tnZqcc4mcuInKodM/WPVuGoN++JG6+DTxhZl8bw3tOAI4GmoAtwPfd/SQzuxz4GPCJ8LiFwGnAYuB+MzsSeB+wz91fZWYlwENmdm94/EnAce7+XObFzOww4KvAK4FmghvtW939i2Z2BvCP7r56UIxXhO39/0H8NfBa4Hh3bzKzfwP+x90/aMEyrY+Y2e+A9wwT39uBe9z9X8NemvIxfF4iIrmie7bu2TKIkmuJFXffb2Y/Bj4OdGT5tlXuvgPAzJ4F+m+0TwKZX/Xd4u5p4Bkz2wIsA94MHJ/RwzINWAJ0A48MvkmHXgU84O57wmveCJwK/FeW8fb7rbs3hdtvBs43s38M90uB+hHiWwVcb2ZJ4L/cfe0Yry0iMm66Z+ueLS+n5Fri6GrgMeCHGW29hMOYzMyAVMZrXRnb6Yz9NC/9O+6DruOAAR9z93syXzCz04G2YeKzUeLPVub5DXiHu28cFMeQ8YWvnQqcA/zEzK5y9x/nKC4RkbG4Gt2z++PQPVs05lriJ+wZuIVgokm/5wm+0gO4AEgewqnfZWaJcEzfEcBG4B7gb8PeBMxsqZlVjHKeh4HTzGxm+PXeRcDvR3lPK1A1wuv3AB8Lb8yY2fKM9pfFZ2aHA7vd/XvAD4AVo1xfRCQvdM/WPVteSj3XEldfBy7L2P8ecIeZPQLcx/A9FCPZSHBDnQ1c6u6dZvZ9gnF9j4U3yT3AW0c6ibvvMLMrgfsJei/udvc7Rrn2E0CvmT0O3EAw7i/Tlwh6f54I43geOBcYLr7TgU+ZWQ9wgGAcoohIVHTP1j1bQuY++FsXERERERE5FBoWIiIiIiKSI0quRURERERyRMm1iIiIiEiOKLkWEREREckRJdciIiIiIjmi5FpEREREJEeUXIuIiIiI5Mj/B6st9eFuS1oUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "make_figure(model_rf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ddf69c5",
   "metadata": {},
   "source": [
    "## Random Forest con SMOTE\n",
    "\n",
    "Repetimos el procedimiento con los datos SMOTE. Esperamos obtener un mal rendimiento, al igual que en redes neuronales."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "fb7500bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from imblearn.over_sampling import SMOTE\n",
    "sm = SMOTE(random_state=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4304cc33",
   "metadata": {},
   "source": [
    "Convertimos los datos de las columnas categóricas a números para poder usar _SMOTE_:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "73fd6150",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wkc</th>\n",
       "      <th>wkr</th>\n",
       "      <th>wrc</th>\n",
       "      <th>wrr</th>\n",
       "      <th>bkc</th>\n",
       "      <th>bkr</th>\n",
       "      <th>opt rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28051</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28052</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28053</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28054</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28055</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28056 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       wkc  wkr  wrc  wrr  bkc  bkr  opt rank\n",
       "0        0    1    1    3    2    2         0\n",
       "1        0    1    2    1    2    2         0\n",
       "2        0    1    2    1    3    1         0\n",
       "3        0    1    2    1    3    2         0\n",
       "4        0    1    2    2    2    1         0\n",
       "...    ...  ...  ...  ...  ...  ...       ...\n",
       "28051    1    1    6    7    4    5        17\n",
       "28052    1    1    6    7    4    6        17\n",
       "28053    1    1    6    7    4    7        17\n",
       "28054    1    1    6    7    5    5        17\n",
       "28055    1    1    6    7    6    5        17\n",
       "\n",
       "[28056 rows x 7 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_columns = ['wkc', 'wrc', 'bkc']\n",
    "\n",
    "data[cat_columns]=data[cat_columns].astype(\"category\")\n",
    "data[cat_columns]=data[cat_columns].apply(lambda x: x.cat.codes)\n",
    "data.astype('int64')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21f4b9e0",
   "metadata": {},
   "source": [
    "Aplicamos _SMOTE_:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "80c3bae9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wkc</th>\n",
       "      <th>wkr</th>\n",
       "      <th>wrc</th>\n",
       "      <th>wrr</th>\n",
       "      <th>bkc</th>\n",
       "      <th>bkr</th>\n",
       "      <th>opt rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81949</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81950</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81951</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81952</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81953</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>81954 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       wkc  wkr  wrc  wrr  bkc  bkr  opt rank\n",
       "0        0    1    1    3    2    2         0\n",
       "1        0    1    2    1    2    2         0\n",
       "2        0    1    2    1    3    1         0\n",
       "3        0    1    2    1    3    2         0\n",
       "4        0    1    2    2    2    1         0\n",
       "...    ...  ...  ...  ...  ...  ...       ...\n",
       "81949    0    1    5    7    3    2        17\n",
       "81950    0    1    1    2    4    3        17\n",
       "81951    0    1    6    6    4    3        17\n",
       "81952    0    1    7    6    5    3        17\n",
       "81953    0    1    3    6    5    3        17\n",
       "\n",
       "[81954 rows x 7 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xsmote, ysmote = sm.fit_resample(data.drop(label, axis=1), data[label])\n",
    "data_smote = pd.concat([xsmote, ysmote], axis=1)\n",
    "data_smote"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "236c5e2c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65416 examples in training, 16538 examples for testing.\n"
     ]
    }
   ],
   "source": [
    "train_smote, test_smote = split_dataset(data_smote)\n",
    "print(\"{} examples in training, {} examples for testing.\".format(\n",
    "    len(train_smote), len(test_smote)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d2212b6",
   "metadata": {},
   "source": [
    "Creamos los datasets train y test para que pueda usarlos la librería. Notemos que los datos de las variables numéricas que hemos convertido se asignan a int8 automáticamente, a diferencia de las demás, que usan int64."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "f6519cd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_smote = tfdf.keras.pd_dataframe_to_tf_dataset(train_smote, label=label, )\n",
    "test_smote = tfdf.keras.pd_dataframe_to_tf_dataset(test_smote, label=label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "801d4459",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BatchDataset shapes: ({wkc: (None,), wkr: (None,), wrc: (None,), wrr: (None,), bkc: (None,), bkr: (None,)}, (None,)), types: ({wkc: tf.int8, wkr: tf.int64, wrc: tf.int8, wrr: tf.int64, bkc: tf.int8, bkr: tf.int64}, tf.int64)>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_smote"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "e8344971",
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "in user code:\n\n    /home/antonio/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:855 train_function  *\n        return step_function(self, iterator)\n    /home/antonio/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow_decision_forests/keras/core.py:646 train_step  *\n        normalized_semantic_inputs = tf_core.normalize_inputs(semantic_inputs)\n    /home/antonio/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow_decision_forests/tensorflow/core.py:255 normalize_inputs  *\n        raise ValueError(\n\n    ValueError: Non supported tensor dtype <dtype: 'int8'> for semantic Semantic.CATEGORICAL of feature wkc\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-25-18e6e4ad8e58>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      4\u001b[0m                 metrics=[\"accuracy\"])\n\u001b[1;32m      5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mmodel_1_smote\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrain_smote\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow_decision_forests/keras/core.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, callbacks, **kwargs)\u001b[0m\n\u001b[1;32m    769\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    770\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 771\u001b[0;31m       history = super(CoreModel, self).fit(\n\u001b[0m\u001b[1;32m    772\u001b[0m           x=x, y=y, epochs=1, callbacks=callbacks, **kwargs)\n\u001b[1;32m    773\u001b[0m     \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m   1181\u001b[0m                 _r=1):\n\u001b[1;32m   1182\u001b[0m               \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_train_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1183\u001b[0;31m               \u001b[0mtmp_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1184\u001b[0m               \u001b[0;32mif\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshould_sync\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1185\u001b[0m                 \u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masync_wait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    887\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    888\u001b[0m       \u001b[0;32mwith\u001b[0m \u001b[0mOptionalXlaContext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jit_compile\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 889\u001b[0;31m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    890\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    891\u001b[0m       \u001b[0mnew_tracing_count\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexperimental_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    931\u001b[0m       \u001b[0;31m# This is the first call of __call__, so we have to initialize.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    932\u001b[0m       \u001b[0minitializers\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 933\u001b[0;31m       \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_initialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0madd_initializers_to\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitializers\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    934\u001b[0m     \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    935\u001b[0m       \u001b[0;31m# At this point we know that the initialization is complete (or less\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_initialize\u001b[0;34m(self, args, kwds, add_initializers_to)\u001b[0m\n\u001b[1;32m    761\u001b[0m     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_graph_deleter\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mFunctionDeleter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lifted_initializer_graph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    762\u001b[0m     self._concrete_stateful_fn = (\n\u001b[0;32m--> 763\u001b[0;31m         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access\n\u001b[0m\u001b[1;32m    764\u001b[0m             *args, **kwds))\n\u001b[1;32m    765\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_get_concrete_function_internal_garbage_collected\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   3048\u001b[0m       \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3049\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3050\u001b[0;31m       \u001b[0mgraph_function\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maybe_define_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3051\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3052\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_maybe_define_function\u001b[0;34m(self, args, kwargs)\u001b[0m\n\u001b[1;32m   3442\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3443\u001b[0m           \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_function_cache\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmissed\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcall_context_key\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3444\u001b[0;31m           \u001b[0mgraph_function\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_graph_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3445\u001b[0m           \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_function_cache\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprimary\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mcache_key\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3446\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_create_graph_function\u001b[0;34m(self, args, kwargs, override_flat_arg_shapes)\u001b[0m\n\u001b[1;32m   3277\u001b[0m     \u001b[0marg_names\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbase_arg_names\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mmissing_arg_names\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3278\u001b[0m     graph_function = ConcreteFunction(\n\u001b[0;32m-> 3279\u001b[0;31m         func_graph_module.func_graph_from_py_func(\n\u001b[0m\u001b[1;32m   3280\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3281\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_python_function\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py\u001b[0m in \u001b[0;36mfunc_graph_from_py_func\u001b[0;34m(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)\u001b[0m\n\u001b[1;32m    997\u001b[0m         \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moriginal_func\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_decorator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munwrap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpython_func\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    998\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 999\u001b[0;31m       \u001b[0mfunc_outputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpython_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mfunc_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfunc_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1000\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1001\u001b[0m       \u001b[0;31m# invariant: `func_outputs` contains only Tensors, CompositeTensors,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36mwrapped_fn\u001b[0;34m(*args, **kwds)\u001b[0m\n\u001b[1;32m    670\u001b[0m         \u001b[0;31m# the function a weak reference to itself to avoid a reference cycle.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    671\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mOptionalXlaContext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcompile_with_xla\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 672\u001b[0;31m           \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mweak_wrapped_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__wrapped__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    673\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    674\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    984\u001b[0m           \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# pylint:disable=broad-except\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    985\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"ag_error_metadata\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 986\u001b[0;31m               \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mag_error_metadata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_exception\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    987\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    988\u001b[0m               \u001b[0;32mraise\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: in user code:\n\n    /home/antonio/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:855 train_function  *\n        return step_function(self, iterator)\n    /home/antonio/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow_decision_forests/keras/core.py:646 train_step  *\n        normalized_semantic_inputs = tf_core.normalize_inputs(semantic_inputs)\n    /home/antonio/anaconda3/envs/tf2.5/lib/python3.8/site-packages/tensorflow_decision_forests/tensorflow/core.py:255 normalize_inputs  *\n        raise ValueError(\n\n    ValueError: Non supported tensor dtype <dtype: 'int8'> for semantic Semantic.CATEGORICAL of feature wkc\n"
     ]
    }
   ],
   "source": [
    "model_1_smote = tfdf.keras.RandomForestModel()\n",
    "\n",
    "model_1_smote.compile(\n",
    "                metrics=[\"accuracy\"])\n",
    "\n",
    "model_1_smote.fit(x=train_smote)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e19c8dfd",
   "metadata": {},
   "source": [
    "Sorpresa. No somos capaces de crear el modelo ya que las variables deben estar codificadas en tf.int64. Pero al usar la función para crear el dataset, como hemos comentado antes, no es posible cambiar el tipo. Aunque definamos inicialmente todas las columnas como int64, la función las cambia a tf.int8.\n",
    "\n",
    "Este bug nos impide comprobar el rendimiento de SMOTE. Suponemos que se solucionará en versiones siguientes. El método con _one-hot_ encoding tampoco funcionará ya que también es convertido a tf.int8."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b64844e",
   "metadata": {},
   "source": [
    "## Datos raw con boosted trees\n",
    "\n",
    "El siguiente algoritmo que vamos a probar es _Gradient Boosted Trees_. En estos modelos, se crean un conjunto de árboles los cuales son mejorados iterativamente mediante _boosting_. A diferencia de _AdaBoost_, se mejoran las predicciones reduciendo el error con una función de coste.\n",
    "\n",
    "Este algoritmo permite modificar ciertos parámetros, por los que realizaremos distintas pruebas y ver su rendimiento."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "3a12579e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "351/351 [==============================] - 0s 1ms/step\n",
      "Model: \"gradient_boosted_trees_model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Total params: 1\n",
      "Trainable params: 0\n",
      "Non-trainable params: 1\n",
      "_________________________________________________________________\n",
      "Type: \"GRADIENT_BOOSTED_TREES\"\n",
      "Task: CLASSIFICATION\n",
      "Label: \"__LABEL\"\n",
      "\n",
      "Input Features (6):\n",
      "\tbkc\n",
      "\tbkr\n",
      "\twkc\n",
      "\twkr\n",
      "\twrc\n",
      "\twrr\n",
      "\n",
      "No weights\n",
      "\n",
      "Variable Importance: NUM_NODES:\n",
      "    1. \"wrr\" 50164.000000 ################\n",
      "    2. \"wrc\" 46244.000000 #############\n",
      "    3. \"bkc\" 39981.000000 ##########\n",
      "    4. \"bkr\" 37646.000000 #########\n",
      "    5. \"wkr\" 27136.000000 ###\n",
      "    6. \"wkc\" 21299.000000 \n",
      "\n",
      "Variable Importance: NUM_AS_ROOT:\n",
      "    1. \"bkr\" 2894.000000 ################\n",
      "    2. \"bkc\" 1723.000000 ########\n",
      "    3. \"wkc\" 980.000000 ###\n",
      "    4. \"wkr\" 932.000000 ###\n",
      "    5. \"wrr\" 499.000000 \n",
      "    6. \"wrc\" 388.000000 \n",
      "\n",
      "Variable Importance: SUM_SCORE:\n",
      "    1. \"bkr\" 9447.080979 ################\n",
      "    2. \"wrr\" 9369.968841 ###############\n",
      "    3. \"bkc\" 9339.258444 ###############\n",
      "    4. \"wrc\" 8868.002580 #############\n",
      "    5. \"wkr\" 6195.950017 ####\n",
      "    6. \"wkc\" 4883.574089 \n",
      "\n",
      "Variable Importance: MEAN_MIN_DEPTH:\n",
      "    1. \"__LABEL\"  6.596429 ################\n",
      "    2.     \"wkr\"  3.999949 #######\n",
      "    3.     \"wkc\"  3.672179 ######\n",
      "    4.     \"wrr\"  3.451561 #####\n",
      "    5.     \"wrc\"  3.013544 ####\n",
      "    6.     \"bkc\"  2.044103 \n",
      "    7.     \"bkr\"  1.752188 \n",
      "\n",
      "\n",
      "\n",
      "Loss: MULTINOMIAL_LOG_LIKELIHOOD\n",
      "Validation loss value: 0.398818\n",
      "Number of trees per iteration: 18\n",
      "Number of trees: 7416\n",
      "Total number of nodes: 452356\n",
      "\n",
      "Number of nodes by tree:\n",
      "Count: 7416 Average: 60.9973 StdDev: 0.232229\n",
      "Min: 41 Max: 61 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 41, 42)    1   0.01%   0.01%\n",
      "[ 42, 43)    0   0.00%   0.01%\n",
      "[ 43, 44)    0   0.00%   0.01%\n",
      "[ 44, 45)    0   0.00%   0.01%\n",
      "[ 45, 46)    0   0.00%   0.01%\n",
      "[ 46, 47)    0   0.00%   0.01%\n",
      "[ 47, 48)    0   0.00%   0.01%\n",
      "[ 48, 49)    0   0.00%   0.01%\n",
      "[ 49, 50)    0   0.00%   0.01%\n",
      "[ 50, 51)    0   0.00%   0.01%\n",
      "[ 51, 52)    0   0.00%   0.01%\n",
      "[ 52, 53)    0   0.00%   0.01%\n",
      "[ 53, 54)    0   0.00%   0.01%\n",
      "[ 54, 55)    0   0.00%   0.01%\n",
      "[ 55, 56)    0   0.00%   0.01%\n",
      "[ 56, 57)    0   0.00%   0.01%\n",
      "[ 57, 58)    0   0.00%   0.01%\n",
      "[ 58, 59)    0   0.00%   0.01%\n",
      "[ 59, 60)    0   0.00%   0.01%\n",
      "[ 60, 61] 7415  99.99% 100.00% ##########\n",
      "\n",
      "Depth by leafs:\n",
      "Count: 229886 Average: 6.59651 StdDev: 1.8032\n",
      "Min: 1 Max: 8 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 1, 2)   4180   1.82%   1.82%\n",
      "[ 2, 3)   6308   2.74%   4.56% #\n",
      "[ 3, 4)   8914   3.88%   8.44% #\n",
      "[ 4, 5)  13219   5.75%  14.19% #\n",
      "[ 5, 6)  19659   8.55%  22.74% ##\n",
      "[ 6, 7)  29002  12.62%  35.36% ###\n",
      "[ 7, 8)  41108  17.88%  53.24% ####\n",
      "[ 8, 8] 107496  46.76% 100.00% ##########\n",
      "\n",
      "Number of training obs by leaf:\n",
      "Count: 229886 Average: 0 StdDev: 0\n",
      "Min: 0 Max: 0 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 0, 0] 229886 100.00% 100.00% ##########\n",
      "\n",
      "Attribute in nodes:\n",
      "\t50164 : wrr [NUMERICAL]\n",
      "\t46244 : wrc [CATEGORICAL]\n",
      "\t39981 : bkc [CATEGORICAL]\n",
      "\t37646 : bkr [NUMERICAL]\n",
      "\t27136 : wkr [NUMERICAL]\n",
      "\t21299 : wkc [CATEGORICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 0:\n",
      "\t2894 : bkr [NUMERICAL]\n",
      "\t1723 : bkc [CATEGORICAL]\n",
      "\t980 : wkc [CATEGORICAL]\n",
      "\t932 : wkr [NUMERICAL]\n",
      "\t499 : wrr [NUMERICAL]\n",
      "\t388 : wrc [CATEGORICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 1:\n",
      "\t5303 : bkr [NUMERICAL]\n",
      "\t4945 : bkc [CATEGORICAL]\n",
      "\t2146 : wrc [CATEGORICAL]\n",
      "\t2073 : wkc [CATEGORICAL]\n",
      "\t1903 : wrr [NUMERICAL]\n",
      "\t1698 : wkr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 2:\n",
      "\t8155 : bkc [CATEGORICAL]\n",
      "\t7709 : bkr [NUMERICAL]\n",
      "\t5435 : wrc [CATEGORICAL]\n",
      "\t4617 : wrr [NUMERICAL]\n",
      "\t4162 : wkc [CATEGORICAL]\n",
      "\t2986 : wkr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 3:\n",
      "\t12463 : bkc [CATEGORICAL]\n",
      "\t11071 : bkr [NUMERICAL]\n",
      "\t9659 : wrc [CATEGORICAL]\n",
      "\t9207 : wrr [NUMERICAL]\n",
      "\t6393 : wkc [CATEGORICAL]\n",
      "\t5349 : wkr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 5:\n",
      "\t26050 : wrr [NUMERICAL]\n",
      "\t24139 : wrc [CATEGORICAL]\n",
      "\t23218 : bkc [CATEGORICAL]\n",
      "\t20910 : bkr [NUMERICAL]\n",
      "\t13953 : wkr [NUMERICAL]\n",
      "\t13024 : wkc [CATEGORICAL]\n",
      "\n",
      "Condition type in nodes:\n",
      "\t114946 : HigherCondition\n",
      "\t107524 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 0:\n",
      "\t4325 : HigherCondition\n",
      "\t3091 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 1:\n",
      "\t9164 : ContainsBitmapCondition\n",
      "\t8904 : HigherCondition\n",
      "Condition type in nodes with depth <= 2:\n",
      "\t17752 : ContainsBitmapCondition\n",
      "\t15312 : HigherCondition\n",
      "Condition type in nodes with depth <= 3:\n",
      "\t28515 : ContainsBitmapCondition\n",
      "\t25627 : HigherCondition\n",
      "Condition type in nodes with depth <= 5:\n",
      "\t60913 : HigherCondition\n",
      "\t60381 : ContainsBitmapCondition\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model_bt_1 = tfdf.keras.GradientBoostedTreesModel(num_trees=500, \n",
    "                                                  growing_strategy=\"BEST_FIRST_GLOBAL\",\n",
    "                                                  max_depth=8)\n",
    "model_bt_1.fit(x=train)\n",
    "model_bt_1.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0fd34a8",
   "metadata": {},
   "source": [
    "Hemos creado y entrenado un modelo con 300 árboles y una profundidad máxima de 8. Veamos las métricas obtenidas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "57b664c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.8913663931501962,\n",
       " 0.8913663931501962,\n",
       " 0.8913663931501962,\n",
       " 0.8785774841048029]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = model_bt_1.predict(test)\n",
    "preds = np.argmax(preds, axis=1)\n",
    "\n",
    "metrics_bt_1= compute_metrics_multiclass(y_test_aux, preds)\n",
    "metrics_bt_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99e41871",
   "metadata": {},
   "source": [
    "Los resultados son mucho mejores. Obtenemos un 89% en Precision, Recall y F1. Hemos alcanzado los mejores resultados de las redes neuronales en el caso de dropout y 6 capas internas de 250 neuronas con un modelo mucho más rápido y simple de construir. Recordemos que los resultados son deterministas y que sólo es necesario una \"época\" a diferencia de las redes neuronales.\n",
    "\n",
    "Creamos otro modelo en el que aportamos más parámetros para el entrenamiento, los cuales se recomiendan en la documentación y tutoriales de la librería."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "29be5230",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "351/351 [==============================] - 0s 1ms/step\n",
      "Model: \"gradient_boosted_trees_model_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Total params: 1\n",
      "Trainable params: 0\n",
      "Non-trainable params: 1\n",
      "_________________________________________________________________\n",
      "Type: \"GRADIENT_BOOSTED_TREES\"\n",
      "Task: CLASSIFICATION\n",
      "Label: \"__LABEL\"\n",
      "\n",
      "Input Features (6):\n",
      "\tbkc\n",
      "\tbkr\n",
      "\twkc\n",
      "\twkr\n",
      "\twrc\n",
      "\twrr\n",
      "\n",
      "No weights\n",
      "\n",
      "Variable Importance: NUM_NODES:\n",
      "    1. \"bkr\" 80207.000000 ################\n",
      "    2. \"wrc\" 44066.000000 #######\n",
      "    3. \"bkc\" 39708.000000 ######\n",
      "    4. \"wkr\" 24787.000000 ###\n",
      "    5. \"wkc\" 19797.000000 ##\n",
      "    6. \"wrr\" 9051.000000 \n",
      "\n",
      "Variable Importance: NUM_AS_ROOT:\n",
      "    1. \"bkr\" 3933.000000 ################\n",
      "    2. \"bkc\" 1479.000000 #####\n",
      "    3. \"wkc\" 933.000000 ###\n",
      "    4. \"wkr\" 606.000000 ##\n",
      "    5. \"wrc\" 251.000000 \n",
      "    6. \"wrr\" 52.000000 \n",
      "\n",
      "Variable Importance: SUM_SCORE:\n",
      "    1. \"bkr\" 16701.633258 ################\n",
      "    2. \"bkc\" 10920.684873 #########\n",
      "    3. \"wrc\" 9608.707738 ########\n",
      "    4. \"wkr\" 6900.316011 #####\n",
      "    5. \"wkc\" 4510.972891 ##\n",
      "    6. \"wrr\" 1757.327341 \n",
      "\n",
      "Variable Importance: MEAN_MIN_DEPTH:\n",
      "    1. \"__LABEL\"  6.491117 ################\n",
      "    2.     \"wrr\"  5.667415 #############\n",
      "    3.     \"wkr\"  4.415438 #########\n",
      "    4.     \"wkc\"  3.703789 #######\n",
      "    5.     \"wrc\"  2.917051 #####\n",
      "    6.     \"bkc\"  2.008529 ##\n",
      "    7.     \"bkr\"  1.172906 \n",
      "\n",
      "\n",
      "\n",
      "Loss: MULTINOMIAL_LOG_LIKELIHOOD\n",
      "Validation loss value: 0.37488\n",
      "Number of trees per iteration: 18\n",
      "Number of trees: 7254\n",
      "Total number of nodes: 442486\n",
      "\n",
      "Number of nodes by tree:\n",
      "Count: 7254 Average: 60.9989 StdDev: 0.0939229\n",
      "Min: 53 Max: 61 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 53, 54)    1   0.01%   0.01%\n",
      "[ 54, 55)    0   0.00%   0.01%\n",
      "[ 55, 56)    0   0.00%   0.01%\n",
      "[ 56, 57)    0   0.00%   0.01%\n",
      "[ 57, 58)    0   0.00%   0.01%\n",
      "[ 58, 59)    0   0.00%   0.01%\n",
      "[ 59, 60)    0   0.00%   0.01%\n",
      "[ 60, 61)    0   0.00%   0.01%\n",
      "[ 61, 61] 7253  99.99% 100.00% ##########\n",
      "\n",
      "Depth by leafs:\n",
      "Count: 224870 Average: 6.49114 StdDev: 1.77628\n",
      "Min: 1 Max: 8 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 1, 2)  3757   1.67%   1.67%\n",
      "[ 2, 3)  5718   2.54%   4.21% #\n",
      "[ 3, 4)  9162   4.07%   8.29% #\n",
      "[ 4, 5) 14169   6.30%  14.59% ##\n",
      "[ 5, 6) 22241   9.89%  24.48% ##\n",
      "[ 6, 7) 33055  14.70%  39.18% ####\n",
      "[ 7, 8) 43372  19.29%  58.47% #####\n",
      "[ 8, 8] 93396  41.53% 100.00% ##########\n",
      "\n",
      "Number of training obs by leaf:\n",
      "Count: 224870 Average: 0 StdDev: 0\n",
      "Min: 0 Max: 0 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 0, 0] 224870 100.00% 100.00% ##########\n",
      "\n",
      "Attribute in nodes:\n",
      "\t80207 : bkr [NUMERICAL]\n",
      "\t44066 : wrc [CATEGORICAL]\n",
      "\t39708 : bkc [CATEGORICAL]\n",
      "\t24787 : wkr [NUMERICAL]\n",
      "\t19797 : wkc [CATEGORICAL]\n",
      "\t9051 : wrr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 0:\n",
      "\t3933 : bkr [NUMERICAL]\n",
      "\t1479 : bkc [CATEGORICAL]\n",
      "\t933 : wkc [CATEGORICAL]\n",
      "\t606 : wkr [NUMERICAL]\n",
      "\t251 : wrc [CATEGORICAL]\n",
      "\t52 : wrr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 1:\n",
      "\t7577 : bkr [NUMERICAL]\n",
      "\t4510 : bkc [CATEGORICAL]\n",
      "\t2157 : wrc [CATEGORICAL]\n",
      "\t1997 : wkc [CATEGORICAL]\n",
      "\t1328 : wkr [NUMERICAL]\n",
      "\t436 : wrr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 2:\n",
      "\t12508 : bkr [NUMERICAL]\n",
      "\t7941 : bkc [CATEGORICAL]\n",
      "\t5815 : wrc [CATEGORICAL]\n",
      "\t3821 : wkc [CATEGORICAL]\n",
      "\t2771 : wkr [NUMERICAL]\n",
      "\t933 : wrr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 3:\n",
      "\t19810 : bkr [NUMERICAL]\n",
      "\t12619 : bkc [CATEGORICAL]\n",
      "\t10928 : wrc [CATEGORICAL]\n",
      "\t5966 : wkc [CATEGORICAL]\n",
      "\t5059 : wkr [NUMERICAL]\n",
      "\t1813 : wrr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 5:\n",
      "\t45160 : bkr [NUMERICAL]\n",
      "\t25541 : wrc [CATEGORICAL]\n",
      "\t24738 : bkc [CATEGORICAL]\n",
      "\t13406 : wkr [NUMERICAL]\n",
      "\t12158 : wkc [CATEGORICAL]\n",
      "\t4880 : wrr [NUMERICAL]\n",
      "\n",
      "Condition type in nodes:\n",
      "\t114045 : ObliqueCondition\n",
      "\t103571 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 0:\n",
      "\t4591 : ObliqueCondition\n",
      "\t2663 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 1:\n",
      "\t9341 : ObliqueCondition\n",
      "\t8664 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 2:\n",
      "\t17577 : ContainsBitmapCondition\n",
      "\t16212 : ObliqueCondition\n",
      "Condition type in nodes with depth <= 3:\n",
      "\t29513 : ContainsBitmapCondition\n",
      "\t26682 : ObliqueCondition\n",
      "Condition type in nodes with depth <= 5:\n",
      "\t63446 : ObliqueCondition\n",
      "\t62437 : ContainsBitmapCondition\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model_bt_2 = tfdf.keras.GradientBoostedTreesModel(\n",
    "    num_trees=500,\n",
    "    growing_strategy=\"BEST_FIRST_GLOBAL\",\n",
    "    max_depth=8,\n",
    "    split_axis=\"SPARSE_OBLIQUE\",\n",
    "    categorical_algorithm=\"RANDOM\",\n",
    "    )\n",
    "model_bt_2.fit(x=train)\n",
    "model_bt_2.summary()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "a5d47881",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.881198715661791, 0.881198715661791, 0.881198715661791, 0.8672254847683257]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = model_bt_2.predict(test)\n",
    "preds = np.argmax(preds, axis=1)\n",
    "\n",
    "metrics_bt_2= compute_metrics_multiclass(y_test_aux, preds)\n",
    "metrics_bt_2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d039962",
   "metadata": {},
   "source": [
    "Los resultados siguen siendo muy buenos pero empeoran levemente con respecto a los parámetros por defecto.\n",
    "\n",
    "La librería implementa algunas plantillas de parámetros que, según la documentación, proveen mejores resultados que los parámetros por defecto.\n",
    "\n",
    "Estos son:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "426cd5ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[HyperParameterTemplate(name='better_default', version=1, parameters={'growing_strategy': 'BEST_FIRST_GLOBAL'}, description='A configuration that is generally better than the default parameters without being more expensive.'), HyperParameterTemplate(name='benchmark_rank1', version=1, parameters={'growing_strategy': 'BEST_FIRST_GLOBAL', 'categorical_algorithm': 'RANDOM', 'split_axis': 'SPARSE_OBLIQUE', 'sparse_oblique_normalization': 'MIN_MAX', 'sparse_oblique_num_projections_exponent': 1.0}, description='Top ranking hyper-parameters on our benchmark slightly modified to run in reasonable time.')]\n"
     ]
    }
   ],
   "source": [
    "print(tfdf.keras.GradientBoostedTreesModel.predefined_hyperparameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b869941",
   "metadata": {},
   "source": [
    "Podemos probar \"benchmark_rank1\", que ofreció los mejores resultados en sus benchmarks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "46bb0bf8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "351/351 [==============================] - 0s 1ms/step\n",
      "Model: \"gradient_boosted_trees_model_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Total params: 1\n",
      "Trainable params: 0\n",
      "Non-trainable params: 1\n",
      "_________________________________________________________________\n",
      "Type: \"GRADIENT_BOOSTED_TREES\"\n",
      "Task: CLASSIFICATION\n",
      "Label: \"__LABEL\"\n",
      "\n",
      "Input Features (6):\n",
      "\tbkc\n",
      "\tbkr\n",
      "\twkc\n",
      "\twkr\n",
      "\twrc\n",
      "\twrr\n",
      "\n",
      "No weights\n",
      "\n",
      "Variable Importance: NUM_NODES:\n",
      "    1. \"bkr\" 59503.000000 ################\n",
      "    2. \"wrc\" 34820.000000 ########\n",
      "    3. \"bkc\" 33116.000000 ########\n",
      "    4. \"wkc\" 14859.000000 ##\n",
      "    5. \"wkr\" 14090.000000 ##\n",
      "    6. \"wrr\" 5583.000000 \n",
      "\n",
      "Variable Importance: NUM_AS_ROOT:\n",
      "    1. \"bkr\" 3037.000000 ################\n",
      "    2. \"bkc\" 958.000000 ####\n",
      "    3. \"wkc\" 830.000000 ####\n",
      "    4. \"wkr\" 418.000000 #\n",
      "    5. \"wrc\" 90.000000 \n",
      "    6. \"wrr\" 67.000000 \n",
      "\n",
      "Variable Importance: SUM_SCORE:\n",
      "    1. \"bkr\" 16677.602664 ################\n",
      "    2. \"bkc\" 9835.501402 ########\n",
      "    3. \"wrc\" 7970.335011 #######\n",
      "    4. \"wkr\" 4682.182032 ###\n",
      "    5. \"wkc\" 4022.472972 ###\n",
      "    6. \"wrr\" 1099.078474 \n",
      "\n",
      "Variable Importance: MEAN_MIN_DEPTH:\n",
      "    1. \"__LABEL\"  5.490134 ################\n",
      "    2.     \"wrr\"  4.978168 ##############\n",
      "    3.     \"wkr\"  4.237620 ###########\n",
      "    4.     \"wkc\"  3.482901 ########\n",
      "    5.     \"wrc\"  3.052854 #######\n",
      "    6.     \"bkc\"  2.224792 ####\n",
      "    7.     \"bkr\"  0.950627 \n",
      "\n",
      "\n",
      "\n",
      "Loss: MULTINOMIAL_LOG_LIKELIHOOD\n",
      "Validation loss value: 0.441028\n",
      "Number of trees per iteration: 18\n",
      "Number of trees: 5400\n",
      "Total number of nodes: 329342\n",
      "\n",
      "Number of nodes by tree:\n",
      "Count: 5400 Average: 60.9893 StdDev: 0.332047\n",
      "Min: 45 Max: 61 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 45, 46)    1   0.02%   0.02%\n",
      "[ 46, 47)    0   0.00%   0.02%\n",
      "[ 47, 48)    0   0.00%   0.02%\n",
      "[ 48, 49)    0   0.00%   0.02%\n",
      "[ 49, 50)    1   0.02%   0.04%\n",
      "[ 50, 51)    0   0.00%   0.04%\n",
      "[ 51, 52)    1   0.02%   0.06%\n",
      "[ 52, 53)    0   0.00%   0.06%\n",
      "[ 53, 54)    0   0.00%   0.06%\n",
      "[ 54, 55)    0   0.00%   0.06%\n",
      "[ 55, 56)    2   0.04%   0.09%\n",
      "[ 56, 57)    0   0.00%   0.09%\n",
      "[ 57, 58)    1   0.02%   0.11%\n",
      "[ 58, 59)    0   0.00%   0.11%\n",
      "[ 59, 60)    2   0.04%   0.15%\n",
      "[ 60, 61)    0   0.00%   0.15%\n",
      "[ 61, 61] 5392  99.85% 100.00% ##########\n",
      "\n",
      "Depth by leafs:\n",
      "Count: 167371 Average: 5.49019 StdDev: 0.978449\n",
      "Min: 1 Max: 6 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 1, 2)    597   0.36%   0.36%\n",
      "[ 2, 3)   3533   2.11%   2.47%\n",
      "[ 3, 4)   6480   3.87%   6.34% #\n",
      "[ 4, 5)  12596   7.53%  13.87% #\n",
      "[ 5, 6)  23579  14.09%  27.95% ##\n",
      "[ 6, 6] 120586  72.05% 100.00% ##########\n",
      "\n",
      "Number of training obs by leaf:\n",
      "Count: 167371 Average: 0 StdDev: 0\n",
      "Min: 0 Max: 0 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 0, 0] 167371 100.00% 100.00% ##########\n",
      "\n",
      "Attribute in nodes:\n",
      "\t59503 : bkr [NUMERICAL]\n",
      "\t34820 : wrc [CATEGORICAL]\n",
      "\t33116 : bkc [CATEGORICAL]\n",
      "\t14859 : wkc [CATEGORICAL]\n",
      "\t14090 : wkr [NUMERICAL]\n",
      "\t5583 : wrr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 0:\n",
      "\t3037 : bkr [NUMERICAL]\n",
      "\t958 : bkc [CATEGORICAL]\n",
      "\t830 : wkc [CATEGORICAL]\n",
      "\t418 : wkr [NUMERICAL]\n",
      "\t90 : wrc [CATEGORICAL]\n",
      "\t67 : wrr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 1:\n",
      "\t7210 : bkr [NUMERICAL]\n",
      "\t3307 : bkc [CATEGORICAL]\n",
      "\t1801 : wkc [CATEGORICAL]\n",
      "\t1770 : wrc [CATEGORICAL]\n",
      "\t1108 : wkr [NUMERICAL]\n",
      "\t407 : wrr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 2:\n",
      "\t13003 : bkr [NUMERICAL]\n",
      "\t7191 : bkc [CATEGORICAL]\n",
      "\t5527 : wrc [CATEGORICAL]\n",
      "\t3460 : wkc [CATEGORICAL]\n",
      "\t2379 : wkr [NUMERICAL]\n",
      "\t916 : wrr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 3:\n",
      "\t22026 : bkr [NUMERICAL]\n",
      "\t13209 : bkc [CATEGORICAL]\n",
      "\t12257 : wrc [CATEGORICAL]\n",
      "\t5894 : wkc [CATEGORICAL]\n",
      "\t4529 : wkr [NUMERICAL]\n",
      "\t1827 : wrr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 5:\n",
      "\t59503 : bkr [NUMERICAL]\n",
      "\t34820 : wrc [CATEGORICAL]\n",
      "\t33116 : bkc [CATEGORICAL]\n",
      "\t14859 : wkc [CATEGORICAL]\n",
      "\t14090 : wkr [NUMERICAL]\n",
      "\t5583 : wrr [NUMERICAL]\n",
      "\n",
      "Condition type in nodes:\n",
      "\t82795 : ContainsBitmapCondition\n",
      "\t79176 : ObliqueCondition\n",
      "Condition type in nodes with depth <= 0:\n",
      "\t3522 : ObliqueCondition\n",
      "\t1878 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 1:\n",
      "\t8725 : ObliqueCondition\n",
      "\t6878 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 2:\n",
      "\t16298 : ObliqueCondition\n",
      "\t16178 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 3:\n",
      "\t31360 : ContainsBitmapCondition\n",
      "\t28382 : ObliqueCondition\n",
      "Condition type in nodes with depth <= 5:\n",
      "\t82795 : ContainsBitmapCondition\n",
      "\t79176 : ObliqueCondition\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model_bt_3 = tfdf.keras.GradientBoostedTreesModel(\n",
    "    hyperparameter_template=\"benchmark_rank1\")\n",
    "\n",
    "model_bt_3.fit(x=train)\n",
    "model_bt_3.summary()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "977ed630",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.8555119514805566,\n",
       " 0.8555119514805566,\n",
       " 0.8555119514805566,\n",
       " 0.8384821966216377]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = model_bt_3.predict(test)\n",
    "preds = np.argmax(preds, axis=1)\n",
    "\n",
    "metrics_bt_3= compute_metrics_multiclass(y_test_aux, preds)\n",
    "metrics_bt_3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "092177e2",
   "metadata": {},
   "source": [
    "Obtenemos los peores resultados. Es curioso que la mejor configuración de parámetros en las pruebas de Google haya producido estas métricas."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d003297",
   "metadata": {},
   "source": [
    "## Creación de árboles con modelos CART con datos _raw_\n",
    "\n",
    "Probamos ahora la función _CartModel_, variación del famoso C4.5. Este tipo de clasificadores han aparecido en otras asignaturas del máster y provee muy buenos resultados."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "b1520dc0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "351/351 [==============================] - 0s 1ms/step\n",
      "Model: \"cart_model_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Total params: 1\n",
      "Trainable params: 0\n",
      "Non-trainable params: 1\n",
      "_________________________________________________________________\n",
      "Type: \"RANDOM_FOREST\"\n",
      "Task: CLASSIFICATION\n",
      "Label: \"__LABEL\"\n",
      "\n",
      "Input Features (6):\n",
      "\tbkc\n",
      "\tbkr\n",
      "\twkc\n",
      "\twkr\n",
      "\twrc\n",
      "\twrr\n",
      "\n",
      "No weights\n",
      "\n",
      "Variable Importance: NUM_NODES:\n",
      "    1. \"wrr\" 267.000000 ################\n",
      "    2. \"wrc\" 202.000000 ###########\n",
      "    3. \"bkr\" 128.000000 #####\n",
      "    4. \"bkc\" 113.000000 ####\n",
      "    5. \"wkc\" 79.000000 ##\n",
      "    6. \"wkr\" 45.000000 \n",
      "\n",
      "Variable Importance: NUM_AS_ROOT:\n",
      "    1. \"bkr\"  1.000000 \n",
      "\n",
      "Variable Importance: SUM_SCORE:\n",
      "    1. \"bkr\" 6537.829253 ################\n",
      "    2. \"wrc\" 6219.915177 ##############\n",
      "    3. \"wrr\" 6025.801832 #############\n",
      "    4. \"bkc\" 5914.132091 ############\n",
      "    5. \"wkr\" 4833.035964 ######\n",
      "    6. \"wkc\" 3554.173828 \n",
      "\n",
      "Variable Importance: MEAN_MIN_DEPTH:\n",
      "    1. \"__LABEL\" 10.990419 ################\n",
      "    2.     \"wrr\"  6.985629 ##########\n",
      "    3.     \"wrc\"  5.141317 #######\n",
      "    4.     \"wkc\"  5.123353 #######\n",
      "    5.     \"wkr\"  3.002395 ####\n",
      "    6.     \"bkc\"  1.723353 ##\n",
      "    7.     \"bkr\"  0.000000 \n",
      "\n",
      "\n",
      "\n",
      "Winner take all: false\n",
      "Out-of-bag evaluation disabled.\n",
      "Number of trees: 1\n",
      "Total number of nodes: 1669\n",
      "\n",
      "Number of nodes by tree:\n",
      "Count: 1 Average: 1669 StdDev: 0\n",
      "Min: 1669 Max: 1669 Ignored: 0\n",
      "----------------------------------------------\n",
      "[ 1669, 1669] 1 100.00% 100.00% ##########\n",
      "\n",
      "Depth by leafs:\n",
      "Count: 835 Average: 10.9904 StdDev: 1.98072\n",
      "Min: 5 Max: 15 Ignored: 0\n",
      "----------------------------------------------\n",
      "[  5,  6)   1   0.12%   0.12%\n",
      "[  6,  7)   6   0.72%   0.84%\n",
      "[  7,  8)  23   2.75%   3.59% #\n",
      "[  8,  9)  50   5.99%   9.58% ###\n",
      "[  9, 10) 112  13.41%  22.99% ######\n",
      "[ 10, 11) 154  18.44%  41.44% #########\n",
      "[ 11, 12) 173  20.72%  62.16% ##########\n",
      "[ 12, 13) 125  14.97%  77.13% #######\n",
      "[ 13, 14)  88  10.54%  87.66% #####\n",
      "[ 14, 15)  65   7.78%  95.45% ####\n",
      "[ 15, 15]  38   4.55% 100.00% ##\n",
      "\n",
      "Number of training obs by leaf:\n",
      "Count: 835 Average: 24.2132 StdDev: 24.0984\n",
      "Min: 5 Max: 229 Ignored: 0\n",
      "----------------------------------------------\n",
      "[   5,  16) 397  47.54%  47.54% ##########\n",
      "[  16,  27) 195  23.35%  70.90% #####\n",
      "[  27,  38) 108  12.93%  83.83% ###\n",
      "[  38,  50)  50   5.99%  89.82% #\n",
      "[  50,  61)  26   3.11%  92.93% #\n",
      "[  61,  72)  16   1.92%  94.85%\n",
      "[  72,  83)  16   1.92%  96.77%\n",
      "[  83,  95)  10   1.20%  97.96%\n",
      "[  95, 106)   4   0.48%  98.44%\n",
      "[ 106, 117)   2   0.24%  98.68%\n",
      "[ 117, 128)   5   0.60%  99.28%\n",
      "[ 128, 140)   1   0.12%  99.40%\n",
      "[ 140, 151)   0   0.00%  99.40%\n",
      "[ 151, 162)   0   0.00%  99.40%\n",
      "[ 162, 173)   2   0.24%  99.64%\n",
      "[ 173, 185)   2   0.24%  99.88%\n",
      "[ 185, 196)   0   0.00%  99.88%\n",
      "[ 196, 207)   0   0.00%  99.88%\n",
      "[ 207, 218)   0   0.00%  99.88%\n",
      "[ 218, 229]   1   0.12% 100.00%\n",
      "\n",
      "Attribute in nodes:\n",
      "\t267 : wrr [NUMERICAL]\n",
      "\t202 : wrc [CATEGORICAL]\n",
      "\t128 : bkr [NUMERICAL]\n",
      "\t113 : bkc [CATEGORICAL]\n",
      "\t79 : wkc [CATEGORICAL]\n",
      "\t45 : wkr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 0:\n",
      "\t1 : bkr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 1:\n",
      "\t1 : wkr [NUMERICAL]\n",
      "\t1 : bkr [NUMERICAL]\n",
      "\t1 : bkc [CATEGORICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 2:\n",
      "\t3 : bkc [CATEGORICAL]\n",
      "\t1 : wrc [CATEGORICAL]\n",
      "\t1 : wkr [NUMERICAL]\n",
      "\t1 : wkc [CATEGORICAL]\n",
      "\t1 : bkr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 3:\n",
      "\t3 : wkr [NUMERICAL]\n",
      "\t3 : wkc [CATEGORICAL]\n",
      "\t3 : bkr [NUMERICAL]\n",
      "\t3 : bkc [CATEGORICAL]\n",
      "\t2 : wrc [CATEGORICAL]\n",
      "\t1 : wrr [NUMERICAL]\n",
      "\n",
      "Attribute in nodes with depth <= 5:\n",
      "\t16 : wrc [CATEGORICAL]\n",
      "\t10 : wrr [NUMERICAL]\n",
      "\t10 : bkr [NUMERICAL]\n",
      "\t9 : wkc [CATEGORICAL]\n",
      "\t9 : bkc [CATEGORICAL]\n",
      "\t8 : wkr [NUMERICAL]\n",
      "\n",
      "Condition type in nodes:\n",
      "\t440 : HigherCondition\n",
      "\t394 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 0:\n",
      "\t1 : HigherCondition\n",
      "Condition type in nodes with depth <= 1:\n",
      "\t2 : HigherCondition\n",
      "\t1 : ContainsBitmapCondition\n",
      "Condition type in nodes with depth <= 2:\n",
      "\t5 : ContainsBitmapCondition\n",
      "\t2 : HigherCondition\n",
      "Condition type in nodes with depth <= 3:\n",
      "\t8 : ContainsBitmapCondition\n",
      "\t7 : HigherCondition\n",
      "Condition type in nodes with depth <= 5:\n",
      "\t34 : ContainsBitmapCondition\n",
      "\t28 : HigherCondition\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model_CART_1 = tfdf.keras.CartModel()\n",
    "\n",
    "model_CART_1.fit(x=train)\n",
    "model_CART_1.summary()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "c26cbc17",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.6307527648947556,\n",
       " 0.6307527648947556,\n",
       " 0.6307527648947556,\n",
       " 0.5866906674682764]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = model_CART_1.predict(test)\n",
    "preds = np.argmax(preds, axis=1)\n",
    "\n",
    "metrics_CART_1= compute_metrics_multiclass(y_test_aux, preds)\n",
    "metrics_CART_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1367f05a",
   "metadata": {},
   "source": [
    "Obtenemos unos malos resultados con un 63% de Precision, F1 y Recall. Es lógico, ya que tanto RandomForest como BoostedTrees ofrecen mejores resultados que CART al usar éste último un único árbol."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8de872b",
   "metadata": {},
   "source": [
    "## Comparación eficiencia RandomForest de Tensorflow vs Scikit-Learn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb0d72fa",
   "metadata": {},
   "source": [
    "En el apartado de análisis exploratorio de datos usamos RandomForest con la librería Scikit-Learn para estudiar las variables predictoras con más importancia. Podemos pensar qué ventajas aporta esta nueva librería de TensorFlow frente a la ya mencionada anterioremente.\n",
    "\n",
    "Hay aspectos claros, la librería de TensorFlow ofrece una mayor integración en el ecosistema y permite crear modelos de árboles que interactúen con redes neuronales de Keras. Además, el número de métodos de las clases así como los parámetros de customización son mayores (recomendamos ver la documentación de la API).\n",
    "\n",
    "Nos preguntamos si, al usar la librería Yggdrasil _under the hood_, obtenemos una mayor rápidez.\n",
    "\n",
    "Para ello creamos dos modelos de RandomForest con ambas librerías. Los dos tienen la misma configuración: 300 árboles y una profundidad máxima de 8 nodos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "9e0454af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.21 s ± 541 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "train_sk, test_sk = split_dataset(data)\n",
    "\n",
    "clf = RandomForestClassifier(max_depth=8,\n",
    "                             random_state=0,\n",
    "                             n_estimators=300)\n",
    "\n",
    "%timeit clf.fit(train_sk.drop('opt rank', axis=1), train_sk['opt rank'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "28c27ee8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "351/351 [==============================] - 0s 1ms/step\n",
      "351/351 [==============================] - 0s 1ms/step\n",
      "351/351 [==============================] - 0s 1ms/step\n",
      "351/351 [==============================] - 0s 1ms/step\n",
      "WARNING:tensorflow:5 out of the last 92 calls to <function CoreModel.make_predict_function.<locals>.predict_function_trained at 0x7f521052ee50> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:5 out of the last 92 calls to <function CoreModel.make_predict_function.<locals>.predict_function_trained at 0x7f521052ee50> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "351/351 [==============================] - 0s 1ms/step\n",
      "WARNING:tensorflow:6 out of the last 93 calls to <function CoreModel.make_predict_function.<locals>.predict_function_trained at 0x7f521042a8b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:6 out of the last 93 calls to <function CoreModel.make_predict_function.<locals>.predict_function_trained at 0x7f521042a8b0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "351/351 [==============================] - 0s 1ms/step\n",
      "351/351 [==============================] - 0s 1ms/step\n",
      "351/351 [==============================] - 0s 1ms/step\n",
      "449 ms ± 4.69 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "model = tfdf.keras.RandomForestModel(max_depth=8,\n",
    "                                     num_trees=300)\n",
    "\n",
    "%timeit model.fit(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b99c7653",
   "metadata": {},
   "source": [
    "Mientras que la función de la librería Scikit-learn ha necesitado 2.21 s (de media) para realizar el entrenamiento, la función de TensorFlow lo ha hecho en 449 ms, unas cinco veces más rápido.\n",
    "\n",
    "Vemos que la nueva librería ofrece resultados más rápidos. Intuimos que la diferencia en tiempos aumentará de forma notable al usar dataset más grandes y con configuraciones de árboles más complejas.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8169a422",
   "metadata": {},
   "source": [
    "## Conclusiones"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abe034e3",
   "metadata": {},
   "source": [
    "Mostramos los resultados de las métricas obtenidas por los algoritmos usados en este apartado:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "9eb4dae4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Modelo</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1</th>\n",
       "      <th>Cohen kappa</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Random Forest</td>\n",
       "      <td>0.768284</td>\n",
       "      <td>0.768284</td>\n",
       "      <td>0.768284</td>\n",
       "      <td>0.740887</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BT por defecto</td>\n",
       "      <td>0.891366</td>\n",
       "      <td>0.891366</td>\n",
       "      <td>0.891366</td>\n",
       "      <td>0.878577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BT más complejo</td>\n",
       "      <td>0.881199</td>\n",
       "      <td>0.881199</td>\n",
       "      <td>0.881199</td>\n",
       "      <td>0.867225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BT mejor benchmark</td>\n",
       "      <td>0.855512</td>\n",
       "      <td>0.855512</td>\n",
       "      <td>0.855512</td>\n",
       "      <td>0.838482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CART</td>\n",
       "      <td>0.630753</td>\n",
       "      <td>0.630753</td>\n",
       "      <td>0.630753</td>\n",
       "      <td>0.586691</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Modelo  Precision    Recall        F1  Cohen kappa\n",
       "0       Random Forest   0.768284  0.768284  0.768284     0.740887\n",
       "1      BT por defecto   0.891366  0.891366  0.891366     0.878577\n",
       "2     BT más complejo   0.881199  0.881199  0.881199     0.867225\n",
       "3  BT mejor benchmark   0.855512  0.855512  0.855512     0.838482\n",
       "4                CART   0.630753  0.630753  0.630753     0.586691"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metricas = [metrics_rf, metrics_bt_1, metrics_bt_2, metrics_bt_3, metrics_CART_1]\n",
    "precision = []\n",
    "recall = []\n",
    "f1 = []\n",
    "cohen_kappa = []\n",
    "\n",
    "for metrica in metricas:\n",
    "    precision.append(metrica[0])\n",
    "    recall.append(metrica[1])\n",
    "    f1.append(metrica[2])\n",
    "    cohen_kappa.append(metrica[3])\n",
    "    \n",
    "data = {\"Modelo\" : [\"Random Forest\", \"BT por defecto\", \"BT más complejo\", \"BT mejor benchmark\",\"CART\"],\n",
    "        \"Precision\" : precision,\n",
    "       \"Recall\": recall,\n",
    "       \"F1\" : f1,\n",
    "       \"Cohen kappa\" : cohen_kappa}\n",
    "pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa8275a9",
   "metadata": {},
   "source": [
    "En la peor posición encontramos el algoritmo de tipo CART. Lógico, ya que solo crea un árbol mientras que los demás usan ensembles.\n",
    "Random Forest obtiene aproximadamente un 76% en las métricas, pero es superado por rendimiento por todas las pruebas de Boosted Trees.\n",
    "\n",
    "En las configuraciones utilizadas, los parámetros por defecto obtienen mejores resultados (89% en Precision, Recall y F1) que el modelo más complejo y usando la plantilla recomendada por la librería que ofrecía mejores resultados.\n",
    "\n",
    "Además hemos comprobado que esta librería es más eficiente en tiempo computacional que los algoritmos incluidos en la librería Scikit-Learn.\n",
    "\n",
    "En conclusión, hemos obtenido unos resultados que igualan a los mejores conseguidos con redes neuronales con unos modelos mucho más fáciles de configurar y más rápidos de entrenar."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.10 64-bit ('tf2.5': conda)",
   "language": "python",
   "name": "python3810jvsc74a57bd020843789c8bb0501c47bf41ec8f34cc8e8803d6ea02edd6aadc3bbb21aeee5b3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}