week04_[recap]_deep_learning/seminar_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hello, PyTorch\n",
    "\n",
    "![img](https://pytorch.org/tutorials/_static/pytorch-logo-dark.svg)\n",
    "\n",
    "__This notebook__ will teach you to use PyTorch low-level core. If you're running this notebook outside the course environment, you can install it [here](https://pytorch.org).\n",
    "\n",
    "__PyTorch feels__ differently than Tensorflow on almost every level. TensorFlow makes your code live in two \"worlds\" simultaneously:  symbolic graphs and actual tensors. First you declare a symbolic \"recipe\" of how to get from inputs to outputs, then feed it with actual minibatches of data.  In PyTorch, __there's only one world__: all tensors have a numeric value.\n",
    "\n",
    "You compute outputs on the fly without pre-declaring anything. The code looks exactly as in pure numpy with one exception: PyTorch computes gradients for you. And can run stuff on GPU. And has a number of pre-implemented building blocks for your neural nets. [And a few more things.](https://medium.com/towards-data-science/pytorch-vs-tensorflow-spotting-the-difference-25c75777377b)\n",
    "\n",
    "And now we finally shut up and let PyTorch do the talking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys, os\n",
    "if 'google.colab' in sys.modules and not os.path.exists('.setup_complete'):\n",
    "    !wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_RL/master/week04_%5Brecap%5D_deep_learning/notmnist.py\n",
    "    !touch .setup_complete"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.4.1\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X:\n",
      "[[ 0  1  2  3]\n",
      " [ 4  5  6  7]\n",
      " [ 8  9 10 11]\n",
      " [12 13 14 15]]\n",
      "\n",
      "X.shape: (4, 4)\n",
      "\n",
      "add 5:\n",
      "[[ 5  6  7  8]\n",
      " [ 9 10 11 12]\n",
      " [13 14 15 16]\n",
      " [17 18 19 20]]\n",
      "\n",
      "X*X^T:\n",
      "[[ 14  38  62  86]\n",
      " [ 38 126 214 302]\n",
      " [ 62 214 366 518]\n",
      " [ 86 302 518 734]]\n",
      "\n",
      "mean over rows:\n",
      "[ 1.5  5.5  9.5 13.5]\n",
      "\n",
      "cumsum of cols:\n",
      "[[ 0  1  2  3]\n",
      " [ 4  6  8 10]\n",
      " [12 15 18 21]\n",
      " [24 28 32 36]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# numpy world\n",
    "\n",
    "x = np.arange(16).reshape(4, 4)\n",
    "\n",
    "print(\"X:\\n%s\\n\" % x)\n",
    "print(\"X.shape: %s\\n\" % (x.shape,))\n",
    "print(\"add 5:\\n%s\\n\" % (x + 5))\n",
    "print(\"X*X^T:\\n%s\\n\" % np.dot(x, x.T))\n",
    "print(\"mean over rows:\\n%s\\n\" % (x.mean(axis=-1)))\n",
    "print(\"cumsum of cols:\\n%s\\n\" % (np.cumsum(x, axis=0)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X:\n",
      "tensor([[ 0.,  1.,  2.,  3.],\n",
      "        [ 4.,  5.,  6.,  7.],\n",
      "        [ 8.,  9., 10., 11.],\n",
      "        [12., 13., 14., 15.]])\n",
      "X.shape: torch.Size([4, 4])\n",
      "\n",
      "add 5:\n",
      "tensor([[ 5.,  6.,  7.,  8.],\n",
      "        [ 9., 10., 11., 12.],\n",
      "        [13., 14., 15., 16.],\n",
      "        [17., 18., 19., 20.]])\n",
      "X*X^T:\n",
      "tensor([[ 14.,  38.,  62.,  86.],\n",
      "        [ 38., 126., 214., 302.],\n",
      "        [ 62., 214., 366., 518.],\n",
      "        [ 86., 302., 518., 734.]])\n",
      "mean over rows:\n",
      "tensor([ 1.5000,  5.5000,  9.5000, 13.5000])\n",
      "cumsum of cols:\n",
      "tensor([[ 0.,  1.,  2.,  3.],\n",
      "        [ 4.,  6.,  8., 10.],\n",
      "        [12., 15., 18., 21.],\n",
      "        [24., 28., 32., 36.]])\n"
     ]
    }
   ],
   "source": [
    "# PyTorch world\n",
    "\n",
    "x = np.arange(16).reshape(4, 4)\n",
    "\n",
    "x = torch.tensor(x, dtype=torch.float32)  # or torch.arange(0, 16).view(4, 4)\n",
    "\n",
    "print(\"X:\\n%s\" % x)\n",
    "print(\"X.shape: %s\\n\" % (x.shape,))\n",
    "print(\"add 5:\\n%s\" % (x + 5))\n",
    "print(\"X*X^T:\\n%s\" % torch.matmul(x, x.transpose(1, 0)))  # short: x.mm(x.t())\n",
    "print(\"mean over rows:\\n%s\" % torch.mean(x, dim=-1))\n",
    "print(\"cumsum of cols:\\n%s\" % torch.cumsum(x, dim=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NumPy and PyTorch\n",
    "\n",
    "As you can notice, PyTorch allows you to hack stuff much the same way you did with NumPy. No graph declaration, no placeholders, no sessions. This means that you can _see the numeric value of any tensor at any moment of time_. Debugging such code can be done with by printing tensors or using any debug tool you want (e.g. [PyCharm debugger](https://www.jetbrains.com/help/pycharm/part-1-debugging-python-code.html) or [gdb](https://wiki.python.org/moin/DebuggingWithGdb)).\n",
    "\n",
    "You could also notice the a few new method names and a different API. So no, there's no compatibility with NumPy [yet](https://github.com/pytorch/pytorch/issues/2228) and yes, you'll have to memorize all the names again. Get excited!\n",
    "\n",
    "![img](http://i0.kym-cdn.com/entries/icons/original/000/017/886/download.jpg)\n",
    "\n",
    "For example,\n",
    "* If something takes a list/tuple of axes in NumPy, you can expect it to take `*args` in PyTorch\n",
    " * `x.reshape([1,2,8]) -> x.view(1,2,8)`\n",
    "* You should swap `axis` for `dim` in operations like `mean` or `cumsum`\n",
    " * `x.sum(axis=-1) -> x.sum(dim=-1)`\n",
    "* Most mathematical operations are the same, but types an shaping is different\n",
    " * `x.astype('int64') -> x.type(torch.LongTensor)`\n",
    "\n",
    "To help you acclimatize, there's a [table](https://github.com/torch/torch7/wiki/Torch-for-NumPy-users) covering most new things. There's also a neat [documentation page](http://pytorch.org/docs/master/).\n",
    "\n",
    "Finally, if you're stuck with a technical problem, we recommend searching [PyTorch forums](https://discuss.pytorch.org/). Or just googling, which usually works just as efficiently.\n",
    "\n",
    "If you feel like you almost give up, remember two things: __GPU__ and __free gradients__. Besides you can always jump back to NumPy with `x.numpy()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Warmup: trigonometric knotwork\n",
    "_inspired by [this post](https://www.quora.com/What-are-the-most-interesting-equation-plots)_\n",
    "\n",
    "There are some simple mathematical functions with cool plots. For one, consider this:\n",
    "\n",
    "$$ x(t) = t - 1.5 * cos(15 t) $$\n",
    "$$ y(t) = t - 1.5 * sin(16 t) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "t = torch.linspace(-10, 10, steps=10000)\n",
    "\n",
    "# compute x(t) and y(t) as defined above\n",
    "x = <YOUR CODE>\n",
    "y = <YOUR CODE>\n",
    "\n",
    "plt.plot(x.numpy(), y.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're done early, try adjusting the formula and seeing how it affects the function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automatic gradients\n",
    "\n",
    "Any self-respecting DL framework must do your backprop for you. Torch handles this with the `autograd` module.\n",
    "\n",
    "The general pipeline looks like this:\n",
    "* When creating a tensor, you mark it as `requires_grad`:\n",
    "    * `torch.zeros(5, requires_grad=True)`\n",
    "    * `torch.tensor(np.arange(5), dtype=torch.float32, requires_grad=True)`\n",
    "* Define some differentiable `loss = arbitrary_function(a)`\n",
    "* Call `loss.backward()`\n",
    "* Gradients are now available as ```a.grad```\n",
    "\n",
    "__Here's an example:__ let's fit a linear regression on Boston house prices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7fb9a5c1d898>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJztnX+MHOWZ57/PtMu4xyT0eBlypsHYyUX2LuvYA7OJVz5FwbeLsyGQSYA4CFacFC13pz0psNxcJicUm5WjzO4oC/8l4i65IMUBA84OJtbJidaO9tYnnBtnPCE+8OUHxKRB4MhuEpgG2jPv/dFd457q9616q7q6qrr6+5Gs6a6prnqqPP3U8z7v93leUUqBEEJI7zOQtgGEEELigQ6dEEJyAh06IYTkBDp0QgjJCXTohBCSE+jQCSEkJ9ChE0JITqBDJ4SQnECHTgghOWFFkie7/PLL1fr165M8JSGE9DwnTpz4rVJqOGi/RB36+vXrMTMzk+QpCSGk5xGRX9vsx5QLIYTkBDp0QgjJCXTohBCSE+jQCSEkJ9ChE0JITrBSuYjISwB+D2ABwAWl1KiIrAGwH8B6AC8B+KxS6nzcBk7PVjB1+DQq1RoKIlhQCuVSEeM7N2JspLy03wPTz+G7x89g0bNehwjwr4dX41dn57GgFAoiuOMjV2Pv2GZMz1aw5+ApVGt1AMDQoIPdN18LAJg6fBqvVGu4UnOuKPbbHsu0f9jjEEL6D7FZsajp0EeVUr9t2fb3AM4ppSZFZALAkFLqi37HGR0dVWFki9OzFXzpe8+hVl9o+13RKeCrn9mMsZEyHph+Dt959oz1cQFg+wfW4Mcvnkfd8wQYEKAwIKgvXNzeeq4w6Oz3O5Zp/1uvL+PAiYr1cQgh+UJETiilRoP26yTl8ikAjzZfPwpgrINjaZk6fFrrzAGgVl/A1OHTAIDHjr8c+tjHfnmuzZkDwKLCMmfuPVcYdPb7Hcu0/2PHXw51HEJIf2Lr0BWAH4jICRG5p7ntfUqpVwGg+fMK3QdF5B4RmRGRmbNnz4Yy7pVqzer3CwmsixpkS5jPhN1uur4oNhFC8outQ9+ulLoOwF8A+GsR+ajtCZRSjyilRpVSo8PDgZWry7iyVLT6fUEk1HGjEGRLmM+E3W66vig2EULyi5VDV0q90vz5OoB/BPBhAK+JyFoAaP58PW7jxnduRNEpaH9XdAoY37kRAHDHR64OfeztH1gDZ6DdUQ4I4BSWb289Vxh09vsdy7T/HR+5OtRxCCH9SaBDF5HVIvIe9zWAGwH8DMBBAHc3d7sbwNNxGzc2UsZXP7MZZU8kXi4Vl00I7h3bjLu2rYPGP0ME+OAVq5c+WxDBXdvWYd9f/Smmbt+CUtFZ2ndo0ME/fHYrpm7bgnKpCPGca3q2gu2TR7Bh4hC2Tx7B9GzF2n7vscLsv3dsc6jjEEL6k0CVi4i8H42oHGjIHL+rlPqKiPwBgCcArANwBsDtSqlzfscKq3KJk05lf2EVK4QQEhe2Khcr2WJcpOXQdc5YANy5bR32jm22Osb2ySOoaCYhy6Uijk3siMtUQghpIwnZYs+gkwMqAPuePROYNnEJq0whhJCk6QuHbnK6CrDWcodVphBCSNL0hUP3c7q2EXZYxQohhCRNXzj08Z0bYVKq20bYYRUrhBCSNIkuQZcWYyNlzPz6HPY9ewatU8BhI+yxkTIdOCEks/RFhA40tOoP7drKCJsQklv6xqETQkje6YuUC9CuRa9Ua/jS954DAEbphJBc0DcRethWtoQQ0mvkKkL3K+9nYRAhJO/kJkJ3UyqVag0KF1MqbiUoC4MIIXknNxF6UEpl/t0LbZ9hYRAhJE/kxqGbUidupO519qWigz23XMsJUUJIbshNysVvtR/duqSrL1lBZ04IyRW5ceimXium9Tgr1ZrVIhWEENIr5Mahm3qtlH0mPb0Tp4QQ0svkJocOmHut6HLoLu7EKdMvhJBeJ1cOXYfrqKcOn9auOARQi04IyQe5c+im4qKxkbJxGTlq0QkheSBXDl3Xr2X8qTnsOXgKb9TquKzowCkI6gsXJ0qpRSeE5IXcTIoC+uKi+oJCtVaHAlCt1QEFDA06bKFLCMkduYrQTTnyVuqLCoMrV2D2yzcmYBEhhCRHrhx6QcSoO28lqUlQv2ZhhBASN7ly6DbOHEhmEpT91wkhSZOrHLpfEZGLMyCJTIKy/zohJGly5dDHd26EUxDffS5dlUwPF/ZfJ4QkTa4cOgAgIOtSna8nYgb7rxNCkiZXDn3q8GnUF/09elIO1dQsjJp3Qki36NlJUZ2CJCidkaRDbW05QJULISQJRFkqQ+JgdHRUzczMdHwcr4IEaDjrS1YMNIqHNJQNDpXSQkJI1hGRE0qp0aD9ejJCNylIVjkDKDqFNkdvqgaltJAQkid6ModuSq1U5+vanugm50xpISEkT/RkhH5lqWjsmmjqia6D0kJCSJ7oyQg9LgUJpYWEkDzRkw7dtNxc2Lw3pYWEkDxhnXIRkQKAGQAVpdQnRWQDgMcBrAHwEwB/qZR6tztmthMmteJ3DIDSQkJIPgiTQ/8CgOcBvLf5/u8APKSUelxEvgHg8wC+HrN9XSeOB0MQlEYSQpLAKuUiIlcBuAnAf2++FwA7ADzV3OVRAGPdMLDXcaWRlWoNChelkdOzlbRNI4TkDNsc+sMA/guAxeb7PwBQVUpdaL7/DQBtyCki94jIjIjMnD17tiNjexFKIwkhSRHo0EXkkwBeV0qdaN2s2VVbcqqUekQpNaqUGh0eHo5oZu9CaSQhJClscujbAdwiIp8AsAqNHPrDAEoisqIZpV8F4JXumdm7+GnmCSEkTgIjdKXUl5RSVyml1gP4HIAjSqk7ARwFcFtzt7sBPN01K2NmeraC7ZNHsGHiELZPHulqPpvSSEJIUnSiQ/8igL8RkV+gkVP/ZjwmdZekJynj0swTQkgQPdltsRO2Tx7RpkDKpSKOTexIwSJCCPEn190WO8E0GVmp1rB98gi14oSQnqUnS/87wTQZKQC14oSQniZ3Dj1owlM3SSlo11xSK04I6TVy5dB1E5737T+J9S3OXTdJaZpFqFRrjNIJIT1DrnLouqpM11l7VyNqzY+bJkoBYPzJuaXPEEJIlslVhB5UfWlKo+jSMC71RYU9B0/FYh8hhHSTXDl0m+pLndN30zAmTAtPE0JIlsiNQ5+ereCtdy4E7mdy+kmnVJKsViWE9Ae5yKG7k6He/LmXoJL7oUEH5+fbo3ERLE2o2tgS1Pvca683vx/lmIQQkosIXTcZCgClohOq5H73zdfCKbQ3klQKVrr0oLYCblR+7/6T1i112U+dEGJLLiJ002ToG7U6Tu6+0fo4rrO//4k5LHhaIrgO1++BENT7PGgUobsOv2MySieEtJILh95Ji1o3nVGp1lAQaXPkrQSpaPx6n5tGEUH2sp86IcSWXKRcoraobU1nAPB15oD5AeGmUkyfvrJUDHTAJntN52Q/dUKIl1w49Kgtam2iZheTw/U+FEyf83PAfvaynzohxJaeSbkEKT281Z82xzA5YR2rHP2zz++hUPbY6c2hF51C4IPH/R1VLoSQIHrCoYeV+dkeIwzn5+vac5pSKQIs66/eiWO2eVgRQkhPOPQ4lB5h0ismdOcsGbTrpUGnbRsdMyGkm/REDj0OpUdcqhDvcUzzqAkuBEUIIQB6JELvRJYYdAwTuh7punO+YejzYtoeFVaLEkKC6IkIPQ6lh+4YzoBoK0OHBh3cuW2d1TmTkBWyWpQQYkNPROg2E4o2KhjdMfyOO3rNmsCo+IZNw9j37Jll0XzcskJWixJCbBCVYLJ3dHRUzczMxH5cXXMukyRweraCPQdPLbXEHRp0sPvmawHYKVBaK0t1aRkBcOe2ddg7Zm7HG5YNE4e06R8B8OLkTbGdhxCSTUTkhFJqNGi/nojQg7CNYKdnKxh/cg71xYvu8fx8HffuP7nssyZZpPfBoXOyCsDRF8762hs2Hx7HHAIhJP/kwqHbqmCmDp9e5sz9aG2q5TrfgYBeL0H2ANE09eM7N2pHIKwWJYS00hOTokHYTkyGlS66ztadjLRx5n72AMEdGXVEbW1ACOkvchGhmyLYGzYNY/vkkaXUxmVFJ9RycgWR0MVIfpHz9GzFKJ0MetiwKIkQEkQuHLrr6FonOwcE2P/jl5dSLJVqDU5BMABg0eKYRacQ2pkPyPJoW5d/N8F8OCGkU3KRcnF558JFV/3Wuwtt+fL6gsJlgw5KxfayfKChGgEupjTKAU62II1PlIoOnILAPZ1OJ+7XeoD5cEJIHOTGodv2ajk/X8cbtTrKpSLu2rZuWV76oV1b8dLkTUtNtfwWnS6XivjaZ7egXCqiWqujvqBf4cjFL6XCfDghJA5ykXIBwk14utWWB05U8NXPNPTiU4dP4779JzF1+DRu2DSMAycqvhH1DZuGQy0pZ5IelktFOnNCSCzkJkKPkoOu1Rfwpe/9FPftP7msrP47z57x7XH+1c9sxtEXzoZaUo4LVRBCuk1uHLq2V0tB0N6pZTm1+qJx6Tgvbo/zsZFy6CXl0pIeusvjbZg4hO2TR9j/hZAck5uUi6lXy32eKtBOaI24/bo3elcqarUxyfRKHAuDdHJudockJFly49ABvcN0+650ijfiNmnfszTBmVZTrzQfJIT0M4EpFxFZJSI/FpE5ETklIg82t28QkeMi8nMR2S8iK7tvbnh0qZiwFETaHHXcKZRupEbiWBgkClGqYQkhnWMTob8DYIdS6k0RcQD8i4j8TwB/A+AhpdTjIvINAJ8H8PUu2hoJXSomTMTuF3XrRgS6VIP3/LrWv92IaNNq6pXWg4SQficwQlcN3my+dZr/FIAdAJ5qbn8UwFhXLIyBsZEyjk3swItNjXlQwZBL2Kh7eraC8afmlilm7n9yDuNPLt9mU3QUR0SrG51I04ZuTpAmsegHIaQdK5WLiBRE5CSA1wH8EMAvAVSVUm7lzW8AZC45akpjjO/cGKh+KZeKS4oWWx585lRbgdHComqrWLUtOuo0om1NCwHLl9WzXfUoSiqIEk1C0sHKoSulFpRSWwFcBeDDAP5Qt5vusyJyj4jMiMjM2bP+fcLjxG/ZtrGRsq9UUYClxl5hHNn5efvGX96iIx1xRLTu6KRcKrZdc9AoIOrSd+wOSUg6hFK5KKWqIvIjANsAlERkRTNKvwrAK4bPPALgEaCxYlFn5toTpPAYGnSMDlgByypFu6HSGBDBholDuLJU1Famxh3RRhkFdKKSYXdIQpLHRuUyLCKl5usigD8D8DyAowBua+52N4Cnu2VkFPwc2PRsBW++be7TAiBSTtvU9EvHglLLWhDcen25qxFtlFEAJzcJ6S1sIvS1AB4VkQIaD4AnlFLfF5H/C+BxEdkLYBbAN7toZ2j8FB5hVi5qxZ1MNClX9txybdsSd14KmlWPavUFHH3h7FJTMBOdFOtEWfWIS98R0lsEOnSl1E8BjGi2/wqNfHomuWHTMPY9e2ZZ3th1YJ1Uj1aqNYw/OQcIliZAK9Ua7tt/EgqNKN20iIYAWDSsetQa9Zqkj51IG02VtH6f5dJ3hPQWuaoUdZmereDAicoyZy4Abr2+kdfttHpUF4G7W6q1+jI1iXcfXYQOXIx6dZp092HhJWzVZ9i8dpSHACEkPXLp0HWTeQrA0RcaKhtd5BknCjA6dZ0zb416Tbab6CSfbZPC4eQmIb1DLh160GRea+QZR58XHQqNyU3T8QsiWFSqzZGGddBR89nst0JI/shN+9xWbBQdrfrssDgDAqfgX5pUEPF1zotKLVWutjrQsA56/t0LkSo+2W+lAdsLkzyRS4ceplIxbERcLhUxdfsWTN22ZVkFphdXlmjC5LjDNhM7P1+3KvbxQkli9MIpQrJKLlMuYSbzwjTrKogsO477szUXPWCY9GzFTykSJR0UpSVuv0kSdfMFabUXJqRbiApwPnEyOjqqZmZmEjufDd5cMtBY6QhKr2YJ6nm+YeJQYFuBO7etw96xzVb2rZ84ZLWfAHhx8iarfQH9dWetn3tcmK7VNCke9l4S0m1E5IRSajRov1xG6GFwndeDz5xaagWweuUKfHLLWjx2/GVtEdC9zcWkdQVGJZ+WAkBjsvSx4y9j9Jo1Vo7Tb2K1lbCRdT9JEk2ReJCElJBeo+8dusvb9cWl19VaHQdOVHxTJ5VqDeNPzS2L5CvV2tKEqbfrYisLSrUpSkwSQhuJZdRin04lib2yzJxpXmBBqbZInYVTpJfJ5aRoWPwiOD/qC+2tceuLCisGJFA906ooCeoM+dXPbDbaoltNKQl6aULRFHG7PXPYFZLkBTp0BEdwYanVFzG+cyMe3rXV9/PueYMkhGMjZWPLgEWlUnFAvSR79FM9eRc/oTMnvQwdOuwiuLC4Sgm/6No9r42EMGurAPWS7JH92Um/wBw6/JtQuXlmkxrGlCv3VqX6NbmykRBmrVFWr8ke2cKA9AOM0GEfwV2y4uLtGhp0MHXbFgwN6nuge6tS/Y5vUwiVtSiTy8wRkj36XoduolXBURp08ObbF0L3UC97lB9+qhC/3z0w/dyyVsBu4y/v8ZOmV1QuhPQ6tjr03Dv0KE5Hl16JilMQrF65QttW16aQ54Hp5/CdZ88Yf58V504I6R4sLEK0joLTsxXc/8RcYPm+LfUFtbTghWmRZj8nvM/HmbceM6luiYzKCckuuXboYXt1uA+AuJy5DSZViOs4w1jSem2mVY86cca2D0g6fULSIdcOPay07sFnTnVt0QsTOlVIJymfV6q1tpy7adm8sBG9zQOSfdYJSY9cO/Qw0rrp2YpvD5ZuIMBS5By2Y6OJy4pO21qqgL7RWK2+gAefOWUdTds8INnBkJD0yLVsMYy0Lo0KRwUs07i7ZfRRnXnRKUDEf8k6L+fn69bl+zbFTb1UcETigwuFZINcO3Rb7fb0bKVrS9H5IWioWO5/Yi5SemXQGWi7tmqHowy/8n2bB2TWKlpJ9+mlvj55J/eyxSDilChGwbSYdJjPX1Z0IAJU5+sdpWtaj9naD9xNB1WqtWX2Dg062H3ztcsmYXUPRqcgmLptS2wpF9tJV07OJsP2ySPa//dyqYhjEztSsCh/2MoWcx2h26DL+SZJp49ThUa73/Pzdat0TUFkKaIvFfVVrgMiS9FVa/TltddtOezdR2tkTNhGg4wak4NptuzQ9w49jVRLmrhrnb71zgV8cstabTdIt1+7G+GaHnhueibooVhfVLHNUdh2eeylbpC9DtNs2aGvHfr0bEW7wHMWWb2yACfG/61qrY79P34Zt15f1naDdFdmCnrgvVKtWUVifnr7MJNppuNUqrVlx2DUmBzs65Md+tqhhy3cSQMBcNe2dfj0dWW0LKoUC/VFhaMvnDX2WrfhylLRKhLz09uHSYv4nav1GCWLpmkkHrLWOK6fybUOPYheiNYUgO/PvbrUPiBu3AnDKKmn1ihs/Kk5YythP6loWM26zZJ8tfoCLlkxwOXlEoTtibNBXzv0qI4sabrlzIHGA2P+3QtwBsSqm6S7sHK5VMQNm4aNyhYXv6ZhUdIi3sWtTRa/UavjoV1bqXIhfUVfO3SbaK8fOD9fh1MQKwnlv7ps1VKUG3TvTLK1oD41prSIV4b40K6t2HPwlPaBd2Wp2NWokZJIkkWoQzdorOMg7uN1m6FBB9Wm/NEPZ0Bw6aoVVq0SBFjm8IJ0/6aWwtoVowYEiwAWPCMLZ0AwdXt8uncvOltsWiETEhW2z7WkNYrzK46Jwp3b1uHQT19NvEdMVM7P11F0BlALmH2tLyrra2qdqAT8df9+6Rnd50wpoktXreiqY2W/GpJV+lrl4sVdAd5vUWhbmePQoIO9Y5sx++Ub8fCurZEWmu6Uh3dtRTGk1jHImUfFdXim/LgAODaxI3RjMB2dtj8IgpJIklXo0DXodLUAUCo6uHPbumXyrO0fWNPm5AXATR9au/Te5kHRDaYOn16q5swCbr5ZR5CcMIzcsNvSRBbSkKxCh67B1dV6F4Cu1uo4cKKC8Z0b8eLkTTg2sQP7/upPcee2dcucugJw4ERlmZ56eraC+XcvJHMBTSo+DrSbaOqUAAClQUd7D2zkhLqHrDMgKAwsP5lTkK5LE1lIQ7JKoEMXkatF5KiIPC8ip0TkC83ta0TkhyLy8+bPoe6bmxxjI2UMrmyfYtCVjx994axxeTng4iRa0rn0gjScW9LVsEqh3fkWBG++faHtHohcvFd+BUW64pVdH766/Q84gVloFtKQrBKochGRtQDWKqV+IiLvAXACwBiAfwfgnFJqUkQmAAwppb7od6wsqlz82DBxSOsfvN0Ig/YzdaNLgpcmb8L6iUOJn7dV4VN0BnDJikKgnj7sgte91uWPUkcSldhULkqpVwG82nz9exF5HkAZwKcAfKy526MAfgTA16H3GrYrHgXtl9ZkWblUXOpXk7R8svV8tfqi1WRr2AWvk56c7MQhc2k+kgShcugish7ACIDjAN7XdPau078ibuPSxjZXGrRfGnls9/y90K9GR62+gPufmIvU1+XK5oMszhV0Om3Hy+6PJAmsHbqIXArgAIB7lVK/C/G5e0RkRkRmzp49G8XG1LDNlQbtZ1LNdGtGuiCydP4sSel098CP1ja+OkwP0hs2DcfeC71Th2wzmuAybqRTrAqLRMRBw5nvU0p9r7n5NRFZq5R6tZlnf133WaXUIwAeARo59BhsThTb8nG//cZGypj59bm2xZsLBYFaULFH0Hd85OolW7LUr+a6dZfhf//yXKjrbXWapnSHWwxWEEGtvoDHjr/cttBHp4U/fm17p2crgccNSssxJUPiwEblIgC+CeB5pdQ/tPzqIIC7m6/vBvB0/OblB50Spr6gUBp02qLMTlUp3597dem1aXSQBmGduUulWsN9zd7squX9+olDmDp8GjdsGkbRKSw5cdOqTZ2MVvzSZjbRf1BajikZEgc2o/7tAP4SwA4ROdn89wkAkwD+XER+DuDPm++JAZMzqc7X29I1Xl17WKq1+tKQvTUdlBQfvGK1dnsnIxHvZ1snUPc9e8aqwVoncxl+D0YbxxuUlmP1KYkDG5XLv8AcNP7beM3JL5cVHa1s77KiY0zXfOfZM5HP5x2yj42UE5FP3rVtHY6+kOxcic2DotPCH/f/5979J7W/t3G8fmk5W0UVIX6wUjQhTNWTv3u7rh2u7x1rr1QNS62+gAefObU00XburXc6Op4NB05UMpOzdxEAt17feSvdsZGycaTTqeNl9SmJAzr0hDA1jFpU5hxsHE2mzs/Xl3LPcTfeMq1FqtvebfzOqIDYRg3dcrysPu0u/aIg6vv2uUnhpzYxKTCSVKi4KxDZpHkEwEO7tuI+Q/phQalEi5mcAcGuD1/t26o4rly0d8WkOCs+s7aMW14qW/tJQcQIPSGC1CY6h6P7jBuJxjnJ6ZbKj16zJjC6FjT6vI+NlAMXbA5L5Mi++TG/zpKmRaOj4HbPdBu05c0pAJ0XUmWJflIQMUJPCPdLf/8Tc1pZnc45BkWDQZOczgAQlGVx0wXuF9gk+QMuOvO9Y5sBxLuEn7viDwDct/9kqAdCfUFpteetxLUwV5ai1iBbOrE1T4t49JOCiA49Qdwvgm75MlMO1m8YHuRQL13l4O36YtuybZeuWoHqfH3Zl3zrgz8IdMzeXLS3sCcq3mZcJiWJH37OHGgsGu0S1dFlaegeZEuntubJCfaTgogpl4SJc/LLPZYJncZ96vYtmP3yjcvSBdOzlcBOiC7eL7SbfohC0Sng4V1b29IWUdJJQemay4qNlEsnqYQsDd2DbOnU1jwt4tFPCiJG6Cmgi7qjRo1jI2VjhHxlyW7l+zAOSfeFjtLRsbXfjJfxnRuNaZdS0cE7FxbbRji3Xl/GgRMV4yijWqvjgenncPSFs5FTCVmKWoNs6dRW3ejP1glmKS0FdHciO2vQoWeATofHnXz5APsvuemYUTo6Lirl+1AbXFnAW+8ud7xFp4A9t1y7dE7vl3P0mjW+6R9vL51WKtUatk8e8f3CZ2noHmRLp7ZGdYJZSku1kjUFUbegQ88AnU5A2Xz5dFGT+xkbZ+y36ESUCNXrWLyO4K13F+AUBKtXrsAbtTpKgw7eri8s5deHBh08tGvrMnvcL61pwRGFxshAl28XYMkBmpxQpw/OOAmyJQ5bozjBPE2m9iJ06BkgjqG835dPFzWNPzkHSEMh4oczIJi6fYvvlzGsXl6ANseicwT1BYXVl6zAnluuxfiTc6gvXrT1/Hwd40/NAWiP/PzsWVAKRaew7Fy6dJHOCWVp6B5kS1q2Zikt1Y/QoWeAbg/ltc5y0S5JUl9UgdHV+M6NGH9qLvDh4OLq2FsxOeBKtYY9B09p7a0v6G3zy8G7I41WR2c6t84JZWno3okt3cpzZykt1Y/QoWeAbg/lO42Ogj4/NlLGnoOnrJQyzoBg9Jo1y7YFTar6HVfnPEz95wXADZuG2xyhSc/v54SyNvHnxS+XDaBree4spaX6ETr0DNDt4XGnLQRsoqs3LGWPuoi/02Xytj74A4hgmbZ+79hmvHj2TRz75bml/RQazcNGr1nTUW48qxN/rQTJFruV585SWqofERVXCZ0Fo6OjamZmJrHz5ZkwEaLXAQHAgDQag3nxbm+t4PQ7X5jWvALgxcmblt6bJjGj4soYTX1p3FYHrbTez9KgA6UaD6kw16o7blqY7qmr1jf9rvX/hWQHETmhlBoN2o+FRT1I2OIYXTHTe1fpe5u8d5XTVvQEIPB8fn1nvHgjfrfoJy5q9QXsO25uMmbKjR+b2IGHdm3F2/VFVGt147X2wsSfX2FQnoqGyHKYculBokjDvHnjDROHtPu9Uavj5O4bAVyMWnXRqPd8uqH2DZuG24p9dKmMbnTb9Rt4tjou70jnrXcuBN7bXpj4C0ojxZnnzvp8Qj9Bh96DxBEhhl202OZ8OtWFW+zj1b+3FvGYWt52C9cGXS7cROu19sLEn00uOw4nHHY+IS3n3y8PHTr0HiSOCDHIKelGAVHO53XyYZzo0KCDwZUrYu0Jv3plYcmePQdPWXeKbL3Wbk78xel4/GQFZJoEAAAN6UlEQVSNcckvw4wW05pM7oVJ7LigQ+9B4qoCBMxOKSjaD3O+Vic1YKjU1FGdr+OmD6317dESBqcg+MqnNy/ZZNuQTHet3dCAp+14ojxMwowW06oiNZ333v0nMXX4dFej9aRHBnToPUhcEaKfU/KTOvq1AfDidVK2zhy4KDO89fqy72pENnht9mtI5o4M/NJE3UhRpFk2H/VhEma0mNZkst/xu/nQTOMBTYfeo3S7YtE0Cgjb6tcmdeNHrb6Ax46/jEWlMNTs5xJ2bVSdnDDIiXidufeLee/+k7h3/0mUig723HKt9T3xc9rddnh+0WLUh0mY0WJak8lBdRjdemim8YCmbJFo0Ukdo/Rtt3FGAwEqlwWloNDo3xLWmTsFMToXE+fnl0sW/XLt1Vod40/OWS/N5tfiwEQcss4gqWuY9gethPk7SasvedDyj0B3RglpjEgYoRMjcYwCTNFRQQSLSqE06ODNty9gsVsFbp7Dtkoxve0GTE26gkYYNv1uIph60a4YZJ1BlaOm1gtRJr799nNtSVJt0npe04OrG6OENEYkdOikqwSlbrZPHumqbLG+qLDn4CmtE1e46MjKHbZH0EVdrSmOy4pOZMdcjeH++EWLptYLuq6YnZJWczP3vDo5brdGCWnIW5lyIV0laEgex/AzyE9Wa/UlZ+11XK4zPzaxw7j03dCgEzhkN/V3d1Mc1Vo98oMrjojOrzrU9H+gkD9ZX1ypxKydy4UROuk6UdU0Nrh9W46+cDbycdzPmSKq3Tc3VkkydZR0Btrz9J1OBi8duyB4650L2DBxqKMUhV+0aEpFRFnbtRdIcpSQ9IiEETpJFZsJKy9uRO5GPHvHNuPYxA48vGtr6GMBFxeYHhsp49bry0vvCyK49fry0pfy5O4b8fCurRgavDhJWSo62gVAoo48nAHB0KADQWNkAIVlfWXu238S6ycOYfvkEeuJWPfaTNFiPy2inHcYoSdMv5Qgt+J3ze7P+5+YM2rUh5rdD93ouDToYPfN7VJB3aTb/LsXAlMd7nmnZys4cKKy9H5BqbZ2u7YRl83Iw10Oz/1ZaubZ3TbAb71zoW1hD/ddFE2zyfaok5X9+Lecddg+N0FMEzLdzqulie01T89W2paZAxoph11/crW2yZfNfbPpSePm0E1tcXWFRnGctxVnQKyWBDTZnjT9+LecJmyfm0GCpGN5xPaax0bKmLp9C0otmuuhQQdTt23B0RfORr5vrakGoH0CtTW1YEqTnJ+vW7cq1p1X0EjNuKmUgkbuUl9UoZ25n83dph//lnsBplwSpBf6aMdNmGs2pQTu238y1LH9juuXJrCdoLWt9jNdj6l1cRTSatnbj3/LvQAdeoL0Qh/tTtA5yziuOc775pcD1ylBTFSqNYz87Q+WLXtnm2oIo+xp7TjpLf5Jc+Iy73/LvQpTLgmSZzWBqbT8hk3DHV9zUvdNpwQp+ZTde1sE2KpObJU9rmTy2MQOvDR5Ex7atTVRTbMfef5b7mU4KZoweVUG+K2z6WqdO7nmB6afw2PHX15Shdzxkauxd2xzXOYbCTO5WSo6WH2J3eTp9GzFV9lTEMHXPtsuh4xC1L+5oM/l9W85i9hOigY6dBH5FoBPAnhdKfXHzW1rAOwHsB7ASwA+q5Q6H3QyOvT84rcosXfh4bCOIG1FxfRsBfca8vh+BNnod9y4FmyOeu+6cc/5AIhOnCqXbwP4uGfbBIB/Ukp9EMA/Nd+TPsZ24eGwC1wD6SsqxkbKkaomg2wcGykvK1JqJa5cdNR7F/c9j/L/TsIT6NCVUv8M4Jxn86cAPNp8/SiAsZjtIj2GbU41iqPIgqIiSkUrEGzj7puv7TgXPT1bwfbJI9igqSCNeu/ivudpP5T7haiTou9TSr0KAM2fV8RnEulFbBsRRXEUQdG/n0OLC52u3IagSNt73KFBB5esGMB9+09aXUtQ5Gs7crL9fdSRQxYeyv1A12WLInIPgHsAYN26dd0+HUkRm7L4KHI3v8ZSSS7z5b0+00Sw10bb40a5lqBVcaK2cI279StljskQNUJ/TUTWAkDz5+umHZVSjyilRpVSo8PDwxFPR/JCFLmbX/Sf5lBedy3exmGdLtfXaToqagvXuFu/UuaYDFEj9IMA7gYw2fz5dGwWkVwTtRGUKfrv5lA+SJUR9wo8UdNRQZFv1BaucbZ+HRspY+bX55ZJT91OliQ+Ah26iDwG4GMALheR3wDYjYYjf0JEPg/gDIDbu2kkyRdxOoo4hvI6xw20LwytS3+kfS1BqRG/h1KSMkKbTpakc1hYRHqaTvXSD0w/h33PnmkrqV/lDGjb7obtbhjGaZqKmEpFB3tuaW8XHHQOv3sDIFFtv1/hWbe6ReZJ926rQ2cvF9LTdJL2mJ6ttDlzwH9h6DCpnLCTnO62B585texhUq3VAz+n2x6Uk/ebTI2CnwNNWuWS5GR5lqBDJz1P1LSHaXFkP8KkcoIUKDrciV7v6CCKs43iRKM62CAHmrTKJcq9zwNszkX6Fj/nVSq2LwwdVpWRdlGPn5Y8bp150GggaZVLv+reGaGTvsUUNQqAPbc0FobuJAcbNSqN8jlduiNowjROnbmNfBKwv5+d5r/7VffOCJ30LaZy/qLTqNScOnwa4zs34sXJm3BsYkfooXrUqDTs50zVogCMWvJOdOa6ylybiN8tdLqyVMQr1RqmDp/WVsLG0felX3XvVLmQvqY1EiwNOnjz7eULMyfZYdBri1LAG7XgBTSSVJCYlDO3Xl8OXPfVVpEU1/UkoXJJSkkTW/vcOKFDJ1kmDWmdSyfyyzCtizulk773tvc3yevphCTbOlO2SEhI0pxI60SVkWS+2O8eBamNbO9vr+S/s6ikYQ6dkCZxKz/C0MnDJMl8sc09MnW/tL2/vZL/zqKShg6dkCZpOpJOHiZxN9LyI+ge+U1o2t7fJK+nE9IMAEww5UJIk7ibbYWh03a1cfaUCToPYL5HfmkIN09uc3+Tup5OiLvFcBxwUpSQjNBNxURSaoxemdCMi6ypXBihE5IRuhWVJtnXpFcmNOMiayMJ5tAJyTlJLgLSKxOaeYUROiEJkGYr1yTVGKYcO9DQoeehlW2WoUMnpMuk3co16TSINw2R9vX3E0y5ENJl0lz3FEg/DZL29fcTjNAJ6TJpF6CkKccE0r/+foIOnZAukwXlR5pqjCxcf7/AlAshXSbtlEfa9Pv1JwkjdEK6TNopj7Tp9+tPElaKEkJIxrGtFGXKhRBCcgIdOiGE5AQ6dEIIyQl06IQQkhPo0AkhJCckqnIRkbMA3gLw28ROGp3LQTvjpBfs7AUbAdoZN71g5zVKqeGgnRJ16AAgIjM28pu0oZ3x0gt29oKNAO2Mm16x0wamXAghJCfQoRNCSE5Iw6E/ksI5o0A746UX7OwFGwHaGTe9YmcgiefQCSGEdAemXAghJCck5tBF5OMiclpEfiEiE0mdNywi8pKIPCciJ0UkM53ERORbIvK6iPysZdsaEfmhiPy8+XMoTRubNuns3CMileY9PSkin0jTxqZNV4vIURF5XkROicgXmtszdU997MzUPRWRVSLyYxGZa9r5YHP7BhE53ryf+0VkZUbt/LaIvNhyP7emaWdklFJd/wegAOCXAN4PYCWAOQB/lMS5I9j6EoDL07ZDY9dHAVwH4Gct2/4ewETz9QSAv8uonXsA/Oe0bfPYuRbAdc3X7wHw/wD8UdbuqY+dmbqnAATApc3XDoDjALYBeALA55rbvwHgP2bUzm8DuC3t+9jpv6Qi9A8D+IVS6ldKqXcBPA7gUwmdOxcopf4ZwDnP5k8BeLT5+lEAY4kapcFgZ+ZQSr2qlPpJ8/XvATwPoIyM3VMfOzOFavBm863T/KcA7ADwVHN7Fu6nyc5ckJRDLwN4ueX9b5DBP8omCsAPROSEiNyTtjEBvE8p9SrQ+OIDuCJle/z4TyLy02ZKJvXUUCsish7ACBrRWmbvqcdOIGP3VEQKInISwOsAfojGqLyqlLrQ3CUT33uvnUop935+pXk/HxKRS1I0MTJJOXTRbMvqU3G7Uuo6AH8B4K9F5KNpG5QDvg7gAwC2AngVwNfSNeciInIpgAMA7lVK/S5te0xo7MzcPVVKLSiltgK4Co1R+R/qdkvWKo0BHjtF5I8BfAnAJgB/AmANgC+maGJkknLovwFwdcv7qwC8ktC5Q6GUeqX583UA/4jGH2ZWeU1E1gJA8+frKdujRSn1WvNLtAjgvyEj91REHDSc5D6l1PeamzN3T3V2ZvWeAoBSqgrgR2jkpksi4i51manvfYudH2+mtpRS6h0A/wMZup9hSMqh/x8AH2zOeK8E8DkABxM6tzUislpE3uO+BnAjgJ/5fypVDgK4u/n6bgBPp2iLEddBNvk0MnBPRUQAfBPA80qpf2j5VabuqcnOrN1TERkWkVLzdRHAn6GR7z8K4Lbmblm4nzo7X2h5iAsaef7U/0ajkFhhUVNW9TAaipdvKaW+ksiJQyAi70cjKgcaC2h/Nyt2ishjAD6GRme41wDsBjCNhopgHYAzAG5XSqU6IWmw82NopAYUGiqif+/mqdNCRP4NgP8F4DkAi83N/xWN/HRm7qmPnXcgQ/dURD6ExqRnAY1A8Qml1N82v1OPo5HGmAVwVzMKzpqdRwAMo5EePgngP7RMnvYMrBQlhJCcwEpRQgjJCXTohBCSE+jQCSEkJ9ChE0JITqBDJ4SQnECHTgghOYEOnRBCcgIdOiGE5IT/D0kxI7djPluEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb9ad63ea20>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "boston = load_boston()\n",
    "plt.scatter(boston.data[:, -1], boston.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = torch.zeros(1, requires_grad=True)\n",
    "b = torch.zeros(1, requires_grad=True)\n",
    "\n",
    "x = torch.tensor(boston.data[:, -1] / 10, dtype=torch.float32)\n",
    "y = torch.tensor(boston.target, dtype=torch.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = w * x + b\n",
    "loss = torch.mean((y_pred - y)**2)\n",
    "\n",
    "# propagate gradients\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The gradients are now stored in `.grad` of those variables that require them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dL/dw = \n",
      " tensor([-47.3514])\n",
      "dL/db = \n",
      " tensor([-45.0656])\n"
     ]
    }
   ],
   "source": [
    "print(\"dL/dw = \\n\", w.grad)\n",
    "print(\"dL/db = \\n\", b.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you compute gradient from multiple losses, the gradients will add up at variables, therefore it's useful to __zero the gradients__ between iteratons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJztnX+8VHW199/rDHP0AMYBxR8MIGheMEI9iUbRo0LdKEE9pWalXeupfD1Z3bTkdrx1Fc1u57lU1q0nS6ublRYKdiSwh7qC+UipgYBISCgYMqCiclDgCOfH9/ljZg9z5uyfM3v2ntlnvV8vPJyZPXuvvWU+e+211nctMcagKIqiJIuGuA1QFEVRwkfFXVEUJYGouCuKoiQQFXdFUZQEouKuKIqSQFTcFUVREoiKu6IoSgJRcVcURUkgKu6KoigJZEiUBzvmmGPMhAkTojykoihK3bNmzZqXjTGjg3wmUnGfMGECq1evjvKQiqIodY+I/D3oZzQsoyiKkkBU3BVFURKIiruiKEoCUXFXFEVJICruiqIoCcRXtYyIPAe8DvQCPcaYaSIyClgITACeAz5kjNlTHTNzdKzNsmD5ZrKdXaRE6DWGTHMT82ZPorUlU9jmX+97kgPdffbnAhggJcJH3j6OW1qnOu535uTRrHx6Nzs7uxhTcpxybfe7L7vtgUD7UBRl8CJ+JjHlxX2aMeblotf+A3jVGNMuIm3ASGPMl932M23aNFNuKWTH2izX37eBru7eAe81pVN844NTAfjiPevoCzBcasbJo3hi+17b/TodJ6ig2tnuti+77dMpAQPdRSdXrj2KotQXIrLGGDMtyGcqCctcBNyZ//udQGsF+/JkwfLNjgLc1d3LguWbWbB8cyBhB1j17Ku+hL34OEGxs91tX3bbd/eafsJeiT2KoiQfv+JugN+LyBoRuSr/2nHGmF0A+Z/H2n1QRK4SkdUisnr37t1lG7qzs8vzfa9twqCcYzh9JujrYdmjKEry8SvuM4wxbwPeD3xWRM7xewBjzO3GmGnGmGmjRwdaPduPMc1Nnu97bRMG5RzD6TNBXw/LHkVRko8vcTfG7Mz/fAn4DXA28KKInACQ//lStYwEmDd7Ek3plO17TekU82ZPYt7sSTRIsP3OOHmU436djhMUO9vd9mW3fTolpEtOrlx7FEVJPp7VMiIyDGgwxrye//t7gZuBJcCVQHv+5/3VNNRKGnpVywBVr5YJWvlSbLufzzhtH2QfiqIMbjyrZUTkJHLeOuRuBncbY74uIkcD9wDjge3ApcaYV932VUm1TFgEFWa7zwepfFEURamUcqplfJVChkXc4m4nzAJcPn08t7RO9bWPGe0ryNokMTPNTaxqmxWWqYqiKAWiLoWsO+xKDA1w16Pb6Vib9bWPMCpcFEVRqs2gEncnATbgu148jAoXRVGUajOoxN1NgP163kErXxRFUeJgUIn7vNmTcKqU9Ot5t7Zk+MYHp5JpbkLIxdo1maooSq0R6Zi9uGltybD6769y16PbKU4jB/W8W1syKuaKotQ0g8pzB7ildSq3XnaGet6KoiSaQSfuiqIog4FBFZaBgbXu2c4urr9vA4B674qiJIZB57kHbb+rKIpSjyTWc3dqM6CLkBRFGQwkUtzdQi9jmpts2wfoIiRFUZJEIsMybqGXmZNHD6h110VIiqIkjUSKu1OIJdvZxeI12X417gJcfKbWrSuKkiwSKe5OIZaUiG3jsJVPlz/+T1EUpRZJpLg79X/pdWhvnO3sYkb7Ct+dIRVFUWqdRIq7U/+XjEvS1Eq6qsAripIEElktA879X0qHdRRjJV01/q4oSr2TWHG3o3QOqx1a764oShJIrLg7LWKy/jiNy9N6d0VRkkAixd1uEdO8ReuZv2Qje7u6GdPcxMzJo1m8Jjtg0LXWuyuKkgQSmVC1W8TU3Wvo7OrGcLje/eIzM9r6V1GURJJIz91P3Lyru5eVT+9mVdusCCxSFEWJlkSKu1P/mFKiSp46xf8VRVGqRSLDMn7j5lEkT634f7azqxAS0np6RVGqTSLFvbUlw8ihac/tZk4eXXVbtH+8oihxkEhxB5hz2gkDuj+WEkVPGe0fryhKHCRS3DvWZgd0f7QjCoF1Cv1oPb2iKNUkkeJuFwqxIwqBdWpipvX0iqJUk0RUy5RWo/iplIlKYItbHmi1jKIoUVH34m63GlXANiSTEqHPGEeBrVbJolMTM0VRlGpR9+JuF4IxMEDgm9Ip1xWobnNXVZgVRak36j7m7pQUNRCotYCWLCqKkiTq3nN3irFnmpsCtRbQkkVFUZJE3XvuYVWjaMmioihJou7F3WmkXtA4uZYsKoqSJHyHZUQkBawGssaYuSIyEfg1MAp4AviYMeZQdcx0J4xqFC1ZVBQlSQSJuX8B2AS8Kf/7/wZuNcb8WkR+CHwSuC1k+yKl2iWL2h1SUZSo8BWWEZGxwBzgx/nfBZgFLMpvcifQWg0Dk4J2h1QUJUr8xty/A/wL0Jf//Wig0xjTk/99B2DrgorIVSKyWkRW795d/UZdtYqWWiqKEiWe4i4ic4GXjDFril+22dS2T5cx5nZjzDRjzLTRo6vfYrdW0VJLRVGixE/MfQZwoYicDxxJLub+HaBZRIbkvfexwM7qmVn/ONXja6mloijVwNNzN8Zcb4wZa4yZAHwYWGGMuRxYCVyS3+xK4P6qWVklOtZmmdG+golty5jRvqKq8W8ttVQUJUoqqXP/MvBFEXmGXAz+J+GYFA1RJzjDqsdXFEXxgxjjNdIiPKZNm2ZWr14d2fHcmNG+IpS2BYqiKNVGRNYYY6YF+Uzd95YpF6dEZrazixntK7QWXVGUuqbu2w+Ui1MiU0Br0RVFqXsSLe5uCVO7BKfdkA+tRVcUpR5JrLjbJUyvXbiOCXmhBwYkOJ2yD9nOLvXeFUWpKxIr7k4TmqD/lKVVbbPY1j6HVW2zyLjUnF+7cB1f7dhQLXMVRVFCJbHi7rXy0y7cYheqsTDAXY9uVw9eUZS6ILHi7mflZ+kNwKpFd8KAxt8VRakLEifuVhI129ll2wCnGLsbQGtLxjU8E2YvmChXyCqKMrhIlLgXJ1Eh52lbAl8q9G5L/+fNnuR4Y2gQ8SXCXsJdzgpZvRkoiuKXRIm7UxI109zErZed4Xvpf2tLhsunj7cV+F5jfImwm3B3rM3ypXvWB2oBrP3gFUUJQqJWqLq11Q06ZemW1qlMO3EUX7pnPb0lLRosEXban1fv9uvv2zBgn17n4LZPXUGrKEopiRL3StvqdqzNctNvN7LnQDcAzU3pwCLs9t7Ozi5bkfZjq/aDVxQlCIkKy1TSVrdjbZZ5i9YXhB2gs6vbcXs7EbZi4k6LocY0N7mKsZutTqKv/eAVRbEjUeJeSVvdBcs3093rr0OmnQiXJnOdPuMkxikRV1u1H7yiKEGoy7BMx9osC5Zvtu3c6De2XroPJ1G2sPrOpES4+MyBx3ALt2RKbLz+vg39tm1KpzxvQtZ7TuetKIpSTN2Ju+UhW+JY3ErAr9DZ7cMLy6fvNYbFa7JMO3FUv+M5hVsE+vWHr0SkgyaFFUUZvNSduIdRNeKV1PTC7njNQ9P94vXFr5eiIq0oSrWpu5h7GFUjYVSYlO7DaaBVhIOuFEVRCtSd515puaPbPoLaUcxeh8oap9fLxS3foCiKYlF3nnsYVSNu3R8trNWpI4emSTf0X6tqd7woShV1laqiKH6pO8/dT0LSy7u128fMyaNZ+fRu28/48ZZnTh7NXY9u71fjHnapoq5SVRTFL3Un7uCekAxSTbP/YE/BA1725C5uvGAKrS2Zgphfu3BdQcyLK16Kj1W8orUYAduSyUrQVaqKovilLsXdDT/ebcfaLPPuXU9332E/e8+Bbq5ZuI7r73uSnj5TWNDkdHOwVrQ6LXwywMqnd7vaGjR+Hka+QVGUwUHixN2Pd7tg+eZ+wl5MV3efzWu9zF+ysZ8Q7z/Y47mi1c2jLqdef97sSbYLoHSVqqIopdRdQtULP4nNcsIYnV3d/RKZbn1nvGwB786RdlTSXkFRlMFF4jx3J+925uTRzGhfwc7OLhpEHLs9hoWbR92xNutYiul149EFUIqi+CFxnntrS4aLz8yQklz5YkqEt40fweI12YLnXW1hb5DDXrjTBCYnNH6uKEoYJE7cO9ZmWbwmWxDwXmP407Ov2rYbcBqll24QRg5NF0IfI21aCNjRlG4gnRKscL5dHbpb6wONnyuKEhaJE3enUXt2WCP4rpg+vl8ce8Glp7P2hveyrX0O82ZP8mwh0NyUJtPcRFd334Aka2kc3S3sovFzRVHCInEx96DJ0mxnF4vXZAvCWlzj3jw0zb43ehwrayDn5e8/1OOaYC22yamcMdPcpMKuKEpoJM5zd4pZO4Vg4LB3/dWODVy7cF0hNr/nQLetsKdECl7+8COHeJZEFtukQzcURYmCxIm7k3hePn18IclqR7aza0D7ACf6jGFb+xxWtc2i02Z1aumxi4U7jnJGa/zfxLZlzGhfob1oFGUQkLiwjFvvmbse3e76Wb81NMWeuFuHydIJTMU2RhWCCWO4SSXH1g6WihIPiRN3cBbPMFr9lnriTnX1tZIcjavZWJw3FUVRfIRlRORIEXlcRNaLyEYRuSn/+kQReUxEtojIQhFprL65leGn1a8bdkOsww6zhB1CiavZWDkrcBVFCQ8/nvtBYJYxZp+IpIFHROR3wBeBW40xvxaRHwKfBG6roq0VUxqyCbKUyc0bt3tSsAtJFB/bqVVx2N5uXM3GtIOlosSLp+ducuzL/5rO/zHALGBR/vU7gdaqWBgyrS0ZVrXNYlv7HDI+BS6oN251jCzuRfOle9cz797+r/lZ4FSpt2v3tJJuEA4c6qlqgjWK4SWKojjjq1pGRFIisg54CfgD8CzQaYzpyW+yA6jpQKpduMNPmCbT3MSqtlmBPOebfrtxQHlkb58ZUFbpd4FTJd5uadiouSkNkivz9DvNqZxQkZZ8Kkq8+BJ3Y0yvMeYMYCxwNnCq3WZ2nxWRq0RktYis3r3bvb95tXAaTwe5VaFuzJs9KbC42Q3vcKJ0gZMdlXq7xU8rw44YWJfv9nRQ7mg/7WCpKPESqFrGGNMpIg8B04FmERmS997HAjsdPnM7cDvAtGnTqtuxywG3cMeqtllcs3Cd42dX//1VFq/JVq3qo0GEiW3LCqP+io8F4Xu7QZ8OKqm20Q6WihIffqplRotIc/7vTcB7gE3ASuCS/GZXAvdXy0gA9m2Fv1wNi46Bu6XoTwqWvRVefMjxo26C5uWB3vXo9sBx8OYmf43GINfYzPKIF6/JcvGZmap6u0GfDjQxqij1iR/P/QTgThFJkbsZ3GOMWSoifwV+LSK3AGuBn1TNyuwD8Mgl0GsnKH2wdyM8OBPeOh9Ou3HAFm4VI17JSqdHjWxnFzPaV9gO1p5/4ZQBY/xKSdn0lO/q7mXl07tt57UWU8nioKDTnHS0n6LUJ2Kq3Nu8mGnTppnVq1cH+9C+rTnP3FbYbTjiGBh/KZx6HQw/yXGItVXaeO3CdYFKIr1Ip4RhjUNcG4lZTRDsjivAtvY5jgJeWi5ZfC5Bqnn83hzCOJ6iKJUhImuMMdMCfabmxf0vV8OW8srnTeE/cKCvkS/v+AJL955Lc1Oa+RdOobUlw4z2FRWvWi0HO88dDrcsKBVUr5uGVdVTDbSNgKLESzLF/TcZ6LLN1QbGGOgxDXztlTZu/sLXAXvPNC4sj3jB8s2BbziWx18OKt6KUtuUI+613xWya1douxKBdEMfNx3z74WEbOuWU/n5u58JlAQNk+L2wVaoo5xkZbkx8HJLHRVFqW1qv3FY0wmhee4W/Tr/9rzOWTs/z9o3w76+Jq7f8TmW7j031OO5YbUPLqacBmcHDvXQsTYb2OOOq7FYraBPLUpSqX3PfexFkRxGBI5KdfG98QvYNnUuW956IVce7VzdaXnbxSP6Rg5Nk25wGwsyEDuPu5wGZ3sOdJflcQ/mUkd9alGSTO177qdeB1t/5r9apkIsrz4tfcwfcwfzx9xBV18j/5JPxkIulPKtD51u6+FZnqAfz9upBLG0wRmC5xxXKM/jHiyljnYe+mB/alGSTe0nVMGjzj0arMvUB3zvxcv40Z6Pu5YDTmxb5lpiKcDl08dzS6t7+wM/+yrdb5DE6mAodXQ6R6ckeiXJaUWpBslMqAJkzoc5T8Epn4HGo2MxQST3JyXwheMW8tfJ7+ddT76Fz37724XH+OIeNA0uI/0gV6H5q8ee9xUCCOJFB/W4B0MPGCcP3WnsYtKeWpTBSX147k7UkEffLUfQtuOfue/VYMnYUi/ZqQ+8n3LNODzuekhIuj35lHrwSXtqUZJBcj13J4o9+iFHxWKC5dE3cpBvZQYmY92GckP/PjVe3Svd+s/bTYmqNvWSkHTyxK2nlCQ/tSiDl/r23Et58SF45MNw8MXqHcMn1mU1wLaRVzL3zx/2jPE6rZYtXn3q5IXGESf2Y28tMBjyCkqyGXyeeynHnQcXvwAfNfDOuyA1LDZTLI++QeDkzjv566nv596T5jGu8YUB21qepZ+yxFqacFQvZZSDIa+gKKXUfilkuUz4aO4PwF8+C1t+EKs5Apw1fBMPT/oU0L/qxoqr+ylLDNrVsZrUUxml9pZXBhvJ8tydOOv/5Lx5y6OPKT4PNlU3p76f1k1jYf1XfY2mqyUvVEfpKUrtkqyYe1D2bYXHroIXH4zbkgIHeRNffOFGHnhp0oDqk8vv+DOrnn21sG26AXr6iLVKpR6qZRSl3klmV8iQ8BShP10Bz90Vi22ODDkKzv4hTPjoAGG3o7iVsaIoyUHF3YHA1RJPfweeuA5jctt7VDNGwv7eI/jyjn/2bGoWRRWIeuuKEi0q7g6UU7LXsTbLNQvXMXfEH1kw9jsc2ZAbkhGn0Fv/q3qMcMuuT3HnK/ZN1azzcpvmVK44+7lRqvgrSriouDtQTm14y82/HzCa78qj7+crJ/yYtOT2VgtC32fg5l2f7if0Vt+aux7d3u+8m9IpLj4zw+I12bJrvr1ulFpTrijho3XuDgStDe9Ymx0g7AB3vnIR//DUb5m4YSnzd36a7r4GjPHXsTFsClU3DTB/zB1smzqXDVMuZe6IPzKiKT1A2CG3Gvbux7bb9ln50j3rmdi2jBntK1xXmHrVtrt1WlQUJToGhbgHLdnzI0R3vnIRpzy1hIkblvL57fM42DskdqG3+tGvffNsNk1pZe6IPw7Yts/Bvl5jfLUQ8LpR1svCJiUcipvleTkGSrQMCnH3Wxtu/UMNOgVp6d5zmbSxoyD0+3sbYxf6I1M9hcEj9zisjHXCzdP2ulHW0gpapbrUS2+hwUpyV6iW4LVCMaxB2Uv3nluoaLnm2F/w+eMWFu6gUcforeOdnV8Za4DnDo6hLft5Ht/v3kfeztPuWJvlpt9u7HeNRg5Nc+MFUwqJ2gOHegbaAcycPLqSU7G1xStpq4nd6qLDTmqbQeG5+8HuH2qlfOelj3HyhqVM3LCU/3p5Dn15bz4uj75B4KQjd7LwpOvZNnUuf53yQS6yCd0ANIj088A61maZt2j9gFzEvjd6Cu9ff98G21yFARavyYbm0fnxGNWrrD4agqttVNzzBA3FBOXmXZ/hpLzQX7b1G+zpHh576GZo6hDfGb+ArVPncsMJt/XbpteYfmK4YPlmunsHGtvdZ1iwfLPnzTHMpKqfpK0mdquPhuBqGxV3cl5elBGTx/dPpWXTrwtCv/3gsbF79J84Zhnbps5l29S5bJzyQeaO+CNd3b1cs3AdE9qWud78dnZ2+fLWnLYJmpRz2k+2s6vwefUqq4/2FqptVNzJeXkx6CqQE/pzNv+0KBl7RCxCb3nzIjAsdaiQjP35hK94JmPHNDf58tbstiknfOJ2LOvzzUPTgT+rBKOWmtgpAxkUi5i8CDKAOirGNb7A98f+O6cN2wrUxoKpHtPALbs+2W/BVLpBWHDp6QDMW7TeNnQDzguZyl097JX8bm5Kc7CnTxdTKYmgnEVMg6Zaxg2nvuRx8vyh47lo638CMHfEH/nW2AU0NsQj8tYx09LH/DF3MH/MHfQY4du7r+L1iZ9lwfLNrtcv41KpUk74xNqP23H3dnVz62VnaLWMMmhRz53wyiCrzbjGF/hW5pucNfzpwmuxevT5/+zra+L6HZ+zbWrm5IFbZYpO4mz3ObvSxvlLNtLZNbBCp5qj/rTEUoka7S1TAcVf2Oahafa90UO303LOgAhUJewTdx19MdY/o9d6h/KV7Gf7Cb3AgMZlbjdTu/CJ3WfSDUIf0Fvy/8kKFVVDcLV3jhIHKu4hYnWFDIMrpo9n2ZO7bGvAw6IWhb7Uo7dE0Mtjt/OEg6wcHjk0zdob3lv+CbhQL0PBlWSh4h4yTl/kBnHu0VJKsdB4hSLC4uxhG2jPfI+JR+wEakPorWTsf/d+mJ35yphS3Lp0Bkl6u+2nUsrpMKoolaJdIUPGro5XyAn7yKFpmpvShRKwGSePGlArL8Cc004o/N7akmFV2ywyVS7He3z/VGb97fbDC6Z6hsVSQw+HyyvTDblk7CPj3s3GKRfbNjVzK1MMUsJYzXJHXbij1Asq7i4U1/FC/9j5ngPdHOzp49bLzmBV2yzu+vQ7uHz6+H4Cb7fs3qn/SrV4fP9UWv66kHM2/5hfvHw++2qgqdnQ1EG+N34BT73lYs4etgHwXvxid6NNNwiphv631HRKqrqIRhfuKPWCp7iLyDgRWSkim0Rko4h8If/6KBH5g4hsyf8cWX1zo6fY27brj168nH3l07tdt3Hrv1JNRGDHoeP5t51X89aN9xU8+hcONscq9MOHHDzc52by+5mz6RR47m7b7e0WzFx29riB/4CrfC66cEepFzxj7iJyAnCCMeYJETkKWAO0Ah8HXjXGtItIGzDSGPNlt33VW8y9GD+xVq9tymknHAVXHn0//3rCT2iUPqA2YvR9CFuO+TyTZ3/Xcdt6S25qCaVSLlWJuRtjdhljnsj//XVgE5ABLgLuzG92JznBTyx+Yq3lDrKoNpnmJkY6LMeH3OCRSfnBIzUzYUoMk17+T8zdAr8ZCy8+NGDbqPvHVDKYQrtUKlETKOYuIhOAFuAx4DhjzC7I3QCAY8M2rpbwE2std5BFNbGO71eoSydMxT54BKArCw/OhLsF7m0uhG7cbqZhTwiqVJy1S6USNb7FXUSGA4uBa4wxrwX43FUislpEVu/evbscG2sCP7FWr23sxB+ql9UuPv5em1WcXizdey5T8jH67754Gb1GCkIfV/UN3XvhT5fD3cLD49/Dp0bf3+/tpnSKmZNHh+4lVyrOXk8ZOq5OCRtfde4ikgaWAsuNMd/Ov7YZOM8Ysysfl3/IGONaMlDPMfew+GrHhgHDq9MpoafXhJoLPOXYYfzhi+cVfg873j+u8QWWn/K/aGroqYkWCAfNEP5j1yf46SsXkRKh1+bfdSWxeLc6++d81Le75QfmzZ6kq14VV6oScxcRAX4CbLKEPc8S4Mr8368E7i/9rDIQu4qa7l5D89C0bU19uWx5aX8/78/pqaFcnj90PG/Z2BH/hCnyM2Mbevi3MXewbepcVv7DJwollsVUEot3CgEJ+PKy3UJ2GrJRqoGfiMAM4GPALBFZl/9zPtAO/KOIbAH+Mf+74oGTwHQe6B4Q0imtmw/Kl+5ZXxAeK2QUNsUTpmoiRi8w/oiXCiWWa9/y4cKCqUpyHvNmT7L9f2HAlwi7hex0sIhSDbT9QMSccdPvbbsYNjelWXfjwH4ol9/xZ1Y9+2rZxyttohVFOeYV08dz9PO3cfXIH9VWeaWkSb3tP2DyNWXtZ0LbMtvXK209UG8lnUr0aPuBOsBJ5F57o9v28f65VyoT4u4+w/wlGwsJuyjq7BevyfLd7JxCeWUtjBIUgRTd8MS1sOgY2Lc18H6c2kZUWgWlq16VaqDDOiKm02F1ap+B6+/LxYmLk2hhPJp3dnX77lefTonjNCU77JKXXd29/V63RglCLhH79THf411HrS/EyyPn0Cuw5GRAYMRbYNr34bjzPD/mlPisVISLh4/oAqdwGcwLxzQsEzFe3nPpo3iUq1qtyg0/rY4FuPWyM7h24bqyq3zGNb7AF4/7JReOeIgGiTd0U+CoU+Ds2x3FfrCIRRLOM0m99zUsUwd4Va2UeupOnSnBOUxQDsU3lZQPlb18+nhaWzIVhSSeP3Q81z5/HSc/lUvG7otpOHg/Xt+SWzD1wGm2oRur19C29jmsaptVdyLhh6Ssph3sVUgalokYSwy+dM9621rsUrH0emT38uwFGNGUtk3iWlihBetLbWdXMemUMO3EUYB9qKIclu49lwcPzOLiMzM89uRqrh71c+Y0P0xa+uLx6Ds35EM3wHHvhrffDsNPqhmP1suOSux0E8V6upkN9iokFfcYsL4gfuO3rS0Zxy+Vl7g2D01z4wVTBmxjtS8unnx0xk2/9yXS3b2m8EX3M6zajZQIfcb0H8N34iiuWXgs1+64jnGNL/Bf4/+Nk5t25eyOQ+hffBCWnEyvNNLQOYOGA5djOL7g0QKRil5puKHUDq/3vUiKKDoNvh8svfdV3GMirCSatb1TnLzzQLevY3Wszbp696UUf9EtkXcqFXTCKf7Z2pIp3CyeP3Q873nmDiAXo7/qmPu4dOTvOSKGlbEpc4gLR6zkgjetLLx2yKT4wSNXQcsPIrPDy7Ou1PNOiihWKwFeL6i4x4idR17O43SxGJZifSHdvH/wtxDHbr/FdgcdBO6W2Jo3e9KAZO3z+Z7039rzBaam1/L9sV/jTan9kYt88fGOkF6uGXUbLLoH3vc4DD+p6sf38qwr9bzLFcVaCVlZDPYqJBX3GqKSx+lKvZQgj9x2+12wfHPgqhm7cyoWiKGNKfYf6u+BNqVTzL9wCjCF85efxdjux7ntxG8wcshrhZtL1JEbEQ6XV445H6Z9r6oi7+VZV+p5lyOKlYaCqoWXU5NkVNxriEoep/2GXkrftz7jJsypBuGoI4awt6vb8YseNB5rV+lTKhD7D/WSTgnDGofQ2dWNkLse1yxcx8h8LqG1ZRbQVtiHvPgQPH4V5rUklAvTAAAVn0lEQVQtud+jVvqdD8CSB6DxaDjxQ3DqdaELvdeNPIxwRFBRTEoSNkmouNcQlT5Ou30h7TyrefeuB8Fz0VIDMP/CKa5fUidv0Q4BW6GxE4juXpMfsC109x22c8+BbuYtWg+UeIbHnQcX/I13ta8g0/043x23gOPTr0Yv8odegS235f5YpIbBuFY47eaKBN/rRh5HOCIpSdgkoeJeQ1QzkWUrnH3+AindfcbTA5s3exLzFq33tbrVqpEvxenm4DRztrhqx86eaxd28Y6nf15IxJ4/4v8xasjrQExVN7374bm7cn+kEU68tGyhryTcUI3YeFKSsElCFzHVENXsMVKpB+X1+daWDMMavX2FBqFQI1+MlZANitMNobUlw+XTxwOHE7FnbvoV/7BxGX948yo49rwyjhYi5lBO5JecDIuPsx0jWC5ui5CqtUBJ++PUHuq51xDVfJwOEjZx+rwXfqY99Rlsve1yErIWE9qWFXrZZLyumcCBxhPhPStzK1CfvIGe5xaRMgdzb8fh0R98KbcqFiDVBOM+WFHoxmtlZjVi44O9MqUW0d4ydY7fR2y7PhvpBqEP6PUIz1j16OD+5fXbB8euRW7QGnkvey8+M8MvH91u+75dK92OtVmeePgHfGnkd3hT6kDOzth73QhkLoAzbw0k9E5To6zTcXqvkrbFSnXR3jKDjCCP2HbDIhZcejpHHWH/8JYS6TdUAvA8lt9pT0418mHR1d3LXY/ZCzvYh5haWzLc/IWvs3Lq33jL07/jnM0/5jd7zuNQXyrGXjcGsktyoZtfHwl/usJXq2K3weFu7ynJQj33OiaMIQ9uXt629jmFJwMnj7z0WMVPEiOa0uw/1NMvyWq3KjXKzpelNpc++ew/2DNgpe7ZwzbwzXHfZ3xjFmNqwKP3KLN064YI9m0vyu2UWGsLl5JKOZ67xtzrmDDKz9yqHOxEwutYpVUcTl/+4tfd3Iugq179YCX57MpD7Xh8/1TOffpHbGufwyOP3MvYLddwYuPOePvRb7kNtv4M3rUIMuf3e9tP/DsMQQ6ycCmum8BgvvmouNcxYZSfuS14sUvMBT2WU4sFP50kRw5Ns/aG94bq2Q9rTBXsmb9ko+9ultZ5/o93XUrHsHdyxfLNpA5s45rMEuaMeJgj+jpDsS8QvV3wxzkUboFHHpdLxp56Ha0tJzmKWFirNv0uXIpr9WqtrpqNChX3OiaslYhg78ld6zG0I8ixij2oBpvpTXbsOdDNjPYVzJw8mrse3V6xB59OCV//wNSCPX4bpZWeZ39x/GTux76tsOmbsON+6HoB6KvQ2iDkr8wbL+Y8+mduh7N+BG/+pO89lOPh+n1yjGv1qtNxr1m4jgXLN1fNi6+VpwUV9zomzM6Sdp9xK5/0LDksotSD8iPsFtnOLhavyfLOk0fxp2dfLVvgS+11a5Q2cmiaoY1DBrRpmNG+wvk6Dz8JzvpB7g/kxP7RT8JLD5VpcQWYXnj8U/D4p8kJfwOMONVxnGC5Hq7fJ8e4Vq+67b9aXnwtPS1oQlVxJKwxZWGFVUYOTfNGdy9d3cG8YrsEs1Mi2TpO54HufsJuF0ZqEPjo28dzS+tU54Pna+n5+0JMX/RtiosxBrpNAysOzKTh9Jt579vfCZSfmPf77yOMxH85+Pl3F7YN1TpXLYVUQsWufLKcqoqwPLQ9B7oDC3s6JbahI7dcwZ4D3f3KPZ1i830Gfvnodr7ascHZgOEnwTt/CR/p5rKt32D7wWNjGyUoAo0Nfbxv+IO8+5l38cSKbwLOiWQ/q5L9/PuIa/Wqn9LcsJ8eaqnHjoZlFFfCSL45Pb5bU5iah6bZe6C7OlHqEhEtLu0srcSxq8zp6u71TLr+6rHn3b33PGsOnMY5m38KHB4OfsGIP5ISE/3gETG07JqHuXsef51yBMtfewfffvEKnj90fGEbP4l5P/8+4lq96mdKWNj1/bXUY0fDMkrV8Xp8r3ade3NTmmFHDHEV9EyF7Rmes1ndWXwjSbkkkeeO+CPfGPt9hjccPn4c4RvLvK6+nNgPm/b1Quim3gkrxBjXcbTOXalJvDy3Sh9Zm20WSxXT2dVdqIwp3cIS9lVtsxxvMrlYf5+jB5+yUeIgSeSle89l6d5zC79bXv3sN/2ZpoaDkQm9dZyhqYN8YORDsG0WjL1vQB19PRLV00Mt9dhRz12JnUo89+KVl1+6Z32gSpxinsuvxnVf2fmkbcz/iukDk6phPI1Yg0re1Ps812SWcH7znziy9+WK9lke7tU2SvXRhKpSl5SbWBs5NF143G1tyfCtD53uq7dNKZbn3dqS4eIzM4XfUyJcfGamsP9NX3s/V0wf3+99O2GH8p9GLCd95NA0mNxTx/ZDx/PFbVdx6tqf8fFtN/JG3xFl7bt8+mDvxlznyidvivjYSrmo5x4TtbLQISq8zrfl5t87DuXINDcxc/JofvXY8/QaQ0qEj7x9nK2olh7nwKEex/0W4+W5B/1/4+W5FyeTTV7Ei9sW2/W4sRjX+AJXH/cbLj76YRr7Xg9kVyg0jsQc2kOfERowvNrbzN5jLuSkc26MZED4YKQcz13FPQaiSu7UCn7Ot2Ntlnn3rh8wHSqdEi47axyL12TLul5+Wh14xdwtIQ5yE/bbYgFyrZf9jDu0tftzEw6vjH3jBWhoyk18igEDiKRgzJzAbYoVdzQsUyd4DVNIGn7Ot7Ulw4JLT6e5KV14beTQNAsuOZ2VT+8u+3oV12IDA1oLF9dbO4VSeo0JPLWo9LjFoZxSuvtMYGEv2GutjP1AFj7SC5ftg9O/wcAzrT4CudWxZbQpVsJHq2VioJYWOkSB3/N1qpl26nFTzuBwt/CQn2lVQXqi2J3PxBCHkjjWTk9py7UEfvIG2H4v9B0K7ZiB6DuYGyW4fRH8j2RU3dQTKu4xUEsLHcLGTjwrPd8wr5fbohu7Rmx2ZDu7+GrHBlY+vTtwziTIuEOrx41dfb7nCk9rZew7f3m4qdnf78m1C46avoP57pV5GkfmKm8mfDR6WwYRGpaJgaQOE3aaDDVz8uiKzjeq61W6nN4uhGLxy0e3lzVk2u+0qnRKuPGCKaxqm8Vz7XO49bIzym8DYYVuLnkZLnwWJlwOqaH+PlsNDu2BP10OD74nPhsGAZpQjYkkVsu4NU2y+sOXe75f7djgq1omTIIkReHwSlg/82y9avKbm9Ksu/G9ZdtefCzX6241Nnv+N9B7oOLjBSbzQTh3cfTHrTOqskJVRH4KzAVeMsa8Nf/aKGAhMAF4DviQMWZPUIMHM2ENTKgl3GLrdhOaXFvoFtGxNsviNdmCGPYaw+I1WaadOKqq19Da9zUefe0tilfCurV6tX53u3Hs9dlr3g1f7Wet8I3FkzdhnpofXTo2ex+8+JAujqoCfsIyPwPeV/JaG/CgMeYU4MH878ogx+/w5SCDvSHe6qLWlkyh4iUobjZaISCn0E8Y+ZeyrttpN/K5F77F5q7x9BrBGOg10GeqKPerP5f7uW8r/OVq+E0G7m7I/fzL1VptUyae4m6MeRh4teTli4A783+/E2gN2S6lDvEbGw8qOnFXF/mNk9vhZqPTqtqgE65mtK9gYtsyZrSv6HeDLPe6PfDSJGZv+QEnb/gtEzcs5eQNS/mfz91AV7VWxu7dBNkHYNlbc5OkunYCJvdzy22517MPVOfYCabchOpxxphdAPmfx4ZnklKv+O3vHVR0vJ4I3AQuDOzO64rp43191s+M2dJ6eOtG53UeXk9Afp+k/Lz/0Otn8U+7fgynfCY3qzVU+uCRS3IzYe3o7cq9rx58IKpeCikiVwFXAYwf7+8LodQvfnIJQUsb3WbFRjXWzO68Vj6927Ws0a8HbheD93MeXrNJy52x6/S5y98zE1quODxK8MWH4PGr4PUtnufoiZOwF7+/6ZuHj614Uq7n/qKInACQ//mS04bGmNuNMdOMMdNGjx5d5uGUJBG0tNHtiSDOeLzdeViR6aDliuWch9cTULmTtHx/7rjz4IK/wUdNZSWW4tPH3HF/8H0PYsr13JcAVwLt+Z961RXflNPz2umJoJrxeK8ywjB7d5dzHn6egMqtygr8udKqG8jVsb/4oPdnjb9SU954wXsba8HWjvuhaxc0nQBjL4JTrxt0vW78lEL+CjgPOEZEdgA3khP1e0Tkk8B24NJqGqkkj7BKQcNYvWon4uAvTBLneXiFXdxuTpGss3j3f7NjyfvJvP5/nQeOvHU+PHt7PonqwZHHu7+ffWBg7N5Kym65DRAY8ZZB05deFzEpdU2lHTa/2rGBux7dPmBp/5HpBttWwUGn2PsVUbvzKB4B6PY5u/17Dx6JpivpjPYVZLof52tjbuPNRz5PA4Y+hL93n8hJ778TjjsnV+645TbvnZ3yGeeY+76tuaoar9i9xZCjYOIVdePRa8tfZVBSrhfasTbLtQvXDRi954YA22zmpTrtP4iIug3vDiq+bquFAcf3gty4Su22u/4T25bZXt9+19GPMKeaYM5GGD7R/n2/Nwi7/b5rUc03NdMZqsqgpNzQyILlmwMJOwQL93hVs5RinYedMAfpRgnlxfDLyVN4VSv5CjcNPyknsE7lkJYAOwk7lJ9stcos5zxVFx58ELRxmDJocROz5qZ0xc3Kyk32hpEkdqtxL7f+3Q6vKh/flVGZ83MCe8pnoGkMSEPu5ymfyXnsXp51167AthewyiwThoq7MmhxEjMB5l84pawyQj/7L2cRkdvrdou43EQ1zC6boZZjDj+JjiFfYcbmXzBx/RJmbP4FHUO+4u6xF07ghMC29yOBZZYq7sqgxamtQFO6gWsXrmPB8s3Mmz2Jbe1zWNU2K3Dop1wRDfI5p1WqgKOollv/bh2v+EbSPDRtu11pOabV139nZ5fj6tugPYf6MfYi723c8FNmWWdoQlUZ1BQnA5uHptn3Rk+/Oa6VVpEEqZYp3m7m5NG+hoG4JU7LSY56nUtpgthu/qvdfFw/ieWKziVotUwpTWNyowq92LeVrQ/fxFEvL+Po1Ku80juK14+ZU/Xh4FotoygVEKVQFlNJOaevapSQcLo+Xn3s/V7Xis/Frs7dL25llkX773n4YoaYNwa81SNHMuScxVWrutFqGUWpgLi6TwatqikmypGNTtdhb1e362ARv9e14nOxkrKbvgnbfgk9r/v7XKoJTp3nvs2+rfDIJbbCDuRer7GqG425K0qeMKtIglDJTSXKkY1+ro9dctfvdQ3lXKyRgh96Dd69EkZMAbfRI37KLCF3w/DT3OzJG2qmJ72Ku6LkiWu2bSU3lUqSo0Hxuj6VztAN/VyOOy/nSX+0L9fYrNwyS/BfTfPcXTXTk15j7opSRByzbSttoRAlbtenmjN0Y+fuBgi85K2EVFPZYRtNqCpKnVLNm0pUN6wok7uR85uMv+ZmXvhJ3NqgCVVFqVOqNTA9qmEmEG1yN3LGXlRe75pSdtwf2cARjbkrSoKJcphJXDmLSDj1ulxYpVIiXCylnruiREAcsXyItrzTaXgJ5OLxdRtvB+/mZgP6eDrg1ZM+RFTcFaXKRBkaKSXqUElpeCnOcw+d4jr6HffnvPAjj8+FbLpfy1XKeFFpm4QAaFhGUapMrc15jTJUEue5VwWrjv4DWfhIb+7nWT+A0272Dtv4WSwVIiruilJl4lr5CtHWwdsR57lHihW2cRJ4v4ulQkTDMopSZeKuIqlWJY4f4j73SHEL25w6L1JhB/XcFaXqxB0aiZNBd+5OYZuIhR3Uc1eUquNURVJ3CcUyGMznHje6QlVRFKXGKWeFqoZlFEVREoiKu6IoSgJRcVcURUkgKu6KoigJRMVdURQlgURaLSMiu4G/F710DPByZAaUh9oYDmpjeNSDnWpjOFg2nmiMGR3kg5GK+4CDi6wOWt4TNWpjOKiN4VEPdqqN4VCJjRqWURRFSSAq7oqiKAkkbnG/Pebj+0FtDAe1MTzqwU61MRzKtjHWmLuiKIpSHeL23BVFUZQqEIm4i8j7RGSziDwjIm027x8hIgvz7z8mIhOisCugjR8Xkd0isi7/51MR2/dTEXlJRJ5yeF9E5D/z9j8pIm+L0j6fNp4nInuLruENMdg4TkRWisgmEdkoIl+w2SbWa+nTxlq4lkeKyOMisj5v500228T63fZpY6zf7SI7UiKyVkSW2rwX/DoaY6r6B0gBzwInAY3AeuAtJdtcDfww//cPAwurbVcZNn4c+H6UdpUc/xzgbcBTDu+fD/yO3KTe6cBjNWjjecDSuK5h3oYTgLfl/34U8Deb/9exXkufNtbCtRRgeP7vaeAxYHrJNnF/t/3YGOt3u8iOLwJ32/1/Lec6RuG5nw08Y4zZaow5BPwaKJ0SexFwZ/7vi4B3i4hEYFsQG2PFGPMw8KrLJhcBPzc5HgWaReSEaKzL4cPG2DHG7DLGPJH/++vAJqC0uXis19KnjbGTvz778r+m839Kk3ixfrd92hg7IjIWmAP82GGTwNcxCnHPAM8X/b6Dgf9QC9sYY3qAvcDREdg24Ph57GwEuDj/mL5IRMZFY5pv/J5D3Lwj/4j8OxGZEqch+UfbFnLeXDE1cy1dbIQauJb5UMI64CXgD8YYx2sZ03fbj40Q/3f7O8C/AH0O7we+jlGIu93dpfTO6WebauLn+L8FJhhjTgP+m8N30Voh7mvohyfILaM+Hfge0BGXISIyHFgMXGOMea30bZuPRH4tPWysiWtpjOk1xpwBjAXOFpG3lmwS+7X0YWOs320RmQu8ZIxZ47aZzWuu1zEKcd8BFN8JxwI7nbYRkSHACKJ9vPe00RjzijHmYP7XO4AzI7LNL36uc6wYY16zHpGNMQ8AaRE5Jmo7RCRNTjTvMsbcZ7NJ7NfSy8ZauZZF9nQCDwHvK3kr7u92AScba+C7PQO4UESeIxcSniUivyzZJvB1jELc/wKcIiITRaSRXDJgSck2S4Ar83+/BFhh8pmDiPC0sSTmeiG5OGgtsQT4p3ylx3RgrzFmV9xGFSMix1txQhE5m9y/v1citkGAnwCbjDHfdtgs1mvpx8YauZajRaQ5//cm4D3A0yWbxfrd9mNj3N9tY8z1xpixxpgJ5LRnhTHmipLNAl/Hqg/INsb0iMjngOXkqlJ+aozZKCI3A6uNMUvI/UP+hYg8Q+5u9OFq21WGjf8sIhcCPXkbPx6ljSLyK3IVEseIyA7gRnLJIYwxPwQeIFfl8QxwAPhElPb5tPES4DMi0gN0AR+O+CYOOS/pY8CGfBwW4F+B8UV2xn0t/dhYC9fyBOBOEUmRu7ncY4xZWkvfbZ82xvrddqLS66grVBVFURKIrlBVFEVJICruiqIoCUTFXVEUJYGouCuKoiQQFXdFUZQEouKuKIqSQFTcFUVREoiKu6IoSgL5/3CZLlng299LAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb9a4f970f0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss =  44.59417\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "\n",
    "for i in range(100):\n",
    "    y_pred = w * x + b\n",
    "    loss = torch.mean((y_pred - y)**2)\n",
    "    loss.backward()\n",
    "\n",
    "    w.data -= 0.05 * w.grad.data\n",
    "    b.data -= 0.05 * b.grad.data\n",
    "\n",
    "    # zero gradients\n",
    "    w.grad.data.zero_()\n",
    "    b.grad.data.zero_()\n",
    "\n",
    "    # the rest of code is just bells and whistles\n",
    "    if (i + 1) % 5 == 0:\n",
    "        clear_output(True)\n",
    "        plt.scatter(x.numpy(), y.numpy())\n",
    "        plt.scatter(x.numpy(), y_pred.numpy(), color='orange', linewidth=5)\n",
    "        plt.show()\n",
    "\n",
    "        print(\"loss = \", loss.numpy())\n",
    "        if loss.numpy() < 0.5:\n",
    "            print(\"Done!\")\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Bonus quest__: try implementing and writing some nonlinear regression. You can try quadratic features or some trigonometry, or a simple neural network. The only difference is that now you have more variables and a more complicated `y_pred`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# High-level PyTorch\n",
    "\n",
    "So far we've been dealing with low-level PyTorch API. While it's absolutely vital for any custom losses or layers, building large neural nets in it is a bit clumsy.\n",
    "\n",
    "Luckily, there's also a high-level PyTorch interface with pre-defined layers, activations and training algorithms.\n",
    "\n",
    "We'll cover them as we go through a simple image recognition problem: classifying letters into __\"A\"__ vs __\"B\"__.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from notmnist import load_notmnist\n",
    "X_train, y_train, X_test, y_test = load_notmnist(letters='AB')\n",
    "X_train, X_test = X_train.reshape([-1, 784]), X_test.reshape([-1, 784])\n",
    "\n",
    "print(\"Train size = %i, test_size = %i\" % (len(X_train), len(X_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAADHCAYAAAAJSqg8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHWFJREFUeJzt3XmU1NWVB/Dv7eqNpWnpNCCCCGERo0chtuiIxAX3GDHG\nLCQmxsHgEvdkMoTxTBI1GScmLolKBpUBM4boiNsYd8xEExXFFRVRBIRmR0E26aXqzh9dTLr73Wf/\nfl378/s5Jwf6+upXr6pfvfyod999oqogIqLSV1boDhARUXZwQiciCgQndCKiQHBCJyIKBCd0IqJA\ncEInIgoEJ3QiokBwQi9CIlInIveJyA4ReV9EvlnoPhFlSkQuFJGFItIkIrML3Z8QlRe6A2S6GUAz\ngAEAxgD4k4i8pqpvFrZbRBlZA+BqACcA6FHgvgRJuFO0uIhILwCbARygqu+kY78HsFpVpxW0c0RZ\nICJXAxisqt8tdF9Cw69cis8oAMndk3naawD2L1B/iKhEcEIvPr0BfNQp9hGAmgL0hYhKCCf04rMd\nQJ9OsT4AthWgL0RUQjihF593AJSLyMh2sYMAcEGUiD4RJ/Qio6o7ANwL4EoR6SUi4wFMAvD7wvaM\nKDMiUi4i1QASABIiUi0izLTLIk7oxekCtKV1bQAwF8D5TFmkAFwB4GMA0wCcmf77FQXtUWCYtkhE\nFAjeoRMRBYITOhFRIDihExEFghM6EVEgMprQReREEVkiIktFhHVGKBgc21SKup3lIiIJtG2COQ5A\nI4AXAUxW1bd8j6mUKq1Gr249X7boqEonVp1oMdtu3VWd6+78P5HMHh/n1+h7Lusa1WtTdttdTdGf\nME92YQeatSnDd7J0xzasV+4ZF80D3b4e0G+j2TZlXKTMfDKbGo/PRm6ddd2k54NgRZs0YbbdnnI/\n9x8128UhkzvcNPqqD+z5RJub3aDvbezU4ahjO5Ok/nEAlqrqMgAQkT+ibQOMd9BXoxcOlYkZPGXm\nWmcMcWIj+tgDef7Sfd2g2u9prAm1zPiAlNkTZ1SppD041bhsoiJptk22utfY9yq74kByyVI3WGb3\nASn7+bJtgc7P1qVKcmxLuftx1tZWs+3KqYc7sRfOu8Vs26TuBFUlFZH7lTQGYSsyHxO71H1t2zxj\nrcn4fK5orTXbPr19tBN7eJVdG2/Hc/VObNidjWbb1hUrnZj1OwPc31vUsZ3JVy6DAKxq93NjOtaB\niExNF7Vf2ILiu6sjMnBsU0nKZEKP9A88VZ2pqg2q2lCBqgyejihvOLapJGXylUsjgL3b/TwYbSeS\n5E6Mf9KXHbSf2fS/Rt3qxAaW9zbbJgf9zYklpPQTg6x/QgP2P6OH42yz7Ygz3ZiUeb6OyuzbpELI\n/9jOAk1G/xqj3/i1OexJ1+J8ZRPnGrUxPp7DPV89TuzhVtn41/pFZtvE590nfOkc47tyAN/6/SVO\nbJ+fvvBJXYwtk9npRQAjRWSYiFQC+AaAB7PTLaKC4timktTtO3RVbRWRCwE8hrbqabNYQIpCwLFN\npSqj0pWq+jCAh7PUF6KiwbFNpaj0vxAmIiIAnNCJiIJRUqeFxMmiWDehr9n2slWnOrGXVw0227Y2\nGxs1mj3/H2iEEx/ab+/wse7Gg8f2e8iJbU7uNB/fN9HTiX1j+TFm2wWvjXBiT33xOrPtsAo3a+Cd\no2832x75lfOdWK95C8y2cTa8UAQxtvom+tqfg6tH3hf56crhZpdZm4UAOwvs7RY3R/9njaeYj+9V\n7maIJD2b+eoq3c/HIb2Xm21P6Olu6qlP2Dt7fVlglpTxnh9YaWfjvX3ODCc2bM/vmW1HTX0xch/a\n4x06EVEgOKETEQWCEzoRUSA4oRMRBSKvh0T3kTrNSUU6Y5FIEp7qg5kuxsUpP3CAW7UNAE6+6zkn\ndlHf9yN3YeJb7sJu1RkfmW2TW9z4e3eONdsuPfo/I/fhn9a511g0LkfveUQLdD626ocZl8/tjpyN\nbUuMMdh08iFm0ydu/Z0TqxD7unGqJVrb8Yc9co4TGzVlofn4TBfQpcItjw0AGONWTt3ys4/Nps+P\nuceJxVkEblH7vbFKDm9N7TLbnnrxZR1+fu2pG7F986ouxzbv0ImIAsEJnYgoEJzQiYgCwQmdiCgQ\nnNCJiAJRUlv/vYxMHe/KuJEh4CspIFXuKTSpHTvstg0HOLFz595vtj2t13a7b50c8frpZrzm9HVu\nv6wDaD1GXGe3fXO8u+q/f6V9OO61e77ixMaed4HZtv9NzzqxqGcpksubwWVkuaw+yn6frYyWOAef\nWFveffb+nwzvG2OcoK4tns/Bi+4BFXucar83Fzx3mBO7ZdDzZlsro8WXLbQz5fbNKuMBAKs7JUy1\n2ElBDt6hExEFghM6EVEgOKETEQWCEzoRUSAyWhQVkRUAtgFIAmhV1YZsdCqnjIUjNeo9A4AaC6Dl\n++xttAQmznYX/qIufgLAeGMBtGbSarOtuQBqbEEGPFupX7KPxzzlyYuc2PKTbzPbWo482z7B/O1b\n3cVlbXJrZAOwF8DyWJ7i790okrFtvB/aGr1e98hDopeU8Imz8Hf/jt5OrOZFt/6/b+lbk8a2+Ti/\nf88CqlS6JQF8Y3D+40a5hLPtRdEUrJIA9nuTiLG4m6jr1LeEXXqgs2xkuRytqpuycB2iYsOxTSWF\nX7kQEQUi0wldATwuIi+JyNRsdIioSHBsU8nJ9CuX8aq6RkT6A3hCRN5W1afbN0h/GKYCQDXsJHqi\nIsSxTSUnozt0VV2T/nMDgPsAjDPazFTVBlVtqIC7OEZUjDi2qRR1+w5dRHoBKFPVbem/Hw/gyqz1\nLFeslWYj8wUAynq6d1297nRPGgeAy+uWRe7ChasPdWJ9vrrR7ZYvE8Q64MDzGnwZPJZRM93n23SC\nXerAOjH9hoH2/uSDzzrfffxM95APwN7Wnu9yAEU1tq3sJc/vOjFquBP77WfneC7sZqOUe8ZKk7rv\nvy/L5YpFk5zYoNVuVlXOSj94MmK0Jfp1k9W5yapKWn3zJL6k1ld3DLRGu/fO5CuXAQDuk7YJshzA\nH1T10QyuR1QsOLapJHV7QlfVZQAOymJfiIoCxzaVKqYtEhEFghM6EVEgwqiHHkOcRbclM0Y7sWWf\nnRX5ue7YWm/Gl53plg9IbVvqNoxxuruX0da7IPWCWzP6S298x2z73EHzIndhwGR3+3lypt3W3Prt\n2zJdgJIA+WbV6vccQI8NE/o7seEV7uInEG87f1WMaSLxTG3ktsXqwEPey+jxSc8vqGeZW37go5R7\nBgEA7Plsx7G9MWIVEd6hExEFghM6EVEgOKETEQWCEzoRUSA4oRMRBSLYLJc4W4sbf3y42XbZcbc4\nMSs7AABWtrqr1XMumGK2LV/ykhMzD6LI85Z3S9ksO1MHN7oh3+r+3SPvdWJfGfc9+7pGpo0328fz\nuwiK5+ASS/OXtkRuax3MkFQ7myhh9GFlq512Meihde51jXZmNlMWxPncpyaMNdvePPS3RtTOFrLK\nJbSarxhIGPfPP1k/wWzbe17HMhplSbsER2e8QyciCgQndCKiQHBCJyIKBCd0IqJAhLEoaiya+RYU\nU0eMcWJPnv9Ls21S3Xrovu3Rp938Iye211PPmm2lwjiBvKXZbJsLcRakah9bbMZ/sWlfJza9fonZ\ntrdUO7Gl33TrqQPAiBfcmLX9HfBvgS9JnvIG1rhIDHC3+APAJfv+OfLTlRn3cnEW8y5//zSzbfLd\niOcC+Mo2+Mo8WE3LK9zLxhjbO//lIzM+sNxdAG3SlsjX9dWV35lyf5cLf3Gw2bZXakHk52uPd+hE\nRIHghE5EFAhO6EREgeCETkQUiC4ndBGZJSIbROSNdrE6EXlCRN5N/9k3t90kyj6ObQpNlCyX2QBu\nAnBHu9g0APNV9RoRmZb++Z+z372IYhziMPBadxXeWtX2OeiFyWZ8r1+6GS3ebch5zGixO2BnGFjZ\nN8mtW822tz19lBObfrqd5WI5+5j/NeN/q97DiaV27bIv0jkjIv55F7NRJGPbOngFsLO1tk4YZrad\nUvu4E/OVqrCytXb6Dk4xEk9e/+tIs+kwbHJiZT3dbDFttj8DZpaKZ7zG+Ry9c+shTmz5gbeaba1s\nlIQn+6ZK3Ewb33t+6I2XOrG95nky4TrPHRGrgHR5h66qTwP4sFN4EoA56b/PAWDnMBEVMY5tCk13\nv0MfoKprASD9p50YS1R6OLapZOV8Y5GITAUwFQCq4f7Ti6hUcWxTsenuHfp6ERkIAOk/N/gaqupM\nVW1Q1YYKVHXz6YjyhmObSlZ379AfBHAWgGvSfz6QtR59gji1jt/7hbsIAgCP7jMj8vPN297HiQ2+\n1K5LbK1ZaKq0TqWPs216+H+7C0ctX46+AHdF/dtm26OPPMeJVT620GhpLCRm5+0uyNiOM1ZWHxu9\nrVX3HLBrn1un0gNAo1H7fJ8/2afVm33Y1WQEo4+1xGfqzPiWY0c5sSEXv2O2XT7MXQD11e+vEnee\nsWrCA8AdW93zAn73kzPMtnvdZSyAemr9O3NaxF95lLTFuQCeA7CviDSKyBS0DfbjRORdAMelfyYq\nKRzbFJou79BV1c7TAyZmuS9EecWxTaHhTlEiokBwQiciCgQndCKiQBTvARcxDq2Qg/d3Yo9+/Vqz\nrXVohW8F++fXfcuJ9VvxnN0HIwPH19+iFePEiMTzbzmxn210Dw8BgKv7L4p83cZj3K3Un30s8sNL\nV4ysj0smuFv8fayDLLJh23Q38wUAmlpGO7GKcve1VZfbn43htW7pgMn1z5htj+/51Cd1sYPtKbd8\nxMak3YfZWw51Ynffe6TZdtjNbrmLmk3Pm23zMUfwDp2IKBCc0ImIAsEJnYgoEJzQiYgCUbSLotZJ\n7741u1XT3djwiug1zs9471gz3u8/3MUNq7YzYG+b99W4LiW+cgupHW4JhDsXuotJAHD1ydEXRU+a\n6G7z91VZd7bLl0ilhTiLY00nuSUszq79jefKPZxImVXMHHYigG8rfL+EW6fm+TH3ePqQP03a4sSs\n+uQAcNe2oU7sujtON9vuM+NNJzZki123PGX8Lq1zBYD8nIPAO3QiokBwQiciCgQndCKiQHBCJyIK\nROEXRaPWAwaw65RxZttXD7NqnNvXXdy804ltv3iA3Td1d62ZtZ2BWDv9SkqMGumDHrHf85aT3GtY\nNdIB4Nz6p53YD8dMMdumXnV3q5YEz85ky5ovuB/R2jJ38ROwDzf21TiPw7fQWGjlxmfcd0DzlNp1\nTmzy928w2879zhAn9us77QXUIde+5MS0yTNHWAdNew7A7i7eoRMRBYITOhFRIDihExEFghM6EVEg\nopwpOktENojIG+1iPxWR1SLyavp/J+e2m0TZx7FNoYmS5TIbwE0A7ugUv15Vf5X1Hn2C6svWmHFf\nxoTlke0HOLHlX+ljtk1+8x+cmEQvGR4Ge+e4+T601thvzqake0J8/4RdQmH/SjeDY9PYWrNt3at2\n32KYjQKM7ThbwMcf/UbXjdKsz4Ev68NqO3vrXmbb2dMnObGyFjs7Q40ZRY0yHk197HvJ7Xu7bWsP\nX2+2vWv/OU5sSLld8sMqa+DLAPpuH3eemXLBLWbbgw/9mhOrn/Se2dbMhLMyX4BuZ790eYeuqk8D\n+LBbVycqYhzbFJpMvkO/UEReT/+ztW/WekRUeBzbVJK6O6HPADAcwBgAawH82tdQRKaKyEIRWdgC\nT8I9UfHg2KaS1a0JXVXXq2pSVVMAbgVgb+FsaztTVRtUtaECbhlOomLCsU2lrFtb/0VkoKquTf/4\nZQDRVm7ErQXtqwPdfEKDE3ty9EyzbTLG+sHldcvc2D9apQMoe9yFKmubOgD0FHeh6oMx9i+4LrNO\nmbo9tu2L2XFjwavsQPdwZQC4dtDtRrSX2daqfd4KX+kGd1H031450Ww5/L4FTizTmt/2kjjQ1yoF\n4imrMeXIi5zYtFmd17fbTDSqJfgWjFNwF1A/9ozXlw6+24mN+OV5ZtvhP3TPV/CdmdDdw6O7nNBF\nZC6AowDUi0gjgJ8AOEpExqDtSIEVAM7t1rMTFRDHNoWmywldVScbYeu2gaikcGxTaLhTlIgoEJzQ\niYgCwQmdiCgQhT/gwmP12e6qsnVSuc8Nm4ea8duWjHdileX2irKqJ0uBIOJmajS12MNp8kj3EIAr\n6t+O/FxHjLMPsrA3hBcPKbcPhrAyQd4/1c7ZqU+4GS1xtvOnYmwh7/9AdeS2kvB8Fq29/3EYn3Gp\n9Byw8ZdXnNClM+w17EWX21v3LdaBHmUx7n1//MX7zfi8aw90Ysn1G+yLdM6Qivhr5B06EVEgOKET\nEQWCEzoRUSA4oRMRBSK/i6LqbmktH+qesA0ANzXMzeip7r7S3sY86C5j+225/TZ0d/vtp4G19du3\n7fuei45xYlf8OPqi6Jn9njPjN9R/oWOfNkevi58PmvRtu3fVjo++xOtbFLW2/vvOCnh8p7vw1/fZ\nRrOt9SlI+U62z/Ip9gCgyei/14F/22H/h8vdkPV++cQ5c8Gqpw4A9wx0PwfwLop2716bd+hERIHg\nhE5EFAhO6EREgeCETkQUCE7oRESBKPjW/zUnDzbjx/dsiXyN320Z5MT63O9uCwYAWBktnhVlXxF/\nAqTCfR99WS41jW5WhnUKO2CXdxhdudlsu2vM0A4/p54v8KlBnQ9n8BzMkBgxzInNGP0Hz0Xd11Ql\n9se2Sd18FOvAEAD44RtnOLGBqxabba0ssGLNANs1IPoYSHn202eaK+U9VCRlj/ls4h06EVEgOKET\nEQWCEzoRUSA4oRMRBSLKIdF7A7gDwJ4AUgBmquqNIlIH4C4AQ9F2mO7XVNVevfoEqeNjP8Tx78+f\n5MRGNS20G5unikc7qZz+Ls629l6r3O3Y77V+bLYdVeHW/x5S3ttsu3l0xwWw5Gvx6tdndWwLIGUd\nn9+z7ouNE/Z0YmOq7MU8a5u/bxt6VYwcB/lL38htC00S9utVY9G58ZjoYyAF3yKl+3xxatDP3eYm\naQCArP8wct+8g6cLUe7QWwH8QFX3A3AYgO+LyOcATAMwX1VHApif/pmolHBsU1C6nNBVda2qvpz+\n+zYAiwEMAjAJwJx0szkATstVJ4lygWObQhPrO3QRGQpgLIAFAAao6lqg7YMBoL/nMVNFZKGILGyB\np0IbUYFlPLaVY5sKL/KELiK9AcwDcKmqbo36OFWdqaoNqtpQYWySICq0rIxt4dimwos0oYtIBdoG\n/J2qem86vF5EBqb/+0AAnsK+RMWLY5tCEiXLRQDcDmCxql7X7j89COAsANek/3ygy2slEkj0qe0Q\nu2r/Lh/WpbrnPaeCe/rQmbVaTl2IsQqfWL/Fia1prTHbjqqIft2twzu2Tca8Sc7m2E6nuUR63o+/\nFPkfAWYmRlLtTA6rbMLylu1m28H3r3Zivs38cTKaMmZkoflKSiTqP+PEfnTsQ9GfKsY3znGyXK55\n/QSz7T7rFzmxbB+uEyXPaTyAbwNYJCKvpmPT0TbY7xaRKQBWAvhqt3pAVDgc2xSULid0Vf0r4D2r\naWJ2u0OUPxzbFBruFCUiCgQndCKiQOS1Hnqyphrbjh7dITah+hFP655OpLHVXuCpe8veRm7J6wIP\nAQBSH7hbnte11hotASB6KYjKwR1LCkiMBdWsU3UW7xL9+plNL9tvfuTLWgt3vnrbCaPt+e993Wyr\ny993g+L59sla7I1RZaFzSYS4fOvvi68a7sQe3sN+b5vUPV+hSuxkCqtWv78GvXvdvW6LvjqvKbsm\ne3fxDp2IKBCc0ImIAsEJnYgoEJzQiYgCwQmdiCgQec1yae0JbBzb8f9D+ibcbBafN5rdrb4AULl0\nrftcvot0s3A8daLRV+dTO3c6sR2pzItZ7du/Y4mV9RVuxkEhbTnGzcIAgCm1TzixOFvLd/pKVRjJ\nJI2P72M2HQR367+U21kfvq33UalVqsAzfsp6uvPBkt983my7fNLMyH2wMlqsbBYA+Fjd19u7rNps\nO+KB85zYqMdfsDthHq6T3aw73qETEQWCEzoRUSA4oRMRBYITOhFRIPK6KIrqFFIj3QWyqHzbxVs3\nbIp+kRiLeRSTb+u48Z7v0ug17H2Oql/S4edXyndlfM1sWn9K9GPpfCfQW7XPe5ZVmm3XGqUx6t62\n0wOsWuJS09tsqwnjvq/CnjqStT2c2I5BbmzdP9hj5UdffNCJTa191mxrLSSXeWoSWOUSkp65wFoA\nPeTlr5lt9734VSemvs9BHs5d4B06EVEgOKETEQWCEzoRUSA4oRMRBaLLCV1E9haRP4vIYhF5U0Qu\nScd/KiKrReTV9P9Ozn13ibKHY5tCEyXLpRXAD1T1ZRGpAfCSiOzeu3y9qv4q6pPVVO3C0cPf6U4/\nAQDN6uluHlaPKQJP1oB1snlNWfRDSXw+32NFh597lsXeop61sS1lZSjrXdMhduU4N2PDx3cCfcI6\nXMKjLuGWU/jV9TdHfnwC9u+vQtwMnApPVs6exu72OOU9LNtTdvaStZ3f935Zh38kPa9h2EPfc2L7\nXvS62dYsi+DLcsmDKIdErwWwNv33bSKyGMCgXHeMKNc4tik0sb5DF5GhAMYCWJAOXSgir4vILBHp\n63nMVBFZKCILd22OnpdLlE+Zju1mLa4cePp0ijyhi0hvAPMAXKqqWwHMADAcwBi03eX82nqcqs5U\n1QZVbajum3mFPaJsy8bYrhS7Gh9RPkWa0EWkAm0D/k5VvRcAVHW9qiZVNQXgVgDjctdNotzg2KaQ\ndPkduogIgNsBLFbV69rFB6a/gwSALwN4o6tr1SR2YeIeb3W3rxhasdGMlw880Im1rl1nX8RasGA5\ngPhivI+JAf2d2KCKVzLuwp6JHR1+9i3U+WRzbLfU9cDG0w7oEPtWzTOR++Lbsh6HtUg4LoB/FPcQ\nu9SBtQC60ih/AABnLv62+/jr6822ox590YmpVcscKLr5JEqWy3gA3wawSER2Fy6YDmCyiIwBoABW\nADg3Jz0kyh2ObQpKlCyXv8I8CwUPZ787RPnDsU2h4U5RIqJAcEInIgoEJ3QiokDk9YCL3tKMw6s7\nnzZuF9W3HN/TPtX9qsPck8173mdnuUjCXa3WVvsQAPKL8z5++IUhTmxiD7tcg3USu287916d+lBR\nuB3XSNaksGVitHIGTeqOYytDJRusQyCy0Xan8RoAYE3SHRfrWmuc2IqWfubjn/xgPyf28sq9zbZV\ni9ySAkMe3my27fHaYjcoK8y2VqkKTXremyLLkOMdOhFRIDihExEFghM6EVEgOKETEQVCNI9f6ovI\nRgDvp3+sB7Apb0+eP3xdhbOPqtqrbTnWbmyXwvvUXaG+tlJ4XZHGdl4n9A5PLLJQVRsK8uQ5xNf1\n6Rby+xTqawvpdfErFyKiQHBCJyIKRCEn9JkFfO5c4uv6dAv5fQr1tQXzugr2HToREWUXv3IhIgpE\n3id0ETlRRJaIyFIRmZbv58+m9AHCG0TkjXaxOhF5QkTeTf9pHjBczERkbxH5s4gsFpE3ReSSdLzk\nX1suhTK2Oa5L77XtltcJXUQSAG4GcBKAz6HtZJjP5bMPWTYbwImdYtMAzFfVkQDmp38uNa0AfqCq\n+wE4DMD307+nEF5bTgQ2tmeD47ok5fsOfRyApaq6TFWbAfwRwKQ89yFrVPVpAB92Ck8CMCf99zkA\nTstrp7JAVdeq6svpv28DsBjAIATw2nIomLHNcV16r223fE/ogwCsavdzYzoWkgG7DxhO/+mekFxC\nRGQogLEAFiCw15ZloY/toH73oY7rfE/oVsVqptkUKRHpDWAegEtVdWuh+1PkOLZLRMjjOt8TeiOA\n9tXqBwNYk+c+5Np6ERkIAOk/NxS4P90iIhVoG/R3quq96XAQry1HQh/bQfzuQx/X+Z7QXwQwUkSG\niUglgG8AeDDPfci1BwGclf77WQAeKGBfukVEBMDtABar6nXt/lPJv7YcCn1sl/zv/tMwrvO+sUhE\nTgZwA4AEgFmq+vO8diCLRGQugKPQVq1tPYCfALgfwN0AhgBYCeCrqtp5gamoicgRAJ4BsAjA7jPh\npqPt+8aSfm25FMrY5rguvde2G3eKEhEFgjtFiYgCwQmdiCgQnNCJiALBCZ2IKBCc0ImIAsEJnYgo\nEJzQiYgCwQmdiCgQ/wfQEz56iywmUgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fda9174f048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i in [0, 1]:\n",
    "    plt.subplot(1, 2, i + 1)\n",
    "    plt.imshow(X_train[i].reshape([28, 28]))\n",
    "    plt.title(str(y_train[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with layers. The main abstraction here is __`torch.nn.Module`__:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Base class for all neural network modules.\n",
      "\n",
      "    Your models should also subclass this class.\n",
      "\n",
      "    Modules can also contain other Modules, allowing to nest them in\n",
      "    a tree structure. You can assign the submodules as regular attributes::\n",
      "\n",
      "        import torch.nn as nn\n",
      "        import torch.nn.functional as F\n",
      "\n",
      "        class Model(nn.Module):\n",
      "            def __init__(self):\n",
      "                super(Model, self).__init__()\n",
      "                self.conv1 = nn.Conv2d(1, 20, 5)\n",
      "                self.conv2 = nn.Conv2d(20, 20, 5)\n",
      "\n",
      "            def forward(self, x):\n",
      "               x = F.relu(self.conv1(x))\n",
      "               return F.relu(self.conv2(x))\n",
      "\n",
      "    Submodules assigned in this way will be registered, and will have their\n",
      "    parameters converted too when you call `.cuda()`, etc.\n",
      "    \n"
     ]
    }
   ],
   "source": [
    "from torch import nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "print(nn.Module.__doc__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a vast library of popular layers and architectures already built for ya'.\n",
    "\n",
    "This is a binary classification problem, so we'll train __Logistic Regression__.\n",
    "$$P(y_i | X_i) = \\sigma(W \\cdot X_i + b) ={ 1 \\over {1+e^{- [W \\cdot X_i + b]}} }$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a network that stacks layers on top of each other\n",
    "model = nn.Sequential()\n",
    "\n",
    "# add first \"dense\" layer with 784 input units and 1 output unit.\n",
    "model.add_module('l1', nn.Linear(784, 1))\n",
    "\n",
    "# add softmax activation for probabilities. Normalize over axis 1\n",
    "# note: layer names must be unique\n",
    "model.add_module('l2', nn.Sigmoid())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weight shapes: [torch.Size([1, 784]), torch.Size([1])]\n"
     ]
    }
   ],
   "source": [
    "print(\"Weight shapes:\", [w.shape for w in model.parameters()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 0.4526,  0.4411,  0.5917])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create dummy data with 3 samples and 784 features\n",
    "x = torch.tensor(X_train[:3], dtype=torch.float32)\n",
    "y = torch.tensor(y_train[:3], dtype=torch.float32)\n",
    "\n",
    "# compute outputs given inputs, both are variables\n",
    "y_predicted = model(x)[:, 0]\n",
    "\n",
    "y_predicted  # display what we've got"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now define a loss function for our model.\n",
    "\n",
    "The natural choice is to use binary crossentropy (aka logloss, negative llh):\n",
    "$$ L = {1 \\over N} \\underset{X_i,y_i} \\sum - [  y_i \\cdot log P(y_i=1 | X_i) + (1-y_i) \\cdot log (1-P(y_i=1 | X_i)) ]$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "crossentropy = <YOUR CODE>\n",
    "\n",
    "loss = <YOUR CODE>\n",
    "\n",
    "assert tuple(crossentropy.size()) == (\n",
    "    3,), \"Crossentropy must be a vector with element per sample\"\n",
    "assert tuple(loss.size()) == tuple(\n",
    "), \"Loss must be scalar. Did you forget the mean/sum?\"\n",
    "assert loss.data.numpy() > 0, \"Crossentropy must non-negative, zero only for perfect prediction\"\n",
    "assert loss.data.numpy() <= np.log(\n",
    "    3), \"Loss is too large even for untrained model. Please double-check it.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Note:__ you can also find many such functions in `torch.nn.functional`, just type __`F.<tab>`__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Torch optimizers__\n",
    "\n",
    "When we trained Linear Regression above, we had to manually `.zero_()` gradients on both our variables. Imagine that code for a 50-layer network.\n",
    "\n",
    "Again, to keep it from getting dirty, there's `torch.optim` module with pre-implemented algorithms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt = torch.optim.RMSprop(model.parameters(), lr=0.01)\n",
    "\n",
    "# here's how it's used:\n",
    "opt.zero_grad()      # clear gradients\n",
    "loss.backward()      # add new gradients\n",
    "opt.step()           # change weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dispose of old variables to avoid bugs later\n",
    "del x, y, y_predicted, loss, y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Putting it all together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create network again just in case\n",
    "model = nn.Sequential()\n",
    "model.add_module('first', nn.Linear(784, 1))\n",
    "model.add_module('second', nn.Sigmoid())\n",
    "\n",
    "opt = torch.optim.Adam(model.parameters(), lr=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step #0 | mean loss = 0.573\n",
      "step #10 | mean loss = 0.371\n",
      "step #20 | mean loss = 0.218\n",
      "step #30 | mean loss = 0.159\n",
      "step #40 | mean loss = 0.141\n",
      "step #50 | mean loss = 0.127\n",
      "step #60 | mean loss = 0.131\n",
      "step #70 | mean loss = 0.107\n",
      "step #80 | mean loss = 0.116\n",
      "step #90 | mean loss = 0.101\n"
     ]
    }
   ],
   "source": [
    "history = []\n",
    "\n",
    "for i in range(100):\n",
    "\n",
    "    # sample 256 random images\n",
    "    ix = np.random.randint(0, len(X_train), 256)\n",
    "    x_batch = torch.tensor(X_train[ix], dtype=torch.float32)\n",
    "    y_batch = torch.tensor(y_train[ix], dtype=torch.float32)\n",
    "\n",
    "    # predict probabilities\n",
    "    y_predicted = <YOUR CODE>\n",
    "\n",
    "    assert y_predicted.dim(\n",
    "    ) == 1, \"did you forget to select first column with [:, 0]\"\n",
    "\n",
    "    # compute loss, just like before\n",
    "    loss = <YOUR CODE>\n",
    "\n",
    "    # compute gradients\n",
    "    <YOUR CODE>\n",
    "\n",
    "    # Adam step\n",
    "    <YOUR CODE>\n",
    "\n",
    "    # clear gradients\n",
    "    <YOUR CODE>\n",
    "\n",
    "    history.append(loss.data.numpy())\n",
    "\n",
    "    if i % 10 == 0:\n",
    "        print(\"step #%i | mean loss = %.3f\" % (i, np.mean(history[-10:])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Debugging tips:__\n",
    "* Make sure your model predicts probabilities correctly. Just print them and see what's inside.\n",
    "* Don't forget the _minus_ sign in the loss function! It's a mistake 99% people do at some point.\n",
    "* Make sure you zero-out gradients after each step. Seriously:)\n",
    "* In general, PyTorch's error messages are quite helpful, read 'em before you google 'em.\n",
    "* if you see nan/inf, print what happens at each iteration to find our where exactly it occurs.\n",
    "  * If loss goes down and then turns nan midway through, try smaller learning rate. (Our current loss formula is unstable)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "Let's see how our model performs on test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.96585\n"
     ]
    }
   ],
   "source": [
    "# use your model to predict classes (0 or 1) for all test samples\n",
    "predicted_y_test = <YOUR CODE>\n",
    "\n",
    "assert isinstance(predicted_y_test, np.ndarray), \"please return np array, not %s\" % type(\n",
    "    predicted_y_test)\n",
    "assert predicted_y_test.shape == y_test.shape, \"please predict one class for each test sample\"\n",
    "assert np.in1d(predicted_y_test, y_test).all(), \"please predict class indexes\"\n",
    "\n",
    "accuracy = np.mean(predicted_y_test == y_test)\n",
    "\n",
    "print(\"Test accuracy: %.5f\" % accuracy)\n",
    "assert accuracy > 0.95, \"try training longer\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More about PyTorch:\n",
    "* Using torch on GPU and multi-GPU - [link](http://pytorch.org/docs/master/notes/cuda.html)\n",
    "* More tutorials on PyTorch - [link](http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html)\n",
    "* PyTorch examples - a repo that implements many cool DL models in PyTorch - [link](https://github.com/pytorch/examples)\n",
    "* Practical PyTorch - a repo that implements some... other cool DL models... yes, in PyTorch - [link](https://github.com/spro/practical-pytorch)\n",
    "* And some more - [link](https://www.reddit.com/r/pytorch/comments/6z0yeo/pytorch_and_pytorch_tricks_for_kaggle/)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework tasks\n",
    "\n",
    "There will be three tasks worth 2, 3 and 5 points respectively.\n",
    "If you get stuck with no progress, try switching to the next task and returning later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task I (2 points) - tensormancy\n",
    "\n",
    "![img](https://media.giphy.com/media/3o751UMCYtSrRAFRFC/giphy.gif)\n",
    "\n",
    "When dealing with more complex stuff like neural network, it's best if you use tensors the way samurai uses his sword.\n",
    "\n",
    "\n",
    "__1.1 The Cannabola__\n",
    "[(_disclaimer_)](https://gist.githubusercontent.com/justheuristic/e2c1fa28ca02670cabc42cacf3902796/raw/fd3d935cef63a01b85ed2790b5c11c370245cbd7/stddisclaimer.h)\n",
    "\n",
    "Let's write another function, this time in polar coordinates:\n",
    "$$\\rho(\\theta) = (1 + 0.9 \\cdot cos (8 \\cdot \\theta) ) \\cdot (1 + 0.1 \\cdot cos(24 \\cdot \\theta)) \\cdot (0.9 + 0.05 \\cdot cos(200 \\cdot \\theta)) \\cdot (1 + sin(\\theta))$$\n",
    "\n",
    "\n",
    "Then convert it into cartesian coordinates ([howto](http://www.mathsisfun.com/polar-cartesian-coordinates.html)) and plot the results.\n",
    "\n",
    "Use torch tensors only: no lists, loops, numpy arrays, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "theta = torch.linspace(- np.pi, np.pi, steps=1000)\n",
    "\n",
    "# compute rho(theta) as per formula above\n",
    "rho = <YOUR CODE>\n",
    "\n",
    "# Now convert polar (rho, theta) pairs into cartesian (x,y) to plot them.\n",
    "x = <YOUR CODE>\n",
    "y = <YOUR CODE>\n",
    "\n",
    "\n",
    "plt.figure(figsize=[6, 6])\n",
    "plt.fill(x.numpy(), y.numpy(), color='green')\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task II: The Game of Life (3 points)\n",
    "\n",
    "Now it's time for you to make something more challenging. We'll implement Conway's [Game of Life](http://web.stanford.edu/~cdebs/GameOfLife/) in _pure PyTorch_.\n",
    "\n",
    "While this is still a toy task, implementing game of life this way has one cool benefit: __you'll be able to run it on GPU!__ Indeed, what could be a better use of your GPU than simulating Game of Life on 1M/1M grids?\n",
    "\n",
    "![img](https://cdn.tutsplus.com/gamedev/authors/legacy/Stephane%20Beniak/2012/09/11/Preview_Image.png)\n",
    "If you've skipped the URL above out of sloth, here's the Game of Life:\n",
    "* You have a 2D grid of cells, where each cell is \"alive\"(1) or \"dead\"(0)\n",
    "* Any living cell that has 2 or 3 neighbors survives, else it dies [0,1 or 4+ neighbors]\n",
    "* Any cell with exactly 3 neighbors becomes alive (if it was dead)\n",
    "\n",
    "For this task, you are given a reference NumPy implementation that you must convert to PyTorch.\n",
    "_[NumPy code inspired by: https://github.com/rougier/numpy-100]_\n",
    "\n",
    "\n",
    "__Note:__ You can find convolution in `torch.nn.functional.conv2d(Z,filters)`. Note that it has a different input format.\n",
    "\n",
    "__Note 2:__ From the mathematical standpoint, PyTorch convolution is actually cross-correlation. Those two are very similar operations. More info: [video tutorial](https://www.youtube.com/watch?v=C3EEy8adxvc), [scipy functions review](http://programmerz.ru/questions/26903/2d-convolution-in-python-similar-to-matlabs-conv2-question), [stack overflow source](https://stackoverflow.com/questions/31139977/comparing-matlabs-conv2-with-scipys-convolve2d)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.signal import correlate2d\n",
    "\n",
    "def np_update(Z):\n",
    "    # Count neighbours with convolution\n",
    "    filters = np.array([[1, 1, 1],\n",
    "                        [1, 0, 1],\n",
    "                        [1, 1, 1]])\n",
    "\n",
    "    N = correlate2d(Z, filters, mode='same')\n",
    "\n",
    "    # Apply rules\n",
    "    birth = (N == 3) & (Z == 0)\n",
    "    survive = ((N == 2) | (N == 3)) & (Z == 1)\n",
    "\n",
    "    Z[:] = birth | survive\n",
    "    return Z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def torch_update(Z):\n",
    "    \"\"\"\n",
    "    Implement an update function that does to Z exactly the same as np_update.\n",
    "    :param Z: torch.FloatTensor of shape [height,width] containing 0s(dead) an 1s(alive)\n",
    "    :returns: torch.FloatTensor Z after updates.\n",
    "\n",
    "    You can opt to create a new tensor or change Z inplace.\n",
    "    \"\"\"\n",
    "\n",
    "    <YOUR CODE>\n",
    "\n",
    "    return Z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initial frame\n",
    "Z_numpy = np.random.choice([0, 1], p=(0.5, 0.5), size=(100, 100))\n",
    "Z = torch.from_numpy(Z_numpy).type(torch.FloatTensor)\n",
    "\n",
    "# your debug polygon :)\n",
    "Z_new = torch_update(Z.clone())\n",
    "\n",
    "# tests\n",
    "Z_reference = np_update(Z_numpy.copy())\n",
    "assert np.all(Z_new.numpy() == Z_reference), \\\n",
    "    \"your PyTorch implementation doesn't match np_update. Look into Z and np_update(ZZ) to investigate.\"\n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "plt.ion()\n",
    "\n",
    "# initialize game field\n",
    "Z = np.random.choice([0, 1], size=(100, 100))\n",
    "Z = torch.from_numpy(Z).type(torch.FloatTensor)\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "fig.show()\n",
    "\n",
    "for _ in range(100):\n",
    "    # update\n",
    "    Z = torch_update(Z)\n",
    "\n",
    "    # re-draw image\n",
    "    ax.clear()\n",
    "    ax.imshow(Z.numpy(), cmap='gray')\n",
    "    fig.canvas.draw()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some fun setups for your amusement\n",
    "\n",
    "# parallel stripes\n",
    "Z = np.arange(100) % 2 + np.zeros([100, 100])\n",
    "# with a small imperfection\n",
    "Z[48:52, 50] = 1\n",
    "\n",
    "Z = torch.from_numpy(Z).type(torch.FloatTensor)\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "fig.show()\n",
    "\n",
    "for _ in range(100):\n",
    "    Z = torch_update(Z)\n",
    "    ax.clear()\n",
    "    ax.imshow(Z.numpy(), cmap='gray')\n",
    "    fig.canvas.draw()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More fun with Game of Life: [video](https://www.youtube.com/watch?v=C2vgICfQawE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task III: Going deeper (5 points)\n",
    "<img src=\"http://download.gamezone.com/uploads/image/data/1190338/article_post_width_a88.jpg\" width=360>\n",
    "Your ultimate task for this week is to build your first neural network [almost] from scratch and pure PyTorch.\n",
    "\n",
    "This time you will solve the same digit recognition problem, but at a larger scale\n",
    "\n",
    "* 10 different letters\n",
    "* 20k samples\n",
    "\n",
    "We want you to build a network that reaches at least 80% accuracy and has at least 2 linear layers in it. Naturally, it should be nonlinear to beat logistic regression.\n",
    "\n",
    "With 10 classes you will need to use __Softmax__ at the top instead of sigmoid and train using __categorical crossentropy__  (see [here](http://wiki.fast.ai/index.php/Log_Loss)).  Write your own loss or use `torch.nn.functional.nll_loss`. Just make sure you understand what it accepts as input.\n",
    "\n",
    "Note that you are not required to build 152-layer monsters here. A 2-layer (one hidden, one output) neural network should already give you an edge over logistic regression.\n",
    "\n",
    "\n",
    "__[bonus kudos]__\n",
    "If you've already beaten logistic regression with a two-layer net, but enthusiasm still ain't gone, you can try improving the test accuracy even further! It should be possible to reach 90% without convnets.\n",
    "\n",
    "__SPOILERS!__\n",
    "At the end of the notebook you will find a few tips and frequent errors.\n",
    "If you feel confident enough, just start coding right away and get there ~~if~~ once you need to untangle yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from notmnist import load_notmnist\n",
    "X_train, y_train, X_test, y_test = load_notmnist(letters='ABCDEFGHIJ')\n",
    "X_train, X_test = X_train.reshape([-1, 784]), X_test.reshape([-1, 784])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADeCAYAAADYWw0uAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXeYZEXV/z/V3RN3dmZ3Nue8pCUsOQkrINEAikgGBYmK\nCBh/vqKI4PuqqGRQQZIoKIISVQSJy7Kkhd1ll41szml2YnfX749zq6dvzfRM94Tt2enzeZ55evr2\nDXXPrapb9a1Tp4y1FkVRFEVRFEUpRCL5ToCiKIqiKIqi5AttDCuKoiiKoigFizaGFUVRFEVRlIJF\nG8OKoiiKoihKwaKNYUVRFEVRFKVg0cawoiiKoiiKUrBoY1hRFEVRFEUpWHplY9gY86Ixpt4YUxP8\nzct3mvKJMeZBY8wqY8xWY8x8Y8yF+U5TvjHGnG6MmWuM2W6MWWiM+US+05RPNI+0RPOIkFaPur+E\nMeaWfKcrnxhjdjPG/McYs8UYs8AYc0q+05RP1B5hjDHVxpi/BXXHUmPMmflOU77p6fVpr2wMB3zN\nWlsR/O2S78TkmRuBsdbaSuCzwPXGmP3ynKa8YYz5FPC/wJeBvsARwKK8Jir/aB5JQ/NIM2n1aAUw\nBKgDHs1zsvKGMSYGPAE8CVQDFwEPGmMm5zVheULt0Sq3AY1IeTkLuMMYs0d+k5Q/dob6tDc3hpUA\na+1sa22D+xr8TchjkvLNj4HrrLXTrbVJa+0Ka+2KfCcqn2geaYHmkdY5FVgLvJzvhOSRXYHhwK+s\ntQlr7X+AV4Fz8pusvKH2SMMY0wf4AvA/1toaa+0rwN8pUHsE9Pj6tDc3hm80xqw3xrxqjJmW78Tk\nG2PM7caYWuBDYBXwdJ6TlBeMMVFgf2BQMJy33BhzqzGmLN9pyzeaRwTNI21yHnC/tdbmOyF5xGTY\nNmVHJ6SHoPYIMxlIWGvnp217DyhIZXhnqU97a2P4O8B4YARwN/APY0whq1xYay9Dhic+ATwGNLR9\nRK9lCFCEKFyfAPYBpgI/yGeiegKaR1JoHmkFY8xo4EjgvnynJc98iKjj3zLGFBljjkXsUp7fZOUN\ntUeYCmCLt20LUrcWIjtFfdorG8PW2jestdustQ3W2vuQIZsT852ufBMMYb0CjAQuzXd68kRd8HmL\ntXaVtXY9cBOaPwDNIwGaR1rnXOAVa+3ifCckn1hrm4CTgZOA1cDVwCPA8nymK1+oPVpQA1R62yqB\nbXlIS09gp6hPe2VjuBUsrQ/lFCoxCtQf1Fq7CamkC3mYNxs0j2ge8TkXVYUBsNbOstYeaa0dYK09\nDhmJnJHvdOULtUeI+UDMGDMpbdvewOw8pSev7Cz1aa9rDBtj+hljjjPGlBpjYsaYs5CZi8/lO235\nwBgzOAhpUmGMiRpjjgPOAP6T77TlkXuBrwe26Q9cicyELkg0j7SK5pE0jDGHIm5nBRtFIh1jzF7B\nO6bcGHMNMAz4Q56TlTfUHs1Ya7cjbmbXGWP6GGMOAz4HPJDflOWVHl+fxvKdgG6gCLgemeGaQPyZ\nTrbWFmqsYYsMd9+JdH6WAldaa5/Ia6ryy0+AgUgPvh4Z0vtpXlOUXzSPtETzSJjzgMestYU61Otz\nDnAh8r55GfhUWjSWQkTtEeYy4B7El3oDcKm1tiCV4YAeX5+awp4UrCiKoiiKohQyvc5NQlEURVEU\nRVGyRRvDiqIoiqIoSsHSqcawMeZ4Y8y8IJDyd7sqUTsrao8wao8wao+WqE3CqD3CqD3CqD3CqD3C\nqD06Tod9hoNVReYDn0LCZrwJnGGtndN1ydt5UHuEUXuEUXu0RG0SRu0RRu0RRu0RRu0RRu3ROTrT\nGD4E+FEQUxBjzPcArLU3ZjomWt7HFvWrZsqgdQAkg7BzkV4QArhme5JVa+JsrbHrrbWDsrFHcVWZ\nLRtayaRSWawmH/awwTXdtZuC7FBvJdDI9mQJAJvqZTEhUyuDCcWb43J8fesThhPEaaCOBPHs7WFK\nbCl92k+0M4+fdfvI6o4N1ZLGfhXbAaiOyWeZZ9ZE2gkarfy4MS7X31wr91uyMdhhW204CdEoAPHx\nYqfYx3Iu29DYapK71B4muJGg7MYHyj67D+9cubJp9qgPzl1vi4C0fFAnNo5sD/LBliAfNOQ+cTxB\nnFq2NVlri6H9OiTr/JGJDPnGxOQZxvvJPcb7JgGoLK1P7dM/Ks+/xMj9Fhm5f2frOivfN8cl32yu\nFzsVbZHtkU1B/nH1bStpydke0TJbFq2kfrg8owF9agDoG5F0l5qEXDvILya4qHu2H9dXy+8LmsJp\n6wJMRO67fqTYoV8fKYN9o5K2MiPXdHZ0NFmxfY0tob42wepF9Tsuf+wATJE8q3hlsXyWuh/ko3ir\nPAOztdY/dMeXly6kcZiko7RS6omKqHwOjYXrDVcHubzqytf6hJTNmuBze20JyYYG4mvWZ20P2AE2\n8epmd99Ah9s9vk02JeXds36OlC2blDKTIE4dNSRt0khSdpw9TLHk66YgXyf6SJrLS+X5Vsak3FdE\nmp93UXD7zgoN1n3KuRqSUi9vbpL7bKqT7UVSzRHZ4tWprVDPdhptQ1aG7kxotRHAsrTvy4GD/J2M\nMRcBFwHEqvoz9qtXMePS2wFosFIhlpiiTiSjZ/CXJ2t47oVa7vnj1qXBpnbtUTqkL4fcfTpP7/I0\nkB97NNlE6NorE/J9TuMQAGbUjAfgsY/2BqBopqwoOeofUrATcz+SE0WiofOuSSxlA6tZyZLs7UE5\nB5mjMyfWvdSDhqiNx8O/77UXAB+dKYXns4e9BcA51a8BsJtn1prgngGWxeXHBzYeCsATM6cCMOFP\nYo/oC2+Hjo1W9Qdg480DABjwNUlLfNGSUFpdQV1jl3eZPUyRVDi2SRreGz5/CAAzfnwH0PF85PIC\nwPzg3B+6fLA9yAdz9wGg/A1p9I14Zg0AifkL3Q0En0EjJ2jctFZhrbHLeZ/p6cuWtrBJTvkjE36+\nCfK4S1N04GAA1p0ka4xsOkYWTDp6YnM0xi9UzwRgUtEmAIbH5KW8LSl2mtMkL5QnNu0rnx9KeRn8\nD9mv8m/vyCWDToNrgKenJWd7RPty6NAzmPPDYQCce8DrAHyyYi4AuxRtBaBvRK5VZOT+5zXJNS/9\n8Ez5/TOyUJjLT11BpELqiXnf3R2Azx4sZfGoShGp9ixeC8CQqOTlaPCMlsfFPtPrxzDj2Q3cesWC\n7s8f3YlXD8SGDAdg3TFjANi0e/BzTH4f+W95NiXPvCk/uHo1mdhx5aUr8O572YVSr0781CIADquW\n+uI7Az4KHZYI6otoUH/UBuXr91tk/YpXN0sZfeP9idS+NYv1dz/Ypj0kKTvOJn7d/PFFh6Z+m3FJ\nx9o9rl525fevNbK43e+m7glAcrt0NNewgg/tW+mHdp09vPc7yUToa2z4KEnDp0YCsOEQuccDdpHF\nKo8ZIHXSYWULU8cMicqzLg3ua1HwOv+oUerjBQ3y7nlq5RQAVr4v34e9Jnmqz1PvAmlCTCtpfMM+\n3/69uXvIes+WtNbabvHGs9beDdwNUDp8VK+N45ahc9KmPap2GZJ3e7gC5j4nB22YyUH36+Q+swD4\nyWDJeNHDZYeFl8vvxzx/JQC7fUfaeIl10kgmEgFrfAu0aY9KU926PbyK1TWCG044QLZ/U6754pT7\n5bNO0njRm2cD8N+XDwSgapEU0FidFOR4WXPhqRkuRcEV4u8d8ZSc4zMrAbhipVzr1d/uD8DWYG22\nj/aRBugnJ38VgOKgMZyxwe7dfosN2djDNTADNu0R3i3hq49Z4vIAwB7FZcGnNKi+UCHP/4bB0imI\nThMbv3ulVESnTb8IgLG3Bg3PV2X/5goqc6PYI7RDVvbIhFc5umcRmbIrAB9+XRpsdx39BwCOLf9n\nFietCH0ricoL7YjgUkcME/v8MvhMTJP7/p/vSCfixf+Vl2PfP00Pp9GGXy7pyQ59SbNHnwGj7IZp\no1h84p1Ay5emn1bHXoGK8589/wzAycXT5NydaQx7ZTTSrwqAB06Q8nFYqeSX5sZO62mbECinfSMf\nU19c19ouXZc/upO0RixAzRelTXLtjfcAcGy51DN+46/pdNl/8j+lPO16uTQikrUtleKAnmkPr5yP\n+ql01Bqul+0v9pEGz8i3ZejtrL4bQvuvTUgD76RrrwGg+t6gvBhp+05OzmCNXc76Vq7cMik7wCZB\n/ndlKDpARl2uOeOxFrtGcpyq5SvIB5TKO+me/p8AmhvDGRZ465w9UvVTuP5uOlbeg2suljJ691RZ\nT6RlOQ/fa8KWtLiE22cv6UewV/DOoUI+Ux2mPYNznBHUqf8jdeo//ng4ACNvfw9Is4dp0f5ok85M\noFsOjEr7PhJY2Ynz7dSMHBZl2Yqm0CYK2B4llFFvQy8ztQdqj3RKKAMoTttU0DZRe4QZNiwKao8U\nmj/CqD3ClFBOkpBQUtD2yJXOKMNvApOMMeOAFcDpwJldkqqdkAP2KWXB4iaAYmNMMR2wR649xu7E\n9eySKZ/i4HvQMxwdE+Vw8fG/A+C2g6Vf9NQXDgagcs486tgGnbCHrzaZEulVzrtduoiLT/htaPfJ\n918KwISfiJo9bvusNk+frhu6/uqA38vnXxHl4tarTwbgratuAaDo2jdD53Bq3PYhUpRSNbPXI66k\nP3XUQGfsEWCT4e7u3vsuDH13Q86ZSHjKcjb4+cD1uPcMVMb5R94HwJZPSIP/gAeuAmDc90QRanaf\naH6mlfQHKO3yOsRT5SLl4tIx/3pxo3n9i78EYHBUXBucPRIpn7U21fwQztaxIDf5dnLbUyMrN0me\nHH/oxQBMuuKNVJor7UBIZm+PSNxStr5ZUXZ50VeSfHXG4YZpzZgRsmHO/LZvNheics1RsdogbWXh\n3zMoR463G6qxu1pgQ9fnjzzQ//KPgWZFuCYpPpTuGSSDZ+fyzeLjpCLa+49nADD05LndV152ACYm\n9+mU08aDZHTmuGA0JhHkD5cfDv6b1B+T7nk9OD5wK3IjbR0oL92Jf3+LvrELABdUNa9o31VukAMj\n8pZpHDsIgMjyFQBUmgEkbZKusIdv7+jEcQCsvkmu/fb+v/OOCCvCcYLR1yBfu9HK8kgxPn/YKu/a\n2xZMA2DDhmDUKJjHM2mUuOLdOfFPAIwrkt9/PFhcz274ptSpV3xJRm9nf3s/AGL/CbmMtEuHW1/W\n2jjwNeA5YC7wSCEvNxiLGW6+YRDAZNQeREyEXdgH1B6A2qM1IvLi+xitQwC1h080ZkDtkULzRxi1\nR5iIiVBKOag9OkRnlGGstU8DT3dRWrqc9AlB3vBBt3D0UcUAH1hr9+/2i3UzroferJ6G/S9dD3BL\nUpTAy/vJXMq375WJISuOLGEQY6H+1dzt4SnC0UHSAy5/TL4vniC90vWBX9lnvn01AOMeFhXBPWk3\nmcH3sXXKqom0oqAG923j0osf9kuZfHfc+6Lk/fa3vwZgYOATXBURRWPLRDm8fxu3NdAMA9uJ/OHs\nEiie0UqZSHH60Bmh3dobYcikxrWGm8DiK6DuHC4fuP2Kgt/nnye+olN3Px2AwSc3T0YL3YtlS1eV\nF1/NiOwjs5Om3vsBAM8Mcb61MoXfKTXOXq6+KDFynlzs1OT5/Prqj7tWQ1LStujUuwDYdbOMZoz5\n4etu16ztEWmIU7aw2dfS+Qrnkm6ALVPEv7GiCwMw2ZikZXRMVJyW/sxts6RxYCp5O1V96pXRSKnk\ntT2qVoV2c/nDt0fChuukf+0ndd2Z074uG174y85lj4DURNGAjy+U7wOj4UgGMxqknOz2q9UAuDEa\nfzQsbQJXfu3h+QrHhg0F4GdnPNBi1xit531XN7jJuL5N/PLs1NWaUZK3Kl1SIoZYsgibtJNzvQ2H\nX4fWnSzzba6/6W4Ajgiin7jynKnOjPp1ajBad+Wq5kc180ZRcPs+Kb6+1fUyMlWdSky4HXDJgVJX\nHnfPqwBcVS2TMd275+bhMmp77k/lnbzm0B3nM6woiqIoiqIoOzWdUoZ7OuFed3aKxI7EAolkc3+k\nWb3unrSm+4o6v8iOqGDp+1eZILZv0Lv9/ehXAJhy5WWy488eyi2RxqTU2Ui5dEOrn5BzPzj2xdCu\nhz4kM42dIuxUmGQQaqW92fFtus66cFyBn3LRPyW01sm3fhuA978pYXKcTSv2Cc+GbvvkHSQVrizo\nlU+VkBan9Hkh2EHyTab4lU7Fn9lQEewt3ebSIOZrVVoMyDGBQuD7eLmeftJT+spNEF/SxYkNfCLf\nOSDw87rnAgAmfznw4/LD4HQCX81oOFF8x355220A7FciaXMKgsvzvhbg7tXd4x+2DE399uByiQaw\ndqvYrm+Z2OqcseLz60ZGmv2Pwz6xTglsIqyQPXLurwD4ziPny4b3s7plIZnEbNvO7EZ5ri4CSKaZ\n3JnYPEH2az2+Qwfx/NZzrdsW1g/uytTsOLwyGhkueegL/R4NdpB8kHMM8NZGsXYGPP99s7+EyXrp\n8FsBSFjx53d59UvPXA7A5MUy2tXCV7iH4fsKz/+G+Nae3OdZoPm9CM11gF8+X6iTkvf/5n4OgLf2\neyR0bKru8Orc7cPleKcMS97rWD7x7bzlLJn788zPbgKgf1Sek1+HtuYD3Fpad33lHADGnjk3tU+f\nuNSdSVdXBHnFRWLysTOkcnz24iMAOOWP4iu8LCH2O+/FCwHY/cdrggO2Zr7hVlBlWFEURVEURSlY\nepUy7PdGTlvUHEz6zXnSY4uVBStm2e7qaf+/rPc0Sw3RS4q58k/iR/PrYaI+5upfly3pSpFT8rqL\ngUcHEV1+lsNBRnqFrnc6/y5xfVo49l6guaf8Sr0owBN/Ln6oiaBH6RThLllJy8U0doskBP7Hw38u\nPsSnfvYYAP4y4d8AfH3SiwD8ua/MIk5u2xbcU9jvqTM4H2cnOm/YXezg8omvJPjf939JfK7GnymR\nDSJ9Jc6uCWb+m/79UtdqGCOLiaybKtcYc7L4Zz068R+hc/p51eWxChP2y3Wz4w86V9LQ7/6Uj2yH\n8dWM+NHig3bH7b8BYLfisJrhK78uzU7F3uu18wEYfofsV/Rys0xb3CRxtEd6z/MfMQkEf9/Znwbg\nuh9IHNnjyyXf+CqQ/6z2KhY7zbtY4vLytRwMkLTYhkb+ECwU8/Oh7+RwcDO145rCG7owz2aLb6el\ntdVt7d5j8cvoxoNFGT6wpPXy4uOikbhf790ssVTT8+LOhG+PeZdKfh8WC49DvBvUs7veJfWmG0do\n4SvcU/B9hceOBuDOL94d2q01P2EXacH51V722lmy74ogppFUYy3ixft5p85fpqCjoweRaIs69KEb\nfgFA/6g8J1dfZVKCHf47x7XBxpwm+demjQi2XHhInrqNtz6q6vYniGF/+afOk++bRAGevE7aT/H0\nyEXqM6woiqIoiqIo7dOrlGHXK3E9p4/+uEvqt8m3i6LnL5fY1SzOYV/b0Ejio0U8/+dgycYrpWfT\nUd9hX115tV6+/+jcrwAQ79P8uFcfLHZ48HyJjuD8KnP1N3T4PeCrxkn8yJdzOYkVla/uczKDdeFR\n0svelJBYpc5v6cIXvgzA5A1ir2aFsPtUBH829NbvSGzWVX+SlfjO6hv4mR4ufl9uOdUsV6LLDu+Z\nbDqgKcOOgq8slL9ZHvrd1osak3RlYXPzyqbRxaKEDv2vHNzwKznXpz4nPn1n/+xJAC6qkhGAbEcz\nDrpCntn8PwZ5se1baB2nygQ2darMWbc9ATQrwr6a4afxkRpRY2+7Slb7Gv3kjPD505VRz5/N5Qen\nXPX/gyjdN82T6BnDH5ZoEU75zRQD2HHmJ6R+urHdm0/DWmxDA88vDyaPB8qwry62x5CRm3K5anZE\nOqezrAyezc5O4xkbQ9/bq9v9PPrb50RZm9A0vXsS2F14ZTQ6WeY3PHn0LcEO4bjTnw9GrSa9Jys3\n9nhfYa9e//BKWWb76DJ5fm3FFPbflePuC5Yi/2R4v/YiYCWGNngbEuQkhaYulEi1i/b/hczpmFAU\nVoSzjY3s39uqmyTMUjkypyY9glOuz9bfPzE/HF+/syNaqgwriqIoiqIoBUuvUoZb0FoY2cA/0ia6\nKbpEov1dfPp+3DWRB3xFaFsyWNXnDQkgGklTN0c/J/+fE78SgDmXS3QE358pW3wl+YCStTkdn8IY\n+l31cWhTxJuZPvilotS+8rkD+nRuNrTzW3pNYiMe/heJaLHwdIlhu/SzkqbJz9DlafPV6cN2XdDq\nfk7d9/27qudkGA1pZcW61Ize1Ox4OWfZE6KePvGe+LlveVJiun+rWnrpvrLlz6B2sSAPP1niNvPo\nn1pPU1t4M/ZX3yLq67mV64H2fYRv2xyslniKRIgonRfMXHejRik7p5XLpFOCvQIe2M6tcpd8XfLF\nFx75JgAfnX1HcKZkkAaxh1++LqoWZTkXZdgmkyTrG9i0Mqyi5jqytO+g5QA4nSWlemXjr+mcQb1n\n0ll/403by9rfqScRCSuFLmrCE3vfGewgSlumWLN+mZ3bKKNhk+9eJ793fYq7FT/KwoeXS6x4P+LJ\nqmDUb+Kd4Tvs8b7CnuL98Oec4i333dpz9uuh69fLKnyx50WNjZ90cJuX9lcWHTkkPKJjE4kOCcMA\nS78n9fmzQ6Qd4Neh7eHf29+3S33Y92WZa+Kerv8OA5rr0JKSlr+1hT/6lGzZjjL12ftRqzKsKIqi\nKIqiFCy9WxlupZdkE0HvwVd48ojp5sXxnBqejlO4hrwVdtpsb/WybBkY7YCyU1FGcure3DfexfAV\nxc+t8uZWnBvwRqCWuIgPrfU2uwl3LacQT/6BxDr89XFjAXjyBIlkcJURP/CUb3rzimu5461qFRsh\n/mkXDnk2tJtTI/wRghfr5JmWzfR66cEqe62peJn8uVzc5fgSUe+f/cY0AC69TxTiiog8M9/33B9x\nWHFckOld+NVsMGJ3l7YNFx4CwFv7ifrq+wi7NDi1YlajRI34x3mSZjtPZjh3ah5BYLtkfdh/b+zf\nJfbv7NPks8m6chXeL2Lk+M3JDkb5TSYoXuevYhb2FW+PaVUS+3MhEnGneXZ3LpnVq8SKOvdqSSZ3\nLp3Gj5qw6Gp5JiNj2fleurjvLsrPif++AoDJ88PzIjrkY7+jSIsUkIqyMH4sALefdG9oV1cvHPem\njBCNDEZUdjZf4blXSdQdFy2kLUXVH7F56K9HATAamS9gY22XN//dfMBAmdfxQSd8ZU00QrSikq+e\nFn6X5BrJyr+3+1fL+y+xbl1woVZGcQM12anr3376sZyumQ2XfnZ71vvuXDWOoiiKoiiKonQhvVsZ\nVoBmNTylBKZhMsT06ywdiZHc0C/Cks+WtVjtxqklr9WL3xlr1oUP7I7V3jLh1OjAp83Wis/bc2eK\nSvnbk04EYCSvt3pch/D8MRsmS+zSaWXh+06psKlVj+QZ3Ll6GgCJDcHM9k4oCX7cZefvduhMifAx\n68CHJQ2eEuyrGp+ZKrEib8vp4oEiEyhQx17+auhn/xq+Qn7WrVcBMPzNILJMoHK7e+oU3kiT89P/\n9idPz+74uDv+1zlfOlbbuZjpny6X8vTIy7JyXywSzlexYOiqKJJosa0kIgpZn5jYsDwiZXZYcTiO\nTCZf2UxEozuwTHeUdCU0UAoT0/YF4LXDnQ9pHyB7X+HFTRKdZtdbRNHq8fF20whFCggS/tFFw4Dm\nuNtOIXf2GHpbaegcPfY+PV/hyF7i7/v08a68yjureXVLIX3VVzcqML1eytH4ByQSj9PATbLtcuxH\nojmqSuqYORVSblOx7XOgqbqMNV/Yg6uqXwqlN9f3tz8atXybxKyvQuZxtDkPISa/+e+zrqBvDnKv\nKsOKoiiKoihKwaLKsNJjKC1vZNIBS1Pf/TiLL2+TuNGJLcGa461EQdhhOCUwSEPyPfG7HPle11+q\nuVcdRE84MKym+P6Ifi/9jfcl1uNkgqgJ3kzvriD6z/7yj4SIbqHS+qrGkZWyemBOynDAhi/LRW4Y\nIr7C/kxm//u3Vk8FYPhNb8gJnMqTiyKcY15zozDxxUvb2bPzxOo7d7xTJd1qit1BrnHLI5GdQBlu\nhfrvbgZgYFQU4Vx9hY96XqL7TH7Pj6HeM31ogRaqKUBsmIxeXXdKOFqMs8Phsz4PQJ//yMhSSmXv\nQXN50mkZV1h8wTOtculwzxean/GZL0lc80mL3gpfJMcsv2+xqK6RalFhO6IMJyqSbDmiuQLpaDQp\nn621wXwftyFV/jM/30Q3jPDaHCbpqDKsKIqiKIqiFCy9ShkuM+Fe2X3fvin1/7KrpPdU5PVMkkF/\noNRID/7jpmoA7vnmKUDaSmI7Qw99J6dfUS0nD3034++Lt8vMXaz0iHuEmuBUWLdCWaSlStLV1O1Z\nF/ruK8G+v1e/WV1fzP0IHoPeE9/plip16ysaHlC6ssPX7nt6+Fh/JrP//V9/EH/uIcnAV7gjZbmT\ncXPbpRPRRqL1nUtbTVKUof1eFcWqqT6wT12Qb1wo4USaOu75NxqXBZuC7cPlnAs+KVEEsl2h0BE1\nPdR3lNbzz5azJUbs9L0krrC730yKsO8r/HFcfIV3uUXKtrv7HutDm0Zrq2wuvnA8AKf3lSgFvnJq\n7grmf7BoB6Wyg2SIH/3fY5yvsCjEmXyF05ViZ4Px97X+TCON4TLVXmQnF7GpcfRA2X/psjb3b40B\nZds5a8qMrK+ZLU2Nub9zch09ygaTbUgdVBlWFEVRFEVRCphepQz7PYu9ikvT/m/dsc5XrtYWi4/f\nb8u0n7Cj6RtpYFr5R7gZ2H4vtaYpvEKNH9szr6RWKOvi8xrTwrf3hF3mhL67lYn8mcBOpR0yQ3ys\nU2pTV8Rl9m40tmYLANsC9aMk2vZa9lWR3KONmLJSIrvuzl2TfxtsCc/Ud/fv1LhX6+X7iL8uAZpn\nbXfk/l3kCbdKku0ipdg4n/MgKkmufoMAsbr292mLlYE9xn5plkuUfOZyj94xLnYon5SPXFfFi/XE\naBJ+RIHS5vfLkVdPz+lUvq/wkf8OfIXf2Xl9hSN9+qR+uvLMx0O7OoX00/NPkO9/eyN8rp7qK+y9\nYxZcJc8e+XpKAAAgAElEQVRldBA/OpOvcGu+t6fMEz/p6Atvy7m9aDaRdmJIuzaKX8/XjJLzVGZ9\nV830jdTzyYq5qe/+3I6OYq13HhdppGc+ZkCVYUVRFEVRFKWA6VXKsE82sxP9WKT1gbJhEj3fV6u3\nUWwMI9tQFGsapQfcJ+MevRs3Q/uiQU51EWUq08pzf6mR/SPzZbW4VKe8K+RrL/axLZVnU5TB78sf\ngXmj3ukYK7K+ZGNllOXH9GNykeQA3w/VKeFOjbn6w9MAqFqxQE7Qno+5p266WMoA41+R364e/LSk\nOiHKUHGOUkciUF76RUQNeq5mDwCeP0k+WZLT6QCI1YXrKj8KS3tEPUdlF20kJyLhCB3JvqVt7d0u\nsWg3SEh+RJAc1X3fN3bpNfumfntmiKya2V70CN+X9K0GURZ3+7mMrKRWh9yBq2p2FD8qzYqL9079\ndlGVxAD3ldN194wFoB+rgnP0UAXc8xW2h+0DwPRPuPg3Ugf5vsKO1uJKb7p/FAD9WQ6k5Sd3yaZw\n/oxmiGDjq87bh8lnR5ThskiSKcXbcPfTHX67OwuFe+eKoiiKoihKwdOrlWFl58LQ9mzzFjpOd/Zi\n24sr66tKue6fVRpEPXDqRM3+o4GwL3w6vlJ615IjACjbulh26MboG/UjRCmtipS1+ruvWr9TNzb4\nb25ru7dORQIO25x2zrAfqh9VY8v0wZImRBn24zW3wFO7zR4TUz/9dOjvAOgflfscG3NKd26+z83K\nodjptjpJY7wDM8EdsfrOKf0Jz08wtVJlR3yGAyLbOhf8uEuiSbQT4SWlSrqIDZnyhacSxsZKOfz1\n+b9tuWs7+pLvK3z6o98AYPzc18Np6mlKaSs4RdiNoJxy3n9b7OMU4YuXS0SXfveHV+bcGe4TYPk3\nJZ0ufnQmX2G/Dv7+mr1Sv1U/JJGpMs3daM9n2OHXc3VDOl5WDFDagdVi2z/xzjeyrsqwoiiKoiiK\nUrBoY1hRFEVRFEUpWHq1m0RWzuDeZKJSFy4mmselfguUJptkTaKOkUHYGp+KIm/54O6MqZarW0O3\nLMpgQq4g6/YKF1d/qC7qDXevmDsEgImIm0S7bgK5pMw714Y9/GVIwxOJkjZ8zVc3Tsj5mlUl9Zw4\ntjmsnD8k7U9mGTDHu8928ot/T2sPTC0mSv+oLLu6KVEbvpb32N3waKYQRf5iDM/MlGFUt1R2R4ht\n7/mTrXIl1pnlmFMTIeUcNi4PyYX+MkXy7BKbt4T3zxBSzs8Xc/6flKtjy5vHtdubOOeeuyur92+V\nhRIm/XKhpCXYb+eYOBd25Vjz1f0B+PGg21P7+HXTzN/JBLSB9HB3EM+VLPFJmSQ58+A7gx3kfjJN\nnPNdt/7+58NTv42IB4v+lJT4h8mls3ST8EkMzWFZeY96a5jTFOXAIEmZFknKlYjvJrETLB6jyrCi\nKIqiKIpSsPQqZdjv1cxqbJ7EsSye7XLMY+QcdT0w6Hsvp8YW81r9CE6r2NLq72UxeUZujYHuXKrU\nTQrxF7zI+TydUkBsSM1M7r0t9KsfesdXSgd7ce27Ukn3FazKY1fndPzcN8blfM1+0e18tuptXB/e\nV199NaNiqai4WS9t69ln02EtFRenCPkTZ7KlyAu5NGBm5yevxOp6vpq4w4hEU6peZJ/dAVj6A8kX\n350iSwMPjkk5umbWqQCMuWY7APFFS+QcKaVYjnN1QGKaqIRvHO+W4m0O8thaKK22uOl2Cfs3ZE0n\nlgjf0XiLbLg6cv/z3muxqysfN6zfBYBB974lxwa/7wwKOMDGb0recPfT3sQ5NzLw1xoJdDb6ruYJ\nwin1vzGYeOiFMIx5c04zTcb06/2RQza1ex+Z2Bjvw583HsSBw2QhEH+ic0eJFe0czzcdVYYVRVEU\nRVGUgqVXKcN1VnpcFUZCT533f1elfht8e9ADz1LxK+HN0Pce3WPvJWyOl/P4un05reIFoOXiASPL\nJazWR55PYIeWjyW8qILLDwtuOhiAmV+8CYDXGqoB2JaQUFjv1MrIweamcAixiqioiJPK1gAwqngD\nAPsWrwfgi3POlR2PyyGBNqygnL1rOE865cBfnnNVvAaAfrPFXs6KXaKkez519hAJtP/4Hs5fMLxE\nssOlzSkoo58Vey/K4dKlxrJLUR1+gHh/RCi12E48NyXcV6vG39v8/6FPXwJAJB4Oa5QM5haUbJFj\nl35a0rDo83cBzffrVGynHK1PiOI06A1RdTqj2SeLVNNIkUwQP2o/AH75e8mT+wQ+mi5fOPXrg4Mf\nAuBbf5kKwOxpouYltsry5SYaXoq39EeyUMTgILyW8xOGzL7Cvi/xifNOBGDIzfI+8tXWnoy/6Mim\nM8TOz466A4CaZLO0WRGRd/DD9x8NwPCmHq6Ae/Va4/EHAPDmfne5HYDsfYWveUmU/11q30/t40Ly\nEQ/qmViwWNLqtbK59aiULZPqaZj7DJBFPOZFojkvd7y5vozH5+zNL1PKcG7LpmeipMh7vt05v6eL\n0FpUURRFURRFKVh6lTKs7Nxsry1l+ruTYZwow34P+IiqDwFYWCEz8JPbAh/a9ha8yEBrfmv9PpRz\nnTrv9ND20RWi4N07+uU2z/mnbf0B+PXST4W2r5s5pENpxFqik8YDcErlQ8FGkRCc2uj7ed27WRSb\n5Acfeefq/Az91AIGwakafyz+3QM9tcwpYX4Q+hvW7ynfX/0g5yREMJS3osBlUojrh0sEiJJ3aDXt\nLfBGFqIvvp36v2+Wadz3/w0MfW9WWiRN7hk9uFWWX07O8Z5RB0iUdkPQ/M4SzYPOYkR53O1/JW85\nRXhLUmYZuIUuHE7J/PlQySAHfv5SAPr/QSIeOPVy+fcOBWD25OyWXIbmvOj2cVFImn4k9UCElUGa\nwwu99GT8+nLEhQtC350aDPDrTWMBGHWv1Nk7U7QMgKYrZWTP1SXtPXN/+2vHil95Iq14ZyqlziJ9\nzH+C/6TeyrQAlT9X4piq2QAsqNgfU5NbuYttiTD46RISR4Xza674fsxVZWEH6O6c39NVqDKsKIqi\nKIqiFCztKsPGmFHA/cBQxLXtbmvtb4wx1cCfgbHAEuA0a23HpzXuJNTbWmbzJg3UYzCMYByjzSSa\nbCPvMx1gijHmX/REe6TPtndKRKRz8ZSXrWji/CvWsnptnEjE8NWzK7niq/1YvqGO8y/dADnYo3iz\nZfSTSRpOdr3wcPacViZqyh+GHSMbUspwB9WVVuLtDrw7WC707vD2QMfhO+9IvMzrBov/br23vOqd\nX/wEHyx6jIZtvwvljxF2YYfzx7YpgwDYo1gUYV9tjad8F+X7X5eKH+/A5Hygk356GZa0XfAr8a1e\nOOXOUJp8ZeHjFY1c+I31rF2bJBIxbBwxhQH7HcHA+hdztke9tcxvsuyTISZmPNBYokEff/2ekpYR\nTwUnyDV2ZqRZmTFeOTFl8izc6MSyH4h6OGeCqIe+Pdws9NUrk5x/xVpeW7oZjGFsYjijzSTiMcus\n+KvQgfwRL+ucprE60af9nXLEFmWvVi9b0cR5V6xh9doE0aAO4RBo2loHMMkY8xFZvGMS1X3Y+NkD\n+M3w2+R7IEZlWiLcV9hKN4XrA3uolKO/XfLzYEvr/vCt4S+7vO9TVwIw+b8ST7qtMtnWO4Yc7NFl\neP60zif70QmyHHVDK2X/9/eKb/TwDZ33Fe5We3j3Vv+ZAwF4Za/wCyBXxXRYhlj5XcGyFXG+8o21\nrFmbIBIxnHJGCedd0IfGAeVs37qNXOwR3bSdqkdncvU1ct+/HjYTaPmOyZXhfWTEcIPb4N61kR44\nihWQTS0aB6621u4GHAxcbozZHfgu8Ly1dhLwfPC912MwTGIvDjXHcQCfZDkLqbFbWcKHVDMY4AMK\nyB6xmOHn1w5g9stjeO2pkdz+hy3MmdfITbdt5cjDS6DA7GFMhMmjj9P8ERCNwY0/7J/KHxvffZX6\n9asL1h6uvIz9zdcZfeNXU/ljcWIO1ZEhUGD2ALHJ//1wAO+/NCpVh2xfup7lf54BsE3fMc11CGqP\ngrdHLAb/98Pmd+5D99eyYH6cxZveIBYpptDs0VW0qwxba1cBq4L/txlj5gIjgM8B04Ld7gNeBL7T\nLansQZSYMkoCn82YKaLc9qWBOtaxkv04kgV8ADnaw3a3s4pTtVoRTreO6piPkGPYkBjDhkg26lsR\nYddJxaxYHecfz9Xyp0eq+dGNQJb2MNvqKP33e3xrlahsvxomgXKdb5+bxb3+kMEA9P9IYhF05cpq\nKSU06j7l4SQbRX3dFhd7OaXA+TW7HnSpLaU0Voot2kYRxfSJ96MplmBd4yr2jx3Fgnju+WP93uHe\ntD/jN2HDEQ5q33R+q/OzOX2zP3C0Za/dKTnOz3bhLwJF+EthRdhXEJyP3dihZYwdCt9fI37eVVtm\nUPXQTD7sQHnZEK/ggY2HsE+GmJi+j/kxXxQVbu7/Bqp24K+YtUoVci4OrhKoy04RbjhJZp0/9dX/\nC/arCNLix0CW7668RG/uS5S+lBPUH8nl7F90DAsS70GO+SNR0rnRnSWNA9vfKUdyUYaHDYkxeLDY\n1dUhi9fXsOG1BdAsLrVrk3g5bNjHtvDzjHr5wvf/vOBjWSWs7AnJL7ERwwHY/w5RySYXSb2TjVrm\nrzT3g7XiI7/b98R5NBvf2bbeMeRgj67C97VffH5Q7ozzg5fPb62emjpm5O3vAmmRbDrhK9yt9vAm\nEBRfuSr0PRv/cEiLYOOR9JeobANXZ7S3+pv/zt1tUhF16yxrGhYTLUsp0tnZw0o9+Modh8j362YG\n6c4tqoRf906rngfA45UTgbQILWkjbD0twEROzTBjzFhgKvAGMCRoKLsG8+AMx1xkjJlpjJmZqN3e\nudT2MOrsdraxmSqqaaSBEiMFNlt7NNHxZRR7IkuWNfHu+w0ctG8p69cnGTIkaKRmaw9b39ouOy11\ntoZtdiNVkQE0Ul/w+WPLiu1sS26kKjKwQ+WlblPvskfj5o1p9YfmD2iuQ/ruOozGTbUATZDZJqH3\nS03ver9Ay3cMOdijN+aPXO0BvdsmS5Y1MeuDJvabWkxjYw2RSI7v3F5mj86QdTQJY0wF8FfgSmvt\nVpPlDH5r7d0EHpilw0f1/CmFWRK3cWbxOruwDzFTRLYdwHR7VJpqC7BtdDdLw24mZ1pXLLr7ZAD2\nu/jdLrnElpo4p16wipuuG0hl3whNNsJ/ascCba9M5tvDNjUy/RbppUZvFL/cpmS4C7n+aCnA/e9r\neV+dxsXPdZ+JsE9ZpL0iE5Fn2dRYy3u8xOTIVCJN2ccSbS1/lE7d2OYxvlJV/WHYHiaYVd/sWy2/\np9SaQFkOqTfBtsiUXeXQW6Rnv2AXUYSdYuL7TzqFxG13+73w432Z/cIdTGZvoo2yj4nFgldZZtLt\nUTJ2pH3snX0zxsT0Yxk7/7fxv7kYgElXyEiDTam8niLu2yfteTXbKoixep7k0d/9+FcAjCuqCF3b\nfya+sjTo5c288+49ofojG/WstfwR95ThTCtXZWJ544Cc9s8G24FoEjXbk3zxgtXcdN1Abu5Tkt11\n0vPHuJE2Wdn8zPwRk0zEImL3zefKM/3St58D4KpqGXnKJXqEe+4ugsWr35WRlOJNQYxwz0e1LTry\njmktf3QYLwZybPxYAO489IHQbi7PT//Jgalt5bXB8pc53G97dOU71x8d2n7qQQC8sttdoWOz9RXO\npOZ2h4esu5YrL7++bhCjqvpAZBPJovbbZS3sYQwDfidzZfb69BkAzDrwYblGMCqbHimkNfyRsLP7\nLgHgb2OPkA2ztmZxZ/klqxrLGFOENIQfstY+FmxeY4wZFvw+DFjbPUnseSRtklm8zlBGM9iMAKCY\nEhqsVICFZo+mJsuXLlzDGZ+v4PMnSaOgakARm9cGy04WmD2SNpGWP0YCmj/mvXY/A8dMTSsvpQVt\njw9m/5EhQ/ZWewS4OuTMtDqkuH85QBEUnk0yvWNQe6g9kPJy6gWrQuUlVto3JeIUmj26gmyiSRjg\n98Bca+1NaT/9HTgP+Fnw+US3pLCHYa1lDjPpQ1/GmMmp7YMYziqWuq9Z2cOUFBMdM55jvzQ9tD1X\nZcfhq1GHlMoKZC/NCHxui5t7Z5/vew8Ao71Zr7nOHrXWcuFVa5k8KcbXLk6LxrrfXvzyntQs7uzy\nRxAntP990ks98LQvAjBj6qNAs0Lz6CdEnfzhiJMBiK+SVd+6UoXoKDaZZPbHf++S/IExmKJiThoz\nO7TZ5Q9fhXwp8DKp+q8oWs4KydraIHGtSynRgaIM1hw2IbVt3dnSMPvPQbK6lJsd7SvCTqVwaXFq\nrVNU9vr15ax68mEq12xm0trSlCIzqGkYK+ML3eWyskfxRhj/sGXTcXI//aPlre7n7OHUukWnitqz\n79gvybWvD+LNzpwDpCnA3kqGkdJmNaTmBIksYC6R98uMPe8I7jscLcL5ijr8Z/Tw1n7c+e3FVNQO\nY4IZSQKZ7D2IYayyi91hOdWnTRWd8xle2dAv+K/rhkxtLPs6zFrLRVevY9dJRXzzkv6p7dWHTGDF\nozOdbN2+TZJAY/N1/dinDl/tu2tkEEHmZ/Lp8k1TkA+yUQf96BH7Pyirn45/Vs6ZSzSFtt4xS5mf\nvT1A6pBY6+lvbxVWd5zbb8mXxJf62PLwcM5uD18OwIS/pb3HurAu7lJ7uHN6ozDVX18a+p6tr7Bj\nebDq58akPOdoDr7CiUBVHRqVNLm5MZlw79xdJhXxtYsrU9vL99iDrR+kYqN3qE028gJ5l172rIxo\n3D5CnqlTiMuC/O0r4b6fvqsHPz5JVnAdOUv2c1F4AKyLBtVDyMZN4jDgHOB9Y4wbU/8+0gh+xBhz\nAfAx8MXuSWLPYgsbWM3HVFDFdPsvACYyhTHskgoVBWyhQOzx6ox6HvzLNqbsVsRBx6wkYgzXf28A\nw047iQU3PA4FZo/N25exatP7zfkjYZgY2bNg80fd8sVsmT2ThLNHk2FidK+Ctcf8t2p45YkNVEQt\nr235K9YmpP4wu/K+LTx7gNQhD/2lhim7FbPvMR8DED9jEaNOP4gVj86sDEJF6TuGXVjKfLUHhW2P\n9HfugceswBi4/nsDGHDo0Wx+63UKzR5dRTbRJF4hs9fV0V2bnJ5PPzOQYzi11d/240j+bf/ygbU2\nK7vYMZbEnY0pH8hce6Pt4eJr3jBkVovfElZUtc7GEzzkwBIaV45PzZp15/nxd0azy95XMPO9q7O2\nBzZYqSZQFQZ+RWIV/uA5mZF9/WBZ532fYlFu5vxIhssmf1WiADvf2FTY3S5QJVL+pBG32pt8umfl\n/BLdffevGM2x+/wPyffmBsc5f9REzvnDlJZgJo7nourfBVtaj1TgqDSi7H34w3EAWDM2vEOZ2KNP\nlfTyJw8UlfPqkeIjeXDJv1K7Nq/qFs4nft50dnBqtft94sOXADDpj9OZZE4NK2NJwOReXsy2WmLP\nv8XhM74KwOxDZEW+TH5tvmr99v5/lv0fk/1v3ihq71MrZTW4TTVyr5Xl8vtl419MnevcyvDojVOC\nnfrolBA/9rHv1/zSkBOZ9vyJxIJGnylqtst+5oic7OGoG9I519CPa50a27Z/f5t4ow7x8uwXNz38\noDIaV8oqi85u0z4Y736eb63dP5vzRGsNA96Kwmfle7YjbO6ZuTjVfoSYTPvX2WZl1eW93V49B4Dx\n3w3UZjehKYf4um29Y7DZ2yN1SDsKcAtSvsJB2e4jSuW5Z/4rtNuE578MwMRrpGy4Mi7HdiCmeQa6\n1B7B6KNL39YzRQF9fZKMNvorB/r45XtWo9QV3zn5ItnhI1GYTVlQF2UYjQMwRXKN+GpRYz+6WfyW\n3UhWptGmww8qI7FqYovfL99WQtHQgTQsXT4p40Vbwza/cxMbZH7K0pNlFODsR6cB8ODYF4Hm+j6e\n4X3gl7lvnfcXAB79k/jjxxeHFXgAG+3cyFZXoSvQKYqiKIqiKAVL9t13pcsxQDTSPPO/o77C7eGr\nd/K/9MY6qgg7tc2d2ykj4//9FQAmPfB6xxKbTDT3UteIcvnO58YC8LW/iHJ36wiZqbzgROlBT736\nawAM++VroVOlKxXptLZOur/CmNvHV1VigWqS6hEHhzXYsB9dVxAvj7Jx3/4pv25flfDZJ1DGF51y\nV6u/Z8apwM150akObhXAiKeI+0rw/CYJa3X6jdcAMOHO3H0ls2Xs1eJr9tBz4i54Vl8JNZrJr83l\ncZd29/v3B84LfbaFy+81ySCSSeCv7KIG7PoP8Zu845j7ADi+vCF0nLPT8jdkNGMsogznvCqeTyRK\nw+DwCEgmX9lMLNssPsODO6MMezRV7vhXS9GGWgY98A7fv0xiWrsRsWz9Hf14xA5fOXakj0RMmX4W\nAGPOkBEhm/I/z18wVROJEKmoYMlVMrI29PAVACx/W1S/CT94C2iu40xRWIF02z/8hYycPDPgVQDG\n/V1U0MmXSFzmlPrdiVjCOwwbrovGXR4u+/4qlj6+b/gpr1wKwMR33wGa67vk9vbD/Pn2jtXkVhf4\n5bzP0O1EijqY37wV4uIrZLR1w1GSx8fdeiEAH54gcyVcfebqN3+E2X0/v1Le4Usflzjbz//w8NQl\nXVxvk+jcyJbDL6cxotic4jwriqIoiqIoSoGiynAB0BEfZNfLcr7AzvfRRRFwSqFTR3Z5+VwAJn9F\n1JhmZaQDCfZ7qUuXAbDwKIlWMeF28VVbePS9AMy6+nYAJu4r2ydfK1E0EgtSs/PbJZOAYw4QVWXD\ntaL03Tz8EaD5fvuUyXbnj9qVxPvAuoOaE9aeauFwSphPppGH1pREf1sm9ezUhccAsO3qYQAMmtF9\nijAAkSjxJaKqPnDWCQDUP/BvAC6oEmXTqRJxT8X2YyL7qobL4wnn/562v1PInSL8z1opUz/5tsQw\n3v118YUbf4KLCS1+lv7Iy6B3vIzWGeXQGExRjNIBdaHNmZ6zX6ZTMXGXVwEZIvR3kKbycP5xvvUJ\nwqMbfpqcteLJ3HUaay3J+npmXCUupDPuEeXzwBKpo1LP2XvePs5+vm++q0dd+ZrywBWp38Z/T/xm\nW1R3bfiMdjf1I8uY953dWfR5qR/d/bO7fBz9uqiaZY8HCq8XX3v+7RI3ePZnbgVgz5uuBGDyL2QE\nLlXGvXjlOwMupvRz41xUmNZ9YB0unzr/3NQKm/dkyKfZjM54o5HRhtxGdPxyPmXIKlYXdXKE0r17\ng/S7lVcnXygx2088WuaCjP6prG567+iXg7S0vnqes9O1gyRyz/dvfz/12903jgXg5U3rgJYjn/6q\nfn5bxOFHNnLvqE2J2lRdng2qDCuKoiiKoigFiyrDPYhc1wPvTpp7Waktod/v3Cy+j3ffLFO3xwY+\nojbShWn3FOJkEJdw4jninzXtJIkqEP+6+Iwu+KQoxVuOFKXsM7PPBGDTv0W1HPCB9FKLN4svnC1u\n7gvWV0uPf+1+sm3PI8XH6aYxohzcvVGUhAN+IGrK2HvkfiN9A7X6Oukx77J6UfgeOqH8lZY1svse\nH6e+Z+tT3t5qQR3B+QSf/cH5AJTfKn6mJc8EK2sZeQbdpgg70nzK7cwPAHjskxIV4vobPg3Af47+\nNdC8KlwmX2uXU331trX9FzdJHNGjnpL4sbv9eAkA5WvEfz0xVfwqx8XCtndK07sNMoJQ9bqMcjjr\ndMbP0kQMkZISDhjZcoZ2a7Qs08KoZzz1pAsUvoZ+YVs7G2fy1/XLycCyDi6tHIkSfUGi81z7GYns\nsPUXUt6f3ONBID0+ddvRIlza3mqQ4895S+ZDjPo/UcHGz2ieF9ETVVLTBKWrmu/RqZ9O3Vz2abnP\n3acPAWD9sRLBo+Jc8S3eJSKfX/j0+QAMfzdQhANfVxdtoifca65Muez99ndKwx+VO3HuFwCI/UdG\nH/zV+rLCm7sS8YJ+tOf7749cfKp6DjOirY8K5kzqmaatFgrEnpf7XfWS5IEjThL/8VWnSeJ/e9D9\nAEwrC48kO9Lr2sv7LQt9+tpspvo6U7l9sU72v3D6eQCM/Z1h0eLbWt23NVQZVhRFURRFUQoWVYYL\nHKcW1AYxMzcGysYb9aMAeGWrrPjz1PviOzv8WckyVc+KD9CgrZ6PaHcoI54fk5uBX/LUm8GnbD5+\n/7MBWHC6KILHHilrxJx/mfj57l4k56kP7nlNorkvuDkpkRge2yT+hn/7YB8Azr3xm3KNp+Va1YTv\nN6VWf1N8BlvofJ2ww9DiLXx79DOp75niC+eKi4DwQaPc87v1YwCYu314ap9nPhTHwuoXROkc9IrM\nCu7/0QLZwd2XNxLQbYpwOr5P+dr1AEz+SrB60hSZ+bzwbImfu8/h4t923lCZDX9QiajYfQOFbF1C\nVNs36uX+H14jvpLvvdocrnPig7JS3OQPxL8y4d23WSAK/lFfv0w2BOaxQRYr3hbMtl75VvheOlNO\ngpnxM1eIordppKzMVxmMDDjf1m1JeSYrE3K/79aPBuBfG+UZl/9b/Py7Mu5BtE7u6/m6QBEOdJfx\nRbIK5pCo5D1n+w8aJTLI05slEsSH/x1Ph0gbOUjOlkgBFcdLWr50iIzqLD9K/LkbdpNy0K9KVOja\neklT08fye7+5Ut6G/Gs5AKOWykiEq4dCcXV7kCLsKF61nVE/fY09GyVPXnS+VJRnVErd/e7xtwBw\n3dTDAHh8ruT/4ttlJC32mIx6WBPEcXd1fK5xi3sQ8YF92HDKIfxt5G8AWBUPxwvPhPN5HxiVfLz1\njzIyWh1EhXHx6DtT/8Vqw99d2Sg2bdu7KHgv7Fm6jDJfXu4swX27+/Lf887ffPzjsvvPR8o8ju8f\nLu2HdftK2obtLfM5jh82J3XqMSVSb08qlt+mFIV9gpfF5XNNQt7nr2yXtsg/V+0mvy8eBMDQ/8oz\nqX5xCQATVr2Tln7PqG2gyrCiKIqiKIpSsBi7A3uyxph1wHZg/Q67aPcykJb3MsZaOyibg9UeYdQe\nYXpOSVwAACAASURBVNQeYdQeYdQeYdQeYQJ7LM1wnp2RTtkDel0eUXuE6Vx52ZGNYQBjzMxcl5Ps\nqXTFvag9uv4cPQW1Rxi1Rxi1Rxi1R5iuupfeYhO1Rxi1R5jO3oe6SSiKoiiKoigFizaGFUVRFEVR\nlIIlH43hu/Nwze6iK+5F7dH15+gpqD3CqD3CqD3CqD3CdNW99BabqD3CqD3CdOo+drjPsKIoiqIo\niqL0FNRNQlEURVEURSlYtDGsKIqiKIqiFCw7rDFsjDneGDPPGLPAGPPdHXXdrsAYM8oY84IxZq4x\nZrYx5hvB9h8ZY1YYY94N/k7M4Zxqj5bn3SltovYIo/YIo/YIo/YIo/Zoib5zw6g9wnRLmbHWdvsf\nEAUWAuOBYuA9YPcdce0uSv8wYN/g/77AfGB34EfANWqPztljZ7eJ2kPtofZQe6g9eq5N1B5qj/b+\ndpQyfCCwwFq7yFrbCPwJ+NwOunansdausta+Hfy/DZgLjOjEKdUeLdlpbaL2CKP2CKP2CKP2CKP2\naIm+c8OoPcJ0R5nZUY3hEcCytO/L6XxhzwvGmLHAVOCNYNPXjDGzjDH3GGP6Z3katUdLeoVN1B5h\n1B5h1B5h1B5h1B4t0XduGLVHmK4qMzuqMWxa2bbTxXQzxlQAfwWutNZuBe4AJgD7AKuAX2Z7qla2\nFbI9oBfYRO0RRu0RRu0RRu0RRu3REn3nhlF7hOnKMrOjGsPLgVFp30cCK3fQtbsEY0wRYvSHrLWP\nAVhr11hrE9baJPBbZOghG9QeLdmpbaL2CKP2CKP2CKP2CKP2aIm+c8OoPcJ0dZnZUY3hN4FJxphx\nxphi4HTg7zvo2p3GGGOA3wNzrbU3pW0flrbbKcAHWZ5S7dGSndYmao8wao8wao8wao8wao+W6Ds3\njNojTHeUmVjXJS8z1tq4MeZrwHPILMZ7rLWzd8S1u4jDgHOA940x7wbbvg+cYYzZBxleWAJcnM3J\n1B4t2cltovYIo/YIo/YIo/YIo/Zoib5zw6g9wnR5mdHlmBVFURRFUZSCRVegUxRFURRFUQoWbQwr\niqIoiqIoBYs2hhVFURRFUZSCRRvDiqIoiqIoSsGijWFFURRFURSlYNHGsKIoiqIoilKwaGNYURRF\nURRFKVi0MawoiqIoiqIULNoYVhRFURRFUQoWbQwriqIoiqIoBYs2hhVFURRFUZSCRRvDiqIoiqIo\nSsGijWFFURRFURSlYNHGsKIoiqIoilKwaGNYURRFURRFKVi0MawoiqIoiqIULNoYVhRFURRFUQoW\nbQwriqIoiqIoBYs2hhVFURRFUZSCRRvDiqIoiqIoSsGijWFFURRFURSlYNHGsKIoiqIoilKwaGNY\nURRFURRFKVi0MawoiqIoiqIULNoYVhRFURRFUQoWbQwriqIoiqIoBYs2hhVFURRFUZSCRRvDiqIo\niqIoSsGijWFFURRFURSlYNHGsKIoiqIoilKwaGNYURRFURRFKVh6bWPYGHO6MWauMWa7MWahMeYT\n+U5TvjDGjDXGPG2M2WSMWW2MudUYE8t3uvKJ5o9mjDE13l/CGHNLvtOVbzSPNGOMedEYU5+WR+bl\nO035xBhTbYz5W5A3lhpjzsx3mvKJMeZBY8wqY8xWY8x8Y8yF+U5TvtH6QzDGlBhjfh+Uk23GmHeM\nMSfkO10+vbJBZIz5FPC/wJeAGcCw/KYo79wOrEXs0A/4F3AZcHM+E5UvNH+EsdZWuP+NMX2ANcCj\n+UtR/tE80ipfs9b+Lt+J6CHcBjQCQ4B9gKeMMe9Za2fnN1l540bgAmttgzFmV+BFY8w71tq38p2w\nfKD1R4gYsAw4EvgYOBF4xBizp7V2ST4Tlk6vbAwDPwaus9ZOD76vyGdiegDjgFuttfXAamPMs8Ae\neU5TPtH8kZlTkY7Ty/lOSJ7RPKK0StBh/AIwxVpbA7xijPk7cA7w3bwmLk94nQAb/E0ACrIxjNYf\nKay124EfpW160hizGNgPWJKPNLVGr3OTMMZEgf2BQcaYBcaY5YFbQFm+05ZHfgOcbowpN8aMAE4A\nns1zmvKC5o92OQ+431pr852QfKF5JCM3GmPWG2NeNcZMy3di8shkIGGtnZ+27T0KW2DAGHO7MaYW\n+BBYBTyd5yTlBa0/2sYYMwQpQz1qFKXXNYaRYasiROH6BDKENRX4QT4TlWf+i1TUW4HlwEzg8bym\nKH9o/siAMWY0MpR1X77Tkmc0j7TkO8B4YARwN/APY8yE/CYpb1QAW7xtW4C+eUhLj8Faexlig08A\njwEN+U1R3tD6IwPGmCLgIeA+a+2H+U5POr2xMVwXfN5irV1lrV0P3IT4qRQcxpgI8BxSOfUBBgL9\nEX+mQkTzR2bOBV6x1i7Od0LyjOYRD2vtG9babdbaBmvtfcCrFK49aoBKb1slsC0PaelRWGsT1tpX\ngJHApflOT57Q+qMVgrbIA4iv/dfynJwW9LrGsLV2E6J+Fuwwr0c1MArxGW6w1m4A7qVAC6bmjzY5\nF1WFNY9khwVMvhORJ+YDMWPMpLRte9PDhn3zTAzxGS44tP5oiTHGAL9HVPMvWGub8pykFvS6xnDA\nvcDXjTGDjTH9gSuBJ/OcprwQ9EoXA5caY2LGmH6IX+h7+U1ZXtH84WGMORQZAi/oKBJpaB4JMMb0\nM8YcZ4wpDeqQs4AjkBGngiOYEPQYcJ0xpo8x5jDgc4jqVXAEZeR0Y0yFMSZqjDkOOAP4T77Tlke0\n/ghzB7Ab8BlrbV17O+eD3toY/gnwJtKDnwu8A/w0rynKL58HjgfWAQuAOPDNvKYov2j+aMl5wGPW\n2oIf6g3QPNJMEXA9Un+sB74OnGytLeRYw5cBZUjklYeBSws4rJpFXCKWA5uAXwBXWmufyGuq8ovW\nHwHGmDHAxYjv9Oq0WOVn5TlpIUwBTxpXFEVRFEVRCpzeqgwriqIoiqIoSrtoY1hRFEVRFEUpWDrV\nGDbGHG+MmRcEli7IlXfSUXuEUXuEUXu0RG0SRu0RRu0RRu0RRu0RRu3RcTrsMxyssjIf+BTiOP8m\ncIa1dk7XJW/nQe0RRu0RRu3RErVJGLVHGLVHGLVHGLVHGLVH54h14tgDgQXW2kUAxpg/IeFlMhq+\n2JTYUvq0e+KGUbLP8MpNAPSNyEI2rtm+Oi7xzutWlwMQ2bS9I+nvUsrpSwN1JIi/Ya0d1JX2yJb4\nIDnXiCHrAYiaJAClRiwX6wKvGBs8BROEGF0ZL5Vr26hcKyLhA8v6DKSxYRuJeMMOs4eJyP0lKmXV\ny6YqSWtluURyqYhKPqow8hkzzWFS66z8vzFeAcCWWjlH8ebg3FtqvYsFn1n2JTuSP6J9+9jYoH7s\n2XcDAEnvYpEgEQsbZOGrxLxEkLYgcTticqx3regukg8mlEhQikxpfn/bAGLDBhJftb4p2zok2/xh\nYpKGREUJAE0Vcs1oaRyAiqJGACqjki+KjNitGCkvMRPko7S0NwX/xq38VpOUfL8tLtdoaCqSa2yX\naxVtk3Pauvp20+sopy+1bOt6e7hnFIu5DcFnsENQbmzEhD9j8pkMDguKeGgbMTFMJCq2K44GtoyK\nrUsi8llqmoI0y/dYYNtokBZXn8zdPkD2WxqnPNqP2sTmLrdHd+Dqnoaxkh8ikaT7xdtT7rtoRWDj\nbswfsapyWzK4qsPRoE2WlVtzdmq5v0ntY73tNnRsJPjuylfjBrFjbH3md3uu9oBuyCM5vgfCxwZ5\nPxoNbbaJoB7Psf4upy911JC0yR1vj3bsYIqkfmyqKk5tS1QEZaEoqBOCMpNIBnkgLnaJbJfvxZul\n3raNXohi79qmrDT4bqlr2kJjvDarEtCZxvAIYFna9+XAQf5OxpiLgIsASinnIHN0+o/y6T30Bdcc\nDMC1x/8FgKPKlwDNL6Sfrz0GgHd/sQ8Aff88XX6IBJkqmejA7XSONXY5G1jNSpYsDTblbo9c8ey3\n7ouHAHDD1fcA0C8iDbiJRVLhDox2PtMnrGTYaNBguHbdHgBsapKOyaSytQDcec4wNq2bx5qP3+x6\ne7jnHKTF3X+kTNJQc9SecsET5PcTpr4PwGGVHwFwaKkkaVC0Oft/0CiF9Y8bJe/94729ARj9uNxn\n6T9mhNMdNCyyrbg6kj9iA6sYecNlzJj2BwAavDjlJUbSfNoisdmWw6XRbIqkwrFNjW2mqSvwr1V1\njzRmHhn/PJA5zZNePJ+aNz5g7a//lL6sbQubtJo/Um/foHPnlfdof0lDzWES83/lEbJf5STpXB86\nbAkAx/STSFijYhsBGB6VexgYlY5QTbJ5NdnVwSXWJKSz9Np2WW/hpfUTAZi/fIhc402piIf9V86Z\nnOWtOJoh74LkkfeZnrs9fLx6MFIqaYoMFLsQdBZsUZCHS+QZJiqlAZIok+0N/eSzbqDYr35A8zul\nYUDw4qqW59u3v9Q1I6sk+WMr5P7Hla0DYFLJGtkekzxaHZXjqoK0lhq51r7TzwdgzCVrWV2/kPe2\n/rPz9uhqWnlvRSqkQ7rwhvEAVPSpD3Yx3qFyzJD/EZsm35sbnKD9d1eu+aN4cCW73fxlIqbtuslv\nqKbuqZ3josHv0aARE0t1AJqFGHeO4kiQF01431jwvSzIDxsa5B217CGx48DfBfVusmVHPxt7yCHd\nl0dS74GkDaczm2NLgo5Tv6rgWDlHcutWOWdDhtWsM9R/a+xyPuTt9D13mD3aex/Gho6QNJ4wOrVt\nwyHyzIcPl7piQJnUIVsbpL5avq4/AH1mSH088nFpbsaXLnM3Ih9BZ8LGpVEdmbirbLeW1xf8Put7\n6ExjuLXWdovSY629G1nLnkpTHe4OBkSDzDDkWSkYJQ1LALj9+lMBePh9ye8NQ6SgrPqqZJK5v7oT\ngHFHXgTA5Muk4LR4MJKQbO+rK8nOHh0+e/jwwfdIQbj50cNlwwDJTPEBYrfGtF5Z3SCx0QXfk1CQ\nF1WtBKDJis2KTLi36ho1rjFz9JzPAlD8aWn8Juul8p9nquX45AKirGuR4pa3kL09Us81yPSOdZdK\nJ+Azl7wEwI8H3dXWaYCKFlsODjqTBw9/E4Cbg8/a46SBdOY1nwFg08/HAFD6ZLhx3MGOWNv2qBxp\nx9wdYfahomDuUewaaWJr9yx6Mi4/VUTEwLMaJe1j7o6wdq1hbctDQjZpLX+0qPz23g2ADy+X5/q/\n0x4B4LSK57NMZbH3KfSPlqf9L5+7BerxtDIJsfv9gUGo3V2DHaWfzqpv1QBw8WKpw1b8cRwAgx+c\nBUBye6B4ta/it2sPH2efNZfIe/DICySvnlj1z9B+fYIRt+qIPJOhUXev5XQfJd6n4OqX2Yc8BMD4\nay9m+9vbZM2qMDnbo6tJ5b+098vGk6cAMP/IO7I6x6G7XQJA32DpI+PU+GSmIzKS0R77711qZ0zd\nudbQcWJL/Fqx7ZfO+jQAG34zFoDyx2fKjjZjPdu979yIp+J676LIXlIRrDxa3oPb9m5u0E4YJbXd\nblWrAdi7TyDKxKTxmwxGbrcmpK6cvk062v/8SM5Z/ZzU/wP+/I7sH7xzU2lq3STdYw+vQe7s4Br4\nWz4/VX4/V9oAT0+R9Wg+ije/s8598ysAbP33UAAaN0pSGvoFZWGk5IU9TpMO45VXSv11/ltfBmDM\n9fJ78l0RvjefI+2AZ2/4JQAHPXg1Dbdk38TtzLj5cmSZX8dIYGUnzrdTU0IZ9YQWVlF7qD1SqD1a\nUlJSBeEWaEHbpIQyUHukiPXT/JGO5o8wao8wJZSRJNSbKmh75EpnlOE3gUnGmHHACuB04Mx2jzKm\nhbJT+ZR8f/G9yQBMvlhUuSqkB+Uer+tTjA4Ejl2vuxSAxRdKb3xc9Kuh452SmH6t7qKS/tRRA1Bs\njPn/7Z13eBzV1cZ/s6tuSW5ykatcZDAYMAYMoROaAwFCDTVAAJuEZtpHQhIg+RLyEUoIvQQInYCp\nSYhpodoYA46NDQZj3Issd3Vpy3x/nHtnNVdaaVe7WsvWfZ9Hz2pnp9w5c+fO3Pec854cErVHGqHd\nKhHtXtkgLsmAcmXnNnObN54h4QA/LvpOLZFZZ8Ag/PVMPWDMm+r/OgiArIYVQEvWttjpQ72bHnuY\n+9az774PSP94oywxNmZDRNi49REVC9nsXPsoN15plp81LgiI7V4pl8qzoQdlvXEfyex01M9XARDZ\nuEk1tnWmr0P9o7qO4Htz+M0KYeFfGv2WtKED9NG2gtnWm1YIwx58bw695Le8pMcQNX4su3EfAKb/\n5FYARmTLtYt4xwyoNsg10w+KLEyvh/Qr7Q3Rn5Fmbdfr5Dr+ITNs0DF637ofvTz6dWnyjdKWC84X\nr82aydKHvTCKQJBitwSiHbCH4ZX45j4J8Vl67H2+8ze9PTEupHUmWJ+/Gfcty1SYhOrnxkO4ZRMT\n5V307ZMbJXvMYOhI/+hkeC7xZvd4358Ky6dtrUNsCgLy1Gpwdey09J/qYWKPIr0Dp337FNNbdpGg\nPerdKF821TNGjf/tXaNMINF+oO+jv4+WCsa594gdxw/+OQAD7p6ZtD1SQaveZqDqTHmOll4sz9GH\nRogrI7WQRPEq/aRYcn8ig2YCEDxUbHfn/5QB8NqV4obKflPY8uJACdFolE61h8GM6zGn5jSxw/6/\n+ASAWwc+4FttxBuXATD2ysXesuFbJHwxrndMLd+oll/zo0sAeOCOhwDo94qEVZw5TxjmtybcBkBv\nZfv+n0WpMFJ92jy1xFf1w3XdMHApUp9+IfB8Ny5HScAJsBPjAcZg7SH2cPYEaw/A9o/WEJAXgBXY\nMQSw9jChSBNrDwXbP/yw9vAj4ATIk4mttUcHkAozjOu6rwOvJ7yBIwOcZviW3CIxHnmVEhPsMbpq\nJqsDqPRszMu6VJ/Db/gYgMP3F+Zs6XEPAzDxY2GMe//t49ih48SbphMlTim4LHBdd+9OO0giMAPL\nFSMcKPJ4CIovFEazZ0AY4bqorKOZUA3NiOnlx3xzjGz/osQn67mcOWPGdSlhINBxe5jXrP6EiQDc\neIfMvg/Pl2OaSX1v1gmLMOWDcwEo+VC+910gM+5ghSRSEYold0UHSnLR1p1EqaTiYDmzqw+T7n1J\nLwna1+zaNwc9Ib+/OQGAeVfLZ/D9/6rGt5ztdpn+0bWwNVl7LLpVEmeXnCIegRolbaDjTjX7pDmM\nGCNqMqOCAien1eXBZmxd3HXi8Ale7KNijhuj0ocfGfYRALc+I8l9754q/Say8Fu9aXL2CAQ9diZ6\nkMTpzZn0F91qWe6xuH6vT72rxgVtL9VnTfZOe4uCPvZS2bKDSgUa5r179rJDARh7m9yjKzrQPzoL\n5ngU/v5e3m8vl0tfDKhHarGKkdfnpRl0HedfM7LDz6GE7RHBoTqa027/N2F6UtIJvU/dx1p6K/yo\niqj8iKDYrXZICy9Fp/YP85oHdxHvdfQe8TJ+vLMwoLF+LKykHov+d/0Eb1/PfCGerPyvpW/kbdDy\nB/JRJTmDnHrUDABuHiD5Bd59qpQ2pvZeBsCRD90NwFWnT5ENZ31BFtm4rjum42fcOlrYYUB/AL67\nS+J9vznIzwTr94n9/zQVgDF3CbsdacYsN/fct35QNd4o2+a/IrkPf1wmZPcvX3oGgP/u85zaQGy/\nsEno4OJ5lQTrDOWJNmAr0FlYWFhYWFhYWHRbpMQMJw1XGMSskWUAHHuEMMGLjhI2LmrG5RiZ+V7G\nol6gmIysX/cCoGaazCKv/5VkLj70j4netpFNm33bbCN1icxAnZsX26aw9Zhdvf9njpWZnJ7Bmoyw\nnunq5ZotWPtiGQD9QxKX3xkyXuYstOE4uY4v3fNnoGU81rKwzASPeepaAEbfL3HMY1Z95ltPW6NV\nTqZCpJ+K5srXor/L57+KRQXgvksk1vWuC0WpQrPSt5cKQz76WMncH/We2+o5WKSOaO8e1B6+L0tO\nkWug+26+Ym2DCcRdQssY2v3miuKD81SJ7K9S9tvYOzY8Vp4oMaDfKpk7k9E0oZd7zLEadrZGJYny\n2j4SY3jfFSJrNOZnelxK6BTUPiX/wlXj5Hc/FjtoNYh43h6NQicviYPF9gewKiI2WhMWb1NFWBSB\nVoYki35lg3wuqRGbrqmSMX7rVmmbWyVtyt4s9imdoWKyp2u1gC1JtS0TMMfTwK9iWijaxtpGZmx5\n0FBQGlPuz2tq4VlLAyJugI3RHkDiWsbQviclk9Ax1xrhImWnthUUUoY5fteeLOP7bbdKHP5+eXJ8\nreyj1XJOX/p9ACr+IJ6fvDf/6+2zPOyTPWuBEvX5+XVyT4x47AIAlh4tnlA9bukxRKsLLb1C+taI\nWUmcYIJo8XyfKLKlRz32IQBTe0sCl2mHCY8IIzxcMcJaZcINxZ6HyT4btUSkVo+4+BGJH//qkvt8\n671cJR6y6Mo1LTWJ29p/Uq2xsLCwsLCwsLCw2IGQWWYYwHX55melACz+t3yWbZTYXi9WOGqwjAab\n6zHEevY2S2Jrvj/vHAC0tuL1F+/s7WLoH2b6jpGJggTbHFH/LHqnK1vG0puZ9d6mukKU+n79OgnL\n6n/PTN8+027HQLCFasRv7/wrAD0Dfibr5g07AfDuZfsDUPa+9CM93zRjkmJZ4K3EwikmT+t9akSU\nAPqQP8p53/aPUwGY/oToy077TOwy5tqPvfaDZYQ7A4F+TeT/PMaoxWJdE5vTm0zpDxf9AIC+P5Vq\neeEKlemsrmF2M89U0SuyzejHzwNgsWKITUakPRQ6fn3d+494HIC7hxwpC1YktBuB6+KGmggUCNt6\nzL5zfT9rNtJksdeGJXZ+/9evAiBvrdwn2WIGcjfLfZK/WVWT26oqRNXEWJZAlbBTTo14ZaJVquJg\nTY1qm153LQD9vM924OmXdoAp7ySYLGHdScISfjg2pmeuWbt4LLw5zl4+/G0A7imSONJotTJ+Gj2X\nITfI+nAx7THDZv94sUZY/Fk1o3zrRVXMeUiVIoyqGFatyhOKBlusq3/T8epNKr5/Ut8FAJxXXNlq\nG+Ihq5fcw17+UJqZYfP9QKtFvHqLaNf2V15J874f/azoRo+6WujZXERdyG3mEfDebzTM55DO8VFq\nUGOvkDyC5z6VegGnF4l3O894bbtmvKgMvZQ9BEIpBvHrtuo+r+zQeKz00z/dIyzsxFxh7DdH5P7X\n3qhDF/wIgOE3zvTvJ14BkSQQNfYx4tElALx5vrTlqAIZc15YKsxw/8avk7qPLDNsYWFhYWFhYWHR\nbZFRZthxHAJ5eQzZQ1iCHpNVxrWaPbnhkF5RfRrlVs3lsR0DkPeQzKBQsrOXn/Wqt8qrt0t9EDci\nx9yhYzoN3dEtZ0nM7etD722+EhB/Jm5m+b79qCh/9GdmGhsaB6r9w/4q2p2H5usZtCy/ZaOUwp1x\njFToCaySuCwvLqlJ1TBP5toqZqcFaayVObJk9hldILqw8/eVWf6Y0KfGfra9lueOikG5W7hpxKuQ\nYCa6iZBBI321QEqDlleINqZWW4nWCtsR6BGLTdcV43q+J3F6HCofkSSpy5i6gPSTSQXCdtyV2/Fq\ngu4ukoZ+z+CnfMu1ckHE6JPzm2ScHHOxUUWxveM0+79dQs7QIzU9Ls1+UDv3Kwd1pZwOz6OkxoI9\nr48f+6kZwzfqJNv+5MKqVtc7Kl/6050T1BjmqdBoe6ROeUYIsilcCK3VeWwGTzFIxd7f8NjZQMwb\nFoO+JqmPcbdfI3H65111n9qj3xOpYSqb9O4pdnNUKXHS5Zg0PJ2ho8Tj9+jNdwAxRljH62oVpjF/\nE+WqUdcb3m3dn5s9g9r1oup1VVu0V/KWr48G4HSlmlCt49KVwsbg7E3q2GU44dSY4RaKKYeLYsq9\n994FxOKUtZdNM8La25R7Q7Fvf2acfUrQY4KyT3it1Bm4doH0paMmPgtAzTcyvvVPcveWGbawsLCw\nsLCwsOi2yCgzHC3Op+7g3Vm9QWYVo5ZJjFtcllbNjoN9JTPZq/KlWTxjHlnwL5mx/36DxJr+uuRr\n77dnjj4WgPxXDTbEYFF3BJh17sdfJnZuzgLHq0plxm7dsUlYpwEPSpa3N8/rLHtFI6z/mbDQbwy5\n39fWjxulre+fJjqzkVUSU+XFeaUhLqkFdJy6ntW3FyvdhRitHQ09HJiYmz77OmZ8ndadVn27tUzk\nvM1yf3gxgyoGOGQweWaFNhOajdPMCoGO8xLhIn8sYvyKc354MfWmd8hgkttkd0xPiD5fUwloO3SY\nmM+lDVPUuDRI61vHYnG1osmJ30hOwcoZQwA4WVVH1dc7qGJoNWtfOUGYtoHvq2PquNE0jK9hN8CG\nUGH7KxqI5ra/jg9OEmyk6h/BDjK6w4olbrY2mJxXKC5025W9g1ICnKPv+A8AY3Pk/jQZ4V1mCns+\nQjPCXl9RY0YqzwHjZhnay6+sUuD4vUjfNorWb7SuDrejN5qR6xIYJ+9QVzwgbKtmhLWCj6mOctBH\nlwIwalY773RpgHmPNM1VEQFKPKx4cWtbtQ/LDFtYWFhYWFhYWHRbZJQZbip0WHNgkPy5ig3Rsws9\ny1MMhI4HWvK4qAVctfs7AHy0RWJF118lsX4oFQlPf65BZurPPitaf7++LMYMb/yJivnLk+zQ/PUy\nw8n6z+eyQjyGeDvSJTYzYSt/LioLrw3WFaliM8pAnNJRZvzYkw9MAqB/SMWPdaI9nGCAYGExZ1wi\n2oWapdYM16X3iq5g6UJDu7AzGOF4MOPXt4N+saPAwUk6TrhNtHfpWmFZcrfK9Z/TJGPOwUpEItqC\njW27QlvQqAaXFLtmIJzXMZvskPkS6YDOYVHPIx1LPmXqq77Vmsezak9awz2DACgJ+PuO7heeB0Fd\n7poJ9f5jp5FCD7sBNoZice+JKjaE8+LcGPHGvA6Mgdk1/m1i1e78fdl8To0q3ADA/KzeSR+zNehc\nEP3M/Pr3woi+3keoepMRvnvzcADKpoiqjX5biCkVpeF5YOzj649F637UyvMBiDaIjbIK5P4tZRSb\n9AAAIABJREFUnCnvUynl8+h+p65x34ckHvf4HpI/Yepn6z60ISLvVSPv9rc5rbHCZlPDfo/diGcl\nJn7XJnk/KHtTqusmO7pZZtjCwsLCwsLCwqLbIrNqErlRgiNrKHlXBSWZsyjFun17i2jaFeZtBeC1\nUw4AYNlJUqPl/578GwAP/ECyLKPLVqoDyKym7CkR65xxUWyWPXe/JwC4tXwXAEbnycznF2+eDkD5\npZJR7jHEeqa0PTB/mslQs1vNlA/9sejw6fi05hWkTD1MsxKdjrse9A+xbSY4pEhRPtXfH8u1fT5Q\nS2SupvWEB/1F4r1dHd/UtA21ortQvwg4/rZ4mdYZCNTUx/L6ntN17JIqWmNOc2csBOD3Z50LQP1A\nuddCPeQebCyWPhsSMpFo67KzHisdVc6akZULO9zOSF6KnEagdWbZU4Boi0mM08fiMUPxVCW6Ekvt\nxSSqNi27UqpuTe4pVbda05bWY1TBy/IciR4oeQ1m/HbUUGI4aVeJsfxCfU9nJbqIG2BrKD/p7aKa\nGTaZ4DSOedm1HdvX2HxhZOcH+6bWACP3w9lHrvFbx92uVpBYa+0h1fjbX44BoMSojdCZdQtG/OLj\nTts3jsT36r6+9GYVF18mse6mNrsXM6yezcd8cR4AvT+eJ/vLRA6W0Q8j30iQ8JCb5bOjI4llhi0s\nLCwsLCwsLLotMsoMZwUj9C2upeBryY7Ub/Ca4dOZnEN3EdY27xiJ/YioWcvQrxYBcMXQswDo9xfZ\nT+9jVUU6NUsLr5TtXtg80Tv2AaWihvDWtQcBMGOeMJ5nTpc4mzfPPVD29bh/FuZpj+oKQV0QJpOx\n8XSpwDK7XGcy+1nf1uBlvSsy4JkXJO566PLMVe4L9XJZdVzLGeWjbx0GwKjwLKMtO44CSCqIGvGn\nuv57Rtg2gzU027KjQesNO4oJKUjTflPpyZGcFG0eh8XpDMdC3H12hRh87WFT7GxWqWTp33/uA0As\n5tbUvgV44aHDgVjcZs538gz7okn2tVdusNVtz+sj6//PCPFQhpeKtno6GLZwNMCWxg4ww7k6fjR9\nmscmcmo6ts/dctWzPWvPdDaHtb+SsXJUtjDCJvs/ZZUwpv0eFV153UvN+NXOgFlJ1YTnhelIX3Hl\nOaGrvb56pmbGZWTTMcIaZv+NvFai/lPKToaSVUagawHo96AOapVbZtjCwsLCwsLCwqLbIqPMsItD\n1HWIrFln/CBv8JGdJVOzYot8H64171QMrGa6xv56GQCXfCwxXHfvebL8/t8vAWiaJDHHi6pWe4eo\nGSAzvbUXiPLA8NOkDbOulHULfilV8XhcPoK7SgzYT14SZYObnjtDtjNrbpsxXpkkNrzKfUrDsliq\nv+w85Uvfaq0xGRoma3zrJqlHP+IJf6xwOmPZ4qF3fh2njP+8RduGvGPaOIPTzhSy/FtFGvpHwGjT\n+GJhS6b/6BAAGnrJDNnJAHHuKgIrb4sc7JDi932/m23tUjCbprR+vSpSrSHgZyFa7rNj5xutk6zt\njvSPcH5yx8xRHSOrTFR53Fw530hPYRFDxfK9qaeMcQ29Aup77DhNvaShTb3VvdhT7tXCYsm+71Mg\nn3lZsjwUFXtVN0q+yJYqYZ76vSzHLHxevD7bkiE2PWwLfyXPI10Bs0bFT2pN4Qe2DPa2HfiYeAqi\nqv3hdesBeG7zvgDsNfC/vmPpWGKt31o1XljoAsUMp4Nhi7oONU3tiwabmrHkSdvSqXlsIrsmMa+V\nWbFxnPaCtMOWtolA0GNRo4cIw/zOXveoH0V9w1St+eRJWW9A2Hj+x/G+tTmGpIpWqtuliq+nCiOu\ndZXNWGFT2enNOkl2GPCieOs9VY0MvCe0gK4FkKI9LDNsYWFhYWFhYWHRbZFRZhgXwpFgi2peXvWX\n74SdHXLfUFmuZqxRpSOrZ6qR9TLrvmv5EQB8e77MasrV5HvFUbJe6b1DvUOfcPFpALw48SEAri0W\nJQrelap1VdeIhnG+ihFe8mOpevebVySWK3c3UbbwquFt3qrOqetkz288YVcAXh/ur9zWljZrlqHr\n+MCbRwIwermKz82glm9BsJG9eizzvq8JyzELlkuN9haajplAF7q+GnlGHNf1Jd/I533fbIvmtAmz\nrV0K5qWNKsYlgbj4rtQrGvq0zQyburIH5QmDMvKDZwDIU+NsgRonmqskdBY8zdtDpG2jJ14MwKhr\nFEOsnw2ZIJoC/ljD4C5jAHhHKQuEXGFvtYdN2/Oex0/wdjG4VhhDU/P+nVXiYUQxw2alQj02r9tH\n9jniZfVDO1rAiSDqBqhtTJ6hzCkQNt8JShvcdITFGox/sDa5nUbVHaeVkdy8DjKvjrDumnFffIbs\nryQojLAZK/x8jeQxDXpSvK3tMqCGslNXh5OTQ9aQ4Tx16ENqiVxzM1bYrD9w7YJTABi4Qang7ACV\nfC0zbGFhYWFhYWFh0W2R4ZhhiETjsxia8Q2+t771FfRsWc2+ls0U5veHPxClCM2LuX1lVla4NMZm\n1t0jMVm73iez/A0nCYva+wnRrt2wXuJt++ysdEMLZSY6+jnJHl80RbZzhwyQHW7cBMRijpYfrdiU\nX06Le35ph5Hte+hVhhJGvPJXtKxG9E69zOx2ukW0ib0ZcAa1fAudJg7MX4nWeFwZkU8qjP6QiZhh\n1ceCfVSlozharEmjMj27sUgDTCpg55EALD63uOWqTVpz17/cE89Q+squ3qe+NQNu69818uVOG/sL\npZVekWDbm6GxT3I8tb7nh2UVtrmeHiPCajSINPOSmHq5cfehGL2o4WGpVfseotpQMHqr73dPjzgD\nRJMXnxuWNn53o4zlI5SygFmF7LVaiasc9tdYhdOojhFV3ksdU7plqRo/9m67DX3G+8e4dMReRqMO\nDU3Z7a9oYEAvUU5yctQ5KZY7rUjR4xYpUULe3yZ7XIktDQ7oD8D9Rzze5upeHYItoh/dXqxwIF/6\nyLqf7AFAuCA2YDiJPrbMx7YWi1Bva7p634AnVJy6zjdwnKRdVk19sll2+mAOUFrl8bzJZlx5+BN/\nBcDOjC/PFCwzbGFhYWFhYWFh0W2R4YA+B7ctHVI9+9CMpzHLMPXj+s+RqdaZZ0mc2Y3sJT83yiyl\nqXcs9i3/FWGAn75FKtfs+XOp/LPsb7JPt07VjleZ1D1WSRsWXaD2EVLxy/MkRiZYLizSXneK+kH1\nbfsD8F38s0sbzNlp5aVy7N/1/4taI3k2YMorFwEwat22y+p2MCvTK2QiQ9WMaesp7OAPPpArekQP\nue4hNX/MbocZCxqV2CoiEpP2xojUm9qg4rd0/Nx166Ta1dwpUkWpsUT6rBPp/GvnBsVuuRuEPRr/\n4HwAbhkwt9W2dimY5vlO2NlHjp8BxFQEoKUnJVVopRRtl2P+T2LwOsIMN/X13x8x1rZtb4ZmgfSn\n3k6zQjqfwLt2aRAG0Xbsrew4vU5yEgb+yR8Dmom8AFMRKHqQ0mc/4D4AQq60Kduw4/V/PQ+AwRtn\nttinGSva73NltFPlQzNspsLPxSOl6uZzuTJAeDkaKYzDrgtNje0/4s22lPcUlnqtqYiQxmdCqmNT\nQ7/U4torThLVpEkFohal1RN0rPDmiLCto17wX88W/dKIla09ahwAc264P6X2NYc5Vly0Uiryrnig\nGSMMHbougaIwRQfF3JXm2KHvV7OK7aAP63z72SYqEmmGZYYtLCwsLCwsLCy6LezLsIWFhYWFhYWF\nRbdFhsMkXAKBNtzLmuZPsPxj4XeSdDEoSxIcdDnn3Ao5rfV7xvx6g96Wz3tuEn/Vx7dLic1J+5wD\nQM5GFVqhhOVL7xAX2O++XQzAf+tEgP3ZPx4MwO9PFlmim++W0tADnm7pMks7jCIb2kWz9zkSSK9d\nGaZbRSPSLPFMu3u/aBL39k73iKxdBgr4xkW1m8OHDYM5rVCu68CgJC/SX5V83KKSbDqxTKgHdYz9\nCyRDQ4uRdxRLQu2L3ycKMxlpaa2E/jBbQhTyMlA6W8Ms0+21RcFsa5eC0TRdcn3Ks1MA+OanMVen\nvl90SIG+x9oqaNMWtFRRblDdoynYKbtXcrKHuu06DKKtMu0QGzfq3Vh/qo5K+xtUs7dG5TxWhCWx\nZnlTPwAW1pUCsGCTfK5cLf2j8Cs55tBXJS7E+VbGMM/lm8lEHG37mzYAsUQ5s/DAwiZxDRcvF3s0\nHd0yK85VyXiBkKxjyt6ZUpYapxSuAOC5cSJtyeeqcFIqY13UIdqYfOLvbkVSxKcie6fkj5kooqkl\nQdf3SS2hue6wGt93MyH0xnVSwCjwvr9YitkvzcSxVYfL9a6MyLNrbmMvb928QGJycnvnSL/TITVm\nqNlnT0pyni79bRaLSQYDcquYOuod73u8/qmxQIWLZn8pxWE8a2S0/nLnwDLDFhYWFhYWFhYW3RYZ\nZYYdB7KD0XblSeLCmH0EKjcD0KCS8sJjywAY/k+Z9a34QVGLXRQ/Kwli5cedB8C9zz4NwFVfSFGO\nomdkVlZ1ipTR/LBK5gtvr5RZcr89pYzzPf/zYwAGvOIvz0g6BMrjQEvd6OSKyouljW8M1Qkfqsxq\nnNldc2an0JFEgRNfmgrA6KXbvhxqXTSXz2tHcFqhJF8Nz5Lzrd1J2KS8b0X2LR2lStuF2vnbNSLB\nF+ErAEKq/nB2nFrHUaWtNVbN7gsdYYTfqdpFrfFla5ulhICRrOdkq76Ygdm6PpZmhs22dGkYCWGa\n5R79oCTS3X3icO+3oqB4n8pzhMk8QBVX6GhiXdBMJE6hbPXgki0JrWcmwywNyTh57tfiHaucJext\n8RJV4KBK1s/ZKuN0dnVs/AhUiT2cWvl069VnkwyAboOMUW5IPE/5LAVgjPr02uTtMHOi/ebzp/Zk\nGUc/GvsgEBtHTcZ8ZLbY7cPbZbztSDKluY0+lk7cWr+3JO6WqKr0KUlWuUBT+200JTjH5wlL/Vb+\nHr7lntRavOTGBMYb75mf5NBktrGxnUIz8eAEAwQLi/nD+Fd9yyOGm+iNN4T1L0PkSuO9s5jet51/\nI3J75/32eLVCG+Oh2mekUhIWN5+7HwCzb/YXzdKeivfq5VqWvlGh2qwOkUKyaXEgxBEFq9BlqM3+\nqWUVg4o3fbjyUDn25s0dPmZXhWWGLSwsLCwsLCwsui0yywzjkhWIEuwnMaDhtUpHqINsZLRWYrgi\nimWpL5XZdcFLIpA9fF4sTlPvWc/wRp4p7OPFTwgrsuSIRwH428sixr0lIjGi026Uss0Dp31iHF1Y\nSq+UZwfidRKGjhVWjHCwRJjSXc/+yrdaTBrJHytssg8AMxpk3VHT6v2HSiH+KFVsrivghXl7cctR\ncm00g7X6YGnTqH+oq5gmeSsfjL4XUfHJ7+0vxVred0pb307ZS7Miod1Ecu+xp+8GoGeWtHXavAlq\ngxfS1WIPUYNldENy7TJyDY1rYbalS8MYbnT52fBKiZn89yGjvd8iGzYCsOYakUCbf5Wwg1WqfGvv\nYGox5algQp+Vbf5uCunv+akUEij9hSzPXyhx8cOdZbJBnHG4+dJkeUrPc+bFwKrS15rVymCMsD6m\nbtOEX87x/W7KSzUa9Yg1W9YYTfz+0uVt4zHD+tps2lO+l8Qam/AxWsB1CDQkPlZqz8Heua0X2fAK\nMHXEa2h4Ppym1FyozUJxk0K4Zz6bjtmF43u8q5aI3TX7qjHg0471R/3cSAjaG6LsGTrRz7bWKU9u\nT0fadvl88UaXfisynx32sDdvAg5FbeQMeIV21OX7ZO0waQPbvgyz9uR5Y0mKzzvLDFtYWFhYWFhY\nWHRbZJQZjkQDbK3PI3+UxEUFPGa4YxmzmsnxvmuWQc9CmwtBmwU9FMqeku+7F58BQJ8HVXnQGVLc\nuUeVKsNozEI0Msq+KfusP34MAK+XSWyRmfVsorUSi2e/L0U2xsyU4DRTFWBbIHurw6B/ZsFR/uWX\nHvNvAN74jbC0Uc0qZCC+WSsMxIXRhpVHCvs+xCh1O+h1udVWpLd5Fp0BdU2j1TUtfgomJ9zQqXCC\nQYI9ezMyf7FvualwYTKd+c8IrRZZKHkCTq7yoJmxhyZ72xoSZC63hafJRItiRZO/B8D0QTKOmio8\nZox1yx2m3iazzO2h44VxW6O+pxIP6kQh2Nh+I02lFO1BXHi1jLdZVYMAiOSrkuNZ/k/UZyAvdo2z\nc5UCQo58FuXJjdMjW54vZcWrfW1oT8XARFOvjjHm4UKXdQdFvWdhjfLs6HN+p16WF30q3iF9RgkX\nlUgg9t/0voa/L8XCZu/9kFpDqbw4/ud5j+d7JtaGJFDvwrwmmKiGgPZyIGqq/Ax6RvJ3TGhvfJrf\nVdp9GXYcZyjwBDAQCXt/yHXdvziO0wf4O1AGLANOc113x4uqNtDg1rIg8glNrtxEgxnBMKeckNvE\nfGYBjHMc5y26jT3q+JJPaaQBB6fb2yO0dTPrXnyWXR9fQSDgcNHZxVx+US/CjXV8O+sp6Gb2WL0m\nws+v2MzG9S6BgENg0heMOnX3bts/Vq4Ocd7llaytDBEIOEw+uyeXX9SLpkg9X6x+FbqZPaDtMQQo\ndxznW7rRM6Z2XQ2fu++LPSIOg51RDGNUt7XHytUhzr18HRWVEYJqTKUXRCRMstvZQ98vTSGZYDS/\nX+qoprvZI11IhBkOA1e7rjvHcZwi4HM1WJ8HvOO67v85jvML4BfAdW3tKNoUpHZlERv2kDf7/jNk\nVtHRjFmnUNi3qJqiZ1f7yzW3xkaYs4khv10k/9wobGv2m58CLbOcve3cKOXsRrHTm7AbYjbv0Mcd\nwFqW0Yf+bKJyAfAOCdgjYSi7aPb2mCs+8P2s49FMmFnRsxpi9h39qMFwd7CcooNDObunxR7BqnqK\n31zInZvLALisl8RlT+29DIBHf34MENOAzgibHWemH+uzqnxzSR8AfnPSi2w+0OHyiUOpromy05GV\nrBk/mg3Tp9E/0JOtkP7+0YURDMLvbijm0PG9qK6JMviwBfTbewjL+Lrz7pdU0Ymehqwsh1tv7Muu\nuwWproly4KQKjji4gKUbZ9Gnx3A21i1L3B452bhDB7Bz7get/qxZHpN1C4SN81P3/rZib9saQ4Bq\n13XLE33GtH4AzV4pBrOHZM5ffIUoCmg7aUbdZMd+sly05T9aJDHkTkA9X5KIj3/uIFGqmJjrZ51N\nFv/C/u+zgTBrSg6jZ05/GtasZLb7Dn2c/snbw4VAAsywRnPPIcCSkx5MeNtU0ZYyR1aWw59u6Mue\nu+dSVwv7HL2Spskrqf1wDiTZP3LzQuw0JsZKm/ZfGZJ8nPDqNb7lCY8JHRg7qq4S72M8tvqRrcLQ\n937rOwDq9P0S6CP3i/u2d78EySbshhK2R0VTMbetnsTzI0VrOKoyA3RPMPWXqTa8JJ2RvxMPhgb5\n6uv2B2LeuoF3plbrod0zcV13reu6c9T/1cBCYDBwAvC4Wu1x4EcptWQ7Qa6TT7EjwvJZTjYFFNFI\nPetZQymeFJO1Rze1R5/+2YwaJ8lURYUBBowsYGtlI5WhFQzOKderdRt7DBwQZI/dZNJSVBigqKw3\nDRtqu23/KB2QxYTd5SFXVBhg5/IcVleEqaz5lsE9d9OrdRt7QNtjCLBRrdZtbFLSP4ueOZLI7bfH\nauiG9igdkMWeu4sfX98zkU1V1M1ZCN3QHm3dL9l4oRXdxh7pQlIxw47jlAF7Ap8AA1zXXQvywuw4\nTv92D1YH/T4NUHmIMHn9vYpzCQacGLGzkYHSIbIdFdu1SjI5NccZGLezt2mgWirCuDkys6m/T9aa\nsVgYvdFvfiaHUPFzXuZsG2x1vVtLNVvoSR+aaCTXyQc3cXu0CyNTc91k0T78bb/7fKvFm1Wb2owX\nzTvH+3/Qh1JZJ53saqr2cCNRIlVVPH6fMMBTfyXnqWP5HrpUFBp+N13iuyNfLUr7ObRslCk5YMSe\nq3jtRXcOAeD4HnVqxQDLVoZY9GkYd6dTaYreSFaV0mBNV//YzrBsZYitizbSe5cBHesfuITcSAsG\nq8NojzRrLV4zTWIZAdU/5s5vZN8JeTSFN5IbFMYyUXtEswPUDy6iPFtnsBeqffsbaY4P2bVdt1qU\nOYaglNtTuWfMGM1lV4t+7uSeMwBoVM8TzRJqduxLVZ1z4yli1/LVftWJZHDxa2cDMGfvvwMxRQoz\nPvmAPGlD3R5DZfma76hmC72C/WgKJ2cPJwrB1oUhEoLORekIzFjoeIgbjx0H+p7pd34JG+6thiT7\nR14wzJjiyri/N0RN5jN9eSneu4VShaq4QpjNeRP8dQJMtvr3M34IwJj16h2l2fPOvF/y1RiQqD3q\nq/OY9+4Y8Jhhf36BCTeQeR158/le/WPRY15whdhtxL8vBCSOt7X1E0XCHLfjOIXAi8BU13Wrkthu\nsuM4nzmO81m4oTapxnVlhN0wX/AxOzGerCRu6Ob2CNGFsnFShLWHHzW1UU69oILSQ35EMDev/Q0U\ndnR7jLv8ALJ7tF3+tzma22P9xszL93QWamqjnHFhJXf8roTiosRdjb7+0bTjjKfQsTFkR71fACLh\nxpTsEa7bsfqHHkPu+F0JwYLEy9s3t0n95hRmB10M6XjmRmp3rD6SChJihh3HyUZehJ92XfcltXid\n4zilagZSCrQ63XJd9yHgIYCe2f3dvtMXk3+OqnZSLKoSkaoqfSC9UevtMDIXt46RWdDKsOzHXb7K\nt/6K4/p4///u/NeBWAWxX39+AgCjz1G1x3VscAJailE3yhd8zECG0d8ZDEAOuTS6qipTgvYodvrE\nOVFth6jv+/GT3/et1ppKRHNoRkjHIJX+qeUN44ZTL5mXVnsEgvS/XxQ8pv5UmPA7S2VGvJ96pxz7\npGTPLzxGJr6RdbLrFqx+GmM/vUx0HV+pZp3L/iAZ6YsPu99bNxRy2eOcWsqO2om+dzfBW5/xHXk0\nOk3gpqF/bGcIhVzOuGAtZ55UyIeHiA5zR/pH+W757jv1BUwqSM9LjxM22CtVYQylVtJcrUZLzYZ6\n+DcJdKByXCjkcsaFlZxxUiGH/yDI1mg9OcECGhuE4U3UHj36DXVrS7MYppRLIu142DZHxGsRbOx6\nk4p4Y0iYUDbEt0mb94sRK5xVKtzRg+cJo2TG7WpWTLOVJ750CQCjV4vqho41dhPQyHWUB1KvW7NA\nPYtkSGuh32pW+qoYH2DV359msLaHEyCHvKTskT9wqJuK+kk8daJtgVDI5ccXruPMkwo56dhC/vbt\nFr7rk09tbWOb9gC/TfqM7eduaCxsbTUA8gKGUlEHla70+4RXDZQYI+weMB6Ax6f+Wf3if7E37V42\nzdCRj0Ti3i+6Dyf8TpZf6o56bhOzzpLz2y+v7Zj27F5Gh+pMGQkjX0vfv+fd9JpvtdI3jdfYDrap\nXUrCcRwHeARY6LruHc1+eg04V/1/LvCque2OCNd1+YrP6EERw50x3vJ+DGIty/VXa49ubI8Lr6qk\nz4gi9j6nmT2cQax1l+mv3coeV1yzhbHlOVx5cW9veXfvHzuXZzN1SqxyQP+CUayp9kp1dxt7QNtj\nCNBXfe02NnFdl4p//p3ckv5+ewSGQDe1x+Sr17NzebZvDBlwwAjopvb4yp3d6v0SwgsN6Db2SBcS\nYYYPAM4B5juOM1ctux74P+B5x3EuQORTT21vR244TGT9elaskSomvc6Uql79HtD1v9VsOl6sh56l\nqVnbuv1kdv1wxSEARBs2qf3IaQ35Yyy78OHpxwEQaJB9j1o4Fx8SVLLYykYqWEEhPZnlvgXAaMYx\nnJ08qShgKwnYIx50DXg9k1xzrcQW6Vjh2KytdVbKZIz3/+x8AEo/ntdy5RTZ006xh7oWX1+4KwCz\nXhSmeC81gb69VGL3rp++OwBzz9wJgIiqpKXhVb1S8DQ7W5s5qr6lvQ/6u2bOdbyh3uc39+8JwNJj\nhRHW1+RvM2p4alo1hfkFLHrlI2iqF3sExjI/OhPS0D+2J3zyaRPPv1jPbmMjTDhiBUsbn2eXyft2\nqH+squ/N1fNOYdL3ngZa6sImi0gvv3qC2+BnPaINMZdqsF8/APp/36+Pmp2kPuqHn9Tz1LRqdhub\nwz5HrqIynMvRU8sZmbcHczdPhyTsEcmGmiGxMaDRlfPRzJIeB3QLvw2LnbKVq1jfBalo2aYDbY0h\ny1lUrKSiEnrGNIcZK7zw15KwebDyMJmxwlp14716+b7zn5YCENa603UqHyCRMdOoitV3vn8b05vX\nnCmeMbuBqvmfkVfWn1nuFwCMjuxBWWBnlkcXJmwP0RmOfTeVArYXzJjdwNPTahg3NocJR4hSe9mU\neRw5eRgPPvffpPpHbWMOs5cPhxGt/35OkdQ+eORHJwJQ8LKqM2BqcZvPEP280KpM+vo3xi5A/QkT\nAbju9icAGK/2qWOzzT5x3TphkPPemy+HVsu3uuupcJe3er+sYgnJ2MNtaCTy5TecM+1SAL49W55n\n9ar6nTm2nrCT9Mf5aah+FxdxvG0r75eJ0OSeovShq+j2fl/dp2q9jipjtfsy7LruR8RPGzm8Q0fd\njtHLKeEITmn1t704hLfdaQtc1+02drH28GP8PnnMXjaMX512AQDupzKQ4QTZK3gYb0f+3q3ssd/E\nXDasGuSVKz5p8ZEA1Drru2X/OGDfPJrWjPRCmG7dNAqAikAeE/v+iOlr7+1W9oC2xxBcFrmuu3dm\nW7RtceC++ezxj18B0O94Kf7kBLxHdbe0R9MaCa/S983Va0v1z93OHr2cEo5wTm11UlbgFlHlbipv\nZTOLdpDRCnQaIx+Xi1j8W4n9rH0gse1isy7Z/vSDhFF++eUDARiGYoJbUVdw/ysuSD1nSEdd77RD\nx8io2WSwRDxAE0/+wreayQC1h5J7CuIea1vUFG8T0Ujs2qhrdu3VPwfgn3ffCcRqtd88QOzy5j+k\natOVj0hVvWF3CQMeTSY5QLFE8cKNtp4lGayHXCPxg9MHPOz7XV+Ty6+7HIDCT1V1r65RyMSeAAAO\nY0lEQVTYz7ZTZFc4DLklQOWLcl37BqQfJMoQmxWdXj7iHgBO/+2VAJS9IjG7gah0gtqyIm/dLeeL\nFugXuz4LtNTwThQ1rtzbug8/+G8ptziq4uOk9gPg5kVpKq/3vreXwT+nvgyAQIWoUXldPaPlozof\n5j3n7D0OgBnH3Q5AyBXbayZYx+tmq/4z5dkpAJRVfNzq/hKByU71/nw9AJUR6bv9lXKI9iiZrOCt\nu04D4PZe8myLbNlK0nAh2LhDpB20UETZqaCiQ/vJ3hqg/2t5NB4iY4a2uzmGnPWHfwLw4ga5PwNK\nfSluXpOxPLiLhC98fV0sPnnB4TLe6DFjzPsSZXrBOHlvua6v37P50puSjzKyQfXDTlRNGn2DnN9P\nDhJN7SeGi3b51qiML7FnruTvTDp4MgBZ/0lfBdsWnlx1v317974ALJno172+YYkox2VVqLquKSp/\nZFAx2cLCwsLCwsLCwqJrIbPMsCNv/8F3JeZz6VTRAW76pbD6OsY3kCdBXVHFkJoxtNEDJZbm5F5S\ny3vew1I73YsZaU0hIeCfeXdFps5Uy1h7msTCvj5MYoV1bFE8Nsr8fY/Zosc78D8qProDme/bAmZ8\nro7bOjYwFYCbb5UZoo79O6pArveXl4md7j1HNDpv++AHAAx8X+Z8PRcLKxOskAqVvhjREklmqh4r\nWd/r9pZtTpgkDO+tA1t3X8xulGNP/aUwwkXPW0a401BbD7Pn88NfXwPAS/97KwBDlJqCZnc0NPOn\nWSWTfdMxe19fpHQ+LxQ2z1QVaA2Jah1rBlm3radis+/YJG7fMX8VxrAj/pn8nBB7DI8p6JiZ36Zm\n6Be1cl+EK9bJ4u1kPEgKrZxTr79IjGFpi34i9tLX+fNGGT9HP7gSiMUKdzQGsTmiS2WfL1TLM++S\nXvJdx/Hq/qT7y+HSTbjhSFm/cNpsWZAE6eVEIat+x2CGQ4aaw6Qeizq0n8CmWoqem8Wuxwmzufiw\nxwBoVA9drbx0cS/JDTjhadG2P+Nr0YmumCXhGQWqQJ0blD5SM1TsPHIfua4PlT8KxMYmgFVh6V/7\n3SHPsfJXZCfj3lzpa6P2HoyaViPHUMvT0Q9bRSDo5UdsPK0EgOtfk3wc7X3VSjQ63I3rZdziP6pt\n6p1LM8TJeJtcowKmfv9b8qTkDC05RJ73G5RdSpRXZdUsUdEoQ5jhdnPO2oFlhi0sLCwsLCwsLLot\nMssMuypzWbG0g66U2ciZ/5aqPA9+dTIA+a/O9m/W6M/yPvBe+f3U1y8DoHytyvhsi43rarGxGs2Y\nDN3uYF9hJ4+d/KFv1XhslI4704zw2rDMKIueEv1lfe7erI1OqtaWZpgMcQ+lKnHzkrMAePx+keZ6\nZNhHvu0063LJ8eI54Hj50DPLTWrS2uDG7NkrIMcalhVfg7I5pq6VnI2FPxsLQNHszmOEswI7Tkxn\nyucSCNL7cYmhO2+FsPGjbpGY8QeHyHJ9P5hxhia7FGNOBTEmONhifb1ulpGLr5m9ePvSywsDwnbc\nubkMgDd+eoCs+M1877yk8SSM0pyt/Grov0CVYDXVZRpUHLtuy7IarbsujFSqTEqXg+PgZGV757P0\njxJz+cYIyZA340JjDLHY/vJvTgegcOUSWZzGvArdpjlVqgS5xwz72ftYFTK5lnVnS6xw4QvJM7yO\nC1nNYob1+Zp91fQodBTx1I2Sgb5nTc1s89n3Vu1o9V/HYofLJwuzPOpBUVr67vvCEJveIO1NeG/c\nK7JgXOv7M8ectWr432Xm2d46w38v65TOFQ94zQ9FXeLYAn8hkIuXSiysl3yt0VnvMNGI19fDK8XT\nNHeSaPru9oCc8Px9nwFi/fOdXUTrd9cX5Vk8fPJaACIbRdErmfhdrdSx4ZwJAPz6uicBKM9+F4Bx\ns6TC3OyJj/m2G/yef9xKlTm3zLCFhYWFhYWFhUW3RebVJJrPQpYsA+Dhq04C4Pq/PA7A5fv+FIBR\nz0hsZ9MAiRHpfZMwgQtrZIZVfplkNnoqDJ0VU9OZcN0WcW5rzpI4sdf7S0BOe5XmzOpF924SRiR3\nk0xP3e/tIb9vqfO2cTZL1b9otbDISWloZhgmQ6xVJlZ9T+w2/jJRmzjjItFcNLNyNXSsUUkHhDbv\n3iyMzsOPHgvAoLtU3wvN97ctjYywW1xA0wH7cNswXalIWIrsVtRSuirMtt42THTgz50kCg78e1py\nO2w2fujcgxWHyXX93gkXA9B4uowb940TNmO/vKBqi3nh2+4I/vXjVHls5/enqiRO95aXRbe0/G7R\nxGStltxT934HWJ9N4R48uWl/9lIVGk0dWVM947uZ0ofL0AGPO47HAQDXxQ01sfECGf8WnSuMsB4/\n21MbqZw7AIBCR66Rp1OcBkZOe+UmFC/3LTc9DWY+yD/2/CsAF0z8mSz4JIn7xXUJhGLjuT7/ROPd\ntyXMWOpHtgpTeceT8q4w/D5doOajFtsmtP96YWNHny0qCkd9T5QdFp8tHpz99xRJu+NL5Pc8R1j1\nHEf6wuqQaN7OrhbB4g+WC1Md/EwUaIa9JjG1QxcuiB3TyFsqWCXP3pEvi3qJG5RzHiZCFuSh4nIz\nofyk963Go8h6UZwZdKIUsTtskig11VwqnooHdn0KgC+V5vuiOeJ1PX2evLs1fCpKWPnr/JUWG3vH\n3nXqdhaP/2E7i613zv4UgGtfOgeAMX8WD83gCrnWB74my+8cJ5EEOe/LGOr18BTHs+3nqWphYWFh\nYWFhYWGRZmwTnWEvhlWxabn/khnB7VVnAnDEHTIbu/qstwHYGJWYknNmSyGDET+RmUSLmUAXZDXb\nQtPIfJbevAcX7yaxwRtCwvxd1OdPag3NBLY9kzcZj9/3lxlT5EnR261UmaArI7Ea6NVRmQHfvuJo\nAIKnyG8difnJFDzW1ZgpD7hLYrDee0zYg38c/X0AVh8r65+wu9hhv8LvADgwX+L1mlt1ZoMoksys\nlhn+y1+KYsnAfwlT0+sNiUst3SLH8qyivRKdoBoRKnFZc36jF8esY/62B2ZHw9Tx1Oey5nyVB/Dv\nDuxUsxjK9prlKX5G4rYRQpj/HS6xd5v3k6zjdUIY0me09PF9BkgW8lG9hL0Zmb1Bflfx4wOC+d4h\n61RFplXqMldE5Dw+qhHFl+lrJHZ83WLJxh6ssqyLPhAvxYiNEs8c1n03DWxPw7I8vvlpOaffJ7kB\nfx4qcXx5io2/f7NUSfz7Y1LDo+zPM33b72hKJ9FeBdQfOpHHf3MHAJUReT7o6m7ZhgeuTi0foC5F\nwdrOG/OCA6SC4fg86aMbIqLfqp9gJivVoNrQRzHFS69Ua5yexDEbo/RYXsNhX54AQCSq1DOy5Lpn\nB6Tv5QUV6xmU7zmq/+er5bnqu/4sCOrqZPI9LyDrFQRiuT16mWZUW3x3/NsE1YjaT+17TqOM5f/z\ngrCBo//0FQBD1PgbSVUJRb07eNq2qjpruZL7VpwsjxWIooFTVOhbP7JBmFO3Ua7jMPzxvWY9Azmk\nv19F58o5lV/STlszme/k9X3VM9U4lTNd3tH6TJfFvxn9YwDWHyTXacPBct32GCkxxzlHy1i6sUG8\ndptqRYWidk2xd6jiufLOsfJheeY6M0T1aiRqrDSucenkLQD8caC8J7qhr+K0vWOwzLCFhYWFhYWF\nhUW3heNmkPlzHGc9UAtsyNhBOxcltDyX4a7r9ktkY2sPP6w9/LD28MPaww9rDz+sPfxQ9lgeZz/b\nI1KyB+xwfcTaw4/U7pdMvgwDOI7z2Y5SSzwd52Ltkf59dBVYe/hh7eGHtYcf1h5+pOtcdhSbWHv4\nYe3hR6rnYcMkLCwsLCwsLCwsui3sy7CFhYWFhYWFhUW3xbZ4GX5oGxyzs5COc7H2SP8+ugqsPfyw\n9vDD2sMPaw8/0nUuO4pNrD38sPbwI6XzyHjMsIWFhYWFhYWFhUVXgQ2TsLCwsLCwsLCw6LawL8MW\nFhYWFhYWFhbdFhl7GXYcZ5LjON84jrPYcZxfZOq46YDjOEMdx3nXcZyFjuN86TjOFWr5TY7jrHYc\nZ676OyaJfVp7tNzvdmkTaw8/rD38sPbww9rDD2uPlrDPXD+sPfzolHvGdd1O/0Mq334HjARygHnA\nLpk4dpraXwpMUP8XAYuAXYCbgGusPVKzx/ZuE2sPaw9rD2sPa4+uaxNrD2uP9v4yxQxPBBa7rrvE\ndd0m4DnghAwdO2W4rrvWdd056v9qYCEwOIVdWnu0xHZrE2sPP6w9/LD28MPaww9rj5awz1w/rD38\n6Ix7JlMvw4OBlc2+ryL1m32bwHGcMmBP4BO16FLHcb5wHOdRx3F6J7gba4+W2CFsYu3hh7WHH9Ye\nflh7+GHt0RL2meuHtYcf6bpnMvUy7LSybLvTdHMcpxB4EZjqum4VcD8wChgPrAVuT3RXrSzrzvaA\nHcAm1h5+WHv4Ye3hh7WHH9YeLWGfuX5Ye/iRznsmUy/Dq4Chzb4PAdZk6NhpgeM42YjRn3Zd9yUA\n13XXua4bcV03CjyMuB4SgbVHS2zXNrH28MPaww9rDz+sPfyw9mgJ+8z1w9rDj3TfM5l6Gf4UKHcc\nZ4TjODnA6cBrGTp2ynAcxwEeARa6rntHs+WlzVY7EViQ4C6tPVpiu7WJtYcf1h5+WHv4Ye3hh7VH\nS9hnrh/WHn50xj2Tlb7mxYfrumHHcS4F3kCyGB91XffLTBw7TTgAOAeY7zjOXLXseuAMx3HGI+6F\nZcCURHZm7dES27lNrD38sPbww9rDD2sPP6w9WsI+c/2w9vAj7feMLcdsYWFhYWFhYWHRbWEr0FlY\nWFhYWFhYWHRb2JdhCwsLCwsLCwuLbgv7MmxhYWFhYWFhYdFtYV+GLSwsLCwsLCwsui3sy7CFhYWF\nhYWFhUW3hX0ZtrCwsLCwsLCw6LawL8MWFhYWFhYWFhbdFv8PSCysRqLbFWMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f625a9f4278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "plt.figure(figsize=[12, 4])\n",
    "for i in range(20):\n",
    "    plt.subplot(2, 10, i+1)\n",
    "    plt.imshow(X_train[i].reshape([28, 28]))\n",
    "    plt.title(str(y_train[i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<YOUR CODE: a whole lot of it>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SPOILERS!\n",
    "\n",
    "Recommended pipeline:\n",
    "\n",
    "* Adapt logistic regression from previous assignment to classify one letter against others (e.g. A vs the rest)\n",
    "* Generalize it to multiclass logistic regression.\n",
    "  - Either try to remember lecture 0 or google it.\n",
    "  - Instead of weight vector you'll have to use matrix (feature_id x class_id)\n",
    "  - Softmax (exp over sum of exps) can be implemented manually or as `nn.Softmax` (layer) or `F.softmax` (function)\n",
    "  - Probably better to use STOCHASTIC gradient descent (minibatch) for greater speed\n",
    "    - You can also try momentum/rmsprop/adawhatever\n",
    "    - in which case the dataset should probably be shuffled (or use random subsamples on each iteration)\n",
    "* Add a hidden layer. Now your logistic regression uses hidden neurons instead of inputs.\n",
    "  - Hidden layer uses the same math as output layer (ex-logistic regression), but uses some nonlinearity (e.g. sigmoid) instead of softmax\n",
    "  - You need to train both layers, not just the output layer :)\n",
    "  - 50 hidden neurons and a sigmoid nonlinearity will do for a start. Many ways to improve.\n",
    "  - In ideal case this totals to 2 `torch.matmul`'s, 1 softmax and 1 ReLU/sigmoid\n",
    "  - __Make sure this neural network works better than logistic regression!__\n",
    "\n",
    "* Now's the time to try improving the network. Consider layers (size, neuron count), nonlinearities, optimization methods, initialization — whatever you want, but please avoid convolutions for now.\n",
    "\n",
    "* If anything seems wrong, try going through one step of training and printing everything you compute.\n",
    "* If you see NaNs midway through optimization, you can estimate $\\log P(y \\mid x)$ as `F.log_softmax(layer_before_softmax)`."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}