02_imagewoof.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#default_exp imagewoof"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reproducing the [notebook](https://github.com/muellerzr/Conference_Posters/blob/master/Talks/The%20fastai%20API%20and%20You/01_Image_Classification.ipynb) created by [muellerzr](https://github.com/muellerzr) and also collect all the different techniques into one notebook\n",
    "\n",
    "**Techniques**\n",
    "- Fit OneCycle with resnet34 \n",
    "- LearningRate(LR) Finder, Discriminative LR and Mixed Precision\n",
    "- MixUp during training by adding it as a callback\n",
    "  `learn.fine_tune(1, cbs=[MixUp()]`\n",
    "- Apply labelsmoothing as regularization\n",
    "- Mixup using callbacks\n",
    "- Test Time Augmentation\n",
    "- ranger as an optimizer, a mix of LookAhead and RAdam instead of the default AdamW\n",
    "\n",
    "\n",
    "**Results**\n",
    "\n",
    "Experiment | Accuracy \n",
    "-----------| ------\n",
    "resnet34 with CrossEntropy |  0.944515\n",
    "resnet50 with LR + Discriminative, MP | 0.93889\n",
    "resnet34 with LabelSmooting | \n",
    "resnet34 with MixUp | \n",
    "resnet34 with CrossEntropy + TTA |\n",
    "resnet34 with Ranger + Fit Flat Cosine + Discrim LR | 0.930\n",
    "resnet34 with CrossEntropy + NativeMP | 0.93991\n",
    "\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Performing image classification on [ImageWoof](https://github.com/fastai/imagenette) dataset. This is a subset of ImageNet containing 10 different dog breeds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "- Install & Import the library\n",
    "- Applying DataBlock \n",
    "- Create a baseline using Resnet34 model\n",
    "- Showing examples using techniques such as MixUp and Test Time-Augmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install and Import the library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With pytorch, different releases are published are based on the computation backends such as (CPU, CUDA). One has to install appropriate pytorch version based on CUDA version. light-the-torch package simplifies the problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install light-the-torch\n",
    "!ltt install torch torchvision\n",
    "!pip install fastai==2.1.4 fastcore==1.3.2 nbdev --upgrade\n",
    "\n",
    "#Ensure that you restart the runtime if it still shows the old version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Enable this variable if you want to run fine_tune\n",
    "FINE_TUNE=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use vision module in the fastai library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.1.4'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import fastai\n",
    "fastai.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "from fastai.vision.all import *\n",
    "from nbdev.showdoc import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying the DataBlock"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Download the data\n",
    "path = untar_data(URLs.IMAGEWOOF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at how this dataset is structured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#2) [Path('/root/.fastai/data/imagewoof2/val'),Path('/root/.fastai/data/imagewoof2/train')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two directories containing training and validation files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training directory contains a list of directories containinig images corresponding to a particular breed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#2) [Path('/root/.fastai/data/imagewoof2/train/n02087394'),Path('/root/.fastai/data/imagewoof2/train/n02088364')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(path/'train').ls()[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The directory names contain the labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the datablock to prepare for modeling\n",
    "dblk = DataBlock(\n",
    "    blocks = (ImageBlock, CategoryBlock),\n",
    "    get_items = get_image_files,\n",
    "    splitter = GrandparentSplitter(train_name='train', valid_name='val'),\n",
    "    get_y = parent_label,\n",
    "    item_tfms = [Resize(size=448, method='crop')], #data augmentation see more here https://docs.fast.ai/vision.augment.html\n",
    "    batch_tfms = [Normalize.from_stats(*imagenet_stats)]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting-up type transforms pipelines\n",
      "Collecting items from /root/.fastai/data/imagewoof2/train/n02087394\n",
      "Found 942 items\n",
      "2 datasets of sizes 942,0\n",
      "Setting up Pipeline: PILBase.create\n",
      "Setting up Pipeline: parent_label -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}\n",
      "\n",
      "Building one sample\n",
      "  Pipeline: PILBase.create\n",
      "    starting from\n",
      "      /root/.fastai/data/imagewoof2/train/n02087394/n02087394_46.JPEG\n",
      "    applying PILBase.create gives\n",
      "      PILImage mode=RGB size=500x451\n",
      "  Pipeline: parent_label -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}\n",
      "    starting from\n",
      "      /root/.fastai/data/imagewoof2/train/n02087394/n02087394_46.JPEG\n",
      "    applying parent_label gives\n",
      "      n02087394\n",
      "    applying Categorize -- {'vocab': None, 'sort': True, 'add_na': False} gives\n",
      "      TensorCategory(0)\n",
      "\n",
      "Final sample: (PILImage mode=RGB size=500x451, TensorCategory(0))\n",
      "\n",
      "\n",
      "Setting up after_item: Pipeline: Resize -- {'size': (448, 448), 'method': 'crop', 'pad_mode': 'reflection', 'resamples': (2, 0), 'p': 1.0} -> ToTensor\n",
      "Setting up before_batch: Pipeline: \n",
      "Setting up after_batch: Pipeline: IntToFloatTensor -- {'div': 255.0, 'div_mask': 1} -> Normalize -- {'mean': tensor([[[[0.4850]],\n",
      "\n",
      "         [[0.4560]],\n",
      "\n",
      "         [[0.4060]]]], device='cuda:0'), 'std': tensor([[[[0.2290]],\n",
      "\n",
      "         [[0.2240]],\n",
      "\n",
      "         [[0.2250]]]], device='cuda:0'), 'axes': (0, 2, 3)}\n",
      "\n",
      "Building one batch\n",
      "Applying item_tfms to the first sample:\n",
      "  Pipeline: Resize -- {'size': (448, 448), 'method': 'crop', 'pad_mode': 'reflection', 'resamples': (2, 0), 'p': 1.0} -> ToTensor\n",
      "    starting from\n",
      "      (PILImage mode=RGB size=500x451, TensorCategory(0))\n",
      "    applying Resize -- {'size': (448, 448), 'method': 'crop', 'pad_mode': 'reflection', 'resamples': (2, 0), 'p': 1.0} gives\n",
      "      (PILImage mode=RGB size=448x448, TensorCategory(0))\n",
      "    applying ToTensor gives\n",
      "      (TensorImage of size 3x448x448, TensorCategory(0))\n",
      "\n",
      "Adding the next 3 samples\n",
      "\n",
      "No before_batch transform to apply\n",
      "\n",
      "Collating items in a batch\n",
      "\n",
      "Applying batch_tfms to the batch built\n",
      "  Pipeline: IntToFloatTensor -- {'div': 255.0, 'div_mask': 1} -> Normalize -- {'mean': tensor([[[[0.4850]],\n",
      "\n",
      "         [[0.4560]],\n",
      "\n",
      "         [[0.4060]]]], device='cuda:0'), 'std': tensor([[[[0.2290]],\n",
      "\n",
      "         [[0.2240]],\n",
      "\n",
      "         [[0.2250]]]], device='cuda:0'), 'axes': (0, 2, 3)}\n",
      "    starting from\n",
      "      (TensorImage of size 4x3x448x448, TensorCategory([0, 0, 0, 0], device='cuda:0'))\n",
      "    applying IntToFloatTensor -- {'div': 255.0, 'div_mask': 1} gives\n",
      "      (TensorImage of size 4x3x448x448, TensorCategory([0, 0, 0, 0], device='cuda:0'))\n",
      "    applying Normalize -- {'mean': tensor([[[[0.4850]],\n",
      "\n",
      "         [[0.4560]],\n",
      "\n",
      "         [[0.4060]]]], device='cuda:0'), 'std': tensor([[[[0.2290]],\n",
      "\n",
      "         [[0.2240]],\n",
      "\n",
      "         [[0.2250]]]], device='cuda:0'), 'axes': (0, 2, 3)} gives\n",
      "      (TensorImage of size 4x3x448x448, TensorCategory([0, 0, 0, 0], device='cuda:0'))\n"
     ]
    }
   ],
   "source": [
    "#Let's pass a single directory \n",
    "dblk.summary((path/'train').ls()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above, we can observe it sets a pipeline for X and y. And to perform a transformation on y it first applies `parent_label`(get the name of directory containing this file) followed by `Categorize` and converting it to TensorCategory.\n",
    "\n",
    "\n",
    "```\n",
    " Pipeline: parent_label -> Categorize -- \n",
    "{'vocab': None, 'sort': True, 'add_na': False}\n",
    "    \n",
    "    starting from\n",
    "      /root/.fastai/data/imagewoof2/train/n02096294/n02096294_1539.JPEG\n",
    "\n",
    "    applying parent_label gives\n",
    "      n02096294\n",
    "\n",
    "    applying Categorize -- {'vocab': None, 'sort': True, 'add_na': False} gives\n",
    "      TensorCategory(0)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "fastai provides **Transform** pipeline construct that allows not only to augment the data but also allows one to change the folder name into a more readable one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_dict = dict(\n",
    "    n02086240= 'Shih-Tzu',\n",
    "    n02087394= 'Rhodesian ridgeback',\n",
    "    n02088364= 'Beagle',\n",
    "    n02089973= 'English foxhound',\n",
    "    n02093754= 'Australian terrier',\n",
    "    n02096294= 'Border terrier',\n",
    "    n02099601= 'Golden retriever',\n",
    "    n02105641= 'Old English sheepdog',\n",
    "    n02111889= 'Samoyed',\n",
    "    n02115641= 'Dingo'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'n02086240': 'Shih-Tzu',\n",
       " 'n02087394': 'Rhodesian ridgeback',\n",
       " 'n02088364': 'Beagle',\n",
       " 'n02089973': 'English foxhound',\n",
       " 'n02093754': 'Australian terrier',\n",
       " 'n02096294': 'Border terrier',\n",
       " 'n02099601': 'Golden retriever',\n",
       " 'n02105641': 'Old English sheepdog',\n",
       " 'n02111889': 'Samoyed',\n",
       " 'n02115641': 'Dingo'}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('n02087394', 'Rhodesian ridgeback', 'Rhodesian ridgeback')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item = (path/'train').ls()[0]\n",
    "item.name, label_dict[item.name], label_dict.__getitem__(item.name) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding this as part of transformation to the target using get_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pipeline is composed of functions/transforms for encoding/decoding purpose.\n",
    "get_y = Pipeline([parent_label, label_dict.__getitem__])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's rewrite the datablock and run the summary if it transform the y as we expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dblk = DataBlock(\n",
    "    blocks = (ImageBlock, CategoryBlock),\n",
    "    get_items = get_image_files,\n",
    "    splitter = GrandparentSplitter(train_name='train', valid_name='val'),\n",
    "    get_y = get_y,\n",
    "    item_tfms = [Resize(size=448, method='crop')], #data augmentation see more here https://docs.fast.ai/vision.augment.html\n",
    "    batch_tfms = [Normalize.from_stats(*imagenet_stats)]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting-up type transforms pipelines\n",
      "Collecting items from /root/.fastai/data/imagewoof2/train/n02087394\n",
      "Found 942 items\n",
      "2 datasets of sizes 942,0\n",
      "Setting up Pipeline: PILBase.create\n",
      "Setting up Pipeline: parent_label -> dict.__getitem__ -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}\n",
      "\n",
      "Building one sample\n",
      "  Pipeline: PILBase.create\n",
      "    starting from\n",
      "      /root/.fastai/data/imagewoof2/train/n02087394/n02087394_46.JPEG\n",
      "    applying PILBase.create gives\n",
      "      PILImage mode=RGB size=500x451\n",
      "  Pipeline: parent_label -> dict.__getitem__ -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}\n",
      "    starting from\n",
      "      /root/.fastai/data/imagewoof2/train/n02087394/n02087394_46.JPEG\n",
      "    applying parent_label gives\n",
      "      n02087394\n",
      "    applying dict.__getitem__ gives\n",
      "      Rhodesian ridgeback\n",
      "    applying Categorize -- {'vocab': None, 'sort': True, 'add_na': False} gives\n",
      "      TensorCategory(0)\n",
      "\n",
      "Final sample: (PILImage mode=RGB size=500x451, TensorCategory(0))\n",
      "\n",
      "\n",
      "Setting up after_item: Pipeline: Resize -- {'size': (448, 448), 'method': 'crop', 'pad_mode': 'reflection', 'resamples': (2, 0), 'p': 1.0} -> ToTensor\n",
      "Setting up before_batch: Pipeline: \n",
      "Setting up after_batch: Pipeline: IntToFloatTensor -- {'div': 255.0, 'div_mask': 1} -> Normalize -- {'mean': tensor([[[[0.4850]],\n",
      "\n",
      "         [[0.4560]],\n",
      "\n",
      "         [[0.4060]]]], device='cuda:0'), 'std': tensor([[[[0.2290]],\n",
      "\n",
      "         [[0.2240]],\n",
      "\n",
      "         [[0.2250]]]], device='cuda:0'), 'axes': (0, 2, 3)}\n",
      "\n",
      "Building one batch\n",
      "Applying item_tfms to the first sample:\n",
      "  Pipeline: Resize -- {'size': (448, 448), 'method': 'crop', 'pad_mode': 'reflection', 'resamples': (2, 0), 'p': 1.0} -> ToTensor\n",
      "    starting from\n",
      "      (PILImage mode=RGB size=500x451, TensorCategory(0))\n",
      "    applying Resize -- {'size': (448, 448), 'method': 'crop', 'pad_mode': 'reflection', 'resamples': (2, 0), 'p': 1.0} gives\n",
      "      (PILImage mode=RGB size=448x448, TensorCategory(0))\n",
      "    applying ToTensor gives\n",
      "      (TensorImage of size 3x448x448, TensorCategory(0))\n",
      "\n",
      "Adding the next 3 samples\n",
      "\n",
      "No before_batch transform to apply\n",
      "\n",
      "Collating items in a batch\n",
      "\n",
      "Applying batch_tfms to the batch built\n",
      "  Pipeline: IntToFloatTensor -- {'div': 255.0, 'div_mask': 1} -> Normalize -- {'mean': tensor([[[[0.4850]],\n",
      "\n",
      "         [[0.4560]],\n",
      "\n",
      "         [[0.4060]]]], device='cuda:0'), 'std': tensor([[[[0.2290]],\n",
      "\n",
      "         [[0.2240]],\n",
      "\n",
      "         [[0.2250]]]], device='cuda:0'), 'axes': (0, 2, 3)}\n",
      "    starting from\n",
      "      (TensorImage of size 4x3x448x448, TensorCategory([0, 0, 0, 0], device='cuda:0'))\n",
      "    applying IntToFloatTensor -- {'div': 255.0, 'div_mask': 1} gives\n",
      "      (TensorImage of size 4x3x448x448, TensorCategory([0, 0, 0, 0], device='cuda:0'))\n",
      "    applying Normalize -- {'mean': tensor([[[[0.4850]],\n",
      "\n",
      "         [[0.4560]],\n",
      "\n",
      "         [[0.4060]]]], device='cuda:0'), 'std': tensor([[[[0.2290]],\n",
      "\n",
      "         [[0.2240]],\n",
      "\n",
      "         [[0.2250]]]], device='cuda:0'), 'axes': (0, 2, 3)} gives\n",
      "      (TensorImage of size 4x3x448x448, TensorCategory([0, 0, 0, 0], device='cuda:0'))\n"
     ]
    }
   ],
   "source": [
    "dblk.summary((path/'train').ls()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, we have introduced an additional transform and converted the y from n02096294 to Border terrier\n",
    "```\n",
    "Pipeline: parent_label -> dict.__getitem__ -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}\n",
    "\n",
    "    starting from\n",
    "      /root/.fastai/data/imagewoof2/train/n02096294/n02096294_1539.JPEG\n",
    "\n",
    "    applying parent_label gives\n",
    "      n02096294\n",
    "\n",
    "    applying dict.__getitem__ gives\n",
    "      Border terrier\n",
    "\n",
    "    applying Categorize -- {'vocab': None, 'sort': True, 'add_na': False} gives\n",
    "      TensorCategory(0)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Flip -- {'size': None, 'mode': 'bilinear', 'pad_mode': 'reflection', 'mode_mask': 'nearest', 'align_corners': True, 'p': 0.5}:\n",
       " encodes: (TensorImage,object) -> encodes\n",
       " (TensorMask,object) -> encodes\n",
       " (TensorBBox,object) -> encodes\n",
       " (TensorPoint,object) -> encodes\n",
       " decodes: ,\n",
       " Brightness -- {'max_lighting': 0.2, 'p': 1.0, 'draw': None, 'batch': False}:\n",
       " encodes: (TensorImage,object) -> encodes\n",
       " decodes: ,\n",
       " RandomResizedCropGPU -- {'size': (224, 224), 'min_scale': 0.75, 'ratio': (1, 1), 'mode': 'bilinear', 'valid_scale': 1.0, 'p': 1.0}:\n",
       " encodes: (TensorImage,object) -> encodes\n",
       " decodes: ]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aug_transforms(size=224, min_scale=0.75)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#If you want more fine grained control,explictly define your batch tfms\n",
    "\n",
    "# batch_tfms = [FlipItem(), RandomResizedCrop(128, min_scale=0.35),\n",
    "#                   Normalize.from_stats(*imagenet_stats)\n",
    "#                   ]\n",
    "\n",
    "# Or go with the defaults\n",
    "#batch_tfms = [aug_transforms(size=128, min_scale=0.75)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the datablock to prepare for modeling\n",
    "dblk = DataBlock(\n",
    "    blocks = (ImageBlock, CategoryBlock),\n",
    "    get_items = get_image_files,\n",
    "    splitter = GrandparentSplitter(train_name='train', valid_name='val'),\n",
    "    get_y = get_y,\n",
    "    #Resize images to relatively “large” dimensions\n",
    "    item_tfms = [Resize(size=448, method='crop')], \n",
    "    #data augmentation see more here https://docs.fast.ai/vision.augment.html\n",
    "    batch_tfms = [FlipItem(), RandomResizedCrop(128, min_scale=0.35),\n",
    "                  Normalize.from_stats(*imagenet_stats)\n",
    "                  ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting-up type transforms pipelines\n",
      "Collecting items from /root/.fastai/data/imagewoof2\n",
      "Found 12954 items\n",
      "2 datasets of sizes 9025,3929\n",
      "Setting up Pipeline: PILBase.create\n",
      "Setting up Pipeline: parent_label -> dict.__getitem__ -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}\n",
      "\n",
      "Building one sample\n",
      "  Pipeline: PILBase.create\n",
      "    starting from\n",
      "      /root/.fastai/data/imagewoof2/train/n02087394/n02087394_46.JPEG\n",
      "    applying PILBase.create gives\n",
      "      PILImage mode=RGB size=500x451\n",
      "  Pipeline: parent_label -> dict.__getitem__ -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}\n",
      "    starting from\n",
      "      /root/.fastai/data/imagewoof2/train/n02087394/n02087394_46.JPEG\n",
      "    applying parent_label gives\n",
      "      n02087394\n",
      "    applying dict.__getitem__ gives\n",
      "      Rhodesian ridgeback\n",
      "    applying Categorize -- {'vocab': None, 'sort': True, 'add_na': False} gives\n",
      "      TensorCategory(7)\n",
      "\n",
      "Final sample: (PILImage mode=RGB size=500x451, TensorCategory(7))\n",
      "\n",
      "\n",
      "Setting up after_item: Pipeline: Resize -- {'size': (448, 448), 'method': 'crop', 'pad_mode': 'reflection', 'resamples': (2, 0), 'p': 1.0} -> ToTensor\n",
      "Setting up before_batch: Pipeline: \n",
      "Setting up after_batch: Pipeline: FlipItem -- {'p': 0.5} -> RandomResizedCrop -- {'size': (128, 128), 'min_scale': 0.35, 'ratio': (0.75, 1.3333333333333333), 'resamples': (2, 0), 'val_xtra': 0.14, 'p': 1.0} -> IntToFloatTensor -- {'div': 255.0, 'div_mask': 1} -> Normalize -- {'mean': tensor([[[[0.4850]],\n",
      "\n",
      "         [[0.4560]],\n",
      "\n",
      "         [[0.4060]]]], device='cuda:0'), 'std': tensor([[[[0.2290]],\n",
      "\n",
      "         [[0.2240]],\n",
      "\n",
      "         [[0.2250]]]], device='cuda:0'), 'axes': (0, 2, 3)}\n",
      "\n",
      "Building one batch\n",
      "Applying item_tfms to the first sample:\n",
      "  Pipeline: Resize -- {'size': (448, 448), 'method': 'crop', 'pad_mode': 'reflection', 'resamples': (2, 0), 'p': 1.0} -> ToTensor\n",
      "    starting from\n",
      "      (PILImage mode=RGB size=500x451, TensorCategory(7))\n",
      "    applying Resize -- {'size': (448, 448), 'method': 'crop', 'pad_mode': 'reflection', 'resamples': (2, 0), 'p': 1.0} gives\n",
      "      (PILImage mode=RGB size=448x448, TensorCategory(7))\n",
      "    applying ToTensor gives\n",
      "      (TensorImage of size 3x448x448, TensorCategory(7))\n",
      "\n",
      "Adding the next 3 samples\n",
      "\n",
      "No before_batch transform to apply\n",
      "\n",
      "Collating items in a batch\n",
      "\n",
      "Applying batch_tfms to the batch built\n",
      "  Pipeline: FlipItem -- {'p': 0.5} -> RandomResizedCrop -- {'size': (128, 128), 'min_scale': 0.35, 'ratio': (0.75, 1.3333333333333333), 'resamples': (2, 0), 'val_xtra': 0.14, 'p': 1.0} -> IntToFloatTensor -- {'div': 255.0, 'div_mask': 1} -> Normalize -- {'mean': tensor([[[[0.4850]],\n",
      "\n",
      "         [[0.4560]],\n",
      "\n",
      "         [[0.4060]]]], device='cuda:0'), 'std': tensor([[[[0.2290]],\n",
      "\n",
      "         [[0.2240]],\n",
      "\n",
      "         [[0.2250]]]], device='cuda:0'), 'axes': (0, 2, 3)}\n",
      "    starting from\n",
      "      (TensorImage of size 4x3x448x448, TensorCategory([7, 7, 7, 7], device='cuda:0'))\n",
      "    applying FlipItem -- {'p': 0.5} gives\n",
      "      (TensorImage of size 4x3x448x448, TensorCategory([7, 7, 7, 7], device='cuda:0'))\n",
      "    applying RandomResizedCrop -- {'size': (128, 128), 'min_scale': 0.35, 'ratio': (0.75, 1.3333333333333333), 'resamples': (2, 0), 'val_xtra': 0.14, 'p': 1.0} gives\n",
      "      (TensorImage of size 4x3x448x448, TensorCategory([7, 7, 7, 7], device='cuda:0'))\n",
      "    applying IntToFloatTensor -- {'div': 255.0, 'div_mask': 1} gives\n",
      "      (TensorImage of size 4x3x448x448, TensorCategory([7, 7, 7, 7], device='cuda:0'))\n",
      "    applying Normalize -- {'mean': tensor([[[[0.4850]],\n",
      "\n",
      "         [[0.4560]],\n",
      "\n",
      "         [[0.4060]]]], device='cuda:0'), 'std': tensor([[[[0.2290]],\n",
      "\n",
      "         [[0.2240]],\n",
      "\n",
      "         [[0.2250]]]], device='cuda:0'), 'axes': (0, 2, 3)} gives\n",
      "      (TensorImage of size 4x3x448x448, TensorCategory([7, 7, 7, 7], device='cuda:0'))\n"
     ]
    }
   ],
   "source": [
    "dblk.summary(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls = dblk.dataloaders(path, bs=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#dls.show_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = dls.one_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, torch.Size([64, 3, 448, 448]), torch.Size([3, 448, 448]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(b), b[0].shape, b[0][0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline using Resnet34"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can leverage Transfer Learning to speed up the training and also improve the metrics.\n",
    "\n",
    "fastai provides Learner object which wraps the dataloaders, architecture, hyperparameters, metrics etc.\n",
    "\n",
    "For images, we can use cnn_learner(Convolutional Neural Network)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading: \"https://download.pytorch.org/models/resnet34-333f7ec4.pth\" to /root/.cache/torch/hub/checkpoints/resnet34-333f7ec4.pth\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7c52c0a688b04c829315d98ee4373841",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=87306240.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "learn = cnn_learner(dls, arch=resnet34, metrics=accuracy, loss_func=CrossEntropyLossFlat())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learner by default tries to download the pretrained weights for resnet architecture and apply custom head on top. Let's look at the model summary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of CrossEntropyLoss()"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.loss_func"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 1 - Baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.468834</td>\n",
       "      <td>0.210816</td>\n",
       "      <td>0.939425</td>\n",
       "      <td>02:49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.262372</td>\n",
       "      <td>0.188912</td>\n",
       "      <td>0.944515</td>\n",
       "      <td>03:38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if FINE_TUNE:\n",
    "  learn.fine_tune(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 2 - LR Finder, Discriminative LR, Mixed Precision, Resnet 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading: \"https://download.pytorch.org/models/resnet50-19c8e357.pth\" to /root/.cache/torch/hub/checkpoints/resnet50-19c8e357.pth\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b729c12b6fdf442aa290ba09bfc7dafd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=102502400.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "learn = cnn_learner(dls, arch=resnet50, metrics=accuracy, loss_func=CrossEntropyLossFlat()).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/fastai/learner.py:53: UserWarning: Could not load the optimizer state.\n",
      "  if with_opt: warn(\"Could not load the optimizer state.\")\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.010000000149011612, lr_steep=0.0003311311302240938)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEKCAYAAADn+anLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3xV9f3H8dcnm2xGmAHCUvaMDHGg1bpHrVvrwlJHq1attf39fmrtr/VnbbXapbi1DpwtirjqRIaEvRVZAQIEsslOvr8/ctGICSQk5557k/fz8bgP7z33nHvfuV7yzplfc84hIiLSkAi/A4iISOhSSYiISKNUEiIi0iiVhIiINEolISIijVJJiIhIo6L8DtBcXbp0cRkZGX7HEBEJK4sWLdrtnEtr7nJhVxIZGRlkZWX5HUNEJKyY2eZDWU6bm0REpFEqCRERaZRKQkREGqWSEBGRRnleEmYWaWZLzOzNBp6LNbMZZrbezBaYWYbXeUREpOmCsSZxI7CmkeemAvnOuYHAA8C9QcgjIiJN5GlJmFk6cBrwWCOznAU8Hbj/CvA9MzMvM0nLOedYua2QwtIqv6OIiMe8Pk/iz8BtQFIjz/cCsgGcc9VmVgh0BnZ7nKvVbMgtoai8mtG9U/2O8h27Syq46cWlLM0uIC0p9utbXFQkMVFGVEQEXRJjOX1UDwakJR709aprapm1IodHPt7A6pwiUjpEc8P3BvGjiX2JidLuLZG2yLOSMLPTgV3OuUVmNqWFrzUNmAbQp0+fVkjXMs455qzfzRNzNvLhulwiDP568VhOHdHjW/Mt3JTHjIXZ3HHGUJLjooOace2OIqY+lcXukgp+OC6dorIqdhVXsCaniIqqWqpq6m6FZVU88P4XjO2TynmZvTlpWHc6JcR867VyCsv415Lt/HP+ZrYVlDEgLYG7zhjKf9bu4rdvruaf8zdz0wmDGNg1kY7xMXSMj6FDTGRQf14R8YZ5NTKdmd0D/AioBuKAZOA159yl9eZ5B7jLOTfPzKKAHUCaO0CozMxM5+cZ19sKypj61ELW7iimS2IMP5qYwSdf5rIsu4Dpl43j+MHdAHhj2XZueWkZlTW1/PS4gdx60uFBy/je6p3c9OISEuOiePSyTEamN76Ws6uonNeXbOPlRVtZv6sEgMO6JTKhX2cGpCXwn7W7mLN+N87B+H6dmHZ0f44f3JWICMM5x0df5PK7WWu+Xnaf7slxHDmwM5MHdGHywC50T4nz9GcWkQMzs0XOucxmLxeM4UsDaxK3OudO32/69cAI59w1ZnYhcI5z7vwDvZafJVFZXct5j8xjw64S7jhjKGeO7klsVCRF5VVc8ugC1u0s5skrjmD51kLufXst4zM6kdwhms/W7+bjX0yha7K3vyj3VlRz/3tf8MRnGxnZK4Xpl2XSrYnv6ZxjxbZCPv1yNws25rFoUx57K2tI79iBc8amc86YXmR0SWhw2eqaWpZkF7CnpIL80iry9layensRc7/aTX5gv8WRAzpz2aS+nDCkG1GR2jQlEmxhUxJmdjeQ5ZybaWZxwLPAGCAPuNA5t+FAr+VnSdw1cxVPzd3Ew5eO5eTh3960lL+3kosenc+Xu0qoqXWcMaon9507kh2F5Zxw/8dcNL4Pvz17uGfZ3l21g7tmriKnqJxLJvThv08bSlz0oW/yqa6pZVtBGb07xhMRcWjHEtTWOtbsKOKDNbt4cWE22wrK6JESx3mZvTlmUBdGpqdqX4ZIkIR0SbSmQy2JVdsLeXXRNorKqygur6KorBqA1PhoUuOjSe4QTWxUJNERRmSk0TkhhpOH9SAlvm5fwqzlOVz//GKumtyPO84Y2uB75BZXcP3zi5nQrxM/P+Gwr3+5/s+/VvLC51t4/+ZjG/1r/FCVVFRz2yvLeGvFDgZ3T+J3PxjBuL4dW/U9WkNNreM/a3byzLzNzFlfd1xCh+hIMjM6csrwHvxgTC/txxDxkEriIN5ZtYNbXlpGclwUSXHRJHeIwjkoLKuioKyKwtIqKmtqv7VMbFQEp43owfFDunL7qys4rFsiL06b1Oy/fncVlXPsfR9x4tBuPHTRmGZnb0xOYRlXPrmQL3eVcPOJhzHtmP5Eh8GmnPy9lSzYuId5X+1hzvrdfJW7l9T4aC4e34fLJmVo/4WIB1QSrcA5R02to7rWsX5XCS8u3MK/l2ynuKKajvHRzLrhaHqmdjik177vnbX87cOvePNnRzG8V0qLs67cVsjUpxeyt6KGv10ylmMPa/Zl4kOCc46Fm/J5fM4G3l29EwNGpqcyeWBnJg/swri+HYmN0hqGSEupJDxSWlnNu6t2MrBrYot+uReWVXHMHz4EoH9aAj1TO9ArtQNdk2LpnBhD54RY0jt2oF+XBA50PmF5VQ2vLt7K72atIbVDNE9ceQSDuycfcq5QsmVPKa8symbO+t0s21pITa2jR0ocf79kLGP6hN4mNJFwopIIA/M37OHVRVvZXljG9oJythWUUVn97U1c/bskcNLw7pwyvDuDuiaxry/27K3kufmbeeHzLeSXVjGub0f+cclYz4+Y8ktxeRVzv9rDb99cza6iCu48cygXj+9zwAIVkcapJMKQc47iimr2lFSyu6SCtTlFvLNqJ/M27KGm9rv/XyIMThjSjSsn92Ni/07t4hdmQWklN764lI+/yOW8celcfXR/OsZHkxofoyOjRJpBJdGG5O+t5IO1u9hVXIGj7v9PTGQEJw3rTu9O8T6nC76aWseD73/BQx+s/9b0rkmx3Pr9wzkvM71dFKZIS6gkpM1bk1PEV7kl5JdWUbC3ko++yGXR5nwm9OvE788Z0aTrT4m0VyoJaXdqax0zsrK55601lFfVMvXofkw7uj8d97v2lIgcekloo66ErYgI46LxffjPLVM4bWQPHv74Kybf+wH3zF7D7pIKv+OJtAkqCQl7aUmxPHDBaN696RhOHNqNRz/ZwFH3fsBLC7P9jiYS9lQS0mYM6pbEgxeO4b2bjyWzbydue3U598xeQ20DR4qJSNOoJKTNGZCWyFNXHsElE/rwyMcbuO65xZRV1vgdSyQsqSSkTYqKjOB/zx7O/5w+lHdW7+Ccf8xl8ZZ8v2OJhB2VhLRZZsbUo/rx+OWZ7Cmp4Jy/z+XnM5ayo7Dc72giYUMlIW3e8YO78cGtU7huygBmLc/huD9+xNNzNxFuh3+L+EElIe1CYmwUt508mPdvPpYJ/Ttx58xV/Pr1Fd+5dpaIfJtKQtqVPp3jeeLyI7huygBe+DybSx9fQN7eSr9jiYQslYS0OxERxm0nD+bBC0ezNLuAs/42h+y8Ur9jiYQklYS0W2eN7sVLP5lEUVk1lz/5OQWlWqMQ2Z9KQtq10b1Tmf6jcWzNK+PHz2RRXqXzKUTqU0lIuzehf2f+eP4oFm7K55aXl+kMbZF6ovwOIBIKzhzVk5yCMu6ZvZYuCTHcccYwIiM0RoWISkIkYNox/cktruCxORv5YmcJD140mq5JbXN4WJGm0uYmkQAz479PH8p9545kSXY+pz44h7nrd/sdS8RXKgmR/ZyX2Zt/X38UKR2iuPTxBby2eKvfkUR8o5IQacDh3ZOY+dOjGN+vE//1+krW7yrxO5KIL1QSIo1IiI3iwQvHEBcdwQ0vLKGiWofHSvujkhA5gG7JcfzxvFGszini3tnr/I4jEnQqCZGD+N6QblxxZAZPfLaRD9fu8juOSFCpJESa4PZTBjO4exK3vLyMbQVlfscRCRqVhEgTxEVH8teLx1JVXcuPn86itLLa70giQaGSEGmigV0TeejiMazdUcTNM3T5DmkfVBIizXDc4V359alDeHvVDv78/hd+xxHxnGclYWZxZva5mS0zs1Vm9psG5rnCzHLNbGngdrVXeURay9Sj+nFBZm8e+mA9M5dt9zuOiKe8vHZTBXC8c67EzKKBOWY22zk3f7/5ZjjnfuphDpFWZWb89uzhbNy9l1+8vIx+nRMYkZ7idywRT3i2JuHq7DtNNTpw00ZcaRNioiL4+6Vj6ZIYy7Rns8gtrvA7kognPN0nYWaRZrYU2AW855xb0MBsPzSz5Wb2ipn19jKPSGvqkhjLIz8aR35pJdf+cxGV1bV+RxJpdZ6WhHOuxjk3GkgHxpvZ8P1meQPIcM6NBN4Dnm7odcxsmpllmVlWbm6ul5FFmmV4rxTuO3cUWZvzuXPmSpzTyrK0LUE5usk5VwB8CJy83/Q9zrl96+mPAeMaWX66cy7TOZeZlpbmbViRZjpjVE+umzKAFz7P5l9Lt/kdR6RVeXl0U5qZpQbudwBOBNbuN0+Peg/PBNZ4lUfES7d8/3DG9knlf99cQ0Fppd9xRFqNl2sSPYAPzWw5sJC6fRJvmtndZnZmYJ4bAofHLgNuAK7wMI+IZyIjjN/9YAQFZVXc+/bagy8gEiYs3LahZmZmuqysLL9jiDTo92+tYfonG3j12kmM69vJ7zgiXzOzRc65zOYupzOuRVrRjd8bRM+UOH792kqqanS0k4Q/lYRIK0qIjeKuM4exbmcxT8zZ6HcckRZTSYi0su8P686JQ7vx5/e/JKdQlxWX8KaSEPHAHacPpcY5/vC2RrOT8KaSEPFA707xXH1UP15fso3FW/L9jiNyyFQSIh657riBpCXFcvcbqzX2hIQtlYSIRxJjo7jtpMNZml3Av5fpTGwJTyoJEQ/9cGw6I9NTuHf2Og15KmFJJSHioYgI447Th7KjqJxHPt7gdxyRZlNJiHgsM6MTpwzvzuNzNuq6ThJ2VBIiQXDjCYMoqajmsU91gp2EF5WESBAM7p7MaSN68ORnG8nfq7UJCR8qCZEgufGEQZRW1fDop9o3IeFDJSESJId1S+K0ET14eu4m8rQ2IWFCJSESRDd+r25tYvonWpuQ8KCSEAmiQd2SOGNkT56Zt4nc4oqDzi/iN5WESJDddMIgqmscd85c6XcUkYNSSYgEWf+0RG48YRBvrdjBWyty/I4jckAqCREfTDumP8N7JXPHv1dqJ7aENJWEiA+iIyO479xRFJRW8Zs3VvkdR6RRKgkRnwzpkcz1xw3k30u3897qnX7HEWmQSkLER9cfN5DB3ZO4498rKa+q8TuOyHeoJER8FBMVwV1nDiOnsJyn5m7yO47Id6gkRHw2sX9njjs8jb9/uF5XiZWQo5IQCQG/PGUwxRXV/O3D9X5HEfkWlYRICBjcPZkfjk3n6bmb2Zpf6nccka+pJERCxM0nHgYG97/3hd9RRL6mkhAJET1TO3DlkRm8vmQba3KK/I4jAqgkRELKdVMGkhQbxZ/e1dqEhAaVhEgISYmPZtox/Xl/zU6WZRf4HUdEJSESaq6Y3I/U+GgeeF9rE+I/lYRIiEmMjeInxwzgo3W5LNqc73ccaedUEiIh6PIj+9I5IYYHdKST+MyzkjCzODP73MyWmdkqM/tNA/PEmtkMM1tvZgvMLMOrPCLhJD4mimunDGDO+t0s2LDH7zjSjnm5JlEBHO+cGwWMBk42s4n7zTMVyHfODQQeAO71MI9IWLl0Yl/SkmL503tf4JzzO460U56VhKtTEngYHbjt/00/C3g6cP8V4HtmZl5lEgkncdGR/PS4gXy+MY+P1uX6HUfaKU/3SZhZpJktBXYB7znnFuw3Sy8gG8A5Vw0UAp29zCQSTi4a34eMzvH8/q01VNfU+h1H2iFPS8I5V+OcGw2kA+PNbPihvI6ZTTOzLDPLys3VX1TSfsRERXD7KYP5clcJM7Ky/Y4j7VBQjm5yzhUAHwIn7/fUNqA3gJlFASnAd/bSOeemO+cynXOZaWlpXscVCSknDevO+IxOPPDeFxSXV/kdR9oZL49uSjOz1MD9DsCJwNr9ZpsJXB64fy7wgdMeOpFvMTP+67Qh7C6p5OGPv/I7jrQzXq5J9AA+NLPlwELq9km8aWZ3m9mZgXkeBzqb2XrgZuB2D/OIhK1RvVM5a3RPHvt0I9sLyvyOI+2Ihdsf7pmZmS4rK8vvGCJBtzW/lOP/9DGnj+jB/ReM9juOhBkzW+Scy2zucjrjWiRMpHeM58rJGby+VJcSl+BpUkmYWYKZRQTuH2ZmZ5pZtLfRRGR/1x1bdynxP76zzu8o0k40dU3iEyDOzHoB7wI/Ap7yKpSINCwlPpprpgzgP2t3sXBTnt9xJIh+88Yq3l21I+jv29SSMOdcKXAO8Hfn3HnAMO9iiUhjrjyyH12TYrl39lpdrqOdqKiu4am5m1iTUxz0925ySZjZJOASYFZgWqQ3kUTkQDrERHLjCYPI2pzPB2t3+R1HgmBbfhnOQe9OHYL+3k0tiZuAXwGvO+dWmVl/6k6OExEfnJ/Zm4zO8fzh7XXU1Gptoq3bklcKQJ9O8UF/7yaVhHPuY+fcmc65ewM7sHc7527wOJuINCI6MoJbvn8463YW87Iu19HmZefXnRvTO1RLwsyeN7NkM0sAVgKrzewX3kYTkQM5bUQPxvfrxO/fWsOu4nK/44iHtuaVEhMVQVpibNDfu6mbm4Y654qAs4HZQD/qjnASEZ9ERBj3nDOC8upa7n5jtd9xxENb8krp3bEDERHBH0mhqSURHTgv4mxgpnOuiu+ODSEiQTYgLZGfHTeQN5fn8MHanX7HEY9k55f6sqkJml4SjwCbgATgEzPrC+iUT5EQ8JNjB3BYt0T++/WVlFRU+x1HPLBlT6kvO62h6TuuH3LO9XLOnRoYcW4zcJzH2USkCWKiIrjnnJHkFJXrTOw2qLC0iqLyanp3DOGSMLMUM7t/38A/ZvYn6tYqRCQEjOvbkcsm9uXpeZv4bP1uv+NIK8rOrzv81Y9zJKDpm5ueAIqB8wO3IuBJr0KJSPP98pTB9O+SwM0vLSVvb6XfcaSVZOftK4kQXpMABjjn7nTObQjcfgP09zKYiDRPfEwUD144hvy9Vfzy1eW6ZEcb8c2aRGiXRJmZHbXvgZlNBjTyiUiIGd4rhdtOPpz3Vu/kuQVb/I4jrWBLXimp8dEkx/lz4e2oJs53DfCMmaUEHufzzbCjIhJCrprcj0++3M1v31zN+H6dOKxbkt+RpAWy88p822kNTT+6aZlzbhQwEhjpnBsDHO9pMhE5JBERxh/PG0libBS3v7qcWl3bKaxl55X6ttMamjkynXOuKHDmNdSNSS0iIahrUhy/OnUIi7cU8OrirX7HkUNUW+vYml/m2/4IaNnwpcE/P1xEmuycMb0Y2yeVe99eS2FZld9x5BDsKq6gsqY29Dc3NULrsCIhLCLCuPus4ezZW8mf3//C7zhyCLb4fPgrHKQkzKzYzIoauBUDPYOUUUQO0fBeKVwyoQ/PzNvM2h26kk64yfZxHIl9DlgSzrkk51xyA7ck51xTj4wSER/d+v3DSY6L4o5/r9K5E2FmS14pZtAzNc63DC3Z3CQiYSA1PobbTh7M5xvzeOFzDVAUTrLzS+mRHEdslH+jRaskRNqBCzJ7c/SgLvz2zdVs3L3X7zjSRFvzykj3cVMTqCRE2oWICOO+c0cRExXBTTOWUlVT63ckaYK6wYZUEiISBN1T4rjnnBEsyy7gLx+s9zuOHER5VQ07i8t93WkNKgmRduXUET344dh0/vbhehZtzvc7jhzAtoIynPPvEuH7qCRE2pm7zhxKj5Q4bn5pKWWVNX7HkUaEwuGvoJIQaXeS4qK579xRbN5Tyh/f1Uh2ocrvcST2UUmItEOTBnTm0ol9eOKzjdrsFKKy88uIiYogLTHW1xwqCZF26vZThtAzpQO3vbKM8iptdgo163YU079LAhER/l4mTyUh0k4lxkZxzzkj+Cp3L39+/0u/40g9zjlWbS9keK+Ug8/sMc9Kwsx6m9mHZrbazFaZ2Y0NzDPFzArNbGngdodXeUTku445LI0LMnsz/ZOvWJpd4HccCdhVXMHukkqG90z2O4qnaxLVwC3OuaHAROB6MxvawHyfOudGB253e5hHRBrw69OG0C05jp/PWMreimq/4wiwclshAMPa8pqEcy7HObc4cL8YWAP08ur9ROTQpHSI5v7zR7Npz17ufmO133EEWLW9CDMY0qNtr0l8zcwygDHAggaenmRmy8xstpkNC0YeEfm2SQM6c82xA5iRlc3sFTl+x2n3Vm4rpF/nBBJj/b/YtuclYWaJwKvATfWGPt1nMdA3MH72X4B/NfIa08wsy8yycnNzvQ0s0k7dfOJhjEpP4fbXVrC9oMzvOO3aqu1FIbGpCTwuCTOLpq4gnnPOvbb/84Exs0sC998Cos2sSwPzTXfOZTrnMtPS0ryMLNJuRUdG8OCFY6iqqeXml5ZSW6uxJ/yQv7eSbQVlDAuBndbg7dFNBjwOrHHO3d/IPN0D82Fm4wN59niVSUQOLKNLAnedMYz5G/J4cu4mv+O0S6tz6ja4DO8ZGmsSXm7wmgz8CFhhZksD034N9AFwzj0MnAtca2bVQBlwodPQWSK+Oi8znbdX7eC+d9Zy/OCu9OuS4HekduXrI5tCZE3Cs5Jwzs0BDniqoHPur8BfvcogIs1nZtxzzghOvP9jfvHyMmb8ZBKRPp/1256s3F5Er9QOdEyI8TsKoDOuRaQB3ZLjuOvMYWRtzufJzzb6HaddWbW9kKEhshYBKgkRacQPxvTihCFdue+ddXyVW+J3nHZhb0U1G3fvDZn9EaCSEJFGmBm//8EI4qIjueWlZRryNAjW5BThXOjsjwCVhIgcQNfkOH7/gxEszS7gof/oIoBe27fTOhQu7LePSkJEDui0kT04PzOdv364nvkbdIS6l1ZtL6JzQgzdkv0dQ6I+lYSIHNSdZwwjo3MCP5+xlMLSKr/jtFkrA2daB04fCwkqCRE5qITYKB68cDS5xRX86vXl6HSm1ldRXcOXO4tDan8EqCREpIlGpqdy60mH89aKHfxzwRa/47Q563YUU13rVBIiEr6mHd2f4w5P4+43Vmls7Fa2cFPd5zmub0efk3ybSkJEmiwiwvjzBWPomdqBa/+5iF3F5X5HajMWbswjvWMHeqR08DvKt6gkRKRZUuKjefjScRSXV3P9c4uprNb5Ey3lnGPhpjzGZ3TyO8p3qCREpNmG9Ejm/344goWb8vndLI1m11Ibdu9lz95KjuinkhCRNuKs0b24anI/np63mbdX7vA7TlhbuDEPgCO0JiEibcntpwxmRK8Ubn9tOTmFGs3uUH2+KY/OCTEMSAu9y7KrJETkkMVERfDQRWOorK7l5zOWUqPR7A7Jwk15ZGZ0DKmT6PZRSYhIi/TrksBvzqwbze7hj7/yO07Y2VFYTnZeGeP7dfY7SoNUEiLSYueOS+f0kT24/70vWLxF5080x+eb6vZHhOKRTaCSEJFWYGb87gcj6JESx8+eX0JBaaXfkcLGwo15JMREMqRHkt9RGqSSEJFWkdIhmr9ePJZdxeXc+rKu79RUCzflMbZvR6IiQ/PXcWimEpGwNLp3Kr86ZQjvr9nJY59q2NODKSitZN3O4pDd1AQqCRFpZVdOzuDkYd259+21ur7TQWRtysc5QvIkun1UEiLSqsyMP5w3kp6pHfjp84vZU1Lhd6SQtXBTHtGRxujeqX5HaZRKQkRaXXJcNH+/ZCx5eyu57rnFGh+7EQs25jEyPZW46Ei/ozRKJSEinhjeK4V7fziSBRvz+M0bq/yOE3J2FJazbGsBxx6W5neUA4ryO4CItF1nj+nFmpwiHvlkA0N6JHPJhL5+RwoZs1fm4FzdGOKhTGsSIuKp204ezJTD07jz36tYsGGP33FCxqzlOQzunsSAtES/oxyQSkJEPBUZYTx44Rj6dI5n2rOLWLej2O9IvsspLCNrcz6nh/haBKgkRCQIUjpE89QV44mLjuDSxxewafdevyP5avaKukurnzpCJSEiAkCfzvH8c+oEqmtqueSxBe360uKzVuQwpEcy/UN8UxOoJEQkiAZ1S+KZqyZQWFbFpY8tYHc7PIdie0EZi8JkUxOoJEQkyEakp/D45ZlsKyjjvIfnkZ1X6nekoJq9Mnw2NYFKQkR8MKF/Z567egL5pZX84O9zWbmt0O9IQTNr+XaG9kimX5fQG4WuISoJEfHFuL6deOWaScRGRXDBI/P49MtcvyN5bltBGYu3FIT8uRH1eVYSZtbbzD40s9VmtsrMbmxgHjOzh8xsvZktN7OxXuURkdAzsGsSr157JL07xXPVUwt5d9UOvyN5avaKHABOC5NNTeDtmkQ1cItzbigwEbjezIbuN88pwKDAbRrwDw/ziEgI6p4Sx4yfTGJYzxSue24xb69su0Uxa0UOw3omkxEmm5rAw5JwzuU45xYH7hcDa4Be+812FvCMqzMfSDWz8KlYEWkVKR2ieWbqeEakp/DT5xd//Rd3W7K9oIwlWwrCZof1PkHZJ2FmGcAYYMF+T/UCsus93sp3i0RE2oHkuGieuWo8o3qn8tMXljBredsqirfCcFMTBKEkzCwReBW4yTlXdIivMc3MsswsKze37e/cEmmvkuKiefqq8Yztk8pNM5Yw58vdfkdqNbNW5DC0R3htagKPS8LMoqkriOecc681MMs2oHe9x+mBad/inJvunMt0zmWmpYX2ZXVFpGUSY6N47PIj6N8lkZ88m9UmDo/dt6kpnI5q2sfLo5sMeBxY45y7v5HZZgKXBY5ymggUOufa1jqmiDRbSoe6NYrU+BiueHIhW/aE9wl34bqpCbxdk5gM/Ag43syWBm6nmtk1ZnZNYJ63gA3AeuBR4DoP84hIGOmeEsfTVx1BdW0tlz2xgNzi8L2Ex1thuqkJPBx0yDk3B7CDzOOA673KICLhbWDXJB6//AgufWwBF06fxws/nkjX5Di/YzXL9sAJdL846XC/oxwSnXEtIiFtXN+OPHnlEeQUlnPh9PnsLCr3O1Kz7NvUFG6Hvu6jkhCRkDexf2eevmo8O4vqimJHYfgUxb5NTeFyrab9qSREJCwckdGJZ6aOJ7e4ggumz2NbQeiPR7EhtyTsrtW0P5WEiISNcX078ezU8eTtreSCR0L/MuOPfLyB2KgIzs/sffCZQ5RKQkTCypg+HXn+6omUVFRz/iPz2BiiQ6HmFJbx2pKtXHBEb9KSYv2Oc8hUEiISdkakp/D81ROpqK7l/EfmsX5Xsd+RvuPRTzZS64Gob2sAAArZSURBVODHR/f3O0qLqCREJCwN7ZnMjGkTAbhw+ny+3Bk6RZG3t5IXPt/CWaN60rtTvN9xWkQlISJha1C3JF6cNpEIMy6cPp91O0KjKJ6au4myqhqunTLA7ygtppIQkbA2IC2RF6dNJCrSuPjR+azdcUjXEW01JRXVPPXZRr4/tBuDuiX5mqU1qCREJOz1T0vkxWmTiI6M4OJHF/i6RvH8gs0UlVdz3XEDfcvQmlQSItIm9OuSwIvTJhIdaVzy2HzW7yoJeoaaWsdTn21iUv/OjO6dGvT394JKQkTajIwuCTx3dd3O7Esem8/mPcE9PPY/a3ayvbCcy4/MCOr7ekklISJtysCuiTx39UQqq2u5+NEFbM0P3gl3z87fTPfkOE4Y0jVo7+k1lYSItDmHd0/i2akTKC6v4of/mMuCDXs8f88NuSV8+uVuLp7Qh6jItvOrte38JCIi9QzvlcKL0ybRITqSix6dz4Pvf0lNrfPs/Z5bsIWoCOPC8eF7CY6GqCREpM0a2jOZN284mjNH9eSB97/g0scWeHKp8bLKGl7Oyubk4d3pmhRe410cjEpCRNq0xNgoHrhgNH84dyRLsws46c+ffD3GQ2uZuWwbReXVXDYpo1VfNxSoJESkzTMzzs/szawbjqJvp3iue24xN7+0lOLyqha/tnOOZ+Zt5vBuSRyR0bEV0oYWlYSItBv90xJ55dojueF7g/jXkm2c8uCnLN9a0KLXXLAxj1Xbi7h0Ul/MDjhic1hSSYhIuxIdGcHNJx7Gy9cciXNw7j/m8fyCLTjX/J3aReVV3PryMnqlduCcMb08SOs/lYSItEvj+nbkjZ8dxYT+nfj16yu49eXllFXWNHl55xy/fm0FOYXlPHTRGBJiozxM6x+VhIi0W50SYnjqyvHc+L1BvLZkK8f98SOmf/JVk/ZVvJy1lTeX53DziYcxrm/b2xexjx3KKpafMjMzXVZWlt8xRKSNmffVHh76z5fM27CHpNgoLprQh+MHd2VkegrxMd9eS1i/q4Qz/jKH0b1T+efVE4iMCP19EWa2yDmX2ezlVBIiIt9YvrWARz7ZwOwVOdQ6iIwwhvRIom+nBEoqqikur2LznlJqnePtm46hW3J4nBdxqCXRNjeiiYgcopHpqfzt4rHk761kSXY+S7YUsHhLPmt2FJEUG0ViXBRHZHTi6qP7hU1BtIRKQkSkAR0TYjh+cDeOH9zN7yi+0o5rERFplEpCREQapZIQEZFGqSRERKRRKgkREWmUSkJERBqlkhARkUapJEREpFFhd1kOM8sFNgMpQGFg8sHu7/tvF2D3Ibxt/ddszvP7Tz/Q4/2z1p92KLmDmbn+fT8+64NNC8XMDU1vq9+P1sp8oFzNydxYzsbut5XvdF/nXFpzw+OcC8sbML2p9+v9N6ul79Wc5/effqDH+2dtae5gZvb7sz7YtFDM3J6+H62V+WC5m5o5HL4ffmeufwvnzU1vNON+/Wktfa/mPL//9AM9bihrS3IHM3P9+3581gebFoqZG5reVr8frZX5YMs3NfP+00Lx++F35q+F3eamljCzLHcIV0H0WzjmVubgCcfcyhwcrZE5nNckDsV0vwMconDMrczBE465lTk4Wpy5Xa1JiIhI87S3NQkREWkGlYSIiDRKJSEiIo1SSQSY2dFm9rCZPWZmc/3O0xRmFmFmvzOzv5jZ5X7naSozm2JmnwY+7yl+52kqM0swsywzO93vLE1hZkMCn/ErZnat33mayszONrNHzWyGmX3f7zxNYWb9zexxM3vF7ywHEvgOPx34fC9pyjJtoiTM7Akz22VmK/ebfrKZrTOz9WZ2+4Fewzn3qXPuGuBN4Gkv8waytTgzcBaQDlQBW73KWl8r5XZACRBHEHK3UmaAXwIveZPy21rpO70m8J0+H5jsZd56+Voj97+ccz8GrgEu8DJvIFtrZN7gnJvqbdKGNTP/OcArgc/3zCa9QUvPxguFG3AMMBZYWW9aJPAV0B+IAZYBQ4ER1BVB/VvXesu9BCSFQ2bgduAngWVfCZfPGogILNcNeC5MMp8IXAhcAZweDpkDy5wJzAYuDpfvR73l/gSMDbPMQfl32IL8vwJGB+Z5vimvH0Ub4Jz7xMwy9ps8HljvnNsAYGYvAmc55+4BGtxcYGZ9gELnXLGHcYHWyWxmW4HKwMMa79J+o7U+64B8INaLnPW10mc9BUig7h9amZm95ZyrDeXMgdeZCcw0s1nA817lrfd+rfFZG/B/wGzn3GJvE7f6dzrompOfujX3dGApTdyS1CZKohG9gOx6j7cCEw6yzFTgSc8SHVxzM78G/MXMjgY+8TLYQTQrt5mdA5wEpAJ/9TZao5qV2Tn3XwBmdgWw28uCOIDmfs5TqNu8EAu85WmyA2vu9/pnwAlAipkNdM497GW4RjT3s+4M/A4YY2a/CpSJnxrL/xDwVzM7jSZeuqMtl0SzOefu9DtDczjnSqkrtrDinHuNuoILO865p/zO0FTOuY+Aj3yO0WzOuYeo+2UWNpxze6jbhxLSnHN7gSubs0yb2HHdiG1A73qP0wPTQlk4ZobwzK3MwROOucMxc32tlr8tl8RCYJCZ9TOzGOp2Os70OdPBhGNmCM/cyhw84Zg7HDPX13r5g70n3qO9+y8AOXxzKOjUwPRTgS+o28v/X37nDPfM4ZpbmZW7rWUOZn5d4E9ERBrVljc3iYhIC6kkRESkUSoJERFplEpCREQapZIQEZFGqSRERKRRKglpE8ysJMjv1ypjjljd2BqFZrbUzNaa2R+bsMzZZja0Nd5f5GBUEiINMLMDXtfMOXdkK77dp8650cAY4HQzO9jYD2dTdzVaEc+pJKTNMrMBZva2mS2yupHwBgemn2FmC8xsiZm9b2bdAtPvMrNnzewz4NnA4yfM7CMz22BmN9R77ZLAf6cEnn8lsCbwXOBS15jZqYFpi8zsITN780B5nXNl1F3CuVdg+R+b2UIzW2Zmr5pZvJkdSd0YEfcF1j4GNPZzirQGlYS0ZdOBnznnxgG3An8PTJ8DTHTOjQFeBG6rt8xQ4ATn3EWBx4Opu6z5eOBOM4tu4H3GADcFlu0PTDazOOAR4JTA+6cdLKyZdQQG8c1l319zzh3hnBsFrKHucgtzqbsGzy+cc6Odc18d4OcUaTFdKlzaJDNLBI4EXg78YQ/fDHCUDswwsx7Ujdq1sd6iMwN/0e8zyzlXAVSY2S7qRtPbf8jVz51zWwPvuxTIoG541g3OuX2v/QIwrZG4R5vZMuoK4s/OuR2B6cPN7H+pG3cjEXinmT+nSIupJKStigAKAtv69/cX4H7n3MzAwDx31Xtu737zVtS7X0PD/2aaMs+BfOqcO93M+gHzzewl59xS4CngbOfcssBgR1MaWPZAP6dIi2lzk7RJzrkiYKOZnQd1Q2Ka2ajA0yl8c239yz2KsA7oX29YyQsOtkBgreP/gF8GJiUBOYFNXJfUm7U48NzBfk6RFlNJSFsRb2Zb691upu4X69TAppxV1I3xC3VrDi+b2SJgtxdhApusrgPeDrxPMVDYhEUfBo4JlMv/AAuAz4C19eZ5EfhFYMf7ABr/OUVaTJcKF/GImSU650oCRzv9DfjSOfeA37lEmkNrEiLe+XFgR/Yq6jZxPeJzHpFm05qEiIg0SmsSIiLSKJWEiIg0SiUhIiKNUkmIiEijVBIiItIolYSIiDTq/wGblDzJv+C2bAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.333751</td>\n",
       "      <td>0.282139</td>\n",
       "      <td>0.931280</td>\n",
       "      <td>03:27</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.582499</td>\n",
       "      <td>0.896619</td>\n",
       "      <td>0.761517</td>\n",
       "      <td>04:05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.384746</td>\n",
       "      <td>0.459956</td>\n",
       "      <td>0.869941</td>\n",
       "      <td>04:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.165085</td>\n",
       "      <td>0.246784</td>\n",
       "      <td>0.933571</td>\n",
       "      <td>04:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if FINE_TUNE:\n",
    "  learn.fine_tune(3, 1e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=3.311311302240938e-05, lr_steep=1.0964781722577754e-06)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3zdddn/8deV2aYjbdO9J3TS1g5ZRTZlCNzKFAUUQUREbxVF3Kj3/fNWqzfIrVSWLAGxQoHKEBltodBBd2mbpnQ3q21G0+zr98c5adP0JDlJzmryfj4eeZDznVe+nJ7rfLa5OyIiIg0lxTsAERFJTEoQIiISkhKEiIiEpAQhIiIhKUGIiEhIShAiIhJSSrwDiJTevXv78OHD4x2GiMhxZfny5QXu3ifUvnaTIIYPH86yZcviHYaIyHHFzLY1tk9VTCIiEpIShIiIhKQEISIiISlBiIhISEoQIiISkhKEiIiEpAQRJ+7O2l1FaLp1EUlUUU0QZjbbzDaaWbaZ3RVi/xlmtsLMqs3sihD7u5vZTjP7QzTjjIcPdxzgkvsW8e+P8uIdiohISFFLEGaWDNwPXAiMB641s/ENDtsO3Ag81chlfg68E60Y42l7YRkAL6/ZE+dIRERCi2YJYiaQ7e457l4JPA1cVv8Ad//Y3VcDtQ1PNrNpQD/gtSjGGDd5JeUAvLEhj+qaY/58EZG4i2aCGATsqPd6Z3Bbs8wsCfgt8J1mjrvFzJaZ2bL8/PxWBxoPucUVABQdquKDrfviHI2IyLEStZH6NmCBu+9s6iB3n+vu0919ep8+IeeaSli5xeX0796JTqlJvLpub7zDERE5RjQTxC5gSL3Xg4PbwnEKcLuZfQz8BrjezP5fZMOLr7ySCoZmZTBrTB9eW5+r3kwiknCimSCWAmPMbISZpQHXAPPDOdHdr3P3oe4+nEA102PufkwvqONZXnE5/bp34oIJ/dlTVM7qnUXxDklE5ChRSxDuXg3cDrwKbACedfd1ZnaPmV0KYGYzzGwncCXwgJmti1Y8icTdyS2uoG+3dM4d15fkJFM1k4gknKiuB+HuC4AFDbb9uN7vSwlUPTV1jUeBR6MQXtyUVlRzqKqGft3T6ZGRxidH9OLVdXv57uyx8Q5NROSwRG2kbtfqejD1694JgAsm9GdL/kGy80rjGZaIyFGUIOKgbgxEn27pAJw3vh8Ar61XNZOIJA4liDjIa1CCGNijMycNzuTVdbnxDEtE5ChKEHGQWxwoQfQNliAgUM20ascB9haVxyssEZGjKEHEQV5JBRlpyXRNP9JH4IIJqmYSkcSiBBEHucExEGZ2eNvovt0Y2aeLuruKSMJQgoiDvOKKww3U9V0woT9LcvZxoKwyDlGJiBxNCSIO8krKDzdQ13f++H7U1DpvbNAaESISf0oQMVY3irpfiBLE5ME96Nc9Xe0QIpIQlCBirCQ4irpv92MTRFKScf74/ry9KZ9DlTVxiE5E5AgliBhrOAaioXPH96O8qpZl27RGhIjElxJEjOUdHgMROkFMGdwDgDW7NLuriMSXEkSM5ZUEShChqpgAMjNSGdKrM2uVIEQkzpQgYqxuFHVjVUwAkwZlqgQhInGnBBFjucXHjqJuaOKgTHbsO6TxECISV0oQMdbYGIj6Jg3KBGDd7uJYhCQiEpISRIzlBVeSa8rEgYEEoWomEYknJYgYyy0pp28zJYieXdIY1KOzEoSIxJUSRAy5O3mNjKJuaNKgTPVkEpG4UoKIoZLDa1E3XYIAmDQ4k22FZRQdqopBZCJyvPr78p08s3R7VK6tBBFDdaOoGxsDUd/Eww3VKkWISOP+vDCHF1bujsq1lSBiqLlR1PVNHNgdQNVMItKo/JIKPtpbwuljekfl+koQMZRbEkwQYZQgsrqmMzCzE2t2Nd/V1d3bHJuIHH8WZxcAMGt0n6hcXwkihpqbqK+hiWE0VC/aXMBJP3uNNTtV0hDpaBZuLqBnRioTgjUOkaYEEUO5xRV0aWYUdX2TBmWyteAgJeWNN1Q/9cE2Ssqr+c9nV1JepSnCRToKd2dRdj6nju5NUpI1f0IrKEHEUDhjIOqb2MyI6uLyKv61IY8pQ3qQnVfKb17dGJE4RSTxZeeVkltcwazR0Wl/ACWImMoPYxR1fXUJorFqplfW7KWyupafXjqBL5w8jIcWb+W9LYURiVVEEtvCzYH2h2g1UIMSREzlhjEPU319uqXTv3unRkdUP79yF8OzMpg8OJPvXzSWYb0y+M7fVjVZJSUi7cOi7AJG9O7C4J4ZUbuHEkSMBNaiLm9RCQICpYhQCWJvUTnv5RRy+dRBmBkZaSn89qop7Ck6xM9fWh+psEUkAVVW17Ikp5DTo1i9BEoQMVNSUU15VW2LShBwpKG6tKL6qO3zV+3CHS6fMujwtmnDenLrp0bx7LKd/Gt9bkTiFpHE8+H2/ZRV1kS1egmUIGLm8CC5MMZA1DdxUHfcYX2DhurnP9zN5CE9GN67y1Hbv3nuCYwb0J275q2msLSibUGLSEJalF1AcpJxyqisqN5HCSJGDk+zEcYo6vrq1oaoX820KbeE9XuK+Y8pA485Pi0liTlXTab4UDU/+MfaNg+im7diJ9c//MHhATkiEn8LNxcweXAm3TulRvU+ShAxUjeKul8LSxB9u3eib7f0o3oyPf/hLpKTjEsmH5sgAMYN6M43zh3DK+v2tnrKcHfn/jez+dazq1iSU8h1D77P9Q9/cExJRkRiq6isitU7D3D6mOiMnq5PCSJGcg9P1NeyEgQcvUZ1ba3zwsrdzBrTm95dG082131yKClJxoI1e1t8v5pa58cvrOPXr27k8ikDWf7Dc/nBReNYteMAF9+3kG89s5Kd+8tafF0Rabv3cgqodZgV5fYHUIKImbwWjqKub8KgTLbkl1JWWc2ybfvZdeDQUY3TofTISOOUUVn8c+2eFlUzlVfV8LUnV/D4km3ccsZI5lw1hW6dUrn5jJG8c+dZ3HLGSF5as4ezf/M2c17f1OK/RUTaZuHmArqmpzBlSI+o3yuqCcLMZpvZRjPLNrO7Quw/w8xWmFm1mV1Rb/sUM3vPzNaZ2WozuzqaccZCS8dA1DdpUObhhurnV+6ic2oy543v1+x5F00awLbCMtbvCa9aqKisiusf/oBX1u3lhxeP4+6Lxh01hD8zI5XvXziOt75zJueM68u9b2zWdOQiMbYou4CTR/YiNTn63++jdgczSwbuBy4ExgPXmtn4BodtB24EnmqwvQy43t0nALOB35tZ9NNlFOUVl9OnhWMg6tQ1VK/Yvp+XV+/hggn96BJGSeT88f1IMvhnGNVMeSXlXPnAu3y4fT/3XjuVL88a2eixA3t05r8/M4m0lCSeXboj/D9ERNpkx74ythWWcVqUxz/UiWYKmglku3uOu1cCTwOX1T/A3T9299VAbYPtm9x9c/D33UAeEP0WmSjKK6lodQmiX/d0endN56FFWyk6VMVlU5uuXqqT1TWdk0dmsWBN89VMc17bxMeFZfzlizO5tJHG7/p6ZKQxe0J//vHhLk0SKBIjddNrxKL9AaKbIAYB9b9e7gxuaxEzmwmkAVtC7LvFzJaZ2bL8/PxWBxptdaOoW9qDqY6ZMWlQd3KLK8jqktaiybkunDSAnIKDbMotbfSYgtIK5n24iyumDebUFlz7mhlDKC6v5pW1LW8IF5GWW5SdT//unRjVp2tM7pfQjdRmNgB4HPiiu9c23O/uc919urtP79MncQsYxeWBUdQtHQNRX93EfZ+ePJCUFtQ9XjChH2awYM2eRo95/L1tVFbXctPpI1oU08kjsxjaK4Ono7QerogcUVPrLM4u5PQxvTGLzvTeDUUzQewChtR7PTi4LSxm1h14GfiBuy+JcGwxld+CleQa88kRWSQZXDFtcIvO69utEzOG9+Kfa0MniPKqGh5fso1zx/Vt8beSpCTj6hlDWJKzj48LDrboXBFpmbW7iig6VBWz6iWIboJYCowxsxFmlgZcA8wP58Tg8f8AHnP356IYY0zktnAluVBOH9ObD35w7uGSREtcNLE/m3JLyc4rOWbfvBW72HewsslG6aZcMW0wSQbPLlNjtUg0LQrOZhCrBmqIYoJw92rgduBVYAPwrLuvM7N7zOxSADObYWY7gSuBB8xsXfD0q4AzgBvNbGXwZ0q0Yo0Ed+eVtXs5VHlsg21eXQmilb2Y6jQ1MK4psycOAI7tzVRb6zy4KIdJgzL55Iherbp2v+6dOHtsX/62fCfVNcfUAopIBFRU1/Dc8p1MGpTZ6s+B1ohqG4S7L3D3E9x9lLv/Mrjtx+4+P/j7Uncf7O5d3D0r2K0Vd3/C3VPdfUq9n5XRjLWt1u4q5tYnlvO9v68+psdQW0ZRR0L/zE5MG9aTfzZoTH5zYx45+Qf58qwRbarTvHrGUPJLKnhzY+J2FBA5nj24cCtbCw5y5wUnxvS+Cd1IfTzJKQj0Epq/ajePvbftqH25xeWtHkUdKRdO7M/6PcVHtRX8eWEOAzI7cdGkAW269lkn9qFvt3SeUWP1YT97cR3/tWDDMdO0i7TUrgOHuO/fm5k9oT9nnBDbzjhKEBGyvTAwN9GsMb35xcvrWb5t/+F9bRkDESmzJ/YHOFyKWLuriCU5+/jiacPbPCIzJTmJK6YN5t8f5bG3qLzNsR7vtheW8cjij5n7Tg7nzXmbV9epG7C03i+CC4D96NMNxxlHnxJEhGzbV0bfbun84dpP0D+zE197csXh9Rjyisvb1IMpEgb3DCxNWteb6cGFOXRNT+GamUMjcv2rpg+h1uHvK3ZG5HrHsxdX7wbgD5+bSmbnVL7y+HJufmwZuw8cinNkcrx5Z1M+/1y7l6+fPYZBPTrH/P5KEBGyvbCMYVkZZGak8sfrprGvrJI7nv6Qmlont7iiTWMgIuXCSQNYvbOIpR/v46XVe7h6xpCIzSc/vHcXThmZxTNLd1Bb27Y1KI53L67azSeG9uCSkwby4tdP5+6LxrJocwHnznmbBxfmqDFfwlJRXcNP569jRO8ufHlWy8YoRYoSRIRs23eQob0Cq7tNHJTJLy6byOLsQn73+ibySlo/ijqSLgxWM33tyRXUuvPF04ZH9PrXzBzC9n1lLMkpjOh1jyebckv4aG/J4elKUpOTuOWMUbz2n2dw8sgsfvHyBm57cgU1HTyJSvMeWrSVnIKD/OTT40lPSY5LDPFrNW1HyqtqyC2uYFhWxuFtV80YwvJt+/nDm9lA28ZARMqwrC5MGNiddbuLufikAQzumdH8SS1wwYT+ZHZO5f63svlobwm5xeXsLS5nb1E5ucXlDM3qwtwvTKNTanze7LHw4qrdJBlcdNLRDf9DemXw0A3TeXDhVn65YAO/euUj7r5oXJyilES3+8Ah7nsjmwsm9OPME/vGLQ6VICJg+75AA3X9BAHws8smMGFgd4BWz+QaaRcHP7hubuXAuKZ0Sk3mimmDWZxdyD0vreeRdz9mxfb91LpzQr9uvLMpn1+/ujHi900U7s6Lq3ZzyqiskFWKZsaXZ43g+lOGMfedHJ56X72+JLRfvrwBx/nRJbFvmK5PJYgIqOvBNLTX0QmiU2oyf/r8NH724npmDG/dQLRIu+n0EZw8Mitqi418d/aJXD1jCL27ptMzI/Wo8RU/en4tDy/eyrnj+kV9sfV4WLurmI8Ly7j1U6MaPcbM+PEl49m+r4wfvbCWIb06MysGS0fK8WPR5gJeXrOHb593QsRL+S2lEkQEbNsXOkFAoGrhwRumMzAOPRBCSU9J5hNDe0b1+if060avLmnHDL77/kVjGdYrg+/8bRUl5VVRiyFe5q/aRWqyHe5S3JiU5CTuu3YqY/p25bYnVrA599gpUKRj2ltUzrf/tpLhWRncfEbkS/ktpQQRAdsLD9I1PYVeXdLiHUpCy0hL4bdXTWZP0SF+8dKGeIcTUbW1zkur93DGmD70yGj+fdCtUyoP3TiDTmnJfPHRpRQEu0RLx3WosoabH1tGaXk1f0qQtjpVMUXAtn1lDO2VEbMpeI9n04b14iufGsUf39rCBRP7cfbY5pdODcfzH+7iqQ+2U1Prx/ycNbYv35t9YlT//yzbtp89ReXcdeHYsM8Z1KMzD14/navnvsfNjy3jrzefnBAfCu7OB1v3MXlIj4SIpyOorXW+/beVrN1dxIPXT2ds/+7xDglQCSIi6sZASHi+ee4Yxvbvxvf+vob9ByvbfL28knLumrea3OJyOqcm071zKn26pTOwR2d6ZKTyp7e38GSUG4RfXLWbTqlJnDuuZQlv8pAe/P7qKazccYDrH/og7iPR3Z17XlrP1XOXcO2flxwe7CnR9fs3NrNgzV7uvnAc57TwPRRNShBtVFPr7NhfxlAliLClpyQz56opHCir5IcvrG3z9f7vzS1U1TiPfnEmT3z5kzz2pZk8fOMMHrxhOn+9+WTOOrEP97y4njU7iyIQ/bGqa2pZsGYP54wLb63whmZPHMDvr57C2t1FXHTvQt7amBeFKJtXW+v86IW1PLL4Yy6Y0I/1u4v5zB/fZWsr1/r4cPt+Zv/+He5/M5uySs1J1ZgXVu7i3jc2c9X0wXEbENcYJYg22lN0iKoaZ1hwkJyEZ/zA7nzz3BN4efUe5q/a3err7D5wiKfe384VnxjMiN7H/j9ISjLmXDWFPt3Sue2p5RSVRb5x/N0thRQerOTTJzW/lndjLpsyiPm3n07fbunc+MhSfvXKR1TFcMR1Ta3z/XlreGLJdm791Cj+9PlpPHXzyRQfquKzf3z3qLnFwrFzfxk3P7aMnfsP8etXN/KpX7/F40u2xfRvOh58uH0/dz63mpkjevGLyyclXDW1EkQb1XVxVRVTy33ljJFMHdqD7zy7ivve2Exldcs/PO7792Yc5+vnjG70mJ5d0vjD56YGe4isOmY69rZ6cdVuuqWncOaJbeuuOrpvV57/2mlcO3Mof3xrC9fMXRKT+Ztqap07/7aKZ5bt4I6zRx9ur5k2rCfzbjuNbp1S+Nyfl4S99nhJeRU3PbqMiupanv/aaTx36ykMz8rgR8+v5bw5b/Piqt0dfjoWCHy5ueXx5fTrns6fPj+NtJTE+zhOvIiOM9ub6OIqTUtJTuLB66dz/oR+/Pb1TXz6vkWs2B7+N9WPCw7y7LKdfG7m0Gb7i08d2pO7LxrHvzbkMvednLaGflhFdQ2vrNvL+RP6R6RBt1NqMv/9mUnce+1UPtpTzEX3LuTtTdFbZ6OqppZvPrOSeR/u4lvnncC3zj+6MX9E7y7M++qpjBvQna8+uZxHFm9t8nrVNbXc/tSHbMkv5U+fn8bovl2ZPrwXz37lFB66YTrpKcl8/a8fcun9i1i7KzpVfseDyupavvrkCg5V1vDwDTMStgekEkQbbdtXRkqSJcw4h+NNVtd0/vC5T/DQDdMpLg9UZ/x0/rqw1lH43zc2k5psfO2sxksP9d146nAunjSA/3l1Ix9s3dfW0AF4e2M+JeXVfHpy29bUaOjSyQN56Y5Z9O/eiS8+8gF/effjiF4fAsnhjr9+yIurdnPXhWO545wxIY/L6prOX28+mfPG9eNnL67nCw+9z7rdoT/cf/7Set7elM/PL5941NKYZsY54/qx4BuzmHPVZPKKK7j8/sX84d+bO+Tkhb95bSOrdhzg11ecxJh+3eIdTqMs0sXteJk+fbovW7Ys5vf92pMrWLe7iLfuPCvm925vSiuq+fUrH/HYkm0M6N6JX/7HJM4aG3oems25JZz/+3e4ZdZIvt+COY1Kyqu49A+LKaus5uU7ZoW1fKO788LK3Ud94637kv1eTiG7D5Tz/t3ntHldjVAOVlTzjadX8q8NuVx/yjB+fMl4UiJwn5pa5z+fWcn8Vbv54cXjwlqTvKbWeWTxVu77dzbF5VX8x5RBfOv8I6N9H128lZ++uJ6bZ43gBxc3PUXEgbJKfvj8Wl5avYdPDO3BnKumMDxEG1J79ObGPL74yFI+f/JQfnH5pHiHg5ktd/fpIfcpQbTNJfctpFeXdB770syY37u9Wr5tP9+ft5pNuaVcO3MoP7x43DG9g776xHIWbi5g4XfPomcLi+cb9hRz+f2LGdWnK/97zZQmv8GVVVbz/XlreGFloBtrcjAz1P2rcYdbzhjJf553QotiaImaWud/XvmIB97JYdaY3tx/3SfaNE27e6BB+umlO/ju7BO57czwSmB1ig5V8ce3tvDI4q24ww2nDmPioEz+85mVnD22Hw98YRrJSeE1ts5ftZsf/mMNVTXODy4ex3WfHJpwDbWRlFtczoX/u5C+3dJ5/munJcQ4EyWIKHF3TvrZa1w+ZRA/v3xiTO/d3lVU1zDn9U3MfSeHYb0y+N3VU5ganCJk7a4iLrlvEXecM4ZvtfKD+c2P8vj231ZRWlHNneefyJdOH3HMh1p2XilffWI5W/JL+fb5J/LVT40iKcwPvmh4Zul2fvCPtYzo3YWHbpjRqq7V7s4vXt7AQ4u28rWzRnHnBeEP7Gto94FDzHl9E39fsRN3mDCwO89+5ZQWd/XdW1TOnc+tYuHmAmaN6c0XTh7G6WN6k5HWvsbx1tQ6n3/wfVbuOMCLXz+N0X0To2pJCSJK9h+sZOrPXw+7iC4ttySnkG8/u4q9xeXcftZobj97NLc8towV2w+w8HtntembdEFpBXfPW8Nr63OZMbwnv7lyMsOyAtUcL6/ew3efW0V6ajL3XjOV08f0buZqsfHelkJufWI5yUnGbWeOYtaYPpzQr2vY37rnvL6Je9/YzI2nDucnnx4fkW/rH+0tZt6KXXzptBH0z2zdtPbuzhNLtvE/r26kpLyatJQkTh2VxTnj+nHO2L7too3v3jc2M+f1TfzPFSdx1fQh8Q7nMCWIKFm54wCX37+YuV+YxvkTmp6gTVqvuLyKn76wjnkf7mJM365szittVdVIKO7OPz7cxU/mr6O6xrn7orF8XFjGQ4u2MnVoD/7vuk8wIDOxPpy2Fhzk639dwdpdxUBgKvnTR/fmtNG9OX1070Y/pOe+s4X/WvARV04bzK8+e1JcS0ONqaqpZenWffxrQx5vfJTLtmA38gkDu/O5Tw7lM1MH0zktctUy7s6GPSUcKKvklFFZUaveej+nkGv/vIRLJw/kd1dPSahqNCWIKHlh5S6+8fRKXv3mGZzYPzGKi+3Zy6v3cPc/1pCWksTbd54Z0SqIPUWH+O5zq1m4uQAI9Hi6+6JxCdk3vc6uA4dYvLmAhdkFvJtdQGFw2pIeGakM7ZXBkF4Zgf/2zKCgtII5r2/i4pMGcO81U8NuI4gnd2dL/kH+/VEuL6zczbrdxWR2TuWamUO4/pThrV6juS4pLFizh5fX7Dk8UvzzJw/lp5+e0KpOAO7O1oKDbMk/SGqykZaSRHpKEmnJyTjOLY8tp1NqEi/dMYuurRhtH01KEFFy3xub+e3rm9hwz+yIfquRxu0/WElFdW2rqzKa4u7MW7GLrp1SuOA4KxHW1job9hbz3pZCcgoOsmNfGTv2lbHrQGCkP8DZY/sm7ICs5rg7y7bt55HFW3ll7V7MjAsm9OPK6UNIT0mioqqWiuoayqtqKa+qaXTE9p6icv65di9bCw6SZHDKqCwunjSQbYUHeeCdHM4Z25f7Pjc1rC8fhyprWJJTyFsb83hzY/7hMVGhpCUnMe+2U5k4KLPVzyBamkoQiZXKjjPb95XRt1u6kkMMtbTHUkuYGZ+dNjhq14+mpCRjwsBMJgw8+gOoptbZW1xOYWkF4wd0j0gX2XgwM2YM78WM4b3YdeAQj733MU9/sIMFa8Ib3V2nLincPGskF0zoR1a9bs6De3bmJ/PXce3cJTx4w4yQq0CWV9Xw8uo9vLh6N+9tKaSiupZOqUmcNqo3N88awUmDe1Bd61RW11JZUxv4b3Uto/p2SZgZWltCJYg2uOqB93B3/nbrqTG9r4gEuiB/uP0AyUlGekoSnVKTD/83NTmJUNX8nVOTm+xl9fr6XL7+1xX06ZbOo1+cyag+XYFAu8+TS7bx3IqdHCirYlhWBmeP7ctZJ/Zl5oheCdFdtbVUgoiS7YVlnDq6/S2dKXI8yEhLOWq0diScN74fT99yCjc9upTP/vFdvnHOGP61IZfF2YWkJBkXTOjPdScP5ZSR0WvQTiRKEK1UXlXD3uJyzeIq0s5MGdKDebedyo2PLOVnL65nUI/OfOf8E7hqxhD6dot821ciU4JopR37NIurSHs1LKsLz992Guv3FDNzRK/jotdXNChBtFJd/2wtFCTSPmVmpHLKqI5dhXx8dmlIANvqShCa5ltE2ikliFbaXniQrukpCTuPu4hIWylBtNK2fWUM7ZXRIXoyiEjHpATRStv3lamBWkTaNSWIVqipdXbuO6QGahFp16KaIMxstpltNLNsM7srxP4zzGyFmVWb2RUN9t1gZpuDPzdEM86W2ltcTmVNrdahFpF2LWoJwsySgfuBC4HxwLVm1nAdwu3AjcBTDc7tBfwE+CQwE/iJmfWMVqwtta0wMPujBsmJSHsWzRLETCDb3XPcvRJ4Gris/gHu/rG7rwYaTr14AfC6u+9z9/3A68DsKMbaItsLNUhORNq/sBKEmXUxs6Tg7yeY2aVm1txSXoOAHfVe7wxuC0dY55rZLWa2zMyW5efnh3npttu2r4yUJGNAFKacFhFJFOGWIN4BOpnZIOA14AvAo9EKKlzuPtfdp7v79D59+sTsvtsLyxjcs/NxO3WyiEg4wv2EM3cvAz4D/J+7XwlMaOacXUD9hVcHB7eFoy3nRt22fQcZmqX2BxFp38JOEGZ2CnAd8HJwW3MToC8FxpjZCDNLA64B5od5v1eB882sZ7Bx+vzgtoSwvbBMU2yISLsXboL4JvB94B/uvs7MRgJvNnWCu1cDtxP4YN8APBs89x4zuxTAzGaY2U7gSuABM1sXPHcf8HMCSWYpcE9wW9wdKKukuLxaDdQi0u6FNZuru78NvA0QbKwucPc7wjhvAbCgwbYf1/t9KYHqo1DnPgw8HE58sZQTXOB8mKqYRKSdC7cX01Nm1t3MugBrgfVmdmd0Q0tM2XmlAIzu2zXOkYiIRFe4VUzj3b0YuBz4JzCCQE+mDmdLXqbo7+0AAA8PSURBVClpyUkM6dk53qGIiERVuAkiNTju4XJgvrtXAR69sBJXdl4pI3p3URdXEWn3wv2UewD4GOgCvGNmw4DiaAWVyLbkl6p6SUQ6hLAShLvf6+6D3P0iD9gGnBXl2BJOeVUN2/eVMaqPGqhFpP0Lt5E608zm1E1rYWa/JVCa6FA+LjxIrcMolSBEpAMIt4rpYaAEuCr4Uww8Eq2gEpV6MIlIRxLWOAhglLt/tt7rn5nZymgElMiy80oxg1F9lCBEpP0LtwRxyMxOr3thZqcBh6ITUuLakn+QwT070ym1uVlGRESOf+GWIG4FHjOzzODr/UBCrfIWC9l5pSo9iEiHEW4vplXuPhk4CTjJ3acCZ0c1sgRTU+vk5JcyWglCRDqIFo32cvfi4IhqgG9FIZ6EtWv/ISqqa9VALSIdRluGA1vEojgOZOeXAOrBJCIdR1sSRIeaamNLXmAWV7VBiEhH0WQjtZmVEDoRGNChZqvLzislq0saPbukxTsUEZGYaDJBuHu3WAWS6LLzSzWCWkQ6FE1JGgZ3JztPk/SJSMeiBBGGgtJKig5VqYuriHQoShBh2JKvOZhEpONRggiDJukTkY5ICSIM2XmlZKQlMyCzU7xDERGJGSWIMGzJD8zBZNahxgaKSAenBBEG9WASkY5ICaIZpRXV7CkqV4IQkQ5HCaIZOcEeTJpiQ0Q6GiWIZhzpwdThluAWkQ5OCaIZ2XmlpCQZw7KUIESkY1GCaEZ2XinDsjJITdajEpGORZ96zdiSrx5MItIxKUE0oaqmlm2FZUoQItIhKUE0YVvhQaprXQlCRDokJYgm1PVgUhdXEemIlCCaoAQhIh2ZEkQTtuQfZGBmJ7qkN7nwnohIu6QE0YTsPC0zKiIdV1QThJnNNrONZpZtZneF2J9uZs8E979vZsOD21PN7C9mtsbMNpjZ96MZZyi1ta4uriLSoUUtQZhZMnA/cCEwHrjWzMY3OOwmYL+7jwZ+B/wquP1KIN3dJwHTgK/UJY9YySupoKyyhpFqfxCRDiqaJYiZQLa757h7JfA0cFmDYy4D/hL8/TngHAssuuBAFzNLAToDlUBxFGM9Rm5xOQADumuRIBHpmKKZIAYBO+q93hncFvIYd68GioAsAsniILAH2A78xt33NbyBmd1iZsvMbFl+fn5Egy8orQCgT7f0iF5XROR4kaiN1DOBGmAgMAL4tpmNbHiQu8919+nuPr1Pnz4RDSC/JJAgeitBiEgHFc0EsQsYUu/14OC2kMcEq5MygULgc8Ar7l7l7nnAYmB6FGM9xuEE0TUtlrcVEUkY0UwQS4ExZjbCzNKAa4D5DY6ZD9wQ/P0K4N/u7gSqlc4GMLMuwMnAR1GM9RgFpRVkdk4lPSU5lrcVEUkYUUsQwTaF24FXgQ3As+6+zszuMbNLg4c9BGSZWTbwLaCuK+z9QFczW0cg0Tzi7qujFWso+aUVKj2ISIcW1SHC7r4AWNBg24/r/V5OoEtrw/NKQ22PpfySCjVQi0iHlqiN1HFXUFpJn27q4ioiHZcSRCPyS1TFJCIdmxJECGWV1ZRWVKuKSUQ6NCWIEApKKgHo01UJQkQ6LiWIEPJLNUhOREQJIoS6QXIqQYhIR6YEEUJdCaKvShAi0oEpQYRQUFKBGfTqol5MItJxKUGEkF9aQa+MNFKS9XhEpOPSJ2AIGkUtIqIEEVJBaQW91UAtIh2cEkQIKkGIiChBHMPdlSBERFCCOEZpRTUV1bWah0lEOjwliAYOD5JTCUJEOjgliAaOjKLWVN8i0rEpQTRQUBqYqK93N1UxiUjHpgTRQH5JOaB5mERElCAayC+tIDnJ6JmhEoSIdGxKEA0UlFSS1SWNpCSLdygiInGlBNFAfqnGQIiIgBLEMQJrUStBiIgoQTRQoBKEiAigBHGU2lpXghARCVKCqKfoUBVVNa4qJhERlCCOUlCqaTZEROooQdRzZJoNJQgRESWIevIPlyA0SE5ERAmiHk3UJyJyhBJEPfmlFaQlJ9G9c0q8QxERiTsliHoCg+TSMNM0GyIiShD1FJRWqgeTiEiQEkQ9WotaROQIJYh6NA+TiMgRShBBNbXOvoMqQYiI1IlqgjCz2Wa20cyyzeyuEPvTzeyZ4P73zWx4vX0nmdl7ZrbOzNaYWVT7nu47WEmtaxS1iEidqCUIM0sG7gcuBMYD15rZ+AaH3QTsd/fRwO+AXwXPTQGeAG519wnAmUBVtGKFI2MgVMUkIhIQzRLETCDb3XPcvRJ4GriswTGXAX8J/v4ccI4F+pieD6x291UA7l7o7jVRjFXzMImINBDNBDEI2FHv9c7gtpDHuHs1UARkAScAbmavmtkKM/tuFOMENA+TiEhDiTpkOAU4HZgBlAFvmNlyd3+j/kFmdgtwC8DQoUPbdMO6eZh6qwQhIgJEtwSxCxhS7/Xg4LaQxwTbHTKBQgKljXfcvcDdy4AFwCca3sDd57r7dHef3qdPnzYFW1BSQefUZLqkJbfpOiIi7UU0E8RSYIyZjTCzNOAaYH6DY+YDNwR/vwL4t7s78CowycwygonjU8D6KMZKfnAlOU2zISISELUqJnevNrPbCXzYJwMPu/s6M7sHWObu84GHgMfNLBvYRyCJ4O77zWwOgSTjwAJ3fzlascKReZhERCQgqm0Q7r6AQPVQ/W0/rvd7OXBlI+c+QaCra0wUlFYwoneXWN1ORCThaSR1kOZhEhE5mhIEUFVTy/6yKg2SExGpRwkCKCytBDRITkSkPiUINEhORCQUJQggv7Qc0CA5EZH6lCCAgpJgFZNKECIihylBcGSaDbVBiIgcoQRBoA2iW3oKnVI1zYaISB0lCI5MsyEiIkcoQaC1qEVEQlGCIDCTq0oQIiJHU4JAVUwiIqF0+ARRXlVDSXm1ZnIVEWmgwyeI4vIqRvTuwqCeneMdiohIQknUJUdjpm+3Trz5nTPjHYaISMLp8CUIEREJTQlCRERCUoIQEZGQlCBERCQkJQgREQlJCUJEREJSghARkZCUIEREJCRz93jHEBFmlg9sq7cpEyhqweveQEEUQmt4n0ie09Rxje0Ltb0lzypaz6mx2CJxTrSeU8Ntek81vq0jvaea2p+Iz2qYu/cJucfd2+UPMLeFr5fFIo5IntPUcY3tC7W9Jc8qWs8pms8qWs8pxLPReyqM91CI59au3lPH67MK9dOeq5hebOHrWMURyXOaOq6xfaG2t/dnFa3n1HDb8f6cmjtO76nwjzken9Ux2k0VU1uZ2TJ3nx7vOBKdnlP49KzCo+cUvlg/q/ZcgmipufEO4Dih5xQ+Pavw6DmFL6bPSiUIEREJSSUIEREJSQlCRERCUoIQEZGQlCCaYWazzOxPZvagmb0b73gSmZklmdkvzew+M7sh3vEkMjM708wWBt9bZ8Y7nkRmZl3MbJmZXRLvWBKZmY0Lvp+eM7OvRuKa7TpBmNnDZpZnZmsbbJ9tZhvNLNvM7mrqGu6+0N1vBV4C/hLNeOMpEs8KuAwYDFQBO6MVa7xF6Fk5UAp0op0+qwg9J4DvAc9GJ8rEEKHPqg3Bz6qrgNMiEld77sVkZmcQ+Ef4mLtPDG5LBjYB5xH4h7kUuBZIBv67wSW+5O55wfOeBW5y95IYhR9TkXhWwZ/97v6AmT3n7lfEKv5YitCzKnD3WjPrB8xx9+tiFX+sROg5TQayCCTSAnd/KTbRx1akPqvM7FLgq8Dj7v5UW+NKaesFEpm7v2Nmwxtsnglku3sOgJk9DVzm7v8NhCzCmtlQoKi9JgeIzLMys51AZfBlTfSija9Iva+C9gPp0Ygz3iL0njoT6AKMBw6Z2QJ3r41m3PEQqfeUu88H5pvZy4ASRCsMAnbUe70T+GQz59wEPBK1iBJXS5/VPOA+M5sFvBPNwBJQi56VmX0GuADoAfwhuqEllBY9J3f/AYCZ3Uiw1BXV6BJLS99TZwKfIfCFY0EkAuiICaLF3P0n8Y7heODuZQSSqTTD3ecRSKgSBnd/NN4xJDp3fwt4K5LXbNeN1I3YBQyp93pwcJscS88qfHpW4dFzCl/cn1VHTBBLgTFmNsLM0oBrgPlxjilR6VmFT88qPHpO4Yv7s2rXCcLM/gq8B5xoZjvN7CZ3rwZuB14FNgDPuvu6eMaZCPSswqdnFR49p/Al6rNq191cRUSk9dp1CUJERFpPCUJEREJSghARkZCUIEREJCQlCBERCUkJQkREQlKCkHbNzEpjfL+IrBkSXC+iyMxWmtlHZvabMM653MzGR+L+IqAEIdIiZtbk/GXufmoEb7fQ3acAU4FLzKy5Of4vJzDrqUhEKEFIh2Nmo8zsFTNbHlzVbWxw+6fN7H0z+9DM/hVcqwEz+6mZPW5mi4HHg68fNrO3zCzHzO6od+3S4H/PDO5/LlgCeNLMLLjvouC25WZ2r5k1ucaBux8CVhKY3RMzu9nMlprZKjP7u5llmNmpwKXAr4OljlGN/Z0i4VKCkI5oLvB1d58GfAf4v+D2RcDJ7j4VeBr4br1zxgPnuvu1wddjCUzXPRP4iZmlhrjPVOCbwXNHAqeZWSfgAeDC4P37NBesmfUExnBkCvV57j7D3ScTmILhJnd/l8A8PXe6+xR339LE3ykSFk33LR2KmXUFTgX+FvxCD0cW7BkMPGNmA4A0YGu9U+cHv8nXedndK4AKM8sD+nHs0qEfuPvO4H1XAsMJrBqW4+511/4rcEsj4c4ys1UEksPv3X1vcPtEM/sFgbUkuhKYq6clf6dIWJQgpKNJAg4E6/Ybuo/A8p/zg4uv/LTevoMNjq2o93sNof8thXNMUxa6+yVmNgJYYmbPuvtK4FHgcndfFVxI58wQ5zb1d4qERVVM0qG4ezGw1cyuBLCAycHdmRyZb/+GKIWwERhZb3nJq5s7IVja+H/A94KbugF7gtVa9deyLgnua+7vFAmLEoS0dxnB6ZPrfr5F4EP1pmD1zTrgsuCxPyVQJbMcKIhGMMFqqtuAV4L3KQGKwjj1T8AZwcTyI+B9YDHwUb1jngbuDDayj6Lxv1MkLJruWyTGzKyru5cGezXdD2x299/FOy6RhlSCEIm9m4ON1usIVGs9EOd4REJSCUJEREJSCUJEREJSghARkZCUIEREJCQlCBERCUkJQkREQlKCEBGRkP4/amCk8FmNkJAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "if FINE_TUNE:\n",
    "  learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.121878</td>\n",
       "      <td>0.230724</td>\n",
       "      <td>0.937134</td>\n",
       "      <td>04:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.076736</td>\n",
       "      <td>0.243553</td>\n",
       "      <td>0.934334</td>\n",
       "      <td>04:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.055963</td>\n",
       "      <td>0.231417</td>\n",
       "      <td>0.938916</td>\n",
       "      <td>04:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if FINE_TUNE:\n",
    "  learn.fit_one_cycle(3, slice(1e-5, 5e-4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The performance seemse to be much lower than the resnet34."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 2a - LabelSmoothing\n",
    "\n",
    "Change the loss function with label smoothing applied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h2 id=\"LabelSmoothingCrossEntropy\" class=\"doc_header\"><code>class</code> <code>LabelSmoothingCrossEntropy</code><a href=\"https://github.com/fastai/fastai/tree/master/fastai/losses.py#L79\" class=\"source_link\" style=\"float:right\">[source]</a></h2><blockquote><p><code>LabelSmoothingCrossEntropy</code>(<strong><code>eps</code></strong>:<code>float</code>=<em><code>0.1</code></em>, <strong><code>reduction</code></strong>=<em><code>'mean'</code></em>) :: <code>Module</code></p>\n",
       "</blockquote>\n",
       "<p>Same as <code>nn.Module</code>, but no need for subclasses to call <code>super().__init__</code></p>\n",
       "<p><a href=\"https://docs.fast.ai/losses#LabelSmoothingCrossEntropy\" target=\"_blank\" rel=\"noreferrer noopener\">Show in docs</a></p>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc(LabelSmoothingCrossEntropyFlat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From \n",
    "https://rickwierenga.com/blog/machine%20learning/LabelSmoothing.html\n",
    "\n",
    "> Label smoothing is a variation on one hot encoding where the negative labels have a value slightly higher than 0, say 0.1  and the positive labels have a value slightly lower than 1, say 0.9\n",
    "\n",
    "\n",
    "```\n",
    "class LabelSmoothingCrossEntropy(Module):\n",
    "    y_int = True\n",
    "    def __init__(self, eps:float=0.1, reduction='mean'): self.eps,self.reduction = eps,reduction\n",
    "\n",
    "    def forward(self, output, target):\n",
    "        c = output.size()[-1]\n",
    "        log_preds = F.log_softmax(output, dim=-1)\n",
    "        if self.reduction=='sum': loss = -log_preds.sum()\n",
    "        else:\n",
    "            loss = -log_preds.sum(dim=-1) #We divide by that size at the return line so sum and not mean\n",
    "            if self.reduction=='mean':  loss = loss.mean()\n",
    "        return loss*self.eps/c + (1-self.eps) * F.nll_loss(log_preds, target.long(), reduction=self.reduction)\n",
    "\n",
    "    def activation(self, out): return F.softmax(out, dim=-1)\n",
    "    def decodes(self, out):    return out.argmax(dim=-1)\n",
    "```\n",
    "\n",
    "```\n",
    "@delegates()\n",
    "class LabelSmoothingCrossEntropyFlat(BaseLoss):\n",
    "    \"Same as `LabelSmoothingCrossEntropy`, but flattens input and target.\"\n",
    "    y_int = True\n",
    "    @use_kwargs_dict(keep=True, eps=0.1, reduction='mean')\n",
    "    def __init__(self, *args, axis=-1, **kwargs): super().__init__(LabelSmoothingCrossEntropy, *args, axis=axis, **kwargs)\n",
    "    def activation(self, out): return F.softmax(out, dim=-1)\n",
    "    def decodes(self, out):    return out.argmax(dim=-1)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = cnn_learner(dls, arch=resnet34, metrics=accuracy, loss_func=LabelSmoothingCrossEntropyFlat())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of LabelSmoothingCrossEntropy()"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.loss_func"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.172003</td>\n",
       "      <td>0.827896</td>\n",
       "      <td>0.927971</td>\n",
       "      <td>02:48</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.874614</td>\n",
       "      <td>0.734942</td>\n",
       "      <td>0.938916</td>\n",
       "      <td>03:38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if FINE_TUNE:\n",
    "  learn.fine_tune(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 3 - Mixup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Source: https://towardsdatascience.com/2-reasons-to-use-mixup-when-training-yor-deep-learning-models-58728f15c559\n",
    "\n",
    "Mixup is a form of regularization technique. The idea is instead of passing raw images to your model, you pass a blend of 2 images from your training set. \n",
    "\n",
    "newImage = alpha * image1 + (1-alpha) * image2\n",
    "\n",
    "But "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h2 id=\"MixUp\" class=\"doc_header\"><code>class</code> <code>MixUp</code><a href=\"https://github.com/fastai/fastai/tree/master/fastai/callback/mixup.py#L18\" class=\"source_link\" style=\"float:right\">[source]</a></h2><blockquote><p><code>MixUp</code>(<strong><code>alpha</code></strong>=<em><code>0.4</code></em>) :: <code>Callback</code></p>\n",
       "</blockquote>\n",
       "<p>Basic class handling tweaks of the training loop by changing a <code>Learner</code> in various events</p>\n",
       "<p><a href=\"https://docs.fast.ai/callback.mixup#MixUp\" target=\"_blank\" rel=\"noreferrer noopener\">Show in docs</a></p>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc(MixUp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = cnn_learner(dls, arch=resnet34, metrics=accuracy, loss_func=CrossEntropyLossFlat())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of CrossEntropyLoss()"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.loss_func"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.421824</td>\n",
       "      <td>0.240442</td>\n",
       "      <td>0.930008</td>\n",
       "      <td>02:49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.046022</td>\n",
       "      <td>0.199377</td>\n",
       "      <td>0.942225</td>\n",
       "      <td>03:38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fine_tune(1, cbs=MixUp())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 4 - Test Time Augmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " This is a type of data augmentation. By selecting a number of areas to crop from the original rectangular image, pass each of them through our model, and take the **maximum** or **average** of the predictions. \n",
    "- `preds, _ = learn.tta(dl=test_dl, beta=0)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"Learner.tta\" class=\"doc_header\"><code>Learner.tta</code><a href=\"https://github.com/fastai/fastai/tree/master/fastai/learner.py#L559\" class=\"source_link\" style=\"float:right\">[source]</a></h4>\n",
       "\n",
       "> <code>Learner.tta</code>(**`ds_idx`**=*`1`*, **`dl`**=*`None`*, **`n`**=*`4`*, **`item_tfms`**=*`None`*, **`batch_tfms`**=*`None`*, **`beta`**=*`0.25`*, **`use_max`**=*`False`*)\n",
       "\n",
       "Return predictions on the `ds_idx` dataset or `dl` using Test Time Augmentation"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(Learner.tta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='0' class='' max='1' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      \n",
       "    </div>\n",
       "    \n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tta_preds = learn.tta(ds_idx=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds = learn.get_preds(ds_idx = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4 id=\"accuracy\" class=\"doc_header\"><code>accuracy</code><a href=\"https://github.com/fastai/fastai/tree/master/fastai/metrics.py#L99\" class=\"source_link\" style=\"float:right\">[source]</a></h4><blockquote><p><code>accuracy</code>(<strong><code>inp</code></strong>, <strong><code>targ</code></strong>, <strong><code>axis</code></strong>=<em><code>-1</code></em>)</p>\n",
       "</blockquote>\n",
       "<p>Compute accuracy with <code>targ</code> when <code>pred</code> is bs * n_classes</p>\n",
       "<p><a href=\"https://docs.fast.ai/metrics#accuracy\" target=\"_blank\" rel=\"noreferrer noopener\">Show in docs</a></p>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc(accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 3929, 3929)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(preds), len(preds[0]), len(preds[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorImage([5.8873e-04, 4.8669e-04, 3.7679e-04, 9.7222e-04, 8.8172e-04, 2.8819e-03,\n",
       "         4.1782e-04, 3.6273e-04, 9.0794e-04, 9.9212e-01]), TensorCategory(9))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds[0][0], preds[1][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorImage(0.9422)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(*preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorImage(0.9443)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(*tta_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 5 - Native Mixed Precision\n",
    "\n",
    "- Native Mixed Precision `to_native_fp16()` during training and back to `to_native_fp32` before export"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading: \"https://download.pytorch.org/models/resnet34-333f7ec4.pth\" to /root/.cache/torch/hub/checkpoints/resnet34-333f7ec4.pth\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c94e9e199e8448f0a34569c80e0f0e32",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=87306240.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "learn = cnn_learner(dls, arch=resnet34, metrics=accuracy, loss_func=CrossEntropyLossFlat()).to_native_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.012022644281387329, lr_steep=0.001737800776027143)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXgV5fn/8fednSyEEMKaQCCsQVk0Iru4IVpFa933VktdWrVaq/7autV+a2vVVm2rVO1isWrFBXdFUUEUCMge1rCGJSErIXty//44g43xJATInMk5uV/XdS7OmXnmzOfkCufOM/PMPKKqGGOMMU2FeR3AGGNM+2QFwhhjjF9WIIwxxvhlBcIYY4xfViCMMcb4ZQXCGGOMXxFeB2hL3bp10/T0dK9jGGNM0Fi6dOk+VU3xty6kCkR6ejrZ2dlexzDGmKAhItuaW2eHmIwxxvhlBcIYY4xfViCMMcb4ZQXCGGOMX1YgjDHG+GUFwhhjjF9WIIyrqmrrWbyliM0F5dit5Y0JLq5fByEi4UA2kKeqZzdZFw38CzgeKAQuVtWtzrq7gWuBeuBmVX3f7aymbewoqmD2sp18sbmQr7aXUFPfAED3hGjGZSST1S+J6MhwcOpFYmwkWf2SSI6P9jC1MaapQFwodwuQA3T2s+5aoFhVB4rIJcDvgItFJBO4BBgO9AbmishgVa0PQF5zFBZvKWLG89mUVtYyvHdnrhrXjzH9u1J4oIaFmwv5fFMhbyzf5XfbQd3jOaF/V8BXZLYXVVCwv5reXToxoFscGd3jOSE9iZOHdEdEAvmxjOmQXC0QIpIKfAf4DXCbnybnAvc5z18BnhTf//xzgRdVtRrYIiKbgDHAF27mNf4dqK7j8Y828tX2EjK6xzGoewJDeiYwIjWRhJjIr9u9sTyPO/67ktSunXj9xgmkd4v7xvtcOqYvqsqesirq6hUREBF2l1SyeGsRi3KLmLN8F5HhQt+usRzbJ5Fu8dHklVSyuaCceevz+esnypj+Xbn3nEyG904M9I/CmA7F7R7EH4GfAwnNrO8D7ABQ1ToRKQWSneVfNmq301lmAuyT9fn84rXV5JVUMjI1kfdW7+E/FTsACA8TRqV1YeLAbtTWN/CXTzYzpn9XZl55PF1io/y+n4jQK7HTN5b16dKJrPSu3DgFVLXZ3kFtfQMvZ+/gkQ82cPYTC7g4K43vjOhF55hIEjtFkhQXRWKnSL/bGmMOn2sFQkTOBvJVdamITHFxPzOAGQB9+/Z1azcdSkODsnxnCf9cuJU3lu8iIyWO/14/jhPSu6KqFJRXs37PfhblFjF/0z4e/3gjqvDd0X146HvHEh0RfsT7bunQUWR4GJef2I+zR/Tm8Y828s+FW3lxyY5vtMns1ZlThnbnlGHdGZnahfAwOxRlzJESt0aWiMhvgSuBOiAG3zmIV1X1ikZt3gfuU9UvRCQC2AOkAHcBqOpvm7ZraZ9ZWVlqN+trXnl1Ha99lUfJgRoO1NRTUVOHKnTuFEFip0jioyNZsaOEj9btZV95DVHhYdwwJYMbT85o8Uu/pKKGncWVDO/dOaDnBvaWVbG9qIKyylpKK2vZVVLJZxv2sXR7MfUNSkJ0BMP7dObYPokc0yeRKUO6Ww/DmCZEZKmqZvldF4ihh04P4md+RjHdBByrqtc7J6nPV9WLRGQ48AK+8w69gY+AQYc6SW0Fonmr80r58QvL2FpYAUBUeBix0b4v/bLKWhqcX4OE6AhOGpLC6Zk9mDK4O4mxwfeFWlpRy6cbC1i8pZBVeWXk7C6jpq6B5Lgo7jpzKN87LpUw61kYA7RcIAJ+u28ReQDIVtU5wLPA885J6CJ8I5dQ1TUi8jKwFl8P5CYbwXRkVJV/f7mNX7+dQ1JsJC/88EROSO9KZPj/LoFpaFDKa+ooq6yle0IMURHBfXlMYmwk00f2ZvrI3oDv3MWKHSX83zs53PHKSl5asoP7zx1OZq/A9niMCTYB6UEEivUgvqmmroHb/7uCN1fsYsqQFB65cGSHvtagoUF5ZdlOHnp3HUUHaoiNCqdn5xh6JsbQOSaSmvoGquvqqa1TBveMZ9rwXpw44JvF1JhQ4/khpkCxAvE/NXUN3DhrKXNz8rnjjCHccFKGHVZxlFTU8OqyPPJKKtlTWsWesirKq+qIiggjKiKMcBFW5ZVSWVtPYqdIThvWg2nH9GTSoG7ERB75CXhj2qN2dYjJuM9XHJYxNyefB84dzlXj0r2O1K50iY3iBxP7t9imqraezzYU8N6aPXy4dg+zl+0kNiqcKUNSOGdEb07P7EGE9SxMiLMCEeTqG5SVO0uIDA8jsVMkcdER3Dl7JXNz9lpxOAoxkeFMHd6TqcN7UlvfwJe5hby3eg8frN3LO6v2kJrUiWsn9ueirDTiou2/kQlNdogpiG3cu587Z69k2faSb62z4uCO+gblo5y9/G1+Lku2FtM5JoJLx/TlirH9SOsa63U8Yw6bnYMIMrkF5dw7Zw03TMlgfEa3b62vqWvgr59s5s/zNhEbHc7Ppg6he0I0pZW1lFXVMaBbHCcP7e5B8o5l2fZinp2/hffW7KFBlVOGdOfKcf2YPCjFzveYoGEFIsjc8O+lvLt6DyLwk1MGccupgwgPE+oblLdW7uLxjzayueAA00f25p5zMunWgUcmtQe7Syv5z6LtvLB4B/vKq+mXHMsVJ/bjwqzUZm85Ykx7YQXCA2VVtbywaDvH90vihPSurd5uza5SvvP4An44qT9FB2qZvWwnJ/bvyjkje/Psgi1s2XeAwT3iuXPaUE4d1sPFT2AOV01dA++u3s2/v9zGkq3FREeEcVFWGj+fNuQbNzU0pj2xUUwBVFVbz7+/3MaT8zZRUlFLeJhw57Qh/HDSgFZdlPWnuRtJiIngx6cMIrFTJOMzkvnVG6tZtKWI4b0789QVxzE1s6cdwmiHoiLCOHdUH84d1Yec3WX864utzFq0jY/X5fPIRSMZOyDZ64jGHBbrQbShJVuLuPXF5eSVVDJ5cAo/OWUgzy3Ywrur93DmMT35/QUjqK1XPt2Qz7x1BcRFR/DL7wz7ehTM6rxSzn5iAT89bTC3nDbo6/fdVniAncWVjM9Itit/g8zSbcXc/vJythVVcN3E/tw+dYhdS2HaFTvEFCBXPruIDXv389hFoxg/0HdyWVV5Zv4WHnpvHQkxEZRW1qIKyXFRlFTWMqxXAs9dcwLdE2K47p9LWLyliAV3nUJnOyQRMg5U1/F/7+Qwa9F2RqQm8tQVx9O7S6dDb2hMANghpgCob1C+2l7CeaN7f10cwHf76h9OHsCI1ESeXbCFYc7tqI/tk8inGwq4cdYyzv/LQm6fOpi5OfncfvpgKw4hJi46gt9891hOGpzCbS+vYPqTC/jzZcdxoh1yMu2cXQraRtbtKaO8uo6sfv5PSJ84IJmZV2Xx09MHMzKtC2FhwslDu/PSj8ZSVVvPT19aQZfYSK6ZkB7Y4CZgpg7vyes3TaBzTCSXP7OI57/Y6nUkY1pkBaKNLNtWDMDx/ZIOa7sRqV147cYJZPVL4q5pQ220S4gb2D2e1388gcmDU/jVG2t4YdF2ryMZ0ywrEG0ke1sxPTpHk5p0+MeW07rG8soN47lkjM2I1xF0jonkb1dlMWlQN+57cw1rdpV6HckYv6xAtJHsrcVk9etqo4xMq4SHCX+8eBRdY6O4adYy9lfVeh3JmG9xrUCISIyILBaRFSKyRkTu99PmMRFZ7jw2iEhJo3X1jdbNcStnW9hTWkVeSSXHHebhJdOxJcdH88Rlo9lRXMlds1cRSiMKTWhwcxRTNXCKqpaLSCSwQETeVdUvDzZQ1Z8efC4iPwFGN9q+UlVHuZivzWRvKwIgywqEOUwnpHfljjOG8NC768hamMT3J7R8G3JjAsm1HoT6lDsvI51HS38iXQr8x608R6O+oeW/7LK3FtMpMpzM3p0DlMiEkhmTBnDasO7c/+Zanvhoo/UkTLvh6jkIEQkXkeVAPvChqi5qpl0/oD/wcaPFMSKSLSJfish5buZsjqrys/+u4Lw/f95ikVi6rZiRaYk2NaU5ImFhwpOXHcf5o/vwyIcb+PF/vqKyxqZgN95z9RtNVeudw0SpwBgROaaZppcAr6hq4/8V/Zyr+y4D/igiGf42FJEZTiHJLigoaNP8ry/P45WlO1mVV8oHa/b4bXOguo61u8uavf7BmNaIiQznkYtGcveZQ3ln1W4ueGohu0srvY5lOriA/MmrqiXAPGBaM00uocnhJVXNc/7NBT7hm+cnGrebqapZqpqVkpLSZpnzSiq55/U1HN8viX7JsTz9Wa7frv+KnSXUNyjHp9v5B3N0RIQfnZTBc1efwLbCCq5+brGNbjKecnMUU4qIdHGedwJOB9b5aTcUSAK+aLQsSUSinefdgAnAWreyNtXQoPzs5RU0qPLYRaO4bmJ/lu8oIdu5GK6xpVt9y45LswJh2sbJQ7vz9JXHs7ngALe+uPyQ58CMcYubPYhewDwRWQkswXcO4i0ReUBEpjdqdwnwon7zz/NhQLaIrMDX83hIVQNWIJ77fAtf5BZyzzmZ9E2O5YLj00iKjeTpT3O/1TZ7WzGDe8STGGtXQJu2M2FgN+49J5OP1uXz8PvrvY5jOijXhrmq6kr8HBZS1XuavL7PT5uFwLFuZWtJbkE5v39/PacN68FFWWkAdIoK58px6c5MbuVkpMQDvp7Gsu3FnD2itxdRTYi7cmw/1u/Zz1OfbmZwj3jOPy7V60img7FhN008+fEmwkX47fnHfuOq6KvG9SM6Ioxn5vt6EQ0Nyhsr8thfVWfXPxhXiAj3TR/O2AFduWv2KpZt//YhTmPcZAWike2FFbyxYheXn9iXlIRvzvPcLT6a7x2fyuxleTwzP5fTHv2Un760grSunZgypO1OjhvTWGR4GH+9/Hh6JEZz/fNL2VtW5XUk04FYgWjkqc82E+7M3+DPdRP7U1vfwINv5xAfE8GfLhnFx7dPITk+2m97Y9pCUlwUf7sqi/LqOq7/91Kq6+waCRMYNmGQY29ZFa9k7+SCrFR6dI7x22ZASjx/uzKLzp0iOSE9yW7MZwJmaM/OPHLhSG6YtYxfvb6a331vhP3+GddZD8Lxt89yqVflhpP8Xo/3tdMyezCmv9211QTemcf24uZTBvJy9k7+9cU2r+OYDsAKBFB0oIZZi7Zz7sjepHWN9TqOMc269bTBnDq0O795J4cdRRVexzEhzgoE8PfPt1BVV8+NJ7fcezDGa2Fhwm++e6xvpN27OV7HMSGuwxeI/VW1/GPhVs7I7MnA7glexzHmkHomxnDDlAzeWbWHL3MLvY5jQliHLxDx0RH88eJR/PT0wV5HMabVZkweQO/EGB54c63disO4psMXCBHh1GE9GNLTeg8meMREhnPXWcNYu7uM/2bv8DqOCVEdvkAYE6zOGdGLrH5J/OGD9XbXV+MKKxDGBCkR4Z5zMtlXXsOf5m70Oo4JQVYgjAliI1K7cOmYvjz3+RZW7CjxOo4JMVYgjAlyd581lJSEaO6cvZKaugav45gQYgXCmCDXOSaSB887lnV79vP0p5u9jmNCiBUIY0LA6Zk9OHtEL574eBOb8vd7HceECDenHI0RkcUiskJE1ojI/X7aXCMiBSKy3Hlc12jd1SKy0Xlc7VZOY0LFfdOHExsdzp2zV9Fg10aYNuBmD6IaOEVVRwKjgGkiMtZPu5dUdZTzeAZARLoC9wInAmOAe0XEZuUxpgXd4qP51XcyWbqtmP8utWsjzNFzrUCoT7nzMtJ5tPbPmjPwzWFdpKrFwIfANBdiGhNSzj+uD8f17cIfPtjAgeo6r+OYIOfqOQgRCReR5UA+vi/8RX6afU9EVorIKyKS5izrAzT+E2ins8zfPmaISLaIZBcUFLRpfmOCjYjwy7MzKdhfbSeszVFztUCoar2qjgJSgTEickyTJm8C6ao6Al8v4Z9HsI+ZqpqlqlkpKTb1pzHH9U3inJG9mTk/l92llV7HMUEsIKOYVLUEmEeTw0SqWqiq1c7LZ4Djned5QFqjpqnOMmNMK/z8jCE0KDz8/nqvo5gg5uYophQR6eI87wScDqxr0qZXo5fTgYM3uH8fmCoiSc7J6anOMmNMK6R1jeXaif15dVkeK3faFdbmyLjZg+gFzBORlcASfOcg3hKRB0RkutPmZmcI7ArgZuAaAFUtAn7tbLcEeMBZZoxppRunZJAcF8WDb+egasNezeGTUPrFycrK0uzsbK9jGNNuzFq0jV+8tpqnrjiOacf0OvQGpsMRkaWqmuVvnV1JbUwIuzgrjcE94vntu+uorqv3Oo4JMlYgjAlhEeFh/PI7mWwrrOBfC7d5HccEGSsQxoS4yYNTmDIkhcc/3khhefWhNzDGYQXCmA7gF2cNo6Kmnj99ZBMLmdazAmFMBzCoRwKXjenLrEXb2bjX7vZqWscKhDEdxE9PH0xsVDi/fXfdoRsbgxUIYzqMrnFR3DhlIB+vy+eLzYVexzFBwAqEMR3I9yek0ysxht++m2NzRphDsgJhTAcSExnO7VOHsHJnKW+t2u11HNPOWYEwpoP57ug+DOvVmYfft4vnTMusQBjTwYSHCXefOZQdRZX8+8vtXscx7ZgVCGM6oMmDU5g0qBtPfLyR0spar+OYdsoKhDEd1J3ThlJSUcvfP9/idRTTTlmBMKaDOqZPIqdn9uC5BVsoq7JehPk2KxDGdGC3nDqIsqo6/vn5Vq+jmHbIzRnlYkRksYiscCYFut9Pm9tEZK2IrBSRj0SkX6N19SKy3HnMcSunMR3ZMX0SOXVod55ZsIX91oswTbjZg6gGTlHVkcAoYJqIjG3S5isgS1VHAK8Av2+0rlJVRzmP6RhjXHHLaYMorazlX1/Y7cDNN7lWINSn3HkZ6Ty0SZt5qlrhvPwSSHUrjzHGvxGpXTh5SAp/m59LeXWd13FMO+LqOQgRCReR5UA+vjmpF7XQ/Frg3UavY0QkW0S+FJHzWtjHDKdddkFBQRslN6ZjueW0wZRU1PK89SJMI64WCFWtV9VR+HoGY0TkGH/tROQKIAt4uNHifs48qZcBfxSRjGb2MVNVs1Q1KyUlpY0/gTEdw6i0Lpw02NeLqKyxq6uNT0BGMalqCTAPmNZ0nYicBvwCmK6q1Y22yXP+zQU+AUYHIqsxHdVNJw+k6EANL2fv8DqKaSfcHMWUIiJdnOedgNOBdU3ajAaexlcc8hstTxKRaOd5N2ACsNatrMYYGNO/K1n9kpj5WS619Q1exzHtgJs9iF7APBFZCSzBdw7iLRF5QEQOjkp6GIgH/ttkOOswIFtEVuDreTykqlYgjHHZDVMyyCup5K2Vu7yOYtoBUQ2de8JnZWVpdna21zGMCVoNDcqZf5qPorx3y2TCwsTrSMZlIrLUOd/7LXYltTHma2FhwvVTBrBhbznz1ucfegMT0qxAGGO+4ewRvenTpRN//WSz11GMx6xAGGO+ITI8jB+dNIDsbcUs2VrkdRzjISsQxphvufD4NJLjonjy401eRzEesgJhjPmWTlHhXDupP59uKGDFjhKv4xiPWIEwxvh11bh0EjtF8oT1IjosKxDGGL/ioyP4wYT+zM3Zy9pdZV7HMR6wAmGMadY1E9JJiI7gyXkbvY5iPGAFwhjTrMROkVw9Pp13V+9h4979XscxAdaqAiEicSIS5jwfLCLTRSTS3WjGmPbgBxP70ykynCfn2bmIjqa1PYjP8M3P0Af4ALgS+IdboYwx7UfXuCiuGNuPN1fsIreg/NAbmJDR2gIhzsxv5wN/UdULgeHuxTLGtCc/nDSAqIgwG9HUwbS6QIjIOOBy4G1nWbg7kYwx7U1KQjRXjUvnjeV5bMq3XkRH0doCcStwN/Caqq4RkQH4bsNtjOkgfjR5ADGR4Tz+kY1o6ihaVSBU9VNVna6qv3NOVu9T1ZtdzmaMaUeS46O5enw6b67cxQYb0dQhtHYU0wsi0llE4oDVwFoRueMQ28SIyGIRWSEia0Tkfj9tokXkJRHZJCKLRCS90bq7neXrReSMw/tYxhg3zJg0gNjIcP4013oRHUFrDzFlqmoZcB7wLtAf30imllQDp6jqSGAUME1ExjZpcy1QrKoDgceA3wGISCZwCb4T4dOAv4iInfMwxmNJcVF8f0J/3l61m3V77OrqUNfaAhHpXPdwHjBHVWuBFqeiU5+DZ7MinUfTbc4F/uk8fwU4VUTEWf6iqlar6hZgEzCmlVmNMS66blJ/EqIjeOzDDV5HMS5rbYF4GtgKxAGfiUg/4JB/PohIuIgsB/LxzUm9qEmTPsAOAFWtA0qB5MbLHTudZf72MUNEskUku6CgoJUfxxhzpLrERvGDif15f81eVueVeh2nw3tv9R6e+Ggjbkwf3dqT1I+rah9VPcvpGWwDTm7FdvWqOgpIBcaIyDFHmdffPmaqapaqZqWkpLT12xtj/PjBxP50jongj3OtF+G1d1bt5qXsHfgOvrSt1p6kThSRRw/+pS4ij+DrTbSKqpbgGxY7rcmqPCDN2UcEkAgUNl7uSHWWGWPagcROkcyYPIC5Ofkst/kiPJW7r5yMlHhX3ru1h5ieA/YDFzmPMuDvLW0gIiki0sV53gk4HVjXpNkc4Grn+QXAx+rrJ80BLnFGOfUHBgGLW5nVGBMA10zoT1JsJI/auQjPqCq5BQcYkNLqv9cPS0Qr22Wo6vcavb7fObfQkl7AP53RR2HAy6r6log8AGSr6hzgWeB5EdkEFOEbuYRzMd7LwFqgDrhJVetb/7GMMW6Lj47gRydl8NC768jeWkRWelevI3U4e8qqqKipZ4BLPYjWFohKEZmoqgsARGQCUNnSBqq6EhjtZ/k9jZ5XARc2s/1vgN+0Mp8xxgNXjevHM/NzefTDDbzww6aj2I3bNucfACDDpR5Eaw8xXQ/8WUS2ishW4EngR64kMsYEjdioCG6YMpCFmwv5YnOh13E6nNx9visJPD0HoaornAveRgAjVHU0cIoriYwxQeXyE/vSo3M0j3643pWhlqZ5m/PLiY+OoHtCtCvvf1gzyqlqmXNFNcBtLuQxxgSZmMhwbjp5IEu2FjN/4z6v43Qouft8J6jdGOIKRzflqDuJjDFB5+IT0uidGMMjH26wXkQAbc53b4grHF2BsN8CYwwA0RHh/OTUQazYUcLH6/K9jtMhVNTUsau0igHd3DlBDYcoECKyX0TK/Dz2A71dS2WMCToXHJ9K366xPGq9iIDILfCNYHJriCscokCoaoKqdvbzSFDV1g6RNcZ0AJHhYdx86iDW7Crj/TV7vI4T8nL3OUNcu3vUgzDGmMNx3qjeDEiJ47EPN9LQYL0IN+UWlCMC6clWIIwxQSAiPIxbTxvM+r37eXPlLq/jhLTNBQdITepETKR7U+VYgTDGtKmzj+1FZq/OPPLBBmrqGryOE7JyC8oZ0M298w9gBcIY08bCwoSfTxvC9qIK/rN4u9dxQlJDg+8mfW4OcQUrEMYYF5w0OIWxA7ryxMcbOVBd53WckLOnrIrK2nrX7uJ6kBUIY0ybExHunDaUfeU1PDN/i9dxQs7mAt89mKxAGGOC0ui+SUwb3pOZn22msLza6zgh5eA1EAPtEJMxJlj97IwhVNbW8+S8TV5HCSmbC3w36Utx6SZ9B7lWIEQkTUTmichaEVkjIrf4aXOHiCx3HqtFpF5EujrrtorIKmddtls5jTHuGdg9nouy0pj15XZ2FFV4HSdk+E5Qu3eTvoPc7EHUAberaiYwFrhJRDIbN1DVh1V1lKqOAu4GPlXVokZNTnbWZ7mY0xjjoltOG4QIPDbXpiZtK5sLyl29xcZBrhUIVd2tqsuc5/uBHKBPC5tcCvzHrTzGGG/0SuzENePTee2rPNbv2e91nKB3oLqO3aVVrs0i11hAzkGISDq+6UcXNbM+FpgGzG60WIEPRGSpiMxwO6Mxxj03TMkgPjqCh99f53WUoLdln/s36TvI9QIhIvH4vvhvbTTZUFPnAJ83Obw0UVWPA87Ed3hqcjPvP0NEskUku6CgoE2zG2PaRpfYKK4/KYO5Oflkby069AamWdudczn9kmNd35erBUJEIvEVh1mq+moLTS+hyeElVc1z/s0HXgPG+NtQVWeqapaqZqWkpLRNcGNMm/v+hHRSEqL53Xvr7HbgRyGvuBKA1C5BXCDEd3r9WSBHVR9toV0icBLwRqNlcSKScPA5MBVY7VZWY4z7YqMiuPnUQSzZWmyTCh2FncUVxEdH0LmT+zMuuNmDmABcCZzSaCjrWSJyvYhc36jdd4EPVPVAo2U9gAUisgJYDLytqu+5mNUYEwCXnJBG/25xPPh2DtV19V7HCUp5JZX06dLJ9SGuAK6VIFVdQCvmrVbVfwD/aLIsFxjpSjBjjGciw8O4f/pwrnpuMTM/zeUnpw7yOlLQ2VlcSWpSp4Dsy66kNsYE1OTBKXzn2F48OW8T2wvt4rnDlVdSSR8rEMaYUPWrszOJCBPunbPaTlgfhrKqWvZX1dGnixUIY0yI6pkYw09PH8y89QV8sHav13GCxsERTNaDMMaEtGvGpzO0ZwL3z1lDRY3NGdEaXxcI60EYY0JZRHgYD553DLtKq3hugc0Z0Rp5JdaDMMZ0EFnpXTk9swdPf5pL8YEar+O0e3kllURFhNEtzt3bfB9kBcIY46k7zhhCeU0df/10s9dR2r28Yt81EGFh7l8DAVYgjDEeG9wjgfNHp/KPhVvZXVrpdZx2bWdJ4K6BACsQxph24NbTBoHCn+Zu9DpKu5ZXXBGwE9RgBcIY0w6kdY3lshP78nL2DjYXlHsdp12qqq1nX3mNFQhjTMfz41MGEhMZziMfrPc6SrsU6BFMYAXCGNNOdIuPZsbkAbyzag+Lcgu9jtPuBPoaCLACYYxpR340OYPeiTHc9+Za6hvsFhyNWQ/CGNOhdYoK5+6zhpGzu4wXl2z3Ok67kldcSXiY0LNzTMD2aQXCGNOunD2iF2P6d+UP76+ntKLW6zjtRl5JJT07xxARHrivbTdnlEsTkXkislZE1ojILX7aTBGR0kYTCt3TaN00EVkvIptE5C63chpj2hcR4b5zhlNaWctjczd4HafdyDoAl9gAABKNSURBVCsO3G2+D3KzFNUBt6tqJjAWuElEMv20m6+qo5zHAwAiEg78GTgTyAQubWZbY0wIyuzdmUvH9OX5L7exfs9+r+O0C3kllaQG8AQ1uFggVHW3qi5znu8HcoA+rdx8DLBJVXNVtQZ4ETjXnaTGmPbo9qlDiI+O4O5XV3b4E9Z19Q3sKasKqR7E10QkHRgNLPKzepyIrBCRd0VkuLOsD7CjUZudtL64GGNCQNe4KO6fPpxl20uY+Vmu13E8taesivoGDegQVwhAgRCReGA2cKuqljVZvQzop6ojgSeA14/g/WeISLaIZBcUFBx9YGNMu3HuqN6ceUxPHvtwA+v2NP366DgCPVHQQa4WCBGJxFccZqnqq03Xq2qZqpY7z98BIkWkG5AHpDVqmuos+xZVnamqWaqalZKS0uafwRjjHRHhwfOOoXOnCG57aQU1dQ1eR/LETg8ukgN3RzEJ8CyQo6qPNtOmp9MOERnj5CkElgCDRKS/iEQBlwBz3MpqjGm/kuOj+b/vHsva3WU8+XHHvJnfwYvkege4QES4+N4TgCuBVSKy3Fn2/4C+AKr6FHABcIOI1AGVwCXqm8G8TkR+DLwPhAPPqeoaF7MaY9qxqcN78r3jUvnzJ5s5eWh3RvdN8jpSQOUVV9ItPpqYyPCA7te1AqGqC4AWZ7VQ1SeBJ5tZ9w7wjgvRjDFB6N7pmXyZW8htL6/g7ZsnEhvl5t+37UtegOeBOMiupDbGBIXOMZH84cKRbC08wINv53gdJ6DySgJ/kRxYgTDGBJFxGcnMmDSAFxZt56OcvV7HCYiyqlq2F1WQ0S0u4Pu2AmGMCSq3TR3MsF6duXP2SvaVV3sdx3VfbC6kvkEZP7BbwPdtBcIYE1SiI8L548WjKKuq467Zq/CNawldCzbuIzYqnOM8ODFvBcIYE3SG9Ezg52cMYW7OXmYv83uJVMiYv7GAsQOSiYoI/Ne1FQhjTFD6/oT+nJCexP1vrmF3aaXXcVyxo6iCrYUVTBoU+MNLYAXCGBOkwsOEP1w4krp65c4QPdQ0f+M+ACYN8uYuEVYgjDFBq19yHHefNZTPNhTw4pIdh94gyCzYVECvxBgyUgI/ggmsQBhjgtwVJ/ZjfEYyD761lh1FFV7HaTP1DcrnmwqZNKgbzh2JAs4KhDEmqIWFCb+/YAQiwl2vrgyZQ02r8kopraxlokeHl8AKhDEmBKQmxXLXmUP5fFMhL2eHxqGm+RsKEIGJHlz/cJAVCGNMSLhsTF/GDujKg2/lsKe0yus4R23+pn0M792ZrnFRnmWwAmGMCQlhYcLvvjeC2oYGfvl6cI9qKq+uY9m2Ys9GLx1kBcIYEzL6Jcfxs6lDmJuTz5wVu7yOc8QW5RZS16BM8vDwEliBMMaEmO9P6M+otC7cN2dN0N6raf7GfcREhnF8urfzXliBMMaElPAw4eELRnCgup573wjOecY+37SPMf2TiY4I7ARBTbk55WiaiMwTkbUiskZEbvHT5nIRWSkiq0RkoYiMbLRuq7N8uYhku5XTGBN6BvVI4NbTB/H2qt28vXK313EOS/7+KjbmlzM+I9nrKK72IOqA21U1ExgL3CQimU3abAFOUtVjgV8DM5usP1lVR6lqlos5jTEhaMakAYxMTeRXb6ymMIgONX2xuRCACRnenn8AFwuEqu5W1WXO8/1ADtCnSZuFqlrsvPwSSHUrjzGmY4kID+PhC0dSXlXHPUF0qGnhpkI6x0SQ2buz11ECcw5CRNKB0cCiFppdC7zb6LUCH4jIUhGZ0cJ7zxCRbBHJLigoaIu4xpgQMbhHArecFlyHmj7fvI9xGcmEh3lze43GXC8QIhIPzAZuVdWyZtqcjK9A3Nlo8URVPQ44E9/hqcn+tlXVmaqapapZKSnejhk2xrQ/P5o8gBGpifzy9VXsLWvfF9DtKKpgZ3El49vB4SVwuUCISCS+4jBLVV9tps0I4BngXFUtPLhcVfOcf/OB14AxbmY1xoSmiPAwHr1oFFW1Ddz28nIaGtrvBXSfb/Ld3nvCQO9PUIO7o5gEeBbIUdVHm2nTF3gVuFJVNzRaHiciCQefA1OB1W5lNcaEtoHd47lveiafbyrk6c9yvY7TrIWbC+meEE1GSrzXUQCIcPG9JwBXAqtEZLmz7P8BfQFU9SngHiAZ+ItzO9s6Z8RSD+A1Z1kE8IKqvudiVmNMiLsoK43PNuzjkQ/WMy4jmVFpXbyO9A2qysLNhUwcmOzZ7b2bcq1AqOoCoMVPqarXAdf5WZ4LjPz2FsYYc2REhP87/1iW7yjh5v98xds3TyQhJtLrWF/bsLecfeXVjPf49hqN2ZXUxpgOI7FTJH+6ZBQ7iyu4478r29X5iIWbfecf2sMFcgdZgTDGdChZ6V35xXcyeW/NHn73/jqv43zt802F9EuOJTUp1usoX3PzHIQxxrRLP5iQzpZ95Tz9aS7pyXFcOqavp3nq6htYlFvI2SN7eZqjKSsQxpgOR0S475zh7Ciq5JevryYtKZaJg7w79r8yr5T91XXt5vqHg+wQkzGmQ4oID+PJy0YzMCWeG2YtJWe33+t4A2LO8l1ERYQxeXD7utjXCoQxpsNKiInk2WuyiIuK4MpnF5NbUB7wDLX1DcxZsYvTM3uQ2Kn9jKoCKxDGmA4uNSmWf193Ig2qXPHMInYWVwR0/5+uL6DoQA3nj+5z6MYBZgXCGNPhDewez79+MIb91XVc8cwi8vcH7p5Nr361k+S4qHZ3eAmsQBhjDADH9EnkH98fQ/7+ai7/2yLyA3Bjv9KKWuauzWf6qN5Ehre/r+P2l8gYYzxyfL8knrk6i7ySSi58+gt2FLl7uOntVbupqW/g/NHtcyocKxDGGNPI+IxuzLruREoqarnwqS/YlL/ftX29umwng7rHc0wf7ycH8scKhDHGNDG6bxIvzhhLXYNy0dNfsmJHSZvvY1vhAbK3FXP+cant5uZ8TVmBMMYYP4b16swr148jNiqci57+gte+2tmm7//qsjxE4LzRvdv0fduSFQhjjGlGerc43rhpAqPSuvDTl1bw4FtrqatvOOr3PVBdxytLdzI+I5leiZ3aIKk7rEAYY0wLkuOj+fd1J3LN+HSeWbCFq/+++KiGwaoqd7+6it2llfzklEFtmLTtuTmjXJqIzBORtSKyRkRu8dNGRORxEdkkIitF5LhG664WkY3O42q3chpjzKFEhodx3/Th/P6CESzZWszUxz7jjeV5qB7+7cL/vWg7c1bs4rbTBzN2QPu5tbc/bvYg6oDbVTUTGAvcJCKZTdqcCQxyHjOAvwKISFfgXuBEfHNR3ysiSS5mNcaYQ7ooK413bp5Ev+Q4bnlxOTfOWsa+8upWb79yZwm/fnMtU4akcOOUgS4mbRuuFQhV3a2qy5zn+4EcoOm15OcC/1KfL4EuItILOAP4UFWLVLUY+BCY5lZWY4xprYHd45l9/TjunDaUj3LyGf/Qx/zgH0t4YdF29rZwcV3B/mpunLWMlIRoHrtoFGFh7XPkUmMBud23iKQDo4FFTVb1AXY0er3TWdbccn/vPQNf74O+fb29p7sxpmOICA/jhikZnDasOy8s3s6Ha/fy8bp8eA26xUfTLT6KlIRoEmIi2FVSxfaiCooO1BAZLrz8o3EkxUV5/RFaxfUCISLxwGzgVlVt8/vpqupMYCZAVlZW+5k/0BgT8gb1SODec4Zzz9mZbNhbzkfr9rK9sIJ95TXsK68mr6SSXokxnDG8B327xjEuI5lRaV28jt1qrhYIEYnEVxxmqeqrfprkAWmNXqc6y/KAKU2Wf+JOSmOMOToiwpCeCQzpmeB1lDbl5igmAZ4FclT10WaazQGuckYzjQVKVXU38D4wVUSSnJPTU51lxhhjAsTNHsQE4EpglYgsd5b9P6AvgKo+BbwDnAVsAiqA7zvrikTk18ASZ7sHVLXIxazGGGOacK1AqOoCoMXT9OobRHxTM+ueA55zIZoxxphWsCupjTHG+GUFwhhjjF9WIIwxxvhlBcIYY4xfViCMMcb4JUdyN8L2SkQKgBKgtNHixEavDz73t6wbsO8Idtv4vVq7/lDLmq5v69yHytyajP6WHeq5Fz/r5pYHY+6WXlvuQ+c61PojyR0K3yWDVDXR7x5VNaQewMzmXh983syy7LbYX2vWH2pZc5+hrXIfKnNb5G7p5x7In3Uo5W7N77blDmzuUPou8fcIxUNMb7bw+s0WlrXV/lqz/lDLmvsMbZW7Ndsebe5D/dyPxJH8rJtbHoy5W/O73fi55T689UeSO6S/S0LqENPREJFsVc3yOsfhCsbcwZgZLHegWW7vhWIP4kjN9DrAEQrG3MGYGSx3oFluj1kPwhhjjF/WgzDGGOOXFQhjjDF+WYEwxhjjlxWIVhCRSSLylIg8IyILvc7TGiISJiK/EZEnRORqr/O0lohMEZH5zs97itd5DoeIxIlItoic7XWW1hKRYc7P+hURucHrPK0lIueJyN9E5CURmep1ntYSkQEi8qyIvOJ1ltYI+QIhIs+JSL6IrG6yfJqIrBeRTSJyV0vvoarzVfV64C3gn27mdbIddWbgXHxTtdYCO93K2lgb5VagHIghuHID3Am87E7Kb2uj3+0c53f7InyTfLmujXK/rqo/BK4HLnYzb6N8bZE7V1WvdTdp2wn5UUwiMhnfF86/VPUYZ1k4sAE4Hd+X0BLgUiAc+G2Tt/iBquY7270MXKuq+9t7ZudRrKpPi8grqnqBm5nbMPc+VW0QkR7Ao6p6eZDkHgkk4yts+1T1rWDIrar5IjIduAF4XlVfCJbcznaP4JvzflmQ5Q7I/8mj5eaUo+2Cqn4mIulNFo8BNqlqLoCIvAicq6q/BfweHhCRvvjmzHa1OEDbZBaRnUCN87LevbT/01Y/a0cxEO1Gzqba6Oc9BYgDMoFKEXlHVRvae27nfeYAc0TkbcD1AtFGP28BHgLeDURxgDb//Q4KIV8gmtEH2NHo9U7gxENscy3wd9cSHdrhZn4VeEJEJgGfuRnsEA4rt4icD5wBdAGedDdaiw4rt6r+AkBErsHpBbmarnmH+/OeApyPrxi/42qylh3u7/dPgNOARBEZqL457r1wuD/vZOA3wGgRudspJO1WRy0Qh01V7/U6w+FQ1Qp8RS2oqOqr+IpbUFLVf3id4XCo6ifAJx7HOGyq+jjwuNc5DpeqFuI7bxIUQv4kdTPygLRGr1OdZe1ZMGYGyx1oljuwgjV3q3TUArEEGCQi/UUkCrgEmONxpkMJxsxguQPNcgdWsOZunSO5b3kwPYD/ALv533DPa53lZ+EbfbAZ+IXXOYM9s+W23Ja7feY+mkfID3M1xhhzZDrqISZjjDGHYAXCGGOMX1YgjDHG+GUFwhhjjF9WIIwxxvhlBcIYY4xfViBMSBOR8gDvr03mCxHfvBilIrJcRNaJyB9asc15IpLZFvs3BqxAGHNYRKTF+5ep6vg23N18VR0FjAbOFpFDzddwHr67yRrTJqxAmA5HRDJE5D0RWSq+2euGOsvPEZFFIvKViMx15qRARO4TkedF5HPgeef1cyLyiYjkisjNjd673Pl3irP+FacHMMu5RTUicpazbKmIPC4iLc4doaqVwHJ8dw5FRH4oIktEZIWIzBaRWBEZD0wHHnZ6HRnNfU5jWssKhOmIZgI/UdXjgZ8Bf3GWLwDGqupo4EXg5422yQROU9VLnddD8d2WfAxwr4hE+tnPaOBWZ9sBwAQRiQGeBs509p9yqLAikgQM4n+3bX9VVU9Q1ZFADr5bPizEdw+gO1R1lKpubuFzGtMqdrtv06GISDwwHviv8wc9/G9iolTgJRHpBUQBWxptOsf5S/6gt1W1GqgWkXygB9+eInWxqu509rscSMc3I1muqh587/8AM5qJO0lEVuArDn9U1T3O8mNE5EF8c2bEA+8f5uc0plWsQJiOJgwocY7tN/UEvmlO5zgT6dzXaN2BJm2rGz2vx///pda0acl8VT1bRPoDX4rIy6q6HPgHcJ6qrnAmKJriZ9uWPqcxrWKHmEyHoqplwBYRuRB8U1eKyEhndSL/u5f/1S5FWA8MaDR15cWH2sDpbTwE3OksSgB2O4e1Gs/Zvd9Zd6jPaUyrWIEwoS5WRHY2etyG70v1WufwzRrgXKftffgOySwF9rkRxjlMdSPwnrOf/UBpKzZ9CpjsFJZfAYuAz4F1jdq8CNzhnGTPoPnPaUyr2O2+jQkwEYlX1XJnVNOfgY2q+pjXuYxpynoQxgTeD52T1mvwHdZ62uM8xvhlPQhjjDF+WQ/CGGOMX1YgjDHG+GUFwhhjjF9WIIwxxvhlBcIYY4xfViCMMcb49f8Bn55SY902T4QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.407955</td>\n",
       "      <td>0.325616</td>\n",
       "      <td>0.924154</td>\n",
       "      <td>02:17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.460510</td>\n",
       "      <td>1.098424</td>\n",
       "      <td>0.767116</td>\n",
       "      <td>02:24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.482724</td>\n",
       "      <td>0.715637</td>\n",
       "      <td>0.783660</td>\n",
       "      <td>02:24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.336754</td>\n",
       "      <td>0.416998</td>\n",
       "      <td>0.883685</td>\n",
       "      <td>02:26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.176540</td>\n",
       "      <td>0.237083</td>\n",
       "      <td>0.933825</td>\n",
       "      <td>02:25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.074207</td>\n",
       "      <td>0.211133</td>\n",
       "      <td>0.939170</td>\n",
       "      <td>02:26</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fine_tune(5, 1e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We may need to convert back full precision before exporting the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = learn.to_native_fp32()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment 6 - Optimizers (LookAhead + RAdam)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Source: https://www.kaggle.com/muellerzr/cassava-fastai-starter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"ranger\" class=\"doc_header\"><code>ranger</code><a href=\"https://github.com/fastai/fastai/tree/master/fastai/optimizer.py#L327\" class=\"source_link\" style=\"float:right\">[source]</a></h4>\n",
       "\n",
       "> <code>ranger</code>(**`p`**, **`lr`**, **`mom`**=*`0.95`*, **`wd`**=*`0.01`*, **`eps`**=*`1e-06`*, **`sqr_mom`**=*`0.99`*, **`beta`**=*`0.0`*, **`decouple_wd`**=*`True`*)\n",
       "\n",
       "Convenience method for `Lookahead` with `RAdam`"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(ranger)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = cnn_learner(dls, arch=resnet34, metrics=accuracy, opt_func=ranger)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.025118863582611083, lr_steep=0.004365158267319202)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3gc5bn38e+tXi3LklxlFTfccENxwTa9GEJLAgQw3cQh5KSRl5yQ5ISQchJCAjmQBEJCS4DQHQwBgxMMAYyN5Yo7LrLcZDVbvet+/9ixWeSVLNmanV3p/lzXXt6deWbnt4vQreeZmWdEVTHGGGPaivA6gDHGmNBkBcIYY0xAViCMMcYEZAXCGGNMQFYgjDHGBGQFwhhjTEBRXgfoTunp6ZqTk+N1DGOMCRsrV64sVdWMQOt6VIHIyckhPz/f6xjGGBM2RGRXe+tsiMkYY0xAViCMMcYEZAXCGGNMQFYgjDHGBGQFwhhjTEBWIIwxxgRkBcIc5UBlPUu3lbLnYC2trTYdvDG9VY+6DsIcP1Xlw+1l/G3ZLt7aeIAWpzDEREUwLD2RH1w4htNGBbyWxhjTQ1mBMBysaeT6xz7i470V9E2I5pZZucwamc6eg3UUlNbwr00HmPfkCu67chIXTxzc7vs0NrdS19RCbFQEMZERRERIED+FMaa7WYHo5eoaW5j35Aq2HKjini+dzKWThhAXHfmZNredOYJbnlzBN59dTUVdE9dOz/7M+s1FlTy9rJAFq/dS3dB8ZPnAPnH8+vIJ1vMwJkxJT7rlaF5entpUG53X3NLKrU+t4t+bD/DQ3CnMGT+o3bZ1jS18/ZlVvL25mLNH9ycuOpKWVmV/ZT1rdx8iJiqCi04exLghKTQ2t9LY3MrrH+9na3EVXz9jBN8+ZyRRkXbIy5hQIyIrVTUv4DorEL2PqtLQ3Mrdr27k7x8V8tNLx3H9jJxjbtfU0spPX93I+9tKiRCIjBASYqK4aMIgvjQlk9TEmM+0r2ts4ScLN/Bc/m6m5vTjvHEDaGpRmltaiYgQkuOiSIqNIjUhhinZqaTER7v0iY0x7bECEQZUlZKqBpLjoomPiTz2Bl1QUdfEP1bv5bkVu9lZWkNdU8uRdbedMZzvzRndrftra8HqPfxowXpqGlvabRMVIUwflsa5YwcwITOFlPho+sRHExMVQVl1IyVVDZRWN9A3IZrRA/vQzylGJVUNrCgoZ83uQ9T7fa6U+GjGDU7h5MwUBqfEIXL08ZCq+ia2l9QwZlAysVHH950f/v8n0PsbEw6sQHhsZ2kNfeOjj/oLe+O+Shas3sP6vZVsLqrkYG0TGcmx/O7Lk5g5Iv2E91tcVc9v39zKK2v3Ut/UyvghfZiem0ZCTCRxMZEM6RvPJRMHB+WXW31TC40trURHRBAZIbSqUt3QTFV9Mwcq63lnSwmLNxaxvaSmU++XkRxLYkwkBWW1gO9sq0S/wlpR18ThM3TTEmM4bVQG54wZwGmjfAffn1q2iwWr91Lb2EJCTCSnDk/jtFEZNLcoO0qr2VlaQ31TK6MGJDNmUDI5aYlsK65mZeFBVu06yIHK+iPvHxUhZKUlMCIjieH9k4iPjqSuqYW6xhb6xEdz/Yxs0pNiu/X7NKa7WIHwyMGaRn7+z028tGoPkRHC9GH9mDNuIHHRkTzzUSGrC31j92MG9WHMwGRG9E/i7x8VsqO0htvOGM53zhl13OP2SzYX8/9eWEt1QzNfmDyEa6ZlMSGzbzd/wu63s7SGgtIaKuubqKxror6plbSkGDKSY0lLjKWkuoEtRZVsLqqiur6ZU7JT+VxuP8YPTiEm6tPvqq6xhc1FlazfV8mqXQd5Z0sxB2ubiIoQmluV2KgILp44mNkj08kvOMg7W4vZXV4HQJ+4KIZlJBEbFcHmoioq6pqOvO+QvvGckp1KVr8EIgQQobG5lZ2l1Wwv8WVvblViIiOIi46guqGZ+OhIbj19OPNm55IQY+eFmNDiSYEQkTjgP0AsvrOlXlTVu9q0uR8403mZAPRX1b7OuhbgY2ddoapecqx9hkqBUFUWrt3HT1/dSEVdE/Nm5xIVIbyxvogdzl/IwzISmTstm8unZJKS8OnYe21jM3cv3Mhz+bs5JTuVR2/Io29CTHu7Okp9Uwv3LNrM4x8UMHpgMg9ePZmRA5K7/TOGm5ZWZVXhQZZsLqZfYgyXn5L5me9VVdlzsI74mEjSEmOO9KpUlaLKenaW1JCbkciglPgO99Pc0gpwpLBvL6nm14s28+aGA/RPjmX+acO4amoWSbFWKExo8KpACJCoqtUiEg28D3xLVZe10/4bwGRVvdl5Xa2qSV3ZZ6gUiOdX7OZ7L61j4tC+/OqLJzNmUB/A98tmW3E1lfXNTMnq2+HQzitr9nLHC+uYkJnCU7dM+8ypp1X1TazcdZCYqAjioyOJjBBWFx7ivU9KWbajjOqGZm48NYfvXzD6qFNWjTdW7irn14u2sHxnOclxUcydls31M7IZ3LfjgmOM2zwfYhKRBHwF4muqurydNkuBu1R1sfM6bAvEtX9Zzr6KOhZ/53QiT+BisdfW7eMbf1/NOWMG8NDcKURFRrB8Rxm3P7+WvYfqjmo/tF88s0ZkcPHEQZw6/MSPYZjut7rwIH95bydvrN9Pq8LI/knMGpnO6aMyOG1khl1caIKuowLhaj9XRCKBlcAI4A8dFIdsIBd4229xnIjkA83Ar1T1H+1sOx+YD5CVldWN6Y/21LJd7Cqr4c4LxrT7P3JlfRPLdpQxb3buCRUHgIsmDKa0qoGfvLqR/3llPf0SY/jjO9vJ7pfAYzfmkRATRV1TCw1NrYwZlEx2WuIJ7c+4b3JWKn+Ym0phWS1vrN/P+9tKeWZ5IY9/UMBVnxvKL75w8gn/3BjTXVwtEKraAkwSkb7AAhEZr6rrAzS9Ct8xCv/zILNVda+IDAPeFpGPVXV7gH08AjwCvh6ECx8D8I1h3794K2U1jSTERPGdc0cFbPfulhKaW5Vzxwzolv3eODOXA1UNPPSO76NfmZfJXRePI9HGsMNaVloCXz19OF89fTj1TS08+PYn/GHJduqbWvjNFRPtokITEoLyW0ZVD4nIEmAO0F6B+HqbbfY6/+4QkXeAycBRBSJY8gvKKatpZET/JP7v358wakAyn59w9JXH/9p0gLTEGCZnpXbbvr93/kn0jY8mOy2ROeMHdtv7mtAQFx3JHeePJiEminvf3EJDcyv/d9Xkz5yVZYwXXPsJFJEMp+eAiMQD5wKbA7QbDaQCH/otSxWRWOd5OjAT2OhW1s5YtKGImKgIXvjqDKZk9eW7L6xh/d6Kz7RpamllyeZizhrdv1uHCUSEr54+3IpDD/f1M0fwo8+P4Y31RVzz52VsLqr0OpLp5dz8E2UQsERE1gErgMWq+pqI/FRE/E9ZvQp4Vj97tHwMkC8ia4El+I5BBKVAbD1QxTPLCz+zTFV5c30Rp43MIDUxhj9dl0e/hBjm/zWfsuqGI+1WFJRTWd/MOWO7Z3jJ9D63zB7G7748ie0l1Xz+gff5ycINn7kOw5hgcm2ISVXX4RsWarv8x21e/yRAm6XAyW5l68jjHxTw948KGZ6RyLRhaQCs21PBvop6bj/vJMB3Fe8j1+fxxYeW8r0X1/GXG/IQEf61sZiYqAhmj7QziMzxu2zyEE4flcFvF2/hyQ8LeG3dPu67cpLNimuCzgY529hV5ruQ7Z5Fm4/Ms7NoQxGREcI5Y/ofaTd+SAp3XjCaf28u5qnlhagqizcVMWtEul0ta05YamIMP7/sZF79r1n0S4zhhsc/4rdvbTlyIydjgsEKRBsFpTWkxEezqvAQizceQFVZtL6IGcPSjrqi+cZTczh9VAY/f20jr39cxO7yOs7pprOXjAHfHyKvfH0Wl0/J5MG3tzH3L8soqWo49obGdAMrEH7qm1rYV1HP9TOyGZaeyL1vbmFzURU7S2sCHiAWEe69YgJJsVF889nVAJzt18swpjvEx0Ry7xUT+c0VE1mz+xA3PfHRZ2auNcYtViD87C73zQw6on8S/+/8k/ikuJrvPLcGETivnQPP/ZN9d01raVUmZqYwoE9cMCObXuTyUzL5/dVT2LCvku+/tI5gzIJgejcbLPezs9R3/CE7LZGJmSlMyExh3Z4K8rJT6d/BL/6zxwzgl188mRy7ktm47JyxA7j9nFH8dvFWxg9J4ZbZw7yOZHow60H42eXcWyA3LRER4b+dG+lccHL7t+I87OqpWcwYnuZqPmPAd73EnHED+d/XN/HeJyVexzE9mBUIPzvLauibEH1k+u2ZI9JZcNupXDc92+NkxnwqIkL47ZUTGdk/mW/8fTXFVfVeRzI9lBUIP7vKao6a8G5yVqpNeWBCTmJsFH+YO4XaxhZ+uGC9HY8wrrDffH4KSmvJTUvwOoYxnTKifxJ3nHcSizce4B9r9nodx/RAViAcvlNc62zKbBNWbp6VyynZqfxk4UYOVNpQk+leViAcew7Wogq56VYgTPiIjBDuvXwC9U0t/ODlj22oyXQrKxCOnaW+M5iybYjJhJlhGUnccf5J/HtzMS+vsqEm032sQDgOz8FkPQgTjm6amcvnclK5+9UNNtRkuo0VCMdOZw6mtvMtGRMOIiOEX18+kcaWVu60oSbTTaxAOHaV1ZJjvQcTxnLTE7nj/NG8vbmYl2yoyXQDKxCOnaU15NjxBxPmbjo158hQU1GFDTWZE2MFAmho9p3ianMpmXAX4Qw1NbW0cseLa+3+EeaEWIEAdpfXoQo56daDMOEvNz2R/7loLO99Uspv3tridRwTxlwrECISJyIfichaEdkgIncHaHOjiJSIyBrncYvfuhtE5BPncYNbOcF3kyDALpIzPcbcadlcMy2Lh97Zzit2lbU5Tm5O990AnKWq1SISDbwvIm+o6rI27Z5T1f/yXyAi/YC7gDxAgZUislBVD7oRtODwKa5WIEwP8pOLx7HtQDXfe3Edw9KTODkzxetIJsy41oNQn2rnZbTz6OyA6PnAYlUtd4rCYmCOCzEBX4HoExdFX2cWV2N6gpioCP547RTSEmOY/7d8u1Wp6TJXj0GISKSIrAGK8f3CXx6g2ZdEZJ2IvCgiQ51lQ4Ddfm32OMtcsausltx03z0gjOlJ0pNieeT6PA7VNvGVv+bbrUpNl7haIFS1RVUnAZnAVBEZ36bJq0COqk7A10t4sqv7EJH5IpIvIvklJcd385SCANN8G9NTjB+Swu+umsTaPYf47gtrabUzm0wnBeUsJlU9BCyhzTCRqpap6uF+71+AU5zne4Ghfk0znWWB3vsRVc1T1byMjIwuZ2tpVZpb1C6SMz3a+eMG8v05o/nnuv3c/6+tXscxYcK1g9QikgE0qeohEYkHzgXuadNmkKrud15eAmxynr8J/K+IpDqvzwPudCNnZITw4Z1n219Vpsebf9owdpTU8ODb2xjRP4lLJ7k2amt6CDfPYhoEPCkikfh6Ks+r6msi8lMgX1UXAt8UkUuAZqAcuBFAVctF5GfACue9fqqq5S5mJSLCjj+Ynk1E+Nll49lWUs1dCzcwe2QG/RJt7jHTPulJk3rl5eVpfn6+1zGMCWlbiqq48IH3uDIvk19+cYLXcYzHRGSlquYFWmdXUhvTy5w0MJkbT83h2RW7Wbv7kNdxTAizAmFML/Ttc0aSnhTLj19Zb8ffTLusQBjTCyXHRfODC0ezdk8Fz+XvPvYGpleyAmFML3XZpCFMzenHrxdt5lBto9dxTAiyAmFMLyUi3H3pOCrqmrh/sV0bYY5mBcKYXmzMoD7MnZbNU8sL2VJU5XUcE2KsQBjTy91+7iiSYqO4+9UNdi9r8xlWIIzp5VITY/jueaNYur2MNzcUeR3HhBArEMYYrpmaxeiByfzstU0242uY+d2/tnLlwx+68t5WIIwxREVGcNfF49h7qI4nlxZ4Hcd0wc7SGooq6115bysQxhgAZgxPY+aINJ5YWkBTS6vXcUwnldc0ujanlhUIY8wRN8/MZX9FPYvW27GIcFFWbQXCGBMEZ57Un5y0BB77YKfXUUwnHay1AmGMCYKICOGmmbmsLjzEqsKDXscxx6CqlNU0kmYFwhgTDJefkklyXBSPf1DgdRRzDLWNLTQ2t5JqBcIYEwyJsVFc9bmhvP7xfvZX1Hkdx3SgvMY3h5YNMRljguaGU3NQVf764S6vo5gOHCkQCVYgjDFBkpmawPnjBvLM8kK7cC6EHSkQSVYgjDFBdP2MHCrqmnht3X6vo5h2hG0PQkTiROQjEVkrIhtE5O4AbW4XkY0isk5E/i0i2X7rWkRkjfNY6FZOY0xg04f1Y0T/JJ5aZsNMoSqcexANwFmqOhGYBMwRkelt2qwG8lR1AvAi8Gu/dXWqOsl5XOJiTmNMACLC3GlZrNl9iPV7K7yOYwIoq2kkOlJIjo1y5f1dKxDqU+28jHYe2qbNElWtdV4uAzLdymOM6bovTskkPjrSehEh6mBNI6kJMYiIK+/v6jEIEYkUkTVAMbBYVZd30Hwe8Ibf6zgRyReRZSJyWQf7mO+0yy8pKemm5MYYgJT4aC6ZOJhX1uyjoq7J6zimjTIX52EClwuEqrao6iR8PYOpIjI+UDsRuRbIA+71W5ytqnnANcDvRGR4O/t4RFXzVDUvIyOjmz+BMeba6dnUNbXw8qo9Xkcxbbg5zQYE6SwmVT0ELAHmtF0nIucAPwQuUdUGv232Ov/uAN4BJgcjqzHms07OTGHi0L48vbzQ7jgXYtycyRXcPYspQ0T6Os/jgXOBzW3aTAb+hK84FPstTxWRWOd5OjAT2OhWVmNMx66dlsW24mo+3F7mdRTjJ2wLBDAIWCIi64AV+I5BvCYiPxWRw2cl3QskAS+0OZ11DJAvImvx9Tx+papWIIzxyMUTB5OeFMPD/9nhdRTjaGpppaKuydUC4c65UYCqriPAsJCq/tjv+TntbLsUONmtbMaYromLjuSmmbnc++YW1u+tYPyQFK8j9XoHa92dhwnsSmpjTCddOz2bpNgoHn53u9dRDHCwxndWmRUIY4znUuKjmTs9i9c/3k9BaY3XcXq9shrfOT1WIIwxIWHezFyiIiP4kx2L8Jz1IIwxIaV/nzguPyWTl1buobiy3us4vVq59SCMMaFm/uxhNLe28qjdt9pT5U4PItWlmVzBCoQxpoty0hO5YPwgnlleSE1Ds9dxeq3ymgb6xEURHener3ErEMaYLrtpZg5V9c0sWL3X6yi9ltvzMIEVCGPMcTglO5XxQ/rwxNICm37DI27PwwRWIIwxx0FEuOnUXLYVV/PBNpt+wwtl1Y30S4x1dR9WIIwxx+WiiYNIT4rhcTtY7QlfDyLa1X1YgTDGHJfYqEiumZrF21uK7cK5IFNVZ6I+60EYY0LUtdOziRThrx/aHeeCqbqhmaYWtR6EMSZ09e8Tx+cnDOKF/N1U2ymvQVNec3iiPutBGGNC2I2n5lDV0Gx3nAuisiMFwnoQxpgQNmloXyZkpvCknfIaNAetB2GMCQciwg0zctheUmOnvAbJ4R5Eml0HYYwJdRdNHERaYgxPLC3wOkqvcLgHkWoFwhgT6mKjIrl6ahb/3nyA3eW1Xsfp8cprGomJiiAxJtLV/bhWIEQkTkQ+EpG1IrJBRO4O0CZWRJ4TkW0islxEcvzW3eks3yIi57uV0xjTPeZOzyJChKeW2SmvbiuvaaRfQgwi4up+OlUgRCRRRCKc56NE5BIROdbh8wbgLFWdCEwC5ojI9DZt5gEHVXUEcD9wj7OPscBVwDhgDvBHEXG3VBpjTsiglHjOHzeAZ1fspq6xxes4PVp5ECbqg873IP4DxInIEOAt4DrgiY42UJ9q52W082h7isOlwJPO8xeBs8VXEi8FnlXVBlXdCWwDpnYyqzHGIzfMyKGirolX1tgsr24Kxkyu0PkCIapaC3wR+KOqXoHvr/uONxKJFJE1QDGwWFWXt2kyBNgNoKrNQAWQ5r/cscdZZowJYVNz+zFmkM3y6rZgzOQKXSgQIjIDmAv801l2zCEfVW1R1UlAJjBVRMYfX8wOg80XkXwRyS8pKenutzfGdIFvltccNhdV8eEOO+XVLeXVoVUgvg3cCSxQ1Q0iMgxY0tmdqOohp/2cNqv2AkMBRCQKSAHK/Jc7Mp1lgd77EVXNU9W8jIyMzkYyxrjkkkmDSU2I5okPCryO0iM1NrdS1dAcOgVCVd9V1UtU9R7nYHWpqn6zo21EJENE+jrP44Fzgc1tmi0EbnCeXw68rb5+6ULgKucsp1xgJPBRpz+VMcYzcdGRXDMti8Wb7JRXN5TVNACQnuTuVdTQ+bOYnhGRPiKSCKwHNorIHcfYbBCwRETWASvwHYN4TUR+KiKXOG0eBdJEZBtwO/B9AFXdADwPbAQWAV9XVTstwpgwcd30HCJEeNIunOt2pVXOVdRJ7vcgojrZbqyqVorIXOANfL/IVwL3treBqq4DJgdY/mO/5/XAFe1s/wvgF53MZ4wJIQNT4rhg/ECey9/Nd84dRWJsZ3/VmGMprQ6xHgQQ7Vz3cBmwUFWbOPqUVWOMOeKmmblU1dssr92txCkQGSFUIP4EFACJwH9EJBuodCuUMSb8Tcnqy8TMFB5fWkBrq/092V2O9CCSQ+cg9QOqOkRVL3QugNsFnOlyNmNMGBMRbpyZw46SGj7YXup1nB6jtKqRhJhIEmLcH7br7EHqFBG57/D1BiLyW3y9CWOMadeFJw8iPSnGDlZ3o9LqhqAcf4DODzE9BlQBVzqPSuBxt0IZY3qGT2d5LbZTXruJr0C4P7wEnS8Qw1X1LlXd4TzuBoa5GcwY0zNcM803y+vfbJbXbhGKPYg6EZl1+IWIzATq3IlkjOlJBqXEM2fcQJ6zWV67RVl1I+nJoVUgbgX+ICIFIlIA/B74qmupjDE9yvUzsqmoa2LhWpvl9UQ0t7RSXtsYWj0IVV3r3NdhAjBBVScDZ7mazBjTY0zN7cfogck8sXSXzfJ6AsprG1El5I5BAKCqlap6+PqH213IY4zpgUSEG07NYdP+SlYUHPQ6Ttg6PM1GSPUg2uHuve6MMT3KZZOG0Dchmj+/t8PrKGErmNNswIkVCOsnGmM6LT4mkutn5LB44wG2FVd5HScsfVogQmCISUSqRKQywKMKGByUhMaYHuOGGdnERUfwyH+sF3E8Pp1mIwR6EKqarKp9AjySVdWmZzTGdElaUixX5g1lweq9FFXUex0n7JRWNxITFUFykGbHPZEhJmOM6bKvzB5GS6vy+Ac7vY4SdkqrGshIikUkOIeArUAYY4JqaL8EPj9hME8vL6SirsnrOGGltKYxaMcfwAqEMcYDXz1tGNUNzTyzvNDrKGGltCp402yAFQhjjAfGD0lh9sh0HvtgJ/VNNv1GZ5VWNwTlVqOHWYEwxnji1tOHU1LVwILVNv1GZ7S2KmU1wZtmA1wsECIyVESWiMhGEdkgIt8K0OYOEVnjPNaLSIuI9HPWFYjIx866fLdyGmO8cerwNE4eksKf3t1Oi91x7pgO1TXR0qo9o0AAzcB3VXUsMB34uoiM9W+gqveq6iRVnQTcCbyrquV+Tc501ue5mNMY4wER4WtnDKegrJY3NxR5HSfkBfsaCHCxQKjqflVd5TyvAjYBQzrY5Grg727lMcaEnvPHDSQ3PZGH391uk/gdQ2lVcK+ihiAdgxCRHGAysLyd9QnAHOAlv8UKvCUiK0VkfgfvPf/wrVBLSkq6L7QxxnWREcJXZg9j3Z4Klm4v8zpOSCtxehAZPWSICQARScL3i//bfjPBtnUx8EGb4aVZqjoFuADf8NRpgTZU1UdUNU9V8zIyMro1uzHGfV+cMoT0pFgefne711FCWml1cGdyBZcLhIhE4ysOT6vqyx00vYo2w0uqutf5txhYAEx1K6cxxjtx0ZHcPCuH9z4pZf3eCq/jhKyy6gaiIoSU+Oig7dPNs5gEeBTYpKr3ddAuBTgdeMVvWaKIJB9+DpwHrHcrqzHGW3OnZZMQE8ljNv1Guw5fAxEREbw7LbjZg5gJXAec5Xcq64UicquI3OrX7gvAW6pa47dsAPC+iKwFPgL+qaqLXMxqjPFQSnw0V5ySyatr91FcZZP4BVJa3UhaYvCGlwBcmxJQVd+nEzcVUtUngCfaLNsBTHQlmDEmJN04M5e/LtvFU8sKuf3cUV7HCTml1Q1BPcUV7EpqY0yIyE1P5OzR/Xl62S6bfiMA3zxMwTvFFaxAGGNCyM0zcymraWThmn1eRwkpqkppdWNQT3EFKxDGmBAyY3gaowcm89gHO+3COT+V9c00trQG9RRXsAJhjAkhIsLNs3LZXFTFh3bh3BGfTrNhQ0zGmF7skomDSU+K4ZH37L7Vh306zYb1IIwxvVhcdCQ3zczlnS0lduGco6wm+FdRgxUIY0wIum5GNslxUfxhyTavo4SEEutBGGOMT5+4aG6YkcOiDUVsK67yOo7n9hysJSYqgrREOwZhjDHcPCuXuKhI/rjEJvHbVVZLVr+EoE6zAVYgjDEhql9iDNdMy+KVtfsoLKv1Oo6nCstrye6XEPT9WoEwxoSs+acNI1KEh3rxVOCqSmF5LVlpViCMMeaIAX3iuCIvkxdX7mbvoTqv43iipLqB2sYW60EYY0xbt505AhHhd4u3eh3FE4eH17LTEoO+bysQxpiQNqRvPNdPz+alVXv45EDvO6OpsNxXIGyIyRhjArjtzBEkxERx75tbvI4SdLvKahGBzNT4oO/bCoQxJuT1S4xh/mnDeGvjAVYVHvQ6TlAVltcyqE8csVGRQd+3FQhjTFiYNyuX9KQY7nljc6+a6XVXWY0nw0tgBcIYEyYSY6P4xlkjWb6znHe3lngdJ2h810AE/wA1WIEwxoSRq6dmkdUvgV++vpmW1p7fi6huaKa0urHn9SBEZKiILBGRjSKyQUS+FaDNGSJSISJrnMeP/dbNEZEtIrJNRL7vVk5jTPiIiYrg+xeMZsuBKp7P3+11HNd9eoprDysQQDPwXVUdC0wHvi4iYwO0e09VJzmPnwKISCTwB+ACYCxwdTvbGmN6mQvGDyQvO5XfvrWV6oZmr+O46vAprj1uiElV96vqKud5FbAJGNOyqwgAABHESURBVNLJzacC21R1h6o2As8Cl7qT1BgTTkSEH100ltLqBh5+p2dPwVFYXgN4cw0EBOkYhIjkAJOB5QFWzxCRtSLyhoiMc5YNAfz7j3top7iIyHwRyReR/JKS3nPgypjebNLQvlw6aTB/fm8H+3rwFBy7ympJiY8mJT7ak/27XiBEJAl4Cfi2qla2Wb0KyFbVicCDwD+6+v6q+oiq5qlqXkZGxokHNsaEhTvOPwkFfr1os9dRXFNYXuvZ8QdwuUCISDS+4vC0qr7cdr2qVqpqtfP8dSBaRNKBvcBQv6aZzjJjjAEgMzWBW2bl8o81+3rsrUkP3wfCK26exSTAo8AmVb2vnTYDnXaIyFQnTxmwAhgpIrkiEgNcBSx0K6sxJjzdesZw+iXG8L+vb+pxF881tbSy91Bdj+1BzASuA87yO431QhG5VURuddpcDqwXkbXAA8BV6tMM/BfwJr6D28+r6gYXsxpjwlCfuGi+edYIlm4v450edvHcvkN1tLSqZ2cwAUS59caq+j7Q4f3xVPX3wO/bWfc68LoL0YwxPcg107J5fGkBv3p9M6eNzCAyyLfldMuuMu9mcT3MrqQ2xoS1mKgIvne+7+K5l1bt8TpOtzlyDYQVCGOMOX4XnjyQiUP7ct9bW6lrbPE6TrcoLK8lJiqCAclxnmWwAmGMCXsiwg8uGE1RZT2PfbDT6zjdYldZDUNT44nwcMjMCoQxpkeYNiyNc8cO4KF3tlNa3eB1nBO2q6zWk9uM+rMCYYzpMb5/wWjqmlr4v3994nWUE9LSquwsrWF4hhUIY4zpFsMzkpg7LYtnPipkW3G113GOW2F5LQ3NrYwckOxpDisQxpge5VtnjyQ+OpJ7wngKjq0HqgAYZQXCGGO6T1pSLF87YziLNx5g2Y4yr+Mcl0+cAjGyf5KnOaxAGGN6nHmzchmUEsfP/7kxLO88t/VANUP6xpMY69q1zJ1iBcIY0+PERUfygwvHsH5vJc8s3+V1nC7beqCKUQO87T2AFQhjTA910YRBzByRxr1vbgmr016bW1rZUVLj+fEHsAJhjOmhRIS7LxlPXVMLv3w9fA5YF5TV0tji/RlMYAXCGNODjeifxC2zh/HSqj2sKCj3Ok6nfHLkDCYbYjLGGFd946wRDE6J43/+sZ6mllav4xzT1gO+6zdGeHwGE1iBMMb0cAkxUfz44nFsLqrioXe2ex3nmLYWVzG0XzwJMd6ewQRWIIwxvcCc8QO5ZOJgHvj3J3y8J7RvT/rJgSpG9ff++ANYgTDG9BI/u3Q86UmxfOf5NdQ3heaU4E0trewsrQmJA9RgBcIY00ukJERz7xUT2FZcza8XbfE6TkAFpTU0tWhIHKAGFwuEiAwVkSUislFENojItwK0mSsi60TkYxFZKiIT/dYVOMvXiEi+WzmNMb3H7JEZXD8jm8c+2MnSbaVexznK4QPUoXANBLjbg2gGvquqY4HpwNdFZGybNjuB01X1ZOBnwCNt1p+pqpNUNc/FnMaYXuTOC8YwLD2RO15cR2V9k9dxPmPrgSpEfLPShgLXCoSq7lfVVc7zKmATMKRNm6WqetB5uQzIdCuPMcYAxMdE8psrJ7K/oo6fvbrR6zif8UlxFVn9EoiPifQ6ChCkYxAikgNMBpZ30Gwe8IbfawXeEpGVIjLfvXTGmN5mSlYqXztjOC+s3MPijQe8jnPE1gPVjAyRM5ggCAVCRJKAl4Bvq2plO23OxFcg/ttv8SxVnQJcgG946rR2tp0vIvkikl9SUtLN6Y0xPdW3zh7FmEF9uPPldZSFwFxNjc2tFJTWhMwBanC5QIhINL7i8LSqvtxOmwnAX4BLVfXI5O2qutf5txhYAEwNtL2qPqKqeaqal5GR0d0fwRjTQ8VERXD/lydSWdfMDxesR9XbacF3ltbQ3KqcNLAX9CBERIBHgU2qel87bbKAl4HrVHWr3/JEEUk+/Bw4D1jvVlZjTO80emAfbj9vFIs2FPHMR4WeZvnImStq3OAUT3P4c/Na7pnAdcDHIrLGWfYDIAtAVR8GfgykAX/01ROanTOWBgALnGVRwDOqusjFrMaYXmr+7GF8sK2Uu1/dyOShqYwd3MeTHG9tKCI3PZHhGYme7D8Q1wqEqr4PyDHa3ALcEmD5DmDi0VsYY0z3iogQ7v/yJC78v/f4r2dWsfAbs0gK8p3cKuubWLajjJtm5uL8YRwS7EpqY0yvl54UywNXT6agrIYfLfg46Mcj3t1SQlOLct7YAUHd77FYgTDGGGD6sDS+c84o/rFmH08vD+7xiLc2HiAtMYbJWalB3e+xWIEwxhjHbWeO4MyTMrhr4Qbe/yQ4U3E0NrfyzuZizh7Tn8iI0BleAisQxhhzRGSE8MDVkxmRkcTXnl7JtuJq1/e5fGcZVQ3NnDd2oOv76iorEMYY4yc5Lpq/3JBHbFQE855cwcGaRlf399aGA8RHRzJrZLqr+zkeViCMMaaNof0S+NN1eeyvqOe2p1fR2urOQWtVZfHGA5w2Kp246NCYf8mfFQhjjAnglOxUfn7peD7cUcaTHxZ0y3serGnka0+t5LdvbWFbcTUf762gqLKec0NweAncvVDOGGPC2hV5mSzaUMQ9izZz5kn9yUk//ovYmlpaue3pVawoKOfNDUU8+PY2+iXGECFw9uj+3Zi6+1gPwhhj2iEi/O8XTiY6MoLvvbjuhIaafvHPTXy4o4x7vjSBZXeezY8+P4bM1Hi+MDmT1MSYbkzdfawHYYwxHRiYEsePLxrLHS+u44mlBdw8K7fL7/F8/m6eWFrAvFm5fOkU321vbpk9jFtmD+vuuN3KehDGGHMMl5+SyVmj+/PrNzezuSjgXQvatXxHGT9asJ5ZI9K584LRLiV0hxUIY4w5hsNDTSnx0Vz7l+Wdvj7ipZV7uO7Rj8hMjefBqycTFRlev3LDK60xxnhkYEocz3xlOiBc8+dl7CytabdtS6vyyzc28d0X1pKXk8rLt50asscZOmIFwhhjOml4RhLPfGUaza3KNX9exq6yo4vEvkN1zHtyBX96dwfXTs/iyZun0jch/IoDWIEwxpguGTUgmafmTaOuqYVz7/8PP1jwMQWlNTS1tPLIf7Zzzn3vsmxHGT+7bDw/v8x3BlS4Eq9vs9ed8vLyND8/3+sYxpheYHd5LX98ZzsvrdxDc2srA/vEsa+inrNH9+cnl4xjaL8EryN2ioisdG7UdvQ6KxDGGHP8iivrefSDnazedYhbZudy7tgBIXXTn2PpqEDYdRDGGHMC+veJ484LxngdwxXhOzhmjDHGVa4VCBEZKiJLRGSjiGwQkW8FaCMi8oCIbBORdSIyxW/dDSLyifO4wa2cxhhjAnNziKkZ+K6qrhKRZGCliCxW1Y1+bS4ARjqPacBDwDQR6QfcBeQB6my7UFUPupjXGGOMH9d6EKq6X1VXOc+rgE3AkDbNLgX+qj7LgL4iMgg4H1isquVOUVgMzHErqzHGmKMF5RiEiOQAk4HlbVYNAXb7vd7jLGtveaD3ni8i+SKSX1JS0l2RjTGm13O9QIhIEvAS8G1V7dosV52gqo+oap6q5mVkZHT32xtjTK/laoEQkWh8xeFpVX05QJO9wFC/15nOsvaWG2OMCRI3z2IS4FFgk6re106zhcD1ztlM04EKVd0PvAmcJyKpIpIKnOcsM8YYEySuXUktIrOA94CPgVZn8Q+ALABVfdgpIr/HdwC6FrhJVfOd7W922gP8QlUf78Q+S4BdQApQ4Sw+1vPD/6YDpV38mP7v19l1bZd3Np//sq5m7Shne+s7ynmsrG59pyeaNRz/+4dTVvtZ7b6swfzvn62qgcfnVbXHPYBHOvvc79/8E9lPZ9e1Xd7ZfCeStaOc7a3vKGcnvktXvtMTzRqO//3DKav9rIbnz2pHj556JfWrXXjuv+xE9tPZdW2XdyXf8WY91naB1neUs+3rtlnd+k7bW9/ZrOH439//eahntZ/VY68PxZ/VdvWoyfpOhIjkazsTVoWacMkaLjnBsrolXLKGS04Ibtae2oM4Ho94HaALwiVruOQEy+qWcMkaLjkhiFmtB2GMMSYg60EYY4wJyAqEMcaYgKxAGGOMCcgKRCeIyGwReVhE/iIiS73O0x4RiRCRX4jIg6F+Dw0ROUNE3nO+1zO8znMsIpLoTAp5kddZ2iMiY5zv80UR+ZrXeToiIpeJyJ9F5DkROc/rPB0RkWEi8qiIvOh1lkCcn80nne9zbne+d48vECLymIgUi8j6NsvniMgW52ZF3+/oPVT1PVW9FXgNeDJUc+KbPj0TaMI3A64ruimrAtVAXBhkBfhv4Hl3Unbbz+km5+f0SmBmiGf9h6p+BbgV+HKIZ92hqvPcyhhIF3N/EXjR+T4v6dYgx3N1XTg9gNOAKcB6v2WRwHZgGBADrAXGAifjKwL+j/5+2z0PJIdqTuD7wFedbV8M5e8UiHC2G4BvMsdQznoucBVwI3BRqOZ0trkEeAO4JpS/U7/tfgtMCZOsrv0/dYK57wQmOW2e6c4cbt5RLiSo6n+c+1H4mwpsU9UdACLyLHCpqv4SCDiEICJZ+CYTrArVnCKyB2h0Xra4kbO7svo5CMS6kRO67Xs9A0jE9z9jnYi8rqqtbdt5ndN5n4XAQhH5J/BMd2bszqzOPGy/At5Q58ZioZrVC13Jja8HngmsoZtHhXp8gWhHoBsSTTvGNvOAY04Y2M26mvNl4EERmQ38x81gAXQpq4h8Ed+dA/vim7AxmLqUVVV/CCAiNwKl3V0cOtDV7/QMfMMNscDrriY7Wld/Vr8BnAOkiMgIVX3YzXBtdPV7TQN+AUwWkTudQuKF9nI/APxeRD7PiU3HcZTeWiC6TFXv8jrDsahqLb5CFvLUd3+QQPcICVmq+oTXGTqiqu8A73gco1NU9QF8v9hCnqqW4TtWEpJUtQa4yY337vEHqdsRLjckCpecYFndEC45wbIGQ9Bz99YCsQIYKSK5IhKD7wDkQo8zBRIuOcGyuiFccoJlDYbg5w7WUXmvHsDfgf18eurnPGf5hcBWfGcF/NByWlbLaVlD5REquW2yPmOMMQH11iEmY4wxx2AFwhhjTEBWIIwxxgRkBcIYY0xAViCMMcYEZAXCGGNMQFYgTI8mItVB3l+33C9EfPfLqBCRNSKyWUR+04ltLhORsd2xf2PACoQxXSIiHc5fpqqnduPu3lPVScBk4CIROdY9Hi7DN+OsMd3CCoTpdURkuIgsEpGV4rur3Whn+cUislxEVovIv0RkgLP8JyLyNxH5APib8/oxEXlHRHaIyDf93rva+fcMZ/2LTg/gaWeKa0TkQmfZShF5QERe6yivqtbhm8p5iLP9V0RkhYisFZGXRCRBRE7Fdy+Ie51ex/D2PqcxnWUFwvRGjwDfUNVTgP8H/NFZ/j4wXVUnA88C3/PbZixwjqpe7bwejW+68qnAXSISHWA/k4FvO9sOA2aKSBzwJ+ACZ/8ZxworIqnASD6dwv1lVf2cqk4ENuGbhmEpvnl57lDVSaq6vYPPaUyn2HTfplcRkSTgVOAF5w96+PSGRZnAcyIyCN8du3b6bbrQ+Uv+sH+qagPQICLF+O6M1/bWqR+p6h5nv2uAHHy3Wd2hqoff++/A/HbizhaRtfiKw+9UtchZPl5Efo7vXhpJwJtd/JzGdIoVCNPbRACHnLH9th4E7lPVhc7Nd37it66mTdsGv+ctBP5/qTNtOvKeql4kIrnAMhF5XlXXAE8Al6nqWucmRmcE2Lajz2lMp9gQk+lVVLUS2CkiV4Dv1pciMtFZncKn8+vf4FKELcAwv9tJfvlYGzi9jV8B/+0sSgb2O8Nac/2aVjnrjvU5jekUKxCmp0sQkT1+j9vx/VKd5wzfbMB3X1/w9RheEJGVQKkbYZxhqtuARc5+qoCKTmz6MHCaU1j+B1gOfABs9mvzLHCHc5B9OO1/TmM6xab7NibIRCRJVauds5r+AHyiqvd7ncuYtqwHYUzwfcU5aL0B37DWnzzOY0xA1oMwxhgTkPUgjDHGBGQFwhhjTEBWIIwxxgRkBcIYY0xAViCMMcYEZAXCGGNMQP8f8kaBuFzi68EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Learner.fine_tune??\n",
    "\n",
    "```\n",
    "def fine_tune(self:Learner, epochs, base_lr=2e-3, freeze_epochs=1, lr_mult=100,\n",
    "              pct_start=0.3, div=5.0, **kwargs):\n",
    "    \"Fine tune with `freeze` for `freeze_epochs` then with `unfreeze` from `epochs` using discriminative LR\"\n",
    "    self.freeze()\n",
    "    self.fit_one_cycle(freeze_epochs, slice(base_lr), pct_start=0.99, **kwargs)\n",
    "    base_lr /= 2\n",
    "    self.unfreeze()\n",
    "    self.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div, **kwargs)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<h4 id=\"Learner.fit_flat_cos\" class=\"doc_header\"><code>Learner.fit_flat_cos</code><a href=\"https://github.com/fastai/fastai/tree/master/fastai/callback/schedule.py#L128\" class=\"source_link\" style=\"float:right\">[source]</a></h4>\n",
       "\n",
       "> <code>Learner.fit_flat_cos</code>(**`n_epoch`**, **`lr`**=*`None`*, **`div_final`**=*`100000.0`*, **`pct_start`**=*`0.75`*, **`wd`**=*`None`*, **`cbs`**=*`None`*, **`reset_opt`**=*`False`*)\n",
       "\n",
       "Fit `self.model` for `n_epoch` at flat `lr` before a cosine annealing."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_doc(Learner.fit_flat_cos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same can be rewritten using flat cosine annealing. @patch functionality ties it with the Learner. Read more about [patch](https://fastcore.fast.ai/basics.html#patch) . patch decorator adds the decorated function to the class defined in first parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@patch\n",
    "def fine_tune_flat_cos(self: Learner, epochs, base_lr=2e-3, freeze_epochs=1, lr_mult=100, pct_start=0.3, div=5.0, **kwargs):\n",
    "  '''Fine tune with `freeze` for `freeze_epochs` then with `unfreeze` from `epochs` using discriminative LR using flat cosine annealing'''\n",
    "  self.freeze()\n",
    "  self.fit_flat_cos(freeze_epochs, slice(base_lr), pct_start=0.99, **kwargs)\n",
    "  base_lr /= 2\n",
    "  self.unfreeze()\n",
    "  self.fit_flat_cos(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div, **kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.339328</td>\n",
       "      <td>0.220674</td>\n",
       "      <td>0.934080</td>\n",
       "      <td>02:58</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.396408</td>\n",
       "      <td>0.390391</td>\n",
       "      <td>0.880886</td>\n",
       "      <td>03:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.412575</td>\n",
       "      <td>0.349524</td>\n",
       "      <td>0.898956</td>\n",
       "      <td>03:45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.278063</td>\n",
       "      <td>0.359218</td>\n",
       "      <td>0.896920</td>\n",
       "      <td>03:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.142108</td>\n",
       "      <td>0.265442</td>\n",
       "      <td>0.923136</td>\n",
       "      <td>03:45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.072858</td>\n",
       "      <td>0.237780</td>\n",
       "      <td>0.930262</td>\n",
       "      <td>03:45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if FINE_TUNE:\n",
    "  learn.fine_tune_flat_cos(5, 2e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(#2) [0.23778019845485687,0.9302621483802795]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.validate(dl=dls.valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "('Shih-Tzu',\n",
       " TensorImage(9),\n",
       " TensorImage([5.8873e-04, 4.8669e-04, 3.7679e-04, 9.7221e-04, 8.8172e-04, 2.8819e-03,\n",
       "         4.1782e-04, 3.6273e-04, 9.0794e-04, 9.9212e-01]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item = get_image_files(path)[0]\n",
    "learn.predict(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Returning the classname, class index & probabilities"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}