mnist-pytorch-for-beginners-detailed-desc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST using PyTorch for Beginners (Detailed Descriptions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PyTorch\n",
    "\n",
    "PyTorch is a famous python framework for deep learning. It's gaining interest in the deep learning community because of its syntactical similarities with python and operational similarities with numpy - which is the fundamental package for scientific and mathematical computing in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Learning\n",
    "Deep Leaarning is a subset of Machine Learning (Shallow Learning) that, generally, requires huge amount of data in order to puke out good results. However, this statement isn't enitrely true as we will soon see, in this notebook itself, that having huge datasets isn't always a necessary criteria to apply deep learning solutions to specific problems.\n",
    "\n",
    "### What is a Deep Learnig solution anyway?\n",
    "To keep things simple, just remember the following equation\n",
    "\n",
    "`Deep Learning = Neural Networks with more than 3 layers`\n",
    "\n",
    "### Neural Networks\n",
    "Neural Network is, usually, a supervised method of learning. By supervised I mean, we know the target variable and its values for particular records and hence we can feed in this data to machine learning and deep learning algorithms and specify what it needs to learn. In contrast, the unsupervised learning methods require algorithms to just 'figure out' what are the relations between the records in the data that is fed into them.\n",
    "\n",
    "Following 'layers' consists of a Neural Network\n",
    "1. Input Layer\n",
    "2. Hidden Layers\n",
    "3. Output Layer\n",
    "\n",
    "Every network has 1 input and 1 Output layer but can have any number of hidden layers, so in order to implement deep learning, we need at least 2 hidden layers in out network.\n",
    "\n",
    "Details about a Neural Network, their different types and architecture is an active research topic and are already well defined and explained in numerous blogs and videos around the internet. Following are few links that I found relatively easy to understand:\n",
    "1. 3b1b's neural net video: https://www.youtube.com/watch?v=aircAruvnKk&t=9s\n",
    "2. Medium neural net blog: https://towardsdatascience.com/first-neural-network-for-beginners-explained-with-code-4cfd37e06eaf\n",
    "3. Neural Network's Chart: https://towardsdatascience.com/the-mostly-complete-chart-of-neural-networks-explained-3fb6f2367464\n",
    "\n",
    "P.S. I really enjoyed the DeepLizard.com's course on PyTorch. They have really started from the very basic and have ended on a fairly complex note and the learning was so intuitive, probably because they have covered every minute detail that goes on in PyTorch as well as Neural Networks and even Python. Just leaving their course's link here for anyone to get started with... something!\n",
    "\n",
    "Link: https://deeplizard.com/learn/video/v5cngxo4mIg\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MNIST Dataset\n",
    "\n",
    "We are using the most famous Dataset for Computer Vision provided by the Turing Award holder and a very prominent personality in deep learning community: Dr Yann LeCun.\n",
    "\n",
    "The dataset consists of 28x28 pixel, greyscale images. The rest of the details are given clearly in the description of this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation\n",
    "We will be implementing a specific type of Neural Network called <b>'Convolutional Neural Network (CNN)'</b> and we'll follow the following steps:\n",
    "\n",
    "1. Import the libraries\n",
    "2. Load the dataset\n",
    "3. Splitting features and target variables\n",
    "4. Split the data into train and test sets\n",
    "5. Rescaling Values\n",
    "6. Converting to torch tensors\n",
    "7. Dataset and DataLoader\n",
    "8. Define Neural Network Architecture\n",
    "    Input Layer\n",
    "    Convolution Layer\n",
    "    Pooling Layer\n",
    "    Activation Function\n",
    "    Regularization\n",
    "    Linear Layer\n",
    "9. Loss Function\n",
    "10. Optimiser\n",
    "11. Train the Neural Network\n",
    "12. Predict the labels on actual test data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Import the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda:0\n"
     ]
    }
   ],
   "source": [
    "#PyTorch Specific libraries\n",
    "import torch\n",
    "from torch import nn, optim\n",
    "import torch.nn.functional as F\n",
    "from torch.autograd import Variable\n",
    "\n",
    "#Data manipulation and visualisation specific libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# For splitting the data into Train and Test set\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# This piece of code is required to make use of the GPU instead of CPU for faster processing\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(device)\n",
    "\n",
    "#If it prints \"cuda:0\" that means it has access to GPU. If it prints out \"cpu\", then it's still running on CPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Load the dataset\n",
    "\n",
    "After loading the dataset with the help of panda's `read_csv()` function, we check if the data is imported correctly or not. This can be done in many ways, the simplest way is to check the number of rows and column. And we can do so with the help of `shape` attribute. It outputs in the format - (rows, columns)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train.shape:\n",
      " (42000, 785)\n",
      "test.shape:\n",
      " (28000, 784)\n"
     ]
    }
   ],
   "source": [
    "train = pd.read_csv('data/train.csv')\n",
    "test = pd.read_csv('data/test.csv')\n",
    "\n",
    "#Let's check if they have been loaded properly\n",
    "print('train.shape:\\n', train.shape)\n",
    "print('test.shape:\\n', test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Splitting features and target variables\n",
    "\n",
    "The label is the last column and rest of the columns are all the features\n",
    "Here, X -> Features; y -> Labels\n",
    "\n",
    "Why to name \"X\" and \"y\"? They are the standard names used by everyone. You could use anything instead of X and y but since it's used by almost everyone, so it makes your code more readable to others since they would instantly know what they are looking at and would then be able to debug your code fast, in case you need help."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = train.iloc[:,:-1]\n",
    "y = train.iloc[:,-1:] #Could have done like this \n",
    "y = train.label.values # but needed to convert it to np.ndarray for torch tensor conversion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, spliting with `train.iloc[:,-1:]` would have resulted the same as `train.labels.values` have.\n",
    "The only difference between the two is, the former would have resulted in a Series and the latter is preserved as ndarray - which is what we want since we would be convertiing this into a PyTorch Tensor in a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X.shape:  (42000, 784) X.type:  <class 'pandas.core.frame.DataFrame'>\n",
      "y.shape:  (42000,) y.type:  <class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "print('X.shape: ', X.shape, 'X.type: ', type(X) )\n",
    "print('y.shape: ', y.shape, 'y.type: ', type(y) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Split the data into train and test sets\n",
    "\n",
    "We need a test set, apart from the original test.csv that is provided, for the purpose of our training and validation. The test.csv doesn't contain the target variable and hence the model can't be trainied on that. The target (label column) is only present in train.csv. Hence, we will split the data in the training set (training.csv) into two parts - training set (for training the model) and test set (for validation). To do so, we will use train_test_split() method from `sklearn.model_selection`. This method takes in the features (X) and label column (y) separately in the first 2 parameters. The 3rd parameter is test_size, which we can set in order to specify with what percentage we want to split our data. We are going to use \"0.1\" as the argument value. This will split our data into 90% training and 10% test sets. The 4th parameter is the `random_state`. This is an optional parameter which you will find as a part of many ML algorithms. It is required to produce same results and eliminate any 'randomness' that the algorithms can take since a lot of the internal values in ML algorithms have to assume some random values as part of their execution.\n",
    "\n",
    "The function outputs 4 values, essentially splitting the X and y in to 2 parts each (test and training sets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train.shape:  (37800, 784)\n",
      "y_train.shape:  (37800,)\n",
      "X_test.shape:  (4200, 784)\n",
      "y_test.shape:  (4200,)\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y , test_size = 0.1, random_state = 1)\n",
    "\n",
    "print('X_train.shape: ', X_train.shape)\n",
    "print('y_train.shape: ', y_train.shape)\n",
    "print('X_test.shape: ', X_test.shape)\n",
    "print('y_test.shape: ', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Rescaling values\n",
    "\n",
    "The pixel values that we have got in the dataset is in the range of (0,255). For neural network (or any machine learning algorithm for that matter) to be efficient, we rescale the values to (0,1). That means, all the values will be between 0 and 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Rescaling values\n",
    "X_train = X_train.values/255\n",
    "X_test = X_test.values/255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Converting to torch tensors\n",
    "\n",
    "PyTorch works with tensors. Tensors can be visualised as multidimensional arrays. Just like numpy arrays, or Tensorflow's tensors. PyTorch has its own form of arrays.\n",
    "\n",
    "torch.from_numpy() function converts a numpy array or a panda's dataframe to torch tensor (remember why we needed 'y' to be a numpy array and not series?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train.dtype: torch.float64\n",
      "X_test.dtype: torch.float64\n",
      "y_train.dtype: torch.int64\n",
      "y_test.dtype: torch.int64\n"
     ]
    }
   ],
   "source": [
    "#Converting to Tensors\n",
    "X_train = torch.from_numpy(X_train)\n",
    "X_test = torch.from_numpy(X_test)\n",
    "\n",
    "y_train = torch.from_numpy(y_train).type(torch.LongTensor)\n",
    "y_test = torch.from_numpy(y_test).type(torch.LongTensor)\n",
    "\n",
    "print('X_train.dtype:', X_train.dtype)\n",
    "print('X_test.dtype:', X_test.dtype)\n",
    "print('y_train.dtype:', y_train.dtype)\n",
    "print('y_test.dtype:', y_test.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Dataset and DataLoader\n",
    "\n",
    "We use Data Loaders because sometimes the datasets can have upto millions of records which may eventually lead to enormous size of those datasets and hence they are generally very difficult to load in memoty all at once. Due to the limited available memory space, we use a technique called batching. Batching is simply accessing records from such huge datasets in `batches` and not all at once.\n",
    "\n",
    "For example, Let's suppose you have 1000 records in a dataset and you want to apply batching on this dataset. So, if you set the batch size to 200, your model will first be trained on the first set of 200 records in 1 iteration of the feedforward network. Then, it will train on the next set of records and so on. So, it will essentially take 5 iterations for the whole dataset to be passed through the feed forward network.\n",
    "\n",
    "Just to clear things:\n",
    "\n",
    "**batch**: Small set of the larger dataset\n",
    "\n",
    "**iteration**: 1 `batch` going from the input layer to the output layer is said to complete one iteration\n",
    "\n",
    "**epochs**: When every record of the dataset is passed through the network, then the network is said to have finised 1 `epoch`\n",
    "\n",
    "Now, the MNIST dataset isn't that large that we have to use dataloader, but since it's a general practice to load the data in batches, so, I am going to implement using dataloader.\n",
    "\n",
    "Head to PyTorch's official documentation on Dataset class, DataLoader and Transforms to know more about the same: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html\n",
    "\n",
    "Now we will make our dataset using TensorDataset class and would set the batch size to be `100`.\n",
    "\n",
    "Then with the help of DataLoader class, that will provide an iterable over our dataset, we will make our `train_loader` and `test_loader`. The shuffle paramtere is set to true in order to specify that the records in a particular batch needs to be randomly selected from the whole dataset and not select in their original sequence as they appear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = torch.utils.data.TensorDataset(X_train, y_train)\n",
    "test = torch.utils.data.TensorDataset(X_test, y_test)\n",
    "\n",
    "batch = 10000\n",
    "# batch = 1000\n",
    "# batch = 100\n",
    "\n",
    "\n",
    "# Set our data loaders\n",
    "train_loader = torch.utils.data.DataLoader(train, batch_size = batch, shuffle = True)\n",
    "test_loader = torch.utils.data.DataLoader(test, batch_size = batch, shuffle = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Define neural network architecture\n",
    "\n",
    "We will be defining the folllowing in the Net() class:\n",
    "\n",
    "* Input Layer\n",
    "* Convolution Layer\n",
    "* Pooling Layer\n",
    "* Activation Function\n",
    "* Regularization\n",
    "* Linear Layer\n",
    "\n",
    "Here's a briefing of everything that's going on in the following cell.\n",
    "\n",
    "**Net() Class:** `Net()` is the main class which inherits `nn.Module` class. Net() class has two methods, namely, `__init__()` and `forward()`.\n",
    "\n",
    "**init() method:** The `__init__()` method, which is one of the 'special methods' that are provided by python, stands for initialisation and as the name tells itself, it is used to initialise everything that we need to. It inherits all the properties of the `nn.Module`'s own `__init__` function when we write `super().__init__()`. All the convolution layers, pooling layers, dropout layers, Linear (Dense/fully connected) layers are defined in this method.\n",
    "\n",
    "**forward() method:** It's the `forward()` method where we set the course of our network and use all the layers that we define in the `__init__` method.\n",
    "\n",
    "When we execute the code, the init method will be called and everything would be initialised. When we'll make the object of the same class, the forward method will be executed.\n",
    "\n",
    "The architecture:\n",
    "\n",
    "`Layer 1: Convolution Layer 1 > Activation Function (RelU) > Pooling Layer 1 > Dropout Layer 1\n",
    "Layer 2: Convolution Layer 2 > Activation Function (RelU) > Pooling Layer 2 > Dropout Layer 2\n",
    "Flatten the output from the 2 layer\n",
    "Layer 3: Linear Layer 1 > Activation Function (RelU) > Dropout Layer 3\n",
    "Layer 4: Linear Layer 2 > Activation Function (RelU) > Dropout Layer 4\n",
    "Layer 5: Output Layer > Activation function (softmax)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(1, 128, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (drop1): Dropout(p=0.3, inplace=False)\n",
      "  (conv2): Conv2d(128, 224, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (pool2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (drop2): Dropout(p=0.4, inplace=False)\n",
      "  (fc3): Linear(in_features=3584, out_features=64, bias=True)\n",
      "  (drop3): Dropout(p=0.4, inplace=False)\n",
      "  (fc4): Linear(in_features=64, out_features=32, bias=True)\n",
      "  (drop4): Dropout(p=0.4, inplace=False)\n",
      "  (fc5): Linear(in_features=32, out_features=10, bias=True)\n",
      "  (softmax): Softmax(dim=1)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "class Net(nn.Module):\n",
    "    \n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.conv1 = nn.Conv2d(1, 128, 5)\n",
    "        self.pool1 = nn.MaxPool2d(2, 2)\n",
    "        self.drop1 = nn.Dropout(p=0.3)\n",
    "        \n",
    "        self.conv2 = nn.Conv2d(128, 224, 5)\n",
    "        self.pool2 = nn.MaxPool2d(2, 2)\n",
    "        self.drop2 = nn.Dropout(p=0.4)\n",
    "        \n",
    "        self.fc3 = nn.Linear(224*4*4, 64)\n",
    "        self.drop3 = nn.Dropout(p=0.4)\n",
    "        \n",
    "        self.fc4 = nn.Linear(64, 32)\n",
    "        self.drop4 = nn.Dropout(p=0.4)\n",
    "        \n",
    "        self.fc5 = nn.Linear(32, 10)\n",
    "        self.softmax = nn.Softmax(dim=1)\n",
    "   \n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = self.drop1(self.pool1(F.relu(self.conv1(x))))\n",
    "        x = self.drop2(self.pool2(F.relu(self.conv2(x))))\n",
    "        \n",
    "        x = x.view(-1,224*4*4)\n",
    "        \n",
    "        x = self.drop3(F.relu(self.fc3(x)))\n",
    "        x = self.drop4(F.relu(self.fc4(x)))\n",
    "        \n",
    "        x = self.softmax(self.fc5(x))\n",
    "        \n",
    "        return x\n",
    "\n",
    "print(Net()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q. Why these many layers? Why not less or more?**\n",
    "\n",
    "A. Choosing the number of layers is up to the network designer (you). It's with hit and trial that you develop an intiution of how many layers (or neurons) a particular problem may require to get to the desired solution. You can play with the number of layers and neurons (numbers like 128, 224 etc) and see how the network is responsding to your changes.\n",
    "\n",
    "**Q. What does Conv2d(1, 128, 5) actually mean?**\n",
    "\n",
    "A. Conv2d() is a function that helps create a convolution layer. The three paramteres mentioned (in order) are: input_channels, output_channels and kernel_size.\n",
    "\n",
    "**input_channels:** input_channels refers to the number of channels the input has. If our dataset contained colored images, this argument would have been equal to 3. But our dataset contains only the black and white pictures (more generally called as the grayscale images), so a grayscale images have only 1 channel. Hence the argument is equal to 1.\n",
    "\n",
    "**output_channels:** output_channels is arbitrary number and can be set to any number you want. It basically sets the number of neurons that its input_channels would be connected to. It also sets the input_channels of the next convolution layer. So, `output_channel (previous layer) = input_channel (next layer)`\n",
    "\n",
    "**kernel_size:** kernel_size is a square matrix of a particular width and height. When we set the third parameter to 5, it basically means we are setting the square kernel of size 5 x 5. A kernel can be thought of as a window (matrix) that slides over our image to extract certain features out of it in order to learn. That's why kernels are also called as feature maps.\n",
    "\n",
    "There's anther parameter that we haven't explicitly defined but by default is set to 1, and that is '**stride**'. A stride means by how much the kernel should slide in order to output the next feature map. `stride = 1` essentially means that it will slide by 1 pixel to right and 1 pixel to down.\n",
    "\n",
    "**Q. What is pooling?**\n",
    "\n",
    "A. Pooling is another type of layer in a neural network that down samples the feature maps created by the convolution layer. It basically summarises a portion of the image in a much lesser space. It is done to reduce variance and computations. But why Max-pooling? It's because max-pooling helps in extracting low-level features like edges, points, etc. There are other type of pooling methods such as average pooling and others.\n",
    "\n",
    "**Q. What is the importance of view() function in the forward() method?**\n",
    "\n",
    "A. The view() function takes in a tensor (for ex a 4D tensor) and outputs a lower dimensional tensor (for example a 1D tensor). This is required because the next set of layers are the linear layers. And linear layers only accept a 1D tensor (a \"flattened\" array) as an input. So, the view function is used to flatten the tensor so that it can then be fed to the linear layers.\n",
    "\n",
    "**Q. What is a Dropout Layer?**\n",
    "\n",
    "A. To help understand what dropout really is, we need to understand what regularisation in general is\n",
    "\n",
    "### Regularization\n",
    "Regularization helps to solve over fitting problem in machine learning. It is nothing but adding a penalty term to the objective function and control the model complexity using that penalty term.\n",
    "\n",
    "Too complex?\n",
    "\n",
    "For now, consider it as something that tells our model to not cross a certain boundry (threshold). Because if it does, it will be overfitted. Overfiting is a problem that machine learning algortihms face when the model is 'too exact' to be used for actual real world data. That simply means that our model, when overfitted, wouldn't be accurate in preidction task when a new datapoint is fed into the model (for example a datapoint from the test dataset). So, in order to avoid overfitting, we use regularization. This is just an intuition. We don't need to get into the math of anything for now.\n",
    "\n",
    "Dropout is one of the regularization methods. Dropout helps avoid overfitting by simply 'switching off' some of the neurons in a particular iteration. Sometimes, the outup of a particular neuron from a particular layer can shoot off to a high value or to a very low value. So, the dropout method helps the network to consider some of the other neurons that it might have been ignoring because of their lower valued outputs. So, by simply turning off some neurons, it makes sure that it takes into account every neuron and hence train in a more vigorous manner and hence avoid overfitting.\n",
    "\n",
    "**Q. What is RelU and softmax?**\n",
    "\n",
    "A. RelU is short for Rectified Linear Unit. RelU and Softmax are both activation functions. What are activation functions, you ask? Well, they can be thought of as something that restricts the values of the neurons after computations to explode to extremes (very high negative or very high positive) values. Activation functions makes sure the values are bounded within a range. For example, the relu function makes sure the values are between (0,max(val)) range, where 'val' is the actual value of the neuron.\n",
    "\n",
    "**Q. But why only relu and Softmax are chosen?**\n",
    "\n",
    "A. You can choose any activation function you want. However, recent researches proves that relu performs well in almost all the cases as comapred to other activation functions. tanh is another very commonly used activation function. We don't have the luxury of choosing just any activation function when it comes to the output layer. We want to choose an activation function according to our desired output. We have chosen softmax because softmax gives out a class score for every class which is basically a probablity distribution that in turn tells us how confident is the neural network about a certain class to be the actual output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note*:**\n",
    "The two modules imported from the torch library: nn and functional can be used interchangeable. Having said that, all the functions that are present in the nn module are there in the functional module as well. The only difference is the names of these funtions.\n",
    "\n",
    "For example, Max pooling function from the functional module can be called as:\n",
    "\n",
    "`F.max_pool2d(x, kernel_size, stride)`\n",
    "\n",
    "and from the nn module:\n",
    "\n",
    "`nn.MaxPool2d(kernel_size, stride)`\n",
    "\n",
    "Why are there two implementations of the same things? It's to suit the coding style of different people. Some prefer a stateful approach while others prefer a more functional approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Loss function\n",
    "\n",
    "A neural network 'learns' by adjusting various parameters through out the network. Weight is one of the paramter that the neural network learns to adjust. Weight is simply the strength of the two adjoining neurons that it is connected to and it is a numeric value. The new weight values are updated duriing backpropagation process. Now, the thing to note here is, how are these new values calculated?\n",
    "\n",
    "The answer is: with the help of loss functions.\n",
    "\n",
    "Loss functions are simply a method to calculate the prediction error of the network. Simply put, it tells the network which weight to be adjusted and how much it needs to be adjusted. Essentially, it calculates if a particular weight value needs to be decreased (subtracting from thhe original value) or increased (adding to the original value) and by how much.\n",
    "\n",
    "There are many loss functions that can be used. The two most common ones are Cross Entropy Loss and NLL Loss for classification problems (just a reminder that we are dealing with a classification problem right now). The choice of which loss function to use is an experimental problem. You can follow <b><a href='https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/'>this</a></b> guide to start using the loss functions and by practice, build your own intiution.\n",
    "\n",
    "For now, I'll be using the Cross Entropy Loss\n",
    "\n",
    "We are now making an object of our Net() class and setting the loss function. We are using the Cross Entropy Loss. You can use any other loss function as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Making an object of the Net class\n",
    "# model = Net()\n",
    "model = Net().to(device)\n",
    "\n",
    "#Loss function\n",
    "criterion = nn.CrossEntropyLoss ()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. Optimizer\n",
    "An optimizer ties the loss function and model parameters together by updating the model in response to the output of the loss criterion. We will be using Adam as our optimizer with a learning rate of 0.0015. Why 0.0015? Again, it's an experimental value. You need to hit and trial with all of these parameters that are changable. After a while dealiing with and constructing neural networks, you'll develop an intiution of which values could possibly work for a particular problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optimizer\n",
    "optimizer = optim.Adam(model.parameters(), lr = 0.0015)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11. Train the Neural Network\n",
    "\n",
    "Now it's finally time to train our network. We will be setting some variablbes first and then start training our network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialising variables\n",
    "epochs = 30\n",
    "steps = 0\n",
    "print_every = 100\n",
    "trainLoss = [] \n",
    "testLoss = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 25/30..  Test Accuracy: 0.972\n"
     ]
    }
   ],
   "source": [
    "for e in range(epochs):\n",
    "    running_loss = 0\n",
    "    for images, labels in train_loader:\n",
    "        steps += 1   # Forward pass\n",
    "        \n",
    "        images = (images.view(-1,1,28,28)).type(torch.DoubleTensor)\n",
    "        optimizer.zero_grad()\n",
    "        log_ps = model(images.type(torch.FloatTensor).to(device))\n",
    "        labels = labels.to(device)\n",
    "        loss = criterion(log_ps, labels)\n",
    "        loss.backward()   # Backward pass\n",
    "        optimizer.step()\n",
    "        \n",
    "        running_loss += loss.item()\n",
    "        if steps % print_every == 0:\n",
    "            test_loss = 0\n",
    "            accuracy = 0\n",
    "\n",
    "            with torch.no_grad():\n",
    "                model.eval()\n",
    "                for images, labels in test_loader:\n",
    "                    images = (images.view(-1,1,28,28)).type(torch.DoubleTensor)\n",
    "                    log_ps = model(images.type(torch.FloatTensor).to(device))\n",
    "                    labels = labels.to(device)\n",
    "                    test_loss += criterion(log_ps, labels)\n",
    "                    ps = torch.exp(log_ps)\n",
    "                    \n",
    "                    top_p, top_class = ps.topk(1, dim = 1)\n",
    "                    equals = top_class == labels.view(*top_class.shape)\n",
    "                    accuracy += torch.mean(equals.type(torch.FloatTensor))\n",
    "\n",
    "            model.train()\n",
    "\n",
    "            trainLoss.append(running_loss/len(train_loader))\n",
    "            testLoss.append(test_loss/len(test_loader))\n",
    "\n",
    "            print(\"Epoch: {}/{}.. \".format(e + 1, epochs),\n",
    "                  \"Test Accuracy: {:.3f}\".format(accuracy/len(test_loader)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are using a `for` loop to go through a total of 30 epochs.\n",
    "\n",
    "The `images` tensor is a 2D tensor, we first convert it into a 4D tensor since our neural network (the Net() class) requires it to be a 4D tensor. Hence we use the `view()` function.\n",
    "\n",
    "`optimizer.zero_grad()` sets all the gradient's values to zero. It's an essential step while training the network.\n",
    "\n",
    "`log_ps = model(images.type(torch.FloatTensor))` simply gives a set of images to the network for training. Here the set is of 100, as we set it earlier while defining the `DataLoader`\n",
    "\n",
    "`loss = criterion(log_ps, labels)` compare the predicted and the actual labels and calculates the loss.\n",
    "\n",
    "`loss.backward()` is the step where the network starts backpropagating in order to adjust the weights and other parameters of the network and start the training process all over again.\n",
    "\n",
    "*Please note that the loss is calculated by the loss function but the weights are **updated** only during backpropagation.\n",
    "*\n",
    "\n",
    "`running_loss += loss.item()` is used to calculate `trainLoss` which eventually would be helpful in plotting the graph of training and validation loss\n",
    "\n",
    "Since we don't want every iteration to get printed on the screen, we only want every 100th iteration to get prinited with the accuracy score, hence we have used an `if` statment for the same.\n",
    "\n",
    "`with torch.no_grad():` the `with` keyword is an elegant way of handling exceptions in python. The wrapper `torch.no_grad` temporarily sets all the `requires_grad` flag to false. Eventually, it will reduce the memory usage and speed up computations.\n",
    "\n",
    "`model.eval()` it indicates the model that nothing new is to be learnt and the model is used for testing.\n",
    "\n",
    "`torch.exp(log_ps)` returns a new tensor with the exponential of the elements of the input tensor\n",
    "\n",
    "`ps.topk(1, dim = 1)` returns a sparse copy of the tensor\n",
    "\n",
    "`labels.view(*top_class.shape)` sees how many of the classes were correct?\n",
    "\n",
    "`torch.mean(equals.type(torch.FloatTensor))` calculate the mean (gets the accuracy for this batch) and add it to the running accuracy for this epoch\n",
    "\n",
    "`model.train()` tells your model that you are training the model. So effectively, layers like dropout etc, which behave different on the train and test procedures, know what is going on and hence can behave accordingly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now visualise the training and validation loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[23], line 4\u001b[0m\n\u001b[1;32m      1\u001b[0m get_ipython()\u001b[38;5;241m.\u001b[39mrun_line_magic(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmatplotlib\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124minline\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m      3\u001b[0m plt\u001b[38;5;241m.\u001b[39mplot(trainLoss, label \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mTraining Loss\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m----> 4\u001b[0m \u001b[43mplt\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mplot\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtestLoss\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlabel\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mValidation Loss\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m      5\u001b[0m plt\u001b[38;5;241m.\u001b[39mlegend(frameon \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m)\n",
      "File \u001b[0;32m~/anaconda3/envs/mnist/lib/python3.8/site-packages/matplotlib/pyplot.py:2812\u001b[0m, in \u001b[0;36mplot\u001b[0;34m(scalex, scaley, data, *args, **kwargs)\u001b[0m\n\u001b[1;32m   2810\u001b[0m \u001b[38;5;129m@_copy_docstring_and_deprecators\u001b[39m(Axes\u001b[38;5;241m.\u001b[39mplot)\n\u001b[1;32m   2811\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mplot\u001b[39m(\u001b[38;5;241m*\u001b[39margs, scalex\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m, scaley\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m, data\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m-> 2812\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mgca\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mplot\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   2813\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mscalex\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mscalex\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mscaley\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mscaley\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2814\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m{\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mdata\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mdata\u001b[49m\u001b[43m}\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mif\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mdata\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mis\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mnot\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43;01melse\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43m{\u001b[49m\u001b[43m}\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/anaconda3/envs/mnist/lib/python3.8/site-packages/matplotlib/axes/_axes.py:1688\u001b[0m, in \u001b[0;36mAxes.plot\u001b[0;34m(self, scalex, scaley, data, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1445\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m   1446\u001b[0m \u001b[38;5;124;03mPlot y versus x as lines and/or markers.\u001b[39;00m\n\u001b[1;32m   1447\u001b[0m \n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m   1685\u001b[0m \u001b[38;5;124;03m(``'green'``) or hex strings (``'#008000'``).\u001b[39;00m\n\u001b[1;32m   1686\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m   1687\u001b[0m kwargs \u001b[38;5;241m=\u001b[39m cbook\u001b[38;5;241m.\u001b[39mnormalize_kwargs(kwargs, mlines\u001b[38;5;241m.\u001b[39mLine2D)\n\u001b[0;32m-> 1688\u001b[0m lines \u001b[38;5;241m=\u001b[39m [\u001b[38;5;241m*\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_get_lines(\u001b[38;5;241m*\u001b[39margs, data\u001b[38;5;241m=\u001b[39mdata, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)]\n\u001b[1;32m   1689\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m line \u001b[38;5;129;01min\u001b[39;00m lines:\n\u001b[1;32m   1690\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39madd_line(line)\n",
      "File \u001b[0;32m~/anaconda3/envs/mnist/lib/python3.8/site-packages/matplotlib/axes/_base.py:311\u001b[0m, in \u001b[0;36m_process_plot_var_args.__call__\u001b[0;34m(self, data, *args, **kwargs)\u001b[0m\n\u001b[1;32m    309\u001b[0m     this \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m args[\u001b[38;5;241m0\u001b[39m],\n\u001b[1;32m    310\u001b[0m     args \u001b[38;5;241m=\u001b[39m args[\u001b[38;5;241m1\u001b[39m:]\n\u001b[0;32m--> 311\u001b[0m \u001b[38;5;28;01myield from\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_plot_args\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    312\u001b[0m \u001b[43m    \u001b[49m\u001b[43mthis\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mambiguous_fmt_datakey\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mambiguous_fmt_datakey\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/anaconda3/envs/mnist/lib/python3.8/site-packages/matplotlib/axes/_base.py:496\u001b[0m, in \u001b[0;36m_process_plot_var_args._plot_args\u001b[0;34m(self, tup, kwargs, return_kwargs, ambiguous_fmt_datakey)\u001b[0m\n\u001b[1;32m    494\u001b[0m     y \u001b[38;5;241m=\u001b[39m _check_1d(xy[\u001b[38;5;241m1\u001b[39m])\n\u001b[1;32m    495\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 496\u001b[0m     x, y \u001b[38;5;241m=\u001b[39m \u001b[43mindex_of\u001b[49m\u001b[43m(\u001b[49m\u001b[43mxy\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m-\u001b[39;49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    498\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39maxes\u001b[38;5;241m.\u001b[39mxaxis \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    499\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39maxes\u001b[38;5;241m.\u001b[39mxaxis\u001b[38;5;241m.\u001b[39mupdate_units(x)\n",
      "File \u001b[0;32m~/anaconda3/envs/mnist/lib/python3.8/site-packages/matplotlib/cbook/__init__.py:1656\u001b[0m, in \u001b[0;36mindex_of\u001b[0;34m(y)\u001b[0m\n\u001b[1;32m   1654\u001b[0m     \u001b[38;5;28;01mpass\u001b[39;00m\n\u001b[1;32m   1655\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m-> 1656\u001b[0m     y \u001b[38;5;241m=\u001b[39m \u001b[43m_check_1d\u001b[49m\u001b[43m(\u001b[49m\u001b[43my\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1657\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (np\u001b[38;5;241m.\u001b[39mVisibleDeprecationWarning, \u001b[38;5;167;01mValueError\u001b[39;00m):\n\u001b[1;32m   1658\u001b[0m     \u001b[38;5;66;03m# NumPy 1.19 will warn on ragged input, and we can't actually use it.\u001b[39;00m\n\u001b[1;32m   1659\u001b[0m     \u001b[38;5;28;01mpass\u001b[39;00m\n",
      "File \u001b[0;32m~/anaconda3/envs/mnist/lib/python3.8/site-packages/matplotlib/cbook/__init__.py:1348\u001b[0m, in \u001b[0;36m_check_1d\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m   1342\u001b[0m \u001b[38;5;66;03m# plot requires `shape` and `ndim`.  If passed an\u001b[39;00m\n\u001b[1;32m   1343\u001b[0m \u001b[38;5;66;03m# object that doesn't provide them, then force to numpy array.\u001b[39;00m\n\u001b[1;32m   1344\u001b[0m \u001b[38;5;66;03m# Note this will strip unit information.\u001b[39;00m\n\u001b[1;32m   1345\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (\u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(x, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mshape\u001b[39m\u001b[38;5;124m'\u001b[39m) \u001b[38;5;129;01mor\u001b[39;00m\n\u001b[1;32m   1346\u001b[0m         \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(x, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mndim\u001b[39m\u001b[38;5;124m'\u001b[39m) \u001b[38;5;129;01mor\u001b[39;00m\n\u001b[1;32m   1347\u001b[0m         \u001b[38;5;28mlen\u001b[39m(x\u001b[38;5;241m.\u001b[39mshape) \u001b[38;5;241m<\u001b[39m \u001b[38;5;241m1\u001b[39m):\n\u001b[0;32m-> 1348\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mnp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43matleast_1d\u001b[49m\u001b[43m(\u001b[49m\u001b[43mx\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1349\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m   1350\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m x\n",
      "File \u001b[0;32m<__array_function__ internals>:200\u001b[0m, in \u001b[0;36matleast_1d\u001b[0;34m(*args, **kwargs)\u001b[0m\n",
      "File \u001b[0;32m~/.local/lib/python3.8/site-packages/numpy/core/shape_base.py:65\u001b[0m, in \u001b[0;36matleast_1d\u001b[0;34m(*arys)\u001b[0m\n\u001b[1;32m     63\u001b[0m res \u001b[38;5;241m=\u001b[39m []\n\u001b[1;32m     64\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m ary \u001b[38;5;129;01min\u001b[39;00m arys:\n\u001b[0;32m---> 65\u001b[0m     ary \u001b[38;5;241m=\u001b[39m \u001b[43masanyarray\u001b[49m\u001b[43m(\u001b[49m\u001b[43mary\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     66\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m ary\u001b[38;5;241m.\u001b[39mndim \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m0\u001b[39m:\n\u001b[1;32m     67\u001b[0m         result \u001b[38;5;241m=\u001b[39m ary\u001b[38;5;241m.\u001b[39mreshape(\u001b[38;5;241m1\u001b[39m)\n",
      "File \u001b[0;32m~/anaconda3/envs/mnist/lib/python3.8/site-packages/torch/_tensor.py:970\u001b[0m, in \u001b[0;36mTensor.__array__\u001b[0;34m(self, dtype)\u001b[0m\n\u001b[1;32m    968\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m handle_torch_function(Tensor\u001b[38;5;241m.\u001b[39m__array__, (\u001b[38;5;28mself\u001b[39m,), \u001b[38;5;28mself\u001b[39m, dtype\u001b[38;5;241m=\u001b[39mdtype)\n\u001b[1;32m    969\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m dtype \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 970\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mnumpy\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    971\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    972\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mnumpy()\u001b[38;5;241m.\u001b[39mastype(dtype, copy\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m)\n",
      "\u001b[0;31mTypeError\u001b[0m: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first."
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGdCAYAAAAxCSikAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAkI0lEQVR4nO3df3iVdeH/8dfZJgOMc8awA2dxjq5SsAEbhjPguoLJSHfRAu3KAsyFXSqKkY3Li+0q4CJ/TNJ0XoSQkoxRicZidVUmwaLJQGHgbIZMJktobpDldtiKM9ve3z/6cj6d2Gi37GzvHZ6P67ovvX+/7/tanmdn99lxGWOMAAAALBY30AMAAAD4XwgWAABgPYIFAABYj2ABAADWI1gAAID1CBYAAGA9ggUAAFiPYAEAANZLGOgB9JWuri69++67GjFihFwu10APBwAA9IIxRqdPn1ZKSori4np+HyVmguXdd9+V3+8f6GEAAIAP4cSJExo7dmyP62MmWEaMGCHp3xfsdrsHeDQAAKA3gsGg/H5/+HW8JzETLGd/DeR2uwkWAAAGmf/1OAcP3QIAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALCe42CprKxUbm6uUlJS5HK5VF5eft7td+/eLZfLdc7U3Nwc3qazs1MrVqxQamqqhg0bpk984hN64IEHZIxxfEEAACD2JDjdob29Xenp6br99tt1880393q/uro6ud3u8LzX6w3/+5o1a7R+/Xpt3rxZaWlpqq6u1qJFi+TxeLR06VKnQwQAADHGcbDk5OQoJyfH8Ym8Xq+SkpK6Xbd3717NnTtXc+bMkSRdccUVeu6557R//37H5wEAALGn355hycjIkM/n0+zZs1VVVRWxbtq0adq1a5feeustSdLrr7+uPXv2nDeMQqGQgsFgxAQAAGKT43dYnPL5fNqwYYOmTJmiUCikjRs3aubMmXr11Vd1zTXXSJIKCgoUDAY1fvx4xcfHq7OzUw899JAWLlzY43GLioq0evXqaA8fAABYwGUu4MlWl8ul7du3a968eY72mzFjhgKBgLZs2SJJ2rp1q+6//349+uijSktLU01Nje677z49/vjjysvL6/YYoVBIoVAoPB8MBuX3+9Xa2hrxrAwAALBXMBiUx+P5n6/fUX+HpTuZmZnas2dPeP7+++9XQUGBvvKVr0iSJk6cqHfeeUdFRUU9BktiYqISExP7ZbwAAGBgDcjfYampqZHP5wvP/+Mf/1BcXORQ4uPj1dXV1d9DAwAAFnL8DktbW5vq6+vD8w0NDaqpqVFycrICgYAKCwvV2Nio0tJSSVJxcbFSU1OVlpamM2fOaOPGjaqoqNCOHTvCx8jNzdVDDz2kQCCgtLQ0vfbaa3r88cd1++2398ElAgCAwc5xsFRXVysrKys8n5+fL0nKy8tTSUmJmpqadPz48fD6jo4OLVu2TI2NjRo+fLgmTZqknTt3Rhxj7dq1WrFihe655x6dOnVKKSkpuuuuu7Ry5coLuTYAABAjLuihW5v09qEdAABgj96+fvNdQgAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6joOlsrJSubm5SklJkcvlUnl5+Xm33717t1wu1zlTc3NzxHaNjY269dZbNWrUKA0bNkwTJ05UdXW10+EBAIAYlOB0h/b2dqWnp+v222/XzTff3Ov96urq5Ha7w/Nerzf87++//76mT5+urKwsvfjii/roRz+qo0ePauTIkU6HBwAAYpDjYMnJyVFOTo7jE3m9XiUlJXW7bs2aNfL7/dq0aVN4WWpqquNzAACA2NRvz7BkZGTI5/Np9uzZqqqqilj3y1/+UlOmTNGXvvQleb1eTZ48Wc8888x5jxcKhRQMBiMmAAAQm6IeLD6fTxs2bFBZWZnKysrk9/s1c+ZMHTp0KLzNsWPHtH79el155ZV66aWXdPfdd2vp0qXavHlzj8ctKiqSx+MJT36/P9qXAgAABojLGGM+9M4ul7Zv36558+Y52m/GjBkKBALasmWLJGnIkCGaMmWK9u7dG95m6dKlOnDggPbt29ftMUKhkEKhUHg+GAzK7/ertbU14lkZAABgr2AwKI/H8z9fvwfkY82ZmZmqr68Pz/t8Pn3qU5+K2Obqq6/W8ePHezxGYmKi3G53xAQAAGLTgARLTU2NfD5feH769Omqq6uL2Oatt97S5Zdf3t9DAwAAFnL8KaG2traId0caGhpUU1Oj5ORkBQIBFRYWqrGxUaWlpZKk4uJipaamKi0tTWfOnNHGjRtVUVGhHTt2hI/xrW99S9OmTdPDDz+sW265Rfv379fTTz+tp59+ug8uEQAADHaOg6W6ulpZWVnh+fz8fElSXl6eSkpK1NTUFPGrnI6ODi1btkyNjY0aPny4Jk2apJ07d0Yc49prr9X27dtVWFio7373u0pNTVVxcbEWLlx4IdcGAABixAU9dGuT3j60AwAA7GH1Q7cAAABOECwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6zkOlsrKSuXm5iolJUUul0vl5eXn3X737t1yuVznTM3Nzd1u/8gjj8jlcum+++5zOjQAABCjHAdLe3u70tPTtW7dOkf71dXVqampKTx5vd5ztjlw4IB++MMfatKkSU6HBQAAYliC0x1ycnKUk5Pj+ERer1dJSUk9rm9ra9PChQv1zDPP6MEHH3R8fAAAELv67RmWjIwM+Xw+zZ49W1VVVeesX7JkiebMmaPs7OxeHS8UCikYDEZMAAAgNkU9WHw+nzZs2KCysjKVlZXJ7/dr5syZOnToUHibrVu36tChQyoqKur1cYuKiuTxeMKT3++PxvABAIAFHP9KyKlx48Zp3Lhx4flp06bp7bff1hNPPKEtW7boxIkT+uY3v6nf/e53Gjp0aK+PW1hYqPz8/PB8MBgkWgAAiFFRD5buZGZmas+ePZKkgwcP6tSpU7rmmmvC6zs7O1VZWakf/OAHCoVCio+PP+cYiYmJSkxM7LcxAwCAgTMgwVJTUyOfzydJmjVrlmprayPWL1q0SOPHj9fy5cu7jRUAAHBxcRwsbW1tqq+vD883NDSopqZGycnJCgQCKiwsVGNjo0pLSyVJxcXFSk1NVVpams6cOaONGzeqoqJCO3bskCSNGDFCEyZMiDjHpZdeqlGjRp2zHAAAXJwcB0t1dbWysrLC82efI8nLy1NJSYmampp0/Pjx8PqOjg4tW7ZMjY2NGj58uCZNmqSdO3dGHAMAAOB8XMYYM9CD6AvBYFAej0etra1yu90DPRwAANALvX395ruEAACA9QgWAABgPYIFAABYj2ABAADWI1gAAID1CBYAAGA9ggUAAFiPYAEAANYjWAAAgPUIFgAAYD2CBQAAWI9gAQAA1iNYAACA9QgWAABgPYIFAABYj2ABAADWI1gAAID1CBYAAGA9ggUAAFiPYAEAANYjWAAAgPUIFgAAYD2CBQAAWI9gAQAA1iNYAACA9QgWAABgPYIFAABYj2ABAADWI1gAAID1CBYAAGA9ggUAAFiPYAEAANYjWAAAgPUIFgAAYD3HwVJZWanc3FylpKTI5XKpvLz8vNvv3r1bLpfrnKm5uTm8TVFRka699lqNGDFCXq9X8+bNU11dneOLAQAAsclxsLS3tys9PV3r1q1ztF9dXZ2amprCk9frDa/7wx/+oCVLluiVV17R7373O33wwQf63Oc+p/b2dqfDAwAAMSjB6Q45OTnKyclxfCKv16ukpKRu1/32t7+NmC8pKZHX69XBgwf12c9+1vG5AABAbOm3Z1gyMjLk8/k0e/ZsVVVVnXfb1tZWSVJycnKP24RCIQWDwYgJAADEpqgHi8/n04YNG1RWVqaysjL5/X7NnDlThw4d6nb7rq4u3XfffZo+fbomTJjQ43GLiork8XjCk9/vj9YlAACAAeYyxpgPvbPLpe3bt2vevHmO9psxY4YCgYC2bNlyzrq7775bL774ovbs2aOxY8f2eIxQKKRQKBSeDwaD8vv9am1tldvtdjQeAAAwMILBoDwez/98/Xb8DEtfyMzM1J49e85Zfu+99+pXv/qVKisrzxsrkpSYmKjExMRoDREAAFhkQIKlpqZGPp8vPG+M0Te+8Q1t375du3fvVmpq6kAMCwAAWMpxsLS1tam+vj4839DQoJqaGiUnJysQCKiwsFCNjY0qLS2VJBUXFys1NVVpaWk6c+aMNm7cqIqKCu3YsSN8jCVLluinP/2pfvGLX2jEiBHhv9Hi8Xg0bNiwC71GAAAwyDkOlurqamVlZYXn8/PzJUl5eXkqKSlRU1OTjh8/Hl7f0dGhZcuWqbGxUcOHD9ekSZO0c+fOiGOsX79ekjRz5syIc23atElf+9rXnA4RAADEmAt66NYmvX1oBwAA2KO3r998lxAAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwnuNgqaysVG5urlJSUuRyuVReXn7e7Xfv3i2Xy3XO1NzcHLHdunXrdMUVV2jo0KG67rrrtH//fqdDAwAAMcpxsLS3tys9PV3r1q1ztF9dXZ2amprCk9frDa97/vnnlZ+fr1WrVunQoUNKT0/XDTfcoFOnTjkdHgAAiEEJTnfIyclRTk6O4xN5vV4lJSV1u+7xxx/XHXfcoUWLFkmSNmzYoF//+td69tlnVVBQ4PhcAAAgtvTbMywZGRny+XyaPXu2qqqqwss7Ojp08OBBZWdn/9+g4uKUnZ2tffv29Xi8UCikYDAYMQEAgNgU9WDx+XzasGGDysrKVFZWJr/fr5kzZ+rQoUOSpPfee0+dnZ0aPXp0xH6jR48+5zmX/1RUVCSPxxOe/H5/VK8DAAAMHMe/EnJq3LhxGjduXHh+2rRpevvtt/XEE09oy5YtH/q4hYWFys/PD88Hg0GiBQCAGBX1YOlOZmam9uzZI0m67LLLFB8fr5MnT0Zsc/LkSY0ZM6bHYyQmJioxMTGq4wQAAHYYkL/DUlNTI5/PJ0kaMmSIPv3pT2vXrl3h9V1dXdq1a5emTp06EMMDAACWcfwOS1tbm+rr68PzDQ0NqqmpUXJysgKBgAoLC9XY2KjS0lJJUnFxsVJTU5WWlqYzZ85o48aNqqio0I4dO8LHyM/PV15enqZMmaLMzEwVFxervb09/KkhAABwcXMcLNXV1crKygrPn32OJC8vTyUlJWpqatLx48fD6zs6OrRs2TI1NjZq+PDhmjRpknbu3BlxjC9/+cv661//qpUrV6q5uVkZGRn67W9/e86DuAAA4OLkMsaYgR5EXwgGg/J4PGptbZXb7R7o4QAAgF7o7es33yUEAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArOc4WCorK5Wbm6uUlBS5XC6Vl5f3et+qqiolJCQoIyMjYnlnZ6dWrFih1NRUDRs2TJ/4xCf0wAMPyBjjdHgAACAGOQ6W9vZ2paena926dY72a2lp0W233aZZs2ads27NmjVav369fvCDH+jNN9/UmjVr9L3vfU9r1651OjwAABCDEpzukJOTo5ycHMcnWrx4sRYsWKD4+Phz3pXZu3ev5s6dqzlz5kiSrrjiCj333HPav3+/4/MAAIDY0y/PsGzatEnHjh3TqlWrul0/bdo07dq1S2+99ZYk6fXXX9eePXvOG0ahUEjBYDBiAgAAscnxOyxOHT16VAUFBXr55ZeVkND96QoKChQMBjV+/HjFx8ers7NTDz30kBYuXNjjcYuKirR69epoDRsAAFgkqu+wdHZ2asGCBVq9erWuuuqqHrd74YUX9JOf/EQ//elPdejQIW3evFmPPfaYNm/e3OM+hYWFam1tDU8nTpyIxiUAAAALuMwFfBTH5XJp+/btmjdvXrfrW1paNHLkSMXHx4eXdXV1yRij+Ph47dixQ9dff738fr8KCgq0ZMmS8HYPPvigfvzjH+vIkSO9GkswGJTH41Fra6vcbveHvSQAANCPevv6HdVfCbndbtXW1kYse+qpp1RRUaFt27YpNTVVkvSPf/xDcXGRb/bEx8erq6srmsMDAACDhONgaWtrU319fXi+oaFBNTU1Sk5OViAQUGFhoRobG1VaWqq4uDhNmDAhYn+v16uhQ4dGLM/NzdVDDz2kQCCgtLQ0vfbaa3r88cd1++23X8ClAQCAWOE4WKqrq5WVlRWez8/PlyTl5eWppKRETU1NOn78uKNjrl27VitWrNA999yjU6dOKSUlRXfddZdWrlzpdHgAACAGXdAzLDbhGRYAAAaf3r5+811CAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsJ7jYKmsrFRubq5SUlLkcrlUXl7e632rqqqUkJCgjIyMc9Y1Njbq1ltv1ahRozRs2DBNnDhR1dXVTocHAABikONgaW9vV3p6utatW+dov5aWFt12222aNWvWOevef/99TZ8+XZdccolefPFFHT58WN///vc1cuRIp8MDAAAxKMHpDjk5OcrJyXF8osWLF2vBggWKj48/512ZNWvWyO/3a9OmTeFlqampjs8BAABiU788w7Jp0yYdO3ZMq1at6nb9L3/5S02ZMkVf+tKX5PV6NXnyZD3zzDPnPWYoFFIwGIyYAABAbIp6sBw9elQFBQX68Y9/rISE7t/QOXbsmNavX68rr7xSL730ku6++24tXbpUmzdv7vG4RUVF8ng84cnv90frEgAAwACLarB0dnZqwYIFWr16ta666qoet+vq6tI111yjhx9+WJMnT9add96pO+64Qxs2bOhxn8LCQrW2toanEydOROMSAACABRw/w+LE6dOnVV1drddee0333nuvpH/HiTFGCQkJ2rFjh66//nr5fD596lOfitj36quvVllZWY/HTkxMVGJiYjSHDwAALBHVYHG73aqtrY1Y9tRTT6miokLbtm0LP1g7ffp01dXVRWz31ltv6fLLL4/m8AAAwCDhOFja2tpUX18fnm9oaFBNTY2Sk5MVCARUWFioxsZGlZaWKi4uThMmTIjY3+v1aujQoRHLv/Wtb2natGl6+OGHdcstt2j//v16+umn9fTTT1/ApQEAgFjh+BmW6upqTZ48WZMnT5Yk5efna/LkyVq5cqUkqampScePH3d0zGuvvVbbt2/Xc889pwkTJuiBBx5QcXGxFi5c6HR4AAAgBrmMMWagB9EXgsGgPB6PWltb5Xa7B3o4AACgF3r7+s13CQEAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrESwAAMB6BAsAALAewQIAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsR7AAAADrOQ6WyspK5ebmKiUlRS6XS+Xl5b3et6qqSgkJCcrIyOhxm0ceeUQul0v33Xef06EBAIAY5ThY2tvblZ6ernXr1jnar6WlRbfddptmzZrV4zYHDhzQD3/4Q02aNMnpsAAAQAxzHCw5OTl68MEHddNNNznab/HixVqwYIGmTp3a7fq2tjYtXLhQzzzzjEaOHOl0WAAAIIb1yzMsmzZt0rFjx7Rq1aoet1myZInmzJmj7OzsXh0zFAopGAxGTAAAIDYlRPsER48eVUFBgV5++WUlJHR/uq1bt+rQoUM6cOBAr49bVFSk1atX99UwAQCAxaL6DktnZ6cWLFig1atX66qrrup2mxMnTuib3/ymfvKTn2jo0KG9PnZhYaFaW1vD04kTJ/pq2AAAwDIuY4z50Du7XNq+fbvmzZvX7fqWlhaNHDlS8fHx4WVdXV0yxig+Pl47duxQMBjUTTfdFLFNZ2enXC6X4uLiFAqFItb1JBgMyuPxqLW1VW63+8NeEgAA6Ee9ff2O6q+E3G63amtrI5Y99dRTqqio0LZt25Samqqurq5ztlm0aJHGjx+v5cuX9ypWAABAbHMcLG1tbaqvrw/PNzQ0qKamRsnJyQoEAiosLFRjY6NKS0sVFxenCRMmROzv9Xo1dOjQiOX/vc2ll16qUaNGnbMcAABcnBwHS3V1tbKyssLz+fn5kqS8vDyVlJSoqalJx48f77sRAgCAi94FPcNiE55hAQBg8Ont6zffJQQAAKxHsAAAAOsRLAAAwHoECwAAsB7BAgAArEewAAAA6xEsAADAegQLAACwHsECAACsF9UvP+xPZ/9gbzAYHOCRAACA3jr7uv2//vB+zATL6dOnJUl+v3+ARwIAAJw6ffq0PB5Pj+tj5ruEurq69O6772rEiBFyuVwDPZwBFQwG5ff7deLECb5XKYq4z/2He90/uM/9g/scyRij06dPKyUlRXFxPT+pEjPvsMTFxWns2LEDPQyruN1u/sfQD7jP/Yd73T+4z/2D+/x/zvfOylk8dAsAAKxHsAAAAOsRLDEoMTFRq1atUmJi4kAPJaZxn/sP97p/cJ/7B/f5w4mZh24BAEDs4h0WAABgPYIFAABYj2ABAADWI1gAAID1CJZB6u9//7sWLlwot9utpKQkff3rX1dbW9t59zlz5oyWLFmiUaNG6SMf+Yi++MUv6uTJk91u+7e//U1jx46Vy+VSS0tLFK5gcIjGfX799dc1f/58+f1+DRs2TFdffbWefPLJaF+KVdatW6crrrhCQ4cO1XXXXaf9+/efd/uf/exnGj9+vIYOHaqJEyfqN7/5TcR6Y4xWrlwpn8+nYcOGKTs7W0ePHo3mJQwKfXmfP/jgAy1fvlwTJ07UpZdeqpSUFN1222169913o30Zg0Jf/0z/p8WLF8vlcqm4uLiPRz3IGAxKN954o0lPTzevvPKKefnll80nP/lJM3/+/PPus3jxYuP3+82uXbtMdXW1+cxnPmOmTZvW7bZz5841OTk5RpJ5//33o3AFg0M07vOPfvQjs3TpUrN7927z9ttvmy1btphhw4aZtWvXRvtyrLB161YzZMgQ8+yzz5o//elP5o477jBJSUnm5MmT3W5fVVVl4uPjzfe+9z1z+PBh853vfMdccsklpra2NrzNI488YjwejykvLzevv/66+cIXvmBSU1PNP//5z/66LOv09X1uaWkx2dnZ5vnnnzdHjhwx+/btM5mZmebTn/50f16WlaLxM33Wz3/+c5Oenm5SUlLME088EeUrsRvBMggdPnzYSDIHDhwIL3vxxReNy+UyjY2N3e7T0tJiLrnkEvOzn/0svOzNN980ksy+ffsitn3qqafMjBkzzK5duy7qYIn2ff5P99xzj8nKyuq7wVssMzPTLFmyJDzf2dlpUlJSTFFRUbfb33LLLWbOnDkRy6677jpz1113GWOM6erqMmPGjDGPPvpoeH1LS4tJTEw0zz33XBSuYHDo6/vcnf379xtJ5p133umbQQ9S0brXf/nLX8zHPvYx88Ybb5jLL7/8og8WfiU0CO3bt09JSUmaMmVKeFl2drbi4uL06quvdrvPwYMH9cEHHyg7Ozu8bPz48QoEAtq3b1942eHDh/Xd735XpaWl5/0SqotBNO/zf2ttbVVycnLfDd5SHR0dOnjwYMT9iYuLU3Z2do/3Z9++fRHbS9INN9wQ3r6hoUHNzc0R23g8Hl133XXnveexLBr3uTutra1yuVxKSkrqk3EPRtG6111dXfrqV7+q+++/X2lpadEZ/CBzcb8iDVLNzc3yer0RyxISEpScnKzm5uYe9xkyZMg5/2EZPXp0eJ9QKKT58+fr0UcfVSAQiMrYB5No3ef/tnfvXj3//PO68847+2TcNnvvvffU2dmp0aNHRyw/3/1pbm4+7/Zn/+nkmLEuGvf5v505c0bLly/X/PnzL+ov8IvWvV6zZo0SEhK0dOnSvh/0IEWwWKSgoEAul+u805EjR6J2/sLCQl199dW69dZbo3YOGwz0ff5Pb7zxhubOnatVq1bpc5/7XL+cE7hQH3zwgW655RYZY7R+/fqBHk7MOXjwoJ588kmVlJTI5XIN9HCskTDQA8D/WbZsmb72ta+dd5uPf/zjGjNmjE6dOhWx/F//+pf+/ve/a8yYMd3uN2bMGHV0dKilpSXi//2fPHkyvE9FRYVqa2u1bds2Sf/+5IUkXXbZZfr2t7+t1atXf8grs8tA3+ezDh8+rFmzZunOO+/Ud77znQ91LYPNZZddpvj4+HM+ndbd/TlrzJgx593+7D9Pnjwpn88XsU1GRkYfjn7wiMZ9PutsrLzzzjuqqKi4qN9dkaJzr19++WWdOnUq4p3uzs5OLVu2TMXFxfrzn//ctxcxWAz0QzRw7uzDoNXV1eFlL730Uq8eBt22bVt42ZEjRyIeBq2vrze1tbXh6dlnnzWSzN69e3t82j2WRes+G2PMG2+8Ybxer7n//vujdwGWyszMNPfee294vrOz03zsYx877wOKn//85yOWTZ069ZyHbh977LHw+tbWVh667eP7bIwxHR0dZt68eSYtLc2cOnUqOgMfhPr6Xr/33nsR/y2ura01KSkpZvny5ebIkSPRuxDLESyD1I033mgmT55sXn31VbNnzx5z5ZVXRnzc9i9/+YsZN26cefXVV8PLFi9ebAKBgKmoqDDV1dVm6tSpZurUqT2e4/e///1F/SkhY6Jzn2tra81HP/pRc+utt5qmpqbwdLG8AGzdutUkJiaakpISc/jwYXPnnXeapKQk09zcbIwx5qtf/aopKCgIb19VVWUSEhLMY489Zt58802zatWqbj/WnJSUZH7xi1+YP/7xj2bu3Ll8rLmP73NHR4f5whe+YMaOHWtqamoifnZDodCAXKMtovEz/d/4lBDBMmj97W9/M/Pnzzcf+chHjNvtNosWLTKnT58Or29oaDCSzO9///vwsn/+85/mnnvuMSNHjjTDhw83N910k2lqaurxHARLdO7zqlWrjKRzpssvv7wfr2xgrV271gQCATNkyBCTmZlpXnnllfC6GTNmmLy8vIjtX3jhBXPVVVeZIUOGmLS0NPPrX/86Yn1XV5dZsWKFGT16tElMTDSzZs0ydXV1/XEpVuvL+3z2Z7276T9//i9Wff0z/d8IFmNcxvz/BxUAAAAsxaeEAACA9QgWAABgPYIFAABYj2ABAADWI1gAAID1CBYAAGA9ggUAAFiPYAEAANYjWAAAgPUIFgAAYD2CBQAAWI9gAQAA1vt/IYXvgRPGsFEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "plt.plot(trainLoss, label = 'Training Loss')\n",
    "plt.plot(testLoss, label = 'Validation Loss')\n",
    "plt.legend(frameon = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 12. Predict the labels on actual test data\n",
    "\n",
    "Now, we apply all the steps for the actual test data from the test.csv and finally make a 'submission.csv' with all out predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\n",
    "\n",
    "finalTest = test.values/255\n",
    "\n",
    "finalTest = torch.from_numpy(finalTest)\n",
    "\n",
    "temp = np.zeros(finalTest.shape)\n",
    "temp = torch.from_numpy(temp)\n",
    "\n",
    "data = torch.utils.data.TensorDataset(finalTest, temp)\n",
    "\n",
    "submissionLoader = torch.utils.data.DataLoader(data, batch_size = batch, shuffle = False)\n",
    "\n",
    "submission = [['ImageId', 'Label']]\n",
    "\n",
    "with torch.no_grad():\n",
    "    model.eval()\n",
    "    image_id = 1\n",
    "    for images, _ in submissionLoader:\n",
    "        images = (images.view(-1,1,28,28)).type(torch.DoubleTensor)\n",
    "        log_ps = model(images.type(torch.FloatTensor).to(device))\n",
    "        ps = torch.exp(log_ps)\n",
    "        top_p, top_class = ps.topk(1, dim = 1)\n",
    "        \n",
    "        for prediction in top_class:\n",
    "            submission.append([image_id, prediction.item()])\n",
    "            image_id += 1\n",
    "            \n",
    "\n",
    "\n",
    "pytorchSubmission = pd.DataFrame(submission)\n",
    "pytorchSubmission.columns = pytorchSubmission.iloc[0]\n",
    "pytorchSubmission = pytorchSubmission.drop(0, axis = 0)\n",
    "\n",
    "pytorchSubmission.to_csv(\"submission.csv\", index = False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*I have tried to provide every possible minute detail that I found was necessary for any beginner, who is just getting started with either neural networks or PyTorch, to know and understand.*\n",
    "\n",
    "*I'll keep on updating this notebook as and when I feel the need to. You are welcome to point out any mistakes, share any suggestoins, imporvements or anything you feel like with me in the comments. Please consider liking this notebook if you feel so and share it with anyone you might think would benefit from this or can improve this further.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}