Introduction_to_pytorch-.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "pqBnnotyafAg",
        "DfcJscqNVNzP",
        "9XYWGxjmcvDv",
        "SNXHU33PdD-4",
        "dkarZOqudt0B",
        "Oxena08xk6W8",
        "02zhHKtKl3HD",
        "_yQ6iGdgm120",
        "O8FuKNvUnJ2c",
        "CaJ7cbuGnwvD",
        "4NssJ2iDrdbE",
        "M88ycz7Fri4F",
        "eqrkWtgesfmM",
        "u9XQaAgws3Dj",
        "xeOzQcsXtFF8"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Objectives :\n",
        "\n",
        "* Introduction to Pytorch tensors\n",
        "* Automatic differentiation and computational graphs \n",
        "* Introduction to nn.Module and Building the first neural network model in Pytorch\n"
      ],
      "metadata": {
        "id": "vu888NoI4nJt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Introduction to Pytorch**\n",
        "\n",
        "\n",
        "PyTorch is an open-source machine learning framework that enables developers to create and deploy deep learning models. Developed by Facebook AI Research, PyTorch has gained immense popularity due to its ease of use, flexibility, and dynamic computational graph feature. It offers a Python-based interface that allows users to build and train neural networks and perform various other machine learning tasks. PyTorch has a large and active community of contributors who constantly improve and expand its functionalities.\n",
        "\n",
        "One of the key features of PyTorch is its *dynamic computational graph*, which enables users to define and modify their models on-the-fly during runtime. This is in contrast to other popular frameworks like TensorFlow, which require users to define a *static computational graph* before runtime. Additionally, PyTorch offers a variety of optimization techniques, including stochastic gradient descent, Adam, and RMSprop. PyTorch also has excellent support for GPUs, making it an ideal choice for deep learning tasks that require significant computational power. Overall, PyTorch is a powerful and intuitive framework that has become a popular choice for deep learning practitioners and researchers alike."
      ],
      "metadata": {
        "id": "cd7jWOuXuCec"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Recall :** In the context of neural networks, a computational graph is a directed acyclic graph (DAG) that represents the mathematical operations performed by the network to transform the input data into output predictions. Each node in the graph represents a mathematical operation, and each edge represents the flow of data from one operation to another. The input data and the parameters of the neural network are also represented as nodes in the graph. The computational graph is used to compute the gradient of the loss function with respect to the parameters of the neural network during backpropagation, which is used to update the parameters to minimize the loss.\n",
        "\n",
        "In PyTorch, computational graphs are constructed dynamically as the model is executed. This means that the graph is generated on-the-fly based on the input data and the parameters of the model, allowing for flexibility in the construction of complex neural network architectures. The computational graph in PyTorch is represented by a data structure called a \"computational graph node\", which contains information about the operation performed by the node, the inputs to the operation, and the outputs of the operation. The computational graph is an essential component of PyTorch's autograd package, which is used to automatically compute gradients and update the parameters of the model during training."
      ],
      "metadata": {
        "id": "b8MTK2Yuutey"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Pytorch main packages: \n",
        "\n",
        "PyTorch consists of four main packages:\n",
        "* torch: a general-purpose array library similar to NumPy that can do computations on GPU.\n",
        "* torch.autograd: a package for automatically obtaining gradients.\n",
        "* torch.nn: a neural net library with common layers and cost functions.\n",
        "* torch.optim: an optimization package with common optimization algorithms like SGD, Adam, etc."
      ],
      "metadata": {
        "id": "Yn4Tnt2v3wTU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Installing Pytorch**\n",
        "\n",
        "First, you need to install PyTorch. You can install PyTorch using pip by running the following command:"
      ],
      "metadata": {
        "id": "pqBnnotyafAg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# You can run the code in this notebook without installing anything. However, if you want to run the code on your local machine, you need to install the required\n",
        "# packaes :\n",
        "# pip install torch"
      ],
      "metadata": {
        "id": "zV9ZHR_wal3w"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Creating Tensors in Pytorch**\n",
        "\n",
        "Tensors are a fundamental data structure in PyTorch, used to represent data and computations. They are similar to arrays in other programming languages, but with some additional features that make them well-suited for working with deep learning models.\n",
        "\n",
        "In PyTorch, a tensor is a multi-dimensional array of values, with a fixed size and data type. Tensors can have any number of dimensions, from zero to any positive integer. The size of each dimension can also vary, as long as it is consistent across all elements in the tensor.\n",
        "\n",
        "Here's an example of how to create a tensor in PyTorch:"
      ],
      "metadata": {
        "id": "DfcJscqNVNzP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "# create a 1-dimensional tensor of size 3\n",
        "x = torch.tensor([1, 2, 3])\n",
        "\n",
        "# create a 2-dimensional tensor of size 2x3\n",
        "y = torch.tensor([[1, 2, 3], [4, 5, 6]])\n",
        "\n",
        "# create a 3-dimensional tensor of size 2x2x2\n",
        "z = torch.tensor([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])"
      ],
      "metadata": {
        "id": "ePdY8sKtb1DJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the above example, we have created three tensors with different dimensions using the torch.tensor() function. We can also specify the data type of the tensor using the dtype argument, which defaults to float32 if not specified."
      ],
      "metadata": {
        "id": "LG14wNQPgDlq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "A new tensor can be created from an existing one. So, if we wanted, we could create a new Tensor of zeros with the same properties (shape and data type) as the x we created\n"
      ],
      "metadata": {
        "id": "sEHnY5Trf2Sz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.zeros_like(x)\n",
        "print(x) # tensor([0, 0, 0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iLcMIK8ff5LJ",
        "outputId": "dfb44242-bd0b-46e1-a4d3-b077476278b1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0, 0, 0])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "Tensors in PyTorch can be manipulated using a variety of mathematical operations, such as addition, subtraction, multiplication, and division. Here's an example of how to perform arithmetic operations on tensors:"
      ],
      "metadata": {
        "id": "MLDzIUmPb2X3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create two tensors\n",
        "a = torch.tensor([1, 2, 3])\n",
        "b = torch.tensor([4, 5, 6])\n",
        "\n",
        "# perform addition\n",
        "c = a + b\n",
        "\n",
        "# perform multiplication\n",
        "d = a * b"
      ],
      "metadata": {
        "id": "C8lci-3Wb7WY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"a + b\",c)\n",
        "print(\"a * b \",d) # notice this is element-wise multiplication"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F2F_4RwccAPI",
        "outputId": "55aa6292-6f38-453a-9906-c05c4256421a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "a + b tensor([5, 7, 9])\n",
            "a * b  tensor([ 4, 10, 18])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        " the above example, we have created two tensors and performed addition and multiplication operations on them. PyTorch also supports many other mathematical operations on tensors, such as matrix multiplication, dot product, and element-wise operations like sin, cos, tanh, etc.\n",
        "\n",
        "PyTorch tensors can also be used for deep learning models, as they can store both input data and model parameters. Tensors can be passed as input to a neural network model, and the model will perform computations on them to generate the output.\n",
        "\n",
        "Overall, tensors are a fundamental building block in PyTorch, and understanding their properties and operations is essential for working with deep learning models.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "HqwnAKh-cWlo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lz0Fxkpbfe4C",
        "outputId": "d047635b-fc0f-40ce-f253-6c154126ad60"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([0, 0, 0])"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Shape of tensor\n",
        "In PyTorch, the shape of a tensor refers to the number of dimensions and the size of each dimension. For example, a 2-dimensional tensor with a shape of (3, 4) has two dimensions, with the first dimension having a size of 3 and the second dimension having a size of 4.\n",
        "\n",
        "The shape of a tensor can be inspected using the .shape attribute, which returns a tuple containing the size of each dimension. Here's an example:"
      ],
      "metadata": {
        "id": "9XYWGxjmcvDv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "# create a 2-dimensional tensor of size 3x4\n",
        "x = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])\n",
        "\n",
        "# get the shape of the tensor\n",
        "print(x.shape)  # prints (3, 4)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M2WTm7svVK6f",
        "outputId": "c929eb32-4ca9-46c5-bcdf-fa7a8a12602a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([3, 4])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the above example, we create a 2-dimensional tensor x of size 3x4 and then print its shape using the .shape attribute."
      ],
      "metadata": {
        "id": "vv8-qgqPdUlw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Additional properties\n",
        "\n",
        "Additionally, we can also inspect other properties of a tensor, such as its data type, number of elements, and storage size. Here's an example:"
      ],
      "metadata": {
        "id": "SNXHU33PdD-4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "# create a 1-dimensional tensor of size 5 with random values\n",
        "x = torch.randn(5)\n",
        "\n",
        "# get the data type of the tensor\n",
        "print(x.dtype)  # prints torch.float32\n",
        "\n",
        "# get the number of elements in the tensor\n",
        "print(x.numel())  # prints 5\n",
        "\n",
        "# get where the tensor is stored:\n",
        "\n",
        "x.device\n",
        "# device(type='cpu')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "shfdDvK5VRAO",
        "outputId": "06ab8c2f-adae-40cb-f785-5028453ef70f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.float32\n",
            "5\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "device(type='cpu')"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Tensor multiplication in PyTorch\n",
        "\n",
        "##Dot Product\n",
        "\n",
        "The dot product is a basic operation in linear algebra that takes two vectors and produces a scalar. In PyTorch, the dot product of two tensors can be computed using the torch.dot() function. However, this function only works for 1-dimensional tensors."
      ],
      "metadata": {
        "id": "dkarZOqudt0B"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "# create two 1-dimensional tensors of size 3\n",
        "x = torch.tensor([1, 2, 3])\n",
        "y = torch.tensor([4, 5, 6])\n",
        "\n",
        "# compute the dot product of the two tensors\n",
        "z = torch.dot(x, y)\n",
        "\n",
        "print(z)  # prints 32"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xKy4ahkXdKkB",
        "outputId": "c2bab0a9-ce93-401b-e9e3-ce92e964b260"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(32)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the above example, we create two 1-dimensional tensors x and y of size 3 and compute their dot product using the torch.dot() function.\n",
        "\n",
        "## Matrix multiplication\n",
        "\n",
        "Matrix multiplication is a common operation in linear algebra and is used extensively in deep learning models. In PyTorch, matrix multiplication can be performed using the torch.mm() function or the torch.matmul() function."
      ],
      "metadata": {
        "id": "k6dMPbGXeBMg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "# create two 2-dimensional tensors of size 2x3 and 3x4\n",
        "x = torch.tensor([[1, 2, 3], [4, 5, 6]])\n",
        "y = torch.tensor([[7, 8, 9, 10], [11, 12, 13, 14], [15, 16, 17, 18]])\n",
        "\n",
        "# perform matrix multiplication\n",
        "z = torch.mm(x, y)\n",
        "\n",
        "print(z)  # prints tensor([[ 74,  80,  86,  92], [173, 188, 203, 218]])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XZtIUjc-d-Aw",
        "outputId": "9b037e43-c334-46ae-d2f6-65adf4695f27"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[ 74,  80,  86,  92],\n",
            "        [173, 188, 203, 218]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the above example, we create two 2-dimensional tensors x of size 2x3 and y of size 3x4 and perform their matrix multiplication using the torch.mm() function. The resulting tensor z has a size of 2x4.\n",
        "\n",
        "Note that the torch.matmul() function is more general than torch.mm(), as it can handle matrices with more than two dimensions and can automatically broadcast the input tensors if needed."
      ],
      "metadata": {
        "id": "igEz1WbweVrS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "# create a 3-dimensional tensor of size 2x3x4 and a 2-dimensional tensor of size 4x5\n",
        "x = torch.randn(2, 3, 4)\n",
        "y = torch.randn(4, 5)\n",
        "\n",
        "# perform matrix multiplication\n",
        "z = torch.matmul(x, y)\n",
        "\n",
        "print(z.shape)  # prints torch.Size([2, 3, 5])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xdgah7ZKeMzn",
        "outputId": "4661f064-41dc-41d1-b17f-2cdb11df3ec0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([2, 3, 5])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this example, we create a 3-dimensional tensor x of size 2x3x4 and a 2-dimensional tensor y of size 4x5. The first dimension of x represents the batch size, the second dimension represents the number of rows, and the third dimension represents the number of columns. The y tensor represents a weight matrix of a neural network with 4 inputs and 5 outputs.\n",
        "\n",
        "We perform matrix multiplication between x and y using the torch.matmul() function. Since x is a 3-dimensional tensor and y is a 2-dimensional tensor, PyTorch automatically broadcasts y along the first dimension of x to perform the matrix multiplication. The resulting tensor z has a size of 2x3x5, where the first dimension represents the batch size, the second dimension represents the number of rows, and the third dimension represents the number of outputs.\n",
        "\n",
        "###Remark :\n",
        " In a neural network, the output of each layer is obtained by performing matrix multiplication between the input and the weight matrix, followed by the addition of a bias term and the application of an activation function. The torch.matmul() function is a fundamental building block for implementing this operation efficiently in PyTorch."
      ],
      "metadata": {
        "id": "3AtryTume2sB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Computional graphs in Pytorch\n",
        "\n",
        "In PyTorch, computational graphs are built using the autograd package, which allows automatic differentiation for training neural networks. Here's an example of how to create a simple computational graph in PyTorch:\n",
        "\n"
      ],
      "metadata": {
        "id": "DdKPRny28K7F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "# Define the input data\n",
        "x = torch.tensor(2.0, requires_grad=True)\n",
        "\n",
        "# Define the computation graph\n",
        "y = x ** 2 + 3 * x - 1\n",
        "\n",
        "# Compute the gradients\n",
        "y.backward()\n",
        "\n",
        "# Access the gradients\n",
        "grad_x = x.grad\n",
        "\n",
        "# Explanation using math symbols\n",
        "\"\"\"\n",
        "Given:\n",
        "x = 2.0\n",
        "\n",
        "Computation Graph:\n",
        "y = x^2 + 3x - 1\n",
        "\n",
        "Expanding the terms:\n",
        "y = x^2 + 3x - 1\n",
        "  = 2.0^2 + 3(2.0) - 1\n",
        "  = 4.0 + 6.0 - 1\n",
        "  = 9.0\n",
        "\n",
        "Gradients:\n",
        "∂y/∂x = 2x + 3\n",
        "∂y/∂x (at x = 2.0) = 2(2.0) + 3\n",
        "                     = 4.0 + 3\n",
        "                     = 7.0\n",
        "\"\"\"\n",
        "\n",
        "# Print the results\n",
        "print(\"Input x:\", x.item())\n",
        "print(\"Output y:\", y.item())\n",
        "print(\"Gradient of x:\", grad_x.item())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dDxg6k1C8Qqb",
        "outputId": "fe921451-57bb-48bb-fb30-62c83e7e108a"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input x: 2.0\n",
            "Output y: 9.0\n",
            "Gradient of x: 7.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now lets see a more complex example 🇰\n"
      ],
      "metadata": {
        "id": "3l7SNsM39i6a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "# Define the input data\n",
        "x1 = torch.tensor(2.0, requires_grad=True)\n",
        "x2 = torch.tensor(3.0, requires_grad=True)\n",
        "\n",
        "# Define the computation graph\n",
        "y = x1 ** 2 + 3 * x2 - torch.sin(x1 * x2)\n",
        "\n",
        "# Compute the gradients\n",
        "y.backward()\n",
        "\n",
        "# Access the gradients\n",
        "grad_x1 = x1.grad # ∂y/∂x1 \n",
        "grad_x2 = x2.grad # ∂y/∂x2\n",
        "\n",
        "# Explanation using math symbols\n",
        "\"\"\"\n",
        "Given:\n",
        "x1 = 2.0\n",
        "x2 = 3.0\n",
        "\n",
        "Computation Graph:\n",
        "y = x1^2 + 3x2 - sin(x1 * x2)\n",
        "\n",
        "Expanding the terms and applying sin element-wise:\n",
        "y = x1^2 + 3x2 - sin(x1 * x2)\n",
        "  = 2.0^2 + 3(3.0) - sin(2.0 * 3.0)\n",
        "  = 4.0 + 9.0 - sin(6.0)\n",
        "  ≈ 13.279415130615234\n",
        "\n",
        "Gradients:\n",
        "∂y/∂x1 = 2x1 - x2 * cos(x1 * x2)\n",
        "∂y/∂x1 (at x1 = 2.0, x2 = 3.0) = 2(2.0) - 3.0 * cos(2.0 * 3.0)\n",
        "                                = 4.0 - 3.0 * cos(6.0)\n",
        "                                ≈  1.1194891929626465\n",
        "\n",
        "∂y/∂x2 = 3 + x1 * cos(x1 * x2)\n",
        "∂y/∂x2 (at x1 = 2.0, x2 = 3.0) = 3 + 2.0 * cos(2.0 * 3.0)\n",
        "                                = 3 + 2.0 * cos(6.0)\n",
        "                                ≈  1.0796594619750977\n",
        "\"\"\"\n",
        "\n",
        "# Print the results\n",
        "print(\"Input x1:\", x1.item())\n",
        "print(\"Input x2:\", x2.item())\n",
        "print(\"Output y:\", y.item())\n",
        "print(\"Gradient of x1:\", grad_x1.item())\n",
        "print(\"Gradient of x2:\", grad_x2.item())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XPxnfH3v9lld",
        "outputId": "29937660-f885-4815-c057-b627eaf319ae"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input x1: 2.0\n",
            "Input x2: 3.0\n",
            "Output y: 13.279415130615234\n",
            "Gradient of x1: 1.1194891929626465\n",
            "Gradient of x2: 1.0796594619750977\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Automatic differentiation \n",
        "\n",
        "\n",
        "Automatic differentiation is a technique used in computational graphs to compute the derivatives of functions or expressions with respect to their input variables. It allows for efficient and accurate calculation of gradients, which are crucial for training machine learning models.\n",
        "\n",
        "In the context of neural networks, automatic differentiation is used to determine the gradients of the loss function with respect to the model's parameters. These gradients are then used in optimization algorithms, such as gradient descent, to update the model's parameters and minimize the loss.\n",
        "\n",
        "There are two main types of automatic differentiation: forward-mode differentiation and backward-mode differentiation (also known as reverse-mode differentiation).\n",
        "\n",
        "Forward-mode differentiation: In forward-mode differentiation, the derivatives are calculated by applying the chain rule in a forward manner. It computes the derivatives of each operation in the computational graph as the values flow forward from inputs to outputs. However, forward-mode differentiation becomes inefficient when the number of input variables is large compared to the number of output variables.\n",
        "\n",
        "Backward-mode differentiation (reverse-mode differentiation): Backward-mode differentiation is the most commonly used method for automatic differentiation in deep learning frameworks like PyTorch and TensorFlow. It computes the derivatives by applying the chain rule in a backward manner. It starts from the final output and computes the gradients of intermediate variables with respect to the output variables by traversing the computational graph in a reverse direction. This method is efficient because it allows for the reuse of intermediate results and is especially useful when the number of output variables is large compared to the number of input variables.\n",
        "\n",
        "PyTorch, as well as other deep learning frameworks, implements automatic differentiation by automatically constructing and evaluating the computational graph during the forward pass and then using backpropagation to compute the gradients efficiently during the backward pass. This makes it easy to compute gradients and perform gradient-based optimization for training neural networks."
      ],
      "metadata": {
        "id": "o8H8NFho8P6a"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Introduction to nn.Module : building a complicated network"
      ],
      "metadata": {
        "id": "uXqlrPO__u6Z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "\n",
        "class MyNetwork(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(MyNetwork, self).__init__()\n",
        "        self.weight1 = nn.Parameter(torch.tensor(2.0))\n",
        "        self.weight2 = nn.Parameter(torch.tensor(3.0))\n",
        "        self.bias = nn.Parameter(torch.tensor(1.0))\n",
        "\n",
        "    def forward(self, x):\n",
        "        y = self.weight1 * x + self.weight2 * x + self.bias\n",
        "        return y\n",
        "\n",
        "\n",
        "# Create an instance of the network\n",
        "model = MyNetwork()\n",
        "\n",
        "# Define the input data\n",
        "x = torch.tensor(2.0)\n",
        "\n",
        "# Pass the input through the network\n",
        "output = model(x)\n",
        "\n",
        "# Print the results\n",
        "print(\"Input x:\", x.item())\n",
        "print(\"Output:\", output.item())\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d8e_ZI0x_uC5",
        "outputId": "5a546348-255d-4218-9c49-f515646ce24a"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input x: 2.0\n",
            "Output: 11.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "1. We import the necessary modules, `torch` and `torch.nn` to work with PyTorch and its neural network components.\n",
        "\n",
        "2. We define a custom neural network architecture `MyNetwork` that inherits from `nn.Module`. This allows us to utilize the functionality provided by `nn.Module` for managing and optimizing the network.\n",
        "\n",
        "3. Inside the `MyNetwork` class, the `__init__` method is called when an instance of the class is created. We use `super()` to initialize the parent class, `nn.Module`. \n",
        "\n",
        "4. We define three learnable parameters, `weight1`, `weight2`, and `bias`, using `nn.Parameter`. These parameters will be automatically registered as tensors that require gradients and will be optimized during training.\n",
        "\n",
        "5. The `forward` method defines the forward pass of the network. It takes an input `x`, and performs a linear operation by multiplying `weight1` with `x`, `weight2` with `x`, and adding the `bias`. The result is returned as the output `y`.\n",
        "\n",
        "6. We create an instance of the network by calling `MyNetwork()`, which initializes the network with the defined parameters.\n",
        "\n",
        "7. We define an input tensor `x` with a value of `2.0`.\n",
        "\n",
        "8. Finally, we pass the input through the network by calling `model(x)` and store the output in the `output` variable.\n",
        "\n",
        "9. We print the input `x` and the output `output` to display the results.\n"
      ],
      "metadata": {
        "id": "FAawWCqdAULJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Creating first model :\n",
        "\n",
        "import the following, we talk about each line as we need later.\n"
      ],
      "metadata": {
        "id": "Oxena08xk6W8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from torch import nn\n",
        "from torch.utils.data import DataLoader\n",
        "from torchvision.utils import make_grid\n",
        "from torchvision.datasets import CIFAR10\n",
        "from torchvision.transforms import ToTensor\n",
        "from torchvision.transforms import Normalize, Compose\n",
        "import os\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n"
      ],
      "metadata": {
        "id": "Q44uxwBJedDn"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# CIFAR-10 dataset\n",
        "\n",
        "This dataset consists of 60,000 32x32 color images, all labeled as one of 10 classes. The training set is 50,000 images, while the test set is 10,000.\n",
        "\n",
        "\n",
        "Source: https://www.cs.toronto.edu/~kriz/cifar.html\n",
        "\n"
      ],
      "metadata": {
        "id": "02zhHKtKl3HD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "After importing CIFAR10 from torchvision, our next step is to download the dataset and prepare it for loading into the neural network.\n",
        "\n",
        "To ensure that the images are normalized before being fed to the model, we define a transformation function and use **torchvision.transforms.Normalize** to normalize all of the images in the training and test datasets. This method requires the desired mean and standard deviation as arguments. Since CIFAR10 images are in color, we need to provide a value for each color channel (R, G, B). In this case, we set the values to 0.5, as we want the image data values to be close to 0. However, there are other, more precise approaches to normalization that could also be used."
      ],
      "metadata": {
        "id": "9LHlMNcgluNL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "transform = Compose(\n",
        "    [ToTensor(),\n",
        "     Normalize((0.5, 0.5, 0.5),  # mean\n",
        "               (0.5, 0.5, 0.5))] # std. deviation\n",
        ")"
      ],
      "metadata": {
        "id": "vutAXxlRlsh6"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Download the dataset"
      ],
      "metadata": {
        "id": "_yQ6iGdgm120"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "training_data = CIFAR10(root=\"cifar\",\n",
        "                        train = True, # train set, 50k images\n",
        "                        download = True,\n",
        "                        transform=transform)\n",
        "test_data = CIFAR10(root = \"cifar\",\n",
        "                    train = False, # test set, 10k images\n",
        "                    download = True,\n",
        "                    transform = transform)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q1DeQNP9lOm4",
        "outputId": "a57d1235-716e-416e-aa5e-f32b0dcc0bc1"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to cifar/cifar-10-python.tar.gz\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 170498071/170498071 [00:05<00:00, 28862648.14it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Extracting cifar/cifar-10-python.tar.gz to cifar\n",
            "Files already downloaded and verified\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Preparing data to feed it to the network:\n",
        "\n",
        "\n",
        "\n",
        "With our dataset downloaded and normalized, we can now prepare it to be fed to the neural network. To do this, we utilize the PyTorch DataLoader, which allows us to specify the batch size for the data."
      ],
      "metadata": {
        "id": "O8FuKNvUnJ2c"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "batch_size = 2\n",
        "train_dataloader = DataLoader(training_data, \n",
        "                              batch_size=batch_size, \n",
        "                              shuffle=True)\n",
        "test_dataloader = DataLoader(test_data, \n",
        "                             batch_size=batch_size, \n",
        "                             shuffle=True)"
      ],
      "metadata": {
        "id": "34d-nX1ImVua"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "The **DataLoader** is an iterable, so to explore it further, we can examine the dimensions of one iteration by looking at train_dataloader."
      ],
      "metadata": {
        "id": "9eVNlZLanVoc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for X, y in train_dataloader:\n",
        "  print(f\"Shape of X [N, C, H, W]: {X.shape}\")\n",
        "  print(f\"Shape of y: {y.shape} {y.dtype}\")\n",
        "  break"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6aBs3xpxmuz5",
        "outputId": "d24905a7-1acc-4c88-b97b-4dbdd81f3231"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape of X [N, C, H, W]: torch.Size([2, 3, 32, 32])\n",
            "Shape of y: torch.Size([2]) torch.int64\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Defining the model :\n",
        "To begin building our neural network, we first define our model class, which we'll call NeuralNetwork. This model will be a subclass of PyTorch's nn.Module, which is the base class for all neural network modules in PyTorch.\n",
        "\n",
        "Since our dataset contains color images, each image has a shape of (3, 32, 32), representing a 32x32 tensor in each of the 3 RGB color channels. Since our initial model will consist of fully-connected layers, we need to flatten the input image data using nn.Flatten(). This will output a linear layer with 3072 (32 x 32 x 3) nodes. We then use nn.Linear() to create additional linear layers and ReLU layers, if desired. The output of our model will be 10 logits, corresponding to the 10 classes in our dataset.\n",
        "\n",
        "Once we've defined the structure of the model, we define the sequence of the forward pass. Since our model is a simple sequential model, our forward method will be straightforward and compute an output tensor from input tensors.\n",
        "\n",
        "Optionally, we can print the model once it's defined to get a summary of its structure."
      ],
      "metadata": {
        "id": "CaJ7cbuGnwvD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class NeuralNetwork(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(NeuralNetwork, self).__init__()\n",
        "        self.flatten = nn.Flatten()\n",
        "        self.linear_relu_stack = nn.Sequential(\n",
        "            nn.Linear(32*32*3, 1024),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(1024, 512),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(512, 10)\n",
        "        )\n",
        "    def forward(self, x):\n",
        "        x = self.flatten(x)\n",
        "        logits = self.linear_relu_stack(x)\n",
        "        return logits\n",
        "        \n",
        "model = NeuralNetwork()\n",
        "\n",
        "print(model)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z7B7CKcun0fz",
        "outputId": "a1687a38-9386-4550-ca83-b35e3ab33b3b"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "NeuralNetwork(\n",
            "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
            "  (linear_relu_stack): Sequential(\n",
            "    (0): Linear(in_features=3072, out_features=1024, bias=True)\n",
            "    (1): ReLU()\n",
            "    (2): Linear(in_features=1024, out_features=512, bias=True)\n",
            "    (3): ReLU()\n",
            "    (4): Linear(in_features=512, out_features=10, bias=True)\n",
            "  )\n",
            ")\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Optimizer \n",
        "In PyTorch, you can use CrossEntropyLoss() as your loss function, which is suitable for many machine learning tasks. However, for other tasks, you may want to use different loss functions that are more appropriate. To optimize our model, we will use stochastic gradient descent, which is available in the torch.optim package, along with other optimizers such as Adam and RMSprop. We simply need to pass the model parameters and the learning rate lr to the SGD() optimizer. If you want to apply momentum or weight decay in your model optimization, you can specify those using the momentum and weight_decay parameters in the SGD() optimizer (both of which default to 0)."
      ],
      "metadata": {
        "id": "4NssJ2iDrdbE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "loss_fn = nn.CrossEntropyLoss()\n",
        "optimizer = torch.optim.SGD( model.parameters(), lr=0.001 ) # momentum=0.9"
      ],
      "metadata": {
        "id": "E3Vu-xz-ncC5"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Training Loop :\n",
        "Let's discuss the definition of the training loop. In this step, we will create our train() function, which will accept train_dataloader, model, loss_fn, and optimizer as arguments during the training process. The size variable is the length of the entire training dataset, which is 50,000. After that, we set the model to training mode by calling model.train(), which is a PyTorch nn.Module method. This enables certain behaviors that are desirable during training, such as dropout and batch normalization. In contrast, if we want to test our model's performance, we would call model.eval() instead.\n",
        "\n",
        "Next, we iterate through each mini-batch and specify that we want to utilize the GPU with to(device). We feed the mini-batch to our model, compute the loss, and then backpropagate. Before starting backpropagation, we need to run optimizer.zero_grad(), which sets the gradient to zero. This step is necessary to ensure that we do not accumulate the gradient over subsequent passes, which can be a desired behavior in some cases, such as RNNs where gradient accumulation is necessary.\n",
        "\n",
        "After the loss.backward() step, which uses the loss to compute the gradient, we use optimizer.step() to update the weights. Finally, we can print updates to the training process, outputting the computed loss after every 2000 training samples."
      ],
      "metadata": {
        "id": "M88ycz7Fri4F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def train(dataloader, model, loss_fn, optimizer):\n",
        "    # Get the size of the training dataset\n",
        "    size = len(dataloader.dataset)\n",
        "    \n",
        "    # Set the model to training mode\n",
        "    model.train()\n",
        "    \n",
        "    # Iterate through each mini-batch of the training data\n",
        "    for batch, (X, y) in enumerate(dataloader):\n",
        "        \n",
        "        # Compute the model predictions\n",
        "        pred = model(X)\n",
        "        \n",
        "        # Compute the loss between the predictions and the true labels\n",
        "        loss = loss_fn(pred, y)\n",
        "        \n",
        "        # Reset the gradients\n",
        "        optimizer.zero_grad()\n",
        "        \n",
        "        # Compute the gradients using backpropagation\n",
        "        loss.backward()\n",
        "        \n",
        "        # Update the model parameters using the optimizer\n",
        "        optimizer.step()\n",
        "        \n",
        "        # Print the current loss and the progress of the training process every 2000 training samples\n",
        "        if batch % 2000 == 0:\n",
        "            loss, current = loss.item(), batch * len(X)\n",
        "            print(f\"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]\")"
      ],
      "metadata": {
        "id": "v4UQWnaMnhQC"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "qdlfWGR3sIij"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def test(dataloader, model, loss_fn):\n",
        "    # Get the size of the dataset and the number of batches\n",
        "    size = len(dataloader.dataset)\n",
        "    num_batches = len(dataloader)\n",
        "    # Set the model to evaluation mode\n",
        "    model.eval()\n",
        "    # Initialize variables for tracking test loss and number of correct predictions\n",
        "    test_loss, correct = 0, 0\n",
        "    # Use torch.no_grad() to disable gradient tracking during testing\n",
        "    with torch.no_grad():\n",
        "        # Iterate through each mini-batch in the data loader\n",
        "        for X, y in dataloader:\n",
        "            # Make predictions using the trained model\n",
        "            pred = model(X)\n",
        "            # Compute the test loss using the specified loss function\n",
        "            test_loss += loss_fn(pred, y).item()\n",
        "            # Count the number of correct predictions by comparing predicted labels to ground truth labels\n",
        "            correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n",
        "    # Compute the average test loss and accuracy across all mini-batches\n",
        "    test_loss /= num_batches\n",
        "    correct /= size\n",
        "    # Print out the test error, accuracy, and average test loss\n",
        "    print(f\"Test Error: \\n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \\n\")"
      ],
      "metadata": {
        "id": "QblahoK4pDVE"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# we just train for 10 epochs\n",
        "epochs = 2\n",
        "for t in range(epochs):\n",
        "    print(f\"Epoch {t+1}\\n-------------------------------\")\n",
        "    train(train_dataloader, model, loss_fn, optimizer)\n",
        "    test(test_dataloader, model, loss_fn)\n",
        "print(\"Done!\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M6o-ItpdpGgc",
        "outputId": "c67c2316-a3eb-497f-823c-db6a1feee826"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1\n",
            "-------------------------------\n",
            "loss: 2.285184  [    0/50000]\n",
            "loss: 1.793684  [ 4000/50000]\n",
            "loss: 1.726945  [ 8000/50000]\n",
            "loss: 1.239966  [12000/50000]\n",
            "loss: 1.765733  [16000/50000]\n",
            "loss: 2.099144  [20000/50000]\n",
            "loss: 1.573969  [24000/50000]\n",
            "loss: 1.326612  [28000/50000]\n",
            "loss: 0.936158  [32000/50000]\n",
            "loss: 1.644593  [36000/50000]\n",
            "loss: 0.942943  [40000/50000]\n",
            "loss: 1.550724  [44000/50000]\n",
            "loss: 2.535042  [48000/50000]\n",
            "Test Error: \n",
            " Accuracy: 44.0%, Avg loss: 1.600083 \n",
            "\n",
            "Epoch 2\n",
            "-------------------------------\n",
            "loss: 1.283306  [    0/50000]\n",
            "loss: 2.011713  [ 4000/50000]\n",
            "loss: 1.411191  [ 8000/50000]\n",
            "loss: 1.717604  [12000/50000]\n",
            "loss: 0.114138  [16000/50000]\n",
            "loss: 1.911854  [20000/50000]\n",
            "loss: 0.846423  [24000/50000]\n",
            "loss: 1.322043  [28000/50000]\n",
            "loss: 2.667917  [32000/50000]\n",
            "loss: 1.373672  [36000/50000]\n",
            "loss: 2.000410  [40000/50000]\n",
            "loss: 1.014187  [44000/50000]\n",
            "loss: 0.506572  [48000/50000]\n",
            "Test Error: \n",
            " Accuracy: 48.4%, Avg loss: 1.459302 \n",
            "\n",
            "Done!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Exercise : try to improve the accuracy of the above model. What would you change?\n",
        "\n",
        "\n",
        "# Save the model "
      ],
      "metadata": {
        "id": "eqrkWtgesfmM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "torch.save(model.state_dict(), \"cifar_fc.pth\")\n"
      ],
      "metadata": {
        "id": "gIvSEwB6pNUK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Load the model \n",
        "When you want to load your model for inference, use torch.load() to grab your saved model, and map the learned parameters with load_state_dict."
      ],
      "metadata": {
        "id": "u9XQaAgws3Dj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "model = NeuralNetwork()\n",
        "model.load_state_dict(torch.load(\"cifar_fc.pth\"))\n",
        "\n"
      ],
      "metadata": {
        "id": "OSa7659Gs9ob"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Evaluating the Model\n"
      ],
      "metadata": {
        "id": "xeOzQcsXtFF8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "correct = 0\n",
        "total = 0\n",
        "\n",
        "with torch.no_grad():\n",
        "   for data in test_dataloader:\n",
        "     images, labels = data\n",
        "     outputs = model(images)\n",
        "     _, predicted = torch.max(outputs.data, 1)\n",
        "     total += labels.size(0)\n",
        "     correct += (predicted == labels).sum().item()\n",
        "     \n",
        "print(f'Model accuracy: {100 * correct // total} %')"
      ],
      "metadata": {
        "id": "Bk6T2RFatGSJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e27bbf24-7acc-4fdc-db2c-e2efa8c869fe"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model accuracy: 48 %\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Excercise : Check how the model performs on individual classes "
      ],
      "metadata": {
        "id": "SJQlOTzTtzgk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "ZgVvTW7rttBl"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "6619vzcmtiyz"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}