nn_Embedding_in_details.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Introduction \n",
        "\n",
        "The embedding layer is a fundamental component in natural language processing (NLP) tasks, such as text classification or language modeling. It maps discrete input indices (typically representing words or tokens) to dense, continuous vectors called embeddings. These embeddings capture semantic relationships between the input elements, allowing the neural network to reason about the meaning of the words in the context of the task.\n",
        "\n",
        "# Inference and training \n",
        "Regarding the handling of parameters during training and inference:\n",
        "\n",
        "Training: During training, the parameters of the embedding layer, i.e., the elements of the weight matrix, are learned through backpropagation. They are updated iteratively to minimize the loss function of the neural network using techniques like gradient descent. The gradients are computed and propagated through the network to adjust the embedding values.\n",
        "\n",
        "Inference: During inference or evaluation, the learned parameters of the embedding layer are fixed and used as is. The network takes input indices, and the corresponding embeddings are looked up from the weight matrix without any further updates or training. The embedding layer acts as a static lookup table, providing fixed embeddings for the given indices.\n",
        "\n",
        "Overall, the embedding layer acts as a bridge between discrete input indices and continuous embeddings. It allows the neural network to represent and process textual data in a meaningful way, capturing the relationships between words or tokens. The correspondence to a lookup table facilitates efficient and flexible access to the embeddings based on the input indices."
      ],
      "metadata": {
        "id": "OfbbwP0YH86h"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "7fu5sRopHJeL"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "class Embedding(nn.Module):\n",
        "    def __init__(self, num_embeddings, embedding_dim):\n",
        "        super(Embedding, self).__init__()\n",
        "        self.num_embeddings = num_embeddings\n",
        "        self.embedding_dim = embedding_dim\n",
        "\n",
        "        # Initialize the embedding matrix\n",
        "        self.weight = nn.Parameter(torch.Tensor(num_embeddings, embedding_dim)) # note that these parameters are trainable\n",
        "        nn.init.xavier_uniform_(self.weight)\n",
        "\n",
        "    def forward(self, input):\n",
        "        if input.dim() == 1:\n",
        "            # If the input is a 1D tensor, expand it to 2D\n",
        "            input = input.unsqueeze(1)\n",
        "        \n",
        "        # Retrieve the embeddings for the input indices\n",
        "        embeddings = self.weight[input] # from this perspective, we are kind of doing a lookup table\n",
        "        \n",
        "        return embeddings.squeeze()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Implementation details\n",
        "Now, let's dive into the implementation details of the embedding layer and its correspondence to a lookup table:\n",
        "\n",
        "Initialization: In the implementation provided earlier, the embedding layer is initialized with a weight matrix (self.weight) of shape (num_embeddings, embedding_dim). num_embeddings represents the size of the vocabulary, i.e., the number of distinct words or tokens. embedding_dim represents the desired dimensionality of the embeddings. Each row of the weight matrix corresponds to the embedding vector for a specific index.\n",
        "\n",
        "Lookup Operation: During the forward pass, given an input tensor of indices, the embedding layer retrieves the corresponding embeddings from the weight matrix. In the implementation, this is achieved by indexing the self.weight matrix with the input indices: embeddings = self.weight[input]. The resulting embeddings tensor has a shape of (batch_size, embedding_dim), where batch_size is the number of input indices provided.\n",
        "\n",
        "Correspondence to a Lookup Table: The embedding layer can be seen as a lookup table, where each index corresponds to a row in the table (weight matrix), and the embedding vector associated with that index is retrieved. This lookup operation is similar to accessing values in a table or dictionary based on the provided key/index."
      ],
      "metadata": {
        "id": "ck_S3ZoYIF_6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create an instance of the custom Embedding module\n",
        "vocab_size = 1000\n",
        "embedding_dim = 50\n",
        "embedding_layer = Embedding(vocab_size, embedding_dim)\n",
        "\n",
        "# Generate some dummy input\n",
        "input_indices = torch.tensor([1, 3, 5, 2])\n",
        "\n",
        "# Pass the input through the embedding layer\n",
        "embeddings = embedding_layer(input_indices)\n",
        "\n",
        "print(embeddings.shape)  # Output: torch.Size([4, 50])\n",
        "print(embeddings)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vMh-Rzy0HKNI",
        "outputId": "1a2918e3-56db-4a3f-eec0-238646290695"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([4, 50])\n",
            "tensor([[-0.0400,  0.0118, -0.0004, -0.0688, -0.0588, -0.0314, -0.0043, -0.0013,\n",
            "         -0.0626,  0.0684,  0.0536, -0.0084, -0.0579,  0.0711,  0.0380,  0.0638,\n",
            "         -0.0657,  0.0045, -0.0316,  0.0601, -0.0629,  0.0234, -0.0057, -0.0171,\n",
            "          0.0056, -0.0239, -0.0229, -0.0548, -0.0599, -0.0509, -0.0168, -0.0189,\n",
            "         -0.0182, -0.0198, -0.0480,  0.0316, -0.0603, -0.0523, -0.0556, -0.0048,\n",
            "          0.0647,  0.0057,  0.0633,  0.0470, -0.0298,  0.0455,  0.0719, -0.0164,\n",
            "          0.0709,  0.0517],\n",
            "        [ 0.0137, -0.0645,  0.0155,  0.0660,  0.0663, -0.0663,  0.0383, -0.0498,\n",
            "         -0.0475, -0.0259,  0.0004, -0.0655, -0.0519,  0.0132, -0.0687,  0.0634,\n",
            "         -0.0730, -0.0432,  0.0569,  0.0094, -0.0417,  0.0394,  0.0326, -0.0216,\n",
            "         -0.0416, -0.0361, -0.0409,  0.0336,  0.0238,  0.0486,  0.0540,  0.0294,\n",
            "         -0.0115, -0.0469, -0.0315, -0.0095, -0.0549, -0.0307,  0.0636, -0.0743,\n",
            "          0.0197, -0.0616,  0.0110,  0.0066,  0.0736,  0.0299, -0.0392,  0.0080,\n",
            "          0.0440, -0.0398],\n",
            "        [ 0.0065, -0.0545,  0.0541,  0.0213, -0.0211,  0.0094,  0.0484, -0.0385,\n",
            "         -0.0176,  0.0146,  0.0349,  0.0352,  0.0448,  0.0582,  0.0615,  0.0168,\n",
            "         -0.0295, -0.0321,  0.0596, -0.0534,  0.0136,  0.0576,  0.0739,  0.0674,\n",
            "         -0.0427,  0.0594, -0.0740,  0.0205, -0.0007,  0.0395,  0.0587,  0.0301,\n",
            "          0.0408, -0.0263, -0.0426, -0.0548,  0.0392,  0.0404,  0.0207, -0.0484,\n",
            "          0.0742, -0.0642,  0.0096, -0.0081, -0.0518, -0.0670, -0.0717, -0.0570,\n",
            "         -0.0414, -0.0737],\n",
            "        [-0.0256,  0.0070,  0.0548, -0.0377, -0.0224,  0.0177, -0.0152, -0.0512,\n",
            "          0.0310,  0.0165,  0.0211,  0.0689,  0.0283, -0.0272, -0.0741,  0.0516,\n",
            "         -0.0343, -0.0702, -0.0698,  0.0212, -0.0021,  0.0195, -0.0597, -0.0558,\n",
            "         -0.0534,  0.0061, -0.0311, -0.0199, -0.0344,  0.0619,  0.0399, -0.0348,\n",
            "         -0.0039,  0.0598,  0.0548, -0.0726,  0.0388, -0.0452, -0.0528,  0.0633,\n",
            "          0.0602,  0.0716,  0.0602,  0.0198,  0.0691, -0.0506, -0.0473,  0.0147,\n",
            "         -0.0487,  0.0185]], grad_fn=<SqueezeBackward0>)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "7MgzAD38HSD_"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}