gensim_word2vec.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jAeWR-HzqzFY",
        "outputId": "cf6e67b3-8469-4334-8d44-a4cb2a9a9bce"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Vector representation of 'cats': [-9.5785465e-03  8.9431154e-03  4.1650687e-03  9.2347348e-03\n",
            "  6.6435025e-03  2.9247368e-03  9.8040197e-03 -4.4246409e-03\n",
            " -6.8033109e-03  4.2273807e-03  3.7290000e-03 -5.6646108e-03\n",
            "  9.7047603e-03 -3.5583067e-03  9.5494064e-03  8.3472609e-04\n",
            " -6.3384566e-03 -1.9771170e-03 -7.3770545e-03 -2.9795230e-03\n",
            "  1.0416972e-03  9.4826873e-03  9.3558477e-03 -6.5958775e-03\n",
            "  3.4751510e-03  2.2755705e-03 -2.4893521e-03 -9.2291720e-03\n",
            "  1.0271263e-03 -8.1657059e-03  6.3201892e-03 -5.8000805e-03\n",
            "  5.5354391e-03  9.8337233e-03 -1.6000033e-04  4.5284927e-03\n",
            " -1.8094003e-03  7.3607611e-03  3.9400971e-03 -9.0103243e-03\n",
            " -2.3985039e-03  3.6287690e-03 -9.9568366e-05 -1.2012708e-03\n",
            " -1.0554385e-03 -1.6716016e-03  6.0495257e-04  4.1650953e-03\n",
            " -4.2527914e-03 -3.8336217e-03 -5.2816868e-05  2.6935578e-04\n",
            " -1.6880632e-04 -4.7855065e-03  4.3134023e-03 -2.1719194e-03\n",
            "  2.1035396e-03  6.6652300e-04  5.9696771e-03 -6.8423809e-03\n",
            " -6.8157101e-03 -4.4762576e-03  9.4358288e-03 -1.5918827e-03\n",
            " -9.4292425e-03 -5.4504158e-04 -4.4489228e-03  6.0000787e-03\n",
            " -9.5836855e-03  2.8590010e-03 -9.2528323e-03  1.2498009e-03\n",
            "  5.9991982e-03  7.3973476e-03 -7.6214634e-03 -6.0530235e-03\n",
            " -6.8384409e-03 -7.9183402e-03 -9.4990805e-03 -2.1254970e-03\n",
            " -8.3593250e-04 -7.2562015e-03  6.7870365e-03  1.1196196e-03\n",
            "  5.8288667e-03  1.4728665e-03  7.8936579e-04 -7.3681297e-03\n",
            " -2.1766580e-03  4.3210792e-03 -5.0853146e-03  1.1307895e-03\n",
            "  2.8833640e-03 -1.5363609e-03  9.9322954e-03  8.3496347e-03\n",
            "  2.4156666e-03  7.1182456e-03  5.8914376e-03 -5.5806171e-03]\n",
            "Similar words to 'dogs': [('loyal', 0.1459505707025528), ('Dogs', 0.041577354073524475), ('cute', 0.03476494178175926), ('Cats', 0.01915225386619568), ('are', 0.01613469421863556), ('cats', 0.008826159872114658), ('adore', 0.004842504393309355), ('love', 0.001951066660694778), ('I', -0.11410722881555557)]\n"
          ]
        }
      ],
      "source": [
        "import gensim\n",
        "from gensim.models import Word2Vec\n",
        "\n",
        "# Step 1: Prepare the Training Data\n",
        "sentences = [\n",
        "    [\"I\", \"love\", \"cats\"],\n",
        "    [\"I\", \"adore\", \"dogs\"],\n",
        "    [\"Cats\", \"are\", \"cute\"],\n",
        "    [\"Dogs\", \"are\", \"loyal\"]\n",
        "]\n",
        "\n",
        "# Step 2: Train the Word2Vec Model\n",
        "model = Word2Vec(sentences, min_count=1)  # min_count specifies the minimum frequency of a word to be included in the model\n",
        "\n",
        "# Step 3: Explore Word Embeddings\n",
        "# Get the vector representation of a word\n",
        "vector = model.wv[\"cats\"]\n",
        "print(\"Vector representation of 'cats':\", vector)\n",
        "\n",
        "# Find similar words\n",
        "similar_words = model.wv.most_similar(\"dogs\")\n",
        "print(\"Similar words to 'dogs':\", similar_words)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "similar_words = model.wv.most_similar(\"cats\")\n",
        "print(\"Similar words to 'cats':\", similar_words)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4E8GKwtEqzwJ",
        "outputId": "8fbda320-1d38-4dfd-ccb5-9cc0eace49e2"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Similar words to 'cats': [('loyal', 0.19912061095237732), ('Dogs', 0.07497556507587433), ('Cats', 0.060591842979192734), ('love', 0.044689226895570755), ('I', 0.03364058583974838), ('are', 0.027057481929659843), ('dogs', 0.008826158009469509), ('cute', -0.06900332123041153), ('adore', -0.14454564452171326)]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Using word2vecs in LM\n",
        "\n",
        "To use the gensim Word2Vec model in language models, you can follow these steps:\n",
        "\n",
        "* Preprocess your text data: Prepare your text data by tokenizing it into sentences or words, and perform any necessary preprocessing steps like removing stopwords or punctuation.\n",
        "\n",
        "* Train the Word2Vec model: Use the Word2Vec class from gensim.models to train the Word2Vec model on your preprocessed text data. You can specify the desired hyperparameters such as vector size, window size, and minimum count. For example:"
      ],
      "metadata": {
        "id": "pW-A9tjhsiLv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = gensim.models.Word2Vec(sentences, size=100, window=5, min_count=1)"
      ],
      "metadata": {
        "id": "qFTU0N4NrWIP"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "\n",
        "* Obtain word embeddings: Once the model is trained, you can access the word embeddings using model.wv[word], where word is the specific word you want the embedding for.\n",
        "\n",
        "* Integrate word embeddings into your language model: Incorporate the word embeddings into your language model architecture. This could involve initializing an embedding layer with the pretrained word embeddings or fine-tuning the embeddings during training.\n",
        "\n",
        "For example, if you're using PyTorch, you can create an embedding layer and initialize it with the pretrained word embeddings as follows:\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "85ZSxGdzsQrA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "embedding_layer = nn.Embedding.from_pretrained(torch.FloatTensor(model.wv.vectors))"
      ],
      "metadata": {
        "id": "0LfXwAKTsdvf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "* You can then use this embedding layer as part of your language model architecture.\n",
        "\n",
        "* Train and evaluate your language model: Train your language model using your preferred architecture and training setup. During training, the word embeddings can be updated along with other model parameters to improve the language model's performance on your specific task.\n",
        "\n",
        "By using the pre-trained Word2Vec embeddings in your language model, you can benefit from the semantic information captured by the embeddings, which can help improve the model's understanding and generation of text.\n"
      ],
      "metadata": {
        "id": "w4I-eFcHst0G"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "8_ldJ-gIsxfu"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}