6. Named Entity Recognition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Named Entity Recognition (NER)\n",
    "Named Entity Recognition (NER) is a technique used in NLP to recognize named entities in text into categories like person, place, company or date.<br><br>\n",
    "For example, let's consider the following sentence:<br><br>\n",
    "\n",
    "\"Tesla opened new office in Tokyo today\"<br><br>\n",
    "\n",
    "1. <b>Organization:</b> Tesla<br>\n",
    "2. <b>Location:</b> Tokyo<br>\n",
    "3. <b>Date:</b> today"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NER Techniques:\n",
    "There are several approaches to NER, but two main categories are:<br>\n",
    "#### 1. Rules - based NER:\n",
    "This method involves recognizing entities based on predefined set of rules and dictionaries.<br><br>\n",
    "<b>a. Pattern-based rules:</b><br><br>\n",
    "These rules deal with the morphological structure of words.<br><br>\n",
    "e.g: If a word ends with the suffix '-field' or '-land', it is probably a location as it is commonly associated with city/town names.<br><br>\n",
    "<b>b. Context-based rules:</b><br><br>\n",
    "These rules deal with the surrounding words of a particular word to classify it.<br><br>\n",
    "e.g: If \"Dr.\", \"Mr.\", \"Mrs.\" or \"Miss\" appears before a word, it is probably the name of a person.<br><br>\n",
    "<b>c. Dictionary approach:</b><br><br>\n",
    "In this approach, we define a dictionary containing words and their possible entity names and for every word, we check every token/word against this dictionary to assign possible entity names to it.\n",
    "\n",
    "#### 2. ML - based NER:\n",
    "In this approach, we train ML models on large chunks of labelled data.<br>\n",
    "\n",
    "There are two main models used for NER:<br><br>\n",
    "<b>a. Conditional Random Fields (CRFs):</b><br><br>\n",
    "In CRF, y_i (tag for current word) depends only on y_i-1 (tag for previous word).<br><br>\n",
    "We give a sequence of words as input and the model computes the probability of a sequence of tags. <br><br>\n",
    "For example, for the sequence \"Ali is an Engineer\", the goal of the model would be to maximize probability of \"Person O O Profession\".<br><br>\n",
    "Here, O means not a named entity.<br><br>\n",
    "\n",
    "<b>b. Bi-LSTM combined with CRF:</b><br><br>\n",
    "The problem with CRF is, it is blind to words that come after the target word. <br><br>\n",
    "For example, \"I went to Harvard University.\"<br><br>\n",
    "In this sentence, a CRF will recognize Harvard as a person name because it can't see the \"University\" that comes after it. <br><br>\n",
    "Hence, we connect Bi-directional LSTM between the inputs and CRF to make our system more context aware.<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NER with spaCy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "def ner_spacy(sentence):\n",
    "  nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "  doc = nlp(sentence)\n",
    "  named_entities = [token.ent_iob_ + \"-\" + token.ent_type_ for token in doc]\n",
    "  \n",
    "  return named_entities, doc\n",
    "\n",
    "entities, doc = ner_spacy(\"Tesla opened a new office in Tokyo today\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['B-ORG', 'O-', 'O-', 'O-', 'O-', 'O-', 'B-GPE', 'B-DATE']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Visualizing the results\n",
    "We can also visualize the results with displacy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Tesla\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " opened a new office in \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Tokyo\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    today\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       "</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "  \n",
    "from spacy import displacy\n",
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NER with nltk\n",
    "Here's a simple function that takes in a piece of text and returns a dictionary of named entities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "def ner_nltk(text):\n",
    "    entities = {}\n",
    "\n",
    "    tokens = nltk.sent_tokenize(text)\n",
    "    for sent in tokens:\n",
    "        word_tokens = nltk.word_tokenize(sent)\n",
    "        for i, chunk in enumerate(nltk.ne_chunk(nltk.pos_tag(word_tokens))):\n",
    "            if hasattr(chunk, 'label'):\n",
    "                entities[word_tokens[i]] = chunk.label()\n",
    "                \n",
    "    return entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Tesla': 'PERSON', 'Tokyo': 'GPE'}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"Tesla opened a new office in Tokyo today\"\n",
    "ner_nltk(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "demo",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}