5. Let's Learn NLP - Extraction of Keywords & Summarization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9aa5dbbc",
   "metadata": {},
   "source": [
    "![](./resources/images/header_logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4447f721",
   "metadata": {},
   "source": [
    "# Chapter 5 Extraction of Keywords & Summarization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aeb941ee",
   "metadata": {},
   "source": [
    "In this chapter we will learn how to express the most important facts and ideas in a concise way using keyword extraction and summarization."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2a651ec",
   "metadata": {},
   "source": [
    "## Keyword Extraction\n",
    "\n",
    "**Keywords** refer to the \"**important phrases/expressions that are representative of the underlying document**\". [1]\n",
    "\n",
    "[1] 1. Mishra P. 10 Popular Keyword Extraction Algorithms in Natural Language Processing [Internet]. MLearning.ai. 2022 [cited 2022 Mar 11]. Available from: https://medium.com/mlearning-ai/10-popular-keyword-extraction-algorithms-in-natural-language-processing-8975ada5750c\n",
    "\n",
    "There are many different approaches to extracting keywords from a document. These include graph algorithms, keyword scoring techniques and many, many others. But there are also simple methods that you will see in this chapter.\n",
    "\n",
    "The basic method of keyword extraction is to get the most frequent words out of the text."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73b4f180",
   "metadata": {},
   "source": [
    "### Exercise 5.1\n",
    "\n",
    "Write a function `freq` that takes in a spacy doc object and returns a dictionary where each unique token is a key and the value attached to it is the number of occurences of this word in the whole doc. Ignore tokens that are whitespaces, punctuation marks, and stop_words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "631d7bf0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en_core_web_md')\n",
    "# FEEL FREE TO CHANGE THIS TEXT, YOU CAN EVEN PASTE HERE A WHOLE ARTICLE\n",
    "text = \"\"\"This is an example sentence. Use it to check your answer.\"\"\"\n",
    "doc = nlp(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "70dd9d85",
   "metadata": {},
   "outputs": [],
   "source": [
    "def freq(doc):\n",
    "    d = dict()\n",
    "    for token in doc:\n",
    "        if token.is_alpha and not(token.is_space) and not(token.is_punct) and not(token.is_stop):\n",
    "            key = token.text.lower()\n",
    "            if key in d.keys():\n",
    "                d[key] += 1\n",
    "            else:\n",
    "                d[key] = 1\n",
    "    return d"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7aefef3",
   "metadata": {},
   "source": [
    "### Exercise 5.2\n",
    "Write a function `rel_freq` that takes in a doc and uses the function `freq` to change the dictionary values from frequencies into relative frequencies. It should return a dict with unique tokens as keys and their relative frequencies as values. Relative frequency is the number of occurences of the token divided by the number of all the tokens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "82ad87e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def rel_freq(doc):\n",
    "    d = freq(doc)\n",
    "    s = 0 # s is the total number of tokens in the text\n",
    "    for value in d.values():\n",
    "        s += value\n",
    "    for key, values in d.items():\n",
    "        d[key] = value / s\n",
    "    return d"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8f370bf",
   "metadata": {},
   "source": [
    "## Summarization\n",
    "\n",
    "There are two main types of summarization methods:\n",
    "- *extractive*: the program extracts the most important sentences from a given text\n",
    "- *abstractive*: the program tries to understand the text and generates a new summary\n",
    "\n",
    "### Extractive Summarization\n",
    "\n",
    "The **extractive summarization** is much easier than **abstractive**, because it focuses on scoring the sentences based on their importance and choosing the sentences with highest score. \n",
    "\n",
    "One approach to do extractive summarization is based on word statistics:\n",
    "1. Almost as always in text analysis tasks, we do some text preprocessing. We can for instance remove stop_words and punctuation  marks, as they will not hold any valuable information.\n",
    "2. We calculate the **relative frequency** of each word in text, where relative frequence is just the number of occurences of this word over the total number of words in a text. Note: All relative frequencies together should some up to one.\n",
    "3. Next, for each sentence we calculate so-called **sentence strength** by summing up the relative frequencies of its words.\n",
    "4. Finally, we take the sentences with top sentence strength scores and concatenate them together to create a summary\n",
    "\n",
    "Let's test this idea in practice."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71ee5e60",
   "metadata": {},
   "source": [
    "### Exercise 5.3\n",
    "Write a function `top_sentences(doc, n)` that takes in a spacy `doc` object and an integer `n` and returns a list of top `n` most \"important\" sentences from the text using sentence scoring by summing up the relative frequencies of each word in a sentence. The list should have sentences in order from most to less important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "b4674a51",
   "metadata": {},
   "outputs": [],
   "source": [
    "def top_sentences(doc, n):\n",
    "    d = rel_freq(doc)\n",
    "    scores = [] # it will store sentence \"strengths\"\n",
    "    for sent in doc.sents:\n",
    "        score = 0\n",
    "        for token in sent:\n",
    "            if token.text in d.keys():\n",
    "                score += d[token.text]\n",
    "        scores.append((sent.text, score))\n",
    "    scores = sorted(scores, key=lambda item: -item[1]) # descending order\n",
    "    return [item[0] for item in scores]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "896bc19c",
   "metadata": {},
   "source": [
    "## [Video Solutions for 5.1-5.3](https://youtu.be/s_HhM0HqatY)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ca6da69",
   "metadata": {},
   "source": [
    "### Abstractive Summarization\n",
    "\n",
    "**Abstractive summarization** is much more difficult. It uses advanced pre-trained models to paraphrase the most important content of a given text and generate a summary based on it. This involves using Neural Networks and ML mathods, hence due to its high complexity, we will not be focusing on that in this workshop.\n",
    "\n",
    "However, what you should know is that there are already many models for abstractive summarization available online and most of them need just a couple of Python code to start using them. If you want to see some examples, [huggingface](https://huggingface.co/) is a platform for AI community where thousands of different models for various purposes and domains are shared, including some popular and high-performance ones such as T5, GPT2 among others."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44e7f740",
   "metadata": {},
   "source": [
    "## Quiz 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "55f8ca18",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div id=\"xgTLnkbrhemP\" data-shufflequestions=\"False\"\n",
       "               data-shuffleanswers=\"True\"\n",
       "               data-numquestions=\"1000000\"> <style>:root {\n",
       "    --medium-slate-blue: #6f78ffff;\n",
       "    --orange-pantone: #f75c03ff;\n",
       "    --russian-violet: #392061ff;\n",
       "    --maximum-yellow-red: #ffc857ff;\n",
       "    --viridian-green: #119da4ff;\n",
       "    --incorrect-red: #c80202;\n",
       "    --correct-green: #009113;\n",
       "}\n",
       "\n",
       ".Quiz {\n",
       "    max-width: 600px;\n",
       "    margin-top: 15px;\n",
       "    margin-left: auto;\n",
       "    margin-right: auto;\n",
       "    margin-bottom: 15px;\n",
       "    padding-bottom: 4px;\n",
       "    padding-top: 4px;\n",
       "    line-height: 1.1;\n",
       "    font-size: 16pt;\n",
       "}\n",
       "\n",
       ".QuizCode {\n",
       "    font-size: 14pt;\n",
       "    margin-top: 10px;\n",
       "    margin-left: 20px;\n",
       "    margin-right: 20px;\n",
       "}\n",
       "\n",
       ".QuizCode>pre {\n",
       "    padding: 4px;\n",
       "}\n",
       "\n",
       ".Answer {\n",
       "    margin: 10px 0;\n",
       "    display: grid;\n",
       "    grid-template-columns: auto auto;\n",
       "    grid-gap: 10px;\n",
       "}\n",
       "\n",
       ".Feedback {\n",
       "    font-size: 16pt;\n",
       "    text-align: center;\n",
       "    min-height: 2em;\n",
       "}\n",
       "\n",
       ".Input {\n",
       "    align: left;\n",
       "    font-size: 20pt;\n",
       "}\n",
       "\n",
       ".Input-text {\n",
       "    display: block;\n",
       "    margin: 10px;\n",
       "    color: inherit;\n",
       "    width: 140px;\n",
       "    background-color: #c0c0c0;\n",
       "    color: #fff;\n",
       "    padding: 5px;\n",
       "    padding-left: 10px;\n",
       "    font-family: inherit;\n",
       "    font-size: 20px;\n",
       "    font-weight: inherit;\n",
       "    line-height: 20pt;\n",
       "    border: none;\n",
       "    border-radius: 0.2rem;\n",
       "    transition: box-shadow 0.1s);\n",
       "}\n",
       "\n",
       ".Input-text:focus {\n",
       "    outline: none;\n",
       "    background-color: #c0c0c0;\n",
       "    box-shadow: 0.6rem 0.8rem 1.4rem -0.5rem #999999;\n",
       "}\n",
       "\n",
       ".MCButton {\n",
       "    background: #fafafa;\n",
       "    border: 1px solid #eee;\n",
       "    border-radius: 10px;\n",
       "    padding: 10px;\n",
       "    font-size: 16px;\n",
       "    cursor: pointer;\n",
       "    text-align: center;\n",
       "}\n",
       "\n",
       ".MCButton p {\n",
       "    color: inherit;\n",
       "}\n",
       "\n",
       ".MultipleChoiceQn {\n",
       "    padding: 10px;\n",
       "    background: var(--medium-slate-blue);\n",
       "    color: #fafafa;\n",
       "    border-radius: 10px;\n",
       "}\n",
       "\n",
       ".ManyChoiceQn {\n",
       "    padding: 10px;\n",
       "    background: var(--orange-pantone);\n",
       "    color: #fafafa;\n",
       "    border-radius: 10px;\n",
       "}\n",
       "\n",
       ".NumericQn {\n",
       "    padding: 10px;\n",
       "    background: var(--russian-violet);\n",
       "    color: #fafafa;\n",
       "    border-radius: 10px;\n",
       "}\n",
       "\n",
       ".NumericQn p {\n",
       "    color: inherit;\n",
       "}\n",
       "\n",
       ".InpLabel {\n",
       "    line-height: 34px;\n",
       "    float: left;\n",
       "    margin-right: 10px;\n",
       "    color: #101010;\n",
       "    font-size: 15pt;\n",
       "}\n",
       "\n",
       ".incorrect {\n",
       "    color: var(--incorrect-red);\n",
       "}\n",
       "\n",
       ".correct {\n",
       "    color: var(--correct-green);\n",
       "}\n",
       "\n",
       ".correctButton {\n",
       "    /*\n",
       "    background: var(--correct-green);\n",
       "   */\n",
       "    animation: correct-anim 0.6s ease;\n",
       "    animation-fill-mode: forwards;\n",
       "    color: #fafafa;\n",
       "    box-shadow: inset 0px 0px 5px #555555;\n",
       "    outline: none;\n",
       "}\n",
       "\n",
       ".incorrectButton {\n",
       "    animation: incorrect-anim 0.8s ease;\n",
       "    animation-fill-mode: forwards;\n",
       "    color: #fafafa;\n",
       "    box-shadow: inset 0px 0px 5px #555555;\n",
       "    outline: none;\n",
       "}\n",
       "\n",
       "@keyframes incorrect-anim {\n",
       "    100% {\n",
       "        background-color: var(--incorrect-red);\n",
       "    }\n",
       "}\n",
       "\n",
       "@keyframes correct-anim {\n",
       "    100% {\n",
       "        background-color: var(--correct-green);\n",
       "    }\n",
       "}</style><script type=\"text/Javascript\">var questionsxgTLnkbrhemP=[\n",
       "    {\n",
       "        \"question\": \"What is a keyword in NLP?\",\n",
       "        \"type\": \"many_choice\",\n",
       "        \"answers\": [\n",
       "            {\n",
       "                \"answer\": \"The longest word in a document\",\n",
       "                \"correct\": false,\n",
       "                \"feedback\": \"False. The length of the word does not inform us about its importance to the whole text.\"\n",
       "            },\n",
       "            {\n",
       "                \"answer\": \"A word that acts as a key in Python dicionary\",\n",
       "                \"correct\": false,\n",
       "                \"feedback\": \"False. By the way, not only Python is used for NLP tasks.\"\n",
       "            },\n",
       "            {\n",
       "                \"answer\": \"A word of highest emotional intensity\",\n",
       "                \"correct\": false,\n",
       "                \"feedback\": \"False. Words expressing too much emotions are usually not very informative.\"\n",
       "            },\n",
       "            {\n",
       "                \"answer\": \"An important word, phrase or expression that can represent an underlying document in a short and clear way\",\n",
       "                \"correct\": true,\n",
       "                \"feedback\": \"Correct.\"\n",
       "            }\n",
       "        ]\n",
       "    },\n",
       "    {\n",
       "        \"question\": \"How does abstractive summarization work?\",\n",
       "        \"type\": \"multiple_choice\",\n",
       "        \"answers\": [\n",
       "            {\n",
       "                \"answer\": \"It chooses the most important sentences using a scoring algorithm and concatanates them\",\n",
       "                \"correct\": false,\n",
       "                \"feedback\": \"False.\"\n",
       "            },\n",
       "            {\n",
       "                \"answer\": \"It uses advanced AI methods to paraphrase existing text and generate a summary\",\n",
       "                \"correct\": true,\n",
       "                \"feedback\": \"True.\"\n",
       "            },\n",
       "            {\n",
       "                \"answer\": \"It searches for an abstract of a paper in a document and prints it.\",\n",
       "                \"correct\": false,\n",
       "                \"feedback\": \"False. It would be useless. Also, not every text has an abstract.\"\n",
       "            },\n",
       "            {\n",
       "                \"answer\": \"Generates sentences using extracted keywords\",\n",
       "                \"correct\": false,\n",
       "                \"feedback\": \"False. Access to keywords only is not enough. Generating a sentence based on them will usually be totally unrelated to the original text.\"\n",
       "            }\n",
       "        ]\n",
       "    },\n",
       "    {\n",
       "        \"question\": \"How does extractive summarization work?\",\n",
       "        \"type\": \"multiple_choice\",\n",
       "        \"answers\": [\n",
       "            {\n",
       "                \"answer\": \"It chooses the most important sentences using a scoring algorithm and concatanates them\",\n",
       "                \"correct\": true,\n",
       "                \"feedback\": \"True. For example, relative frequency can be used.\"\n",
       "            },\n",
       "            {\n",
       "                \"answer\": \"It uses advanced AI methods to paraphrase existing text and generate a summary\",\n",
       "                \"correct\": false,\n",
       "                \"feedback\": \"False. This is abstractive summarization.\"\n",
       "            },\n",
       "            {\n",
       "                \"answer\": \"It searches for an abstract of a paper in a document and prints it.\",\n",
       "                \"correct\": false,\n",
       "                \"feedback\": \"False. It would be useless. Also, not every text has an abstract.\"\n",
       "            },\n",
       "            {\n",
       "                \"answer\": \"Generates sentences using extracted keywords\",\n",
       "                \"correct\": false,\n",
       "                \"feedback\": \"False. Extractive summarization chooses existing sentences.\"\n",
       "            }\n",
       "        ]\n",
       "    }\n",
       "   \n",
       "];\n",
       "    // Make a random ID\n",
       "function makeid(length) {\n",
       "    var result = [];\n",
       "    var characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';\n",
       "    var charactersLength = characters.length;\n",
       "    for (var i = 0; i < length; i++) {\n",
       "        result.push(characters.charAt(Math.floor(Math.random() * charactersLength)));\n",
       "    }\n",
       "    return result.join('');\n",
       "}\n",
       "\n",
       "// Choose a random subset of an array. Can also be used to shuffle the array\n",
       "function getRandomSubarray(arr, size) {\n",
       "    var shuffled = arr.slice(0), i = arr.length, temp, index;\n",
       "    while (i--) {\n",
       "        index = Math.floor((i + 1) * Math.random());\n",
       "        temp = shuffled[index];\n",
       "        shuffled[index] = shuffled[i];\n",
       "        shuffled[i] = temp;\n",
       "    }\n",
       "    return shuffled.slice(0, size);\n",
       "}\n",
       "function check_mc() {\n",
       "    var id = this.id.split('-')[0];\n",
       "\n",
       "    //console.log(\"In check_mc(), id=\"+id);\n",
       "    //console.log(event.srcElement.id)           \n",
       "    //console.log(event.srcElement.dataset.correct)   \n",
       "    //console.log(event.srcElement.dataset.feedback)\n",
       "\n",
       "    var label = event.srcElement;\n",
       "    //console.log(label, label.nodeName);\n",
       "    var depth = 0;\n",
       "    while ((label.nodeName != \"LABEL\") && (depth < 20)) {\n",
       "        label = label.parentElement;\n",
       "        console.log(depth, label);\n",
       "        depth++;\n",
       "    }\n",
       "\n",
       "\n",
       "\n",
       "    var answers = label.parentElement.children;\n",
       "\n",
       "    //console.log(answers);\n",
       "\n",
       "\n",
       "    // Split behavior based on multiple choice vs many choice:\n",
       "    var fb = document.getElementById(\"fb\" + id);\n",
       "    //console.log(id, \", got numcorrect=\",fb.dataset.numcorrect);\n",
       "    if (fb.dataset.numcorrect == 1) {\n",
       "        for (var i = 0; i < answers.length; i++) {\n",
       "            var child = answers[i];\n",
       "            //console.log(child);\n",
       "            child.className = \"MCButton\";\n",
       "        }\n",
       "\n",
       "\n",
       "\n",
       "        if (label.dataset.correct == \"true\") {\n",
       "            // console.log(\"Correct action\");\n",
       "            if (\"feedback\" in label.dataset) {\n",
       "                fb.textContent = jaxify(label.dataset.feedback);\n",
       "            } else {\n",
       "                fb.textContent = \"Correct!\";\n",
       "            }\n",
       "            label.classList.add(\"correctButton\");\n",
       "\n",
       "            fb.className = \"Feedback\";\n",
       "            fb.classList.add(\"correct\");\n",
       "\n",
       "        } else {\n",
       "            if (\"feedback\" in label.dataset) {\n",
       "                fb.textContent = jaxify(label.dataset.feedback);\n",
       "            } else {\n",
       "                fb.textContent = \"Incorrect -- try again.\";\n",
       "            }\n",
       "            //console.log(\"Error action\");\n",
       "            label.classList.add(\"incorrectButton\");\n",
       "            fb.className = \"Feedback\";\n",
       "            fb.classList.add(\"incorrect\");\n",
       "        }\n",
       "    }\n",
       "    else {\n",
       "        var reset = false;\n",
       "        var feedback;\n",
       "        if (label.dataset.correct == \"true\") {\n",
       "            if (\"feedback\" in label.dataset) {\n",
       "                feedback = jaxify(label.dataset.feedback);\n",
       "            } else {\n",
       "                feedback = \"Correct!\";\n",
       "            }\n",
       "            if (label.dataset.answered <= 0) {\n",
       "                if (fb.dataset.answeredcorrect < 0) {\n",
       "                    fb.dataset.answeredcorrect = 1;\n",
       "                    reset = true;\n",
       "                } else {\n",
       "                    fb.dataset.answeredcorrect++;\n",
       "                }\n",
       "                if (reset) {\n",
       "                    for (var i = 0; i < answers.length; i++) {\n",
       "                        var child = answers[i];\n",
       "                        child.className = \"MCButton\";\n",
       "                        child.dataset.answered = 0;\n",
       "                    }\n",
       "                }\n",
       "                label.classList.add(\"correctButton\");\n",
       "                label.dataset.answered = 1;\n",
       "                fb.className = \"Feedback\";\n",
       "                fb.classList.add(\"correct\");\n",
       "\n",
       "            }\n",
       "        } else {\n",
       "            if (\"feedback\" in label.dataset) {\n",
       "                feedback = jaxify(label.dataset.feedback);\n",
       "            } else {\n",
       "                feedback = \"Incorrect -- try again.\";\n",
       "            }\n",
       "            if (fb.dataset.answeredcorrect > 0) {\n",
       "                fb.dataset.answeredcorrect = -1;\n",
       "                reset = true;\n",
       "            } else {\n",
       "                fb.dataset.answeredcorrect--;\n",
       "            }\n",
       "\n",
       "            if (reset) {\n",
       "                for (var i = 0; i < answers.length; i++) {\n",
       "                    var child = answers[i];\n",
       "                    child.className = \"MCButton\";\n",
       "                    child.dataset.answered = 0;\n",
       "                }\n",
       "            }\n",
       "            label.classList.add(\"incorrectButton\");\n",
       "            fb.className = \"Feedback\";\n",
       "            fb.classList.add(\"incorrect\");\n",
       "        }\n",
       "\n",
       "\n",
       "        var numcorrect = fb.dataset.numcorrect;\n",
       "        var answeredcorrect = fb.dataset.answeredcorrect;\n",
       "        if (answeredcorrect >= 0) {\n",
       "            fb.textContent = feedback + \" [\" + answeredcorrect + \"/\" + numcorrect + \"]\";\n",
       "        } else {\n",
       "            fb.textContent = feedback + \" [\" + 0 + \"/\" + numcorrect + \"]\";\n",
       "        }\n",
       "\n",
       "\n",
       "    }\n",
       "\n",
       "    if (typeof MathJax != 'undefined') {\n",
       "        var version = MathJax.version;\n",
       "        console.log('MathJax version', version);\n",
       "        if (version[0] == \"2\") {\n",
       "            MathJax.Hub.Queue([\"Typeset\", MathJax.Hub]);\n",
       "        } else if (version[0] == \"3\") {\n",
       "            MathJax.typeset([fb]);\n",
       "        }\n",
       "    } else {\n",
       "        console.log('MathJax not detected');\n",
       "    }\n",
       "\n",
       "}\n",
       "\n",
       "function make_mc(qa, shuffle_answers, outerqDiv, qDiv, aDiv, id) {\n",
       "    var shuffled;\n",
       "    if (shuffle_answers == \"True\") {\n",
       "        //console.log(shuffle_answers+\" read as true\");\n",
       "        shuffled = getRandomSubarray(qa.answers, qa.answers.length);\n",
       "    } else {\n",
       "        //console.log(shuffle_answers+\" read as false\");\n",
       "        shuffled = qa.answers;\n",
       "    }\n",
       "\n",
       "\n",
       "    var num_correct = 0;\n",
       "\n",
       "\n",
       "\n",
       "    shuffled.forEach((item, index, ans_array) => {\n",
       "        //console.log(answer);\n",
       "\n",
       "        // Make input element\n",
       "        var inp = document.createElement(\"input\");\n",
       "        inp.type = \"radio\";\n",
       "        inp.id = \"quizo\" + id + index;\n",
       "        inp.style = \"display:none;\";\n",
       "        aDiv.append(inp);\n",
       "\n",
       "        //Make label for input element\n",
       "        var lab = document.createElement(\"label\");\n",
       "        lab.className = \"MCButton\";\n",
       "        lab.id = id + '-' + index;\n",
       "        lab.onclick = check_mc;\n",
       "        var aSpan = document.createElement('span');\n",
       "        aSpan.classsName = \"\";\n",
       "        //qDiv.id=\"quizQn\"+id+index;\n",
       "        if (\"answer\" in item) {\n",
       "            aSpan.innerHTML = jaxify(item.answer);\n",
       "            //aSpan.innerHTML=item.answer;\n",
       "        }\n",
       "        lab.append(aSpan);\n",
       "\n",
       "        // Create div for code inside question\n",
       "        var codeSpan;\n",
       "        if (\"code\" in item) {\n",
       "            codeSpan = document.createElement('span');\n",
       "            codeSpan.id = \"code\" + id + index;\n",
       "            codeSpan.className = \"QuizCode\";\n",
       "            var codePre = document.createElement('pre');\n",
       "            codeSpan.append(codePre);\n",
       "            var codeCode = document.createElement('code');\n",
       "            codePre.append(codeCode);\n",
       "            codeCode.innerHTML = item.code;\n",
       "            lab.append(codeSpan);\n",
       "            //console.log(codeSpan);\n",
       "        }\n",
       "\n",
       "        //lab.textContent=item.answer;\n",
       "\n",
       "        // Set the data attributes for the answer\n",
       "        lab.setAttribute('data-correct', item.correct);\n",
       "        if (item.correct) {\n",
       "            num_correct++;\n",
       "        }\n",
       "        if (\"feedback\" in item) {\n",
       "            lab.setAttribute('data-feedback', item.feedback);\n",
       "        }\n",
       "        lab.setAttribute('data-answered', 0);\n",
       "\n",
       "        aDiv.append(lab);\n",
       "\n",
       "    });\n",
       "\n",
       "    if (num_correct > 1) {\n",
       "        outerqDiv.className = \"ManyChoiceQn\";\n",
       "    } else {\n",
       "        outerqDiv.className = \"MultipleChoiceQn\";\n",
       "    }\n",
       "\n",
       "    return num_correct;\n",
       "\n",
       "}\n",
       "function check_numeric(ths, event) {\n",
       "\n",
       "    if (event.keyCode === 13) {\n",
       "        ths.blur();\n",
       "\n",
       "        var id = ths.id.split('-')[0];\n",
       "\n",
       "        var submission = ths.value;\n",
       "        if (submission.indexOf('/') != -1) {\n",
       "            sub_parts = submission.split('/');\n",
       "            //console.log(sub_parts);\n",
       "            submission = sub_parts[0] / sub_parts[1];\n",
       "        }\n",
       "        //console.log(\"Reader entered\", submission);\n",
       "\n",
       "        if (\"precision\" in ths.dataset) {\n",
       "            precision = ths.dataset.precision;\n",
       "            // console.log(\"1:\", submission)\n",
       "            submission = Math.round((1 * submission + Number.EPSILON) * 10 ** precision) / 10 ** precision;\n",
       "            // console.log(\"Rounded to \", submission, \" precision=\", precision  );\n",
       "        }\n",
       "\n",
       "\n",
       "        //console.log(\"In check_numeric(), id=\"+id);\n",
       "        //console.log(event.srcElement.id)           \n",
       "        //console.log(event.srcElement.dataset.feedback)\n",
       "\n",
       "        var fb = document.getElementById(\"fb\" + id);\n",
       "        fb.style.display = \"none\";\n",
       "        fb.textContent = \"Incorrect -- try again.\";\n",
       "\n",
       "        answers = JSON.parse(ths.dataset.answers);\n",
       "        //console.log(answers);\n",
       "\n",
       "        var defaultFB = \"\";\n",
       "        var correct;\n",
       "        var done = false;\n",
       "        answers.every(answer => {\n",
       "            //console.log(answer.type);\n",
       "\n",
       "            correct = false;\n",
       "            // if (answer.type==\"value\"){\n",
       "            if ('value' in answer) {\n",
       "                if (submission == answer.value) {\n",
       "                    fb.textContent = jaxify(answer.feedback);\n",
       "                    correct = answer.correct;\n",
       "                    //console.log(answer.correct);\n",
       "                    done = true;\n",
       "                }\n",
       "                // } else if (answer.type==\"range\") {\n",
       "            } else if ('range' in answer) {\n",
       "                //console.log(answer.range);\n",
       "                if ((submission >= answer.range[0]) && (submission < answer.range[1])) {\n",
       "                    fb.textContent = jaxify(answer.feedback);\n",
       "                    correct = answer.correct;\n",
       "                    //console.log(answer.correct);\n",
       "                    done = true;\n",
       "                }\n",
       "            } else if (answer.type == \"default\") {\n",
       "                defaultFB = answer.feedback;\n",
       "            }\n",
       "            if (done) {\n",
       "                return false; // Break out of loop if this has been marked correct\n",
       "            } else {\n",
       "                return true; // Keep looking for case that includes this as a correct answer\n",
       "            }\n",
       "        });\n",
       "\n",
       "        if ((!done) && (defaultFB != \"\")) {\n",
       "            fb.innerHTML = jaxify(defaultFB);\n",
       "            //console.log(\"Default feedback\", defaultFB);\n",
       "        }\n",
       "\n",
       "        fb.style.display = \"block\";\n",
       "        if (correct) {\n",
       "            ths.className = \"Input-text\";\n",
       "            ths.classList.add(\"correctButton\");\n",
       "            fb.className = \"Feedback\";\n",
       "            fb.classList.add(\"correct\");\n",
       "        } else {\n",
       "            ths.className = \"Input-text\";\n",
       "            ths.classList.add(\"incorrectButton\");\n",
       "            fb.className = \"Feedback\";\n",
       "            fb.classList.add(\"incorrect\");\n",
       "        }\n",
       "        if (typeof MathJax != 'undefined') {\n",
       "            var version = MathJax.version;\n",
       "            console.log('MathJax version', version);\n",
       "            if (version[0] == \"2\") {\n",
       "                MathJax.Hub.Queue([\"Typeset\", MathJax.Hub]);\n",
       "            } else if (version[0] == \"3\") {\n",
       "                MathJax.typeset([fb]);\n",
       "            }\n",
       "        } else {\n",
       "            console.log('MathJax not detected');\n",
       "        }\n",
       "        return false;\n",
       "    }\n",
       "\n",
       "}\n",
       "\n",
       "function isValid(el, charC) {\n",
       "    //console.log(\"Input char: \", charC);\n",
       "    if (charC == 46) {\n",
       "        if (el.value.indexOf('.') === -1) {\n",
       "            return true;\n",
       "        } else if (el.value.indexOf('/') != -1) {\n",
       "            parts = el.value.split('/');\n",
       "            if (parts[1].indexOf('.') === -1) {\n",
       "                return true;\n",
       "            }\n",
       "        }\n",
       "        else {\n",
       "            return false;\n",
       "        }\n",
       "    } else if (charC == 47) {\n",
       "        if (el.value.indexOf('/') === -1) {\n",
       "            if ((el.value != \"\") && (el.value != \".\")) {\n",
       "                return true;\n",
       "            } else {\n",
       "                return false;\n",
       "            }\n",
       "        } else {\n",
       "            return false;\n",
       "        }\n",
       "    } else if (charC == 45) {\n",
       "        edex = el.value.indexOf('e');\n",
       "        if (edex == -1) {\n",
       "            edex = el.value.indexOf('E');\n",
       "        }\n",
       "\n",
       "        if (el.value == \"\") {\n",
       "            return true;\n",
       "        } else if (edex == (el.value.length - 1)) { // If just after e or E\n",
       "            return true;\n",
       "        } else {\n",
       "            return false;\n",
       "        }\n",
       "    } else if (charC == 101) { // \"e\"\n",
       "        if ((el.value.indexOf('e') === -1) && (el.value.indexOf('E') === -1) && (el.value.indexOf('/') == -1)) {\n",
       "            // Prev symbol must be digit or decimal point:\n",
       "            if (el.value.slice(-1).search(/\\d/) >= 0) {\n",
       "                return true;\n",
       "            } else if (el.value.slice(-1).search(/\\./) >= 0) {\n",
       "                return true;\n",
       "            } else {\n",
       "                return false;\n",
       "            }\n",
       "        } else {\n",
       "            return false;\n",
       "        }\n",
       "    } else {\n",
       "        if (charC > 31 && (charC < 48 || charC > 57))\n",
       "            return false;\n",
       "    }\n",
       "    return true;\n",
       "}\n",
       "\n",
       "function numeric_keypress(evnt) {\n",
       "    var charC = (evnt.which) ? evnt.which : evnt.keyCode;\n",
       "\n",
       "    if (charC == 13) {\n",
       "        check_numeric(this, evnt);\n",
       "    } else {\n",
       "        return isValid(this, charC);\n",
       "    }\n",
       "}\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "function make_numeric(qa, outerqDiv, qDiv, aDiv, id) {\n",
       "\n",
       "\n",
       "\n",
       "    //console.log(answer);\n",
       "\n",
       "\n",
       "    outerqDiv.className = \"NumericQn\";\n",
       "    aDiv.style.display = 'block';\n",
       "\n",
       "    var lab = document.createElement(\"label\");\n",
       "    lab.className = \"InpLabel\";\n",
       "    lab.textContent = \"Type numeric answer here:\";\n",
       "    aDiv.append(lab);\n",
       "\n",
       "    var inp = document.createElement(\"input\");\n",
       "    inp.type = \"text\";\n",
       "    //inp.id=\"input-\"+id;\n",
       "    inp.id = id + \"-0\";\n",
       "    inp.className = \"Input-text\";\n",
       "    inp.setAttribute('data-answers', JSON.stringify(qa.answers));\n",
       "    if (\"precision\" in qa) {\n",
       "        inp.setAttribute('data-precision', qa.precision);\n",
       "    }\n",
       "    aDiv.append(inp);\n",
       "    //console.log(inp);\n",
       "\n",
       "    //inp.addEventListener(\"keypress\", check_numeric);\n",
       "    //inp.addEventListener(\"keypress\", numeric_keypress);\n",
       "    /*\n",
       "    inp.addEventListener(\"keypress\", function(event) {\n",
       "        return numeric_keypress(this, event);\n",
       "    }\n",
       "                        );\n",
       "                        */\n",
       "    //inp.onkeypress=\"return numeric_keypress(this, event)\";\n",
       "    inp.onkeypress = numeric_keypress;\n",
       "    inp.onpaste = event => false;\n",
       "\n",
       "    inp.addEventListener(\"focus\", function (event) {\n",
       "        this.value = \"\";\n",
       "        return false;\n",
       "    }\n",
       "    );\n",
       "\n",
       "\n",
       "}\n",
       "function jaxify(string) {\n",
       "    var mystring = string;\n",
       "\n",
       "    count = 0;\n",
       "    var loc = mystring.search(/([^\\\\]|^)(\\$)/);\n",
       "\n",
       "    count2 = 0;\n",
       "    var loc2 = mystring.search(/([^\\\\]|^)(\\$\\$)/);\n",
       "\n",
       "    //console.log(loc);\n",
       "\n",
       "    while ((loc >= 0) || (loc2 >= 0)) {\n",
       "\n",
       "        /* Have to replace all the double $$ first with current implementation */\n",
       "        if (loc2 >= 0) {\n",
       "            if (count2 % 2 == 0) {\n",
       "                mystring = mystring.replace(/([^\\\\]|^)(\\$\\$)/, \"$1\\\\[\");\n",
       "            } else {\n",
       "                mystring = mystring.replace(/([^\\\\]|^)(\\$\\$)/, \"$1\\\\]\");\n",
       "            }\n",
       "            count2++;\n",
       "        } else {\n",
       "            if (count % 2 == 0) {\n",
       "                mystring = mystring.replace(/([^\\\\]|^)(\\$)/, \"$1\\\\(\");\n",
       "            } else {\n",
       "                mystring = mystring.replace(/([^\\\\]|^)(\\$)/, \"$1\\\\)\");\n",
       "            }\n",
       "            count++;\n",
       "        }\n",
       "        loc = mystring.search(/([^\\\\]|^)(\\$)/);\n",
       "        loc2 = mystring.search(/([^\\\\]|^)(\\$\\$)/);\n",
       "        //console.log(mystring,\", loc:\",loc,\", loc2:\",loc2);\n",
       "    }\n",
       "\n",
       "    //console.log(mystring);\n",
       "    return mystring;\n",
       "}\n",
       "\n",
       "\n",
       "function show_questions(json, mydiv) {\n",
       "    //var mydiv=document.getElementById(myid);\n",
       "    var shuffle_questions = mydiv.dataset.shufflequestions;\n",
       "    var num_questions = mydiv.dataset.numquestions;\n",
       "    var shuffle_answers = mydiv.dataset.shuffleanswers;\n",
       "\n",
       "    if (num_questions > json.length) {\n",
       "        num_questions = json.length;\n",
       "    }\n",
       "\n",
       "    var questions;\n",
       "    if ((num_questions < json.length) || (shuffle_questions == \"True\")) {\n",
       "        //console.log(num_questions+\",\"+json.length);\n",
       "        questions = getRandomSubarray(json, num_questions);\n",
       "    } else {\n",
       "        questions = json;\n",
       "    }\n",
       "\n",
       "    //console.log(\"SQ: \"+shuffle_questions+\", NQ: \" + num_questions + \", SA: \", shuffle_answers);\n",
       "\n",
       "    // Iterate over questions\n",
       "    questions.forEach((qa, index, array) => {\n",
       "        //console.log(qa.question); \n",
       "\n",
       "        var id = makeid(8);\n",
       "        //console.log(id);\n",
       "\n",
       "\n",
       "        // Create Div to contain question and answers\n",
       "        var iDiv = document.createElement('div');\n",
       "        iDiv.id = 'quizWrap' + id + index;\n",
       "        iDiv.className = 'Quiz';\n",
       "        mydiv.appendChild(iDiv);\n",
       "        // iDiv.innerHTML=qa.question;\n",
       "\n",
       "        var outerqDiv = document.createElement('div');\n",
       "        outerqDiv.id = \"OuterquizQn\" + id + index;\n",
       "        //qDiv.textContent=qa.question;\n",
       "        iDiv.append(outerqDiv);\n",
       "\n",
       "        // Create div to contain question part\n",
       "        var qDiv = document.createElement('div');\n",
       "        qDiv.id = \"quizQn\" + id + index;\n",
       "        //qDiv.textContent=qa.question;\n",
       "        qDiv.innerHTML = jaxify(qa.question);\n",
       "        outerqDiv.append(qDiv);\n",
       "\n",
       "        // Create div for code inside question\n",
       "        var codeDiv;\n",
       "        if (\"code\" in qa) {\n",
       "            codeDiv = document.createElement('div');\n",
       "            codeDiv.id = \"code\" + id + index;\n",
       "            codeDiv.className = \"QuizCode\";\n",
       "            var codePre = document.createElement('pre');\n",
       "            codeDiv.append(codePre);\n",
       "            var codeCode = document.createElement('code');\n",
       "            codePre.append(codeCode);\n",
       "            codeCode.innerHTML = qa.code;\n",
       "            outerqDiv.append(codeDiv);\n",
       "            //console.log(codeDiv);\n",
       "        }\n",
       "\n",
       "\n",
       "        // Create div to contain answer part\n",
       "        var aDiv = document.createElement('div');\n",
       "        aDiv.id = \"quizAns\" + id + index;\n",
       "        aDiv.className = 'Answer';\n",
       "        iDiv.append(aDiv);\n",
       "\n",
       "        //console.log(qa.type);\n",
       "\n",
       "        var num_correct;\n",
       "        if (qa.type == \"multiple_choice\") {\n",
       "            num_correct = make_mc(qa, shuffle_answers, outerqDiv, qDiv, aDiv, id);\n",
       "        } else if (qa.type == \"many_choice\") {\n",
       "            num_correct = make_mc(qa, shuffle_answers, outerqDiv, qDiv, aDiv, id);\n",
       "        } else if (qa.type == \"numeric\") {\n",
       "            //console.log(\"numeric\");\n",
       "            make_numeric(qa, outerqDiv, qDiv, aDiv, id);\n",
       "        }\n",
       "\n",
       "\n",
       "        //Make div for feedback\n",
       "        var fb = document.createElement(\"div\");\n",
       "        fb.id = \"fb\" + id;\n",
       "        //fb.style=\"font-size: 20px;text-align:center;\";\n",
       "        fb.className = \"Feedback\";\n",
       "        fb.setAttribute(\"data-answeredcorrect\", 0);\n",
       "        fb.setAttribute(\"data-numcorrect\", num_correct);\n",
       "        iDiv.append(fb);\n",
       "\n",
       "\n",
       "    });\n",
       "    //console.log(\"At end of show_questions\");\n",
       "    if (typeof MathJax != 'undefined') {\n",
       "        console.log(\"MathJax version\", MathJax.version);\n",
       "        var version = MathJax.version;\n",
       "        setTimeout(function(){\n",
       "            var version = MathJax.version;\n",
       "            console.log('After sleep, MathJax version', version);\n",
       "            if (version[0] == \"2\") {\n",
       "                MathJax.Hub.Queue([\"Typeset\", MathJax.Hub]);\n",
       "            } else if (version[0] == \"3\") {\n",
       "                MathJax.typeset([mydiv]);\n",
       "            }\n",
       "        }, 500);\n",
       "if (typeof version == 'undefined') {\n",
       "        } else\n",
       "        {\n",
       "            if (version[0] == \"2\") {\n",
       "                MathJax.Hub.Queue([\"Typeset\", MathJax.Hub]);\n",
       "            } else if (version[0] == \"3\") {\n",
       "                MathJax.typeset([mydiv]);\n",
       "            } else {\n",
       "                console.log(\"MathJax not found\");\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "    return false;\n",
       "}\n",
       "\n",
       "        {\n",
       "        show_questions(questionsxgTLnkbrhemP,  xgTLnkbrhemP);\n",
       "        }\n",
       "        </script>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from jupyterquiz import display_quiz\n",
    "display_quiz(\"resources/quizzes/questions5.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77c6bc20",
   "metadata": {},
   "source": [
    "# Summary and Feedback\n",
    "\n",
    "Well done! You made it! This is the end of this workshop. \n",
    "\n",
    "We hope you have learned something and gained interest in NLP or generally Computer Science.\n",
    "Give us some feedback by rating the workshop in the form below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "89039721",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"750px\"\n",
       "            src=\"https://docs.google.com/forms/d/e/1FAIpQLScnerXm7The1YtcGzE3EvFNALf83SGpvb9fhhbKK-kOiIYdPg/viewform?embedded=true\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1cfd0e675b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import IFrame\n",
    "\n",
    "link = \"https://docs.google.com/forms/d/e/1FAIpQLScnerXm7The1YtcGzE3EvFNALf83SGpvb9fhhbKK-kOiIYdPg/viewform?embedded=true\"\n",
    "iframe = IFrame(link, width=\"100%\", height=\"750px\")\n",
    "display(iframe)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42a5a7fd",
   "metadata": {},
   "source": [
    "# Thank you!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nlp_env",
   "language": "python",
   "name": "nlp_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}