Research_Project.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"
    },
    "colab": {
      "name": "Research_Project.ipynb",
      "provenance": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iPrXUcQXaikc"
      },
      "source": [
        "# Research Project\n",
        "\n",
        "------------------------------------------------------\n",
        "\n",
        "## Neural Networks\n",
        "\n",
        "**Enrique Botía Barberá** - 100406632\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "buOjq9acaikn"
      },
      "source": [
        "# Introduction\n",
        "For this project was required to select a paper about a deep learning topic and reproduce one or more illustrative experiments. I have chosen the paper [Deep Learning for Symbolic Mathematics](https://arxiv.org/pdf/1912.01412v1.pdf), using the code in the github repository [SymbolicMathematics](https://github.com/facebookresearch/SymbolicMathematics), developed by facebook. In the following I will explain the content of the paper and then show its implementation and some experiments that I have performed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PmSIQ3WMLOQp"
      },
      "source": [
        "# Explanation of the paper\n",
        "The authors face the challenge to map symbolic reasoning into the real space. Two symbolic mathematical problems are considered: symbolic integrations and differential equations.The content of the paper is focused mainly in how to represent and generate the data to train the model, to later apply a seq2seq model to predict the solution of the given operation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iTDVEXBALOTD"
      },
      "source": [
        "## Mathematics as a natural language\n",
        "Trees are a suitable representation of mathematical expressions, but tree-to-tree models are more involved and slower than seq2seq models. So, to get the dataset (composed by the problems and its corresponding solutions) random trees are generated and then converted to the prefix notation, e.g. the expression \"$2 + 3∗(5+ 2)$\" in the prefix notation is represented as the sequence \"$[+ 2 ∗ 3 + 5\\,2]$\" and also as a tree:\n",
        "\n",
        "<img src='https://drive.google.com/uc?id=1uILwHqYen9QoGTuaFFHkBgtXB-EcwOce'>\n",
        "\n",
        "The algorithm that they implement to compute random trees with equal probability can be found in the appendix of the paper.\n",
        "\n",
        "\n",
        "They also analyse the number all of possible to trees, under different constraints. This will allow them to know the needed size the sample, so that it is representative in terms of the total problem space. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lOZIqLTdLOVV"
      },
      "source": [
        "## Generating datasets\n",
        "For the task handled, solutions of random problems sometimes cannot be easily derived or don't exist. Because of this, it is need to design a specialized way to generate the data in each of the problems.\n",
        "\n",
        "### Integration\n",
        "- **Foward generation (FWD)**: generate random expressions with the algorithm that is showed in the paper and solve the integrations with an external math framework.\n",
        "- **Backward generation (BWD)**: as FWD, but computing the derivative instead, which is always possible, faster and independent of external resources. Once computed, the derivative is added the dataset as the problem and the original expression as its corresponding solution.\n",
        "- **Backward generation with integration by parts (IBP)**: use integrations by parts to generate examples only by derivating, as FWD.\n",
        "\n",
        "In FWD and IBP the integral (the solution) tends to be longer than the derivative (the problem), while BWD generation favors the opposite. So, a mixture of BWD and IBP generated data should therefore provide a better representation of problem space, without resorting to external tools.\n",
        "\n",
        "### ODEs\n",
        "The procedure followed can be understood with the examples that are presented in the paper.\n",
        "For the first order differential equation (ODE 1):\n",
        "<img src='https://drive.google.com/uc?id=1NzQHeMba52XxgiUB5l9YqPUYdG1MUQ_d'>\n",
        "\n",
        "And for the second order differential equation (ODE 2):\n",
        "\n",
        "<img src='https://drive.google.com/uc?id=1Ysn4IAFJ0M5a2o3NkE4fSDNXTAXlGFGh'>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dJIodtSWLOYD"
      },
      "source": [
        "## Experiments performed by the authors of the paper\n",
        "### Dataset\n",
        "The datasets created were restricted to a maximum size, only one variable, and integer numbers in the region {-5,...,5}.\n",
        "### Model\n",
        "It was used a [transformer model](https://arxiv.org/pdf/1706.03762.pdf) with 8 attention heads, 6 layers, and a dimensionality of 512. They used the Adam optimizer with a learning rate of $10^{-4}$, and 256 equations per batch. At inference, expressions are generated by a beam search, with early stopping. They normalize the log-likelihood scores of the hypotheses in the beam by their sequence length. Results were reported with beam widths of 1, 10 and 50.\n",
        "\n",
        "## Evaluation\n",
        "Since the correctness of generated expression can be easily verifyed (only it is needed to derivate if the problem was an integrations or susbtitute if it was a differential equation) all hypotheses in the beam are checked, if one of them is correct means that the model has successfully solved the probem.\n",
        "\n",
        "## Results\n",
        "This table show the accuracies obtained, also by comparing with other popular external frameworks:\n",
        "\n",
        "![results](https://drive.google.com/uc?id=1fQ8gyobxLrfqUsNAc-AMquvWoNdR0jLe)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A2aloSzSMbBc"
      },
      "source": [
        "# Implementation\n",
        "After having explained the contents of the paper, I attach here the the code of the official github repository, adding some explanations in each part. Then I have implemented a couple of experiments taking advantage of this code. This code is required to be executed using the gpu.\n",
        "## Setting the enviroment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6P5WhhoVd1A4",
        "outputId": "c216f9ab-e647-496b-d1b3-c502044f8f28"
      },
      "source": [
        "import os\n",
        "if 'google.colab' in str(get_ipython()):\n",
        "    # mount drive (it's required be located in the folder where you have all the needed files)\n",
        "    from google.colab import drive\n",
        "    drive.mount(\"/content/drive\")\n",
        "    \n",
        "# input folder where you have store the needed files\n",
        "local_folder = input(\"Enter here your local folder: \")\n",
        "%cd $local_folder\n",
        "%ls\n",
        "assert os.path.isfile('fwd_bwd.pth')\n",
        "assert os.path.isfile('ode1.pth')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n",
            "Enter here your local folder: drive/MyDrive/UNI/3º/Colab Research Project/\n",
            "/content/drive/MyDrive/UNI/3º/Colab Research Project\n",
            " fwd_bwd.pth   \u001b[0m\u001b[01;34mimg\u001b[0m/   ode1.pth  'Research Project.ipynb'   \u001b[01;34msrc\u001b[0m/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jpZKi31Baikp"
      },
      "source": [
        "# importing the needed libraries\n",
        "import numpy as np\n",
        "import sympy as sp\n",
        "import torch\n",
        "\n",
        "from src.utils import AttrDict\n",
        "from src.envs import build_env\n",
        "from src.model import build_modules\n",
        "\n",
        "from src.utils import to_cuda\n",
        "from src.envs.sympy_utils import simplify"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "trDP83e7l0VO"
      },
      "source": [
        "## Code from SymbolicMathematics repository by facebookresearch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "de-nWxijaikp"
      },
      "source": [
        "### Build environment / Reload model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uaa6VWY-aikq"
      },
      "source": [
        "# setting the parameters\n",
        "params = params = AttrDict({\n",
        "\n",
        "    # environment parameters\n",
        "    'env_name': 'char_sp',\n",
        "    'int_base': 10,\n",
        "    'balanced': False,\n",
        "    'positive': True,\n",
        "    'precision': 10,\n",
        "    'n_variables': 1,\n",
        "    'n_coefficients': 0,\n",
        "    'leaf_probs': '0.75,0,0.25,0',\n",
        "    'max_len': 512,\n",
        "    'max_int': 5,\n",
        "    'max_ops': 15,\n",
        "    'max_ops_G': 15,\n",
        "    'clean_prefix_expr': True,\n",
        "    'rewrite_functions': '',\n",
        "    'tasks': 'prim_fwd',\n",
        "    'operators': 'add:10,sub:3,mul:10,div:5,sqrt:4,pow2:4,pow3:2,pow4:1,pow5:1,ln:4,exp:4,sin:4,cos:4,tan:4,asin:1,acos:1,atan:1,sinh:1,cosh:1,tanh:1,asinh:1,acosh:1,atanh:1',\n",
        "\n",
        "    # model parameters\n",
        "    'cpu': False,\n",
        "    'emb_dim': 1024,\n",
        "    'n_enc_layers': 6,\n",
        "    'n_dec_layers': 6,\n",
        "    'n_heads': 8,\n",
        "    'dropout': 0,\n",
        "    'attention_dropout': 0,\n",
        "    'sinusoidal_embeddings': False,\n",
        "    'share_inout_emb': True,\n",
        "    'reload_model': './fwd_bwd.pth',\n",
        "\n",
        "})"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RGlHJx52aikr"
      },
      "source": [
        "env = build_env(params)\n",
        "x = env.local_dict['x']"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wayzsjxuaikt"
      },
      "source": [
        "#torch.load(map_location=torch.device('cpu'))\n",
        "modules = build_modules(env, params)\n",
        "\n",
        "encoder = modules['encoder']\n",
        "decoder = modules['decoder']"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JI_Dm4dUaiku"
      },
      "source": [
        "### Start from a function F, compute its derivative f = F', which  will be tried to recover F from f"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nMYcKq4yaiku"
      },
      "source": [
        "# here you can modify the integral function the model has to predict, F\n",
        "F_infix = 'x * tan(exp(x)/x)'\n",
        "F_infix = 'x * cos(x**2) * tan(x)'\n",
        "F_infix = 'cos(x**2 * exp(x * cos(x)))'\n",
        "F_infix = 'ln(cos(x + exp(x)) * sin(x**2 + 2) * exp(x) / x)'"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "id": "4CSdb-p-g88p",
        "outputId": "8b4d037f-106e-488d-cd2d-65a65380703e"
      },
      "source": [
        "# F (integral, that the model will try to predict)\n",
        "F = sp.S(F_infix, locals=env.local_dict)\n",
        "F"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/latex": "$\\displaystyle \\log{\\left(\\frac{e^{x} \\sin{\\left(x^{2} + 2 \\right)} \\cos{\\left(x + e^{x} \\right)}}{x} \\right)}$",
            "text/plain": [
              "log(exp(x)*sin(x**2 + 2)*cos(x + exp(x))/x)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        },
        "id": "XfKjRIZmhBFd",
        "outputId": "2d477b0e-5288-4523-c334-2c4472412afc"
      },
      "source": [
        "# f (F', that the model will take as input)\n",
        "f = F.diff(x)\n",
        "f"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/latex": "$\\displaystyle \\frac{x \\left(2 e^{x} \\cos{\\left(x + e^{x} \\right)} \\cos{\\left(x^{2} + 2 \\right)} - \\frac{\\left(e^{x} + 1\\right) e^{x} \\sin{\\left(x + e^{x} \\right)} \\sin{\\left(x^{2} + 2 \\right)}}{x} + \\frac{e^{x} \\sin{\\left(x^{2} + 2 \\right)} \\cos{\\left(x + e^{x} \\right)}}{x} - \\frac{e^{x} \\sin{\\left(x^{2} + 2 \\right)} \\cos{\\left(x + e^{x} \\right)}}{x^{2}}\\right) e^{- x}}{\\sin{\\left(x^{2} + 2 \\right)} \\cos{\\left(x + e^{x} \\right)}}$",
            "text/plain": [
              "x*(2*exp(x)*cos(x + exp(x))*cos(x**2 + 2) - (exp(x) + 1)*exp(x)*sin(x + exp(x))*sin(x**2 + 2)/x + exp(x)*sin(x**2 + 2)*cos(x + exp(x))/x - exp(x)*sin(x**2 + 2)*cos(x + exp(x))/x**2)*exp(-x)/(sin(x**2 + 2)*cos(x + exp(x)))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EEv_cDj1aikv"
      },
      "source": [
        "### Compute prefix representations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xew41Do_hEWt",
        "outputId": "e2583964-fc9d-4fdc-9c7b-5f7b4e46ff20"
      },
      "source": [
        "F_prefix = env.sympy_to_prefix(F)\n",
        "f_prefix = env.sympy_to_prefix(f)\n",
        "print(f\"F prefix: {F_prefix}\")\n",
        "print(f\"f prefix: {f_prefix}\")"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F prefix: ['ln', 'mul', 'pow', 'x', 'INT-', '1', 'mul', 'cos', 'add', 'x', 'exp', 'x', 'mul', 'exp', 'x', 'sin', 'add', 'INT+', '2', 'pow', 'x', 'INT+', '2']\n",
            "f prefix: ['mul', 'x', 'mul', 'pow', 'cos', 'add', 'x', 'exp', 'x', 'INT-', '1', 'mul', 'pow', 'sin', 'add', 'INT+', '2', 'pow', 'x', 'INT+', '2', 'INT-', '1', 'mul', 'add', 'mul', 'INT+', '2', 'mul', 'cos', 'add', 'INT+', '2', 'pow', 'x', 'INT+', '2', 'mul', 'cos', 'add', 'x', 'exp', 'x', 'exp', 'x', 'add', 'mul', 'pow', 'x', 'INT-', '1', 'mul', 'cos', 'add', 'x', 'exp', 'x', 'mul', 'exp', 'x', 'sin', 'add', 'INT+', '2', 'pow', 'x', 'INT+', '2', 'add', 'mul', 'INT-', '1', 'mul', 'pow', 'x', 'INT-', '2', 'mul', 'cos', 'add', 'x', 'exp', 'x', 'mul', 'exp', 'x', 'sin', 'add', 'INT+', '2', 'pow', 'x', 'INT+', '2', 'mul', 'INT-', '1', 'mul', 'pow', 'x', 'INT-', '1', 'mul', 'add', 'INT+', '1', 'exp', 'x', 'mul', 'exp', 'x', 'mul', 'sin', 'add', 'INT+', '2', 'pow', 'x', 'INT+', '2', 'sin', 'add', 'x', 'exp', 'x', 'exp', 'mul', 'INT-', '1', 'x']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dIsfkSjKaikw"
      },
      "source": [
        "### Encode input"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aZQwjnBohH9i"
      },
      "source": [
        "x1_prefix = env.clean_prefix(['sub', 'derivative', 'f', 'x', 'x'] + f_prefix)\n",
        "# note that the expressions are converted to ids to improve the process\n",
        "x1 = torch.LongTensor(\n",
        "    [env.eos_index] +\n",
        "    [env.word2id[w] for w in x1_prefix] +\n",
        "    [env.eos_index]\n",
        ").view(-1, 1)\n",
        "len1 = torch.LongTensor([len(x1)])\n",
        "x1, len1 = to_cuda(x1, len1)\n",
        "\n",
        "with torch.no_grad():\n",
        "    encoded = encoder('fwd', x=x1, lengths=len1, causal=False).transpose(0, 1)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tT6m_q9Qaikw"
      },
      "source": [
        "### Decode with beam search"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZveMuvwRhMzf"
      },
      "source": [
        "beam_size = 10\n",
        "with torch.no_grad():\n",
        "    _, _, beam = decoder.generate_beam(encoded, len1, beam_size=beam_size, length_penalty=1.0, early_stopping=1, max_len=200)\n",
        "    assert len(beam) == 1\n",
        "hypotheses = beam[0].hyp\n",
        "assert len(hypotheses) == beam_size"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CdW0e2x6aikx"
      },
      "source": [
        "### Print results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 524
        },
        "id": "c-x2hxaThZUJ",
        "outputId": "de5b1330-c347-4834-fd51-1139f879ca55"
      },
      "source": [
        "print(\"Input function f:\\n\")\n",
        "display(f)\n",
        "print(\"\\nReference function F:\\n\")\n",
        "display(F)\n",
        "print(\"\")\n",
        "\n",
        "firstCorrect = None\n",
        "for score, sent in sorted(hypotheses, key=lambda x: x[0], reverse=True):\n",
        "\n",
        "    # parse decoded hypothesis\n",
        "    ids = sent[1:].tolist()                  # decoded token IDs\n",
        "    tok = [env.id2word[wid] for wid in ids]  # convert to prefix\n",
        "\n",
        "    try:\n",
        "        hyp = env.prefix_to_infix(tok)       # convert to infix\n",
        "        hyp = env.infix_to_sympy(hyp)        # convert to SymPy\n",
        "\n",
        "        # check whether we recover f if we differentiate the hypothesis\n",
        "        # note that sometimes, SymPy fails to show that hyp' - f == 0, and the result is considered as invalid, although it may be correct\n",
        "        res = \"OK\" if simplify(hyp.diff(x) - f, seconds=1) == 0 else \"NO\"\n",
        "        if (firstCorrect==None and res==\"OK\"):\n",
        "            firstCorrect = hyp\n",
        "\n",
        "    except:\n",
        "        res = \"INVALID PREFIX EXPRESSION\"\n",
        "        hyp = tok\n",
        "\n",
        "    # print result\n",
        "    print(\"%.5f  %s  %s\" % (score, res, hyp))\n",
        "    \n",
        "print(\"\\nThe well-predicted expression with the highest score is:\\n\")\n",
        "display(firstCorrect)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input function f:\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/latex": "$\\displaystyle \\frac{x \\left(2 e^{x} \\cos{\\left(x + e^{x} \\right)} \\cos{\\left(x^{2} + 2 \\right)} - \\frac{\\left(e^{x} + 1\\right) e^{x} \\sin{\\left(x + e^{x} \\right)} \\sin{\\left(x^{2} + 2 \\right)}}{x} + \\frac{e^{x} \\sin{\\left(x^{2} + 2 \\right)} \\cos{\\left(x + e^{x} \\right)}}{x} - \\frac{e^{x} \\sin{\\left(x^{2} + 2 \\right)} \\cos{\\left(x + e^{x} \\right)}}{x^{2}}\\right) e^{- x}}{\\sin{\\left(x^{2} + 2 \\right)} \\cos{\\left(x + e^{x} \\right)}}$",
            "text/plain": [
              "x*(2*exp(x)*cos(x + exp(x))*cos(x**2 + 2) - (exp(x) + 1)*exp(x)*sin(x + exp(x))*sin(x**2 + 2)/x + exp(x)*sin(x**2 + 2)*cos(x + exp(x))/x - exp(x)*sin(x**2 + 2)*cos(x + exp(x))/x**2)*exp(-x)/(sin(x**2 + 2)*cos(x + exp(x)))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Reference function F:\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/latex": "$\\displaystyle \\log{\\left(\\frac{e^{x} \\sin{\\left(x^{2} + 2 \\right)} \\cos{\\left(x + e^{x} \\right)}}{x} \\right)}$",
            "text/plain": [
              "log(exp(x)*sin(x**2 + 2)*cos(x + exp(x))/x)"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "-0.00003  OK  log(exp(x)*sin(x**2 + 2)*cos(x + exp(x))/x)\n",
            "-0.28475  OK  log(exp(x)*sin((x**3 + 2*x)/x)*cos(x + exp(x))/x)\n",
            "-0.28592  OK  log(exp(x)*sin(x*(x + 2/x))*cos(x + exp(x))/x)\n",
            "-0.35794  OK  log(exp(x)*sin(x*(x + 1) - x + 2)*cos(x + exp(x))/x)\n",
            "-0.37952  NO  log(exp(x)*sin(x**2*(x + 2/x))*cos(x + exp(x))/x)\n",
            "-0.38034  NO  log(exp(x)*sin(x**2 + 2)*cos(x + sinh(x) + cosh(x))/x)\n",
            "-0.39518  OK  atan(tan(log(exp(x)*sin(x**2 + 2)*cos(x + exp(x))/x)))\n",
            "-0.39689  OK  log(exp(x)*sin(x*(x - 1) + x + 2)*cos(x + exp(x))/x)\n",
            "-0.43203  NO  log(exp(x)*sin((x**2 + 2)**2)*cos(x + exp(x))/x)\n",
            "-0.44538  NO  log(exp(x)*sin(x**2 + 2*x)*cos(x + exp(x))/x)\n",
            "\n",
            "The well-predicted expression with the highest score is:\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/latex": "$\\displaystyle \\log{\\left(\\frac{e^{x} \\sin{\\left(x^{2} + 2 \\right)} \\cos{\\left(x + e^{x} \\right)}}{x} \\right)}$",
            "text/plain": [
              "log(exp(x)*sin(x**2 + 2)*cos(x + exp(x))/x)"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e-AlK-DoTSDG"
      },
      "source": [
        "## Own experiments\n",
        "\n",
        "### Exploiding the equivalent solution property\n",
        "\n",
        "It is also mentioned in the paper that this model has the interesting property of generating solutions that are exactly equivalent, but written in different way (it can be observed in the results of the previous code examples). So, without having been trained to do so, the model is able to recover equivalent expressions. Considering this, I have implemented a fragment of code in which it is requested an input expresion and returns you the equivalent expressions in the SimPy format.\n",
        "\n",
        "Here I attach the code:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NZa-ky7cZp9x",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 461
        },
        "outputId": "584744f3-30b7-483d-df99-af18649fe680"
      },
      "source": [
        "# F (integral, that the model will try to predict)\n",
        "F = input(\"You want to find equivalent expression from: \")\n",
        "# I executed this cell for F=x**2+2*x+a8 (where a8 is a constant)\n",
        "F = sp.S(F, locals=env.local_dict)\n",
        "F_prefix = env.sympy_to_prefix(F)\n",
        "print(\"Your input expression (that is the problem solution) is:\")\n",
        "display(F)\n",
        "# f (F', that the model will take as input)\n",
        "f = F.diff(x)\n",
        "print(\"Your derivative of your problem (that is the problem) is:\")\n",
        "display(f)\n",
        "print(\"\")\n",
        "\n",
        "F_prefix = env.sympy_to_prefix(F)\n",
        "f_prefix = env.sympy_to_prefix(f)\n",
        "x1_prefix = env.clean_prefix(['sub', 'derivative', 'f', 'x', 'x'] + f_prefix)\n",
        "x1 = torch.LongTensor(\n",
        "    [env.eos_index] +\n",
        "    [env.word2id[w] for w in x1_prefix] +\n",
        "    [env.eos_index]\n",
        ").view(-1, 1)\n",
        "len1 = torch.LongTensor([len(x1)])\n",
        "x1, len1 = to_cuda(x1, len1)\n",
        "\n",
        "beam_size = 50\n",
        "with torch.no_grad():\n",
        "    encoded = encoder('fwd', x=x1, lengths=len1, causal=False).transpose(0, 1)\n",
        "    _, _, beam = decoder.generate_beam(encoded, len1, beam_size=beam_size, length_penalty=1.0, early_stopping=1, max_len=200)\n",
        "    assert len(beam) == 1\n",
        "hypotheses = beam[0].hyp\n",
        "assert len(hypotheses) == beam_size\n",
        "\n",
        "print(\"Printing equivalent expressions...\")\n",
        "for score, sent in sorted(hypotheses, key=lambda x: x[0], reverse=True):\n",
        "    # parse decoded hypothesis\n",
        "    ids = sent[1:].tolist()                  # decoded token IDs\n",
        "    tok = [env.id2word[wid] for wid in ids]  # convert to prefix\n",
        "    try:\n",
        "        hyp = env.prefix_to_infix(tok)       # convert to infix\n",
        "        hyp = env.infix_to_sympy(hyp)        # convert to SymPy\n",
        "        # check whether we recover f if we differentiate the hypothesis\n",
        "        if simplify(hyp.diff(x) - f, seconds=1)==0:\n",
        "          print(\"\")\n",
        "          display(hyp)\n",
        "          if (input(\"\\n\\nDo you want to display another equivalent expression (Y/y)? \").lower()!=\"y\"):\n",
        "            break\n",
        "\n",
        "    except:\n",
        "        res = \"INVALID PREFIX EXPRESSION\"\n",
        "        hyp = tok"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "You want to find equivalent expression from: x**2+2*x+a8\n",
            "Your input expression (that is the problem solution) is:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/latex": "$\\displaystyle a_{8} + x^{2} + 2 x$",
            "text/plain": [
              "a8 + x**2 + 2*x"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Your derivative of your problem (that is the problem) is:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/latex": "$\\displaystyle 2 x + 2$",
            "text/plain": [
              "2*x + 2"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Printing equivalent expressions...\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/latex": "$\\displaystyle x^{2} + \\frac{2 x^{2} + x}{x}$",
            "text/plain": [
              "x**2 + (2*x**2 + x)/x"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "Do you want to display another equivalent expression (Y/y)? Y\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/latex": "$\\displaystyle - x^{2} + x \\left(2 x + 2\\right)$",
            "text/plain": [
              "-x**2 + x*(2*x + 2)"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "Do you want to display another equivalent expression (Y/y)? y\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/latex": "$\\displaystyle \\frac{x^{3} + 2 x^{2} + x}{x}$",
            "text/plain": [
              "(x**3 + 2*x**2 + x)/x"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "Do you want to display another equivalent expression (Y/y)? n\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SBDQt9-gPwAO"
      },
      "source": [
        "\n",
        "### Playing with ODEs\n",
        "The paper shows, in the results of the experiments performed by the authors, that the accuracy obtained by the model is greater than the one obtained with the mathematica software. I had curiosity about this fact and ask to a friend, who has finished the degree of physics, for differential equations of first order that are not to solvable by mathematica (they are declared below). Will be this model capable to find their solutions?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Y6b_VHvWwyW"
      },
      "source": [
        "# changing some parameters\n",
        "params[\"tasks\"] = 'ode1'\n",
        "params[\"reload_model\"] = './ode1.pth'\n",
        "params[\"n_variables\"] = 2\n",
        "\n",
        "# building the enviroment\n",
        "env = build_env(params)\n",
        "x = env.local_dict['x']\n",
        "y = env.local_dict['y']\n",
        "f = env.local_dict['f']\n",
        "a8 = env.local_dict['a8']\n",
        "\n",
        "modules = build_modules(env, params)\n",
        "encoder = modules['encoder']\n",
        "decoder = modules['decoder']"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OE0MqXtsDOir"
      },
      "source": [
        "# this is an easy ODE problem just for testing\n",
        "ode = x*sp.Derivative(f(x))-f(x)+x\n",
        "# these are all the equations not solvable by mathematica (you can choose anyone):\n",
        "ode = (-x*sp.cos(x)*f(x)+sp.sec(x))/(sp.cos(x)-2*x*sp.sin(x))-sp.Derivative(f(x))\n",
        "ode = (1-x*sp.cos(x)*f(x))/(sp.cos(x)-x*sp.sin(x))-sp.Derivative(f(x))\n",
        "ode = (sp.exp(-x)-x*f(x))/(1+x)-sp.Derivative(f(x))\n",
        "ode = (sp.sqrt((1+x**2))-(1+x**2)*f(x))/x-sp.Derivative(f(x))\n",
        "ode = sp.Derivative(f(x))+x*f(x)+a8\n",
        "ode = 1-x*f(x)-sp.Derivative(f(x))"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zjyIHZHkgpxq"
      },
      "source": [
        "ode_prefix = env.sympy_to_prefix(ode)\n",
        "x1_prefix = env.clean_prefix(ode_prefix)\n",
        "\n",
        "x1 = torch.LongTensor(\n",
        "    [env.eos_index] +\n",
        "    [env.word2id[w] for w in x1_prefix] +\n",
        "    [env.eos_index]\n",
        ").view(-1, 1)\n",
        "len1 = torch.LongTensor([len(x1)])\n",
        "x1, len1 = to_cuda(x1, len1)\n",
        "\n",
        "with torch.no_grad():\n",
        "    encoded = encoder('fwd', x=x1, lengths=len1, causal=False).transpose(0, 1)\n",
        "\n",
        "beam_size = 10\n",
        "with torch.no_grad():\n",
        "    _, _, beam = decoder.generate_beam(encoded, len1, beam_size=beam_size, length_penalty=1.0, early_stopping=1, max_len=200)\n",
        "    assert len(beam) == 1\n",
        "hypotheses = beam[0].hyp\n",
        "assert len(hypotheses) == beam_size"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 358
        },
        "id": "nuQC6z-M3H0w",
        "outputId": "47d106f5-5e70-42f8-a912-bf8018f91460"
      },
      "source": [
        "print(\"Input function ODE:\\n\")\n",
        "display(ode)\n",
        "print(\"\")\n",
        "\n",
        "firstCorrect = None\n",
        "for score, sent in sorted(hypotheses, key=lambda x: x[0], reverse=True):\n",
        "\n",
        "    # parse decoded hypothesis\n",
        "    ids = sent[1:].tolist()                  # decoded token IDs\n",
        "    tok = [env.id2word[wid] for wid in ids]  # convert to prefix\n",
        "\n",
        "    try:\n",
        "        hyp = env.prefix_to_infix(tok)       # convert to infix\n",
        "        hyp = env.infix_to_sympy(hyp)        # convert to SymPy\n",
        "\n",
        "        # check whether we recover f if we differentiate the hypothesis\n",
        "        res = \"OK\" if simplify(simplify(ode.subs(f(x), hyp), seconds=1), seconds=1) == 0 else \"NO\"\n",
        "        if (firstCorrect==None and res==\"OK\"):\n",
        "            firstCorrect = hyp\n",
        "\n",
        "    except:\n",
        "        res = \"INVALID PREFIX EXPRESSION\"\n",
        "        hyp = tok\n",
        "\n",
        "    # print result\n",
        "    print(\"%.5f  %s  %s\" % (score, res, hyp))\n",
        "        \n",
        "print(\"\\nThe well-predicted expression with the highest score is:\\n\")\n",
        "display(firstCorrect)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input function ODE:\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/latex": "$\\displaystyle - x f{\\left(x \\right)} - \\frac{d}{d x} f{\\left(x \\right)} + 1$",
            "text/plain": [
              "-x*f(x) - Derivative(f(x), x) + 1"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "-0.04524  NO  a8*exp(-x**2/2) + 1\n",
            "-0.08929  NO  exp(a8 - x**2/2) + 1\n",
            "-0.13985  NO  (a8*x*exp(-x**2/2) + x)/x\n",
            "-0.15581  NO  1 - exp(a8 - x**2/2)\n",
            "-0.19425  NO  a8*exp(-x*(x + 1)/2 + x/2) + 1\n",
            "-0.19915  NO  exp(x*(a8/x - x)/2) + 1\n",
            "-0.20747  NO  (x*exp(a8 - x**2/2) + x)/x\n",
            "-0.23801  NO  a8*exp(-(x**3 + x)/(2*x)) + 1\n",
            "-0.27655  NO  a8*exp(-(x**3 + 2*x)/(2*x)) + 1\n",
            "-0.28619  NO  a8*exp(-x**2/2 + pi/4) + 1\n",
            "\n",
            "The well-predicted expression with the highest score is:\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "None"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YE-8kIXRZaGC"
      },
      "source": [
        "We can observe that the model is not able to solve the problem.\n",
        "\n",
        "Unfortunately, the result is the same for the rest of the expressions that my friend told me. My final conclusion is that the model is accurate (to the point to be can be comparable with mathematica), but only in a range of problems that, although really wide, is limited considering all the possible ones that could be considered in real life, in contrast to other mathematical frameworks that are of general pourpuse."
      ]
    }
  ]
}