Exercices.ipynb

{
 "metadata": {
  "name": "Exercices",
  "signature": "sha256:77d5115ebf6c96f00122775403f827b913b9144aa466e24779ec3e5729e36d94"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The exercices work that way:\n",
      "\n",
      "1) You have a cell with TODOs that raise errors with a description of what is needed. Do that.\n",
      "2) Then run the cell(ctrl-enter) to execute it.\n",
      "3) It should print \"Success\" at the end (there is validation code in the cell). If not, try again.\n",
      "4) If you want to see the solution, execute the cell that start with \"%load\" after the exercice.\n",
      "\n",
      "First, there are Theano exercices, then 1 scan specific exercics, then some exercices related to the LSTM example."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Exercices 1\n",
      "# This exercice ask you to create Theano variable and do some\n",
      "# computation on them.\n",
      "import numpy as np\n",
      "from theano import function\n",
      "raise NotImplementedError(\"TODO: add any other imports you need\")\n",
      "\n",
      "\n",
      "def make_scalar():\n",
      "    \"\"\"\n",
      "    Returns a new Theano scalar.\n",
      "    \"\"\"\n",
      "\n",
      "    raise NotImplementedError(\"TODO: implement this function.\")\n",
      "\n",
      "\n",
      "def log(x):\n",
      "    \"\"\"\n",
      "    Returns the logarithm of a Theano scalar x.\n",
      "    \"\"\"\n",
      "\n",
      "    raise NotImplementedError(\"TODO: implement this function.\")\n",
      "\n",
      "\n",
      "def add(x, y):\n",
      "    \"\"\"\n",
      "    Adds two theano scalars together and returns the result.\n",
      "    \"\"\"\n",
      "\n",
      "    raise NotImplementedError(\"TODO: implement this function.\")\n",
      "    \n",
      "# The following code use your code and test it.\n",
      "a = make_scalar()\n",
      "b = make_scalar()\n",
      "c = log(b)\n",
      "d = add(a, c)\n",
      "f = function([a, b], d)\n",
      "a = np.cast[a.dtype](1.)\n",
      "b = np.cast[b.dtype](2.)\n",
      "actual = f(a, b)\n",
      "expected = 1. + np.log(2.)\n",
      "assert np.allclose(actual, expected)\n",
      "print \"SUCCESS!\"\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load 01_scalar_soln.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Exercices 2\n",
      "# This exercices ask you to make Theano variable, elemwise\n",
      "# multiplication and matrix/vector dot product.\n",
      "import numpy as np\n",
      "from theano import function\n",
      "raise NotImplementedError(\"TODO: add any other imports you need\")\n",
      "\n",
      "\n",
      "def make_vector():\n",
      "    \"\"\"\n",
      "    Returns a new Theano vector.\n",
      "    \"\"\"\n",
      "\n",
      "    raise NotImplementedError(\"TODO: implement this function.\")\n",
      "\n",
      "\n",
      "def make_matrix():\n",
      "    \"\"\"\n",
      "    Returns a new Theano matrix.\n",
      "    \"\"\"\n",
      "\n",
      "    raise NotImplementedError(\"TODO: implement this function.\")\n",
      "\n",
      "def elemwise_mul(a, b):\n",
      "    \"\"\"\n",
      "    a: A theano matrix\n",
      "    b: A theano matrix\n",
      "    Returns the elementwise product of a and b\n",
      "    \"\"\"\n",
      "\n",
      "    raise NotImplementedError(\"TODO: implement this function.\")\n",
      "\n",
      "\n",
      "def matrix_vector_mul(a, b):\n",
      "    \"\"\"\n",
      "    a: A theano matrix\n",
      "    b: A theano vector\n",
      "    Returns the matrix-vector product of a and b\n",
      "    \"\"\"\n",
      "\n",
      "    raise NotImplementedError(\"TODO: implement this function.\")\n",
      "\n",
      "# The following code use your code and test it.\n",
      "a = make_vector()\n",
      "b = make_vector()\n",
      "c = elemwise_mul(a, b)\n",
      "d = make_matrix()\n",
      "e = matrix_vector_mul(d, c)\n",
      "\n",
      "f = function([a, b, d], e)\n",
      "\n",
      "rng = np.random.RandomState([1, 2, 3])\n",
      "a_value = rng.randn(5).astype(a.dtype)\n",
      "b_value = rng.rand(5).astype(b.dtype)\n",
      "c_value = a_value * b_value\n",
      "d_value = rng.randn(5, 5).astype(d.dtype)\n",
      "expected = np.dot(d_value, c_value)\n",
      "\n",
      "actual = f(a_value, b_value, d_value)\n",
      "assert np.allclose(actual, expected)\n",
      "print \"SUCCESS!\"\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load 02_vector_mat_soln.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Exercices 3\n",
      "# This exercices ask you to create Theano tensor variable, do\n",
      "# broadcastable addition and to compute the max over part of a tensor.\n",
      "import numpy as np\n",
      "from theano import function\n",
      "raise NotImplementedError(\"TODO: add any other imports you need\")\n",
      "\n",
      "\n",
      "def make_tensor(dim):\n",
      "    \"\"\"\n",
      "    Returns a new Theano tensor with no broadcastable dimensions.\n",
      "    dim: the total number of dimensions of the tensor.\n",
      "    (You can use any dtype you like)\n",
      "    \"\"\"\n",
      "\n",
      "    raise NotImplementedError(\"TODO: implement this function.\")\n",
      "\n",
      "\n",
      "def broadcasted_add(a, b):\n",
      "    \"\"\"\n",
      "    a: a 3D theano tensor\n",
      "    b: a 4D theano tensor\n",
      "    Returns c, a 4D theano tensor, where\n",
      "\n",
      "    c[i, j, k, l] = a[l, k, i] + b[i, j, k, l]\n",
      "\n",
      "    for all i, j, k, l\n",
      "    \"\"\"\n",
      "\n",
      "    raise NotImplementedError(\"TODO: implement this function.\")\n",
      "\n",
      "def partial_max(a):\n",
      "    \"\"\"\n",
      "    a: a 4D theano tensor\n",
      "\n",
      "    Returns b, a theano matrix, where\n",
      "\n",
      "    b[i, j] = max_{k,l} a[i, k, l, j]\n",
      "\n",
      "    for all i, j\n",
      "    \"\"\"\n",
      "\n",
      "    raise NotImplementedError(\"TODO: implement this function.\")\n",
      "\n",
      "# The following code use your code and test it.\n",
      "a = make_tensor(3)\n",
      "b = make_tensor(4)\n",
      "c = broadcasted_add(a, b)\n",
      "d = partial_max(c)\n",
      "\n",
      "f = function([a, b], d)\n",
      "\n",
      "rng = np.random.RandomState([1, 2, 3])\n",
      "a_value = rng.randn(2, 2, 2).astype(a.dtype)\n",
      "b_value = rng.rand(2, 2, 2, 2).astype(b.dtype)\n",
      "c_value = np.transpose(a_value, (2, 1, 0))[:, None, :, :] + b_value\n",
      "expected = c_value.max(axis=1).max(axis=1)\n",
      "\n",
      "actual = f(a_value, b_value)\n",
      "\n",
      "assert np.allclose(actual, expected), (actual, expected)\n",
      "print \"SUCCESS!\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load 03_tensor_soln.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Exercices 4\n",
      "# This exercice ask you to compile a Theano functiont and call it to\n",
      "# execute \"x + y\".\n",
      "from theano import tensor as T\n",
      "raise NotImplementedError(\"TODO: add any other imports you need\")\n",
      "\n",
      "\n",
      "def evaluate(x, y, expr, x_value, y_value):\n",
      "    \"\"\"\n",
      "    x: A theano variable\n",
      "    y: A theano variable\n",
      "    expr: A theano expression involving x and y\n",
      "    x_value: A numpy value\n",
      "    y_value: A numpy value\n",
      "\n",
      "    Returns the value of expr when x_value is substituted for x\n",
      "    and y_value is substituted for y\n",
      "    \"\"\"\n",
      "\n",
      "    raise NotImplementedError(\"TODO: implement this function.\")\n",
      "\n",
      "\n",
      "# The following code use your code and test it.\n",
      "x = T.iscalar()\n",
      "y = T.iscalar()\n",
      "z = x + y\n",
      "assert evaluate(x, y, z, 1, 2) == 3\n",
      "print \"SUCCESS!\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load 04_function_soln.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Exercices 5\n",
      "# This exercice make you use shared variable. You must create them and\n",
      "# update them by swapping 2 shared variables values.\n",
      "import numpy as np\n",
      "raise NotImplementedError(\"TODO: add any other imports you need\")\n",
      "\n",
      "\n",
      "def make_shared(shape):\n",
      "    \"\"\"\n",
      "    Returns a theano shared variable containing a tensor of the specified\n",
      "    shape.\n",
      "    You can use any value you want.\n",
      "    \"\"\"\n",
      "    raise NotImplementedError(\"TODO: implement the function\")\n",
      "\n",
      "\n",
      "def exchange_shared(a, b):\n",
      "    \"\"\"\n",
      "    a: a theano shared variable\n",
      "    b: a theano shared variable\n",
      "    Uses get_value and set_value to swap the values stored in a and b\n",
      "    \"\"\"\n",
      "    raise NotImplementedError(\"TODO: implement the function\")\n",
      "\n",
      "\n",
      "def make_exchange_func(a, b):\n",
      "    \"\"\"\n",
      "    a: a theano shared variable\n",
      "    b: a theano shared variable\n",
      "    Returns f\n",
      "    where f is a theano function, that, when called, swaps the\n",
      "    values in a and b\n",
      "    f should not return anything\n",
      "    \"\"\"\n",
      "    raise NotImplementedError(\"TODO: implement the function\")\n",
      "\n",
      "\n",
      "# The following code use your code and test it.\n",
      "a = make_shared((5, 4, 3))\n",
      "assert a.get_value().shape == (5, 4, 3)\n",
      "b = make_shared((5, 4, 3))\n",
      "assert a.get_value().shape == (5, 4, 3)\n",
      "a.set_value(np.zeros((5, 4, 3), dtype=a.dtype))\n",
      "b.set_value(np.ones((5, 4, 3), dtype=b.dtype))\n",
      "exchange_shared(a, b)\n",
      "assert np.all(a.get_value() == 1.)\n",
      "assert np.all(b.get_value() == 0.)\n",
      "f = make_exchange_func(a, b)\n",
      "rval = f()\n",
      "assert isinstance(rval, list)\n",
      "assert len(rval) == 0\n",
      "assert np.all(a.get_value() == 0.)\n",
      "assert np.all(b.get_value() == 1.)\n",
      "\n",
      "print \"SUCCESS!\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load 05_shared_soln.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Exercices 6\n",
      "# This exercice make use Theano symbolic grad\n",
      "from theano import tensor as T\n",
      "\n",
      "\n",
      "def grad_sum(x, y, z):\n",
      "    \"\"\"\n",
      "    x: A theano variable\n",
      "    y: A theano variable\n",
      "    z: A theano expression involving x and y\n",
      "\n",
      "    Returns dz / dx + dz / dy\n",
      "    \"\"\"\n",
      "    raise NotImplementedError(\"TODO: implement this function.\")\n",
      "\n",
      "\n",
      "# The following code use your code and test it.\n",
      "x = T.scalar()\n",
      "y = T.scalar()\n",
      "z = x + y\n",
      "s = grad_sum(x, y, z)\n",
      "assert s.eval({x: 0, y: 0}) == 2\n",
      "print \"SUCCESS!\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load 06_grad_soln.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Exercice 7 #TODO: talk about mode=FAST_COMPILE\n",
      "# This code have a bug. Run this cell to see it.\n",
      "# Use Theano flag (easy in shell, harder in ipython) or extra parameter to a function \n",
      "# to find the cause and fix it.\n",
      "# Do not find the bug by inspecting the code. This is to help you find the bug\n",
      "# in more complicated case when code inspection isn't working well.\n",
      "\n",
      "import numpy as np\n",
      "from theano import function\n",
      "from theano import tensor as T\n",
      "a = T.vector()\n",
      "b = T.log(a)\n",
      "c = T.nnet.sigmoid(b)\n",
      "d = T.sqrt(c)\n",
      "e = T.concatenate((d, c), axis=0)\n",
      "f = b * c * d\n",
      "g = e + f\n",
      "h = g / c\n",
      "fn = function([a], h)\n",
      "fn(np.ones((3,)).astype(a.dtype))\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load 07_mode.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Exercice 8\n",
      "# This exercice is different. The initial version work.\n",
      "# So you must modify it as described bellow and it should still give the same output.\n",
      "\n",
      "# Modify and execute the polynomial example to have the reduction(the sum() call) done by scan.\n",
      "import numpy\n",
      "import theano\n",
      "import theano.tensor as T\n",
      "theano.config.warn.subtensor_merge_bug = False\n",
      "\n",
      "coefficients = theano.tensor.vector(\"coefficients\")\n",
      "x = T.scalar(\"x\")\n",
      "max_coefficients_supported = 10000\n",
      "\n",
      "# Generate the components of the polynomial\n",
      "full_range=theano.tensor.arange(max_coefficients_supported)\n",
      "components, updates = theano.scan(fn=lambda coeff, power, free_var:\n",
      "                                   coeff * (free_var ** power),\n",
      "                                outputs_info=None,\n",
      "                                sequences=[coefficients, full_range],\n",
      "                                non_sequences=x)\n",
      "\n",
      "polynomial = components.sum()\n",
      "calculate_polynomial = theano.function(inputs=[coefficients, x],\n",
      "                                     outputs=polynomial)\n",
      "\n",
      "test_coeff = numpy.asarray([1, 0, 2], dtype=numpy.float32)\n",
      "print calculate_polynomial(test_coeff, 3)\n",
      "# 19.0"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load 08_scan_polynomial_soln.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "LSTM Exercice"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "1) Modif LSTM: Reverse the input sequence and try it like that:\n",
      "   Sutskever-NIPS2014 (No change to Theano code, but useful to better understand how to make 2)\n",
      "2) Modif LSTM: Add to have 2 LSTM layers. The new one take\n",
      "   the input in the reverse order. Then you concatenate the mean\n",
      "   of the outputs of both LSTM to the logistic regression.\n",
      "3) Modif LSTM: Add the V_o parameter and use it. (No solutions provided)\n",
      "    \n",
      "Note. 2) need more epoch before we start to see that it learn something. With max_epochs=16, we start to see it for all version.\n",
      "\n",
      "You can load the original example code in the next cell.\n",
      "Run it once. It will charge the data.\n",
      "At the end of that code, there is in comment example how to run it for a short time (~10m on my laptop, core i5).\n",
      "During that time, we see that is start to learn, but I do not let it go too long for this tutorial."
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load lstm.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load lstm_reverse.diff"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load lstm_double.diff"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    }
   ],
   "metadata": {}
  }
 ]
}