Class_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Statistics with Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Koen Plevoets\n",
    "Last modified: 2020-12-09\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Class 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter 4: Numerical computations with NumPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 4.1: Creating NumPy arrays\n",
    "- 4.2: Computing with NumPy arrays\n",
    "- 4.3: Indexing with NumPy arrays\n",
    "- 4.4: NumPy submodules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python has the module `math` for various mathematical functions and `statistics` for basic functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These functions work on a sequence of numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xx = [6, 9, 5.5, 2, 2.5, 6, 3]\n",
    "statistics.mean(xx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "statistics.median(xx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "statistics.mode(xx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "statistics.variance(xx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "statistics.stdev(xx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the [Python Standard Library](https://docs.python.org/3/library/statistics.html) for other statistical functions. The module `random` has functions for generating random numbers - see the [Python Standard Library](https://docs.python.org/3/library/random.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The proper modules for **advanced calculations** in Python are `NumPy`, `SciPy`, `pandas` etc. These are built **on top of** each other as follows:\n",
    "\n",
    "    +---------------+-------------------------+--------------------+\n",
    "    |               |                         |                    |\n",
    "    |  StatsModels  |         seaborn         |                    |\n",
    "    |               |                         |                    |\n",
    "    +---------------+----+--------------------+       SciPy        |\n",
    "    |                    |                    |                    |\n",
    "    |        pandas      |     Matplotlib     |                    |\n",
    "    |                    |                    |                    |\n",
    "    +--------------------+--------------------+--------------------+\n",
    "    |                                                              |\n",
    "    |                            NumPy                             |\n",
    "    |                                                              |\n",
    "    +--------------------------------------------------------------+\n",
    "    |                                                              |\n",
    "    |                            Python                            |\n",
    "    |                                                              |\n",
    "    +--------------------------------------------------------------+\n",
    "\n",
    "They are **not** part of the **core** Python installation (downloadable from www.python.org), so you have to download them. However, they are part of the so-called \"**Anaconda distribution**\" (downloadable from https://anaconda.org)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Creating NumPy arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the most basic advanced module is **NumPy**. It is conventional in the Python community to **abbreviate** NumPy in your scripts to `np`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason why NumPy is so popular is that it contains data types for **vectors** and **matrices**. Both are subtypes of the general type `array` which can be made with the function `array()`. It takes a sequence as an argument (typically a list or a tuple):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec = np.array( [5, 6, 7, 8, 9] )\n",
    "vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat = np.array( [ [5, 6, 7, 8, 9], [3, 2, 1, 0, -1], [4, 5, 4, 5, 4] ] )\n",
    "mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mul = np.array( [\n",
    "                    [ [ 1,  2,  4,  5], [ 6,  7,  8,  9], [10, 11, 12, 13] ],\n",
    "                    [ [14, 15, 16, 17], [18, 19, 20, 21], [22, 23, 24, 25] ]\n",
    "                ] )\n",
    "mul"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last example makes clear that matrices are created on the basis of a **list of lists**. NumPy always combines the sublists as **rows** into the matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to create arrays with other data types than numbers, but NumPy is meant for numeric computations (although **logical** arrays can be useful for **indexing**, see later):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.array( [ [\"a\", \"b\"] , [\"x\", \"y\"] ] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.array( [True, False, True] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **class** of a NumPy array is `ndarray`. (This stands for \"**n-dimensional array**\" and is also meant to distinguish NumPy arrays from the built-in data type `array`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(mul)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any NumPy array has general **attributes** such as `.ndim`, `.shape` and `.size` which reflect its structure. (There are also the attributes `.data`, `.dtype` and `.itemsize` but these are not often needed.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The attribute `.ndim` gives the **number of dimensions**, which are also called \"**axes**\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec.ndim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat.ndim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mul.ndim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The attribute `.shape` gives more information about the **number of components** in each dimension/axis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mul.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The attribute `.size` gives the total **number of elements** in an array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mul.size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy also contains some **functions** for **generating whole arrays**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, the function `arange()` generates a (zero-based) **range** as an **array**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(5)    # == array(range(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(2, 9)    # == array(range(2,9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(2, 9, 3)    # == array(range(2,9,3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `linspace()` generates a range of **linearly spaced numbers**. The start value is the first argument, the stop value the second and the number of elements as the third argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linspace(2, 9, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linspace(10, 0, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `zeros()` generates an **array of zeros** with the shape of the array specified as the argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.zeros( (2, 3) )    # Matrix of zeros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.zeros(4)    # Vector of zeros"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `ones()` does the same for number **one**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ones( (2, 3) )    # Matrix of ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ones(4)    # Vector of ones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `empty()` creates an **empty array**, although the elements are replaced by a random number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.empty( (2, 3) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.empty(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy has **two** functions for generating **repetitions**! The function `repeat()` repeats **each element** in an array by the amount specified as the second argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.repeat(5, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.repeat(vec, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also specify a **sequence of repeats** which will **match** the corresponding **values** in the array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.repeat(vec, [5, 4, 3, 2, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For **matrices**, `repeat()` creates a **flattened array** by default:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.repeat(mat, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, `repeat()` has an argument `axis` with which you can specify **which dimension** to repeat:\n",
    "\n",
    "- `axis = 0` means: repeat each **row** (i.e. repeat the array **vertically**).\n",
    "- `axis = 1` means: repeat each **column** (i.e. repeat the array **horizontally**).\n",
    "- `axis = 2` (for multi-dimensional arrays)\n",
    "- Etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.repeat(mat, 3, axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.repeat(mat, 3, axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, **sequences of repeats** are also possible if properly combined with the axis argument. Essentially, the number of repeats needs to **match** the number of **rows** if `axis = 0` and **columns** if `axis = 1`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.repeat(mat, [1, 2, 3], axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.repeat(mat, [5, 4, 3, 2, 1], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second function for repetition in NumPy is `tile()` which repeats **whole arrays**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.tile(vec, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.tile(mat, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the function `reshape()` you can **convert** a vector into a matrix. You specify (the tuple of) the number of rows and columns as the second argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat2 = np.reshape(np.arange(1, 9), (4, 2))\n",
    "mat2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat2.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the shape attribute belongs to the `ndarray` class, you can also always work with **assignment**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat2.shape = (2, 4)\n",
    "mat2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The opposite function is `ravel()` which \"**flattens**\" an array to a vector. By default, the order of the elements corresponds to the nesting of the sub-arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ravel(mat2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `ndarray` class also has **methods** for these last two functions. However, the equivalent method for `ravel()` is `.flatten()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat2.reshape(4, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat2.flatten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other methods for the `ndarray` class are `.view()` and (again) `.copy()`. This is related to the fact that **assignment** of one object to another does not create a new object but only a **new name**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec2 = vec\n",
    "vec2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec2[4] = 10\n",
    "vec2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec is vec2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method `.view()` creates a \"**shallow copy**\" or a \"**view**\" of the orginal object. This is an object with **identical content** but possibly in a **different structure**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec3 = vec.view()\n",
    "vec is vec3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec3.shape = (5, 1)\n",
    "vec3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec3[4] = 9\n",
    "vec3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method `.copy()` creates a \"**deep copy**\". This is an entirely **different object**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec4 = vec.copy()\n",
    "vec4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec4[4] = 16\n",
    "vec4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec is vec4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy arrays can also be **combined** into larger arrays with `hstack()`, `vstack()`, `column_stack()` and `row_stack()`. There is even the general function `dstack()` to combine into multi-way arrays but we will not cover it here. All these functions require a **sequence** of arrays as their argument."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `hstack()` combines arrays **horizontally**, but it works differently for 1D-arrays than for 2D-arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec2 = np.array( [100, 100, 100] )\n",
    "np.hstack( [vec, vec2] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.hstack( [mat, vec2] )    # Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat2 = vec2.reshape(3, 1)\n",
    "mat2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.hstack( [mat, mat2] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `column_stack()` always combines arrays into the **columns** of a larger array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.column_stack( [mat, vec2] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec3 = np.array( [1000, 1000, 1000, 1000, 1000] )\n",
    "np.column_stack( [vec, vec3] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `vstack()` and `row_stack()` identically combine arrays into the **rows** of a larger array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.vstack( [mat, vec3] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.row_stack( [mat, vec3] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.vstack( [vec, vec3] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.row_stack( [vec, vec3] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `column_stack()` and `row_stack()` can also be abbreviated to `c_` and `r_`, respectively. However, these abbrevations are used with **square** brackets instead of round brackets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.c_[mat, vec2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.r_[mat, vec3.reshape(1, 5) ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also the general function `concatenate()` to combine arrays along a certain axis. It has an `axis` argument which specifies how you want to combine the arrays:\n",
    "\n",
    "- `axis = 0` means: combine the arrays **along** the **rows**, i.e. **vertically**.\n",
    "- `axis = 1` means: combine the arrays **along** the **columns**, i.e. **horizontally**.\n",
    "- `axis = 2` (for multi-dimensional arrays)\n",
    "- Etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.concatenate( [mat, mat2], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.concatenate( [vec, vec3] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy also has functions for **splitting** an array into smaller arrays. The function `hsplit()` splits along the **columns**, the function `vsplit()` splits along the **rows**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These functions require an argument which is *either* a **single number** *or* a **sequence** of indices:\n",
    "\n",
    "- A **single number** specifies the number of **equally sized sub-arrays** into which you wish to split.\n",
    "- A **sequence** of indices gives the **indices** at which you wish to split."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case of a **single number**, the resulting sub-arrays need to be of **equal size**, otherwise an error occurs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.hsplit(mat, 3)    # Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat3 = np.hstack( [mat, mat2] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.hsplit(mat3, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result of a split is a **list**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type( np.hsplit(mat3, 3) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.vsplit(mat, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.vsplit(mat, 3)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An `hsplit()` with **indices** can be explained by the following example:\n",
    "\n",
    "```\n",
    " Array:      +--------+-------+----+\n",
    "             |  5   6 | 7   8 | 9  |\n",
    "             |  3   2 | 1   0 |-1  |\n",
    "             |  4   5 | 4   5 | 4  |\n",
    "             +--------+-------+----+\n",
    " Index:      0    1   2   3   4    5\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.hsplit(mat, [2, 4] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.hsplit(mat, [2, 4] )[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the same way, `vsplit()` with **indices** can be explained as follows:\n",
    "\n",
    "```\n",
    " Index:      Array:\n",
    "        0    +---------------------+\n",
    "             |  5   6   7   8   9  |\n",
    "        1    +---------------------+\n",
    "             |  3   2   1   0  -1  |\n",
    "        2    +---------------------+\n",
    "             |  4   5   4   5   4  |\n",
    "        3    +---------------------+\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.vsplit(mat, [1, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.vsplit(mat, [1, 2])[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These functions are in fact special versions of the more general `split()` which has an `axis` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.split(mat, [2, 4], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.split(mat, [1, 2], axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.split(mat3, 3, axis = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Computing with NumPy arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculations on NumPy arrays are **vectorized**. That means that any operations on two arrays comes down to doing the operation **element-wise**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec2 = np.array([2, 4, 6, 8, 10])\n",
    "vec2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec + vec2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec * vec2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat2 = np.array( [ [3, 7, 3, 7, 3], [4, 4, 4, 4, 4], [2, 4, 6, 8, 10] ] )\n",
    "mat2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat - mat2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat / mat2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another implication is that operations with a single number (a \"scalar\") are repeated for every array element:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat * 5    # Multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat ** 2    # Exponentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This also shows that the `*` does **not** give the **dot/inner product**, for which there are three options:\n",
    "\n",
    "- The operator `@`\n",
    "- The method `.dot()`\n",
    "- The (universal) function `dot()`\n",
    "\n",
    "(There are even the functions `inner()` and `vdot()` but they work slightly differently.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat3 = np.array( [ [1, 1, 1], [2, 2, 2], [3, 3, 3], [4, 4, 4] ] )\n",
    "mat3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat3 @ mat2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat3.dot(mat2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.dot(mat3, mat2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the number of columns of the first array needs to match the number of rows of the second array, of course:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat2 @ mat3    # Error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy contains many functions and methods for mathematical and statistical computations. These are called \"**universal functions**\" or \"**ufuncs**\" in short. See the whole range of ufuncs at [the ufunc webpage](https://numpy.org/doc/stable/reference/ufuncs.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some **unary** ufuncs are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.unique(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.unique(mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sqrt(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.square(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.exp(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.log(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.log10(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.log2(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.average(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.median(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.var(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.std(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.min(vec)    # == min(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.max(vec)    # == max(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.prod(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.cumsum(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.cumprod(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.diff(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sort( np.linspace(10, 0, 4) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.transpose(mat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Almost all unary ufuncs have the argument `axis` for multi-dimensional arrays:\n",
    "\n",
    "- `axis = 0` means: apply the function for each column **across/along** all **rows**, i.e. **vertically**.\n",
    "- `axis = 1` means: apply the function for each row **across/along** all **columns**, i.e. **horizontally**.\n",
    "- Etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(mat, axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(mat, axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean(mat, axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean(mat, axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sort(mat, axis = 1)    # Disrupts the correspondence among rows and columns!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following functions are also available as **methods**, again with the `axis` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat.sum(axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat.sum(axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec.min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec.var()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec.sort()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat.sort(axis = 1)    # Changes the elements of mat in place!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matrices can be **transposed** with the method `.transpose()`, which can be abbreviated as `.T`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat.transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some **binary** ufuncs are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.add(vec, vec2)    # == vec + vec2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.add(mat, mat2)    # Idem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.subtract(vec, vec2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.multiply(vec, vec2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(vec, vec2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.cov(vec, vec2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.corrcoef(mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.cov(mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.maximum(vec, vec2)    # =/= np.max() !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.minimum(vec, vec2)    # =/= np.min() !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.greater(vec, vec2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.less_equal(vec, vec2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Indexing with NumPy arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because NumPy arrays are defined as sequences of sequences, **indexes** and **slices** apply to whole subsequences. For vectors these just the corresponding elements, but for matrices these are the rows. (In general, a single index for a multi-dimensional array will select the whole sub-array on `axis = 0`.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mul[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, you can select on matrix columns by specifying **two indexes separated by a comma**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[1, 1:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is equivalent to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[1][1:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to leave the rows or columns **unspecified**, then you need to use a `:` or `...` as a placeholder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[1, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[1, ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[, 1:3]    # Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[:, 1:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[..., 1:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[1, ...]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, the dots operator `...` allows you leave an **arbitrary amount** of dimensions unspecified (instead of repeating `:` for every dimension):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mul[..., 3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mul[:, :, 3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Negative** indexes and slices work in their usual way, i.e. **counting from the end**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec[-2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec[-4:-2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[-2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[-2, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[:, -3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[:, -3:-2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[:, -3:-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, indexes or slices can always be used for **assigning** new values to elements in an array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[2, 1:3] = 12\n",
    "mat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy also allows for working with **Boolean indexes**. This is a **sequence** of **Boolean values** of the **same length** as the **number of components** on an axis. The `True` values **select** the corresponding component, the `False` values **deselect** it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ind = [False, True, True]\n",
    "mat[ind, 1:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mat[ind]    # Again, a single index for matrices selects only the rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec[ [True, True, True, False, False] ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The negation operator is `~`, the Boolean \"and\" is `&` and the Boolean \"or\" is `|`. In other words, NumPy does **not** work with the core Python operators `not`, `and` and `or`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ind1 = np.array([True, True, True, False, False])\n",
    "ind2 = np.array([False, True, True, True, False])\n",
    "vec[ ~ind1 ]    # == vec[ np.logical_not(ind1) ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec[ ind1 & ind2 ]    # == vec[ np.logical_and(ind1, ind2) ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec[ ind1 | ind2 ]    # == vec[ np.logical_or(ind1, ind2) ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Such Boolean indexes can, again, also be used for **assigning** new values to specific entries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy has the function `where()` with which you can select values **from two different arrays**. The **Boolean index** specified as the first argument **determines which value** is taken:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.where(ind1, vec, vec2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other functions for Boolean arrays include `all()` and `any()`:\n",
    "\n",
    "- `all()` returns `True` if **all** the elements in an array are `True` (and `False` otherwise).\n",
    "- `any()` returns `True` if **at least one** element in an array is `True` (and `False` otherwise)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.all(ind1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.any(ind1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are also available as **methods**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ind1.all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ind1.any()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some other functions which we will not cover in detail:\n",
    "\n",
    "- `isfinite()`\n",
    "- `isinf()`\n",
    "- `isposinf()`\n",
    "- `isneginf()`\n",
    "- `isnan()`\n",
    "- `isreal()`\n",
    "- `iscomplex()`\n",
    "- Etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 NumPy submodules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy has many different **submodules** for advanced computations. One of them is the submodule `linalg` which contains various functions for linear algebra."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy.linalg import svd\n",
    "svd(mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy.linalg import qr\n",
    "qr(mat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See all the functions at [the webpage of the 'linalg' submodule](https://numpy.org/doc/stable/reference/routines.linalg.html). NumPy also has a submodule `random` for random number generation (see [the webpage of the 'random' submodule](https://numpy.org/doc/stable/reference/random/))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "7. NumPy arrays\n",
    "\n",
    "  7.1 Create the following NumPy arrays:\n",
    "  - A vector `vc1` with the elements 5, 10, 15, ... 100.\n",
    "  - A vector `vc2` with the elements 10, 10, 20, 20,... 100, 100.\n",
    "  - A vector `vc3` with the elements 3, -3, 3, -3,... 3, -3 of length 20.\n",
    "  - A vector `vc4` with the elements 0.25, 0.5, 0.75, ... 5.\n",
    "  - A vector `vc5` with the elements 100, 95, 90, ... 5.\n",
    "  - A vector `vc6` with the elements 1, 1, 1, 2, 3, 3, 3, 4, ... 9, 9, 9, 10.\n",
    "\n",
    "  7.2  Count the number of positive elements in `vc3` in a single line of code (your answer should be 10).\n",
    "\n",
    "  7.3 Combine `vc1`, `vc2` and `vc3` into a 20x3 matrix and `vc4`, `vc5` and `vc6` into a 3x20 matrix. Call these matrices `mt7` and `mt8`, respectively.\n",
    "\n",
    "  7.4 Create the 20x3 matrix `mtA` which contains the element-wise addition of `mt7` and `mt8`. Do the same for matrix `mtD` containing the element-wise difference between `mt7` and `mt8`.\n",
    "\n",
    "  7.5 Compute the mean, median and standard deviation of the first and last columns of `mtA` and `mtD`. Try to use a different way of indexing for `mtA` and `mtD`. Assemble all the statistics in a 6x2 table `mt9`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter 5: Data handling with pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 5.1: Working with pandas Series and DataFrames\n",
    "- 5.2: Exploratory statistics with pandas\n",
    "- 5.3: Reading data with pandas\n",
    "- 5.4: Data wrangling with pandas\n",
    "- 5.5: Aggregate statistics with GroupBy objects\n",
    "- 5.6: Reshaping pandas DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The general module for handling data is **pandas**, conventionally abbreviated as `pd`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Working with pandas Series and DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pandas module has the data types `Series` and `DataFrame`: they are **generalizations** of **vectors** and **matrices**, respectively. The difference is that Series or DataFrames can contain **values** of **different data types**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr1 = pd.Series( [3, 4, 'Bart', 'Marc'] )\n",
    "sr1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = pd.DataFrame( { 'name': ['John', 'James', 'Jonah', 'Jeremiah'],\n",
    "                        'exam' : [70, 40, 60, 70],\n",
    "                        'pass': [True, False, True, True]\n",
    "                    } )\n",
    "print(df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: We use the `print()` command in order to get the \"raw\" DataFrame. The reason for this is that Jupyter Notebook automatically formats any DataFrames as a **table**, in contrast to Spyder. Compare:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, you can create a DataFrame with a **dict** of **lists**. The lists must be of **equal length**. Another way to create a DataFrame is with a **dict** of **dicts**. The **keys** in each sub-dict will be **matched**. That means that **omitted** keys will produce **missing values**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = pd.DataFrame( { 'name': {'stu1':'John', 'stu2':'James', 'stu3':'Jonah', 'stu4':'Jeremiah'},\n",
    "                        'exam' : {'stu1':70, 'stu2':40, 'stu4':70, 'stu3':60},\n",
    "                        'pass': {'stu1':True, 'stu4':True, 'stu3':True}\n",
    "                    } )\n",
    "print(df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, pandas orders the columns in a DataFrame in alphabetical order. You can specify the order with the argument `columns`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = pd.DataFrame( { 'name': ['John', 'James', 'Jonah', 'Jeremiah'],\n",
    "                        'exam' : [70, 40, 60, 70],\n",
    "                        'pass': [True, False, True, True]\n",
    "                    }, columns=['name', 'exam', 'pass'] )\n",
    "print(df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you see, pandas automaticaly creates (row) **indexes** for Series or DataFrames. These indexes are an important, intrinsic attribute of pandas objects, since they are used in computations. More specifically, pandas always **aligns** the elements of different pandas objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr2 = pd.Series( [2, 3] )\n",
    "sr2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr1 + sr2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is why you can specify your own indexes with the argument `index`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr1 = pd.Series( ['Bart', 'Marc', 3, 4] , index = ['a', 'b', 'c', 'd'] )\n",
    "sr1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr2 = pd.Series( [2, 3], index = ['c', 'd'])\n",
    "sr2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr1 + sr2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DataFrames will also **align** on the **columns**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3 = pd.DataFrame( {'exam':[15, 25]}, index=['stu3', 'stu4'])\n",
    "print(df3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2 + df3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use both **numeric** and **character indexes** for selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr1[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr1['c']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr1['c'] = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr1[ ['a','c','d'] ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For DataFrames, indexing works a bit different. Because DataFrames are defined as a dict of columns, the columns names are **attributes** of the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['exam']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.exam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, DataFrames do not have the indexing facilities of NumPy matrices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2[2]    # Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2[:, 2]    # Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['stu3']    # Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['stu3', :]    # Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2[2, :]    # Error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead, you have to use the operators `.loc[]` or `.iloc[]` for indexing on DataFrames:\n",
    "\n",
    "- The operator `.loc[]` is to be used with **character indexes** as arguments.\n",
    "- The operator `.iloc[]` is to be used with **numeric indexes** as arguments.\n",
    "\n",
    "Note that both operators work with **square** brackets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.loc['stu3', :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.loc[:, 'exam']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.loc['stu3', 'exam']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.iloc[2, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.iloc[2, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.iloc[:, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.iloc[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.loc['stu3']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last two examples show that **single values** select **rows** of a DataFrame. The general rule is:\n",
    "\n",
    "- Single indexes with `.loc[]` (e.g. `df2.loc['stu3']`) or `.iloc[]` (e.g. `df2.iloc[2]`) select **rows** in the DataFrame.\n",
    "- Single (string) values between square brackets (e.g. `df2['exam']`) only select **column names** in the DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember to use the placeholder `:` for unspecified rows or columns (just like with NumPy arrays). Otherwise, you will get unexpected results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.loc[, 'exam']    # Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.iloc[, 0]    # Error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.loc[\"stu3\", ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.iloc[2, ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Slicing** in pandas is also somewhat different from ordinary Python slices, although more flexible. For starters, it is also possible to slice with **character indexes**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2.loc['stu2':'stu4', :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2.loc[:,'name':'exam'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr1.loc['b':'c']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Secondly, such character slices also include the **end index**, which **contrasts** to the usual **numeric slices**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2.iloc[1:3, :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2.iloc[:, 0:1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr1.iloc[1:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, both kinds of slices are also possible with ordinary **square brackets**, but slices on **column names** do **not** work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2['stu2':'stu4'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2[1:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2['exam':'name'])    # Empty DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to **deselect** rows or columns, then you can make use of the **method** `.drop()`. This method has an argument `axis` which you can use to deselect columns (since, by default, `axis = 0`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2.drop('stu3'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2.drop( ['stu3', 'stu4'] ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2.drop('stu3':'stu4'))    # Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2.drop('exam', axis = 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2.drop('exam', axis = 'columns'))    # pandas also accepts 'columns' (instead of 1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Underlying a pandas Series or DataFrame there is a NumPy array, which you can access with the attribute `.values`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr1.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Exploratory statistics with pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because pandas objects are built on NumPy arrays, the **ufuncs** are also available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3 = pd.DataFrame(mat)\n",
    "print(df3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.mean(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.var()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method `.describe()` gives a **statistical summary** of the (numeric) columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df3.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The methods `.head()` and `.tail()` give the, by default **5**, **first** and **last rows**, respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df3.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df3.tail())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In pandas the ufuncs automatically **omit missing values**. You can change that behavior with the argument `skipna`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr3 = pd.Series( [3, 4, np.nan, 6, 7] )\n",
    "sr3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr3.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr3.mean(skipna = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the method `.apply()` you can apply **any available function** on your data. This is especially useful with a lambda statement, in which you create your arbitrary function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.apply(lambda x: sum(1/(1+x**3)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fun = lambda x: sum(1/(1+x**3))\n",
    "df3.apply(fun)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3.apply(fun, axis = 1)    # Apply across/along the columns, i.e. row-wise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The usual binary arithmetic operations are also available as **methods**. In fact, pandas also has the **\"reversed\" counterpart** of every binary ufunc. Recall that pandas does **alignment** of (row) indexes and column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df4 = pd.DataFrame(mat2)\n",
    "print(df4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df3.add(df4) )    # == df3 + df4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df3.radd(df4) )    # == df4 + df3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df3.sub(df4) )    # == df3 - df4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df3.rsub(df4) )    # == df4 - df3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df3.mul(df4) )    # Etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df3.rmul(df4) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df4.cov() )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df4.corr() )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df4.corr(method = 'spearman') )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df4.corr(method = 'kendall') )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Reading data with pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DataFrames are suited to **spreadsheet-like data**, so you will usually use them for your **external data file(s)**. (The NumPy module also has functions for reading and writing data, but we will not cover them here.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will work with the `iris` data set (which should be in your working directory). You can read in data with the function `read_table()`, but pandas has many functions for different formats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd    # If not loaded yet\n",
    "iris = pd.read_csv('iris.csv')\n",
    "print(iris)    # Problem with the separator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = pd.read_csv('iris.csv', sep = ';')\n",
    "print(iris)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(iris.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(iris.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(iris.tail())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "8. DataFrames\n",
    "\n",
    "  8.1 Do the same element-wise addition and subtraction as in Exercise 7.3 but with DataFrames. Convert the matrices `mt7` and `mt8` from Exercise 7.3 to DataFrames `pd7` and `pd8`, respectively. Give both DataFrames the column names `One`, `Two`, and `Tri`. Call the resulting DataFrames with element-wise additions and differences `pdA` and `pdD`, respectively. Again, compute the mean, median and standard deviation of the first and the last column. (No need to put the statistics in a new object.)\n",
    "\n",
    "  8.2 Compute the mean and variance for only the last ten rows in `pdA` and `pdD`.\n",
    "\n",
    "\n",
    "9. The `iris`dataset\n",
    "\n",
    "  9.1 Create three separate DataFrames of the `iris` dataset for the species in the column `Species`. You can use logical vectors for indexing. Compute the mean, variance, skewness and kurtosis for the four numeric variables in each subset. (These variables are `Sepal_Length`, `Sepal_Width`, `Petal_Length` and `Petal_Width`.)\n",
    "\n",
    "  9.2 Compute the correlation matrix of the four numeric variables in each subset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4 Data wrangling with pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Series and DataFrames in pandas have many **methods** and **functions** for **data preparation**. We will illustrate:\n",
    "- Removing missing values with the method `.dropna()`\n",
    "- Replacing missing values with the method `.fillna()`\n",
    "- Replacing values in general with the method `.replace()`\n",
    "- Identifying duplicated values with the method `.duplicated()`\n",
    "- Removing duplicated values with the method `.drop_duplicates()`\n",
    "- Discretizing (\"binning\") data with the function `cut()`\n",
    "- Creating dummy variables with the function `get_dummies()`\n",
    "- Combining data row-wise or column-wise with the function `concat()`\n",
    "- Merging or joining data with the function `merge()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can **remove missing values** with the method `.dropna()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df2.dropna() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can **replace missing values** with the method `.fillna()`. You specify the replacement value as the (first) argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df2.fillna('No') )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a **general method** `.replace()` for replacing old values with new values. It works with two arguments, which can be sequences or dicts for replacing more than one value. The argument `inplace` specifies whether you want to modify the pandas object itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['pass']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['pass'].replace(True, 'yes')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['pass'].replace( [True, np.nan], ['yes', 'no'] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['pass'].replace( {True:'yes', np.nan:'no'} )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['pass'].replace( {True:'yes', np.nan:'no'} , inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method `.drop_duplicates()` **removes duplicated values** (the method `.duplicated()` returns a Series of `True` or `False` for every observation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['exam']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['exam'].duplicated()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['exam'].drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df2.drop_duplicates() )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['name'].replace('John', 'Jeremiah', inplace = True)\n",
    "df2.duplicated()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df2.drop_duplicates() )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df2.drop_duplicates(keep = 'last') )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Undo the replacement (for the remainder of this Class):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.loc['stu1', 'name'] = 'John'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then there are also some useful **functions**. For instance, the function `cut()` discretizes a numeric variable into **bins**. You specify the cut-off points as the second argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.cut(df2['exam'], [0, 50, 65, 75, 100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `cut()` bins the intervals with the end-point inclusive. You can override that with the argument `right = False`. You can also give names to your categories with the argument `labels`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.cut(df2['exam'], [0, 50, 65, 75, 100], right = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.cut(df2['exam'], [0, 50, 65, 75, 100], right = False,\n",
    "       labels = ['fail', 'pass', 'cum_laude', 'magna_cum_laude'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `get_dummies()` computes **dummy variables** on the basis of an argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( pd.get_dummies(df2['pass']) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often you can combine `cut()` and `get_dummies()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( pd.get_dummies(\n",
    "        pd.cut(df2['exam'], [0, 50, 65, 75, 100], right = False,\n",
    "        labels = ['fail', 'pass', 'cum_laude', 'magna_cum_laude'])\n",
    "    ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because pandas was developed for handling external data sets, there are various functions for **combining** data. For instance, the function `concat()` combines a sequence of pandas object along an axis (specified as an argument)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df5 = pd.DataFrame( {'name':['Jasmine', 'Jennifer', 'Joyce'],\n",
    "                    'exam':[80, 60, 70],\n",
    "                    'pass':['yes', 'yes', 'yes']\n",
    "                    } )\n",
    "print(df5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df6 = pd.concat( [df2, df5], axis = 0)\n",
    "print(df6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because `axis = 0` is the default, we could also have entered:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.concat( [df2, df5] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It often occurs that data on the same observations are recorded in different data sets. If we need to use information from both (or more) data sets, then we need to **merge** the data sets. This can be done with the function `merge()`. It has the arguments `left_on` and `right_on` for specifying the join column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df7 = pd.DataFrame( {'ID':['Jacob', 'James', 'Jasmine', 'Jennifer', 'Jeremiah', 'John', 'Jonah', 'Joyce', 'Juliet'],\n",
    "                    'gender':['Male', 'Male', 'Female', 'Female', 'Male', 'Male', 'Male', 'Female', 'Female']\n",
    "                    } )\n",
    "print(df7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df8 = pd.merge(df6, df7, left_on = 'name', right_on = 'ID')\n",
    "print(df8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `merge()` function also has a single argument `on` for when the join columns have the same name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df7.columns = ['name', 'gender']\n",
    "print( pd.merge(df6, df7, on = 'name') )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is even possible to join on indexes with the arguments `left_index = True` and/or `right_index = True`. We will not cover that topic here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the `merge()` function retains the elements which appear in **both** data sets. This is called an \"**inner join**\". For **other join types** you have to use the argument `how`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( pd.merge(df6, df7, on = 'name', how = 'inner') )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( pd.merge(df6, df7, on = 'name', how = 'left') )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( pd.merge(df6, df7, on = 'name', how = 'right') )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( pd.merge(df6, df7, on = 'name', how = 'outer') )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additional example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df9 = df6.drop('stu4')\n",
    "print( pd.merge(df9, df7, on = 'name', how = 'outer') )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.5 Aggregate statistics with GroupBy objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A very useful feature in data analysis are **aggregate statistics** for **groups** of observations. In pandas aggregate statistics can be easily computed by first creating a `GroupBy` object. This is done with the method `.groupby()` in which you specify the grouping variable as the argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb1 = df8.groupby('gender')\n",
    "gb1    # GroupBy objects are not informative themselves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb1['exam']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb1.exam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On such a GroupBy object all the **usual methods** can be executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb1.exam.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb1['exam'].mean()    # Different syntax for same result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb1.exam.median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb1.exam.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( gb1.exam.describe() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use **your own functions** by specifying them as the argument of the method `.agg()`. This is an abbreviation of `.aggregate()` and it also has the same result as `.apply()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb1.exam.agg(fun)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb1.exam.aggregate(fun)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb1.exam.apply(fun)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, the `.agg()` (or `.aggregate()`) method is somewhat more flexible. For instance, you also specify **more than one function** in a sequence with `.agg()`. The result is typically a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( gb1.exam.agg( ['mean','std',fun] ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Secondly, you can specify **different functions** for **different columns**. You do this by **mapping** functions to columns in a **dict**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( gb1.agg( {'exam':'mean' , 'pass':'count'} ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aggregate statistics are sometimes also displayed in \"**pivot tables**\". The pandas module has the method `.pivot_table()` for producing these. By default, it computes the means, but you can change that with the argument `aggfunc`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df8.pivot_table(values = 'exam', index = 'gender', columns = 'pass') )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df8.pivot_table(values = 'exam', index = 'gender', columns = 'pass', aggfunc = 'mean') )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df8.pivot_table(values = 'exam', index = 'gender', columns = 'pass', aggfunc = 'count') )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, `.pivot_table()` gives **missing values** for **non-occurring combinations**. You can change that by specifying the argument `fill_value`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df8.pivot_table(values = 'exam', index = 'gender', columns = 'pass', aggfunc = 'count', fill_value = 0) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also add **marginal statistics** with the argument `margins=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df8.pivot_table(values = 'exam', index = 'gender', columns = 'pass', aggfunc = 'mean', fill_value = 0, margins = True) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( df8.pivot_table(values = 'exam', index = 'gender', columns = 'pass', aggfunc = 'count', fill_value = 0, margins = True) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you specify the `count()` function for `aggfunc`, then you basically create a **frequency table**. You can also use the function `crosstab()` on categorical columns to produce a frequency table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( pd.crosstab(df8.gender, df8['pass']) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, the `crosstab()` function also allows for specifying an `aggfunc` and a numeric column for the `values`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( pd.crosstab(index = df8.gender, columns = df8['pass'], values = df8.exam, aggfunc = 'mean', margins = True) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.6 Reshaping pandas DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pandas module also has functions for **reshaping** DataFrames from \"**long format**\" to \"**wide format**\" or vice versa. For instance, the function `melt()` reshapes DataFrames in wide format **to long format**. You specify the columns to be stored in one column as a sequence for the argument `value_vars`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( pd.melt(df3, value_vars = df3.columns) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, the `melt()` function also has the argument `id_vars` which specifies **what counts as one row**. In the example it is the index, so we code this into a separate column. Then this is what `melt()` does:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3['ID'] = list(df3.index)\n",
    "print(df3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( pd.melt(df3, id_vars = 'ID', value_vars = df3.columns[:-1]) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `melt()` function chooses the **names** `variable` and `value` by default for the molten DataFrame. You can change these columns names with the argument `var_name` and `value_name`, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df10 = pd.melt(df3, id_vars = 'ID', value_vars = df3.columns[:-1],\n",
    "               var_name = 'col_cat', value_name = 'number')\n",
    "print(df10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `pivot()` does the opposite transformation, viz. it reshapes from long **to wide format**. You specify three arguments:\n",
    "\n",
    "- `index`: the equivalent of `id_vars`\n",
    "- `columns`: the equivalent of `var_name`\n",
    "- `values`: the equivalent of `value_var`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df11 = pd.pivot(index = df10.ID, columns = df10.col_cat, values = df10.number)    # Error?\n",
    "print(df11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `pivot()` function is also available as a **method**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df11 = df10.pivot(index = 'ID', columns = 'col_cat', values = 'number')\n",
    "print(df11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pandas module also has the methods `.stack()` and `.unstack()` for melting and pivoting, respectively. Both methods also create \"multiple indexes\" which we will not discuss further here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df12 = df11.stack()\n",
    "print(df12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df13 = df12.unstack()\n",
    "print(df13)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "```\n",
    "authors = pd.DataFrame({\n",
    "\t'firstname' : ['Eric', 'Wes', 'Amit', 'Al', 'Al', 'Jake', 'Mahesh'],\n",
    "\t'surname' : ['Matthes', 'Mckinney', 'Saha', 'Sweigart', 'Sweigart', 'VanderPlas', 'Venkitachalam'],\n",
    "\t'title' : ['Python Crash Course', 'Python for Data Analysis', 'Doing Math with Python',\n",
    "               'Automate the Boring Stuff with Python', 'Invent your own Computer Games with Python',\n",
    "\t\t\t   'Python Data Science Handbook', 'Python Playground']\n",
    "\t})\n",
    "books = pd.DataFrame({\n",
    "\t'title' : ['Automate the Boring Stuff with Python', 'Data Science from Scratch: First Principles with Python',\n",
    "               'Doing Math with Python', 'Invent your own Computer Games with Python', 'Python Crash Course',\n",
    "               'Python Data Science Handbook', 'Python for Data Analysis', 'Python Playground'],\n",
    "\t'year' : [2015, 2015, 2015, 2017, 2015, 2016, 2017, 2015],\n",
    "\t'publisher' : ['No Starch', \"O'Reilly\", 'No Starch', 'No Starch', 'No Starch', \"O'Reilly\", \"O'Reilly\", 'No Starch']\n",
    "\t})\n",
    "```\n",
    "\n",
    "10. Data wrangling\n",
    "\n",
    "  10.1 Copy paste the two DataFrames above and perform an inner join, left join, right join and outer join.\n",
    "  \n",
    "  10.2 Construct a frequency table with the columns `surname` and `year` from the inner merged DataFrame.\n",
    "\n",
    "\n",
    "11. Aggregate statistics\n",
    "\n",
    "  11.1 Repeat the computation of the summary statistics in Exercise 9.1 using a `GroupBy` object.\n",
    "  \n",
    "  11.2 **(Optional:)** Create a DataFrame in long format from the `iris` dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}