notebooks/03 Data Representation for Machine Learning.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext watermark\n",
    "%watermark  -d -u -a 'Andreas Mueller, Kyle Kastner, Sebastian Raschka' -v -p numpy,scipy,matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The use of watermark (above) is optional, and we use it to keep track of the changes while developing the tutorial material. (You can install this IPython extension via \"pip install watermark\". For more information, please see: https://github.com/rasbt/watermark)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Webstep Scikit-learn Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Representation and Visualization of Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Machine learning is about fitting models to data; for that reason, we'll start by\n",
    "discussing how data can be represented in order to be understood by the computer.  Along\n",
    "with this, we'll build on our matplotlib examples from the previous section and show some\n",
    "examples of how to visualize data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data in scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data in scikit-learn, with very few exceptions, is assumed to be stored as a\n",
    "**two-dimensional array**, of shape `[n_samples, n_features]`. Many algorithms also accept ``scipy.sparse`` matrices of the same shape."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **n_samples:**   The number of samples: each sample is an item to process (e.g. classify).\n",
    "  A sample can be a document, a picture, a sound, a video, an astronomical object,\n",
    "  a row in database or CSV file,\n",
    "  or whatever you can describe with a fixed set of quantitative traits.\n",
    "- **n_features:**  The number of features or distinct traits that can be used to describe each\n",
    "  item in a quantitative manner.  Features are generally real-valued, but may be Boolean or\n",
    "  discrete-valued in some cases.\n",
    "\n",
    "The number of features must be fixed in advance. However it can be very high dimensional\n",
    "(e.g. millions of features) with most of them being \"zeros\" for a given sample. This is a case\n",
    "where `scipy.sparse` matrices can be useful, in that they are\n",
    "much more memory-efficient than NumPy arrays.\n",
    "\n",
    "As we recall from the previous section (or Jupyter notebook), we represent samples (data points or instances) as rows in the data array, and we store the corresponding features, the \"dimensions,\" as columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Simple Example: the Iris Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example of a simple dataset, we're going to take a look at the iris data stored by scikit-learn.\n",
    "The data consists of measurements of three different iris flower species.  There are three different species of iris\n",
    "in this particular dataset as illustrated below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Iris Setosa\n",
    "<img src=\"figures/iris_setosa.jpg\" width=\"50%\">\n",
    "\n",
    "Iris Versicolor\n",
    "<img src=\"figures/iris_versicolor.jpg\" width=\"50%\">\n",
    "\n",
    "Iris Virginica\n",
    "<img src=\"figures/iris_virginica.jpg\" width=\"50%\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quick Question:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's assume that we are interested in categorizing new observations; we want to predict whether unknown flowers are  Iris-Setosa, Iris-Versicolor, or Iris-Virginica flowers, respectively. Based on what we've discussed in the previous section, how would we construct such a dataset?***\n",
    "\n",
    "Remember: we need a 2D array of size `[n_samples x n_features]`.\n",
    "\n",
    "- What would the `n_samples` refer to?\n",
    "\n",
    "- What might the `n_features` refer to?\n",
    "\n",
    "Remember that there must be a **fixed** number of features for each sample, and feature\n",
    "number *j* must be a similar kind of quantity for each sample."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the Iris Data with Scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For future experiments with machine learning algorithms, we recommend you to bookmark the [UCI machine learning repository](http://archive.ics.uci.edu/ml/), which hosts many of the commonly used datasets that are useful for benchmarking machine learning algorithms -- a very popular resource for machine learning practioners and researchers. Conveniently, some of these datasets are already included in scikit-learn so that we can skip the tedious parts of downloading, reading, parsing, and cleaning these text/CSV files. You can find a list of available datasets in scikit-learn at: http://scikit-learn.org/stable/datasets/#toy-datasets.\n",
    "\n",
    "For example, scikit-learn has a very straightforward set of data on these iris species.  The data consist of\n",
    "the following:\n",
    "\n",
    "- Features in the Iris dataset:\n",
    "\n",
    "  1. sepal length in cm\n",
    "  2. sepal width in cm\n",
    "  3. petal length in cm\n",
    "  4. petal width in cm\n",
    "\n",
    "- Target classes to predict:\n",
    "\n",
    "  1. Iris Setosa\n",
    "  2. Iris Versicolour\n",
    "  3. Iris Virginica"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/petal_sepal.jpg\" alt=\"Sepal\" style=\"width: 50%;\"/>\n",
    "\n",
    "(Image: \"Petal-sepal\". Licensed under CC BY-SA 3.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Petal-sepal.jpg#/media/File:Petal-sepal.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``scikit-learn`` embeds a copy of the iris CSV file along with a helper function to load it into numpy arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "iris = load_iris()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting dataset is a ``Bunch`` object: you can see what's available using\n",
    "the method ``keys()``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The features of each sample flower are stored in the ``data`` attribute of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_samples, n_features = iris.data.shape\n",
    "print('Number of samples:', n_samples)\n",
    "print('Number of features:', n_features)\n",
    "# the sepal length, sepal width, petal length and petal width of the first sample (first flower)\n",
    "print(iris.data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The information about the class of each sample is stored in the ``target`` attribute of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(iris.data.shape)\n",
    "print(iris.target.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(iris.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.bincount(iris.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the NumPy's bincount function (above), we can see that the classes are distributed uniformly in this dataset - there are 50 flowers from each species, where\n",
    "\n",
    "- class 0: Iris-Setosa\n",
    "- class 1: Iris-Versicolor\n",
    "- class 2: Iris-Virginica"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These class names are stored in the last attribute, namely ``target_names``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(iris.target_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data is four dimensional, but we can visualize one or two of the dimensions\n",
    "at a time using a simple histogram or scatter-plot.  Again, we'll start by enabling\n",
    "matplotlib inline mode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_index = 3\n",
    "colors = ['blue', 'red', 'green']\n",
    "\n",
    "for label, color in zip(range(len(iris.target_names)), colors):\n",
    "    plt.hist(iris.data[iris.target==label, x_index], \n",
    "             label=iris.target_names[label],\n",
    "             color=color)\n",
    "\n",
    "plt.xlabel(iris.feature_names[x_index])\n",
    "plt.legend(loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_index = 3\n",
    "y_index = 0\n",
    "\n",
    "colors = ['blue', 'red', 'green']\n",
    "\n",
    "for label, color in zip(range(len(iris.target_names)), colors):\n",
    "    plt.scatter(iris.data[iris.target==label, x_index], \n",
    "                iris.data[iris.target==label, y_index],\n",
    "                label=iris.target_names[label],\n",
    "                c=color)\n",
    "\n",
    "plt.xlabel(iris.feature_names[x_index])\n",
    "plt.ylabel(iris.feature_names[y_index])\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quick Exercise:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Change** `x_index` **and** `y_index` **in the above script\n",
    "and find a combination of two parameters\n",
    "which maximally separate the three classes.**\n",
    "\n",
    "This exercise is a preview of **dimensionality reduction**, which we'll see later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An aside: scatterplot matrices\n",
    "\n",
    "Instead of looking at the data one plot at a time, a common tool that analysts use is called the **scatterplot matrix**.\n",
    "\n",
    "Scatterplot matrices show scatter plots between all features in the data set, as well as histograms to show the distribution of each feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "    \n",
    "iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)\n",
    "pd.tools.plotting.scatter_matrix(iris_df, figsize=(8, 8));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other Available Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Scikit-learn makes available a host of datasets for testing learning algorithms](http://scikit-learn.org/stable/datasets/#dataset-loading-utilities).\n",
    "They come in three flavors:\n",
    "\n",
    "- **Packaged Data:** these small datasets are packaged with the scikit-learn installation,\n",
    "  and can be downloaded using the tools in ``sklearn.datasets.load_*``\n",
    "- **Downloadable Data:** these larger datasets are available for download, and scikit-learn\n",
    "  includes tools which streamline this process.  These tools can be found in\n",
    "  ``sklearn.datasets.fetch_*``\n",
    "- **Generated Data:** there are several datasets which are generated from models based on a\n",
    "  random seed.  These are available in the ``sklearn.datasets.make_*``\n",
    "\n",
    "You can explore the available dataset loaders, fetchers, and generators using IPython's\n",
    "tab-completion functionality.  After importing the ``datasets`` submodule from ``sklearn``,\n",
    "type\n",
    "\n",
    "    datasets.load_<TAB>\n",
    "\n",
    "or\n",
    "\n",
    "    datasets.fetch_<TAB>\n",
    "\n",
    "or\n",
    "\n",
    "    datasets.make_<TAB>\n",
    "\n",
    "to see a list of available functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data downloaded using the ``fetch_`` scripts are stored locally,\n",
    "within a subdirectory of your home directory.\n",
    "You can use the following to determine where it is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import get_data_home\n",
    "get_data_home()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Be warned: many of these datasets are quite large, and can take a long time to download!\n",
    "\n",
    "If you start a download within the IPython notebook\n",
    "and you want to kill it, you can use ipython's \"kernel interrupt\" feature, available in the menu or using\n",
    "the shortcut ``Ctrl-m i``.\n",
    "\n",
    "You can press ``Ctrl-m h`` for a list of all ``ipython`` keyboard shortcuts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Digits Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll take a look at another dataset, one where we have to put a bit\n",
    "more thought into how to represent the data.  We can explore the data in\n",
    "a similar manner as above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_samples, n_features = digits.data.shape\n",
    "print((n_samples, n_features))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(digits.data[0])\n",
    "print(digits.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The target here is just the digit represented by the data.  The data is an array of\n",
    "length 64... but what does this data mean?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a clue in the fact that we have two versions of the data array:\n",
    "``data`` and ``images``.  Let's take a look at them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(digits.data.shape)\n",
    "print(digits.images.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that they're related by a simple reshaping:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "print(np.all(digits.images.reshape((1797, 64)) == digits.data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the data.  It's little bit more involved than the simple scatter-plot\n",
    "we used above, but we can do it rather quickly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up the figure\n",
    "fig = plt.figure(figsize=(6, 6))  # figure size in inches\n",
    "fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.05, wspace=0.05)\n",
    "\n",
    "# plot the digits: each image is 8x8 pixels\n",
    "for i in range(64):\n",
    "    ax = fig.add_subplot(8, 8, i + 1, xticks=[], yticks=[])\n",
    "    ax.imshow(digits.images[i], cmap=plt.cm.binary, interpolation='nearest')\n",
    "    \n",
    "    # label the image with the target value\n",
    "    ax.text(0, 7, str(digits.target[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see now what the features mean.  Each feature is a real-valued quantity representing the\n",
    "darkness of a pixel in an 8x8 image of a hand-written digit.\n",
    "\n",
    "Even though each sample has data that is inherently two-dimensional, the data matrix flattens\n",
    "this 2D data into a **single vector**, which can be contained in one **row** of the data matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generated Data: the S-Curve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One dataset often used as an example of a simple nonlinear dataset is the S-cure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_s_curve\n",
    "data, colors = make_s_curve(n_samples=1000)\n",
    "print(data.shape)\n",
    "print(colors.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "ax = plt.axes(projection='3d')\n",
    "ax.scatter(data[:, 0], data[:, 1], data[:, 2], c=colors)\n",
    "ax.view_init(10, -60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example is typically used with an unsupervised learning method called Locally\n",
    "Linear Embedding.  We'll explore unsupervised learning in detail later in the tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: working with the faces dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we'll take a moment for you to explore the datasets yourself.\n",
    "Later on we'll be using the Olivetti faces dataset.\n",
    "Take a moment to fetch the data (about 1.4MB), and visualize the faces.\n",
    "You can copy the code used to visualize the digits above, and modify it for this data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_olivetti_faces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fetch the faces data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use a script like above to plot the faces image data.\n",
    "# hint: plt.cm.bone is a good colormap for this data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load solutions/03A_faces_plot.py"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}