WhatIsML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "duration": "1.5 hours"
   },
   "source": [
    "# Beginning Machine Learning with scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started with This Course\n",
    "\n",
    "Let us take a look at how we will install the software and learning materials needed for this course...\n",
    "\n",
    "> <font size=\"+2\">https://github.com/DavidMertz/ML-Live-Beginner</font>\n",
    "\n",
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What Is Machine Learning?\n",
    "\n",
    "> **\"If you torture the data enough, nature will always confess.\"** –Ronald Coase\n",
    "\n",
    "As a one line version—not entirely original—I like to think of machine learning as \"statistics on steroids.\"  That characterization may be more cute than is necessary, but it is a good start.  Others have used phrases like \"extracting knowledge from raw data by computational means.\"\n",
    "\n",
    "The lede on the Wikipedia article provides a bit more.\n",
    "\n",
    "![Wikipedia entry](img/ML-Wikipedia.png)\n",
    "\n",
    "Cite: [Wikipedia, 09:29, 2018 October 4](https://en.wikipedia.org/w/index.php?title=Machine_learning&oldid=862453222)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning Libraries\n",
    "\n",
    "There are many software libraries available for machine learning.  Some of them are listed below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For General Machine Learning\n",
    "\n",
    "* **[scikit-learn](http://scikit-learn.org/)**: Free Software (BSD License). The topic of this course\n",
    "* **[Spark MLLib](https://spark.apache.org/mllib/)**: Free Software (Apache License 2.0). Spark based machine learning with interfaces to Java, Scala, Python, and R. MLlib fits into Spark's APIs and interoperates with NumPy in Python and R libraries. You can use any Hadoop data source (e.g. HDFS, HBase, or local files), making it easy to plug into Hadoop workflows.\n",
    "* **[mlpack](https://www.mlpack.org/)**: Free Software (3-clause\n",
    "BSD license;  Mozilla Public License v2.0; Boost Software License, version 1.0). A fast, flexible machine learning library, written in C++, that aims to provide fast, extensible implementations of cutting-edge machine learning algorithms. mlpack provides these algorithms as simple command-line programs, Python bindings, and C++ classes which can then be integrated into larger-scale machine learning solutions.\n",
    "* **[Accord.NET Framework](http://accord-framework.net/)**: Free Software (LGPLv2.1) The Accord.NET Framework is a .NET machine learning framework combined with audio and image processing libraries completely written in C#. It is a complete framework for building production-grade computer vision, computer audition, signal processing and statistics applications even for commercial use. \n",
    "* **[WEKA](https://www.cs.waikato.ac.nz/ml/weka/)**: Free Software (GPL). Data Mining Software in Java. Weka is a collection of machine learning algorithms for data mining tasks. It contains tools for data preparation, classification, regression, clustering, association rules mining, and visualization. \n",
    "* **[Shogun](http://shogun-toolbox.org/)**: Free Software (GPLv3). Shogun is among the oldest of machine learning libraries, but continues to be well maintained and optimized. Shogun was created in 1999 and written in C++. Via SWIG, Shogun can be used in Java, Python, C#, Ruby, R, Lua, Octave, and Matlab. Shogun is designed for unified large-scale learning for a broad range of feature types and learning settings, like classification, regression, or explorative data analysis.\n",
    "* **[Torch](http://torch.ch/)** and **[PyTorch](https://pytorch.org/)**: Free Software (custom BSD-ish license). Torch is a scientific computing framework with wide support for machine learning algorithms, emphasizing GPU computation. Torch is based on the scripting language Lua, PyTorch is Python bindings to the underlying engine and C/CUDA implementation. The goal of Torch is to have maximum flexibility and speed in building your scientific algorithms while making the process extremely simple. Torch comes with a large ecosystem of community-driven packages in machine learning, computer vision, signal processing, parallel processing, image, video, audio and networking among others, and builds on top of the Lua community."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For Deep Learning\n",
    "\n",
    "* **[TensorFlow](https://www.tensorflow.org/)**: Free Software (Apache 2.0 open source license). TensorFlow is an open source software library for numerical computation using data flow graphs. TensorFlow implements what are called data flow graphs, where batches of data (\"tensors\") can be processed by a series of algorithms described by a graph. The movements of the data through the system are called \"flows\". Graphs can be assembled with C++ or Python and can be processed on CPUs or GPUs.\n",
    "* **[Theano](http://deeplearning.net/software/theano/)**: Free Software (BSD License). Theano is a Python library that lets you to define, optimize, and evaluate mathematical expressions, especially ones with multi-dimensional arrays (numpy.ndarray). Using Theano it is possible to attain speeds rivaling hand-crafted C implementations for problems involving large amounts of data. It was written at the LISA lab to support rapid development of efficient machine learning algorithms. Theano is named after the Greek mathematician, who may have been Pythagoras’ wife. \n",
    "* **[Keras](https://keras.io/)**: Free Software (MIT License). Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. Keras allows for easy and fast prototyping (through user friendliness, modularity, and extensibility). It Supports both convolutional networks and recurrent networks, as well as combinations of the two. Runs seamlessly on CPU and GPU.\n",
    "* Apache MXNet\n",
    "* **[Caffe](http://caffe.berkeleyvision.org/)**: Free Software (BSD 2-Clause License). Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR) and by community contributors. Yangqing Jia created the project during his PhD at UC Berkeley. Bindings for Python and MATLAB are part of the library.\n",
    "* **[Chainer](https://chainer.org/)**: Free Software (MIT License). Chainer supports CUDA computation and runs on multiple GPUs with little effort. Chainer supports various network architectures including feed-forward nets, convnets, recurrent nets and recursive nets. It also supports per-batch architectures. Forward computation can include any control flow statements of Python without lacking the ability of backpropagation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cloud Focused\n",
    "\n",
    "* **[Amazon SageMaker](https://aws.amazon.com/sagemaker)**: Commercial. Amazon SageMaker provides fully managed instances running Jupyter notebooks for training data exploration and preprocessing. These notebooks are pre-loaded with CUDA and cuDNN drivers for popular deep learning platforms, Anaconda packages, and libraries for TensorFlow, Apache MXNet, Chainer, and PyTorch.\n",
    "* **[Google Cloud Machine Learning]()**: Commercial. Google Cloud Machine Learning (ML) Engine is a managed service that allows developers and data scientists to build and bring machine learning models to production. Cloud ML Engine offers training and prediction services, which can be used together or individually. Cloud ML provides access to Python libraries TensorFow, Keras, XGBoost, and scikit-learn.\n",
    "* **[Azure ML Studio](https://studio.azureml.net/)**: Commercial and proprietary. Azure ML Studio allows Microsoft Azure users to create and train models, then turn them into APIs that can be consumed by other services. A wide range of algorithms are available, from both Microsoft and third parties. A free-of-cost trial allows evaluation for eight hours."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What Is scikit-learn?\n",
    "\n",
    "Scikit-learn provides a large range of algorithms in machine learning that are unified under a common and intuitive API. Most of the dozens of classes provided for various kinds of models share the large majority of the same calling interface. Very often—as we will see in examples below—you can easily substitute one algorithm for another with nearly no change in your underlying code. This allows you to explore the problem space quickly, and often arrive at an optimal, or at least satisficing$^1$ approach to your problem domain or datasets.\n",
    "\n",
    "* Simple and efficient tools for data mining and data analysis\n",
    "* Accessible to everybody, and reusable in various contexts\n",
    "* Built on NumPy, SciPy, and matplotlib\n",
    "* Open source, commercially usable - BSD license\n",
    "\n",
    "<hr/>\n",
    "\n",
    "<small>$^1$<i>Satisficing is a decision-making strategy of searching through the alternatives until an acceptability threshold is met. It is a portmanteau of satisfy and suffice, and was introduced by Herbert A. Simon in 1956. He maintained that many natural problems are characterized by computational intractability or a lack of information, both of which preclude the use of mathematical optimization procedures.</i></small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview of Techniques Used in Machine Learning\n",
    "\n",
    "The diagram below is from the scikit-learn documentation, but the same general schematic of different techniques and algorithms that it outlines applies equally to any other library.  The classes represented in bubbles mostly will have equivalent versions in other libraries.\n",
    "\n",
    "![Scikit-learn topic areas](img/sklearn-topics.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Difference between \"Deep Learning\" and other ML Techniques\n",
    "\n",
    "### Neural Networks\n",
    "\n",
    "The basic idea of a \"multilayer perceptron\" is a \"feed-forward\" artificial neural network, composed of \"neurons\" arranged in \"layers.\" A common illustration is similar to that at right. This idea of \"Hebbian networks\" has existed since the 1940s, but it really only became a machine learning technique with Paul Werbos' 1975 introduction of \"backpropagation\" as a means to train such networks. Either way, the ideas are fairly old.\n",
    "\n",
    "![Basic perceptron](img/basic-perceptron.png)\n",
    "\n",
    "Included in diagram is a network with 4 layers and 12 connections (i.e. \"parameters\"). If it were \"fully connected\" the diagram would have 16 parameters. What makes a particular trained network special is the set of \"weights\" in the connections, illustrated and commonly named as subscripted  $w$ values.\n",
    "\n",
    "For many decades after neural networks were known, they remained a minor area of interest. Usually a variety of other techniques rooted in statistics and linear algebra were more effective in solving problems of classification, regression, and clustering.\n",
    "\n",
    "Image credit: [\"Feedforward Neural Networks\", John McGonagle and yushi 21](https://brilliant.org/wiki/feedforward-neural-networks/)\n",
    "\n",
    "---\n",
    "\n",
    "### What if We Had a LOT More Neurons?\n",
    "\n",
    "In the last decade or less, neural networks—mathematically not much different from those described in the 1940s—grew much larger. For example, the extremely power Inception v3 image classifier consists of approximately 23.8 million parameters across about 140 layers. Layers generally each have many more neurons than the dozen or fewer shown in textbook illustrations like the one above. Scikit-learn has basic neural network techniques, but their use is mostly for the uses that made sense more than five years ago.\n",
    "\n",
    "![Inception v3](img/inception-v3.png)\n",
    "\n",
    "Classic \"fully connected\" layers make up only a small number of those used. More than anything else, the effect and reason for this is to limit the combinatorial explosion of connections, limiting the parameters to only 24 million.\n",
    "\n",
    "Image credit: [\"Advanced Guide to Inception v3 on Cloud TPU\" (Google)](https://cloud.google.com/tpu/docs/inception-v3-advanced)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification versus Regression versus Clustering\n",
    "\n",
    "### Classification\n",
    "\n",
    "Classification is a type of supervised learning in which the targets for a prediction are a set of categorical values.\n",
    "\n",
    "### Regression\n",
    "\n",
    "Regression is a type of supervised learning in which the targets for a prediction are quantitative or continuous values.\n",
    "\n",
    "### Clustering\n",
    "\n",
    "Clustering is a type of unsupervised learning where you want to identify similarities among collections of items without an *a prior* classification scheme. You may or may not have an *a priori* about the number of categories."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overfitting and Underfitting\n",
    "\n",
    "In machine learning models, we have to worry about twin concerns.  On the one hand, we might **overfit** our model to the dataset we have available.  If we train a model extremely accurately against the data itself, metrics we use for the quality of the model will probably show high values.  However, in this scenario, the model is unlikely to extend well to novel data, which is usually the entire point of developing a model and making predictions.  By training in a fine tuned way against one dataset, we might have done nothing more than memorize that collection of values; or at least memorize a spurious pattern that exists in that particular sample data collection.\n",
    "\n",
    "To some extent (but not completely), overfitting is mitigated by larger dataset sizes.\n",
    "\n",
    "In contrast, if we choose a model that simply does not have the degree of detail necessary to represent the underlying real-world phenomenon, we get an **underfit** model.  In this scenario, we *smooth too much* in our simplification of the data into a model.\n",
    "\n",
    "Some illustrations are useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.over_under_fit import doc, show"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above example is for a regression, but the same concept applies to categorization or clustering problems.  For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.over_under_fit import cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's look at a collection of points about which we have no *a priori* of their clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"Cluster\" everything into just one category\n",
    "cluster(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To the human eye, it would seem reasonable to guess that this represents three categories of observations. Therefore, we can reasonable say that this data is **underfit** by our clustering model.  Indeed, that would also be true if we guessed there were two clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Guess there might be two categories\n",
    "cluster(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model is not terrible, and it indeed seems to identify an important difference in the data.  But looking at the base-line known values for the categories, we can see it really is three types:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the \"known true\" categories\n",
    "cluster(1, known=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we cluster into three categories algorithmically, we almost (but not quite) recover the underlying truth.  The algorithms puts the categories in arbitrary order, so the colors are rotated; but you can seem that most-but-not-all the points are in the same clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moving farther along, if we guessed *more* clusters we would start to **overfit** the data, and impute category distinctions that do not exist in the underlying dataset.  In this case we known the true number because we have specifically generated it as such. In real-world data we usually do not know this in advance, so we can only tell by performing various validations on the strength of the fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Guess there might be 5 categories\n",
    "cluster(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Guess there might be 15 categories\n",
    "cluster(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimensionality Reduction\n",
    "\n",
    "Dimensionality reduction is most often a technique used to assist with other techniques. By reducing a large number of features to relatively few features; very often other techniques are more successful relative to these transformed synthetic features. Sometimes the dimensionality reduction itself is sufficient to identify the \"main gist\" or your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering\n",
    "\n",
    "Very often, the \"features\" we are given in our original data are not those that will prove most useful in our final analysis. It is often necessary to identify \"the data inside the data.\" Sometimes feature engineering can be as simple as normalizing the distribution of values. Other times it can involve creating synthetic features out of two or more raw features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Selection\n",
    "\n",
    "Often, the features you have in your raw data contain some features with little to no predictive or analytic value. Identifying and excluding irrelevant features often improves the quality of a model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorical versus Ordinal versus Continuous Variables\n",
    "\n",
    "Features come in one of three basic types.\n",
    "\n",
    "### Categorical variables \n",
    "\n",
    "Some are **categorical** (also called nominal): A discrete set of values that a feature may assume, often named by words or codes (but sometimes confusingly as integers where an order may be misleadingly implied).\n",
    "\n",
    "### Ordinal variables\n",
    "\n",
    "Some are **ordinal**: There is a scale from low to high in the data values, but the spacing in the data may have little to no relationship to the underlying phenomenon. For example, while an airline or credit card \"reward program\" might have levels of Gold/Silver/Platinum/Diamond, there is probably no real sense in which Diamond is \"4 times as much\" as Gold, even though they are encoded as 1-4.\n",
    "\n",
    "### Continuous variables\n",
    "\n",
    "Some are **continuous** or quantitative: Some quantity is actually measured such that a number represents the amount of it. The distribution of these measurements is likely not to be uniform and linear (in which case scaling might be relevant), but there is a real thing being measured. Measurements might be quantized for continuous variables, but that does not necessarily make them ordinal instead. For example, we might measure annual rainfall in each town only to the nearest inch, and hence have integers for that feature.\n",
    "\n",
    "This notion of types of variables applies to statistics broadly. Some other concepts are genuinely specific to machine learning.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One-hot Encoding\n",
    "\n",
    "For many machine learning algorithms, including neural networks, it is more useful to have a categorical feature with N possible values encoded as N features, each taking a binary value. Several tools, including a couple functions in scikit-learn will transform raw datasets into this format. Obviously, by encoding this way, dimensionality is increased.\n",
    "\n",
    "Let us illustrate using a toy test dataset.  The following whimsical data is suggested in a blog post by [Håkon Hapnes Strand](https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science).  Imagine we collected some data on individual organisms—namely taxonomic class, height, and lifespan.  Depending on our purpose, we might use this data for either supervised or unsupervised learning techniques (if we had a lot more observations, and a number more features)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data= [\n",
    "    ['human', 1.7, 85],\n",
    "    ['alien', 1.8, 92],\n",
    "    ['penguin', 1.2, 37],\n",
    "    ['octopus', 2.3, 25],\n",
    "    ['alien', 1.7, 85],\n",
    "    ['human', 1.2, 37],\n",
    "    ['octopus', 0.4, 8],\n",
    "    ['human', 2.0, 97]\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The data with its original feature, just as a DataFrame\n",
    "import pandas as pd\n",
    "naive = pd.DataFrame(data, columns=['species', 'height (M)', 'lifespan (years)'])\n",
    "naive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The data one-hot encoded\n",
    "encoded = pd.get_dummies(naive)\n",
    "encoded.columns = [c.replace('species_','') for c in encoded.columns]\n",
    "encoded"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters\n",
    "\n",
    "The notion of parameters was introduced to define the way in which a model was trained. For neural networks, parameters are the weights of all the connections between the neurons. But in other models a similar parameterization exists. For example, in a basic linear regression, the coefficients in each dimension are parameters of the trained/fitted model.\n",
    "\n",
    "However, many algorithms used in machine learning take \"hyperparameters\" that tune how the training itself occurs. These may be cut-off values where a \"good enough\" estimate is obtained, for example. Or there may be hidden terms in an underlying equation that can be set. Or an algorithm may actually be a family of closely related algorithms, and a hyperparameter chooses among them. Models in scikit-learn typically have a number of hyperparameters to set before they are trained (with \"sensible\" defaults when you do not specify)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grid Search\n",
    "\n",
    "While scikit-learn usually provides \"sensible\" defaults for hyperparameters, there is often a great deal of domain and dataset specificity for which hyperparameters are most effective. An API is provided to search across the combinatorial space of hyperparameter values and evaluate each collection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metrics\n",
    "\n",
    "After you have trained a model, the big question is \"how good\" is the model.  There is a lot of nuance to answering that question, and correspondingly a large number of measures and techniques.\n",
    "\n",
    "One common technique to look at a combination of successes and failure in a machine learning model is a *confusion matrix*.  Let us look at an example, picking up the whimsical data used above.  Suppose we wanted to guess the taxonomic class of an observed organism and our model had these results:\n",
    "\n",
    "| Predict/Actual | Human    | Octopus  | Penguin  |\n",
    "|----------------|----------|----------|----------|\n",
    "| Human          |  **5**   |    0     |    2     |\n",
    "| Octopus        |    3     |  **3**   |    3     |\n",
    "| Penguin        |    0     |    1     |  **11**  |\n",
    "\n",
    "Giving a single number to describe *how good* the model is is not immediately obvious.  The model is very good at predicting penguins, but it gets rather bad when it predicts octopi.  In fact, if the model predicts something is an octopus, it probably isn't (only 1/3rd of such predictions are accurate)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy versus Precision versus Recall\n",
    "\n",
    "Naïvely, we might simply ask about the \"accuracy\" of a model (at least for classification tasks).  This is simply the number of *right* answers divided by the number of data points.  In our example, we have 28 observations of organisms, and 19 were classified accurately, so that's a **68%** accuracy.  Again though, the accuracy varies quite a lot if we restrict it to just one class of the predictions.  For our multi-class labels, this may not be a bad measure.  \n",
    "\n",
    "Consider a binary problem though:\n",
    "\n",
    "| Predict/Actual | Positive | Negative |\n",
    "|----------------|----------|----------|\n",
    "| Positive       |    1     |    0     |\n",
    "| Negative       |    2     |   997    | \n",
    "\n",
    "Calculating *accuracy*, we find that this model is **99.8%** accurate! That seems pretty good until you think of this test as a medical screening for a fatal disease.  *Two thirds of the people who actually have the disease will be judged free of it by this model* (and hence perhaps not be treated for the condition); that isn't such a happy real-world result.\n",
    "\n",
    "<hr/>\n",
    "\n",
    "In contrast with accuracy, the \"precision\" of a model is defined as:\n",
    "\n",
    "$$\\text{Precision} = \\frac{true\\: positive}{true\\: positive + false\\: positive}$$\n",
    "\n",
    "Generalizing that to the multi-class case, the formula is as follows (for i being the index of the class):\n",
    "\n",
    "$$\\text{Precision}_{i} = \\cfrac{M_{ii}}{\\sum_i M_{ij}}$$\n",
    "\n",
    "Applying that to our hypothetical medical screening, we get a a precision of **1.0**.  We cannot do better than that.  The problem is with \"recall\" which is defined as:\n",
    "\n",
    "$$\\text{Recall} = \\frac{true\\: positive}{true\\: positive + false\\: negative}$$\n",
    "\n",
    "Generalizing that to the multi-class case:\n",
    "\n",
    "$$\\text{Recall}_{i} = \\cfrac{M_{ii}}{\\sum_j M_{ij}}$$\n",
    "\n",
    "Here we do much worse by having a recall of **33.3%** in our medical diagnosis case! This is obviously a terrible result if we care about recall."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### F1 Score\n",
    "\n",
    "There are several different algorithms that attempt to *blend* precision and recall to product a single \"score.\"  Scikit-learn provides a number of other scalar scores that are useful for differing purposes (and other libraries are similar), but F1 score is one that is used very frequently.  It is simply:\n",
    "\n",
    "$$\\text{F1} = 2 \\times \\cfrac{precision \\times recall}{precision + recall}$$\n",
    "\n",
    "Applying that to our medical diagnostic model, we get an F1 score of 50%.  Still not good, but we account for the high precision to some extent.  For intermediate cases, the F1 score provides good balance.\n",
    "\n",
    "F1 score can be generalized to multi-class models by averaging the F1 score across each class, counting only correct/incorrect per class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "import numpy as np\n",
    "\n",
    "y_true = [\"human\",   \"octopus\", \"human\", \"human\", \"octopus\", \"penguin\", \"penguin\"]\n",
    "y_pred = [\"octopus\", \"octopus\", \"human\", \"human\", \"octopus\", \"human\",   \"penguin\"]\n",
    "labels = ['octopus', 'penguin', 'human']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_true, y_pred, labels=labels)\n",
    "print(\"Confusion Matrix (predict/actual):\\n\", \n",
    "      pd.DataFrame(cm, index=labels, columns=labels), sep=\"\")\n",
    "\n",
    "recall = np.diag(cm) / np.sum(cm, axis=1)\n",
    "print(\"\\nRecall:\\n\", pd.Series(recall, index=labels), sep=\"\")\n",
    "\n",
    "precision = np.diag(cm) / np.sum(cm, axis=0)\n",
    "print(\"\\nPrecision:\\n\", pd.Series(precision, index=labels), sep=\"\")\n",
    "\n",
    "print(\"\\nAccuracy:\\n\", np.sum(np.diag(cm)) / np.sum(cm))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this particular case, F1 score is very close to accuracy.  In fact, using the \"micro\" averaging method reduces the result to accuracy.  Using the \"macro\" averaging makes it equivalent to a NumPy reduction from the formula given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "weighted_f1 = f1_score(y_true, y_pred, average=\"weighted\")\n",
    "print(\"\\nF1 score:\\n\", weighted_f1, sep=\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Naive averaging F1 score:\", np.mean(2*(recall*precision)/(recall+precision)))\n",
    "print(\" sklearn macro averaging:\", f1_score(y_true, y_pred, average=\"macro\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Lesson\n",
    "\n",
    "**Exploring a data set**: This lessson got us as far as understading some general concepts in machine learning, with an overview of most of the key ideas.  Next we will start working with a concrete dataset, clean it up and examine it, and being to use scikit-learn APIs.\n",
    "\n",
    "<a href=\"Exploring.ipynb\"><img src=\"img/open-notebook.png\" align=\"left\"/></a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}