2-3-3-ExploringConvolutions.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "GhZOHbwJOvio"
   },
   "source": [
    "##What are Convolutions?\n",
    "\n",
    "What are convolutions? In this lab you'll explore what they are and how they work. In later lessons, you'll see how to use them in your neural network."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "nidI4HtcVQ7i"
   },
   "source": [
    "Together with convolutions, you'll use something called 'Pooling', which compresses your image, further emphasising the features. You'll also see how pooling works in this lab. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "DdBFQswdO-kX"
   },
   "source": [
    "##Limitations of the previous DNN\n",
    "In an earlier exercise you saw how to train an image classifier for fashion items using the Fashion MNIST dataset. This gave you a pretty accuract classifier, but there was an obvious constraint: the images were 28x28, grey scale and the item was centered in the image. \n",
    "\n",
    "For example here are a couple of the images in Fashion MNIST\n",
    "![Picture of a sweater and a boot](https://cdn-images-1.medium.com/max/1600/1*FekMt6abfFFAFzhQcnjxZg.png)\n",
    "\n",
    "The DNN that you created simply learned from the raw pixels what made up a sweater, and what made up a boot in this context. But consider how it might classify this image?\n",
    "\n",
    "![image of boots](https://cdn.pixabay.com/photo/2013/09/12/19/57/boots-181744_1280.jpg)\n",
    "\n",
    "(Image is Public domain CC0 from Pixabay: https://pixabay.com/photos/boots-travel-railroad-tracks-181744/)\n",
    "\n",
    "While it's clear that there are boots in this image, the classifier would fail for a number of reasons. First, of course, it's not 28x28 greyscale, but more importantly, the classifier was trained on the raw pixels of a left-facing boot, and not the features that make up what a boot is.\n",
    "\n",
    "That's where Convolutions are very powerful. A convolution is a filter that passes over an image, processing it, and extracting features that show a commonolatity in the image. In this lab you'll see how they work, but processing an image to see if you can extract features from it!\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "ds0NF5KFVmG2"
   },
   "source": [
    "Generating convolutions is very simple -- you simply scan every pixel in the image and then look at it's neighboring pixels. You multiply out the values of these pixels by the equivalent weights in a filter. \n",
    "\n",
    "So, for example, consider this:\n",
    "\n",
    "![Convolution on image](https://storage.googleapis.com/learning-datasets/MLColabImages/lab3-fig1.png)\n",
    "\n",
    "In this case a 3x3 Convolution is specified.\n",
    "\n",
    "The current pixel value is 192, but you can calculate the new one by looking at the neighbor values, and multiplying them out by the values specified in the filter, and making the new pixel value the final amount.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "tJTHvE8Qe5nM"
   },
   "source": [
    "Let's explore how convolutions work by creating a basic convolution on a 2D Grey Scale image. First we can load the image by taking the 'ascent' image from scipy. It's a nice, built-in picture with lots of angles and lines. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "KTS2sc5nQSCJ"
   },
   "source": [
    "Let's start by importing some python libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DZ5OXYiolCUi"
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "from scipy import datasets\n",
    "i = datasets.ascent()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "SRIzxjWWfJjk"
   },
   "source": [
    "Next, we can use the pyplot library to draw the image so we know what it looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 248
    },
    "id": "R4p0cfWcfIvi",
    "outputId": "075bb08f-613d-4664-fc24-de3ab3d51076"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.grid(False)\n",
    "plt.gray()\n",
    "plt.axis('off')\n",
    "plt.imshow(i)\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "C1mhZ_ZTfPWH"
   },
   "source": [
    "We can see that this is an image of a stairwell. There are lots of features in here that we can play with seeing if we can isolate them -- for example there are strong vertical lines.\n",
    "\n",
    "The image is stored as a numpy array, so we can create the transformed image by just copying that array. Let's also get the dimensions of the image so we can loop over it later. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "o5pxGq1SmJMD"
   },
   "outputs": [],
   "source": [
    "i_transformed = np.copy(i)\n",
    "size_x = i_transformed.shape[0]\n",
    "size_y = i_transformed.shape[1]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "Y7PwNkiXfddd"
   },
   "source": [
    "Now we can create a filter as a 3x3 array. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sN3imZannN5J"
   },
   "outputs": [],
   "source": [
    "# This filter detects edges nicely\n",
    "# It creates a convolution that only passes through sharp edges and straight\n",
    "# lines.\n",
    "\n",
    "#Experiment with different values for fun effects.\n",
    "#filter = [ [0, 1, 0], [1, -4, 1], [0, 1, 0]]\n",
    "\n",
    "# A couple more filters to try for fun!\n",
    "filter = [ [-1, -2, -1], [0, 0, 0], [1, 2, 1]]\n",
    "#filter = [ [-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]\n",
    "\n",
    "# If all the digits in the filter don't add up to 0 or 1, you \n",
    "# should probably do a weight to get it to do so\n",
    "# so, for example, if your weights are 1,1,1 1,2,1 1,1,1\n",
    "# They add up to 10, so you would set a weight of .1 if you want to normalize them\n",
    "weight  = 1"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "JQmm_iBufmCz"
   },
   "source": [
    "Now let's create a convolution. We will iterate over the image, leaving a 1 pixel margin, and multiply out each of the neighbors of the current pixel by the value defined in the filter. \n",
    "\n",
    "i.e. the current pixel's neighbor above it and to the left will be multiplied by the top left item in the filter etc. etc. We'll then multiply the result by the weight, and then ensure the result is in the range 0-255\n",
    "\n",
    "Finally we'll load the new value into the transformed image. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "299uU2jAr90h"
   },
   "outputs": [],
   "source": [
    "for x in range(1,size_x-1):\n",
    "  for y in range(1,size_y-1):\n",
    "      convolution = 0.0\n",
    "      convolution = convolution + (i[x - 1, y-1] * filter[0][0])\n",
    "      convolution = convolution + (i[x, y-1] * filter[1][0])\n",
    "      convolution = convolution + (i[x + 1, y-1] * filter[2][0])\n",
    "      convolution = convolution + (i[x-1, y] * filter[0][1])\n",
    "      convolution = convolution + (i[x, y] * filter[1][1])\n",
    "      convolution = convolution + (i[x+1, y] * filter[2][1])\n",
    "      convolution = convolution + (i[x-1, y+1] * filter[0][2])\n",
    "      convolution = convolution + (i[x, y+1] * filter[1][2])\n",
    "      convolution = convolution + (i[x+1, y+1] * filter[2][2])\n",
    "      convolution = convolution * weight\n",
    "      if(convolution<0):\n",
    "        convolution=0\n",
    "      if(convolution>255):\n",
    "        convolution=255\n",
    "      i_transformed[x, y] = convolution"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "6XA--vgvgDEQ"
   },
   "source": [
    "Now we can plot the image to see the effect of the convolution!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "id": "7oPhUPNhuGWC",
    "outputId": "aa2f8498-015d-4cd9-81c7-64e7c54648d1"
   },
   "outputs": [],
   "source": [
    "# Plot the image. Note the size of the axes -- they are 512 by 512\n",
    "plt.gray()\n",
    "plt.grid(False)\n",
    "plt.imshow(i_transformed)\n",
    "#plt.axis('off')\n",
    "plt.show()   "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "Df7kw1m6XDwz"
   },
   "source": [
    "So, consider the following filter values, and their impact on the image.\n",
    "\n",
    "Using -1,0,1,-2,0,2,-1,0,1 gives us a very strong set of vertical lines:\n",
    "\n",
    "![Detecting vertical lines filter](https://storage.googleapis.com/learning-datasets/MLColabImages/lab3-fig2.png)\n",
    "\n",
    "Using -1, -2, -1, 0, 0, 0, 1, 2, 1 gives us horizontal lines:\n",
    "\n",
    "![Detecting horizontal lines](https://storage.googleapis.com/learning-datasets/MLColabImages/lab3-fig3.png)\n",
    "\n",
    "Explore different values for yourself! "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "xF0FPplsgHNh"
   },
   "source": [
    "## Pooling\n",
    "\n",
    "As well as using convolutions, pooling helps us greatly in detecting features. The goal is to reduce the overall amount of information in an image, while maintaining the features that are detected as present. \n",
    "\n",
    "There are a number of different types of pooling, but for this lab we'll use one called MAX pooling. \n",
    "\n",
    " The idea here is to iterate over the image, and look at the pixel and it's immediate neighbors to the right, beneath, and right-beneath. Take the largest (hence the name MAX pooling) of them and load it into the new image. Thus the new image will be 1/4 the size of the old -- with the dimensions on X and Y being halved by this process. You'll see that the features get maintained despite this compression!\n",
    "\n",
    "![Max Pooling](https://storage.googleapis.com/learning-datasets/MLColabImages/lab3-fig4.png)\n",
    "\n",
    "This code will show (4, 4) pooling. Run it to see the output, and you'll see that while the image is 1/4 the size of the original in both length and width, the extracted features are maintained!\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 268
    },
    "id": "kDHjf-ehaBqm",
    "outputId": "978efe66-a06e-4312-9be1-7d67236445c3"
   },
   "outputs": [],
   "source": [
    "new_x = int(size_x/4)\n",
    "new_y = int(size_y/4)\n",
    "newImage = np.zeros((new_x, new_y))\n",
    "for x in range(0, size_x, 4):\n",
    "  for y in range(0, size_y, 4):\n",
    "    pixels = []\n",
    "    pixels.append(i_transformed[x, y])\n",
    "    pixels.append(i_transformed[x+1, y])\n",
    "    pixels.append(i_transformed[x+2, y])\n",
    "    pixels.append(i_transformed[x+3, y])\n",
    "    pixels.append(i_transformed[x, y+1])\n",
    "    pixels.append(i_transformed[x+1, y+1])\n",
    "    pixels.append(i_transformed[x+2, y+1])\n",
    "    pixels.append(i_transformed[x+3, y+1])\n",
    "    pixels.append(i_transformed[x, y+2])\n",
    "    pixels.append(i_transformed[x+1, y+2])\n",
    "    pixels.append(i_transformed[x+2, y+2])\n",
    "    pixels.append(i_transformed[x+3, y+2])\n",
    "    pixels.append(i_transformed[x, y+3])\n",
    "    pixels.append(i_transformed[x+1, y+3])\n",
    "    pixels.append(i_transformed[x+2, y+3])\n",
    "    pixels.append(i_transformed[x+3, y+3])\n",
    "    pixels.sort(reverse=True)\n",
    "    newImage[int(x/4),int(y/4)] = pixels[0]\n",
    "\n",
    "# Plot the image. Note the size of the axes -- now 128 pixels instead of 512\n",
    "plt.gray()\n",
    "plt.grid(False)\n",
    "plt.imshow(newImage)\n",
    "#plt.axis('off')\n",
    "plt.show()      \n",
    "    \n",
    "    "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "jZWdU6dVYQm-"
   },
   "source": [
    "In the next lab you'll see how to add convolutions to your Fashion MNIST neural network to make it more efficient -- because it will classify based on features, and not on raw pixels. "
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Exploring Convolutions.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}