binary_logistic_regression.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "binary_logistic_regression",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMXm+POOWnt4AJBOp4Oi8r5",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/rares985/machine-learning/blob/master/binary_logistic_regression.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uyc_YVVRh_2d",
        "colab_type": "code",
        "outputId": "dfd7b264-bd79-4984-f066-94be03fe769f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data(\"mnist.mpz\")"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "930uedBjxg7V",
        "colab_type": "text"
      },
      "source": [
        "Since we are implementing binary classification, we are only keeping the two classes for digits $ 0 $ and $ 1 $. Also, we are adding the intercept term as usual:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "79fly1Q0jsgL",
        "colab_type": "code",
        "outputId": "193b2c83-6e10-4d3b-f8b2-0517e2914bcd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "X_train_binary = X_train[(y_train==0)|(y_train==1)]\n",
        "y_train_binary = y_train[(y_train==0)|(y_train==1)].reshape(-1,1)\n",
        "\n",
        "X_test_binary = X_test[(y_test==0)|(y_test==1)]\n",
        "y_test_binary = y_test[(y_test==0)|(y_test==1)].reshape(-1, 1)\n",
        "\n",
        "X_train_binary = X_train_binary.reshape(-1, 28*28)\n",
        "X_test_binary = X_test_binary.reshape(-1, 28*28)\n",
        "m,n = X_train_binary.shape\n",
        "\n",
        "X_intercept = np.hstack((np.ones((m, 1)), X_train_binary))\n",
        "print(X_intercept.shape)\n"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(12665, 785)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n6AoQRJTxsYV",
        "colab_type": "text"
      },
      "source": [
        "Here are the definitions for the *crossentropy* loss function and for the *sigmoid* activation function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kj9GYt03nWt6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def cross_entropy(predictions, targets, epsilon=1e-12):\n",
        "    \"\"\"\n",
        "    Computes cross entropy between targets (encoded as one-hot vectors)\n",
        "    and predictions. \n",
        "    Input: predictions (N, k) ndarray\n",
        "           targets (N, k) ndarray        \n",
        "    Returns: scalar\n",
        "    \"\"\"\n",
        "    predictions = np.clip(predictions, epsilon, 1. - epsilon)\n",
        "    N = predictions.shape[0]\n",
        "    ce = -np.sum(targets*np.log(predictions+1e-9))/N\n",
        "    return ce\n",
        "\n",
        "def sigmoid(x):                                        \n",
        "    return 1 / (1 + np.exp(-x))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nsERkn0Gx0NF",
        "colab_type": "text"
      },
      "source": [
        "And now the actual training: Note that the formula is almost identical to linear regression, with the exception that y_hat is now the result of the application of the sigmoid on the product rather than the product itself. We are also plotting the cost function evolution to see whether the regressor is converging."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SAggqnU2llV_",
        "colab_type": "code",
        "outputId": "e342217a-c559-4b92-b0c6-c3ae0c37dac9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        }
      },
      "source": [
        "import tqdm\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "n_epochs = 25\n",
        "learning_rate = 1e-5\n",
        "\n",
        "theta = np.random.randn(n+1, 1)*np.sqrt(2/m)\n",
        "\n",
        "\n",
        "J_hist = []\n",
        "for n_ep in tqdm.tqdm(range(n_epochs)):\n",
        "    y_hat = sigmoid(X_intercept @ theta)\n",
        "\n",
        "    delta = y_hat - y_train_binary\n",
        "    \n",
        "    J = cross_entropy(y_hat, y_train_binary)\n",
        "    J_hist.append(J)\n",
        "\n",
        "    djdw = (1/m) * X_intercept.T @ delta\n",
        "\n",
        "    theta -= learning_rate * djdw\n",
        "\n",
        "\n",
        "plt.plot(J_hist)"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 25/25 [00:00<00:00, 33.15it/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f38807630b8>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD5CAYAAAAtBi5vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAf6UlEQVR4nO3dd3ydZf3/8dcneydNM5qmbdK9dyiFFlqWULCUIjJUBAQZ4vcHOPgKKgh+EUQFBRQsMpUhIqMqCIhlj27obtPddCRNmr2T6/dHjrWUpknak9xnvJ+Px3nk5Jw7J+/7ceDdO9e57us25xwiIhJ6IrwOICIi3UMFLyISolTwIiIhSgUvIhKiVPAiIiFKBS8iEqKiOtrAzPoDTwLZgAPmOed+c9A2M4GXgc2+h15wzt1+uNfNyMhw+fn5RxBZRCR8LVmyZK9zLrMz23ZY8EAz8F3n3FIzSwaWmNkbzrnVB233rnPui50NmZ+fz+LFizu7uYiIAGa2tbPbdjhE45zb5Zxb6rtfBawBco88noiI9IQujcGbWT4wEfj4EE8fZ2afmNmrZjbaD9lEROQodGaIBgAzSwL+ClzvnKs86OmlQJ5zrtrMzgReAoYe4jWuBK4EGDBgwBGHFhGRjnXqCN7Momkr96eccy8c/LxzrtI5V+27/woQbWYZh9hunnOuwDlXkJnZqc8IRETkCHVY8GZmwCPAGufcPe1s08e3HWY2xfe6pf4MKiIiXdOZIZppwMXACjNb7nvsZmAAgHPuIeA84BozawbqgAudlqkUEfFUhwXvnHsPsA62eQB4wF+hRETk6AXdmaxbS2v45WvrWLSljOaWVq/jiIgErE7PogkUn+yo4MG3N/LAgkKS46I4YWgGM4ZlcuKwTHJS472OJyISMIKu4M8e35cZwzJ5v3Avb68r4e31JbyyYjcAI/okM2NYJjOGZTI5vxexUZEepxUR8Y559VloQUGB88dSBc451u+p5q11xby9voRFW8poanEkxERy/OAMZgzPZOawTPqnJ/ghtYiIt8xsiXOuoFPbBnvBH6ymoZkPN5by9voS3lpfzPayOgBOHJbJI5cUEB0ZdB87iIjs15WCD7ohmo4kxkZx6qhsTh2VjXOOzXtreHn5Tn7z5gZ+9fp6fjBrhNcRRUR6RMgV/IHMjEGZSdxw2jCKqxp46O2NHDe4NzOG6SxaEQl9YTNecevsUQzPTuY7f15OcWW913FERLpd2BR8XHQkD3xlIjWNzVz/5+W0tOpEWxEJbWFT8ABDs5O57ezRfLCxlN8tKPQ6johItwqrggc4v6A/Z4/vy73/Ws/CzWVexxER6TZhV/Bmxh1zx9A/PYHrnl3GvppGryOJiHSLsCt4gOS4aB64aBJ7qxv4/vOfooUvRSQUhWXBA4ztl8pNs0byrzV7eOz9LV7HERHxu7AteIDLpuVz6sgs7nx1DSt2VHgdR0TEr8K64M2MX5w3noykWL79zFKq6pu8jiQi4jdhXfAAvRJj+M2FE9leVsuPXlqp8XgRCRlhX/AAUwamc8Opw3h5+U7+sniH13FERPxCBe/zrZOGcPzg3twyfyUb9lR5HUdE5Kip4H0iI4xfXzCBxJgovv30MuqbWryOJCJyVFTwB8hKieNX549n3Z4qbv/7aq/jiIgcFRX8QWYOz+KqGYN4+uNtvLJil9dxRESOmAr+EL73heGMzU3llpdXUVGrqZMiEpxU8IcQHRnBneeOpaymgZ+/ttbrOCIiR0QF344xual8Y9pAnv54G0u2atVJEQk+KvjDuOG0YfRNjePmF1bS1NLqdRwRkS5RwR9GYmwUt88Zw7o9Vcx7Z5PXcUREukQF34FTR2Vzxug+3PfmBraW1ngdR0Sk01TwnfCTs0cTHRmhtWpEJKio4DuhT2oc3z99OO9u2Mv8T3Z6HUdEpFNU8J30tal5jO+Xyk//vlpz40UkKKjgOykywvjZuWPZV9vEXf9c43UcEZEOqeC7YHTfVL4xLZ9nFm5n0RbNjReRwKaC76IbThtGblo8N7+wgsZmzY0XkcClgu+ihJgobp8zmg3F1Tz8rubGi0jgUsEfgVNGZnPm2La58Vv2am68iAQmFfwRunW25saLSGDrsODNrL+ZLTCz1Wa2ysyuO8Q2Zmb3mVmhmX1qZpO6J27gyE6J48YzhvNe4V5eXq658SISeDpzBN8MfNc5NwqYClxrZqMO2mYWMNR3uxJ40K8pA9RXj81jQv80fvr31ZTXNnodR0TkMzoseOfcLufcUt/9KmANkHvQZnOAJ12bj4A0M8vxe9oAExlh/GzuWMrrmrjrVa0bLyKBpUtj8GaWD0wEPj7oqVxg+wHf7+Dz/wiEpFF9U7hi+kCeXbSdhZs1N15EAkenC97MkoC/Atc75yqP5JeZ2ZVmttjMFpeUlBzJSwSk604d2jY3/sUVNDS3eB1HRAToZMGbWTRt5f6Uc+6FQ2xSBPQ/4Pt+vsc+wzk3zzlX4JwryMzMPJK8ASkhJor/O2cMhcXV/OHdzV7HEREBOjeLxoBHgDXOuXva2Ww+8HXfbJqpQIVzbpcfcwa8k0Zk7V83fntZrddxREQ6dQQ/DbgYONnMlvtuZ5rZ1WZ2tW+bV4BNQCHwMPCt7okb2G6ZPYrICOPW+as0N15EPBfV0QbOufcA62AbB1zrr1DBqm9aPDecOow7XlnD66v3cProPl5HEpEwpjNZ/ezSafmM6JPMbfNXUdPQ7HUcEQljKng/i46M4P/OGcPOinrue3OD13FEJIyp4LtBQX465xf045H3NrNud5XXcUQkTKngu8kPZo0kKS6KH720Qh+4iognVPDdJD0xhptmjWDRln08v2SH13FEJAyp4LvRlyf3Z9KANO58dS37arQYmYj0LBV8N4qIMO6YO5aKuibufk2LkYlIz1LBd7OROSlcdnzbhbqXbtvndRwRCSMq+B5w/WnD6JMSxw9fXElziy7ULSI9QwXfA5Jio7hl9ijW7KrkiQ+3eh1HRMKECr6HzBrThxnDMrnn9XXsrqj3Oo6IhAEVfA8xM26fM5rmVsdP/7Ha6zgiEgZU8D0or3ci1540hH98uot31ofOBU9EJDCp4HvYVTMGMTAjkVteXkl9k67+JCLdRwXfw2KjIvnpnDFsKa3lwbc2eh1HREKYCt4D04dmMHt8Xx58eyOb99Z4HUdEQpQK3iM/PmskMZER3PLySi1GJiLdQgXvkayUOL5/+nDe3bCXl5Z/7vrkIiJHTQXvoa9NzWPigDRu/9tqSqsbvI4jIiFGBe+hyAjj518aR3VDMz/9u+bGi4h/qeA9Niw7mWtmDuGl5TtZsK7Y6zgiEkJU8AHg2pMGMzgzkR+9uFIX6hYRv1HBB4DYqEh+/qVxFJXX8avX13sdR0RChAo+QBTkp3Px1Dwe+2Azy7RuvIj4gQo+gNx4xnCyk+O46YUVNDZr3XgROToq+ACSHBfNT88Zw9rdVcx7R8sYiMjRUcEHmNNGZXPWuBzue7OQwuJqr+OISBBTwQegW2ePIi46gptfWEFrq5YxEJEjo4IPQFnJcfzorFEs3FLGM4u2eR1HRIKUCj5AfbmgH8cP7s1dr6zVJf5E5Iio4AOUmfGzuWNpbGnl1vkrvY4jIkFIBR/A8jMSueG0Yby2ag//XLnL6zgiEmRU8AHuiukDGd03hR+/vIqKuiav44hIEFHBB7ioyAh+/qVxlNU0ctera7yOIyJBRAUfBMbkpnLF9IE8s3A7H24s9TqOiAQJFXyQuP7UYQxIT+DmF1dQ39TidRwRCQIdFryZPWpmxWZ2yKkcZjbTzCrMbLnvdov/Y0p8TCR3njuWzXtruPdfWnFSRDrWmSP4x4EzOtjmXefcBN/t9qOPJYcybUgGF03pz7x3NrFwc5nXcUQkwHVY8M65dwC1SYD40Vmj6N8rge88t5yqes2qEZH2+WsM/jgz+8TMXjWz0X56TTmExNgo7r1gAjvL67jtb7qOq4i0zx8FvxTIc86NB+4HXmpvQzO70swWm9nikpISP/zq8DQ5rxfXnjSE55fs0AlQItKuoy5451ylc67ad/8VINrMMtrZdp5zrsA5V5CZmXm0vzqs/b9ThjI2N5WbXlhBcZXWqhGRzzvqgjezPmZmvvtTfK+pydrdLDoygnsvmEBtYws3Pv8pzmlZYRH5rM5Mk3wG+BAYbmY7zOxyM7vazK72bXIesNLMPgHuAy50apseMSQriZvPHMlb60p46mMtKywinxXV0QbOuYs6eP4B4AG/JZIuuXhqHv9as4c7/rGG4wf3ZlBmkteRRCRA6EzWIBcRYfzyy+OJiYrghj8vp6lFF+sWkTYq+BCQnRLHz+aO5ZMdFfx2QaHXcUQkQKjgQ8RZ43I4d2Iu9/+7kGXb9nkdR0QCgAo+hPxkzmj6pMTxnec+obax2es4IuIxFXwISYmL5lfnj2dLaQ0/e0Vrx4uEOxV8iJk6qDffPGEQf/poGwvWFnsdR0Q8pIIPQd/9wjBG9Enm+89/SllNo9dxRMQjKvgQFBsVyb0XTKCyrombXtBZriLhSgUfokbmpPC904fx2qo9PL9kh9dxRMQDKvgQdsX0QUwdlM5tf1vN1tIar+OISA9TwYewiAjjV+dPIDLCuOZPS3UtV5Ewo4IPcblp8fz6ggms2V3Jj19aqfF4kTCigg8DJ43I4n9OHspfluzgz4u2ex1HRHqICj5MXHfKUE4YmsEt81exYkeF13FEpAeo4MNEZITxmwsnkpkUyzVPLaG8VvPjRUKdCj6MpCfG8LuvTqK4soHr/7yc1laNx4uEMhV8mBnfP41bZo/irXUlPKClhUVCmgo+DH312AGcOzGXe/+1nrfXl3gdR0S6iQo+DJkZd8wdy/DsZK57dhk79tV6HUlEuoEKPkzFx0Ty4Ncm09LiuPappTQ06yQokVCjgg9jAzMS+eX54/lkRwW3/22113FExM9U8GHu9NF9uGrGIJ76eBt/1aJkIiFFBS98/wvDmToonR++tII1uyq9jiMifqKCF6IiI7j/okmkxEVzzZ+WUFHX5HUkEfEDFbwAkJkcy+++Ookd++r43l8+0aJkIiFABS/7FeSnc9OZI3lj9R4eenuT13FE5Cip4OUzvjEtny+Oy+Hu19byz5W7vY4jIkdBBS+fYWb84rzxjO+XxnXPLmPJ1n1eRxKRI6SCl8+Jj4nkkUsKyEmN44onFrGppNrrSCJyBFTwcki9k2J5/LIpmBmXPraIvdUNXkcSkS5SwUu78jMSeeSSAoqr6rn88UXUNjZ7HUlEukAFL4c1cUAv7r9oEiuKKvifp5fR3NLqdSQR6SQVvHTotFHZ3DZnDG+uLeaW+as0R14kSER5HUCCw8VT8yjaV8dDb28kNy2ea08a4nUkEemACl467cbTh7Oroo5fvLaOvmlxzJ3Yz+tIInIYKnjptIgI4+7zxlFc2cCNz39KVnIc04ZkeB1LRNqhMXjpktioSB66eDIDMxK5+o9LtPqkSADrsODN7FEzKzazle08b2Z2n5kVmtmnZjbJ/zElkKTGR/P4ZVNIiI3ksscWsauizutIInIInTmCfxw44zDPzwKG+m5XAg8efSwJdH3T4nn8silUNzRz6aOLqKzXEsMigabDgnfOvQOUHWaTOcCTrs1HQJqZ5fgroASukTkp/P7iyWwsqeaqJ5fQ2Kw58iKBxB9j8LnA9gO+3+F7TMLAtCEZ3H3eOD7cVMp1zy6jSSdCiQSMHv2Q1cyuNLPFZra4pKSkJ3+1dKNzJ/XjR2eN5NWVu/nWU0tpaG7xOpKI4J+CLwL6H/B9P99jn+Ocm+ecK3DOFWRmZvrhV0uguOKEQdx29mjeWL2Hq/+4hPomlbyI1/xR8POBr/tm00wFKpxzu/zwuhJkLjk+n5/NHctb60u44onF1DWq5EW81OGJTmb2DDATyDCzHcCtQDSAc+4h4BXgTKAQqAUu666wEvi+cuwAYqIiuPH5T7j0sYU8eukxJMbqfDoRL3T4f55z7qIOnnfAtX5LJEHvvMn9iI40vvPcJ3z90YU8dtkxpMRFex1LJOzoTFbpFnMm5PLARRP5ZHs5F//hYypqNU9epKep4KXbzBqbw4Nfm8yaXVV85Q8fUVbT6HUkkbCigpduddqobOZ9fTKFxdV85eGPdOk/kR6kgpduN3N4Fo9eegxbSmu4cN5HFFfWex1JJCyo4KVHTBuSwROXTWFXeR3n//5DdpZrgTKR7qaClx5z7KDePHn5sZRWN3LBvA/ZXlbrdSSRkKaClx41Oa8XT33zWCrrmrng9x9SWFzldSSRkKWClx43rl8aT3/zWBpbHOf89gP+vXaP15FEQpIKXjwxum8q8789jfyMBC5/YjEPvrWRtnPmRMRfVPDimb5p8fzlquP54ri+/Pyfa7nu2eVav0bEj7RIiHgqPiaS+y6cwMicZH7x2jo2761h3tcnk5Ma73U0kaCnI3jxnJnxrZlD+MPXC9i8t4bZ97/Pkq2Hu4iYiHSGCl4Cxikjs3nxW8eTFBvJRfM+5rlF2zv+IRFplwpeAsrQ7GReunYaxw5K58a/fsptf1tFsy4DKHJEVPAScNISYnjs0mO4fPpAHnt/C5c+tojyWi1UJtJVKngJSFGREfz4i6P4xXnjWLi5jDm/fZ8Ne3RSlEhXqOAloH25oD/PXDmV2sYW5v7uA95YrZOiRDpLBS8Bb3JeL+Z/exoDMxL55pOL+eGLK6hpaPY6lkjAU8FLUMhJjecvVx/HlScO4umF2zjjN+/w0aZSr2OJBDQVvASNuOhIbj5zJM9ddRwRZlw47yNu+9sqnf0q0g4VvASdY/LTefW6E7jkuDwee38LZ933Lku27vM6lkjAUcFLUEqIieK2OWN4+opjaWhu5csPfcBdr66loVlH8yL/oYKXoHb8kAz+ef0JnF/Qn4fe3sjs+99jxY4Kr2OJBAQVvAS95Lho7vrSOB677Bgq6po453fvc88b62ls1hmwEt5U8BIyThqexevXz2DO+L7c9+YGzvnt+6zdXel1LBHPqOAlpKQmRHPPBRP4/cWTKa6qZ/b973HPG+s1b17CkgpeQtLpo/vw+g0zmDUmh/ve3MDMX77FMwu3aeEyCSsqeAlZ6Ykx3HfRRP56zfEMSE/gphdWcOZ977JgbbEuDyhhQQUvIW9yXi+ev/o4HvzqJBqbW7ns8UV87ZGPWVmk2TYS2lTwEhbMjFljc3j9hhn8ZPYoVu+sZPYD7/GdPy9nZ3md1/FEuoV59adqQUGBW7x4sSe/W6SirokH39rIo+9vxoBvTB/INTMHkxIX7XU0kcMysyXOuYLObKsjeAlLqfHR/GDWCP793RmcOTaHB9/ayMxfvMUTH2yhSR/ESojQEbwIsLKogjv+sYYPN5UyMCORa2YOZs6EvsRGRXodTeQzunIEr4IX8XHOsWBdMXf/cx1rd1eRmRzLJcfl8dVj8+iVGON1PBFABS9yVJxzvF9YysPvbuLt9SXERUdw3uR+XD59EAMzEr2OJ2GuKwUf1d1hRIKNmTF9aAbTh2awfk8Vj7y7mecW7eCpj7dxyohsvnnCQKYMTMfMvI4qclg6ghfphJKqBv744Rb++NFW9tU2Ma5fKpdPH8iZY3OIjtRcBek5fp9FY2ZnmNk6Mys0sx8c4vlLzazEzJb7bld0NbRIIMtMjuU7XxjOBz84hTvmjqG6vpnrnl3OjLsX8PA7m6isb/I6osjndHgEb2aRwHrgNGAHsAi4yDm3+oBtLgUKnHPf7uwv1hG8BLPW1rYPZB9+dxMfbSojISaS00f3Ye7EXKYNySAyQsM30j38PQY/BSh0zm3yvfizwBxg9WF/SiSERUQYp4zM5pSR2awsquCpj7fxj0938uKyIrKSYzl7fF/mTsplVE6KxurFM505gj8POMM5d4Xv+4uBYw88Wvcdwd8JlNB2tH+Dc277IV7rSuBKgAEDBkzeunWrn3ZDxHv1TS0sWFvMC8uKeGtdMU0tjuHZycydlMucCX3JSY33OqKEAL9Ok+xkwfcGqp1zDWZ2FXCBc+7kw72uhmgklO2raeTvK3bx4tIdLN1WjhkcN6g3cyfmMmtsDkmxmsAmR8bfBX8c8BPn3Om+728CcM7d2c72kUCZcy71cK+rgpdwsWVvDS8uK+Kl5UVsLa0lLjqC00b1Yfa4HKYNySBRZS9d4O+Cj6Jt2OUUoIi2D1m/4pxbdcA2Oc65Xb77c4H/dc5NPdzrquAl3DjnWLqtnBeX7eDvn+6ivLaJmMgIpg7uzcnDMzl5RDYDeid4HVMCnN/PZDWzM4FfA5HAo865O8zsdmCxc26+md0JnA00A2XANc65tYd7TRW8hLPG5lYWbynj32uL+fe6YjaV1AAwJCuJk0dkcdLwLArye2mOvXyOlioQCTJb9tbw77XFLFhXzEebSmlqcSTHRXHisExOHp7FzOGZ9E6K9TqmBAAVvEgQq25o5r0Ne1ngO7ovqWrADMb3S2PakN4ck5/O5LxeJGvt+rCkghcJEa2tjlU7K/cf3a8oqqCl1RFhMKpvClPyezNlYDrH5PfSEX6YUMGLhKiahmaWbStn4ZYyFm4uZdm2chqa2y5QMiQriWPy0zl2YDrHDEwnN03z7kORCl4kTDQ0t7CyqIKFm/excHMpi7fuo6q+GYDctHiOye/FmNxUxuamMqpvioZ1QoAKXiRMtbQ61u6uZNHmMhZuKWPJ1n3sqWzY//ygjERG56YyNjeFMX1TGZ2bSmq8Sj+YqOBFZL/iqnpWFVWysqiCFUUVrNpZSVF53f7n83on+Mo+hbG5qYzMSaF3YozW0AlQuuCHiOyXlRxH1og4ThqRtf+x0uoGVu2s9BV+BZ8WlfOPFbv2P98rIZqhWckMzkpiaFYSQ7OTGJKVRJ+UOBV/EFHBi4Sh3kmxnDgskxOHZe5/rKK2iZU7K1i7u4rC4moKi6t4deUunqn971r3SbFR/y39rLbSH5KVRG5aPFE6KSvgaIhGRNrlnKO0ppENe9oKv7C4mg3F1RQWV1Nc9d+x/agIo1+veAb0TiQvPYG83gn0930dkJ5AQoyOJf1FQzQi4hdmRkZSLBlJsRw3uPdnnquobaKwpK30t5TWsq20lm1ltSzfto9K30ye/8hMjmVAegJ56QkM8JV+37R4+qbGk50aS2xUZE/uVthQwYvIEUlNiGZyXjqT89I/91x5bSPbymrZ6iv9raU1bCur5aNNpby4vIiDBw4ykmLJTYsjJzWenLQ4ctPi99/vmxpPZnKsrpJ1BFTwIuJ3aQkxpCXEMK5f2ueeq29qoai8jl3l9eys8H0tr2NnRR2FJdW8s6GE2saWz/xMVISRlRxLZkocWcmxvlscWSmfvd87MUafBRxABS8iPSouOpLBmUkMzkw65PPOOSrrmtvKv6KOovJ6dpXXsbuynpKqBraV1rJ4Sxn7aj9/ofMIg/REX+mntA0t9U6KISOx7Wt6Ysz+x9ITY0J+aEgFLyIBxcxITYgmNSGakTkp7W7X2NxKSXUDxZX1FFc1UFzVQMkB94ur6lm3u4rS6kYaW1oP+RrJsVH7y753UttfAGkJMfRKiKZXQgxpCdH0Smz7Pi0hhrT46KD6C0EFLyJBKSYqgty0+A7X3HHOUd3QTGl1I6U1jZRWN1Ba00hZTSN7qxsorW67v72sluXbyymvbaSppf3ZhclxUfRKOKD0E6JJjW+7pcT5vsb/97HUhGhS4qJIio3q8XMIVPAiEtLMjOS4aJLjosnPSOxwe+ccNY0t7KtppLy2iX21jeyr/e/9/z7W9nXz3hoq65uorGui9TCzziMjjJS4KFLjo/na1DyuOGGQH/fy0FTwIiIHMDOSYtuOuPt/foJQu1pbHdWNzVTUNlFR17S/9CsOulXWNZOZ3DNLO6vgRUT8ICLCSIlrG6bp73UYn+D5tEBERLpEBS8iEqJU8CIiIUoFLyISolTwIiIhSgUvIhKiVPAiIiFKBS8iEqI8u6KTmZUAW4/wxzOAvX6ME2zCef/Ded8hvPdf+94mzzmXebiN/8Ozgj8aZra4s5esCkXhvP/hvO8Q3vuvfe/6vmuIRkQkRKngRURCVLAW/DyvA3gsnPc/nPcdwnv/te9dFJRj8CIi0rFgPYIXEZEOBF3Bm9kZZrbOzArN7Ade5+lJZrbFzFaY2XIzW+x1nu5mZo+aWbGZrTzgsXQze8PMNvi+9vIyY3dpZ99/YmZFvvd/uZmd6WXG7mJm/c1sgZmtNrNVZnad7/Fwee/b2/8uv/9BNURjZpHAeuA0YAewCLjIObfa02A9xMy2AAXOubCYC2xmJwLVwJPOuTG+x+4Gypxzd/n+ge/lnPtfL3N2h3b2/SdAtXPul15m625mlgPkOOeWmlkysAQ4B7iU8Hjv29v/8+ni+x9sR/BTgELn3CbnXCPwLDDH40zSTZxz7wBlBz08B3jCd/8J2v7DDznt7HtYcM7tcs4t9d2vAtYAuYTPe9/e/ndZsBV8LrD9gO93cIQ7HqQc8LqZLTGzK70O45Fs59wu3/3dQLaXYTzwbTP71DeEE5JDFAcys3xgIvAxYfjeH7T/0MX3P9gKPtxNd85NAmYB1/r+jA9brm18MXjGGI/eg8BgYAKwC/iVt3G6l5klAX8FrnfOVR74XDi894fY/y6//8FW8EXwmevZ9vM9Fhacc0W+r8XAi7QNWYWbPb4xyv+MVRZ7nKfHOOf2OOdanHOtwMOE8PtvZtG0ldtTzrkXfA+HzXt/qP0/kvc/2Ap+ETDUzAaaWQxwITDf40w9wswSfR+4YGaJwBeAlYf/qZA0H7jEd/8S4GUPs/So/5Sbz1xC9P03MwMeAdY45+454KmweO/b2/8jef+DahYNgG9q0K+BSOBR59wdHkfqEWY2iLajdoAo4OlQ33czewaYSdtKenuAW4GXgOeAAbStRnq+cy7kPoxsZ99n0vbnuQO2AFcdMCYdMsxsOvAusAJo9T18M23j0OHw3re3/xfRxfc/6ApeREQ6J9iGaEREpJNU8CIiIUoFLyISolTwIiIhSgUvIhKiVPAiIiFKBS8iEqJU8CIiIer/A1RRUgF/iPDRAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QxH3BiY2yFFQ",
        "colab_type": "text"
      },
      "source": [
        "And now some predictions: Accuracy seems to be over 90 \\%."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "82ST0J-il_7T",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "00ebf443-bca4-4bd6-a4bd-4fc4a0d855f7"
      },
      "source": [
        "X_test_intercept = np.hstack((np.ones((X_test_binary.shape[0], 1)), X_test_binary))\n",
        "y_pred_binary = sigmoid(X_test_intercept @ theta)\n",
        "\n",
        "y_pred_labels = np.zeros_like(y_pred_binary)\n",
        "y_pred_labels[y_pred_binary >= 0.5] = 1\n",
        "\n",
        "\n",
        "\n",
        "import pandas as pd\n",
        "df = pd.DataFrame(data={'Predicted': y_pred_labels.reshape(-1, ).astype(np.int32), 'Actual': y_test_binary.reshape(-1,)}).head()\n",
        "\n",
        "guessed = np.sum(y_pred_labels == y_test_binary)\n",
        "total = y_test_binary.shape[0]\n",
        "\n",
        "acc = (guessed / total) * 100\n",
        "\n",
        "print('Accuracy is: {}'.format(acc))"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy is: 94.60992907801419\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Luc4XxKunRv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}