Gradient_Descent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can estimate the coefficient values for our training data using stochastic gradient descent.\n",
    "\n",
    "Stochastic gradient descent requires two parameters:\n",
    "\n",
    "- Learning Rate: Used to limit the amount each coefficient is corrected each time it is updated.\n",
    "- Epochs: The number of times to run through the training data while updating the coefficients.\n",
    "\n",
    "These, along with the training data will be the arguments to the function.\n",
    "\n",
    "There are 3 loops we need to perform in the function:\n",
    "\n",
    "- Loop over each epoch.\n",
    "- Loop over each row in the training data for an epoch.\n",
    "- Loop over each coefficient and update it for a row in an epoch.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, we update each coefficient for each row in the training data, each epoch.\n",
    "\n",
    "Coefficients are updated based on the error the model made. The error is calculated as the difference between the prediction made with the candidate coefficients and the expected output value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "can't assign to function call (<ipython-input-1-5f888d65b8b8>, line 3)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-1-5f888d65b8b8>\"\u001b[0;36m, line \u001b[0;32m3\u001b[0m\n\u001b[0;31m    b1(t+1) = b1(t) - learning_rate * error(t) * x1(t)\u001b[0m\n\u001b[0m                                                      ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m can't assign to function call\n"
     ]
    }
   ],
   "source": [
    "error = prediction - expected\n",
    "\n",
    "b1(t+1) = b1(t) - learning_rate * error(t) * x1(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The special coefficient at the beginning of the list, also called the intercept or the bias, is updated in a similar way, except without an input as it is not associated with a specific input value:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "can't assign to function call (<ipython-input-2-e0f1eb18eb6b>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-2-e0f1eb18eb6b>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    b0(t+1) = b0(t) - learning_rate * error(t)\u001b[0m\n\u001b[0m                                              ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m can't assign to function call\n"
     ]
    }
   ],
   "source": [
    "b0(t+1) = b0(t) - learning_rate * error(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load data \n",
    "\n",
    "def format_data(df,test_size):\n",
    "   \n",
    "    # Targets are final grade of student\n",
    "    labels = df['y_pred']\n",
    "    # Drop the school and the grades from features\n",
    "    df = df.drop(columns=['asd_desc','state','y_pred'],axis = 1)\n",
    "    \n",
    "    # One-Hot Encoding of Categorical Variables\n",
    "    #df = pd.get_dummies(df)\n",
    "    \n",
    "    #df['y'] = list(labels)\n",
    "    #most_correlated = df.corr().abs()['y'].sort_values(ascending=False)\n",
    "    #print(most_correlated)\n",
    "    \n",
    "    # Keep correlations greater than 0.2 in absolute value\n",
    "    #most_correlated = most_correlated[most_correlated >= 0.2][1:]\n",
    "    \n",
    "    #df = df.ix[:, most_correlated.index]\n",
    "    #df = df.drop(columns = 'y')\n",
    "    \n",
    "    # Split into training/testing sets with 25% split\n",
    "    X_train, X_test, y_train, y_test = train_test_split(df, labels, \n",
    "                                                        test_size = test_size,\n",
    "                                                       random_state=42)\n",
    "    \n",
    "    return X_train, X_test, y_train, y_test\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Gradient Descent\n",
    "\n",
    "# Estimate linear regression coefficients using stochastic gradient descent\n",
    "def coefficients_sgd(train, l_rate, n_epoch):\n",
    "    coef = [0.0 for i in range(len(train[0]))]\n",
    "    for epoch in range(n_epoch):\n",
    "        sum_error = 0\n",
    "        for row in train:\n",
    "            yhat = predict(row, coef)\n",
    "            error = yhat - row[-1]\n",
    "            sum_error += error**2\n",
    "            coef[0] = coef[0] - l_rate * error\n",
    "            for i in range(len(row)-1):\n",
    "                #update coefficients\n",
    "                coef[i + 1] = coef[i + 1] - l_rate * error * row[i]\n",
    "        print('>epoch=%d, lrate=%.3f, error=%.3f' % (epoch, l_rate, sum_error))\n",
    "    return coef\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " Convert string column to float\n",
    "def str_column_to_float(dataset, column):\n",
    "    for row in dataset:\n",
    "\t\trow[column] = float(row[column].strip())\n",
    " \n",
    "# Find the min and max values for each column\n",
    "def dataset_minmax(dataset):\n",
    "    minmax = list()\n",
    "    for i in range(len(dataset[0])):\n",
    "        col_values = [row[i] for row in dataset]\n",
    "        value_min = min(col_values)\n",
    "        value_max = max(col_values)\n",
    "        minmax.append([value_min, value_max])\n",
    "    return minmax\n",
    " \n",
    "# Rescale dataset columns to the range 0-1\n",
    "def normalize_dataset(dataset, minmax):\n",
    "    for row in dataset:\n",
    "        for i in range(len(row)):\n",
    "            row[i] = (row[i] - minmax[i][0]) / (minmax[i][1] - minmax[i][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate coefficients\n",
    "\n",
    "l_rate = 0.001\n",
    "n_epoch = 50\n",
    "coef = coefficients_sgd(X, l_rate, n_epoch)\n",
    "print(coef)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'X_train_yield' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-a2c5462dca65>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;31m## Add a columns of 1s as intercept to X. This becomes the 2nd column\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mX_train_yield\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'intercept'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;31m## Transform to Numpy arrays for easier matrix math\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'X_train_yield' is not defined"
     ]
    }
   ],
   "source": [
    "iterations = 1500\n",
    "alpha = 0.01\n",
    "\n",
    "## Add a columns of 1s as intercept to X. This becomes the 2nd column\n",
    "X_train_yield['intercept'] = 1\n",
    "\n",
    "## Transform to Numpy arrays for easier matrix math\n",
    "## and start beta at 0, 0\n",
    "X = np.array(X_train_yield)\n",
    "y = np.array(y_train_yield).flatten()\n",
    "#initialize beta values at 0\n",
    "zeros = [0]*14\n",
    "beta = zeros\n",
    "def cost_function(X, y, beta):\n",
    "    \"\"\"\n",
    "    cost_function(X, y, beta) computes the cost of using beta as the\n",
    "    parameter for linear regression to fit the data points in X and y\n",
    "    \"\"\"\n",
    "    ## number of training examples\n",
    "    m = len(y)\n",
    "\n",
    "    ## Calculate the cost with the given parameters\n",
    "    J = np.sum((X.dot(beta)-y)**2)/2/m\n",
    "\n",
    "    return J\n",
    "\n",
    "cost_function(X, y, beta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(X, y, beta, alpha, iterations):\n",
    "    \"\"\"\n",
    "    gradient_descent() performs gradient descent to learn beta by\n",
    "    taking num_iters gradient steps with learning rate alpha\n",
    "    \"\"\"\n",
    "    cost_history = [0] * iterations\n",
    "\n",
    "    for iteration in range(iterations):\n",
    "        hypothesis = X.dot(beta)\n",
    "        loss = hypothesis-y\n",
    "        gradient = X.T.dot(loss)/m\n",
    "        beta = beta - alpha*gradient\n",
    "        cost = cost_function(X, y, beta)\n",
    "        cost_history[iteration] = cost\n",
    "\n",
    "        ## If you really want to merge everything in one line:\n",
    "        # beta = beta - alpha * (X.T.dot(X.dot(beta)-y)/m)\n",
    "        # cost = cost_function(X, y, beta)\n",
    "        # cost_history[iteration] = cost\n",
    "\n",
    "    return beta, cost_history\n",
    "\n",
    "(b, c) = gradient_descent(X, y, beta, alpha, iterations)\n",
    "\n",
    "print b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_ = [0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hi*14"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 2)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "a = np.array((1,2,3))\n",
    "b = np.array((2,3,4))\n",
    "np.hstack((a,b))\n",
    "a = np.array([[1],[2],[3]])\n",
    "b = np.array([[2],[3],[4]])\n",
    "np.hstack((a,b)).shape\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.column_stack"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}