gradDescent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"toc_15456_5\"></a>\n",
    "# 5 Gradient Descent With Multiple Variables\n",
    "Gradient descent for multiple variables:\n",
    "\n",
    "$$\\begin{align*} \\text{repeat}&\\text{ until convergence:} \\; \\lbrace \\newline\\;\n",
    "& w_j = w_j -  \\alpha \\frac{\\partial J(\\mathbf{w},b)}{\\partial w_j} \\tag{5}  \\; & \\text{for j = 0..n-1}\\newline\n",
    "&b\\ \\ = b -  \\alpha \\frac{\\partial J(\\mathbf{w},b)}{\\partial b}  \\newline \\rbrace\n",
    "\\end{align*}$$\n",
    "\n",
    "where, n is the number of features, parameters $w_j$,  $b$, are updated simultaneously and where  \n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial J(\\mathbf{w},b)}{\\partial w_j}  &= \\frac{1}{m} \\sum\\limits_{i = 0}^{m-1} (f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) - y^{(i)})x_{j}^{(i)} \\tag{6}  \\\\\n",
    "\\frac{\\partial J(\\mathbf{w},b)}{\\partial b}  &= \\frac{1}{m} \\sum\\limits_{i = 0}^{m-1} (f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) - y^{(i)}) \\tag{7}\n",
    "\\end{align}\n",
    "$$\n",
    "* m is the number of training examples in the data set\n",
    "\n",
    "    \n",
    "*  $f_{\\mathbf{w},b}(\\mathbf{x}^{(i)})$ is the model's prediction, while $y^{(i)}$ is the target value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem Dataset\n",
    "\n",
    "| Size (sqft) | Number of Bedrooms  | Number of floors | Age of  Home | Price (1000s dollars)  |   \n",
    "| ----------------| ------------------- |----------------- |--------------|-------------- |  \n",
    "| 2104            | 5                   | 1                | 45           | 460           |  \n",
    "| 1416            | 3                   | 2                | 40           | 232           |  \n",
    "| 852             | 2                   | 1                | 35           | 178           | "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import copy,math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train=np.array([[2104,5,1,45],\n",
    "                  [1416,3,2,40],\n",
    "                  [852,2,1,35]])\n",
    "y_train=np.array([460,232,178])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def derivatives(X,y,w,b):\n",
    "    \"\"\"Calulates partial derivatives for w & b\n",
    "\n",
    "        Arguments:\n",
    "            X(ndarray,(m,n)): Data m examples with n features\n",
    "            y(ndarray,(m,)): target values\n",
    "            w(ndarray,(n,)): Model parameters\n",
    "            b(scalar): model parameter\n",
    "        Returns:\n",
    "            d_w(ndarray,(n,)): gradient of the cost w.r.t parameters w\n",
    "            d_b(scalar): gradient of the cost w.r.t parameter b \n",
    "    \"\"\"\n",
    "    m,n=X.shape #(no of examples,no of features)\n",
    "    d_w=np.zeros(n,dtype=float) # Cz a derivative for each feature\n",
    "    d_b=0.\n",
    "    for i in range(m):\n",
    "        err=np.dot(X[i],w)+b-y[i] # err=f_w,b-y; f_w,b=X.w+b\n",
    "        for j in range(n):\n",
    "            d_w[j]+=err*X[i,j]\n",
    "        d_b+=err\n",
    "    d_w/=m;d_b/=m\n",
    "    return d_w,d_b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(X,y,w_in,b_in,derivative_func,alpha,num_iters):\n",
    "    \"\"\"\n",
    "    Performs batch gradient descent to learn w & b by taking\n",
    "    num_iters gradient steps with learning rate alpha\n",
    "\n",
    "    Args:\n",
    "        X(ndarray,(m,n)): Data m examples with n features\n",
    "        y(ndarray,(m,)): target values\n",
    "        w_in(ndarray,(n,)): Initial model parameters\n",
    "        b_in(scalar): Initial model parameter\n",
    "        derivative_func: function to compute the gradient\n",
    "        alpha(float): Learning rate\n",
    "        num_iters(int): Number of gradient steps to take \n",
    "    \n",
    "    Returns:\n",
    "        w (ndarray (n,)) : Updated values of parameters \n",
    "        b (scalar)       : Updated value of parameter\n",
    "    \"\"\"\n",
    "    w=copy.deepcopy(w_in) # To not change the global w\n",
    "    b=b_in \n",
    "    for i in range(num_iters):\n",
    "        d_w,d_b=derivative_func(X,y,w,b) # Calulate the gradients for value w,b\n",
    "        # Update the values\n",
    "        w=w-alpha*d_w \n",
    "        b=b-alpha*d_b\n",
    "    return w,b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameters: b:-0.002235407530932535,w:[ 0.20396569  0.00374919 -0.0112487  -0.0658614 ]\n",
      "Prediction: 426.19, Target: 460\n",
      "Prediction: 286.17, Target: 232\n",
      "Prediction: 171.47, Target: 178\n"
     ]
    }
   ],
   "source": [
    "w_in=np.zeros(X_train.shape[1],dtype=float) #n=X_train.shape[1]\n",
    "b_in=0\n",
    "alpha=5.0e-7\n",
    "num_iters=1000\n",
    "w,b=gradient_descent(X_train,y_train,w_in,b_in,derivatives,alpha,num_iters)\n",
    "print(f\"Parameters: b:{b},w:{w}\")\n",
    "f_wb=np.zeros_like(y_train,dtype=float)\n",
    "for i in range(X_train.shape[0]):\n",
    "    f_wb[i]=np.dot(w,X_train[i])+b\n",
    "    print(f\"Prediction: {f_wb[i]:0.2f}, Target: {y_train[i]}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}