ch06-梯度下降法/02-线性回归模型中使用梯度下降法.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 线性回归模型中使用梯度下降法\n",
    "\n",
    "\n",
    "输入数据，迭代计算，到一定次数，`theta`就不变了\n",
    "\n",
    "找到最小值了，找到最低点\n",
    "\n",
    "最小值不是0，没有必要做无谓的挣扎"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 不是随机生成，具有可重复性\n",
    "np.random.seed(666)\n",
    "x = 2*np.random.random(size=100)\n",
    "y = x*3 + 4 + np.random.normal(size=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = x.reshape(-1, 1) # -1 代表任意"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.40087424],\n",
       "       [1.68837329],\n",
       "       [1.35302867],\n",
       "       [1.45571611],\n",
       "       [1.90291591]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x11033c278>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAD6CAYAAABXh3cLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAF1lJREFUeJzt3XGMHOV5x/HfY3OIAxoM5KqEq5wSIdkShWBxrVBN2uIUHCkhckAVEUGiTSVLlaqW0F5kpLbAXziiUkCK1Ir/ooSolJJaJqhyE0zVxopLfTKGOsVKSVLKIUVGjUMEF7gcb/+4PXtvb2b2ndmZed935vuRoix7e973Zmefeed5n/d9zTknAECaNoVuAACgOoI4ACSMIA4ACSOIA0DCCOIAkDCCOAAkzCuIm9mUmT09eGxm9hUzO2pmB83svGabCADIMzaIm9m0pAVJNw+e2inpPOfcDZLeJ+mW5poHACgyNog755acc9dKem3w1I8lPTp4/G5TDQMAjFc6FeKc+74kmdmnJZ0v6dDoa8xsr6S9knTRRRddv3379gmbCQD9srCw8IZzbmbc68x32r2Z/bdz7qrB409JulfSrc65nxX93tzcnDt27JjXewAAVpnZgnNubtzrSvfEzewDkuYlfdw591aVxgEA6lGlxPBuSR+UdMjMvmNmn6u5TQAAT9498bVUinPui5K+2FiLAADemOwDAAkjiANAwgjiAJAwpswDPXbg+KIePnRKr59Z0hVbpjW/e5v27JgN3SyUQBAHeurA8UXd942XtLS8IklaPLOk+77xkiQRyBNCOgXoqYcPnTobwNcsLa/o4UOnArUIVRDEgZ56/cxSqecRJ4I40FNXbJku9TziRBAHemp+9zZNT21e99z01GbN794WqEWogoFNoKfWBi+pTkkbQRzosT07ZgnaiSOdAgAJI4gDQMII4gCQMII4ACSMIA4ACSOIA0DCCOIAkDCCOAAkjMk+ADqjj+ujE8QBdEJf10cnnQKgE/q6PjpBHEAn9HV9dII4gE7o6/roBHEAndDX9dEZ2AQQrTLVJn1dH50gDiBKVapN2lofPaZSRoI4gCgVVZs0GTDHBejYShnJiQOIUohqk7UAvXhmSU7nAvSB44tnXxNbKSNBHECUQlSb+ATo2EoZCeIAohSi2sQnQMdWykgQBxClPTtm9dBt12h2y7RM0uyWaT102zWN5p19AnRspYwMbAKIVlvVJmvmd29bN2gpbQzQsZUyEsQBYMA3QOddXEKUHhLEAXRS1YBatfcfqvSQnDiAzskqFbzniRd03YP/vK5csE6hSg+9griZTZnZ04PHF5jZN83shJl91cys0RYCQElZAVWSziwtb6j7rkuo0sOxQdzMpiUtSLp58NRdkl5zzn1E0qVDzwNAFIoCZ1O941Clh2ODuHNuyTl3raTXBk/tkvStwePDkm5qqG0AUMm4wNlE7zhU6WGVnPjlkn46ePympMtGX2Bme83smJkdO3369CTtA4DSsgLqsCZ6xyHq2qVq1SlvSLpk8PiSwX+v45x7TNJjkjQ3N+cqtw4AKlgLnA8+fVI/eXt53c+a7B23XdcuVeuJPyvplsHjXZKeq685AFCPPTtmdfyvbtEjd1zXeu+4TVV64o9Lus3MXpR0QqtBHQCiFKJ33CbvIO6cu2rw/+9I+mRjLQIAeGOyDwAkjGn3AOAhpi3ZhhHEgRJi/SJjVVOfT2xbsg0jiAOeYv4i90VRkG7y8wm136cPcuKAp9j2VuybcftfNvn5xLYl2zCCOOAp5i9yH4wL0k19PgeOL2pTzjp/obZkG0YQBzzFtrdi34wL0k18Pmu9/xW3ceJ5yC3ZhhHEAU+x7a3YN+OCdBOfzwMHT2YuabvZLJqZnwRxwFOoBY6walyQrvvzOXB8UWeWljN/9p5z0XzuVKcAJXR9CnfMfPa/rPPzKRoQjSmFRhAHkIw2L6JFA6IxpdBIpwBAhrze9qUXTkV1N0YQB1DJgeOL2rn/sK7c94x27j/c2AbEoeTl4O+/9epALcpGOgVA6enqfZi96pODjwFBHEhAk2u2VAnIeRNv/uzvTxT+XmpSGMgmnQJEbtx080lVma6eN+i34lytbcN49MSByNWx+NJoT/6m7TN67uXTen1wYchSVJ1xxZZpLeb8fGl5Rfc88YIePnSq8h1DlTuPvq4wSRAHIue7JkheEMtKl3zt6Ktj37eoFnp+97Z1/2aWqnnyKumdvzjwkh4/+urZC1IXc/R5SKcAEcmq+PBZE6Qo5ZLVkx9napMV1kKvzY7cnLMw1JoqqwiWTe8cOL64LoBP8t4poicORCKvB3r79bN6amFxXWAbXROkKPBVWsWvODZLOtfDHdcjL/v+ZVcjfPjQqUopoSIppWboiQORyAvEz718euyaIEWBr8oU8eUV59WLHV6vJE/Z9y+7GuG43H1ZTQ8k140gDkSiKBDv2TGrI/t26Yf7P6Ej+3Zt6BUWBb6sSSuTtGfUWtseueO6WlYRLLsaYd7fbqo2PT61zT8I4kAkLpmeynzepzdZFPiyVve764atZ/87L69dthdb1yqCZf+drL/dJH32hq2VUiCpbf5BThyIwIHji3rr3V9seH7cAOOacbMLiyatjObiperrcNc1OabMv1P3zMq88smYVi4cRhAHIvDwoVNaXtk4PHfxBeeVCmZVAlcq08uL1DmzMqt8MubNPwjiQATybtXPvJ29KUHdUphe3pbULmoEcSACMd7C+5TZpVSKV0ZKFzUGNoEIxLZ/54Hji5p/8sS6Mrv5J0+sK7NLrRSvqwjiQARi27/zgYMntfze+hz98ntODxw8efa/UyvF6yrSKYhKV2/PfcR0C5+3QfDw86mV4nUVQRzR6MNGA10ySR6/axfrkH8P6RREg9vzeFx6YfbEo+Hnq+bxu5ZLD/33EMQRDW7P43H/rVdravP6mZxTm23d/pJV8/h5F+t7nnghyb06Q3c+SKcgGjGW2fWVb610lTx+0UU5xRRa6M4HQRzRSG2mXNc1NdBatCuQVH7XotBCdz5IpyAasZTZZW3MgHxlj5fPqooppdBC1/hX6omb2UWSvi7p/ZKOOOe+UGur0Fuhy+y6WiHTVPVEleM1nKrJ65GnlEILPU3fnMvbE6Pgl8z2SrrcOfeQmT0j6c+dc/+V9dq5uTl37NixCZsJtGPn/sOZgWV2y7SO7NsVoEWTy1ulsI67nEmPV5NtS52ZLTjn5sa9rmo65R1JF5qZSbpA0rsV/x0gKqEHqZrQZPXEpMcrlhRayqoObH5d0ncl/Z6kZ51zrwz/cNBT3ytJW7dunaiBQJtCD1I1ockLUx3HK3QKLXVVe+L3Sfpb59x2SZeZ2W8O/9A595hzbs45NzczMzNxI4G2hB6kakLZPSvL6OLxSk3VnvgvSfr54PE7ki6upzlAWKEHqZpQtnSzzCBoF49XaqoObP6qpMe12pN/VdKdzrmVrNcysAmE5xuYGWiMh+/AZqUgXgZBHEhHF6tzUuUbxJmxieh0bYW7mI0e67y67ZSrc7qOII6odHWyTYyyjrVJyro3T7k6p+uYdo+ohF4Rrk+yjrWTZCOvo9okbgRxRKWLk21ilXdMncTkm4SQTkFUujjZJlZ5x3qzGeMQCaEn3jGpr8DH5JH25K0muOJc0jvt9A1BvENCbxNVB9bSaM/asd5so1lwxiFSQjqlQ4oGBVMKgqyl0Z49O2b1+SdeyPwZ4xBpIIh3CIOCfqhDX49xiLSRTumQJhc66ooupJzqxjhE2gjiHdLWlzHlwVPq0DdiHCJtpFM6pI0V5VKfUUnKKRvjEOkiiHdM01/G1AdPyf+iawjiHdDmQF2sPVnfY1B2be263hdoCkE8cW2nN2LsyZY5BnWmnFJPLaEbCOKJmzS9UbYnWXdPto72lT0GdaWcUkstcdfQTQTxCGR9uSS/3uIk6Y0qPck2t+PybV+oFE8sqSWf4MxdQ3cRxAPL+nLNP3lCMml5xZ19Lu8LN0l6o2pPsq1KBt/2hUrxxJBa8g3Oqd01wB914oFlfbmW33NnA/iavFrmSWrDY+lJrhmtP/fdZSbUZJUYJsn41r3H9lmjPvTEGzbuVrfMlyjrtZOkN2LoSa4ps8vMJdNT2rn/8Lq/96Hbrmk93xvDTu++wTmmzxr1Iog3yOdWt2hfw1F5X7iq6Y22BymLFO0yMxzIpzaZ3nr3FzqztCzp3DF96LZrgmzkG3qSjG9wjumzRr1IpzTI51Y365Z8apNpavP65UGb+MLFNN3ad5eZiy84zzvV1Ae+KZ2YPmvUi554g3xudfNuybOea+ILF6InmZViyutRzm6ZXtfDvnLfM5n/Zoy53TZK+sqkdELfNaAZBPEG+d7q5n25uviFy0sx3X79rJ5aWBx7u59KbrfNkj6Cc7+RTmlQDNULsclLMT338mmv2/02j+kkqzWyWiLaQk+8QUWpktHqir70pIpSTD49yrYqQibtSVPSh7YQxBs2Gpj6MnMuLx9cRzqkzvRBXjsnnRyTStoH6SOd0rI+3GYX7Z4TU4qpqJ2T9qRj+jvRbQTxlvXhNntcLzaWUreidk661V1Mfye6jXRKy/pwmz3uQhVLNUVRO790x3UTT46J5e9Et9ETb1kfbrPr3LC5yf08i9pJTxqpoCfeshjW22haXVO8mx4EHtdOetJIAUE8gK4Hh7ouVE0vn9qHCyq6jyCORtRxoWpjELjrF1R0X9RBnO2k+q0Pg8DApKId2Cyq4UU/9GEQGJhU5SBuZl8ws38zs38ys/PrbJTUj0kxKEaFCDBepXSKmX1Y0tXOuY+a2Z9I+hVJP6izYX2YFIPxyFkDxar2xD8m6VIz+1dJH5X0w/qatKrOWmMA6KqqQXxG0mnn3G9ptRd+4/APzWyvmR0zs2OnT5+u9AbkQ9GWJicUAU2rGsTflLSWnP6BpHX3u865x5xzc865uZmZmUpvQD4UbWAAHamrWmK4IOneweOrVHM+fA35UDSt6QlFQNMq9cSdc9+V9IaZ/YekU8655+ttFtAOBtCRusqTfZxzf1RnQ2LCJKP+YEIRUhftZJ9QyJGmoa7BSAbQkbqop92HQI40LJ+7oDpXN2QRLKSOID6CHGk4vsG57gstA+hIGemUEUwyCsd3qQUutMA5BPER5EjD8Q3OXGiBcwjiI5hkFI5vcOZCC5zT25x40QAaOdIwfLd1YzASOKeXQbzpvRtRTZngzIUWWNWJIF52ck7qZYRdnoxEcAbKST6IV+lVp1TdMBqwb9o+o6cWFqO6i+jyRQWIXfIDm1V2AEqluiFr9ujjR19tdcejcTMjmeEKhJV8EK/Sq06luiHrAuVyXtvEXYRPgGYbPSCs5IN4lV51KmWEZQJzE3cRPgE6pdQU0EXJ58R9y9JGpTCAlrfCnml9j7zKXYRPHtsnQLMKIBBW8j3xVHrVVeSlfT57w9aJ/l7fPLbPXU4qqSmgq5LviUtp9KqraGpSi2+Jpc9dDhNvgLA6EcS7rIkLlG8e2zdAx34RpQQSXUYQr1nIgOH73mXy2LEH6HGYnYuuSz4nPqquHV+qvneomuky792nPDYlkOi6TgXx0BNP8gLGPU+80PgFpUyw6vJg8ChKINF1nUqn5AWyBw6ebCVAFQWGpm/jywar1NMkviiBRNd1qieeF7DOLC230hsfFxiavI1PZSmBtvUpdYR+6lQQLwpYbeRAswLGqKZu4wlW2fqUOkI/dSqdMr97m+554oXMn7WRAx0uycu6hZfyLzSTVrVQr52vL6kj9FOngvieHbN68OmT+snbyxt+thY8my4BXAsYo6VtUn7PuK4yOIIV0D+dSqdI0v23Xp2bVmizeqXMbTxlcACq6lRPXCpOK+zcf7jVHX18e8aUwQGoqnNBXMoPnrEGS8rgAFTVuXRKkVjL8KgsAVBVr4J4rMGSMjgAVUWdTqm7kiTmMjwqSwBUEW0Qb2r1OYIlJJanRXdEG8R9Ny7oghABpc9BjOVp0SXR5sRjrSSpW4iVF0Ov9hgadfnokmiDeKyVJHULEVD6HsT60kFAP0SbTina37HNVEDT7xUioPQ9iFGXjy6ZqCduZp83s2/X1Zhhw2V3krTZ7Oza4PP/cKKVVEAbaYcQdxx9ucvJE2upKVBF5SBuZh+S9Pv1NWWjPTtmz37hVpyTtLo2+PKKW/e6plIBbaQdQgSUvgcx6vLRJZOkUx6VdJ+ke2tqS6asQJqliVRA3r+Zt8xsFSFq12Oul28LpaboikpB3MzulHRC0vdyfr5X0l5J2rp1a+XGSf7BuYlUQF7u1LSaaqkrCIQIKAQxoBuqplM+Keljkv5O0vVm9sfDP3TOPeacm3POzc3MzEzUQJ/g3FQqYH73NlnG807t7BRUhwPHF7Vz/2Fdue+ZxjdrBtC+SkHcOXenc+5GSZ+RtOCc+3K9zTonK387tcl06YVTjecz9+yYlcv5WQqVHH2vBwf6INoSwzWh87ezCZej9WnWK9BXEwVx59yPJP1uPU3J12b+drQu/KbtM3pqYdFrm7XY9L0eHOiDaGdshpCVfnhqYVG3Xz+bZDla3+vBgT4giA/JSz987eirkqQv3XGdjuzblUQAl6gHB/og+px4m4rSDE2tdNfktP7Q4wkAmkcQH5JXF76m7kHBNpZEpR4c6DbSKUOy0g+j6hwU7PtqggAmR098yHD6Ia9HXuegINUjACZFT3zEnh2zOrJvlx6547rGBwWpHgEwKYJ4jjZWuqN6BMCkSKcUaHpQkOoRAJOiJw4ACaMnHhC7rgOYFD3xgCgxBDApgnhAlBgCmBRBPCBKDAFMiiAeECWGACaV1MBmk4tFhUCJIYBJJRPEu1rJwQJVACaRTDqFSg4A2CiZIE4lBwBslEwQp5IDADZKJohTyQEAGyUzsEklBwBslEwQl6jkAIBRyaRTAAAbJdUT79pkHwCYVDJBvKuTfQBgEsmkU5jsAwAbJRPEmewDABslE8SZ7AMAGyUTxJnsAwAbJTOwyWQfANgomSAuMdkHAEYlk04BAGxEEAeAhBHEASBhBHEASBhBHAASZs65Zt/A7LSk/5ngn3i/pDdqak5dYmyTRLvKol3l0K5yJm3Xh5xzM+Ne1HgQn5SZHXPOzYVux7AY2yTRrrJoVzm0q5y22kU6BQASRhAHgISlEMQfC92ADDG2SaJdZdGucmhXOa20K/qcOAAgXwo9cQBAjmBB3MwuMLNvmtkJM/uqmZnPa3x+r4V2mZl9xcyOmtlBMzvPzD5uZq+Z2XcG/6t1jVzPdm1oQyTH63eG2vS/ZnZ308dr8L5TZvZ0mbY3fbw829X6+eXZrtbPL892tXp+ZX0+Ga9p7dwK2RO/S9JrzrmPSLpU0s2er/H5vabbtVPSec65GyS9T9Itg+f/xjl34+B/de8b5/t3j7Yh+PFyzv3LWpskvSjpeE5ba2Nm05IWstozpu2NHi/PdrV+fnm2K6sNwY9XgPMr7/MZ1tq5FTKI75L0rcHjw5Ju8nyNz+813a4fS3p08PjdoedvN7PnzeypBnokvn/3aBtiOF6SJDO7UNJVzrkXc9paG+fcknPuWkmvFbys9fPLs12tn1+e7cpqQwzHS1Kr51fe5zOstXMrZBC/XNJPB4/flHSZ52t8fq/Rdjnnvu+ce97MPi3pfEmHJL0i6S+dc78h6YOSfrvtduW0IfjxGnKzpGcL2tq2EOfXWIHOLx8hzq8yWjm/cj6fUa2dWyE3hXhD0iWDx5coe3pq1msu9vi9ptslM/uUpD+VdKtzbsXM/k/Stwc//pGkXw7Qrqw2eP09Dbdrza2SvjF43PTx8hHi/PIS4PzyEeL8KqO182v088l4SWvnVsie+LM6l0vaJek5z9f4/F6j7TKzD0ial/QJ59zPBk/fK+kzZrZJ0q9J+s+225XThuDHS1odDNLq7ePhgra2LcT5NVag88tHiPPLS5vnV87nM6q1cytkEH9c0qyZvajVq+YrZvbXY17zbM5zbbfrbq3eoh0ajHx/TtKXJf2BpH+X9I/Oue8FaFdWG2I4XpL065JOOud+XtDWxpjZlZGcXz7tCnF++bQrxPnl0y6p3fNr9PP5w5DnFpN9ACBhTPYBgIQRxAEgYQRxAEgYQRwAEkYQB4CEEcQBIGEEcQBI2P8DSIdzy6CgvvIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10e21dd68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用梯度下降法训练\n",
    "![](images/01.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 求偏导(梯度)\n",
    "#### $$ \\frac{\\delta J(\\theta)}{\\delta\\theta_{j}} = \\frac{1}{m}\\sum_{i=1}^{m} ( h_\\theta (x^{(i)})-y^{(i)})x^{(i)}_{j} $$ \n",
    "#### 向量化的偏导(梯度)\n",
    "#### $$ \\frac{\\delta J(\\theta)}{\\delta\\theta_{j}} = \\frac{1}{m} X^T(g(X\\theta)-y)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 损失函数\n",
    "def J(theta, X_b, y):\n",
    "    try:\n",
    "        return np.sum((y - X_b.dot(theta))**2) / len(X_b)\n",
    "    except:\n",
    "        return float('inf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 梯度计算\n",
    "def dJ(theta, X_b, y):\n",
    "    res = np.empty(len(theta))\n",
    "    res[0] = np.sum(X_b.dot(theta) - y) # 第0个参数\n",
    "    \n",
    "    for i in range(1, len(theta)):\n",
    "        # 每一个样本取出 对应特征 对应的列\n",
    "        res[i] = np.sum((X_b.dot(theta) -y).dot(X_b[:, i]))\n",
    "        \n",
    "    return res * 2 / len(X_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(X_b, y, initial_theta, eta, n_iters=1e3 ,epsilon=1e-8):\n",
    "    \n",
    "    theta = initial_theta\n",
    "    i_iter = 0\n",
    "    \n",
    "    while i_iter < n_iters:\n",
    "        gradient = dJ(theta, X_b, y)\n",
    "        last_theta = theta\n",
    "        theta = theta - eta * gradient # 向导数的负方向移1步\n",
    "    \n",
    "        # 是不是最小值的点，导数等于0 的点 --- 可能永远达不到这个精度\n",
    "        # 每一次损失函数都要小一点\n",
    "        if(abs(J(theta, X_b, y) - J(last_theta, X_b, y)) < epsilon):\n",
    "            break\n",
    "        \n",
    "        i_iter += 1\n",
    "        \n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_b = np.hstack([np.ones((len(x), 1)), x.reshape(-1, 1)]) # 变成列向量\n",
    "initial_theta = np.zeros(X_b.shape[1])\n",
    "eta = 0.01\n",
    "\n",
    "theta = gradient_descent(X_b, y, initial_theta, eta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 1.40087424],\n",
       "       [1.        , 1.68837329],\n",
       "       [1.        , 1.35302867],\n",
       "       [1.        , 1.45571611],\n",
       "       [1.        , 1.90291591]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_b[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([8.91412688, 8.89446981, 8.85921604, 9.04490343, 8.75831915])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0.])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "initial_theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.99481236, 3.02953666])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 就是我们要计算的 b w\n",
    "# 截距 斜率\n",
    "# 找到最低点，我的theta就不变了\n",
    "# 此时的theta=[b, w]决定的直线，与原始数据误差最小！！\n",
    "theta "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 封装自己的线性回归算法\n",
    "参见 [代码](playML/LinearRegression.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from playML.LinearRegression import LinearRegression\n",
    "\n",
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit_gd(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4.02145786, 3.00706277])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lin_reg._theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.00706277])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 系数\n",
    "lin_reg.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.021457858204859"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 系数\n",
    "lin_reg.intercept_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lin_reg2 = LinearRegression()\n",
    "lin_reg2.fit_normal(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.00517447])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lin_reg2.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.023696672103897"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lin_reg2.intercept_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 梯度下降法向量化提速\n",
    "\n",
    "以前要for循环\n",
    "\n",
    "形同形式，矩阵之间的乘法\n",
    "![](images/02.png)\n",
    "\n",
    "- 算法表示都是列向量\n",
    "- numpy中向量不区分行、列向量的（没有列向量），基本都是1维向量\n",
    "- 数学表示上，都是列向量，转置变成行向量！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def dJ(theta, X_b, y):\n",
    "    # res = np.empty(len(theta))\n",
    "    # res[0] = np.sum(X_b.dot(theta) - y)\n",
    "    # for i in range(1, len(theta)):\n",
    "    #     res[i] = (X_b.dot(theta) - y).dot(X_b[:, i])\n",
    "    # return res * 2 / len(X_b)\n",
    "    return X_b.T.dot(X_b.dot(theta) - y) * 2. / len(X_b)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}