0128 DL기초 과제.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mlQz3vXAU2QE"
   },
   "source": [
    "# 과제1: tensorflow를 이용한 NN 밑바닥부터 구현\n",
    "input feature가 100개이고,  \n",
    "hidden layer가 2개이고 neuron이 각각 50,10개이고,  \n",
    "output이 5개인 NN를 구현해 보자  \n",
    "* hidden layer는 relu를 activation function으로, output layer는 softmax를 activation function으로 사용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "CeUa71VCU2QL"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "Kfo6oOn8U2QM"
   },
   "outputs": [],
   "source": [
    "n_x = 100\n",
    "n_h1 = 50\n",
    "n_h2 = 10\n",
    "n_y = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "WeNzQkQ-U2QM"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.32244593 -0.5394701   1.1398345  ...  1.6669414  -2.0477543\n",
      "  -0.7643977 ]\n",
      " [ 1.3528959  -0.64019465 -0.47097698 ...  0.9626349  -0.9857793\n",
      "  -0.18307562]\n",
      " [-0.42825982  0.33774576  1.316114   ... -0.2287569  -0.97027117\n",
      "  -0.22762983]\n",
      " ...\n",
      " [-0.04587292  1.0318066   1.1730008  ...  0.13812695  0.6797436\n",
      "   0.69245064]\n",
      " [ 1.267226    0.5102633  -0.20717947 ... -2.033669    0.04772108\n",
      "   0.32280412]\n",
      " [-0.06988382 -1.0796353  -1.8658632  ...  1.271504    1.4803022\n",
      "   0.497322  ]] (500, 100)\n",
      "[[0. 0. 0. 1. 0.]\n",
      " [0. 0. 1. 0. 0.]\n",
      " [0. 0. 1. 0. 0.]\n",
      " ...\n",
      " [0. 0. 0. 0. 1.]\n",
      " [0. 0. 0. 0. 1.]\n",
      " [0. 0. 1. 0. 0.]] (500, 5)\n"
     ]
    }
   ],
   "source": [
    "# Simulate train set\n",
    "m = 500\n",
    "\n",
    "x_train=np.random.randn(m,n_x).astype(np.float32)\n",
    "y_train=np.zeros((m,n_y)).astype(np.float32)\n",
    "y_train[np.arange(m),np.random.randint(n_y,size=m)]=1\n",
    "\n",
    "print(x_train,x_train.shape)\n",
    "print(y_train,y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(500, 5)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XO5ck0OrU2QN"
   },
   "source": [
    "* Initialization of weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "t5gEprk8U2QN"
   },
   "outputs": [],
   "source": [
    "w1=tf.Variable(1e-3*np.random.randn(n_x,n_h1).astype(np.float32),name=\"w1\")\n",
    "## 코드를 작성해 보세요 ##\n",
    "w2= tf.Variable(1e-3*np.random.randn(n_h1,n_h2).astype(np.float32),name=\"w2\")\n",
    "w3= tf.Variable(1e-3*np.random.randn(n_h2,n_y).astype(np.float32),name=\"w3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J0hTdH3IU2QO"
   },
   "source": [
    "* forward propagation을 통해 prediction 값을 구하고, loss를 구하는 function을 만들어 봅시다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "mRZYg6SWU2QO"
   },
   "outputs": [],
   "source": [
    "def forward(x):\n",
    "    z1=tf.matmul(x,w1)   # 500,50\n",
    "    a1=tf.nn.relu(z1)    \n",
    "    ## 코드를 작성해 보세요 ##\n",
    "    z2=tf.matmul(a1,w2)   # 500,10\n",
    "    a2=tf.nn.relu(z2)\n",
    "    z3=tf.matmul(a2,w3)   # 500,5\n",
    "    predictions = tf.nn.sigmoid(z3)\n",
    "    \n",
    "    return predictions\n",
    "\n",
    "def loss_fn(predictions, y):\n",
    "    loss= -tf.reduce_sum(y*tf.math.log(predictions))\n",
    "    return loss    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aiGxU4zYU2QP"
   },
   "source": [
    "* backpropagation & update parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "Br-xEdiDU2QP"
   },
   "outputs": [],
   "source": [
    "learning_rate=1e-2\n",
    "optimizer=tf.keras.optimizers.Adam(learning_rate)\n",
    "\n",
    "def train_step(x, y):\n",
    "    with tf.GradientTape() as tape:\n",
    "        predictions = forward(x)\n",
    "        loss = loss_fn(predictions, y)\n",
    "    ## 코드를 작성해 보세요 ## (hint: tape.gradient를 구글링 해보세요)\n",
    "    gradient = tape.gradient(loss, [w1,w2])\n",
    "    # optimizer와 위에서 구한 경사도를 이용해 가중치들을 업데이트 합니다.\n",
    "    optimizer.apply_gradients(zip(gradient, [w1, w2]))\n",
    "    return loss, w1, w2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OU1oHsFoU2QP"
   },
   "source": [
    "* 간단하게 train loop를 작성해 loss가 줄어나가는지 확인해 봅시다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "YIHNHAjPU2QP"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[346.5737, 346.5704, 346.55972, 346.5393, 346.50766, 346.46323, 346.40503, 346.3322, 346.24377, 346.13873]\n"
     ]
    }
   ],
   "source": [
    "loss_list = []\n",
    "\n",
    "for step in range(10): \n",
    "    loss, w1, w2 = train_step(x_train, y_train)\n",
    "    loss_list.append(loss.numpy())\n",
    "    \n",
    "print(loss_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MN_pVd5cU2QQ"
   },
   "source": [
    "# 과제2: MNIST 데이터를 나만의 NN model로 95 % 이상의 성능으로 training 시켜보자!\n",
    "\n",
    "\n",
    "## Loading MNIST training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "hwZNV5MFU2QQ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 1s 0us/step\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "\n",
    "# import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# Loading the data\n",
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "# Scaling(image data는 min-max scaling 주로 사용)\n",
    "x_train = x_train/255.0\n",
    "x_test = x_test/255.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RDU8J2xRU2QQ"
   },
   "source": [
    "## Training Data\n",
    "28 * 28 pixel 값을 가진 총 60000개의 이미지 데이터"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "jVvXmjQSU2QQ"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-VTAAYKSU2QQ"
   },
   "source": [
    "Neural network 모델에 맞게 이미지 데이터를 벡터 형태로 데이터를 reshape 합니다.  \n",
    "(Model을 만들 때 *keras.layers.Flatten(input_shape=(28, 28)) 이용해도 됨)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "dq36yUX8U2QR"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((60000, 784), (10000, 784))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train, x_test = x_train.reshape((-1, 28*28)), x_test.reshape((-1, 28*28))\n",
    "x_train.shape, x_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "zrQLH9iXU2QR"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAOS0lEQVR4nO3df4xU9bnH8c8jgqgQg7JQYsnd3kZNjcnd4kiuQQiXegnyDxDsTUlsaCTdxh9JMcRcszex/kgMMZdWjKbJ9oLQm15rFRBMzC1KSAyJVkdFBfF31rIFYYlKhSgt8Nw/9nCz4sx3lpkzc4Z93q9kMzPnOWfP47gfzsx8z5mvubsAjHznFN0AgNYg7EAQhB0IgrADQRB2IIhzW7mziRMnemdnZyt3CYTS19enQ4cOWaVaQ2E3s3mSVksaJem/3H1lav3Ozk6Vy+VGdgkgoVQqVa3V/TLezEZJelTSDZKulLTEzK6s9/cBaK5G3rNPl/SBu3/k7n+T9HtJC/JpC0DeGgn7pZL2Dnncny37GjPrNrOymZUHBgYa2B2ARjQS9kofAnzj3Ft373X3kruXOjo6GtgdgEY0EvZ+SVOHPP62pH2NtQOgWRoJ+yuSLjOz75jZGEk/krQln7YA5K3uoTd3P25mt0v6owaH3ta6++7cOgOQq4bG2d39WUnP5tQLgCbidFkgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCaGgWV7S/kydPJuvHjh1r6v7Xr19ftXb06NHktm+//Xay/tBDDyXrPT09VWuPPPJIctvzzz8/WV+1alWyfssttyTrRWgo7GbWJ+kLSSckHXf3Uh5NAchfHkf2f3H3Qzn8HgBNxHt2IIhGw+6StprZq2bWXWkFM+s2s7KZlQcGBhrcHYB6NRr2Ge4+TdINkm4zs1mnr+Duve5ecvdSR0dHg7sDUK+Gwu7u+7Lbg5I2SZqeR1MA8ld32M3sQjMbf+q+pLmSduXVGIB8NfJp/GRJm8zs1O/5H3f/31y6GmEOHz6crJ84cSJZf+ONN5L1rVu3Vq19/vnnyW17e3uT9SJ1dnYm6ytWrEjW16xZU7V20UUXJbedOXNmsj5nzpxkvR3VHXZ3/0jSP+XYC4AmYugNCIKwA0EQdiAIwg4EQdiBILjENQf9/f3JeldXV7L+2Wef5dnOWeOcc9LHmtTQmVT7MtRly5ZVrU2aNCm57bhx45L1s/FsUI7sQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAE4+w5uOSSS5L1yZMnJ+vtPM4+d+7cZL3Wf/vGjRur1s4777zktrNnz07WcWY4sgNBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIyz56DWddXr1q1L1p966qlk/dprr03WFy9enKynXHfddcn65s2bk/UxY8Yk65988knV2urVq5PbIl8c2YEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCHP3lu2sVCp5uVxu2f7OFseOHUvWa41l9/T0VK09+OCDyW23b9+erM+aNStZR3splUoql8tWqVbzyG5ma83soJntGrLsYjN7zszez24n5NkwgPwN52X8OknzTlt2l6Rt7n6ZpG3ZYwBtrGbY3f0FSZ+etniBpPXZ/fWSFubcF4Cc1fsB3WR33y9J2W3VibPMrNvMymZWHhgYqHN3ABrV9E/j3b3X3UvuXjobJ8MDRop6w37AzKZIUnZ7ML+WADRDvWHfImlpdn+ppPR1kAAKV/N6djN7XNJsSRPNrF/SLyStlPQHM1sm6c+SftjMJke6Wt+fXsuECfWPfD788MPJ+syZM5N1s4pDumhDNcPu7kuqlH6Qcy8AmojTZYEgCDsQBGEHgiDsQBCEHQiCr5IeAZYvX1619vLLLye33bRpU7K+e/fuZP2qq65K1tE+OLIDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCMs48Aqa+a7u3tTW67bdu2ZH3BggXJ+sKF6a8fnDFjRtXaokWLktty+Wy+OLIDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBBM2Rxcrevd5807fU7Przt8+HDd+167dm2yvnjx4mR93Lhxde97pGpoymYAIwNhB4Ig7EAQhB0IgrADQRB2IAjCDgTB9ezBTZ8+PVmv9b3xd9xxR7L+5JNPVq3dfPPNyW0//PDDZP3OO+9M1sePH5+sR1PzyG5ma83soJntGrLsHjP7i5ntzH7mN7dNAI0azsv4dZIqnUb1K3fvyn6ezbctAHmrGXZ3f0HSpy3oBUATNfIB3e1m9mb2Mn9CtZXMrNvMymZWHhgYaGB3ABpRb9h/Lem7krok7Ze0qtqK7t7r7iV3L3V0dNS5OwCNqivs7n7A3U+4+0lJv5GU/kgXQOHqCruZTRnycJGkXdXWBdAeal7PbmaPS5otaaKkA5J+kT3ukuSS+iT9zN3319oZ17OPPF999VWy/tJLL1WtXX/99clta/1t3njjjcn6E088kayPRKnr2WueVOPuSyosXtNwVwBaitNlgSAIOxAEYQeCIOxAEIQdCIJLXNGQsWPHJuuzZ8+uWhs1alRy2+PHjyfrTz/9dLL+7rvvVq1dccUVyW1HIo7sQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAE4+xI2rdvX7K+cePGZP3FF1+sWqs1jl7LNddck6xffvnlDf3+kYYjOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EwTj7CFdryq1HH300WX/ssceS9f7+/jPuabhqXe/e2dmZrJtV/EblsDiyA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQjLOfBY4cOZKsP/PMM1Vr9913X3Lb9957r66e8jBnzpxkfeXKlcn61VdfnWc7I17NI7uZTTWz7Wa2x8x2m9nPs+UXm9lzZvZ+djuh+e0CqNdwXsYfl7TC3b8n6Z8l3WZmV0q6S9I2d79M0rbsMYA2VTPs7r7f3V/L7n8haY+kSyUtkLQ+W229pIXNahJA487oAzoz65T0fUl/kjTZ3fdLg/8gSJpUZZtuMyubWbnWedoAmmfYYTezcZI2SFru7n8d7nbu3uvuJXcvdXR01NMjgBwMK+xmNlqDQf+du5/6OtEDZjYlq0+RdLA5LQLIQ82hNxu8TnCNpD3u/sshpS2Slkpamd1ubkqHI8DRo0eT9b179ybrN910U7L++uuvn3FPeZk7d26yfu+991at1foqaC5RzddwxtlnSPqxpLfMbGe2rEeDIf+DmS2T9GdJP2xOiwDyUDPs7r5DUrV/Yn+QbzsAmoXTZYEgCDsQBGEHgiDsQBCEHQiCS1yH6csvv6xaW758eXLbHTt2JOvvvPNOXT3lYf78+cn63Xffnax3dXUl66NHjz7jntAcHNmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IIgw4+x9fX3J+gMPPJCsP//881VrH3/8cT0t5eaCCy6oWrv//vuT2956663J+pgxY+rqCe2HIzsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBBFmnH3Dhg3J+po1a5q272nTpiXrS5YsSdbPPTf9v6m7u7tqbezYscltEQdHdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0Iwtw9vYLZVEm/lfQtSScl9br7ajO7R9JPJQ1kq/a4+7Op31UqlbxcLjfcNIDKSqWSyuVyxVmXh3NSzXFJK9z9NTMbL+lVM3suq/3K3f8zr0YBNM9w5mffL2l/dv8LM9sj6dJmNwYgX2f0nt3MOiV9X9KfskW3m9mbZrbWzCZU2abbzMpmVh4YGKi0CoAWGHbYzWycpA2Slrv7XyX9WtJ3JXVp8Mi/qtJ27t7r7iV3L3V0dOTQMoB6DCvsZjZag0H/nbtvlCR3P+DuJ9z9pKTfSJrevDYBNKpm2M3MJK2RtMfdfzlk+ZQhqy2StCv/9gDkZTifxs+Q9GNJb5nZzmxZj6QlZtYlySX1SfpZUzoEkIvhfBq/Q1KlcbvkmDqA9sIZdEAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSBqfpV0rjszG5D08ZBFEyUdalkDZ6Zde2vXviR6q1eevf2Du1f8/reWhv0bOzcru3upsAYS2rW3du1Lord6tao3XsYDQRB2IIiiw95b8P5T2rW3du1Lord6taS3Qt+zA2idoo/sAFqEsANBFBJ2M5tnZu+a2QdmdlcRPVRjZn1m9paZ7TSzQueXzubQO2hmu4Ysu9jMnjOz97PbinPsFdTbPWb2l+y522lm8wvqbaqZbTezPWa228x+ni0v9LlL9NWS563l79nNbJSk9yT9q6R+Sa9IWuLub7e0kSrMrE9Syd0LPwHDzGZJOiLpt+5+VbbsQUmfuvvK7B/KCe7+723S2z2SjhQ9jXc2W9GUodOMS1oo6Scq8LlL9PVvasHzVsSRfbqkD9z9I3f/m6TfS1pQQB9tz91fkPTpaYsXSFqf3V+vwT+WlqvSW1tw9/3u/lp2/wtJp6YZL/S5S/TVEkWE/VJJe4c87ld7zffukraa2atm1l10MxVMdvf90uAfj6RJBfdzuprTeLfSadOMt81zV8/0540qIuyVppJqp/G/Ge4+TdINkm7LXq5ieIY1jXerVJhmvC3UO/15o4oIe7+kqUMef1vSvgL6qMjd92W3ByVtUvtNRX3g1Ay62e3Bgvv5f+00jXelacbVBs9dkdOfFxH2VyRdZmbfMbMxkn4kaUsBfXyDmV2YfXAiM7tQ0ly131TUWyQtze4vlbS5wF6+pl2m8a42zbgKfu4Kn/7c3Vv+I2m+Bj+R/1DSfxTRQ5W+/lHSG9nP7qJ7k/S4Bl/W/V2Dr4iWSbpE0jZJ72e3F7dRb/8t6S1Jb2owWFMK6u06Db41fFPSzuxnftHPXaKvljxvnC4LBMEZdEAQhB0IgrADQRB2IAjCDgRB2IEgCDsQxP8BwfxNbNfq1cUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x_train[0].reshape(28,28)).set_cmap('Greys')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0YZXzr-AU2QR"
   },
   "source": [
    "## Training Labels\n",
    "이미지 데이터가 나타내는 숫자값을 label로 가지고 있고, 0부터 9까지의 값을 나타냄  \n",
    "마찬가지로, 60000개의 label이 존재"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "V-JVvQcJU2QR"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000,)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "PgAkJK6yU2QR",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show MNIST label for above data\n",
    "y_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000,)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qaI3Kv_GU2QR"
   },
   "source": [
    "## 나만의 모델을 tensorflow keras API 를 이용해 만들어 봅시다~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gSshVnt2U2QS"
   },
   "source": [
    "* parameters for model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "coUZ53nKU2QS"
   },
   "outputs": [],
   "source": [
    "activation_list = [\"sigmoid\", \"relu\", \"softmax\", \"tanh\"]\n",
    "\n",
    "loss_list = [\"sparse_categorical_crossentropy\",\n",
    "             \"categorical_crossentropy\", \n",
    "             \"binary_crossentropy\"]\n",
    "\n",
    "optimizer_list = [\"sgd\", \"adam\", \"rmsprop\", \"adagrad\"]\n",
    "\n",
    "initializer_list = [tf.keras.initializers.RandomNormal(), \n",
    "                    tf.keras.initializers.RandomUniform(), \n",
    "                    tf.keras.initializers.he_normal(), \n",
    "                    tf.keras.initializers.he_uniform(), \n",
    "                    tf.keras.initializers.GlorotUniform(),\n",
    "                    tf.keras.initializers.GlorotNormal()]\n",
    "\n",
    "# dropout\n",
    "dropout_rate = 0.3\n",
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Dense(256, input_dim=784, activation = \"sigmoid\"),\n",
    "    tf.keras.layers.Dense(2, activation = \"sigmoid\"),\n",
    "    tf.keras.layers.Dropout(dropout_rate)\n",
    "])\n",
    "\n",
    "\n",
    "# regularizer\n",
    "regularizer = tf.keras.regularizers.l1(1e-3)\n",
    "regularizer = tf.keras.regularizers.l2(1e-3)\n",
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Dense(256, input_dim=784, activation=\"sigmoid\",\n",
    "                          activity_regularizer=regularizer)\n",
    "])\n",
    "\n",
    "# weight initialization\n",
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Dense(256, input_dim=784, activation=\"sigmoid\",\n",
    "                          kernel_initializer=initializer_list[0])\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d-6ZTz4AU2QS"
   },
   "source": [
    "#### My Own Model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "id": "ziVbGe6sU2QS"
   },
   "outputs": [],
   "source": [
    "#### 자유롭게 Model을 만들고 compile 해봅시다 ####\n",
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Dense(256, input_dim=784, activation=\"sigmoid\"),\n",
    "    tf.keras.layers.Dense(256, activation=\"sigmoid\"),\n",
    "    tf.keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PVhLJHJ9U2QT"
   },
   "source": [
    "내가 만든 모델을 확인해 봅시다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss = 'sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "id": "GChgw-8sU2QT",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_5\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_7 (Dense)              (None, 256)               200960    \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 256)               65792     \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 10)                2570      \n",
      "=================================================================\n",
      "Total params: 269,322\n",
      "Trainable params: 269,322\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f9zWRRHIU2QT"
   },
   "source": [
    "model을 자유롭게 train 해봅시다.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "id": "6uygJ19gU2QT"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1875/1875 [==============================] - 7s 3ms/step - loss: 0.6720 - accuracy: 0.8046\n",
      "Epoch 2/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1698 - accuracy: 0.9490\n",
      "Epoch 3/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.1047 - accuracy: 0.9679\n",
      "Epoch 4/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0746 - accuracy: 0.9781\n",
      "Epoch 5/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0556 - accuracy: 0.9820\n",
      "Epoch 6/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0383 - accuracy: 0.9876\n",
      "Epoch 7/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0281 - accuracy: 0.9915\n",
      "Epoch 8/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0218 - accuracy: 0.9929\n",
      "Epoch 9/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0197 - accuracy: 0.9936\n",
      "Epoch 10/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0130 - accuracy: 0.9961\n",
      "Epoch 11/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0109 - accuracy: 0.9964\n",
      "Epoch 12/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0103 - accuracy: 0.9963\n",
      "Epoch 13/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0075 - accuracy: 0.9979\n",
      "Epoch 14/20\n",
      "1875/1875 [==============================] - 7s 4ms/step - loss: 0.0073 - accuracy: 0.9977\n",
      "Epoch 15/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0057 - accuracy: 0.9982\n",
      "Epoch 16/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0042 - accuracy: 0.9988\n",
      "Epoch 17/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0048 - accuracy: 0.9984\n",
      "Epoch 18/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0031 - accuracy: 0.9992\n",
      "Epoch 19/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0075 - accuracy: 0.9976\n",
      "Epoch 20/20\n",
      "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0025 - accuracy: 0.9993\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x25df1a56348>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train, epochs=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U8A4zKnEU2QT"
   },
   "source": [
    "95%이상의 성능을 가진 모델을 만들면 완성!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "id": "9Xz0qGifU2QU"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 - 1s - loss: 0.0940 - accuracy: 0.9808\n",
      "\n",
      "Accuracy: 0.9807999730110168\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = model.evaluate(x_test,y_test, verbose=2)\n",
    "\n",
    "print('\\nAccuracy:', test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "id": "EbcuzK_PU2QU"
   },
   "source": [
    "![](https://www.tensorflow.org/versions/master/images/mnist_tensorboard.png)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "DL기초 과제.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}