Keras-CNN-Tuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introducing Keras\n",
    "\n",
    "Let's use Keras on the MNIST handwriting data set, using a Convolutional Neural Network that's suited for image processing. CNN's are less sensitive to where in the image the pattern is that we're looking for.\n",
    "\n",
    "We'll start by importing the stuff we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow\n",
    "from tensorflow.keras.datasets import mnist\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Dropout, Conv2D, MaxPooling2D, Flatten\n",
    "from tensorflow.keras.optimizers import RMSprop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll load up our raw data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "(mnist_train_images, mnist_train_labels), (mnist_test_images, mnist_test_labels) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we're treating the data as 2D images of 28x28 pixels, we need to shape it accordingly. Depending on the data format Keras is set up for, this may be 1x28x28 or 28x28x1 (the \"1\" indicates a single color channel, as this is just grayscale. If we were dealing with color images, it would be 3 instead of 1 since we'd have red, green, and blue color channels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import backend as K\n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    train_images = mnist_train_images.reshape(mnist_train_images.shape[0], 1, 28, 28)\n",
    "    test_images = mnist_test_images.reshape(mnist_test_images.shape[0], 1, 28, 28)\n",
    "    input_shape = (1, 28, 28)\n",
    "else:\n",
    "    train_images = mnist_train_images.reshape(mnist_train_images.shape[0], 28, 28, 1)\n",
    "    test_images = mnist_test_images.reshape(mnist_test_images.shape[0], 28, 28, 1)\n",
    "    input_shape = (28, 28, 1)\n",
    "    \n",
    "train_images = train_images.astype('float32')\n",
    "test_images = test_images.astype('float32')\n",
    "train_images /= 255\n",
    "test_images /= 255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to convert our train and test labels to be categorical in one-hot format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_labels = tensorflow.keras.utils.to_categorical(mnist_train_labels, 10)\n",
    "test_labels = tensorflow.keras.utils.to_categorical(mnist_test_labels, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a sanity check let's print out one of the training images with its label:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEzZJREFUeJzt3X+w1XWdx/HnS1EqoBK5IiFyC9HVrc3cI7UjmzqlIVno7mRYNurm0oxZsWNOSqW2MrtuE2arKy4KSaWWjSJE2Eroaq6j080MJMpfi6vIj4uooLK66Hv/+H5vHq/nfM/l/LjnwOf1mDlzz/2+v9/zfZ8vvO7n++Oe+1VEYGbp2aPdDZhZezj8Zoly+M0S5fCbJcrhN0uUw2+WKIe/A0m6WNKP2t1HJ5J0naTZg73s7sjhLyNpsqR7JT0vaYuk/5J0ZLv7aoSkcyT1SHpZ0nX9ah+StDx/r72SfippTFn9HyQ9LmmrpKclfVfSkArrOFpS7EywJP2npLMaenMtJOlYSaskPSfpGUmLJI1td1/N5PDnJL0dWApcAYwExgLfAl5uZ19N8DQwG1hQobYPMA/oBsYD24Dvl9WXAEdExNuB9wLvB75c/gKS9gK+B9zf7Mbb7PfAxyLincC7gEeAue1tqbkc/tcdDBARN0bEqxGxPSJuj4iVAJImSLojHwU2S7pe0jv7Fpa0VtJ5klZKelHSfEmjJd0maZukX0raJ5+3Ox8pZ+Qj6npJX63WWD5C35uPQr+TdMxA31RE3BIRtwLPVKjdFhE/jYitEfEScCVwVFn9sYh4rq8N4DXgoH4vcy5wO/CHgfZUS74HsiHfA7tb0p/3m2VUvseyTdJdksaXLftnZXszf5R0Sj09RMTGiHi6bNKrvPm979Ic/tc9DLwqaaGkE/qCWkbAP5ONAocC44CL+83zt8BxZD9IPgHcBswCusi29Zf7zX8sMBE4HviapI/2byrf1fw52eg9EvgqcLOkrrx+vqSl9bzhCj4MrO63/s9I2gpsJhv5/72sNh74O+Afm7T+PreRbZf9gAeA6/vVPwtcAowCHuyrSxoGLAduyJedDlwl6bBKK8l/mE6u1oSkAyU9B2wn2+7fbuA9dRyHPxcRW4HJQADXAL2SlkgandcfjYjlEfFyRPQClwFH93uZK/IRYx3wK+D+iPhtRPwvsAj4QL/5vxURL0bEKrLd7VMrtHYasCwilkXEaxGxHOgBpuZ9XRoRJzb6/iX9BXAhcF759Ii4Id/tPxi4GthYVv5X4JsR8UKj6++3zgURsS0iXib7Aft+Se8om+XnEXF3Xv868FeSxgEnAmsj4vsRsSMifgvcDHyqynreGRH3FPTxP/lu/yjgGzRx76YTOPxlImJNRJwREQeQHeO+C7gcIN+F/7GkdflI+COy/xTlyoOxvcL3w/vN/2TZ8yfy9fU3HvhUPko9l49Ek4ExFeati6SDyEbbr0TEryrNExGPkO0VXJUv8wlgRET8pFl95K+7p6RLJT2Wb+e1eal8W/9pu+U/eLaQbbvxwAf7bavPAvs30lNEbAEWAosrnfDcVe02b6TZIuIP+dnxL+ST/olsr+B9EbFF0klkx8iNGMfro8mBZCfn+nsS+GFE/H2D66oo33X/JXBJRPywxuxDgAn5848AJUkb8u/fQXbY9L6ImNZAS58BpgEfJQv+O4BnyQ67+owr63842eHQ02Tb6q6IOK6B9VczhOxQ4u1kP2x2eR75c/mJonMlHZB/P45sN/y+fJYRwAvA8/lx+HmVX2mnfFPS2/ITWmcClUbRHwGfkPSxfFR8i6Rj+vqsRdIQSW8B9gT6lh+S18YCdwBXRsTVFZY9S9J++fPDgAuAFX29kx0KHJ4/lpAdLp054HcPQ/J++h57kW3nl8lOUL6N7Iduf1OVXZbdm+zY/76IeJLsas3Bkj4naa/8caSkQ3eip773/jeSDpG0R35+5TLgt/lewG7B4X/dNuCDwP2SXiQL/UNkZ7Mhu+x3BPA82Qm4W5qwzruAR8kC9Z2IuL3/DPl/6mlkJw57yUa388j/7STNknRbwTq+QXbIcT7Z+YPt+TSAs4D3ABdLeqHvUbbsUcCqfHssyx+z8r62RcSGvkf+ui/uZDjm5sv1Pb4P/IDsEGgd2eW2+yosdwNwEdkI/Jf5+yIitpGdPJ1OtiewAfgXYGillefv96+r9DYW+AXZ/4tVZFc6Tt6J99bx5D/mMfgkdQP/DewVETva242lyiO/WaIcfrNEebffLFEe+c0SNajX+UeNGhXd3d2DuUqzpKxdu5bNmzer9pwNhl/SFLJPdO0JXBsRlxbN393dTU9PTyOrNLMCpVJpwPPWvdsvaU/g34ATgMOAU6t9gMLMOk8jx/yTgEcj4vGIeAX4Mdkvo5jZLqCR8I/ljR9MeSqf9gb5Z9Z7JPX09vY2sDoza6aWn+2PiHkRUYqIUldXV6tXZ2YD1Ej411H26SrggHyame0CGgn/r4GJkt6df7pqOtknu8xsF1D3pb6I2CHpHOA/yC71LYiI1TUWM7MO0dB1/ojo+5inme1i/Ou9Zoly+M0S5fCbJcrhN0uUw2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5fCbJcrhN0uUw2+WKN+iezd39tlnF9bnzp1bWL/wwgsL66eddlphfeLEiYV1ax+P/GaJcvjNEuXwmyXK4TdLlMNvliiH3yxRDr9ZonydP3FS8d2cZ8+eXVi/6aabCuvXXHNN1dqRRx5ZuOzQoUML69YYj/xmiXL4zRLl8JslyuE3S5TDb5Yoh98sUQ6/WaJ8nX83d+aZZza0/Pz58wvrDz/8cGH96KOPrlpbs2ZN4bIHH3xwYd0a01D4Ja0FtgGvAjsiotSMpsys9Zox8h8bEZub8DpmNoh8zG+WqEbDH8Dtkn4jaUalGSTNkNQjqae3t7fB1ZlZszQa/skRcQRwAvBFSR/uP0NEzIuIUkSUurq6GlydmTVLQ+GPiHX5103AImBSM5oys9arO/yShkka0fccOB54qFmNmVlrNXK2fzSwKP88+BDghoj4RVO6sqap9Zn5WvXhw4cX1ufMmbPTPfU577zzCuuLFy+u+7WttrrDHxGPA+9vYi9mNoh8qc8sUQ6/WaIcfrNEOfxmiXL4zRLlj/RaoUsuuaSw/ta3vrWwXvSnv++4447CZe+8887C+rHHHltYt2Ie+c0S5fCbJcrhN0uUw2+WKIffLFEOv1miHH6zRPk6vxWqdZvsM844o7BedJ3/pZdeKlx2+/bthXVrjEd+s0Q5/GaJcvjNEuXwmyXK4TdLlMNvliiH3yxRvs5vhS6//PLC+oIFC+p+7UMPPbSwfsghh9T92labR36zRDn8Zoly+M0S5fCbJcrhN0uUw2+WKIffLFG+zr8bWL58edXalVdeWbjsXXfdVViv9Zn6HTt2FNaLTJgwoaG6NabmyC9pgaRNkh4qmzZS0nJJj+Rf92ltm2bWbAPZ7b8OmNJv2vnAioiYCKzIvzezXUjN8EfE3cCWfpOnAQvz5wuBk5rcl5m1WL0n/EZHxPr8+QZgdLUZJc2Q1COpp7e3t87VmVmzNXy2PyICiIL6vIgoRUSpq6ur0dWZWZPUG/6NksYA5F83Na8lMxsM9YZ/CXB6/vx0YHFz2jGzwVLzOr+kG4FjgFGSngIuAi4FbpL0eeAJ4JRWNmnFiv42/j333FO4bHbUVp2kwvqIESMK60uXLq1a23fffQuXtdaqGf6IOLVK6SNN7sXMBpF/vdcsUQ6/WaIcfrNEOfxmiXL4zRLlj/RaQ1555ZXC+jPPPFO1Nnny5Ga3YzvBI79Zohx+s0Q5/GaJcvjNEuXwmyXK4TdLlMNvlihf598N1Prz20XOPvvswvqGDRsK67feemth/eSTT65aO/HEEwuXXbJkSWHdGuOR3yxRDr9Zohx+s0Q5/GaJcvjNEuXwmyXK4TdLlK/zJ+6qq64qrL/44ouF9enTpxfWly1bVrX27LPPFi67ZUv/W0S+0ciRIwvrVswjv1miHH6zRDn8Zoly+M0S5fCbJcrhN0uUw2+WKF/nt0LDhg0rrM+cObOwXnSd/9577y1c9r777iusT506tbBuxWqO/JIWSNok6aGyaRdLWifpwfzhfwWzXcxAdvuvA6ZUmP7diDg8f1T/8W5mHalm+CPibqD49yzNbJfTyAm/cyStzA8L9qk2k6QZknok9fT29jawOjNrpnrDPxeYABwOrAfmVJsxIuZFRCkiSl1dXXWuzsyara7wR8TGiHg1Il4DrgEmNbctM2u1usIvaUzZtycDD1Wb18w6U83r/JJuBI4BRkl6CrgIOEbS4UAAa4EvtLBH62ClUqndLVidaoY/Ik6tMHl+C3oxs0HkX+81S5TDb5Yoh98sUQ6/WaIcfrNE+SO9g2D79u2F9Vofi50zp+ovUAIwfPjwne6pWVatWtW2dVtjPPKbJcrhN0uUw2+WKIffLFEOv1miHH6zRDn8Zonydf4mqHUd/4ILLiisX3vttYX1/fffv7A+a9asqrWhQ4cWLtuoq6++uu5lJ00q/hsw/rhwa3nkN0uUw2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5ev8TbBixYrC+hVXXNHQ68+ePbuwftxxx1WtTZ48uXDZot8RGIiVK1fWvexZZ51VWN9vv/3qfm2rzSO/WaIcfrNEOfxmiXL4zRLl8JslyuE3S5TDb5aogdyiexzwA2A02S2550XE9ySNBH4CdJPdpvuUiHi2da12rilTphTWH3vsscL6Jz/5ycL66tWrC+sf//jHq9b22KP45/vzzz9fWJdUWG/E8ccf37LXttoGMvLvAM6NiMOADwFflHQYcD6wIiImAivy781sF1Ez/BGxPiIeyJ9vA9YAY4FpwMJ8toXASa1q0syab6eO+SV1Ax8A7gdGR8T6vLSB7LDAzHYRAw6/pOHAzcDMiNhaXouIIDsfUGm5GZJ6JPX09vY21KyZNc+Awi9pL7LgXx8Rt+STN0oak9fHAJsqLRsR8yKiFBGlrq6uZvRsZk1QM/zKTvfOB9ZExGVlpSXA6fnz04HFzW/PzFplIB/pPQr4HLBK0oP5tFnApcBNkj4PPAGc0poWO9+QIcWbsbu7u7D+s5/9rLC+aNGiwvpFF11UtbZ169aqtWY48MADC+uf/vSnq9b8kd32qhn+iLgHqHax9yPNbcfMBot/w88sUQ6/WaIcfrNEOfxmiXL4zRLl8Jslyn+6uwOMHz++sD5z5szC+t5771219qUvfamunvpMnDixsL506dLC+kEHHdTQ+q11PPKbJcrhN0uUw2+WKIffLFEOv1miHH6zRDn8ZolS9he4BkepVIqenp5BW59ZakqlEj09PQP6e+se+c0S5fCbJcrhN0uUw2+WKIffLFEOv1miHH6zRDn8Zoly+M0S5fCbJcrhN0uUw2+WKIffLFEOv1miHH6zRNUMv6Rxku6U9HtJqyV9JZ9+saR1kh7MH1Nb366ZNctAbtqxAzg3Ih6QNAL4jaTlee27EfGd1rVnZq1SM/wRsR5Ynz/fJmkNMLbVjZlZa+3UMb+kbuADwP35pHMkrZS0QNI+VZaZIalHUk9vb29DzZpZ8ww4/JKGAzcDMyNiKzAXmAAcTrZnMKfSchExLyJKEVHq6upqQstm1gwDCr+kvciCf31E3AIQERsj4tWIeA24BpjUujbNrNkGcrZfwHxgTURcVjZ9TNlsJwMPNb89M2uVgZztPwr4HLBK0oP5tFnAqZIOBwJYC3yhJR2aWUsM5Gz/PUClvwO+rPntmNlg8W/4mSXK4TdLlMNvliiH3yxRDr9Zohx+s0Q5/GaJcvjNEuXwmyXK4TdLlMNvliiH3yxRDr9Zohx+s0QpIgZvZVIv8ETZpFHA5kFrYOd0am+d2he4t3o1s7fxETGgv5c3qOF/08qlnogota2BAp3aW6f2Be6tXu3qzbv9Zoly+M0S1e7wz2vz+ot0am+d2he4t3q1pbe2HvObWfu0e+Q3szZx+M0S1ZbwS5oi6Y+SHpV0fjt6qEbSWkmr8tuO97S5lwWSNkl6qGzaSEnLJT2Sf614j8Q29dYRt20vuK18W7ddp93uftCP+SXtCTwMHAc8BfwaODUifj+ojVQhaS1Qioi2/0KIpA8DLwA/iIj35tO+DWyJiEvzH5z7RMTXOqS3i4EX2n3b9vxuUmPKbysPnAScQRu3XUFfp9CG7daOkX8S8GhEPB4RrwA/Bqa1oY+OFxF3A1v6TZ4GLMyfLyT7zzPoqvTWESJifUQ8kD/fBvTdVr6t266gr7ZoR/jHAk+Wff8UbdwAFQRwu6TfSJrR7mYqGB0R6/PnG4DR7Wymgpq3bR9M/W4r3zHbrp7b3TebT/i92eSIOAI4AfhivnvbkSI7Zuuka7UDum37YKlwW/k/aee2q/d2983WjvCvA8aVfX9APq0jRMS6/OsmYBGdd+vxjX13SM6/bmpzP3/SSbdtr3RbeTpg23XS7e7bEf5fAxMlvVvS3sB0YEkb+ngTScPyEzFIGgYcT+fdenwJcHr+/HRgcRt7eYNOuW17tdvK0+Zt13G3u4+IQX8AU8nO+D8GfL0dPVTp6z3A7/LH6nb3BtxIthv4f2TnRj4P7AusAB4BfgmM7KDefgisAlaSBW1Mm3qbTLZLvxJ4MH9Mbfe2K+irLdvNv95rliif8DNLlMNvliiH3yxRDr9Zohx+s0Q5/GaJcvjNEvX/W2kYY0HAHQsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def display_sample(num):\n",
    "    #Print the one-hot array of this sample's label \n",
    "    print(train_labels[num])  \n",
    "    #Print the label converted back to a number\n",
    "    label = train_labels[num].argmax(axis=0)\n",
    "    #Reshape the 768 values to a 28x28 image\n",
    "    image = train_images[num].reshape([28,28])\n",
    "    plt.title('Sample: %d  Label: %d' % (num, label))\n",
    "    plt.imshow(image, cmap=plt.get_cmap('gray_r'))\n",
    "    plt.show()\n",
    "    \n",
    "display_sample(1234)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for the meat of the problem. Setting up a convolutional neural network involves more layers.\n",
    "\n",
    "We'll start with a 2D convolution of the image - it's set up to take 32 windows, or \"filters\", of each image, each filter being 3x3 in size.\n",
    "\n",
    "We then run a second convolution on top of that with 64 3x3 windows - this topology is just what comes recommended within Keras's own examples. Again you want to re-use previous research whenever possible while tuning CNN's, as it is hard to do.\n",
    "\n",
    "Next we apply a MaxPooling2D layer that takes the maximum of each 2x2 result to distill the results down into something more manageable.\n",
    "\n",
    "Next we flatten the 2D layer we have at this stage into a 1D layer. So at this point we can just pretend we have a traditional multi-layer perceptron...\n",
    "\n",
    "... and feed that into a hidden, flat layer of 128 units.\n",
    "\n",
    "\n",
    "And finally, we feed that into our final 10 units where softmax is applied to choose our category of 0-9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0925 16:31:45.127260 140268829603584 deprecation.py:506] From /home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "# 64 3x3 kernels\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "# Reduce by taking the max of each 2x2 block\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "# Flatten the results to one dimension for passing into our final layer\n",
    "model.add(Flatten())\n",
    "# A hidden layer to learn with\n",
    "model.add(Dense(128, activation='relu'))\n",
    "# Final categorization from 0-9 with softmax\n",
    "model.add(Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's double check the model description:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 24, 24, 64)        18496     \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 12, 12, 64)        0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 9216)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 128)               1179776   \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 1,199,882\n",
      "Trainable params: 1,199,882\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are doing multiple categorization, so categorical_crossentropy is still the right loss function to use. We'll use the Adam optimizer, although the example provided with Keras uses RMSProp. You might want to try both if you have time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we train our model... to make things go a little faster, we'll use batches of 32.\n",
    "\n",
    "## Warning\n",
    "\n",
    "This will take a few minutes to run on a p3.large instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 - 13s - loss: 0.1098 - acc: 0.9671 - val_loss: 0.0340 - val_acc: 0.9885\n",
      "Epoch 2/10\n",
      "60000/60000 - 12s - loss: 0.0379 - acc: 0.9875 - val_loss: 0.0295 - val_acc: 0.9904\n",
      "Epoch 3/10\n",
      "60000/60000 - 12s - loss: 0.0236 - acc: 0.9928 - val_loss: 0.0340 - val_acc: 0.9888\n",
      "Epoch 4/10\n",
      "60000/60000 - 12s - loss: 0.0157 - acc: 0.9950 - val_loss: 0.0371 - val_acc: 0.9886\n",
      "Epoch 5/10\n",
      "60000/60000 - 12s - loss: 0.0116 - acc: 0.9961 - val_loss: 0.0409 - val_acc: 0.9889\n",
      "Epoch 6/10\n",
      "60000/60000 - 12s - loss: 0.0087 - acc: 0.9969 - val_loss: 0.0438 - val_acc: 0.9897\n",
      "Epoch 7/10\n",
      "60000/60000 - 12s - loss: 0.0068 - acc: 0.9978 - val_loss: 0.0520 - val_acc: 0.9879\n",
      "Epoch 8/10\n",
      "60000/60000 - 12s - loss: 0.0068 - acc: 0.9979 - val_loss: 0.0445 - val_acc: 0.9899\n",
      "Epoch 9/10\n",
      "60000/60000 - 12s - loss: 0.0059 - acc: 0.9979 - val_loss: 0.0522 - val_acc: 0.9897\n",
      "Epoch 10/10\n",
      "60000/60000 - 12s - loss: 0.0043 - acc: 0.9984 - val_loss: 0.0742 - val_acc: 0.9857\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(train_images, train_labels,\n",
    "                    batch_size=32,\n",
    "                    epochs=10,\n",
    "                    verbose=2,\n",
    "                    validation_data=(test_images, test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, you can see that we started overfitting pretty early on, as our accuracy on the test set started exceeding our accuracy on the validation set. Our validation accuracy maxed out at around 99.0% after just a couple of epochs, while our accuracy on the test set kept climbing.\n",
    "\n",
    "To prevent overfitting, we need to perform some sort of regularization. Dropout layers are one such technique in deep learning; they work by \"dropping out\" neurons on each pass to force learning to spread itself out across the network as a whole."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def MakeModel():\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                     activation='relu',\n",
    "                     input_shape=input_shape))\n",
    "    # 64 3x3 kernels\n",
    "    model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "    # Reduce by taking the max of each 2x2 block\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    # Dropout to avoid overfitting\n",
    "    model.add(Dropout(0.25))\n",
    "    # Flatten the results to one dimension for passing into our final layer\n",
    "    model.add(Flatten())\n",
    "    # A hidden layer to learn with\n",
    "    model.add(Dense(128, activation='relu'))\n",
    "    # Another dropout\n",
    "    model.add(Dropout(0.5))\n",
    "    # Final categorization from 0-9 with softmax\n",
    "    model.add(Dense(10, activation='softmax'))\n",
    "    return model\n",
    "\n",
    "model = MakeModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run it again with those two dropout layers added in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 - 14s - loss: 0.2002 - acc: 0.9395 - val_loss: 0.0430 - val_acc: 0.9860\n",
      "Epoch 2/10\n",
      "60000/60000 - 13s - loss: 0.0801 - acc: 0.9756 - val_loss: 0.0332 - val_acc: 0.9889\n",
      "Epoch 3/10\n",
      "60000/60000 - 13s - loss: 0.0638 - acc: 0.9808 - val_loss: 0.0299 - val_acc: 0.9905\n",
      "Epoch 4/10\n",
      "60000/60000 - 13s - loss: 0.0495 - acc: 0.9844 - val_loss: 0.0310 - val_acc: 0.9901\n",
      "Epoch 5/10\n",
      "60000/60000 - 13s - loss: 0.0424 - acc: 0.9867 - val_loss: 0.0294 - val_acc: 0.9916\n",
      "Epoch 6/10\n",
      "60000/60000 - 13s - loss: 0.0371 - acc: 0.9887 - val_loss: 0.0242 - val_acc: 0.9928\n",
      "Epoch 7/10\n",
      "60000/60000 - 13s - loss: 0.0337 - acc: 0.9900 - val_loss: 0.0338 - val_acc: 0.9902\n",
      "Epoch 8/10\n",
      "60000/60000 - 13s - loss: 0.0299 - acc: 0.9908 - val_loss: 0.0286 - val_acc: 0.9922\n",
      "Epoch 9/10\n",
      "60000/60000 - 13s - loss: 0.0271 - acc: 0.9918 - val_loss: 0.0290 - val_acc: 0.9916\n",
      "Epoch 10/10\n",
      "60000/60000 - 13s - loss: 0.0267 - acc: 0.9915 - val_loss: 0.0280 - val_acc: 0.9934\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(train_images, train_labels,\n",
    "                    batch_size=32,\n",
    "                    epochs=10,\n",
    "                    verbose=2,\n",
    "                    validation_data=(test_images, test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's better; our train and test accuracy ended up about the same, at 99.2%. There may still be a tiny bit of overfitting going on, but it's a lot better.\n",
    "\n",
    "Let's also explore the effect the batch size has; as an experiment, let's increase it up to 1000:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 - 5s - loss: 0.5228 - acc: 0.8367 - val_loss: 0.1356 - val_acc: 0.9591\n",
      "Epoch 2/10\n",
      "60000/60000 - 5s - loss: 0.1576 - acc: 0.9541 - val_loss: 0.0656 - val_acc: 0.9791\n",
      "Epoch 3/10\n",
      "60000/60000 - 5s - loss: 0.1021 - acc: 0.9701 - val_loss: 0.0520 - val_acc: 0.9828\n",
      "Epoch 4/10\n",
      "60000/60000 - 5s - loss: 0.0794 - acc: 0.9770 - val_loss: 0.0429 - val_acc: 0.9857\n",
      "Epoch 5/10\n",
      "60000/60000 - 5s - loss: 0.0675 - acc: 0.9805 - val_loss: 0.0351 - val_acc: 0.9872\n",
      "Epoch 6/10\n",
      "60000/60000 - 5s - loss: 0.0574 - acc: 0.9819 - val_loss: 0.0343 - val_acc: 0.9875\n",
      "Epoch 7/10\n",
      "60000/60000 - 5s - loss: 0.0513 - acc: 0.9842 - val_loss: 0.0340 - val_acc: 0.9888\n",
      "Epoch 8/10\n",
      "60000/60000 - 5s - loss: 0.0456 - acc: 0.9858 - val_loss: 0.0340 - val_acc: 0.9886\n",
      "Epoch 9/10\n",
      "60000/60000 - 5s - loss: 0.0422 - acc: 0.9870 - val_loss: 0.0324 - val_acc: 0.9885\n",
      "Epoch 10/10\n",
      "60000/60000 - 5s - loss: 0.0384 - acc: 0.9878 - val_loss: 0.0305 - val_acc: 0.9899\n"
     ]
    }
   ],
   "source": [
    "model = MakeModel()\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "history = model.fit(train_images, train_labels,\n",
    "                    batch_size=1000,\n",
    "                    epochs=10,\n",
    "                    verbose=2,\n",
    "                    validation_data=(test_images, test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you run this block a few times, you'll probably get very different results. Large batch sizes tend to get stuck in \"local minima\", and converge on the wrong solution at random. Smaller batch sizes also have a regularization effect. Sometimes you'll get lucky and the large batch will converge on a good solution; other times, not so much.\n",
    "\n",
    "Let's explore the effect of the learning rate. The default learning rate for Adam is 0.001; let's see what happens if we increase it by an order of magnitude to 0.01:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = MakeModel()\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "adam = tensorflow.keras.optimizers.Adam(learning_rate=0.01)\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=adam,\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 - 14s - loss: 0.6022 - acc: 0.8092 - val_loss: 0.2194 - val_acc: 0.9310\n",
      "Epoch 2/10\n",
      "60000/60000 - 13s - loss: 0.4404 - acc: 0.8649 - val_loss: 0.1900 - val_acc: 0.9447\n",
      "Epoch 3/10\n",
      "60000/60000 - 13s - loss: 0.4128 - acc: 0.8720 - val_loss: 0.1991 - val_acc: 0.9378\n",
      "Epoch 4/10\n",
      "60000/60000 - 13s - loss: 0.3939 - acc: 0.8781 - val_loss: 0.1523 - val_acc: 0.9529\n",
      "Epoch 5/10\n",
      "60000/60000 - 13s - loss: 0.3925 - acc: 0.8792 - val_loss: 0.1614 - val_acc: 0.9504\n",
      "Epoch 6/10\n",
      "60000/60000 - 13s - loss: 0.3821 - acc: 0.8848 - val_loss: 0.1778 - val_acc: 0.9453\n",
      "Epoch 7/10\n",
      "60000/60000 - 13s - loss: 0.3787 - acc: 0.8839 - val_loss: 0.1638 - val_acc: 0.9515\n",
      "Epoch 8/10\n",
      "60000/60000 - 13s - loss: 0.3734 - acc: 0.8856 - val_loss: 0.1787 - val_acc: 0.9491\n",
      "Epoch 9/10\n",
      "60000/60000 - 13s - loss: 0.3688 - acc: 0.8893 - val_loss: 0.1564 - val_acc: 0.9522\n",
      "Epoch 10/10\n",
      "60000/60000 - 13s - loss: 0.3652 - acc: 0.8899 - val_loss: 0.1497 - val_acc: 0.9530\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(train_images, train_labels,\n",
    "                    batch_size=32,\n",
    "                    epochs=10,\n",
    "                    verbose=2,\n",
    "                    validation_data=(test_images, test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yikes! That had a huge, and terrible, effect on the results. Small batch sizes are best paired with low learning rates, and large learning rates have a tendency to overshoot the correct solution entirely - which is probably what happened here. The learning rate is an example of a hyperparameter that you might want to tune by just trying different values; we'll see more of that later in the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}