03-999-03-Gradient Descent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient Descent\n",
    "\n",
    "Inspired by [Chris McCormick on Gradient Descent Derivation](http://mccormickml.com/2014/03/04/gradient-descent-derivation/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Background\n",
    "\n",
    "$h(x) = \\theta_0 + \\theta_1X$\n",
    "\n",
    "Find the values of $\\theta_0$ and $\\theta_1$ which provide the best fit of our hypothesis to a training set. \n",
    "\n",
    "The training set examples are labeled $x$, $y$, \n",
    "\n",
    "$x$ is the input value and $y$ is the output. \n",
    "\n",
    "The $i$th training example is labeled as $x^{(i)}$, $y^{(i)}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MSE Cost Function\n",
    "\n",
    "The cost function $J$ for a particular choice of parameters $\\theta$ is the mean squared error (MSE):\n",
    "\n",
    "$$J(\\theta)=\\frac{1}{m}\\sum_{i=1}^m(h_{\\theta}(x^{(i)})-y^{(i)})^2$$\n",
    "\n",
    "$m$ The number of training examples\n",
    "\n",
    "$x^{(i)}$ The input vector for the $i^{th}$ training example\n",
    "\n",
    "$y^{(i)}$ The class label for the $i^{th}$ training example\n",
    "\n",
    "$\\theta$ The chosen parameter values of \"weights\" ($\\theta_0, \\theta_1, \\theta_2$)\n",
    "\n",
    "$h_{\\theta}(x^{(i)})$ The algorithm's prediction for the $i^{th}$ training example using the parameters $\\theta$\n",
    "\n",
    "The MSE measures the mean amount that the model's predictions deviate from the correct values.\n",
    "\n",
    "It is a measure of the model's performance on the training set. \n",
    "\n",
    "The cost is higher when the model is performing poorly on the training set. \n",
    "\n",
    "The objective of the learning algorithm is to find the parameters $\\theta$ which give the minimum possible cost $J$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This minimization objective is expressed using the following notation, which simply states that we want to find the $\\theta$ which minimizes the cost $J(\\theta)$.\n",
    "\n",
    "$$\\min_{\\theta}J(\\theta)$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example One"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient Descent Minimization\n",
    "\n",
    "Assume the cost function is:\n",
    "\n",
    "$$J(\\theta)=\\theta^2$$\n",
    "\n",
    "And we want to find the value of $\\theta$ which minimises $J(\\theta)$\n",
    "\n",
    "Assume further we starts with $\\theta=3$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient descent is an iterative algorithm which we will run many times. \n",
    "\n",
    "On each iteration, we apply the following \"update rule\" (the $:=$ symbol means replace $\\theta$ with the value computed on the right):\n",
    "\n",
    "$$\\theta:=\\theta-\\alpha\\frac{d}{d\\theta}J(\\theta)$$\n",
    "\n",
    "Alpha is the learning rate. \n",
    "\n",
    "Assume $\\alpha$ = 0.1 for now.\n",
    "\n",
    "$$\\frac{d}{d\\theta}J(\\theta)=2\\theta$$\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set_style(\"whitegrid\")\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.14.3\n",
      "0.23.0\n",
      "3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:44:09) \n",
      "[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]\n",
      "0.19.2\n"
     ]
    }
   ],
   "source": [
    "print(np.__version__)\n",
    "print(pd.__version__)\n",
    "import sys\n",
    "print(sys.version)\n",
    "import sklearn\n",
    "print(sklearn.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.0000 0.6000\n",
      "2.4000 0.4800\n",
      "1.9200 0.3840\n",
      "1.5360 0.3072\n",
      "1.2288 0.2458\n",
      "0.9830 0.1966\n",
      "0.7864 0.1573\n",
      "0.6291 0.1258\n",
      "0.5033 0.1007\n",
      "0.4027 0.0805\n"
     ]
    }
   ],
   "source": [
    "theta = 3\n",
    "alpha = 0.1\n",
    "dat = []\n",
    "for oo in range(0,10):\n",
    "    res = alpha * 2 * theta # update rule\n",
    "    print(\"{0:.4f} {1:.4f}\".format(theta, res))\n",
    "    dat.append([theta, theta ** 2])\n",
    "    theta = theta - res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>9.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.400000</td>\n",
       "      <td>5.760000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.920000</td>\n",
       "      <td>3.686400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.536000</td>\n",
       "      <td>2.359296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.228800</td>\n",
       "      <td>1.509949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.983040</td>\n",
       "      <td>0.966368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.786432</td>\n",
       "      <td>0.618475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.629146</td>\n",
       "      <td>0.395824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.503316</td>\n",
       "      <td>0.253327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.402653</td>\n",
       "      <td>0.162130</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1\n",
       "0  3.000000  9.000000\n",
       "1  2.400000  5.760000\n",
       "2  1.920000  3.686400\n",
       "3  1.536000  2.359296\n",
       "4  1.228800  1.509949\n",
       "5  0.983040  0.966368\n",
       "6  0.786432  0.618475\n",
       "7  0.629146  0.395824\n",
       "8  0.503316  0.253327\n",
       "9  0.402653  0.162130"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp = pd.DataFrame(dat)\n",
    "tmp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAHaCAYAAADc9jeSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4VFXixvF3SpIhpBN6r5fQSZAmxYZi2VVR10UQK4p9V13LuuoWV9e1ratiAURBEdeCiqwNFZEiyNAhXEpCCS3UkMKkzf39AbrADzDAzNwp38/z8JjMJHNeDyF5c+695zosyxIAAAACx2l3AAAAgGhDwQIAAAgwChYAAECAUbAAAAACjIIFAAAQYG67AxzK6/VySSMAAIgYOTk5jqM9HlYFS5JycnKC+vq5ubnKysoK6hixhjkNLOYz8JjTwGI+A485DbxQzKnX6z3mcxwiBAAACDAKFgAAQIBRsAAAAAKMggUAABBgFCwAAIAAo2ABAAAEGAULAAAgwChYAAAAAUbBAgAACDAKFgAAQIBRsAAAAAKMggUAABBgFCwAAIAAo2ABAAAEGAULAAAgwNzBemHDMHpJetI0zTMMw6gnaYykdEkuSSNM01wXrLEBAADsFJQVLMMw7pM0VpLn4EP/lPS2aZoDJP1JUvtgjAsAABAOgnWIcJ2kIYe8f7qkJoZhTJc0TNKMII0LAABiWGW1XxPnrldJRbWtORyWZQXlhQ3DaCFpsmmavQ3DqJR0k2ma4w3DeESS2zTNR478HK/XayUmJgYlz098Pp88Hs8vfyBqjDkNLOYz8JjTwGI+A485DZzp64r1zKwdempQpjo1SgnqWGVlZcrJyXEc7bmgnYN1hF2SPjn49lRJfz/WB2ZlZQU1SG5ubtDHiDXMaWAxn4HHnAYW8xl4zGlgWJal+6bPVtt6SerYMDnoc+r1eo/5XKiuIpwl6YKDbw+QtCJE4wIAgBjh3bBHyzYX6drTW8jhOOrCUsiEqmDdI2mEYRhzJA2W9HiIxgUAADFi/Jz1SvG4dWn3xnZHCd4hQtM010vqffDtDZIGBWssAAAQ27YW7dfny7fphn4tlRgfqjOgjo2NRgEAQMSbOHeDLMvS1b2b2x1FEgULAABEOF9ltd6Zv1GDOtRX04zg7kZQUxQsAAAQ0T5evFl7yip1bd+Wdkf5GQULAABELMuyNH72erVvkKzerTLsjvMzChYAAIhY8/J3a9W2Yl0XBlszHIqCBQAAItb42flKT4zTxd3s35rhUBQsAAAQkTbtLtNXK7draM9m8sS57I5zGAoWAACISBN/2CCHw6HhYbI1w6EoWAAAIOKUllfpnfkbdUHnhmqUVsvuOP8PBQsAAESc970FKvZV6frTW9gd5agoWAAAIKL4/ZbGz85X92Zp6t4s3e44R0XBAgAAEeVbs1Drd5Xp+tPDZ2PRI1GwAABARHl9dr4apno0uFMDu6McEwULAABEjFXb9mn22l0a0aeF4lzhW2PCNxkAAMARxs9aL0+cU0N7NrU7ynFRsAAAQETYVVKuKYs367LsJkpLjLc7znFRsAAAQESYNG+jKqr8ui5Mt2Y4FAULAACEvYoqvyb8sEED29VVm3rJdsf5RRQsAAAQ9qYt26IdxeW6vl/4bs1wKAoWAAAIa5Zlaez3+WpTL0n922TaHadGKFgAACCszcvfrRVb9umGfi3ldDrsjlMjFCwAABDWxn6fr4za8bq0e2O7o9QYBQsAAIStvB0l+nrVdg3v3VyeOJfdcWqMggUAAMLW+NnrFed06ureze2OckIoWAAAICztLavQ+94CXdytkeomJ9gd54RQsAAAQFiaNH+j9ldW64b+kbE1w6EoWAAAIOxUVPn15pz16t82U+0bpNgd54RRsAAAQNiZtmyLtu8r1w0RsrHokShYAAAgrBy6sejAdnXtjnNSKFgAACCsHLqxqMMRGRuLHomCBQAAwsrY7/MibmPRI1GwAABA2FhbWKLpuYW6OsI2Fj0SBQsAAISNcbPyleB26uo+kbWx6JEoWAAAICzsLCnXBwsLdFlOE2UmRdbGokeiYAEAgLAwYe4GVVT5I3ZrhkNRsAAAgO32V1Rr4tz1OiervlrXTbI7zimjYAEAANt9sLBAe8oqNTICb4tzNBQsAABgq2q/pXGz8tW1Sap6tsywO05AULAAAICtpuduV/7OUo0c0CpiNxY9EgULAADYauz3eWqcVkuDOzawO0rAULAAAIBtFm7cox/X79EN/VrK7YqeWhI9/ycAACDijP0+T8ket35zWlO7owRU0AqWYRi9DMOYccRjVxmGMTdYYwIAgMixfmepPlu+TcN7N1dSgtvuOAEVlP8bwzDuk3S1pNJDHusm6QZJ0XH2GgAAOCVjZ+UpzunUdX1b2B0l4IK1grVO0pCf3jEMo46kf0j6XZDGAwAAEWRnSbneW1CgS7s3Vr0Uj91xAs5hWVZQXtgwjBaSJks6XdIHkh6UtF/SZNM0ex/tc7xer5WYmBiUPD/x+XzyeKLvL9JOzGlgMZ+Bx5wGFvMZeLE4pxMX79akJXv12iVN1DQ1PuCvH4o5LSsrU05OzlGPzIXigGeOpLaSXpbkkdTBMIx/maZ51NWsrKysoIbJzc0N+hixhjkNLOYz8JjTwGI+Ay/W5rSsokqfvfeNzsmqr3N7dw3KGKGYU6/Xe8zngl6wTNOcL6mj9L9VrWOVKwAAEP3eW3DgtjijBrayO0rQsE0DAAAImapqv8bOylN2szT1aBEdt8U5mqCtYJmmuV5S7196DAAAxI7Plm/Tpt379dAFHeyOElSsYAEAgJCwLEuvzcxTq8zaGtShvt1xgoqCBQAAQmLuul1atrlIN/ZvJZczurfFpGABAICQeGVmnjKT4jUku7HdUYKOggUAAIJuxZYizVy9Q9ed3lKeOJfdcYKOggUAAILu1e/ylJTg1vDeze2OEhIULAAAEFQbd5Xp06VbdFWvZkqtFWd3nJCgYAEAgKAa832e3E6nbujX0u4oIUPBAgAAQbOzpFz/WbBJl3ZvrPpReFPnY6FgAQCAoHlj9npVVPt1UxTfFudoKFgAACAoSsqrNGHuep3XoYFa102yO05IUbAAAEBQvDNvo/b5qjTqjNZ2Rwk5ChYAAAi48qpqjZ2Vpz6t6qhb0zS744QcBQsAAATcx4u2aPu+ct0Sg6tXEgULAAAEmN9v6dWZ69SxUYr6t820O44tKFgAACCgvlixTet2lGrUwNZyOKL7ps7HQsECAAABY1mWRs9YpxZ1EnVB54Z2x7ENBQsAAATMrLU7tWxzkUYNbC2XMzZXryQKFgAACKCXvl2rBikeXZrd2O4otqJgAQCAgPBu2KMf8nbrxv4tleB22R3HVhQsAAAQEC/PWKu0xDgN7dnM7ii2o2ABAIBTtmrbPk3PLdR1fVuqdoLb7ji2o2ABAIBT9vKMdaod79I1fZvbHSUsULAAAMAp2birTFOXbNGw3s2Vlhhvd5ywQMECAACn5NWZ6+R2OnVDv5Z2RwkbFCwAAHDStu/z6T1vgS7LaaL6KR6744QNChYAADhpY2bmqdpv6ZaBsXlT52OhYAEAgJOyu7RCb8/bqIu7NlKzOol2xwkrFCwAAHBSXp+VL19VtW49k9WrI1GwAADACSvaX6k356zX+Z0aqE29ZLvjhB0KFgAAOGET565XcXmVbj2jjd1RwhIFCwAAnJCyiiqNm5Wvs9rXU6fGqXbHCUsULAAAcEImzduoPWWVuu1MVq+OhYIFAABqzFdZrddm5qlv6zrKaZ5ud5ywRcECAAA19p63QIXF5bqd1avjomABAIAaqaz265UZ65TdLE19WtexO05Yo2ABAIAambJoszbv3a/bz2ojh8Nhd5ywRsECAAC/qKrar5e+XatOjVN0plHP7jhhj4IFAAB+0SdLtmjDrjLdeVZbVq9qgIIFAACOq9pv6cVv16p9g2QN6lDf7jgRgYIFAACOa9qyrcrbUao7z2b1qqYoWAAA4Jj8fksvfL1GbeslaXDHBnbHiRjuYL2wYRi9JD1pmuYZhmF0k/SCpGpJ5ZJGmKa5PVhjAwCAwPh8xTatKSzR87/tJqeT1auaCsoKlmEY90kaK8lz8KHnJd1hmuYZkj6UdH8wxgUAAIHj91v699dr1Cqzti7q0sjuOBElWIcI10kacsj7vzVNc/HBt92SfEEaFwAABMj03O1ata1Yt5/VRi5Wr06Iw7KsoLywYRgtJE02TbP3IY/1lTRO0gDTNHcc+Tler9dKTEwMSp6f+Hw+eTyeX/5A1BhzGljMZ+Axp4HFfAZeOM6pZVm689PNKq30a8wlTSOuYIViTsvKypSTk3PUiQnaOVhHMgzjSkkPSbrwaOXqJ1lZWUHNkZubG/QxYg1zGljMZ+Axp4HFfAZeOM7pN6u2a+3ufD15WWd16tjM7jgnLBRz6vV6j/lcSAqWYRjDJd0s6QzTNHeHYkwAAHByLMvSv6avUZP0WhqS3cTuOBEp6Ns0GIbhkvRvScmSPjQMY4ZhGH8J9rgAAODkfGsWamlBke44q43iXOzodDKCtoJlmuZ6ST+df5URrHEAAEDgsHoVGNRSAADwM1avAoOZAwAAkli9CiQKFgAAkCTNMHewehUgzB4AADi4erWa1asAoWABAADNMHdoCatXAcMMAgAQ41i9CjwKFgAAMY7Vq8BjFgEAiGGWZenZr1i9CjQKFgAAMWx6bqGWbS7SnWe3ZfUqgJhJAABilN9/YPWqRZ1EDene2O44UYWCBQBAjPpixTblbt2nu85pKzerVwHFbAIAEIP8fkvPTV+tVnVr69ddWb0KNAoWAAAxaNqyrVq9vUS/O6edXE6H3XGiDgULAIAYU+0/sO9Vu/pJurBzQ7vjRCUKFgAAMWbqki1at6OU1asgomABABBDqqr9ev7rNWrfIFmDOzawO07UomABABBDpizarPydpbp7UDs5Wb0KGgoWAAAxoqLqwOpV58apGtShvt1xohoFCwCAGPHugk0q2LNf95zbTg4Hq1fBRMECACAG+Cqr9eI3a9SjeboGtqtrd5yoR8ECACAGvPXDBm3fV657zzNYvQoBChYAAFGupLxKo2esU/+2merdqo7dcWICBQsAgCj3xux87S6t0D3nGnZHiRkULAAAolhRWaVenZmnc7Lqq1vTNLvjxAwKFgAAUWzM93kq9lXpnnPb2R0lplCwAACIUjtLyvX67Hxd1KWhshqm2B0nplCwAACIUi/PWCdfZbV+P4jVq1CjYAEAEIW27N2viT9s0GXZTdS6bpLdcWIOBQsAgCj0/PQ1kiX9jtUrW1CwAACIMut2lOg97yYN691MjdNq2R0nJlGwAACIMs9+uVqeOJduO7ON3VFiFgULAIAosnxzkaYt26ob+7VUZlKC3XFiFgULAIAo8s8vTKUlxunGAa3sjhLTKFgAAESJH/J2aebqHbr1jNZK8cTZHSemUbAAAIgClmXpn5+vUoMUj0b0aWF3nJhHwQIAIAp8nVuohRv36s6z28oT57I7TsyjYAEAEOGq/Zae/tJUizqJuqJHE7vjQBQsAAAi3keLNmvVtmLde56hOBc/2sMBfwsAAEQwX2W1nv1qtTo3TtUFnRraHQcHUbAAAIhgb/2wQZv37tcD57eX0+mwOw4OomABABCh9vkq9eK3a9W/baZOb5NpdxwcgoIFAECEevW7ddpbVqn7B7e3OwqO4A7WCxuG0UvSk6ZpnmEYRhtJb0iyJC2XdJtpmv5gjQ0AQLTbvs+ncbPy9euujdSpcardcXCEoKxgGYZxn6SxkjwHH3pW0p9M0+wvySHp4mCMCwBArHj+6zWqqrZ0z7nt7I6CowjWCtY6SUMkTTz4fo6k7w6+/ZmkcyVNOdon5ubmBinSAT6fL+hjxBrmNLCYz8BjTgOL+Qy8E53TgqIKTZ5foAuNFJUVblRuYRDDRSi7v06DUrBM0/zAMIwWhzzkME3TOvh2saRjrmVmZWUFI9LPcnNzgz5GrGFOA4v5DDzmNLCYz8A70Tn991teeeJceuSynqqbnBDEZJErFF+nXq/3mM+F6iT3Q8+3Spa0N0TjAgAQVRZu3KPPlm/TyP6tKFdhLFQFa5FhGGccfPt8Sd+HaFwAAKKGZVl6fFquMpMSdNOAVnbHwXEE7SrCI9wjaYxhGPGSciW9H6JxAQCIGl+u3K4FG/bo75d2Uu2EUP0Ix8kI2t+OaZrrJfU++PZqSQODNRYAANGustqvJz9bpdZ1a+vKHk3tjoNfwEajAABEgHd/3KS8naV64Pwsubmhc9jjbwgAgDBXUl6lf01frZ4tMnROVj2746AGKFgAAIS512bmaWdJhR68oL0cDm7oHAkoWAAAhLHCfT6NmZmnC7s0VPdm6XbHQQ1RsAAACGPPTV+tKr9f951n2B0FJ4CCBQBAmFq9vVjv/rhJw3o1V/M6te2OgxNAwQIAIEz9fVqukhLcuuvstnZHwQmiYAEAEIZmrt6h71bv0B1ntVV67Xi74+AEUbAAAAgz1X5Lj/83V80yEjWib3O74+AkULAAAAgz7y3YpFXbinX/4PZKcLvsjoOTQMECACCMlJZX6ZmvViunebou6NzA7jg4SRQsAADCyKvfrdOO4nI9dGEWm4pGMAoWAABhYmvRfr32fZ4u6tJQ2WwqGtEoWAAAhImnv1gtv1+6f3B7u6PgFFGwAAAIA8s3F+nDRQW67vQWapqRaHccnCIKFgAANrMsS3+dulLpifG69cw2dsdBALjtDgAAQCyqqPJr5IQFkqTT6knz1+9W23pJqhXHtgzRgBUsAABsMHLCAs3L26Uf1u3UM7N2yCFp467Sn0sXIhsrWAAA2Ki82vr5bbZliB6sYAEAYIPRw7Llch1eqOLcTr08PNumRAgkChYAADa49e2F8lVUH/ZYZZVft7y10KZECCQOEQIAYINiX6WqLcnlkDxupw45UogoQMECACDELMuS2+mU2+lQr5YZuqdXsp5fUCpJGjOih83pEAgULAAAQuzz5ds0f/1u/e3ijrq6Twvl5ubqzet72h0LAcQ5WAAAhJCvslqPTctV+wbJGtqzmd1xECQULAAAQui1mXnavHe/HvlVB7ld/BiOVvzNAgAQIlv27tfoGWt1fqcG6ts60+44CCIKFgAAIfLEZ6tkWdIfL8iyOwqCjIIFAEAIzM/fralLtujmAa3UNCPR7jgIMgoWAABBVu239OdPVqhhqkejzmhtdxyEwHG3aTAMwyHpQklnSKojqVDS15K+Mk2TLdEAAKiBd3/cpJVb9+nfQ7srMZ4dkmLBMVewDMM4SwfK1EBJSyVNkuSVdJ6k6YZhnBOShAAARLCisko9/aWpni0y9KsuDe2OgxA5Xo1uK2mQaZrVRzz+H8MwXJJukjQ9aMkAAIgCz01frT1lFXr01x3kcDh++RMQFY5ZsEzTfPVojxuGEWeaZqWkl4OWCgCAKJC7dZ8mzF2vYb2aqWOjVLvjIIR+8UCwYRijJN198GMdkqp0YHULAAAcg2VZevSTFUqtFad7zzXsjoMQq8lVhDfqwHlYn0m6TtKKoCYCACAKfLJki+bn79YfzmuvtMR4u+MgxGpSsHaaprlVUrJpmjMkZQQ3EgAAka2kvEqP/zdXnRun6srTmtodBzaoybWiRYZhXCLJMgzjZkl1g5wJAICI9sI3a7R9X7leHp4jl5MT22NRTQ8RbpD0gKR2kkYFNREAABFsbWGJXp+Vrytymii7WbrdcWCTmqxgPWGa5u0H377HMIwJkr4LYiYAACKSZVn6y9QV8sS5dP/57e2OAxsds2AZhnGbpD9JyjAMY4gOXEEoSStDEQwAgEjzxYrt+n7NTv35Vx2UmZRgdxzY6Hj7YL0k6SXDMP5omubjpzqQYRhxkt6U1EJStaSRpmmuOtXXBQAgHJRVVOmvU1eofYNkDe/d3O44sFlNDhG+aBjG3yQ1kjRN0lLTNNeexFgXSHKbptnXMIxBkv4u6bKTeB0AAMLOv79eqy1FPv17aHe5XTU5xRnRrCZfAeMk5evACe7bDr5/MlZLchuG4ZSUIqnyJF8HAICwsmZ7scZ+n6crcpqoRwt2M0LNVrDqmKb5umEYw03TnGMYxsleb1qiA4cHV0nKlHTR0T4oNzf3JF++Znw+X9DHiDXMaWAxn4HHnAYW83k4y7L0wJdb5XE7NKSN66TmhjkNPLvntCYFS4ZhtD/43yY6cP7Uyfi9pC9M03zQMIymkr4xDKOzaZq+Qz8oKyvrJF++ZnJzc4M+RqxhTgOL+Qw85jSwmM/Dfbx4s5Zuy9djl3RSn+4nd+4Vcxp4oZhTr9d7zOdqUrDulDReUpak9yXdepI59uh/hwV3S4qT5DrJ1wIAwHb7fJX626e56tokVUN7NrM7DsLILxYs0zSXS+oTgLGek/S6YRjfS4qX9EfTNEsD8LoAANji2S9Xa1dpuV6/tgc7tuMwv1iwDMN4VNJtkqp+esw0zUYnOpBpmiWSfnOinwcAQDhasaVIE+au1/BezdWlSZrdcRBmanKI8CJJzU3T3B/sMAAARAK/39KfPlqu9MR43XuuYXcchKGabNNQKLZUAADgZ5Pmb9SijXv1p4uylJoYZ3cchKHj3SrnHUmWpPqSFhmGsfzg+zJN86rQxAMAILwUFvv05Oer1Ld1HV3SrbHdcRCmjneI8JWQpQAAIEI89mmuyiv9+tslneRwcGI7ju54BWuWDmyjMFnSlTpws2eXDtwu56zgRwMAILzMXL1DnyzZorvObqvWdZPsjoMwdryCdb2kP0pqIMk8+JhfB4oXAAAxxVdZrYc/Xq6WmbV1yxmt7Y6DMHfMgmWa5hhJYwzDuN40zddDmAkAgLAz+tu12rCrTJNu7CVPHPtk4/iOeRWhYRh/Nwwj/WjlyjCMuoZhPBHcaAAAhIe1hSV6+bt1urR7Y/Vtk2l3HESA4x0iHC9p/MGbOy+VtF1SmqTeOnA/wvuCHw8AAHtZlqWHpixTYrxbD13I/QJRM8crWE4dKFEeSQMlpUvaL+ku0zTXhSAbAAC2+8+CTZqXv1tPDOmszKQEu+MgQhyvYL2qg/te6cAVhJJU7+BjzwQzFAAA4WBHcbn+Pi1XPVtm6MoeTe2OgwhyvJPczzzyMcMwnJJmiIIFAIgBf5m6Qr5Kv54Y0llObuaME1CTW+VIkgzDcEnqL6lW8OIAABAevlm1XZ8u3arbz2rDnlc4YTUuWDpwLtYdku4MUhYAAMJCaXmVHv5ohdrWS9Kogex5hRN3vHOwDmOaZqmky4OYBQCAsPDMl6u1ee9+vT+qj+LdJ7IWARzAVw0AAIdYsmmv3piTr+G9m6lHiwy74yBCUbAAADiostqvBz5cpsykBN03uL3dcRDBanyIEACAaPfazDzlbt2nV4bnKMUTZ3ccRDBWsAAA0IHb4Tw/fY0u6NxAgzs1sDsOIhwFCwAQ8/x+S/d/sFS14l36y6872R0HUYCCBQCIeRN/2CDvhj165KIOqpvM7XBw6ihYAICYVrCnTE9+vkoD2tXVkOzGdsdBlKBgAQBilmVZ+uOU5ZKkxy/tJIeD2+EgMChYAICY9eHCzZq5eofuH9xeTdIT7Y6DKELBAgDEpMJin/766Ur1aJ6uq3s3tzsOogwFCwAQcyzL0kNTlmt/ZbX+cVkXOZ0cGkRgUbAAADHnkyVb9NXK7br33HZqUy/J7jiIQhQsAEBM2VFcrkc/WaHuzdJ0Q79WdsdBlKJgAQBihmVZ+tNHy1RWUa2nLu8iF4cGESQULABAzJi6dKu+WLFddw9qpzb1ku2OgyhGwQIAxIQdxeV69OPl6to0TTf2a2l3HEQ5ChYAIOpZlqWHP1qu0opqPXNFF7ld/PhDcPEVBgCIelOXbtXnK7bpd+e05dAgQoKCBQCIaoX7fHr4o+Xq3ixNN/XnqkGEBgULABC1LMvSAx8uU3lVtZ65oiuHBhEyfKUBAKLWewsK9M2qQt0/uL1a1WVDUYQOBQsAEJUK9pTpr5+uVO9WGbqmTwu74yDGULAAAFHH77d03/tLZVmWnrq8K/caRMhRsAAAUWfiDxs0Z90uPXxRBzXNSLQ7DmIQBQsAEFXyd5bqic9ydYZRV1ee1tTuOIhRFCwAQNSorPbrd+8uVoLbpScv6yKHg0ODsIfb7gAAAATKS9+u1ZJNe/XSVdmqn+KxOw5iWEgLlmEYD0r6taR4SaNN0xwXyvEBANFr0cY9euGbtRrSvbEu7NLQ7jiIcSE7RGgYxhmS+ko6XdJASRwYBwAERFlFle7+zxI1SPHozxd3tDsOIIdlWSEZyDCMJyRZkjpKSpH0B9M0Fxz6MV6v10pMDO7VHj6fTx4Py8aBxJwGFvMZeMxpYIXjfL4wd4c+W12sf5zXUF0a1LI7zgkLxzmNdKGY07KyMuXk5Bz1RL9QHiLMlNRc0kWSWkr6xDCM9qZpHtbwsrKyghoiNzc36GPEGuY0sJjPwGNOAyvc5vPbVYX67+o83TSgla48M3xynYhwm9NoEIo59Xq9x3wulAVrl6RVpmlWSDINw/BJqiupMIQZAABRZHdphf7w/lK1b5Cse85tZ3cc4Geh3KZhlqTBhmE4DMNoJKm2DpQuAABqpKLKr2ten69rXp+vEl+lBj37nXaVluufl3dRgttldzzgZyErWKZpfippkaT5kqZKus00zepQjQ8AiHwjJyzQvLxdmpe3SzmPTdeu0gq5HA498+Vqu6MBhwnpNg2mad4XyvEAANHJV+X/+W0Xe4kiDLGTOwAgYoweli33EY0qPs6ll4dn25QIODoKFgAgYtz69kLtrzj87JLKKr9ueWuhTYmAo+NWOQCAiLG7tELV1oHDgokJblUecqgQCCesYAEAIsKuknJtK/IpMd6lPq0z9cODZ6tXqzrq1aqOxozoYXc84DCsYAEAwp5lWbrv/aUq2l+pj247XR0apUiS3ry+p83JgKNjBQsAEPbGz16vr1cV6sEL2v9croBwRsECAIS1ZQVFeuKzXJ2TVU/X9m1hdxygRihYAICwVVJepTveWag6tRP01OVd5XCw6RUiA+dgAQDC1sMfLdfG3WV6Z2RvpdeOtzsOUGOsYAEAwtIH3gISRnAFAAAgAElEQVRNWbRZd57dVr1a1bE7DnBCKFgAgLCzbkeJHv54uXq1zNAdZ7W1Ow5wwihYAICw4qus1u2TFine7dS/fttNLifnXSHycA4WACCs/GXqSuVu3afx156mhqm17I4DnBRWsAAAYePjxZv1zvyNGjWwtc5sX8/uOMBJo2ABAMLCuh0levDDZerRPF33ntvO7jjAKaFgAQBst7+iWre9vVCeOJdeuKq73C5+PCGycQ4WAMB2f/5khVZtK9Yb13HeFaIDvyIAAGz14cICvbtgk247s7XOMDjvCtGBggUAsI25rVh/nLJMPVtm6PfncN4VogcFCwBgi2JfpW55y6tkT5xeHMp5V4gunIMFAAg5y7J03/tLtWF3mSbd2Ev1Ujx2RwICil8XAAAhN25Wvj5bvk0PDG7PfQYRlShYAICQmp+/W098tkqDOzbQjf1b2h0HCAoKFgAgZAqLfbpt0kI1y0jUU1d0kcPBfQYRnTgHCwAQEpXVft0+aZFKfFV664ZeSvbE2R0JCBoKFgAgJB7/b67m5+/Wv67sJqNBst1xgKDiECEAIOimLCrQ+Nnrdf3pLXVJ98Z2xwGCjoIFAAiq5ZuL9MAHy9SrZYYevKC93XGAkKBgAQCCZndphW6e6FVG7Xi9NCxbcWwmihjBOVgAgKCoqvbrjncWakdJud67uY8ykxLsjgSEDL9KAACC4qkvTM1eu0uPXdJJXZum2R0HCCkKFgAg4D5evFmvzszT8N7N9JseTe2OA4QcBQsAEFBLC/bqvveXqmfLDD1yUUe74wC2oGABAAKmcJ9PN03wKjMpQS8Py1a8mx8ziE2c5A4ACIjyqmrd/JZXRfsr9cEtfVWHk9oRwyhYAIBTZlmWHpqyXIs27tXLw7LVoVGK3ZEAW7F2CwA4Za/PXq/3vQW66+y2Or9zQ7vjALajYAEATskMs1B/n7ZSgzs20F1nt7U7DhAWKFgAgJNmbivW7ZMWqX2DFD3zm65yOh12RwLCAgULAHBSdpaU64Y3f1SteJfGXdtDtRM4rRf4CQULAHDCfJXVunmiVzuKyzV2RA81TK1ldyQgrIT81w3DMOpJ8koaZJrmqlCPDwA4NZZl6cEPl8m7YY9euiqb2+AARxHSgmUYRpykVyXtD+W4h7Isy66hASAqTF62V1MW7dE9g9rpwi5cMQgcTagPET4t6RVJW0I8riRpztqdGvbeRi3fXGTH8AAQ8aYu2aIJi/bokm6NdPtZbeyOA4QtR6hWdAzDuFZSE9M0HzMMY4akUUceIvR6vVZiYmLQMuzZX6U7phZIcui5Cxurbm1OyAwEn88nj8djd4yowXwGHnMaGMu3+/THL7eqTUac/jG4keJdnMYbKHyNBl4o5rSsrEw5OTlHvXQ2lAVrpiTr4J9uklZL+rVpmtt++hiv12vl5OQENcd/5yzRfV9sU5P0WvrPqD5K8cQFdbxYkJubq6ysLLtjRA3mM/CY01OXt6NEQ16eo/TEeP3j7Ez16t7J7khRha/RwAvFnHq93mMWrJD9+mGa5gDTNAeapnmGpMWSRhxarkKlZXq8Xh6erbWFJbrt7YWqrPaHOgIARJRdJeW67o0f5XQ49MZ1pynF47I7EhD2YnJ9t3/bunr80s76fs1O/WnKck58B4Bj8FVWa+SEBdpW5NOYET3UvE5tuyMBEcGWk5AOrmLZ6jenNdWmPWV64Zu1appRS7efxe0dAKCiyq+RExZIkl4c2l1nPjNDO0sq9MLQ7sppnm5zOiByxPRZ3ncPaqdNu8v09Jer1SitloZkN7E7EgDYauSEBZqXt0uSlP3YV6qstuR2OvS+t0C/6trI5nRA5IjpguVwOPTk5V1UWFyu+95fqjpJCRrYrq7dsQDAdr6q/52f6uL2gsAJi8lzsA6V4Hbplatz1LZ+sm55y6ulBXvtjgQAthk9LFs6olDFx7n08vBsewIBESrmC5YkpXji9OZ1pyk9MV7Xjf9R63eW2h0JAGzx29d+kK/y8KurK6v8uuWthTYlAiITBeugeikeTbihp/yWpWvGz9fOknK7IwFASC3fXKSVW/bJISnB5VCyxy2Pmx8TwMngX84hWtdN0rhrT9P2fT5dN/5HlZRX2R0JAEJi464yXTv+R9VPSVCvlhnq3TpTPzx4tnq1qqNerepozIgedkcEIkpMn+R+NNnN0vXSVdm6aaJXN09coNevPU0JbjbVAxC9Cot9uvr1eaqs9mvyTX3Upl7yz8+9eX1PG5MBkYsVrKM4O6u+nrysi2av3aXfTV6saj8bkQKITkX7KzVi3HztKC7X+OtOO6xcATh5FKxjuDynif50YZY+W75ND01Zxm7vAKLO/opq3fDGj1q3o0SvXp2j7GZsJAoECocIj+PG/q20t6xSL367VmmJ8Xrg/PZ2RwKAgKis9uvWt73ybtyjF4dmq39b9gAEAomC9QvuObed9pRV6JXv1ik9MU43D2xtdyQAOCV+v6V731uib80demJIZ13YpaHdkYCoQ8H6BQ6HQ3+9uJOK9lfqic9WKaVWnIb2bGZ3LAA4KZZl6dFPVujjxVt032CD72dAkFCwasDldOjZ33RTaXmV/jhlmWrFuXRJ98Z2xwKAE2JZlv7x2SpN/GGDbh7YSrewIg8EDSe511C826mXh+eod8s6uue9Jfp8+Va7IwHACXn+6zV6dWaeRvRprgcGt5fDwU0GgWChYJ0AT5xLY6/poa5NUnXHO4v07apCuyMBQI28+t06/Wv6Gl2e00R//lVHyhUQZBSsE1Q7wa3x1/WU0SBZo97yas66nXZHAoDjmjB3vZ74bJUu6tJQT17WRU4n5QoINgrWSUitFacJ1/dS8zqJuvHNBfJu2G13JAA4qv8s2KRHPl6hc7Lq67kru8lFuQJCgoJ1kjJqx+utG3qpfopH17z+oxZu3GN3JAA4zAfeAt3/wVL1b5upF6/qrjgX3/KBUOFf2ymol+LROyN7q05SvK4ZN1+LN+21OxIASJKmLCrQve8vUd/WB27U7InjnqpAKFGwTlGD1AMlK712vK4eN09LCyhZAOz18eLNuuc/S9S7ZR2NHXEa5QqwAQUrABql1dI7N/VWWmKcho+dp2UFRXZHAhCjpi7Zot+/u1g9W2Zo3LU9VCuecgXYgYIVII3Taumdkb2VUitOw8fN0/LNlCwAoTVt6Vb97t3F6tEiQ69fe5oS49lLGrALBSuAmqQn6p2RvZWU4NZVY37QEs7JAhAiHy/erDsnL1J2szSNp1wBtqNgBVjTjERNvqm3Ug8eLvRu4OpCAMH1vrdAv393sXo0T9cb1/VU7QTKFWA3ClYQNM1I1Ls39VFmcoJGjJun+fnskwUgOCbP36g/vL9EfVtnUq6AMELBCpJGabX07k291SDVo2ten685a9nxHUBgTZy7Xg98uEwD29XV2Gs4oR0IJxSsIKqX4tHkm/qoWUairnvjR323eofdkQBEiXGz8vXwwR3aX706h60YgDBDwQqyuskJeuem3mpdN0kj31ygz5dvszsSgAhmWZZe+HqN/vbpSp3fqYFGD8tWgptyBYQbClYIZNSO1zsje6tT4xTd+rZX73sL7I4EIAJZlqXH/5urZ75arcuym+iFod0V7+bbOBCO+JcZIqmJcXrrxl46vU2m7n1vicbPzrc7EoAIUu239OCHyzTm+3xd27eFnrq8i9zcWxAIW/zrDKHEeLfGXtND53Wsr79MXal/f71GlmXZHQtAmKuo8uvOyYs0+cdNuvOsNnr0Vx3kdDrsjgXgOChYIZbgdumlq7I1JLuxnv1qtR6bliu/n5IF4Oj2V1TrpokLNG3pVj10QZbuPteQw0G5AsIdG6bYwO1y6unLuyrFE6dxs/K1u7RC/7y8i+JY7gdiWkWVXyMnLJAkjR6WrZETFmjZ5iKVllfpiSGdNbRnM5sTAqgpCpZNnE6HHv1VB2UmxevpL1drV2mFXh6WzSaBQAwbOWGB5uXtkiT1fHy6ysqrZUnq2DCFcgVEGJZMbORwOHT7WW315GWdNWvNDl015gftKim3OxYAm/mq/Co9WK7iXQ5lJifYHQnACaJghYErT2umV6/uoVXbinX5K3O1aXeZ3ZEA2GD0sGw5jjh5PSHOpZeHZ9uUCMDJomCFiUEd6uvtG3tpd2mFhrw8R8s3F9kdCUCIXfHKXO2vqD7sscoqv255a6FNiQCcLApWGOnRIkPvj+qjeJdTv3l1rr7O3W53JAAhYFmWxszM08qt++SQlOByKNnjlodNRIGIxb/eMNO2frKm3NpXrerW1sgJCzRx7nq7IwEIoqpqvx75eIX+/t9cnd+pgU5vk6nerTP1w4Nnq1erOurVqo7GjOhhd0wAJ4hL1sJQvRSP3r2pj+58Z5Ee/niFNu3ZrwcGt2djQSDKlJZX6Y53FumbVYW6eUAr3X/Ev/M3r+9pYzoAp4IVrDBVO8Gt10b00Ig+zfXazDzdNmnh/zs3A0Dk2r7Ppytfm6sZZqH+dkknPXhBFr9EAVEkZCtYhmHESXpdUgtJCZIeM03zk1CNH4lcTof+8uuOapaRqL//N1ebX5ur167uoQapHrujATgFywqKdOOEH1Xsq9LYa3rorPb17Y4EIMBCuYI1XNIu0zT7Szpf0oshHDtiORwO3di/lV67uofWFZbo4pdmaWnBXrtjAThJ05Zu1RWvzpHb6dT7o/pSroAoFcqC9Z6khw95vyqEY0e8QR3q64Nb+8rtdOqKV+bq06Vb7I4E4ARYlqV/TV+t2yYtVMdGqfrottPVoVGK3bEABInDskJ7o2HDMJIlfSJpjGmakw59zuv1WomJiUEd3+fzyeOJ3ENse/dX628ztmllYbmGdU3TVV3T5bT5xq+RPqfhhvkMPLvntLzKr2dn79DM9aU6u3WS7uxTV/GuyD3fyu75jEbMaeCFYk7LysqUk5Nz1H/MIb2K0DCMppKmSBp9ZLn6SVZWVlAz5ObmBn2MYJvSJUsPTVmut70F2lXl0dO/6aokG+9hGA1zGk6Yz8Czc04L9pTp3olerdxaqgfPb6+bBrSSw+Zfik4VX6OBx5wGXijm1Ov1HvO5kB0iNAyjvqQvJd1vmubroRo3GiW4XXrq8i56+KIO+ip3uy55abbydpTYHQvAEeas3alfvTBLG3eXadw1PXTzwNYRX64A1Ewoz8H6o6R0SQ8bhjHj4J9aIRw/qjgcDt3Qr6Um3tBTu0srdPGLs9n5HQgTlmVp7Pd5Gj5unjKTEvTJ7f04mR2IMSE7rmSa5l2S7grVeLGib+tMTb2jn0ZN9OqGNxfo9+e00x1ntWE/HcAm+yuqdf8HS/XJki06v1MDPXWFvYfwAdiDjUajQOO0WnpvVB8NyW6s56av1o0TFmhPaYXdsYCYs25HiS4dPVtTl27RfYMNjR6WTbkCYhQFK0p44lx65oqu+tvFHTVrzU5d9MIsLd7EfllAqHy6dIt+/cIsFRaX683reurWM9pwvhUQwyhYUcThcOjqPi30/i195HBIV7wyR2/Mzleot+IAYkl5VbUe/Xi5bp+0SO0bpmjanf00oF1du2MBsBkFKwp1aZKmaXf018B2dfXnqSt1+6RFKvZV2h0LiDqbdpfpN6/M1ZtzN+jGfi01+abeapjKtTsAQrwPFkInNTFOY0b00Gsz8/TPL0wt21ykfw/trm5N0+yOBkSciiq/Rk5YIEkaPSxbt769UDuKy7VpT5lkSa8Mz9HgTg1sTgkgnFCwopjD4dDNA1srp3m67pq8WJe/PEf3nmfopv6tuMoQOAEjJyzQvLxdkqRej0/X/opqVVtSssetaXf2V7M6wb0DBYDIwyHCGNCjRYb+e2d/nduxvv7x2SpdM36+Cvf57I4FRBxflV8l5QfKlcshdWuaRrkCcFQUrBiRmhinl67K1hNDOuvH9bt1/vPf65tVbEwK1MSLQ7vryEtFEhPcevXqHFvyAAh/FKwY4nA4NLRnM029vZ/qJifo+jcW6MEPl6m0vMruaEDY2lq0XwOe+lblVf7DHq+s8uuWtxbalApAuKNgxaC29ZP18e2n6+YBrTT5x4264N/fy7tht92xgLDz8eLNOu+5mSraXym306EEl0PJHrc8br51Ajg+vkvEqAS3Sw9ekKXJI3ur2m/pilfm6p+fr1LFEb+lA7Fob1mFbp+0UHdNXqw29ZL0xe8G6PQ2merdOlM/PHi2erWqo16t6mjMiB52RwUQpriKMMb1alVHn93VX499mqvRM9bpm1WFevqKrurUONXuaIAtvlq5XQ9NWabdpRX6w3mGbh7QSm6XU29e3/Pnjzn0bQA4GlawoGRPnJ68vIvGjuih3aUVuvil2frn56vkq6y2OxoQMrtLK3TnO4s0csICZdSO10e3na7bzmwjt4tvkwBOHCtY+Nk5HerrtBYZemzaSo2esU5frNimf17eVTnN0+2OBgSNZVmatmyrHv14hfb5KnX3oHYaNbC14jnPCsAp4DsIDpOaGKenruiqN6/vKV+lX5e/Mkd/mbpCJVxpiCi0rcinmyd6dfukRWqSXkuf3tFfd57dlnIF4JSxgoWjGtiurr74/QA9+dkqvTFnvT5fvk2P/qqjzutYXw4Hu8AjslVV+zVh7gY986WpKr+lB89vrxv6teRwIICAoWDhmJIS3PrbJZ10SffGemjKMo16y6tzsurpz7/uqCbp7F6NyLRk01499NEyLd+8T2cYdfXXX3diN3YAAceva/hFOc3TNfWOfvrjBe01e+0uDXp2pl79bh1bOiCiFO2v1CMfL9clo2ercF+5XroqW+OvPY1yBSAoWMFCjcS5nLppQGtd2KWRHv14hZ74bJXeXbBJj1zUQfXtDgccoaLKr5ETFkiSXhjaXTd/tElbS/JV5bd0TZ8WuvvcdkrxxNmcEkA0o2DhhDROq6Wx1/TQN6u262+f5ura8T+qV5NEPVm3mVpk1rY7HiBJGjlhgebl7ZLfstTtr1/Kb0kOSd2bpenPv+5odzwAMYBDhDgpZ7Wvry9+N0APnt9eS7ft17nPzdSTn69Ssa/S7miAyiurVVHtV0W1Jf/BuzTHuxxKZtUKQIiwgoWTFu926uaBrdUpqUwfrvPr5Rnr9J8fN+muc9pqaM9miuOKLITYPl+lXpmxTos27f25WP0kPs6ll4dn2xMMQMzhJyBOWUaiW8/8pqs+uf10ta2fpEc+XqFzn5upz5dvlWVZv/wCwCmqqPLrjdn5OuOpGRo9Y51Sa8UpwXX4diKVVX7d8tZCmxICiDUULARMlyZpemdkb427podcTodGvbVQl78yV/PydtkdDVHK77f0yZItOve57/TnqSvVvkGyPr2jn7IapsjhcMjjdirZ41a8i73bAIQWhwgRUA6HQ2dn1dfAdnX1nrdAz321Wle+9oP6tcnU3ee2U3YzbruDU2dZlr5YsV3/mr5aq7YVy6ifrPHXnqYzjLpyOBwaM6LHz1cRjh6WrRGvzlRSUpLGjOhhc3IAsYKChaBwu5wa2rOZLu3eWG/9sEGjZ6zTkNFzdFb7erp7UDt1apxqd0REIMuyNMPcoWe/Wq1lm4vUKrO2nv9tN13UpZFczv+tUsW7nXrz+p4/v//YoIbKysqyIzKAGEXBQlB54ly6sX8rDe3ZTG/MWa/XZubpohdm6ZyserrtzDbqzooWjnDoHlajh2Xr1rcXyrIs/bZnM706M09LNu1V04xaevqKrrqkWyNubwMgLFGwEBK1E9y67cw2urpPc42ftV7j5+Tr0tFz1Ld1Hd1+Zhv1aV2HexxC0v/2sJKkXo9PV3mlX1V+SzPX7FSzjEQ9fmlnXdGjCVepAghrFCyEVIonTned01Y39m+pSfM26rXv83TV2Hnq1jRNowa21qAO9Q871IPYZFmWyqstqerA+w7p5xPYWbECEAn4TgVb1E5wa+SAVvr+vjP12CWdtKu0XKPe8urMp2do/Ox8lZRX2R0RNijc55PRIFkV1Ydv71E7waUPb+1LuQIQMVjBgq08cS4N791cvz2tqb5cuV3jZuXrL1NX6tmvVmtoz2Ya0ae5mqRzM95ot3xzkV6fna+pS7aostqS0yEduoVaVbWlW95aeNiJ6wAQzihYCAtul1MXdG6oCzo31KKNezRuVr7GzcrXmO/zdKZRT8N6NdMZRj0OH0aR/RXV+nTpFr09b6MWb9qrxHiXhvVqrpVb9mlpwV5JUpzbqcoqv81JAeDEUbAQdro3S9eLV6Vry979mjx/oyb/uEk3vLlAjdNqaWjPpvpNj6aql+KxOyZq4GhXBJaWV6lT41R9uLBA+3xValMvSY/+qoOGZDdRaq24o36OJPawAhBRKFgIW43Saunucw3dcXZbTV+5XW/P26inv1ytZ79arQHt6urynCY6J6u+PHEuu6PiGH66ItCyLGX/7StVVPllSfJu2KOLujbS8F7N1LNlxmFXkB65hxWHBQFEIgoWwl6cy6nzOzfU+Z0bKn9nqd73btKHCzfr9kmLlOJx66KujXRZdmN1b5ouJ4cQw0ZFlV+7SspVUe0/eOPlAydVuZ0OndYiQy8M7W5rPgAIJgoWIkrLzNr6w3ntdfcgQ3PX7dIHCwv04cICTZq3UY1SPbqgc0Nd2KWhujVNY1+tIDneITyHQ5q9dqemLd2qL1Zs0z5flY78W6gV79K4azncByC6UbAQkVxOh/q1zVS/tpn668UdNT13u6Yt3ao3567X2Fn5apxWSxd0bqBzsuorp3k6l/cH0KEbgfZ+4mtVVFbLb0kD/vmtfFXV2ltWqeQEtwZ1rK/V24q1ZnvxgT2tDqqs8nNFIICoR8FCxEv2xOnS7k10afcmKtpfqekrt2vasq16Y856jfk+X2mJcTrTqKezs+ppQLu6SvHE2R054v20EajvkCv8dpSU61ddGurCLo00oF2mEtwuXfP6fK0tLJHH7eCKQAAxhYKFqJJaK06X5TTRZTlNVOyr1Pdrdmp67nZ9u6pQUxZtltvpUPdmaTq9Tab6tclU16Zp3HLlEMc6/Pf8b7tp8aa9mr12p7bt8x22IiVJteKcmvvg2UpLjD/s8TEjenBFIICYRMFC1Er2xP28t1a139LCjXv07apCzV67U89/vUb/mr5GteNd6tWqjk5rkaEeLdLVuXFqTF+VeOhVfz0eO3DVn9+Suv/1K1k6cIVfYrxLbqdDVf7/lSzLku6avPj/HfbjikAAsYqChZjgOnjl2mktMiRJRWWVmpu3U7PX7tLsdTv1zapCSVK8y6lOjVOU0zxdXZumqWOjVDXPSIzqqxMrq/1aW1ii5ZuLtHp7scoPbqWgg6tUDklN0mvpiSFd1KNFum6e6NW8vF1yOznsBwDHQsFCTEpNjNPgTg01uFNDSdKuknJ5N+yRd8MeLdiwR2/O2aCK6nxJUlKCWx0apahTo1QZDZLUpl6S2tRNVmpiZJ3LZVmWdhSXa21hidbuKFHu1mKt2FKkVduKVXGwJNVOcMnpdKj6kNWpJI9bX/x+gBLjD3y74LAfAPyykBUswzCckkZL6iqpXNKNpmmuDdX4wPHUSUrQuR0b6NyODSQdOBdp9fYDBWT55n1avqVIk+ZvkK/yf6s1mUnxal03SS0za6tJei01Tq+lJumJapJeS/WSPbbc1qeiyq9tRT4V7ClTwZ79Kti7XwV7ypS3o1TrdpSo2Pe/m2ineNzq1DhV1/Rprk6NU9WxUar+OnWF5ufvPqxgHXnVH4f9AOCXhXIF6xJJHtM0+xiG0VvSM5IuDuH4QI3Fu53q1DhVnRqn6srTDjxW7bdUsKfswArQT392lGh67nbtLKk47POdDimjdrzq1E5QZnK8MpMSlJ4Yr2SPW7UT3Eo6+Ccx3qU4l1Mup0Nul0Nup1ObCn3al7BL1X5LlX5L1X6/KqoslVVUqaT84B/fgf/uKq3QzuJy7Swp186SChXtrzwsh8MhNUjxqHmdRF3crZHa1E1Sm3rJal2vthqkeP7fXmE/ve9xOzn8BwCnIJQFq5+kzyXJNM0fDMPgeAIiisvpUPM6tdW8Tm2dnVX/sOf2V1Rr88HVooI9+1W4z6cdJRXaWVKuXSXlWrRxr/aUVqikokqWdYwBDrPluM+6nQ7VTnCrTu0D5c1okKy+tROUmZSghmkeNUk7sJrWINWjeHfNr5Lk8B8ABIbDqtl3+1NmGMZYSR+YpvnZwfc3SmplmubPxyy8Xq+VmJgY1Bw+n08eDzcKDiTmtOYsy1J5laWySr/KKv3yVVmq9luqsg6skFX7LfnKK+TxJMjtOFDqXM4DhaqW26nEOKdqxTkU73KwU/0J4Gs0sJjPwGNOAy8Uc1pWVqacnJyjfjMO5QrWPknJh7zvPLRc/SQrKyuoIXJzc4M+RqxhTgOL+Qw85jSwmM/AY04DLxRz6vV6j/lcKHdYnC3pAkk6eA7WshCODQAAEDKhXMGaImmQYRhzdGBrnetCODYAAEDIhKxgmabplzQqVOMBAADYhZuwAQAABBgFCwAAIMAoWAAAAAFGwQIAAAgwChYAAECAUbAAAAACjIIFAAAQYBQsAACAAKNgAQAABBgFCwAAIMAoWAAAAAFGwQIAAAgwChYA/F979xaiVRVAcfw/pjZF2o1KykgiWmCYgUFFVoIO5EOWkQhRqCWREGEWZZFFUUhQhl3EMotCovsVIgsju0xJzEME0YqkhzAs6UGpLFCnhzPWONhNjrM9Z9bvZeacjwNrDsN869t7z9kRETVLwYqIiIioWUdvb2/pDH/q6ek5cMJERERE/ItJkyZ17O38AVWwIiIiItogU4QRERERNUvBioiIiKhZClZEREREzYaXDjDYJB0OrAFGAyOBRbY/KZuqHSTNBGbZvrx0liaSNAxYAUwEfgfm2/6mbKrmk3QWcJ/tKaWzNJ2kEcCTwDjgYOAe228UDdVwkg4CVgECdgLzbG8sm6r5JB0L9ABdtr8qkWEojmjwrS0AAANlSURBVGAtAtbZvgCYCzxaNk47SFoOLGVo/k7V5RKg0/Y5wGLggcJ5Gk/SzcATQGfpLC1xBfCT7fOA6cAjhfO0wUUAts8F7gCWlY3TfH0fBB4DtpfMMRTfDB+kuvFQjeD9VjBLm3QDC0qHaLjJwNsAtj8FziwbpxU2ApeWDtEiLwJL+h3vKBWkLWy/BlzTd3gS8EPBOG1xP7AS+L5kiFZPEUq6GrhhwOl5tj+TNIZqqnDh4Cdrrn+4p89LmlIgUpuMBrb2O94pabjtvIntI9svSxpXOkdb2P4ZQNIo4CXg9rKJ2sH2DklPAzOBy0rnaTJJc4EtttdKurVkllYXLNurgdUDz0uaADwH3GR7/aAHa7C/u6dRi23AqH7Hw1Ku4kAj6UTgVWCF7WdL52kL23Mk3QJskDTe9i+lMzXUVUCvpGnAGcAzkmbY3jzYQVpdsPZG0niqYe7Ztj8vnSein4+p1mO8IOls4IvCeSL2IOk44B3gOtvrSudpA0lXAmNtLwV+BXZRLXaPfWD7/N3fS3ofuLZEuYIhWLCoFmJ3AsslAWy1fXHZSBFANSrQJakb6ADmFc4TMdBtwJHAEkm712JNt110MXHDvQI8JekDYASw0HbWBrdAtsqJiIiIqNlQ/C/CiIiIiP0qBSsiIiKiZilYERERETVLwYqIiIioWQpWRERERM2G4mMaIqIFJHVS7Y03Fthse+V/uOYo4MI8IDMi9reMYEVEU40B5v/Pa04HZuyHLBERe8hzsCKikSStAmYDhwFrgZHA0cAS229KmgUsonoq9ke2F0t6F5hItYdeN7CM6oPmEcD1trsH/yeJiDbKCFZENNW9wJfA3cAm21OpNm9f0DcVeBcw1fZk4ARJXX3XvGf7ceA04Ebb06iKVp6cHxG1yRqsiGiDnr6vm4FDgVOAY4C3+rbEGgWcDLjfNZuotnzZ3vf6tkFLGxGtlxGsiGiqXfz1N2zgWodvge+ALttTgIeBDQOueQi40/Ycqo21O/Z34IgYOlKwIqKpfqRad3XIwBdsb6Ga9lsvaQMwHfga2AhMkLQQWAO8LulD4FTg+MEKHhHtl0XuERERETXLCFZEREREzVKwIiIiImqWghURERFRsxSsiIiIiJqlYEVERETULAUrIiIiomYpWBERERE1+wPOsLesAZfn2AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,8))\n",
    "plt.plot(np.linspace(-2, 4, 100), np.linspace(-2, 4, 100) ** 2);\n",
    "plt.scatter(tmp.iloc[:,0], tmp.iloc[:,1], marker='X');\n",
    "plt.xlabel('theta');\n",
    "plt.ylabel('J(theta)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Second Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$J(\\theta)=\\theta^4 + \\theta^2$$\n",
    "\n",
    "$$\\frac{d}{d\\theta}J(\\theta)=4\\theta^3 + 2\\theta$$\n",
    "\n",
    "$$\\theta:=\\theta-\\alpha \\times (4\\theta^3 + 2\\theta)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.0000 1.1400\n",
      "1.8600 0.2946\n",
      "1.5654 0.1847\n",
      "1.3807 0.1329\n",
      "1.2478 0.1027\n",
      "1.1451 0.0830\n",
      "1.0621 0.0692\n",
      "0.9930 0.0590\n",
      "0.9339 0.0513\n",
      "0.8827 0.0452\n"
     ]
    }
   ],
   "source": [
    "theta = 3\n",
    "alpha = 0.01\n",
    "dat = []\n",
    "for oo in range(0,10):\n",
    "    res = alpha * (4 * theta ** 3 + 2 * theta) # update function\n",
    "    print(\"{0:.4f} {1:.4f}\".format(theta, res))\n",
    "    dat.append([theta, theta ** 4 + theta ** 2])\n",
    "    theta = theta - res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>90.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.860000</td>\n",
       "      <td>15.428432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.565406</td>\n",
       "      <td>8.455422</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.380657</td>\n",
       "      <td>5.539863</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.247771</td>\n",
       "      <td>3.980968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.145108</td>\n",
       "      <td>3.030703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.062144</td>\n",
       "      <td>2.400869</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.992970</td>\n",
       "      <td>1.958167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.933949</td>\n",
       "      <td>1.633098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.882684</td>\n",
       "      <td>1.386175</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0          1\n",
       "0  3.000000  90.000000\n",
       "1  1.860000  15.428432\n",
       "2  1.565406   8.455422\n",
       "3  1.380657   5.539863\n",
       "4  1.247771   3.980968\n",
       "5  1.145108   3.030703\n",
       "6  1.062144   2.400869\n",
       "7  0.992970   1.958167\n",
       "8  0.933949   1.633098\n",
       "9  0.882684   1.386175"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp = pd.DataFrame(dat)\n",
    "tmp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl0AAAHaCAYAAAA63/zWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYXFWB//93Va9ZOvu+JyScJEAgCSRhB9lkFMFtxAUVREcHZ3R03HedkZ8z6nwd9w0QFUVBxg0EQdYAWTpBQmhu9n3vdNLZOr1U/f6o7tAJSUhCVd2q6vfrefrp2jr1yaHp/uTcc89NpNNpJEmSlFvJuANIkiR1BZYuSZKkPLB0SZIk5YGlS5IkKQ8sXZIkSXlQHneAl1NbW+vplZIkqWhMnz49cbjHC750AUyfPj3n71FXV8ekSZNy/j5dheOZfY5pdjme2eeYZpfjmX35GNPa2tojPufhRUmSpDywdEmSJOWBpUuSJCkPLF2SJEl5YOmSJEnKA0uXJElSHli6JEmS8sDSJUmSlAeWLkmSpDywdEmSJOWBpUuSJCkPLF2SJEl5YOmSJEnKA0uXJElSHli6JEmS8sDSJUmSlAeWLkmSpDywdEmSpJI3e9k26rY2xZrB0iVJkkrex+96lnujxlgzWLokSVJJ27hzH+t37GN8/6pYc1i6JElSSVuwegcAkwZWx5rD0iVJkkpa7eoGqiuSjOtXGWsOS5ckSSpptWsamDKiD+XJRKw5LF2SJKlkNbW08fyGnUwf3TfuKJYuSZJUuhat30lLW5rpoyxdkiRJOVO7ugGAac50SZIk5U7t6gbGDehBvx7xLqIHS5ckSSpR6XSaBasbmFoAhxbB0iVJkkrU6vq91O9pLohF9GDpkiRJJWrBmsx6LkuXJElSDtWubqCmqpwJg3rGHQWwdEmSpBJVu7qBqaP7kox5U9QOli5JklRydjW1EG3exbRRfeKOcoClS5IklZxn1u4gnS6c9Vxg6ZIkSSWodnUDiQScMdKZLkmSpJypXd1AGFxDTXVF3FEOsHRJkqSSkkqleWbNjoI6tAiWLkmSVGKWbtnNrv2tli5JkqRcOnCR6wK5/E8HS5ckSSoptasb6N+jktH9u8cd5SCWLkmSVFIWrGlg2ui+JBKFsSlqB0uXJEkqGVt37Wfltj0Ft54LLF2SJKmE1K7eDsBZY/rFnOSlLF2SJKlkzF3ZQFV5ktOG9447yktYuiRJUsmYv3o7Z4zsQ2V54VWcwkskSZJ0Avbsb2XxhkZmjC28Q4tg6ZIkSSVi4ZodtKXSnFmA67kAyrP9B4YQKoBbgDFAFfAfwDrgj8DS9pd9P4qiO0MIXwBeA7QCH46iaG6280iSpK5h7qrtJBMwbVThXOS6s6yXLuAdQH0URdeFEPoDC4EvA9+MougbHS8KIUwDLgRmAiOBu4GzcpBHkiR1AfNXbWfS0F4FdZHrznJRun4L3NXpfiswHQghhKvJzHZ9GDgPeCCKojSwJoRQHkIYGEXR1hxkkiRJJaylLcXCNTt4y1kj445yRFkvXVEU7QYIIdSQKV+fJXOY8SdRFNWGED4DfAHYAdR3+tJdQG/gJaWrrq4u2zFfoqmpKS/v01U4ntnnmGaX45l9jml2OZ7HJ9raxL6WNoZV7D3iuMU9prmY6SKEMBK4B/heFEV3hBD6RFG0o/3pe4BvA78Hajp9WQ2ZIvYSkyZNykXMg9TV1eXlfboKxzP7HNPscjyzzzHNLsfz+DyxdQWwgWvOPY1BvaoP+5p8jGltbe0Rn8v62YshhMHAA8Anoii6pf3h+0MIM9pvXwLUArOBK0IIyRDCKCAZRdG2bOeRJEmlb96q7Yzu3/2IhasQ5GKm69NAX+BzIYTPtT/2EeD/hRCagU3A+6IoagwhPA48Rab83ZSDLJIkqcSl02nmr27gVRMHxR3lqHKxputDwIcO89Q5h3ntF4EvZjuDJEnqOpZv3cP2Pc2cNabwLnLdmZujSpKkojZvVeFe5LozS5ckSSpq81ZtZ0DPSsYO6BF3lKOydEmSpKI2f1UDZ47uRyKRiDvKUVm6JElS0drc2MSa7Xs5s8DXc4GlS5IkFbGO9Vwzxhb2ei6wdEmSpCI2b+V2uleWMXlor7ijvCxLlyRJKlrzVjUwbVRfyssKv9IUfkJJkqTDaGxqoW5TY1Gs5wJLlyRJKlLzVm4nnYaZY/vHHeWYWLokSVJRmrNyO5VlSaaO6hN3lGNi6ZIkSUVpzop6zhjZh+qKsrijHBNLlyRJKjq7mlpYtH4ns8YV/lYRHSxdkiSp6Mxf3UAqDTPHFcd6LrB0SZKkIjRnxXYqyhJMG1UcZy6CpUuSJBWhp1fUc/qIPnSrLI71XGDpkiRJRWbP/lYWrd/JzCJazwWWLkmSVGRqVzfQlkoXzf5cHSxdkiSpqMxZWU95MsH00cWzngssXZIkqcg8vWI7p43oTY+q8rijHBdLlyRJKhr7mtt4dt2Ooju0CJYuSZJURBasaaClLV1Um6J2sHRJkqSi8fSKesqSCc4cY+mSJEnKmTkrtnPqsF70LLL1XGDpkiRJRaKppY1n1u5gVhFd+qczS5ckSSoKC9Y00NyWKrpNUTtYuiRJUlGYs2I7yQRFuZ4LLF2SJKlIzFlZz+RhvehVXRF3lBNi6ZIkSQWvqaWNBWt2MKsI9+fqYOmSJEkF75m1O2huTTGzSBfRg6VLkiQVgSeX15NMULSL6MHSJUmSisCTy7Zx2og+RbueCyxdkiSpwO3Z38oza3dwzknFe2gRLF2SJKnAzV21ndZUmnNPGhB3lFfE0iVJkgraU8vrqSxLMn1037ijvCKWLkmSVNCeXL6NqaP60K2yLO4or4ilS5IkFawde5tZvKGRc8cX96FFsHRJkqQC9vSKetJpin4RPVi6JElSAXtyeT3dK8uYMqJP3FFeMUuXJEkqWLOXbWPG2H5Ulhd/ZSn+v4EkSSpJmxubWL51T0kcWgRLlyRJKlBPLa8H4Jwi35+rg6VLkiQVpNnLttG7WwWTh/aKO0pWWLokSVLBSafTPLm8nrPH9SeZTMQdJyssXZIkqeCs3b6P9Tv2ce740ljPBZYuSZJUgGYv3wbA2SWyngssXZIkqQA9ubyeQTVVnDSwR9xRssbSJUmSCko6neap5ds4d/wAEonSWM8Fli5JklRglmzezbbdzZxdIvtzdbB0SZKkgvLEssx6rlLZFLWDpUuSJBWUx5duZdzAHozo2z3uKFll6ZIkSQVjf2sbc1Zs5/zxpXPWYgdLlyRJKhi1qxvY19LG+RMGxh0l6yxdkiSpYDy+dBvlyQSzSmw9F1i6JElSAXli6TamjepLz6ryuKNknaVLkiQVhO17mnluw07On1B667nA0iVJkgrE7GXbSKfhPEuXJElS7jy+dCu9qsuZMqJP3FFywtIlSZJil06neXxp5tI/ZcnSufRPZ5YuSZIUu+Vb97BxZ1NJbhXRwdIlSZJi9/jSrQAlu4geLF2SJKkAPLF0G2P6d2dkv9K69E9nli5JkhSr5tYUT62oL+lDi2DpkiRJMVuwpoG9zW0lu1VEB0uXJEmK1RNLt1GWTHB2CV76pzNLlyRJitXjS7cydWQfelVXxB0lp7J+YaMQQgVwCzAGqAL+A3geuA1IA88BN0VRlAohfAF4DdAKfDiKornZziNJkgrXjr3NPLt+Jx+6ZELcUXIuFzNd7wDqoyg6H7gS+A7wTeCz7Y8lgKtDCNOAC4GZwLXAd3OQRZIkFbDZy+pJpyn5RfSQm9L1W+Bzne63AtOBR9vv3wdcCpwHPBBFUTqKojVAeQih9EdckiQd8Ei0hV7V5Zw+onfcUXIu64cXoyjaDRBCqAHuAj4LfD2KonT7S3YBvYFeQH2nL+14fOuhf2ZdXV22Y75EU1NTXt6nq3A8s88xzS7HM/sc0+zqCuOZTqd56PmNnD64mqVLopy/X9xjmvXSBRBCGAncA3wviqI7Qgj/1enpGmAH0Nh++9DHX2LSpEm5iHmQurq6vLxPV+F4Zp9jml2OZ/Y5ptnVFcZz8YadbN+3kqvOOolJk0bm/P3yMaa1tbVHfC7rhxdDCIOBB4BPRFF0S/vDC0MIF7XfvhJ4HJgNXBFCSIYQRgHJKIq2ZTuPJEkqTI9EmYNbF53cNVYX5WKm69NAX+BzIYSOtV0fAv43hFAJ1AF3RVHUFkJ4HHiKTPm7KQdZJElSgXo02srkob0Y1Ks67ih5kYs1XR8iU7IOdeFhXvtF4IvZziBJkgrbzn0t1K5p4J8uGBd3lLxxc1RJkpR3s5dtoy2V5qIwKO4oeWPpkiRJefdItIWa6nKmjeoTd5S8sXRJkqS8SqfTPLpkK+dPGEB5WdepIl3nbypJkgpC3cZdbG7cz0Und51Di2DpkiRJefbIki0AXNjFLkRj6ZIkSXn1SLSVSUN7MbiLbBXRwdIlSZLyprGphdrVDVzUxWa5wNIlSZLyaPbS9q0iusgu9J1ZuiRJUt48Em2lpqqcaaP7xh0l7yxdkiQpLzq2ijhvwgAqutBWER263t9YkiTF4oVNu9jU2NQl13OBpUuSJOXJI9FWAC7sYvtzdbB0SZKkvHj4hS1MGtqLIb271lYRHSxdkiQp5xr2NDN/9XYundQ1Z7nA0iVJkvLgkSVbSKXhkkmD444SG0uXJEnKuYfqtjCgZxVThveOO0psLF2SJCmnWtpSPLpkK6+aOJBkMhF3nNhYuiRJUk7NW7mdXU2tXfrQIli6JElSjj30whYqy5KcN35A3FFiZemSJEk5k06neahuM2ef1J8eVeVxx4mVpUuSJOXMim17WFW/l0u68FYRHSxdkiQpZx6q2wzAqyZauixdkiQpZx6s28LEITWM6Ns97iixs3RJkqSc2LG3mdrVDR5abGfpkiRJOfHokq20pdJdfquIDpYuSZKUEw/WbaF/j0pOH9En7igFwdIlSZKyrqUtxaPRFi6eOIiyLrwLfWeWLkmSlHXzVzXQ2NTKpa7nOsDSJUmSsu6hus2ZXegnDIw7SsGwdEmSpKxKp9P8tW4zs07qT88uvgt9Z5YuSZKUVUs272Z1/V6uOMWzFjuzdEmSpKy6f/EmEgm4zK0iDmLpkiRJWXX/4k1MHdmHQb2q445SUCxdkiQpa9Y17GXxhkYuP2VI3FEKjqVLkiRlzQOLMxe4vsLS9RKWLkmSlDUPPL+JCYN6MnZAj7ijFBxLlyRJyorte5qZu3K7s1xHYOmSJElZ8VDdZlJpDy0eiaVLkiRlxf2LNzOsdzWnDu8Vd5SCZOmSJEmv2N7mVh5fupXLTxlCIuEFrg/H0iVJkl6xx5ZsZX9rissnuyHqkVi6JEnSK/bA4s307lbBjLH94o5SsCxdkiTpFWlpS/Fg3WYumTSI8jKrxZE4MpIk6RWZu3I7jU2tnrX4MixdkiTpFbl/8SaqK5JcMGFg3FEKmqVLkiSdsFQqzf2LN3HBhIF0qyyLO05Bs3RJkqQTtmBNA5sb9/OaKUPjjlLwLF2SJOmE/XnRRirLk7xq4qC4oxQ8S5ckSTohqVSavzyXObRYU10Rd5yCZ+mSJEknZOHaHWzc2cRrpnjW4rGwdEmSpBNy76KNVJYluWSSu9AfC0uXJEk6bul0mvsWbeT8CQPo5aHFY2LpkiRJx+2ZtTvYsLOJfzjNsxaPlaVLkiQdt3sXbaSiLMGlXuD6mFm6JEnScUmn09y7aBPnjR9A724eWjxWli5JknRcnl23k/U79nlo8ThZuiRJ0nG5d9FGypMJLp/sVhHHw9IlSZKOWTqd5t7nNnLu+AH07u6hxeNh6ZIkScfsufWNrN2+j9d4aPG4WbokSdIx+3PHocVTPGvxeFm6JEnSMUmn09z33EbOPqk/fbpXxh2n6Fi6JEnSMXl23U5W1+/lqinD4o5SlCxdkiTpmPz+mQ1UliW54lTPWjwRli5JkvSy2lJp/vTsBi4MA90Q9QSV5+oPDiHMBL4WRdFFIYRpwB+Bpe1Pfz+KojtDCF8AXgO0Ah+OomhurvJIkqQTN2dlPVt27efqMzy0eKJyUrpCCB8HrgP2tD80DfhmFEXf6PSaacCFwExgJHA3cFYu8kiSpFfmj3/fQI/KMi6Z6FmLJypXhxeXA2/odH868JoQwmMhhJ+GEGqA84AHoihKR1G0BigPIQzMUR5JknSCmltT3LtoE5dNHky3yrK44xStnMx0RVF0dwhhTKeH5gI/iaKoNoTwGeALwA6gvtNrdgG9ga2H/nl1dXW5iHmQpqamvLxPV+F4Zp9jml2OZ/Y5ptlVSOM5Z+0edu5rYWr/VMFkOhFxj2nO1nQd4p4oinZ03Aa+DfweqOn0mhoyRewlJk2alNt0ZIpdPt6nq3A8s88xzS7HM/sc0+wqpPH8wd8X0qd7BW+9eCqV5cV7Dl4+xrS2tvaIz+Vr5O4PIcxov30JUAvMBq4IISRDCKOAZBRF2/KUR5IkHYO9za08sHgzV546tKgLVyHI10zXB4DvhBCagU3A+6IoagwhPA48Rab83ZSnLJIk6Rg9WLeFfS1tvO50z1p8pXJWuqIoWgXMar+9ADjnMK/5IvDFXGWQJEmvzB+e2cDgXlXMGNsv7ihFz3lCSZJ0WDv3tvDoki28dsowypKJuOMUPUuXJEk6rL8s3khLW9pDi1ly1MOLIYQEmR3jLwL6A1uAh4C/RlGUznk6SZIUmz/8fQOj+3dnyojecUcpCUec6QohvIpMwboQeBa4g8xZh1cAD4YQLs1LQkmSlHebG5t4ank9rzt9GImEhxaz4WgzXROAy6Ioajvk8d+EEMqA9wEP5iyZJEmKzf8tXE8qDa+fOjzuKCXjiKUriqIfHu7xEEJFFEUtwPdzlkqSJMUmnU7zuwXrmTqqD+MG9ow7Tsl42S0jQgjvBz7S/toE0EpmFkySJJWg5zc2Em3exVeuOTXuKCXlWM5evJHMuq77gOuBxTlNJEmSYvW7BeupKEvw2tOGxh2lpBxL6doWRdFGoCaKokcAd0eTJKlEtbal+P0z63nVxEH07VEZd5ySciyla2cI4RogHUL4J2BgjjNJkqSYPL50G9t2N/OGaSPijlJyjvXw4mrgk8DJwPtzmkiSJMXm7gXr6NO9govDoLijlJxjufbizVEUfbD99kdDCLcDj+YwkyRJisHOfS088Pxmrj1rJJXlXrQm245YukIINwGfBfqFEN5A5sxFgOfzEUySJOXXfYs20tya8tBijhxtn67vAt8NIXw6iqKv5jGTJEmKwe8WrGfcwB6c7mV/cuJYDi9+J4TwFWAY8Gfg2SiKluU2liRJyqc19XuZu2o7H7sieNmfHDmWA7Y/BVaSWUS/qf2+JEkqIfcsXA/ANV72J2eOpXT1j6LoFqAliqIneXFtlyRJKgHpdJrfLVzH2eP6M7xPt7jjlKxjOjUhhDCx/fMI4NALYEuSpCI2f3UDq+v38vppznLl0rGs6fpX4FZgEnAX8M85TSRJkvLqznlr6VFZxmu87E9OvWzpiqLoOeDsPGSRJEl5tquphT8/u5HXnT6MHlXHMhejE/WyoxtC+AJwE9Da8VgURcNyGUqSJOXHn57dyL6WNv7xrJFxRyl5x1JpXwuMjqJoX67DSJKk/Lpz3lrGD+rJtFF94o5S8o5lIf0WoCXXQSRJUn4t2byLZ9bu4C1njnRvrjw42mWAfgWkgcHAwhDCc+33iaLobfmJJ0mScuXOeWspTyY8azFPjnZ48Qd5SyFJkvKquTXFPQvXc+mkwQzoWRV3nC7haKXrCaAM+DXwFjKbopaRuRTQq3IfTZIk5cqDdZvZvqeZt7iAPm+OVrpuAD4NDAGi9sdSZMqYJEkqYnfOW8uQXtVccPLAuKN0GUcsXVEU/Rj4cQjhhvbLAEmSpBKwYcc+Hlu6lZsuGk9Z0gX0+XLEsxdDCP8ZQuh7uMIVQhgYQrg5t9EkSVIu3FW7jnQa/vFMDy3m09EOL94K3BpCSADPApuBPsAsMtdf/Hju40mSpGxKpdL8Zv5azjmpP6P6d487TpdytNKVJFOsqoELgb7APuBDURQtz0M2SZKUZU+tqGddwz4+dkWIO0qXc7TS9UPa9+Uic+YiwKD2x76Ry1CSJCk3fjlnNX26V3DFKUPijtLlHG0h/cWHPhZCSAKPYOmSJKnobGls4oHFm7n+3DFUV5TFHafLOebLiYcQyoDzgG65iyNJknLlznlraU2ledvM0XFH6ZKO5dqLHaqBfwH+NUdZJElSjrSl0vxq7hrOGz+AsQN6xB2nSzrmma4oivYAb8phFkmSlCN/e2ELG3Y28fmrJscdpcs6npkuSZJUpH45ZzWDaqq4ZNLguKN0WZYuSZJK3Nrte3l0yVaunTGKijJ/9cfFkZckqcTdMXcNCeBaL24dK0uXJEklbH9rG7+Zt5ZLJg1mWB83IIiTpUuSpBJ2/+LN1O9p5u0zR8UdpcuzdEmSVMJ+8fRqRvbrxgUTBsYdpcuzdEmSVKKWbt7F3JXbeduM0SSTiZf/AuWUpUuSpBJ1+1OrqSxL8uYzR8QdRVi6JEkqSTv3tXD3gnVcdfowBvSsijuOsHRJklSSfjt/LXub27j+3DFxR1E7S5ckSSWmLZXmZ0+t4qwxfTl1eO+446idpUuSpBLztxe2sHb7Pt59zti4o6gTS5ckSSXm1tkrGdq7mitO8TqLhcTSJUlSCYk27eLJ5fVcd/Zoyr3OYkHxv4YkSSXktidXUVWe5K1nuQN9obF0SZJUInbsbeaehet4/dTh9O1RGXccHcLSJUlSifj1vLU0taR4t9tEFCRLlyRJJaC1LcXtT67i7HH9mTikV9xxdBiWLkmSSsBfn9/Mhp1NboZawCxdkiSVgB8/voKR/bpxySS3iShUli5Jkopc7ertLFizgxvPG0dZMhF3HB2BpUuSpCL3w0dX0Kd7BW8+c0TcUXQUli5JkorYiq27+WvdZq6bNZruleVxx9FRWLokSSpiP358JRVlSd559pi4o+hlWLokSSpS23bv5+4F63jjtBEMrKmKO45ehqVLkqQidfuTq2hpS3Hj+WPjjqJjYOmSJKkI7W1u5fanV3PppMGcNLBn3HF0DCxdkiQVod/OX8eOvS380wXj4o6iY2TpkiSpyLSl0vzkiRVMHdWH6aP7xh1Hxyhn55aGEGYCX4ui6KIQwnjgNiANPAfcFEVRKoTwBeA1QCvw4SiK5uYqjyRJpeIvz21i7fZ9fOYfJpFIuBlqscjJTFcI4ePAT4Dq9oe+CXw2iqLzgQRwdQhhGnAhMBO4FvhuLrJIklRK0uk0P3h0OWP6d+eyyUPijqPjkKvDi8uBN3S6Px14tP32fcClwHnAA1EUpaMoWgOUhxAG5iiPJEkl4dElW1m0ficfuOgkL/lTZHJyeDGKortDCGM6PZSIoijdfnsX0BvoBdR3ek3H41sP/fPq6upyEfMgTU1NeXmfrsLxzD7HNLscz+xzTLPrcOOZTqf5r/s2MLBHGZO67Xa8j1Pc36P5ul5AqtPtGmAH0Nh++9DHX2LSpEm5S9aurq4uL+/TVTie2eeYZpfjmX2OaXYdbjyfWl7P81tX8qXXncKUU8fEE6yI5eN7tLa29ojP5evsxYUhhIvab18JPA7MBq4IISRDCKOAZBRF2/KUR5KkovPdh5cxoGcVbzlrZNxRdALyNdP1UeDHIYRKoA64K4qithDC48BTZMrfTXnKIklS0Vm4poEnlm3j0/8wkeqKsrjj6ATkrHRFUbQKmNV+ewmZMxUPfc0XgS/mKoMkSaXiuw8vo0/3Ct4+c3TcUXSC3BxVkqQCt3jDTh6s28IN546lR1W+DlIp2yxdkiQVuO89vJyaqnLedc6YuKPoFbB0SZJUwJZt2c29z23kurNH07tbRdxx9ApYuiRJKmDfe2QZVeVJ3nPe2Lij6BWydEmSVKCWb93N/y1czztmjqZ/z6q44+gVsnRJklSgvvXgUqrKy3j/RSfFHUVZ4CkQkiQViObWFO+9fT4A14yv4A9/38DIvt3oVe1arlLgTJckSQXivbfPZ86KeuasqOff79sAwJbGpgNFTMXNmS5JkgpMU+uLlyxOJBIxJlE2OdMlSVKB+N7bp1FRfvCv5oryJN9/x7SYEimbLF2SJBWIf/7lAva3tB30WEtrig/8YkFMiZRNHl6UJKmAtKbSAHQvT5DCQ4ulxJkuSZIKxAcuOolUGsYO6MEv/nE0M8f1Z+a4/vz4nWfGHU1Z4EyXJEkF4n8fWsqAnpX8+V/PY/XypfzshhlxR1IWOdMlSVIBeGzJVp5cXs8/XzSe7pXOiZQiS5ckSTFLpdL8f/e9wMh+3Xj7rFFxx1GOWLokSYrZ7/++nuc3NvLvlweqysvijqMcsXRJkhSjppY2vn7/Ek4b3purpgyLO45yyNIlSVKMfvH0atbv2Mcnr5xIMukWEaXM0iVJUkx27mvhOw8v44KTB3Lu+AFxx1GOWbokSYrJ9x9Zzs59LXzy1RPjjqI8sHRJkhSDDTv2cevslVxzxnAmD+sVdxzlgaVLkqQY/M9fl5BOw0cuOznuKMoTS5ckSXm2eMNO7lqwjneePZqR/brHHUd5YumSJCmP0uk0X/7j8/TpVsG/vGpC3HGUR5YuSZLy6C/PbWLOyu189PJA7+4VccdRHlm6JEnKk6aWNv7z3jomDqnh2rNGxh1HeWbpkiQpT376xErWNezj86+dTHmZv4K7Gv+LS5KUB5sbm/juw8u44pTBnONGqF2SpUuSpDz42l9eoLUtzWf+YXLcURQTS5ckSTm2cE0Dv1uwnvecP5ZR/d0ioquydEmSlEOpVJov/+l5BtZUcdPF4+OOoxhZuiRJyqG7Fqxj4ZodfPyKQM+q8rjjKEaWLkmScqRhTzM331vHmaP78sZpI+KOo5hZuiRJypGv/eUFGpta+co1p5JMJuKOo5hZuiRJyoHa1Q38et5abjh3DJOG9oo7jgqApUuSpCxrbUvx2f97jqG9q/nwpSfHHUcFwtIlSVKW3fbkKuo2NvKFqybTw8XzamfpkiQpizbu3Mf//HUJF4eBXHHKkLjjqIBYuiRJyqKv/Ol5WlNpvvS6U0kkXDyvF1m6JEnKkodf2MK9izbxwYvHu/Mh5TUIAAAbsElEQVS8XsLSJUlSFuxqauHT9yxiwqCevO/CcXHHUQGydEmSlAU33/cCmxub+K83TaGqvCzuOCpAli5Jkl6hp5bXc8ecNdxw7limjuobdxwVKEuXJEmvwL7mNj75u2cZ3b87H708xB1HBczNQyRJegW+8UDE6vq9/Oq9s+hW6WFFHZkzXZIknaCFaxq4ZfZK3jZzFGef1D/uOCpwli5Jkk7A/tY2Pn7XswzuVc2nrpwYdxwVAQ8vSpJ0Ar714FKWbtnNre8+i5rqirjjqAg40yVJ0nGat2o7P3h0Of945ggunjgo7jgqEpYuSZKOw+79rXzkN88wvG83Pn/VKXHHURHx8KIkScfhK398nnUN+/jNP51Nzyp/jerYOdMlSdIxemDxJu6cv5b3X3gSZ43pF3ccFRlLlyRJx2Db7v186neLmDS0F/926clxx1ERcl5UkqSXkU6n+eTdi9i1v5VfXXsGleXOWej4+V0jSdLL+PW8tTxYt5lPvHoiJw+uiTuOipSlS5Kko4g27eJLf1zMeeMHcP05Y+KOoyJm6ZIk6Qj2NrfywTsW0LOqgm++5XSSyUTckVTEXNMlSdIRfPEPi1m2dTc/v2Emg2qq446jIudMlyRJh3HPwnX8Zv46PnjxeM6bMCDuOCoBli5Jkg6xYutuPnPPc8wY048PXTIh7jgqEZYuSZI6aWpp46Y7FlJVnuRbbz2D8jJ/VSo7XNMlSVInX/7T89RtbOSWd5/J0N7d4o6jEmJ9lySp3W/mreWOOWv4pwvH8aqJg+OOoxKT15muEMJCYGf73ZXAD4FvAa3AA1EUfSmfeSRJ6vD3tTv47P89x3njB/Cxy0PccVSC8la6QgjVAFEUXdTpsWeANwIrgD+HEKZFUbQgX5kkSYLMdRXf/4taBtZU8e23TnUdl3IinzNdpwPdQwgPtL/vF4GqKIqWA4QQ7gcuASxdkqS8aW1L8cE7FrB9TzN3f+Ac+vaojDuSSlQ+S9de4OvAT4AJwH3Ajk7P7wLGHe4L6+rqch6uqakpL+/TVTie2eeYZpfjmX3FOqY/mlfP0yt28u/nDaSscQN1jRvijgQU73gWsrjHNJ+lawmwLIqiNLAkhLAT6Nfp+RoOLmEHTJo0Kefh6urq8vI+XYXjmX2OaXY5ntlXjGP6+2fWc8/zK3j3OWP44GtPiTvOQYpxPAtdPsa0trb2iM/l86D1DcA3AEIIw4DuwJ4QwkkhhARwBfB4HvNIkrqwBWsa+NhdzzJjTD8+8xrLjXIvnzNdPwVuCyE8AaTJlLAU8EugjMzZi3PymEeS1EWt3b6X990+nyG9qvn+O6ZR4cJ55UHeSlcURc3A2w7z1Kx8ZZAkqbGphff8bB77W1P8+n1n0b9nVdyR1EVY7SVJXUbmTMWFrNi6hx+8YzrjB/WMO5K6EC8DJEnqMr78p+d5bMlWbn7DaZw7fkDccdTFONMlSeoSbpu9ktufWs37LhjHW2eMijuOuiBLlySp5P352Y186U/Pc9nkwXzi1RPjjqMuytIlSSppTy7bxr/d+QzTR/Xlf6+dSlkyEXckdVGWLklSyXpu/U7ee/t8xgzozk/fdRbdKsvijqQuzNIlSSpJq7bt4d23zqVP90puv2EmvbtXxB1JXZylS5JUcrY0NnHdLXNoS6X52Q0zGNK7Ou5IkltGSJJKy869Lbzr1nls29XMr943y724VDCc6ZIklYzGphauu2UOy7fs5gfXTeeMkX3ijiQdYOmSJJWE3ftbedctc6nb2Mj33j6NC08eGHck6SAeXpQkFb29za1cf+tcnl23k+++bRqXTh4cdyTpJZzpkiQVtX3NbbzntvnUrm7gW9eewatPHRJ3JOmwnOmSJBWtppY23vfz+Ty9sp7/+cczeO2UYXFHko7I0iVJKkq797fyntvmMXfVdv7rjVO4ZurwuCNJR2XpkiQVncy2EHNZtH4n/+8tZ3D1GRYuFT5LlySpqGzbvZ/rfjqX5Vt28/23T+PyU1zDpeJg6ZIkFY2NO/fx9p/MYcOOffz03Wdy/gS3hVDxsHRJkorCqm17uO6WOTTsaeH2G2YyY2y/uCNJx8XSJUkqeAvXNPCen80nnU5zx3tnMmWEO82r+Fi6JEkF7a/Pb+ZffrWAwb2que36GYwd0CPuSNIJsXRJkgrWz59ezRd+/xynDe/NT999FgN6VsUdSTphli5JUsFJpdL89wMR339kOZdMHMS33zaV7pX+ylJx8ztYklRQ9ja38rHfPsufF23krTNG8ZWrT6G8zKvWqfhZuiRJBWP9jn2892fzqdvUyKeunMj7LhhHIpGIO5aUFZYuSVJBmLdqO+//eS3NrSlueddZXDxxUNyRpKyydEmSYveruWv4/O+fY0Tf7vz4nWcyflDPuCNJWWfpkiTFpqmljS//6XnumLOGC04eyLevnUrv7hVxx5JywtIlSYrF6vo9/PMvF7B4QyP/dOE4PnZ5cMG8SpqlS5KUd395biMf++2zJJMJfvLOM7l08uC4I0k5Z+mSJOVNc2uKm++r49bZqzh9ZB+++7apjOjbPe5YUl5YuiRJebFsy27+7c5nWLR+J9efO4ZPXTmJynIPJ6rrsHRJknIqnU7z86dX89V76+hWUcYP3jGdV586JO5YUt5ZuiRJObOlsYmP3fUsjy7ZyoUnD+S/3zSFQb2q444lxcLSJUnKiXsXbeQz9yxiX0sbX7n6FN4xa7S7y6tLs3RJkrJqc2MTn//9c9y/eDOnDe/N/7zlDDc7lbB0SZKyJJ1O86u5a/jqvXU0t6b45JUTufG8se69JbWzdAENe5ppS6XjjiFJRWvVtj188oGNPLupiVnj+nHzG6YwdkCPuGNJBaXLl65UKs2l33yUmoo0N1cP5uyT+scdSZKKxr7mNr73yDJ++NgKyhNpbn7DabzlzJEkk67dkg7V5ed8k8kE//3mKextSfPWHz/NB+9YwMad++KOJUkFLZ1Oc++ijVzyjUf49t+WceWpQ/jRNSN564xRFi7pCLr8TBfAqyYO5kfXjODhTRX84NHlPFS3hQ++ajw3nj+WqvKyuONJUkFZunkXX/zjYmYvq2fikBr+37VTmTG2H3V1dXFHkwqapatdVXmSf7vsZN40fQT/8efn+e/7I341dw3/fnngdacP819ukrq8LY1N/M+DS/nN/LX0qCzjy1efwttmjHKhvHSMLF2HGNmvOz+87kyeWLqNr95bx4fvfIYfP76CT105ifMmDIg7niTlVHNrivfePh+A7719Gv/8ywW0plKcMaIPt8xeRUtbiutmjeZfXjWe/j2rYk4rFRdL1xGcN2EAf/qX8/jD3zfw9Qci3vHTOZw/YQCfePVETh3eO+54kpQT7719PnNW1AMw86sP0tSSojWVZvayel47ZSgfuyIwur9nJUonwtJ1FMlkgmumDufK04bw86dW852Hl/Habz/BpZMG8aFLTua0EZYvSaUnnU6zvy0NrZn7yQScPrIP33nbtHiDSUXOA/HHoKq8jBvPH8djH7+Yj152MvNWNXDVd57ghtvm8fe1O+KOJ0lZ0dTSxjkn9ae57eB9C7tXlvHLG2fGlEoqHZau49CruoJ/uWQCT3ziYj52RWDBmgau/u5s3nnLXGYv20Y67QarkorPzn0t/ODR5VzwXw9z830vvOT51rY0H/jFghiSSaXFw4snoKa6gpsuHs+7zhnDz55cxa2zV/H2n8xh8tBevO+CcbxmylAqPJtHUoFb17CXW2ev4tdz17CnuY1zx/dnaO9qok27AKgoT9LSmoo5pVQ6LF2vQM+qcm66eDzvOW8sv39mPT96bAUfvvMZvvaXF7j+3DH845kj6dO9Mu6YknRAOp1m4dod3DZ7FX9etBGAq6YM5cbzx3Hq8N6HPXsR4MfvPDO2zFKpsHRlQXVFGW85axRvnj6Sh6Mt/OixFXz13hf4xgNLuOr0YVw3azSnj+wTd0xJXdje5lZ+/8wGfvH0ahZvaKRnVTk3nDuGd587luF9uh14XWV5kp/dMOPA/c63Jb0ylq4sSiYTXDJpMJdMGkzdxkZ+8fRq7lm4nrtq13Ha8N68Y9YoXjNlGD2rHHZJ+RFt2sWv5q7h7gXr2NXUysQhNXzlmlN5/dTh/iyS8sz/43Jk0tBe/OfrT+OTV07knoXr+cXTq/nE3Yv44h+e58rThvCm6SOYNba/O91LyrqGPc384e8buKt2HYvW76SiLMGVpw7lurNHc+boviQS/tyR4mDpyrGa6greefYYrps1mgVrGrirdj1/+vsGfrdgPcP7dOON04bzujOGMX5QTdxRJRW4o623SqXTPLZkK/csXM+DdZtpaUszeWgvPv/ayVx9xjB3j5cKgKUrTxKJBNNH92P66H584arJ3L94E3fVruPbDy/jf/+2jIlDanjtlKG8dsowxgxwt2dJL9V5t/hZNz9Ec0sbqTSc/7W/sbe5jV37W+nXo5LrZo3hjdOHc8owN3CWComlKwbVFWVcfcZwrj5jOJsbm7h30Ub+9OxGvv7AEr7+wBJOHd6LyycP4bLJg5k4pMZDAZIO6NgtvqnTVg71e5p5/dThvPb0YZxzUn+3rJEKlKUrZoN7VXP9uWO5/tyxrN+xj/vaC9g3/7qEb/51CcP7dOOyyYO5bPJgzhrTj8pyf5hKXc3mxiYeqtsCkLk8TyfdKpI89alL3J5GKgKWrgIyvE83bjx/HDeeP44tu5r4W90WHqzbzK/mruG2J1fRvbKMs8f15/wJA7jg5IGMHdDDWTCpBDW1tDFv1XYeW7KVx5du44X2zUqry5OUJaBz70qn4UO/fsatHaQiYOkqUINqqrl2xiiunTGKvc2tPLF0G48tzfwAfuiFzL94R/Ttxjkn9WfWuP7MHNf/oL12JBWmwy2GT6XT3HTxeGpXN/D0inrmrtzO/tYUlWVJzhzTl09eOZGLwkC++uc65q7cTgXuFi8VI0tXEeheWc7lpwzh8lOGALC6fg+PLd3GY0u2cv/izfxm/jogU8Jmju3PjLF9mTaqLycN7OmWFFKBee/t83l6+TZSaZj6lb/S2pYilYbHl24D4OTBPXnrjFFcePJAZo7rR/fKF39M/+RdZ7lbvFTELF1FaHT/HlzXvwfXzRpNKpXmhU27mLOynqdX1PO3FzZz94JMCaupLueMkX2YOqovU0f24ZThvRhUUx1zeqm0HG0bh8ryJK1tKZZu2c2i9TtZuKaB+au2v7guK5X5XJaAMKQXP3/PjKNu7eBu8VJxs3QVuWQyweRhvZg8rBfXnzuWdDrNim17WLC6gYVrd7BgdQPf+dvSjp/tDO5VxanDenPK8N6cMqwXE4fUMLJvd2fEpCN4uVLVeRuHmV99kObWzMzVhf/9MIN6VfPCxkb2tx8G7FVdzhmj+jBvVQPNnQ4Ndq8q564PnH3QrJak0uP/4SUmkUhw0sCenDSwJ28+cyQAu/e3snj9Tp7b0Nj+eScPR1sOFLFuFWWcPLgnYUgNJw+u4aRBPTlpQE+G9+1GmWVMXcDRitWhe2O1tKZIp9Nc+6OneOfZY1i5bQ/N7YcIaX3xz9yyaz9j+vfgnWeP5tThvTllWG/GDejB9bfN49BzkFtaU3zgFwucuZJKnKWrC+hZVc7M9sX2HfY2txJt2sWSzbt4YdMuok27eKhuy4H1YZA5lDG2fw/GDezBqP7dGdWvO6P79WB0/+4M7V1NuXsBqYAdacE6QDKR4ENn9uBdt8wFIJVOM2/lduClxaqpJUVrKk1r6uC9sRas2cGCNc8c9h8mPSrLmPuZS+hRVXHEfNXlSRfDS12MpauL6l5ZnlnrNarvQY/X797Pim17WLF1Nyu27mH51t1EmzOFrLntxV8O5ckEg3tVM7xvN4b36cawPtUM79OdIb2rGFRTTeO+VlKptIctlReHK1jPrtvB3v2tJBKJA0WquS1FOg2VZQnmrthGWzqz5UJ5WYKWthRtaV5SrA6nsjzJf79pCpOH9uLLf3yeeau2H/R1bak0//zLhYedueqYPeucteNxSaXN0qWD9O9ZRf+eVZw1pt9Bj7el0mxqbGJN/V7WbN/Dmu172bCjifU79jF35XY2NTbRljp408byu9YyqKaKgTWZP3NAz0oG9KxiQM8q+vespG/3Svr1qKRvj0r6da+kW2VZPv+qKiE3/mwec1bUkwbO/M8HaW3NzE5lviMPnqGCl24w2nLI9y5kitV/vfE0bn9qNYvX7zzoa5LA7xas5+obhh/4h8Wxzly5GF7qumIvXSGEJPA94HRgP3BjFEXL4k2lQ5UlEwzvk5nVOvuk/i95vi2VZsuuJjbtbGJz437+vmQVie592NTYxNZd+9m0s4nn1u+kfk/zS8pZh8ryJL27VdCrupze3Soyt7tVUFNdTs+qjs+Zjx5V5fSoKqN7ZeZzj8pyqivK6FZZRreKMteiFYF0Ok1zW4qm5hT7WtrY29zK3uY29uxvZU9zK3v2Z27v3t/KrqaOzy3samqlsamFnfvaP/a20NjUaTFVW9sR37MsmSDJwSWre2UZU4b35pm1Ow4qZ0ngnoUbqKmuIJFIUF2eOGypcuZK0rGKvXQB1wDVURSdHUKYBXwDuDrmTDpOZckEQ3t3Y2jvzAato8samDRp4ktel0ql2bGvhe17mmnY28z2Pc3s2NtM/Z5mdu5roXHfi79Mt+1uZvnWPezZ38qu/a0Hne31cirLk1SXJ6muKKO6ooyq8iRVFUmqy8uoLE9SUZaksjzzUVWWuV9Rnsh8LktSUZagPJmkPJmgvKzjc4LyZIKyZJKyJJS1P59MJihLJChLZk5kyNxOkEhk1g5lPoD2+wkyr0smIHNBgQQdFxboeK5zZex4buW2/bSs20H7siQ6akM6nZnRSR/oEWlS7YfNUuk06TSkSR+4nzrweJq2VKYwZx5P05Y6+KO10+fWtva1TW1p2lIpmtsyj7W0vXi7uS1Fc2v7R/vt/R0fLW3sb03R1NJGU0sb+9ov1nysOgp3z/ZSPqimmvEDe9K7WwXdKsu4dfaqA2cJAiQTUJ6A5k7fNul0mtZD3jOVShNtfnHH90OL1cuVKmeuJB2rQihd5wF/AYii6OkQgv88LGHJZIJ+PTKHFY/X/tY2drfPeOzZn5kZ2dPcxt79mc/7Wtpoav+8r6WNfc1t7G9tY39LiqZOn5tbU+zZ38r+TsWgpS1Fa1tm5qWlLUVLW/qIM3LxWh93gAMqO8ppp6JaWZ6kslOhrSxL0qtbBdXlSaray291RZKq8jK6V2YKcbeKzO1ulQfPXHbMZPasLqdnZflR1we+65a5HO7ZltTBRapjXeKhl9M5dXhvku3t9nDbQliqJGVDIZSuXsDOTvfbQgjlURQdOF5QV1eX8xBNTU15eZ+uItfjmQB6tH9QzhG+k5PtHycmnU7TlqZ9piczO9SayswUtaWgLd1xu2NmKfP6AzNJdMw2HTwblcr84Qfuvzhj9eLtzhkg83hLSwuVFZmz4Q695mZmhowDxSOReHHGrOO5ZPtjyU6vTSYzM24ds3HJBAdm7ZKdPpcnOTCDV5YgS4dvU+0fLQc/3Jz5aNmd+cGw86Vf+BK7d+8mlU5TWZagIpk4cPiwujzBhP5VfOaiwfznI5sPzPolEwk+Oqs333g686d/bGYNFWWZv9OaFUv55Nk1ACxfGmXh79l1+HM0uxzP7It7TAuhdDUCNZ3uJzsXLoBJkyblPERdXV1e3qercDyzzzE9sl9NCEfdwBTg7imnHPQ1dXV13P2v0/MbtMT5PZpdjmf25WNMa2trj/hcIZSu2cBVwG/a13QtijmPpCLjIUBJxaAQStc9wGUhhCfJHPG4PuY8kiRJWRd76YqiKAW8P+4ckiRJueR1XCRJkvLA0iVJkpQHli5JkqQ8sHRJkiTlgaVLkiQpDyxdkiRJeWDpkiRJygNLlyRJUh5YuiRJkvLA0iVJkpQHli5JkqQ8sHRJkiTlgaVLkiQpDyxdkiRJeWDpkiRJyoNEOp2OO8NR1dbWFnZASZKkTqZPn5443OMFX7okSZJKgYcXJUmS8sDSJUmSlAeWLkmSpDwojztAIQgh9AZ+AfQCKoGPRFH0VLypSkMI4fXAm6MoelvcWYpRCCEJfA84HdgP3BhF0bJ4U5WGEMJM4GtRFF0Ud5ZiFkKoAG4BxgBVwH9EUfSHWEMVuRBCGfBjIABtwPVRFC2PN1XxCyEMAmqBy6IoeiGODM50ZXwEeCiKoguBdwPfjTdOaQghfAu4Gb/PXolrgOoois4GPgl8I+Y8JSGE8HHgJ0B13FlKwDuA+iiKzgeuBL4Tc55ScBVAFEXnAp8HvhlvnOLX/o+DHwL74szhL8OM/yHzHwMys39NMWYpJU8CH4g7RJE7D/gLQBRFTwNnxhunZCwH3hB3iBLxW+Bzne63xhWkVERR9H/A+9rvjgY2xxinVHwd+AGwIc4QXe7wYgjhPcC/HfLw9VEUzQshDCFzmPHD+U9WvI4ypneGEC6KIVIp6QXs7HS/LYRQHkWRv9hegSiK7g4hjIk7RymIomg3QAihBrgL+Gy8iUpDFEWtIYSfAa8H3hR3nmIWQng3sDWKovtDCJ+KM0uXK11RFP0U+Omhj4cQTgN+Dfx7FEWP5j1YETvSmCorGoGaTveTFi4VmhDCSOAe4HtRFN0Rd55SEUXRu0IInwDmhBAmR1G0J+5MReoGIB1CuBQ4A7g9hPC6KIo25TtIlytdhxNCmExmivwtURT9Pe48Uiezyazv+E0IYRawKOY80kFCCIOBB4APRlH0UNx5SkEI4TpgRBRFNwN7gRSZBfU6AVEUXdBxO4TwCPD+OAoXWLo63ExmQe23QggAO6MoujreSBKQmT24LITwJJAAro85j3SoTwN9gc+FEDrWdl0ZRVGsC5aL3O+AW0MIjwEVwIejKHKtcQnwMkCSJEl54NmLkiRJeWDpkiRJygNLlyRJUh5YuiRJkvLA0iVJkpQHbhkhqWSEEKrJXAtwBLApiqIfHMPX9ANe7aaeknLNmS5JpWQIcONxfs0U4HU5yCJJB3GfLkklI4TwY+AtQE/gfqAS6A98LoqiP4YQ3gx8hMzu3k9EUfTJEMJfgdPJXDPwSeCbZP5B2gf41yiKnsz/30RSKXKmS1Ip+U/geeDLwPooii4hcwH7D7QfRvwScEkURecBw0MIl7V/zd+iKPoRcArw0SiKLiVTvrwCgKSscU2XpFJV2/55E9AdGA8MBO5tv9xXDTAOiDp9zXoyl7PZ1/58Y97SSip5znRJKiUpXvy5dujaiZXAWuCyKIouAr4NzDnka/4X+EIURe8ic3HxRK4DS+o6LF2SSskWMuu4uh36RBRFW8kcMnw0hDAHuBJYAiwHTgshfBj4BfD7EMLjwMnAsHwFl1T6XEgvSZKUB850SZIk5YGlS5IkKQ8sXZIkSXlg6ZIkScoDS5ckSVIeWLokSZLywNIlSZKUB/8/iFPsJH3zZFMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,8))\n",
    "x_grid = np.linspace(-2, 4, 100)\n",
    "plt.plot(x_grid, x_grid ** 4 + x_grid ** 2);\n",
    "plt.scatter(tmp.iloc[:,0], tmp.iloc[:,1], marker='X');\n",
    "plt.xlabel('theta');\n",
    "plt.ylabel('J(theta)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cost Function\n",
    "\n",
    "$$J(\\theta)=\\frac{1}{m}\\sum_{i=1}^m(h_{\\theta}(x^{(i)})-y^{(i)})^2$$\n",
    "\n",
    "## Derivative of Cost Function\n",
    "\n",
    "$$\\frac{d}{d\\theta}J(\\theta)=\\frac{2}{m}\\sum_{i=1}^m(h_{\\theta}(x^{(i)})-y^{(i)}) \\times x^{(i)} $$\n",
    "\n",
    "## Update Rule\n",
    "\n",
    "$$\\theta_i:=\\theta_i-\\alpha\\frac{d}{d\\theta}J(\\theta)$$\n",
    "\n",
    "Substitute derivative of cost function to update rule\n",
    "\n",
    "$$\\theta_i:=\\theta_i-\\alpha \\times \\frac{2}{m}\\sum_{i=1}^m(h_{\\theta}(x^{(i)})-y^{(i)}) \\times x^{(i)} $$\n",
    "\n",
    "Simplifying everything, our gradient for the coefficient of x is just the average of our predicted values multiplied by their respective x value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "Quite often you see it in the following form:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cost Function\n",
    "\n",
    "$$J(\\theta)=\\frac{1}{2m}\\sum_{i=1}^m(h_{\\theta}(x^{(i)})-y^{(i)})^2$$\n",
    "\n",
    "## Derivative of Cost Function\n",
    "\n",
    "$$\\frac{d}{d\\theta}J(\\theta)=\\frac{1}{m}\\sum_{i=1}^m(h_{\\theta}(x^{(i)})-y^{(i)}) \\times x^{(i)} $$\n",
    "\n",
    "## Update Rule\n",
    "\n",
    "$$\\theta_i:=\\theta_i-\\alpha\\frac{d}{d\\theta}J(\\theta)$$\n",
    "\n",
    "Substitute derivative of cost function to update rule\n",
    "\n",
    "$$\\theta_i:=\\theta_i-\\alpha \\times \\frac{1}{m}\\sum_{i=1}^m(h_{\\theta}(x^{(i)})-y^{(i)}) \\times x^{(i)} $$\n",
    "\n",
    "Simplifying everything, our gradient for the coefficient of x is just the average of our predicted values multiplied by their respective x value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Applying Gradient Descent to Boston Housing Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1.0</td>\n",
       "      <td>296.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.90</td>\n",
       "      <td>4.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>392.83</td>\n",
       "      <td>4.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>394.63</td>\n",
       "      <td>2.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.33</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \\\n",
       "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   \n",
       "1  0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   \n",
       "2  0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   \n",
       "3  0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   \n",
       "4  0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   \n",
       "\n",
       "   PTRATIO       B  LSTAT  \n",
       "0     15.3  396.90   4.98  \n",
       "1     17.8  396.90   9.14  \n",
       "2     17.8  392.83   4.03  \n",
       "3     18.7  394.63   2.94  \n",
       "4     18.7  396.90   5.33  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "boston_data = load_boston()\n",
    "df = pd.DataFrame(boston_data.data, columns=boston_data.feature_names)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = df[['LSTAT']].values\n",
    "y = boston_data.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "sc_x = StandardScaler()\n",
    "sc_y = StandardScaler()\n",
    "X_std = sc_x.fit_transform(X)\n",
    "y_std = sc_y.fit_transform(y.reshape(-1,1)).flatten()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "alpha = 0.0001\n",
    "w_ = np.zeros(1 + X_std.shape[1])\n",
    "cost_ = []\n",
    "n_ = 100\n",
    "\n",
    "for i in range(n_):\n",
    "    y_pred = np.dot(X_std, w_[1:]) + w_[0]\n",
    "    errors = (y_std - y_pred)\n",
    "    \n",
    "    w_[1:] += alpha * X_std.T.dot(errors)\n",
    "    w_[0] += alpha * errors.sum()\n",
    "    \n",
    "    cost = (errors**2).sum() / 2.0\n",
    "    cost_.append(cost)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl0AAAHeCAYAAAChTr7AAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmcXHWd7/9XVa/ppNd09nQWSPimMWELEgQCKPBD5I46jo6Og+hVcRlH0Zk76qjc8c7AOM4dvcq4jANyXQYHFcWroI6CooBAoAEh0HyBkJCQkH3vpPf6/VHVpBM7ZKuqU931ej7oR9U5dc7pTz++Dzrv/n6/53tSmUwGSZIkFVY66QIkSZLKgaFLkiSpCAxdkiRJRWDokiRJKgJDlyRJUhEYuiRJkoqgMt8XDCFUATcAc4Aa4GrgPuA6oBmoAC6PMa4IIVwBvBfoB66OMd6a73okSZJKQSrf63SFEP47cHKM8cMhhInAw8CvgJ/GGL8XQnglUAd0AL8ETgdqgbuB02OMPXktSJIkqQQUYnjx+8BVw7b7gbOBmSGE24E/B+4EzgDuiTH2xBh3AM8AJxWgHkmSpMTlfXgxxrgbIIRQD9wMfAr4JrAtxnhhCOF/Ah8DngJ2DDt1F9B44PU6OjpcMl+SJI0aixcvTo20P++hCyCE0AbcAnwlxvidEMLngR/nPv4JcA3wIFA/7LR6YPtI11u8eHFe6+vs7KS9vT2v19Sxs11Kl21TmmyX0mXblKZitEtHR8dBPyvERPopwC+Av4wx3pHbfTfwGuDbwLnA48Ay4JoQQi3ZCfftwPJ81yNJklQKCtHT9QmydyleFUIYmtv1duD6EML7yQ4pvjXGuC2EcC1wF9m5ZZ+MMXYXoB5JkqTEFWJO15XAlSN8dNEIx15HdikJSZKkMc3FUSVJkorA0CVJklQEhi5JkqQiMHRJkiQVgaFLkiSpCAxdkiRJRWDokiRJKgJDlyRJUhEYuiRJkorA0CVJklQEhi5JkqQiMHRJkiQVgaFLkiSpCAxdkiRJRWDoAn722As8u2l30mVIkqQxzNAF/MOtT/C13zybdBmSJGkMM3QBUxtrWb11T9JlSJKkMczQBcxqqTN0SZKkgjJ0AW0tdbywYy99A4NJlyJJksYoQxfZ0DWYgXXb9yZdiiRJGqMMXWSHFwGHGCVJUsEYusj2dAGs2WpPlyRJKgxDFzC1oZaqipQ9XZIkqWAMXUBFOsXM5jrWbDN0SZKkwjB05cxsHscae7okSVKBGLpyXKtLkiQVkqErZ1ZLHdv39LGzuy/pUiRJ0hhk6MrZdwejvV2SJCn/DF05swxdkiSpgAxdOW3NrtUlSZIKx9CV01hXRUNtpZPpJUlSQRi6hpk10bW6JElSYRi6hmlrdtkISZJUGIauYWa11PH81r0MDmaSLkWSJI0xhq5h2lrq6B0YZOOunqRLkSRJY4yha5ihtbocYpQkSflm6BpmlqFLkiQViKFrmBlN40ilXCBVkiTln6FrmOrKNNMaag1dkiQp7wxdB2hrca0uSZKUf4auA7S1uFaXJEnKP0PXAWa11LFhZw/dfQNJlyJJksYQQ9cBhu5gfH6bD76WJEn5Y+g6QFvLOMA7GCVJUn4Zug7gAqmSJKkQDF0HmDShhtqqtD1dkiQprwxdB0ilUrQ1ewejJEnKL0PXCGa11LHGifSSJCmPDF0jaGupY83WPWQymaRLkSRJY4ShawRtLXXs7uln256+pEuRJEljRGW+LxhCqAJuAOYANcDVMcYf5z57K/DBGOMrcttXAO8F+nPH3Zrveo7G0Fpda7buoWV8dcLVSJKksaAQPV2XAVtijEuBS4AvAYQQTgHeBaRy21OBDwFnAxcDnwkh1BSgniM2y2UjJElSnhUidH0fuGrYdn8IYSLwT8CHh+0/A7gnxtgTY9wBPAOcVIB6jtjM5twCqT74WpIk5UnehxdjjLsBQgj1wM1kA9jXgY8Aw28JbAB2DNveBTSOdM3Ozs681tjd3X3IazbVVvDoinV0TnFeV7EcTrsoGbZNabJdSpdtU5qSbpe8hy6AEEIbcAvwFeBpYD7wVaAWODGE8AXgV0D9sNPqge0jXa+9vT2v9XV2dh7ymnMnb2XHQEXev7cO7nDaRcmwbUqT7VK6bJvSVIx26ejoOOhnhZhIPwX4BfCXMcY7crtflvtsDnBTjPHDuTld14QQaslOuG8Hlue7nqM1d+J47n12S9JlSJKkMaIQc7o+ATQDV4UQ7sx9jTvwoBjjeuBa4C6yvV6fjDF2F6CeozK3dTwv7Ohmb+9A0qVIkqQxoBBzuq4ErjzIZ6uAM4dtXwdcl+8a8mFO63gAntvaxYKpDQlXI0mSRjsXRz2IubnQtXJTV8KVSJKkscDQdRBDPV0rtxi6JEnSsTN0HcSEmkom1dewarOhS5IkHTtD10uYO3E8Kw1dkiQpDwxdL2Fu63hWbnZVekmSdOwMXS9hTut4Nu/uYVe3q9JLkqRjY+h6CXNbsw++XmVvlyRJOkaGrpfgHYySJClfDF0vYc7EbOjyDkZJknSsDF0vobaqgumNtd7BKEmSjpmh6xDmtLpshCRJOnaGrkOY2zqeVc7pkiRJx8jQdQhzW8ezfU8f27p6ky5FkiSNYoauQxiaTO8djJIk6VgYug5h7iTvYJQkScfO0HUIbc11pFM4mV6SJB0TQ9chVFemmdlcZ+iSJEnHxNB1GLyDUZIkHStD12GY2zqelZu6yGQySZciSZJGKUPXYZgzsY6u3gE27e5JuhRJkjRKGboOw9xJEwBYtXlPwpVIkqTRytB1GOYOrdW1eXfClUiSpNHK0HUYpjfVUlWRYqU9XZIk6SgZug5DZUWaWS119nRJkqSjZug6THNbxzunS5IkHTVD12GaMzG7VtfgoMtGSJKkI2foOkxzWsfT0z/ICzu7ky5FkiSNQoauw3Rcqw++liRJR8/QdZjmtA4tG2HokiRJR87QdZimNtRSU5k2dEmSpKNi6DpM6XQqdwejoUuSJB05Q9cRmNs6nmcNXZIk6SgYuo7AvMkTeG5LFz39A0mXIkmSRhlD1xGYN3kCgxkffC1Jko6coesIHD9pAgDPbPRxQJIk6cgYuo7A8ZMmkErB0xt3JV2KJEkaZQxdR2BcdQUzm8fZ0yVJko6YoesIzZ9cb+iSJElHzNB1hOZNnsCzm7sY8MHXkiTpCBi6jtC8SRPo7R9kzVbvYJQkSYfP0HWE5k3J3sH4tEOMkiTpCBi6jtC8yS4bIUmSjpyh6wg11FYxpaHGZSMkSdIRMXQdhXmTJ7DCni5JknQEDF1HYd6kCTyzcTeZjHcwSpKkw2PoOgrzptTT1TvACzu6ky5FkiSNEoauozDPZzBKkqQjZOg6CvNdNkKSJB0hQ9dRmDi+mqa6Knu6JEnSYavM9wVDCFXADcAcoAa4GlgN/CswAPQAl8cYN4QQrgDeC/QDV8cYb813PYWQSqWYN8k7GCVJ0uErRE/XZcCWGONS4BLgS8AXgQ/GGM8Hfgh8LIQwFfgQcDZwMfCZEEJNAeopiPlTJrhWlyRJOmyFCF3fB64att0PvCXG+EhuuxLoBs4A7okx9sQYdwDPACcVoJ6COH7SBLbt6WPL7p6kS5EkSaNA3ocXY4y7AUII9cDNwKdijC/k9p0F/CVwLtnerR3DTt0FNI50zc7OzrzW2N3dfczXrOnJPvD69gce56Sp4/JRVtnLR7uoMGyb0mS7lC7bpjQl3S55D10AIYQ24BbgKzHG7+T2vRn4JHBpjHFTCGEnUD/stHpg+0jXa29vz2t9nZ2dx3zNhml7uer29fSNm0h7++w8VVbe8tEuKgzbpjTZLqXLtilNxWiXjo6Og35WiIn0U4BfAH8ZY7wjt+8yshPmz48xbs0dugy4JoRQS3bCfTuwPN/1FMr0xlrqqiu8g1GSJB2WQvR0fQJoBq4KIVwFVAALgeeAH4YQAH4TY/y7EMK1wF1k55Z9MsY4apZ4T6VSzJs8wdAlSZIOSyHmdF0JXHmYx14HXJfvGopl3qQJ/G7FlqTLkCRJo4CLox6D4ydPYP3ObnZ29yVdiiRJKnGGrmMwf3L2cUAukipJkg7F0HUM5k32wdeSJOnwGLqOwayWOqor0oYuSZJ0SIauY1BZkWZu63hDlyRJOiRD1zGaN3kCTxu6JEnSIRi6jtEJU+pZs20Pe3r7ky5FkiSVMEPXMVowrZ5MBp7aYG+XJEk6OEPXMWqf2gDAky/sTLgSSZJUygxdx2hm8zjGV1fw5PpdSZciSZJKmKHrGKXTKU6YWk+nPV2SJOklGLryYMHUBp5cv4tMJpN0KZIkqUQZuvKgfVo9O/b2sX5nd9KlSJKkEmXoyoMFL06md16XJEkamaErD8LUegA61zuvS5IkjczQlQeN46qY0TTOni5JknRQhq48WTC1nift6ZIkSQdh6MqTBdPqWbGpi57+gaRLkSRJJcjQlScLpjYwMJhhxcaupEuRJEklyNCVJ+3TspPpHWKUJEkjMXTlyZyJ46muTPs4IEmSNCJDV55UVqQ5YcoEHwckSZJGZOjKo6HHAUmSJB3I0JVHC6bWs2lXD5t39yRdiiRJKjGGrjxqn5Z9HFC0t0uSJB3A0JVHC4YeB+S8LkmSdABDVx5NnFDDpPoa53VJkqQ/YOjKMx8HJEmSRmLoyrP2aQ08tWE3/QODSZciSZJKiKErzxZMrae3f5BVW3wckCRJ2sfQlWcLpmbvYOx8wXldkiRpH0NXnh0/eTyV6ZTLRkiSpP0YuvKsprKC4ydNcDK9JEnaj6GrAMLUeocXJUnSfgxdBbBgWj1rt+9lx56+pEuRJEklwtBVAAunNwLw+LodCVciSZJKhaGrABbNyIaux9YauiRJUpahqwCax1czo2mcoUuSJL3I0FUgi2Y0stzQJUmScgxdBbJoZiOrtuxhZ7eT6SVJkqGrYBbm5nXZ2yVJksDQVTALp2cfB2TokiRJYOgqmIkTapjeWMtja12ZXpIkGboKaqGT6SVJUo6hq4AWzWhk5eYuJ9NLkiRDVyEtnJlbmd4hRkmSyp6hq4CGVqb3cUCSJMnQVUCtE2qY1ljryvSSJMnQVWgLZzQauiRJkqGr0IYm0+/u6U+6FEmSlKDKQlw0hFAF3ADMAWqAq4EngG8AGWA58IEY42AI4e+AS4F+4MMxxmWFqCkpi2Y0ksnA42t3sOS4iUmXI0mSElKonq7LgC0xxqXAJcCXgM8Dn8rtSwGvCyGcBpwHLAHeAny5QPUkZuhxQA4xSpJU3grS0wV8H7h52HY/sBj4TW77Z8D/B0TgFzHGDLA6hFAZQpgUY9w0/GKdnZ15La67uzvv13wpE+squKdzDWe39hTte45GxW4XHT7bpjTZLqXLtilNSbdLQUJXjHE3QAihnmz4+hTwL7lwBbALaAQagC3DTh3av1/oam9vz2t9nZ2deb/mSzl1dhcrN+8u6vccjYrdLjp8tk1psl1Kl21TmorRLh0dHQf9rGAT6UMIbcCvgW/HGL8DDA77uB7YDuzMvT9w/5iyaEYjzzqZXpKkslaQ0BVCmAL8AvhYjPGG3O6HQwjn595fAtwF3ANcHEJIhxBmAekY4+ZC1JSkRTMbyGTgiXWuTC9JUrkq1JyuTwDNwFUhhKty+64Erg0hVAOdwM0xxoEQwl3AvWQD4AcKVE+iFk7fN5n+jLktCVcjSZKSUKg5XVeSDVkHOm+EYz8NfLoQdZSKyQ21TK6vYbl3MEqSVLZcHLVIFrkyvSRJZc3QVSQLZzSyYtNuupxML0lSWTJ0FclJM7Mr0zvEKElSeTJ0FckpbU0APLxmzK2IIUmSDoOhq0gmTqhhzsQ6HnpuW9KlSJKkBBi6iujUWc08vGY7mUzm0AdLkqQxxdBVRKfNamLTrh6e37Y36VIkSVKRGbqK6NRZzYDzuiRJKkeGriJaMLWe2qq087okSSpDhq4iqqxIc9LMJnu6JEkqQ4auIjttVjNPrNtBd99A0qVIkqQiMnQV2amzmugbyPD4OhdJlSSpnBi6iuy03GT6h55ziFGSpHJi6CqySfU1tLWM46HVTqaXJKmcGLoScGpbMw+vtqdLkqRyYuhKwGmzmli/s5t1210kVZKkcmHoSsCLi6Ta2yVJUtkwdCWgfVoDNZVp53VJklRGDF0JqK5Ms2hGIw8buiRJKhuGroScNruZ5Wt30tPvIqmSJJUDQ1dCTm1rondgkCfW7Uy6FEmSVASGroScNju3SKqT6SVJKguGroRMaahlRpOLpEqSVC4MXQk6ZVYTj9jTJUlSWTB0Jei0Wc2s3b6XDTu7ky5FkiQVmKErQafOagKg4zmHGCVJGusMXQlaOL2RcVUVLFu5NelSJElSgRm6ElRdmea02U3c9+yWpEuRJEkFZuhK2JK5E4kbdrF9T2/SpUiSpAIydCVsydwWMhl4YJXzuiRJGssMXQk7ua2J6so09zvEKEnSmGboSlhtVQWntDWxbJWT6SVJGsteMnSFEKYdZP+SwpRTns6c28LytTvY1d2XdCmSJKlADtXTdePQmxDCt4ft/0xhyilPS46byGAGHnS9LkmSxqxDha7UsPczD7Jfx+jUWU1UplOu1yVJ0hh2tHO6MnmtoszVVVdy0sxGJ9NLkjSGHSp0ZQ7yXnm25LiJPPr8Dvb09iddiiRJKoDKQ3x+TghhHdnhxJZh75sLXlmZWTK3ha/euYKHV2/n7HmtSZcjSZLy7CVDV4yxuliFlLvFs5tJp+D+Z7cYuiRJGoNeMnSFEGqA9wHXAtOBLwA9wP+IMa4vfHnlo762ioUzGrnPyfSSJI1Jh5rT9a/A7NxxXwF+D/wA+GqB6ypLS+a28Mia7XT3DSRdiiRJyrNDha7ZMca/AqqApcBnY4y3AJMKXlkZWjJ3Ir39g/x+zfakS5EkSXl2qNA1mHs9G1gWYxxaMn1c4UoqXy+f00IqBfc7xChJ0phzqLsXu0II7wHeCHwnhJAG3gmsLnhlZaixrooFUxtcJFWSpDHoUD1d7wOOB34UY/wG8Drgz4D3F7iusrVkbgsdz22jb2Dw0AdLkqRR41ChaxZwMXBdCOENwL8BbcDLC11YuTrzuBb29g3w6PM7ki5FkiTl0aFC1zXA5bm5XFcDlwCnAx8vdGHl6uVzWgC4z0cCSZI0phwqdKVjjI+GEKYD42OMD8UYd7Jvgr3ybOKEGk6c1sBdT29KuhRJkpRHhwxduddXA7fDiwum1heyqHK39IRWOp7b5nMYJUkaQw519+LtIYR7yM7jem0I4XiyC6N+91AXDiEsIbuu1/khhFPIzgfrB54C3h1jHAwhXAG8N7f/6hjjrcfws4wZS+dN4mu/eZb7n93KKxdMTrocSZKUBy/Z0xVj/CzwbuDUGOMjud1fjTF+5qXOCyF8FLgeqM3t+jvg72OM5wA1wKUhhKnAh8iuAXYx8JlcL1rZO31OMzWVaX7rEKMkSWPGoXq6iDF2Dnu/AlhxGNddAbwB+HZu+2GgJYSQIjs02QecAdwTY+wBekIIzwAnAQ8c0U8wBtVWVXDG3Bbufnpz0qVIkqQ8OWToOhoxxh+EEOYM2/U08GXgU8AO4E6yC64OXxdhF9A40vU6OztH2n3Uuru7837NfAuNg9z19G5+++BjTBpfkGYqOaOhXcqVbVOabJfSZduUpqTbpVj/mn8RWBpjfDyE8AHgc8B/sf+E/HpgxIcOtre357WYzs7OvF8z397YvJPrH7yLDTRxbntb0uUUxWhol3Jl25Qm26V02TalqRjt0tHRcdDPDnX3Yr5sBXbm3q8DmoFlwNIQQm0IoRFoB5YXqZ6SF6bUM6m+hrscYpQkaUwoVk/Xu4GbQgj9QC9wRYxxfQjhWuAusuHvkzHG7iLVU/JSqRRL57Vy51ObGBzMkE6nki5JkiQdg4KFrhjjKuDM3Pu7yd6leOAx1wHXFaqG0e6c+a388OG1PPHCThbOGHG6myRJGiWKNbyoo3DOvFYAhxglSRoDDF0lbHJDLQum1nP3M67XJUnSaGfoKnFL57fywMpt7O0dSLoUSZJ0DAxdJW7p/En0Dgxy/8otSZciSZKOgaGrxJ0xt4XqyrSr00uSNMoZukpcbVUFZ8xpcTK9JEmjnKFrFDhnfitxwy427HQZM0mSRitD1yiwdH526QiHGCVJGr0MXaNA+9QGWifUcOdTLh0hSdJoZegaBdLpFK9aMIk740b6BgaTLkeSJB0FQ9cocUH7FHZ19/PAyq1JlyJJko6CoWuUWDq/lerKNLd3bky6FEmSdBQMXaNEXXUlZx0/kTue3EAmk0m6HEmSdIQMXaPIBe1TeG7LHlZs2p10KZIk6QgZukaRCxZMBnCIUZKkUcjQNYpMbxrHy6Y3cPsTG5IuRZIkHSFD1yhzQfsUHlq9ja1dvUmXIkmSjoCha5S5sH0ygxn49ZMOMUqSNJoYukaZhdMbmVxfwx1POsQoSdJoYugaZdLpFBe0T+a3T22mp38g6XIkSdJhMnSNQhe2T2F3Tz/3P+vq9JIkjRaGrlHo7Hmt1FaluaPTIUZJkkYLQ9coVFtVwTnzWrm9c6Or00uSNEoYukapC9qnsHb7XuKGXUmXIkmSDoOha5QaWp3+DlenlyRpVDB0jVKTG2o5ua2Jny9fn3QpkiTpMBi6RrHXLJzKY2t3sHrLnqRLkSRJh2DoGsVes2gaAD9d/kLClUiSpEMxdI1ibS11nNzWxG2PGrokSSp1hq5R7tJFDjFKkjQaGLpGuaEhxtses7dLkqRSZuga5WY213FKWxO3PbYu6VIkSdJLMHSNAZcumsbytTt5bktX0qVIkqSDMHSNAZcsmgo4xChJUikzdI0BM5vrOHWWdzFKklTKDF1jxKWLpvH4up2s2uwQoyRJpcjQNUZc4l2MkiSVNEPXGDGjaRynOcQoSVLJMnSNIZeeNJ0nXtjJSocYJUkqOYauMeQ1ubsYf+oQoyRJJcfQNYZMaxzH4tnN/PiRdWQymaTLkSRJwxi6xpjXnzKduGEXj6/bmXQpkiRpGEPXGPNHJ0+nuiLNDx56PulSJEnSMIauMaaprpoL2ifz40fW0TcwmHQ5kiQpx9A1Bv3JaTPZ0tXLnXFT0qVIkqQcQ9cYdF6YxMTx1fygwyFGSZJKhaFrDKqqSPO6U2Zwx5Mb2NbVm3Q5kiQJQ9eY9cbFM+kbyPCTR9clXYokScLQNWadOL2B9mkNDjFKklQiKgt14RDCEuCzMcbzQwiTgeuAZqACuDzGuCKEcAXwXqAfuDrGeGuh6ilHf3LaDK6+rZNnNu5i3uT6pMuRJKmsFaSnK4TwUeB6oDa365+BG2OM5wKfAhaEEKYCHwLOBi4GPhNCqClEPeXqdafMoCKd4uaOtUmXIklS2SvU8OIK4A3Dts8GZoYQbgf+HLgTOAO4J8bYE2PcATwDnFSgesrSpPoazj9hErc8/DwDgz4WSJKkJBVkeDHG+IMQwpxhu+YA22KMF4YQ/ifwMeApYMewY3YBjSNdr7OzM6/1dXd35/2apWrJFLjjyR7+81cPsXhGXdLlvKRyapfRxrYpTbZL6bJtSlPS7VKwOV0H2AL8OPf+J8A1wIPA8IlG9cD2kU5ub2/PazGdnZ15v2apOm7+AF9edgcPbE5z2YWl/TOXU7uMNrZNabJdSpdtU5qK0S4dHR0H/axYdy/eDbwm9/5c4HFgGbA0hFAbQmgE2oHlRaqnbNRUVvDak6fz8+Xr2b7HNbskSUpKsULXXwOXhxB+B7wa+McY43rgWuAu4FfAJ2OM3UWqp6z82Rmz6Okf5GaXj5AkKTEFG16MMa4Czsy9fw64aIRjriO7lIQK6MTpDSye3cyN96/mnWfPJZ1OJV2SJEllx8VRy8RlZ85i5eYufrdiS9KlSJJUlgxdZeKShdNoGV/Nf9z3XNKlSJJUlgxdZaK2qoI3nT6TX3ZuYP0Op85JklRshq4y8udnzGYwk+E/l61OuhRJksqOoauMzJpYx3knTOKmB1bTNzCYdDmSJJUVQ1eZuWzJbDbs7OH2JzYkXYokSWXF0FVmXrlgMjOaxvEf9zuhXpKkYjJ0lZmKdIq3LpnFPc9sYcWm3UmXI0lS2TB0laE/Pb2NqooUN97nhHpJkorF0FWGJtXX8OqF07i5Yw17evuTLkeSpLJg6CpT7zhrDju7+/neA2uSLkWSpLJg6CpTi2c3s3h2M9ffvZJ+l4+QJKngDF1l7D3nHsfz2/bys+Xrky5FkqQxz9BVxi5qn8JxreP5998+SyaTSbocSZLGNENXGUunU7x76XE8tnYH9z67JelyJEka0wxdZe4Np82gdUI1//7bZ5MuRZKkMc3QVeZqqyp4+yvmcGfcRFy/K+lyJEkaswxd4rIzZzOuqsLeLkmSCsjQJZrHV/Pml7fx49+vZf2O7qTLkSRpTDJ0CYB3nTOXgcEM//eelUmXIknSmGToEgBtLXW8ZtE0vnP/anZ29yVdjiRJY46hSy9633nHs6unn2/csyrpUiRJGnMMXXrRwhmNXHTiFK6761l27LW3S5KkfDJ0aT8fvnA+u7r7ueFu53ZJkpRPhi7t52XTG7lk4VRuuHsl2/f0Jl2OJEljhqFLf+DKC+ezq6ef6++yt0uSpHwxdOkPLJjawKUnTeP/3rOSrV32dkmSlA+GLo3owxfMZ0/fANfd5Sr1kiTlg6FLI5o/pZ7Xnjydb/5uFZt39yRdjiRJo56hSwf1oQvm09034DMZJUnKA0OXDur4SRN4/Skz+Na9q9i4y2cySpJ0LAxdekkfumA+/QMZvnD700mXIknSqGbo0kua0zqet71iNjctW01cvyvpciRJGrUMXTqkKy+Yz4SaSq75aWfSpUiSNGoZunRITXXVfOiC+fz2qU3cGTcmXY4kSaOSoUuH5fJXzGHOxDquua2T/oHBpMuRJGnUMXTpsFRXpvn4Je08vXE3Nz2wJulyJEkadQxdOmwXv2wKZ8xt4f/88il2dvclXY4kSaOKoUuHLZVKcdWlJ7Klq5ev/HpF0uVIkjSqGLp0RBbNbOQNp83ghrtXsmbrnqTLkSRp1DB06Yj9zcWBinSK//WTx8lkMkmXI0nSqGDo0hGb1jjhWke3AAAWxklEQVSOj1w0n9s7N/Jfj29IuhxJkkYFQ5eOyn8/ey7t0xr49I8fZ5eT6iVJOiRDl45KVUWaf/zjhWzY1c3nfvFU0uVIklTyDF06aqfOauayJbP55r2rePT57UmXI0lSSTN06Zj8zasDrRNq+MQtj7lSvSRJL8HQpWPSUFvFp//oZSxfu5Nv3vtc0uVIklSyDF06Zq9ZNJVXhkl87heRddv3Jl2OJEklydClY5ZKpfj71y1kMJPhb3/4mGt3SZI0AkOX8qKtpY6/vaSd3zy1iRvvX510OZIklZyCha4QwpIQwp0H7HtrCOHeYdtXhBAeDCHcF0L4b4WqRcXxtjNns3R+K9fc1snKzV1JlyNJUkkpSOgKIXwUuB6oHbbvFOBdQCq3PRX4EHA2cDHwmRBCTSHqUXGk0yn+9xtPproyzUe++4h3M0qSNEyherpWAG8Y2gghTAT+CfjwsGPOAO6JMfbEGHcAzwAnFageFcnUxlr+4fULeWTNdr5654qky5EkqWRUFuKiMcYfhBDmAIQQKoCvAx8Bht/a1gDsGLa9C2gc6XqdnZ15ra+7uzvv19Q+86vhvDnj+cLtTzG7pov5Ew+vA9N2KV22TWmyXUqXbVOakm6XgoSuAywG5gNfJTvceGII4QvAr4D6YcfVAyMua97e3p7Xgjo7O/N+Te3vi7OP5+Iv/JZrl+3g1g+eQ21VxSHPsV1Kl21TmmyX0mXblKZitEtHR8dBPyv43YsxxmUxxpfFGM8H3gI8EWP8MLAMWBpCqA0hNALtwPJC16PiaKqr5n+/8WSe2bibf/rZk0mXI0lS4hJbMiLGuB64FriLbK/XJ2OM3UnVo/w794RJvPPsuXzjd6u49dF1SZcjSVKiCja8GGNcBZz5UvtijNcB1xWqBiXv45cs4JE12/jYzY/SPq2B4ydNSLokSZIS4eKoKqjqyjRfeutp1FRV8P7/6GBPb3/SJUmSlAhDlwpuetM4vviWU3h6424+ectyHxMkSSpLhi4VxdL5k/jwBSdwy8Nr+c4yHxMkSSo/hi4VzQdfNY9zT5jE//rxEzz2/I5DnyBJ0hhi6FLRpNMpvvDmU2idUM17v/0gG3d5s6okqXwYulRULeOr+ffLT2fbnj6u+FYHe3sHki5JkqSiMHSp6BbOaOQLbzmFR5/fzl9//xEGB51YL0ka+wxdSsTFL5vKJy5p56ePredffhGTLkeSpIIrxrMXpRG9e+lcnt3cxVfuXMGc1vEsGp90RZIkFY49XUpMKpXi71/3MpbOb+UTP3yM37+wN+mSJEkqGEOXElVVkV2xfm7reP7h1xt4fJ1LSUiSxiZDlxLXOK6Kb7zzDOqqU1z+9WU8s3F30iVJkpR3hi6VhBlN4/jHi6aRSsHbvn4/a7buSbokSZLyytClkjGzsZpvv2sJXT39XPb1+9m408VTJUljh6FLJaV9WgPffOcZbNrVw2Vfv59tXb1JlyRJUl4YulRyTp3VzPVvP51VW/Zw+Q3L2LGnL+mSJEk6ZoYulaSzjm/l3y47jbh+F2+57j427+5JuiRJko6JoUsl61ULpvD1d5zOys27+dOv3cv6Hc7xkiSNXoYulbSl8yfxrXcuYePOHt70td95V6MkadQydKnknTG3hRvfvYRd3f286d/udR0vSdKoZOjSqHByWxM3vedM+gcHefPX7uX3a7YnXZIkSUfE0KVRY8HUBr733ldQV1PBm//9Xn6+fH3SJUmSdNgMXRpVjps0gVv+4mwWTG3g/Td2cN1vnyWTySRdliRJh2To0qjTOqGGm95zJpcsnMo1P+3kUz9aTv/AYNJlSZL0kgxdGpVqqyr40p+dxvvOO54b71/Nu775IDu7XURVklS6DF0atdLpFB+/ZAGfecMi7n5mM6/917vpfGFn0mVJkjQiQ5dGvT87YxY3vedM9vQO8MdfuYdbHn4+6ZIkSfoDhi6NCS+f08KtHzqHk2c28ZHv/p5P/egxevoHki5LkqQXGbo0Zkyur+XGdy/hvecex3/ct5o//dp9rmAvSSoZhi6NKZUVaf72Ne3822Wn8ezG3Vzyxbv4QcfzLishSUqcoUtj0qsXTuOnVy6lfVo9f/393/MXNz7Etq7epMuSJJUxQ5fGrLaWOm56zyv42KsXcHvnBi7+wm/5zVObki5LklSmDF0a0yrSKd5//vHc8hdn0ziuirffsIy//eFj7Njrml6SpOIydKksLJzRyE8+eA5XLJ3Ldx9YzQWf+w0/+f0653pJkorG0KWyUVtVwScvPZH/94FzmNpYwwf/82He+Y0HeH6bdzhKkgrP0KWys2hmIz/6i7P51KXt3L9yKxd9/rd89c4VruslSSooQ5fKUmVFmncvPY5f/tV5nD2vlc/+/Eku+vxv+dljLzjkKEkqCEOXytqMpnFc//bT+fa7zmBcVQXvv/Eh3vzv9/HY8zuSLk2SNMYYuiRg6fxJ3Pahc7jmjxeyYuNuXvvlu/mr7z7Cc1u6ki5NkjRGVCZdgFQqKivS/PmS2fzRydP58q+f4Rv3rOL//X4db1o8k7981TxmNtclXaIkaRSzp0s6QENtFX97STt3ffSVvO3M2fzwobW88l/u5KofLWf9ju6ky5MkjVKGLukgJjfU8unXvow7/+Z83nR6G/+5bDXn/vOv+djNj/LMxt1JlydJGmUMXdIhTG8axz/+8SJ+/T/O580vb+NHj6zlws//hiu+9SAPrtqadHmSpFHCOV3SYWprqeMfXr+QD184n2/e+xzfuncVv3xiA6fNauLtZ83hkoXTqK707xhJ0sj8F0I6QhMn1PBXF53A7z7+Kj79RyeypauXK296hLP+6Q7+5b8ia7fvTbpESVIJsqdLOkp11ZW84+y5XP6KOdz1zGa+fe8qvnznM3zlzme4oH0Kbz69jfPCJKoq/NtGkmToko5ZOp3ivBMmcd4Jk1izdQ//uWw133twDb98YgOtE6p5/Skz+JPFM2mf1pB0qZKkBBm6pDxqa6njo69ewEcuOoHfxE3c3PE837x3FdffvZKXTW/gtSdP5zWLptHW4ppfklRuDF1SAVRVpLnwxClceOIUtnX18uPfr+OHDz3PZ372JJ/52ZOc0tbEfztpGq9ZNI3pTeOSLleSVAQFC10hhCXAZ2OM54cQTgH+FRgAeoDLY4wbQghXAO8F+oGrY4y3FqoeKSnN46t5+1lzePtZc1izdQ+3PvoCtz22jqtv6+Tq2zo5aWYjF7ZP4cL2KbRPqyeVSiVdsiSpAAoSukIIHwXeBgw9uO6LwAdjjI+EEN4LfCyE8M/Ah4DTgVrg7hDCL2OMPYWoSSoFbS11vP/843n/+cezcnMXP33sBW7v3MDnf/kUn//lU8xoGseF7ZM5f8Fklsxtoa7azmhJGisK9Rt9BfAG4Nu57bfEGF8Y9j27gTOAe3IhqyeE8AxwEvBAgWqSSsrc1vF84JXz+MAr57FxVze/fnIjv3xiI999cA3fvPc5qivSvHxuM0vnT2Lp/FbapzaQTtsLJkmjVSqTyRTkwiGEOcBNMcYzh+07C/g6cC5wMbAoxvix3GffAr4VY7x9+HU6OjoydXX5nXTc3d1NbW1tXq+pY2e7ZPX0D7J8QzcPrdvLQy/sZdW2XgAaatIsmlLLoqnjOGlKLbObq0kXaSjStilNtkvpsm1KUzHaZc+ePSxevHjEX85FG7sIIbwZ+CRwaYxxUwhhJ1A/7JB6YPtI57a3t+e1ls7OzrxfU8fOdtnnlEVwWe79hp3d3PX0Zu5dsYX7V27hntVbAGiqq+L02c2cOquZ02Y1c3JbY8GGI22b0mS7lC7bpjQVo106OjoO+llRQlcI4TKyE+bPjzEOPaxuGXBNCKEWqAHageXFqEcaTaY01PLGxTN54+KZAKzdvpf7n93Cfc9u4cHntnF750YAKtIpFkyt59RZTZw0o4mFMxqZP2WCi7NKUokoeOgKIVQA1wKrgR+GEAB+E2P8uxDCtcBdZB9H9MkYY3eh65FGuxlN43jDaTN5w2nZELatq5dH1mznodXb6HhuGz96eB3/cd9qAGoq07RPa2DRjEbapzWwYFo9YUo942ucoC9JxVaw37wxxlXA0HyuloMccx1wXaFqkMpB8/hqXrlgMq9cMBmAwcEMq7Z08djaHTz2/A4eW7uDHz28lm/f9xwAqRTMbqkjTK3nhCn1zJs8gXmTJ3D8pAnUVlUk+aNI0pjmn7vSGJNOpzhu0gSOmzSB150yA4BMJsPz2/by5PpddL6wkyfX7+TJF3Zxe+dGBgazN9OkUtDWXMdxk8Yzt3X/r6FjJElHz9AllYFUKkVbSx1tLXVcdOKUF/f39A+wavMent64i6c37OaZTbtZuamLZSu3sqd34MXjKtPQ1rKRmc3jmNVSx6yWOmY21zG9qZYZzeNoHV/jchaSdAiGLqmM1VRWEKbWE6bW77c/k8mwcVcPz27qYtWWLh5+ag1d6XGs2bqH2x57ge17+vY7vroyzfTGWqY3jWNqQy1TG3NfDbVMaahlckMNrRNqnNQvqawZuiT9gVQqxZRcYHrF8RM5pb5rv9usd3b3sXbbXtZt38va7XtZu20vz2/fy/od3dy/cisbdnbTf8CQZCoFLXXVTKqvYVJ9NoS1Tqhm4oTs+4kTqmmpq6ZlfDXN46sZX13hI5EkjSmGLklHrKG2ioZpVbRPaxjx88HBDJu7eli/o5sNO3vYtKuHjbu62birh407e9i0u4eVm7vYvLuH7r7BEa9RXZGmeXwVzXXVNI6roqmuiqZx1TTVVdEwLvdVW0njsPf1tVXU11YyrsrAJqn0GLok5V06nWJyfS2T6w+98vOe3n427+plc1cP27p62drVy7Y9vWzp6mVbVy/b9/SxfW8fqzbvYfve7Wzb00dv/8hBbUhFOkV9bSXjqyuzrzXZrwk1FYyvzr6vq6548bWuuoJx1ZXUVQ29r6CuupLaqjTjqiqora5gXFWFw6OSjomhS1Ki6qormTWxklkTD/9xX919A+zs7mPn3n527O1j594+dvX0s6u7j13d+1539/Szu7ufrt5+duzpZe22fvb0DtDV009X78AR35VZkU5RW5mmtqqCmqHX3PuayvR+76uHXiuy76sr01RXVFBVmdq3ryJNVUWaqso0VenU/u8r01Tm9lVWpKhMp6mqSFFZkd1fmU6xO/ezVOSOS6ewh08qYYYuSaNObVUFtVUVTK4/9LEHk8lk6B0YpKtngD29/eztHWBv3wB7egfY25t97e7L7uvuG/5+MLc9SE//vtfe/kF27u2jp3/fdm//ID25196BwQItvfHcfluV6RQVuVCWzr1WpNNUpKEynSY99JrKhsh0av/jK1LZfen0vs+Hjkmn2G87ldseej/83FRq/+NTKUiR25c+YHvo81SKFLx43RS8eFfs8H1D77P7s9dI5d6ncu8ZdmyKffuHMmmK7I4Xzxs6Nnc8w48d+l4HnL/vPcOOz26tXdvF6sH1wz5Lvfh+/2sPqyd3+v7H7dsaKU7/wfnD9u133AgbqRGueMhzR6jraM4/3HMP9+zD/VtjYNhd2UkwdEkqS6lUiprKCmoqK2gZX12U7zkwmKFvIBvAhkJZ/0A2/PXt95WhfyCz//Zg9tj+wUH6B7Ofr133AhMnTaZ/MMPAYCa3f5CBTIaBgez2YCZD30CGwcFMdv/gAV+Z/T8bHHod5MVrDQ5mGMzw4ufZL3L7932WGdqfe81kstfN5PZlDtiXyUCGfedkxtxycBuSLkAHePmMcXz/5OS+v6FLkoqkIp2iIl2Rt5X/Ozv30t5+fF6uVSoyQ4GMfUFsMJfGXgxvueMGM8AB+zLsC3O5/17cHgp1mWHfhwM+zwyvI/fZ0Fn7n//i3n3XGLZv5cqVzJkzd9jPNexnHHbO0PWGf8+RzmG/T/b//ODn7LvugVcZ8bgRvsdh7jqyax72uYcnM9LJB1HVlWwQNnRJkkrG0BAhQMVhD0yVnvSOGtpnNCZdhg7Q2bkl0e/vrTiSJElFYOiSJEkqAkOXJElSERi6JEmSisDQJUmSVASGLkmSpCIwdEmSJBWBoUuSJKkIDF2SJElFYOiSJEkqAkOXJElSERi6JEmSisDQJUmSVASGLkmSpCIwdEmSJBWBoUuSJKkIDF2SJElFkMpkMknX8JI6OjpKu0BJkqRhFi9enBppf8mHLkmSpLHA4UVJkqQiMHRJkiQVQWXSBRRLCCENfAU4GegB3h1jfCbZqspXCKEKuAGYA9QAVwNPAN8AMsBy4AMxxsGESixrIYTJQAdwEdCP7VISQgh/C7wWqCb7++w32DaJyv0u+ybZ32UDwBX4/0ziQghLgM/GGM8PIcxjhPYIIfwdcCnZ9vpwjHFZoesqp56u1wO1McZXAB8HPpdwPeXuMmBLjHEpcAnwJeDzwKdy+1LA6xKsr2zl/hH5GrA3t8t2KQEhhPOBs4CzgfOANmybUvAaoDLGeBbw98A12C6JCiF8FLgeqM3t+oP2CCGcRvb/oyXAW4AvF6O2cgpd5wA/B4gx3gecnmw5Ze/7wFXDtvuBxWT/cgf4GXBhsYsSAP8C/BuwLrdtu5SGi4HHgFuAnwC3YtuUgqeAytxoSgPQh+2StBXAG4Ztj9Qe5wC/iDFmYoyrybbhpEIXVk6hqwHYMWx7IIRQNsOrpSbGuDvGuCuEUA/cDHwKSMUYh26n3QU0JlZgmQohvAPYFGP8r2G7bZfS0Er2j8U3Ae8DbgTStk3idpMdWnwSuA64Fv+fSVSM8Qdkw++QkdrjwExQlHYqp9C1E6gftp2OMfYnVYwghNAG/Br4dozxO8DwOQ/1wPZECitv7wQuCiHcCZwCfAuYPOxz2yU5W4D/ijH2xhgj0M3+/0jYNsn4CNl2OYHsnOFvkp1zN8R2Sd5I/7YcmAmK0k7lFLruITv2TgjhTLLd9EpICGEK8AvgYzHGG3K7H87NW4HsPK+7kqitnMUYz40xnhdjPB94BLgc+JntUhLuBl4dQkiFEKYD44E7bJvEbWNfj8lWoAp/l5WakdrjHuDiEEI6hDCLbEfM5kIXUk7Da7eQ/Qv+d2Qn0v33hOspd58AmoGrQghDc7uuBK4NIVQDnWSHHZW8vwaus12SFWO8NYRwLrCM7B/MHwBWYtsk7f8AN4QQ7iLbw/UJ4EFsl1LyB7/DYowDuTa7l33/PxWcK9JLkiQVQTkNL0qSJCXG0CVJklQEhi5JkqQiMHRJkiQVgaFLkiSpCMppyQhJY1RuDZ7vkX1o+pBNMcY3HeN1vwHcFGP8+bFcR5LA0CVp7PhVjPEtSRchSQdj6JI0ZuUeZ/QksIDsoshvjjGuDyF8juwDbwG+E2P8YghhPnA92QUu9wBDAe69IYSPkn3kzvtjjMuK+TNIGjsMXZLGilflQtaQ23Kvv4sxvi+E8BfAJ0IIvwDmAmeS/R14dwjhV8DVwGdijD8PIfwpcGru/I4Y49W5h4G/g+yK8JJ0xAxdksaKPxheDCFcCvwqt/k74HXAGuCuGGMG6Ash3AecCASyjwQhxvi93PlvBTpy568H6gr9Q0gau7x7UdJYtzj3ejbwONlnr50DEEKoAs4Cns7tf3lu/5+HED6YO89npUnKC3u6JI0VBw4vAowD3hFC+CugC3hbjHFLCOH8EMK9ZOdvfS/G+FAI4W+Ar4UQPkV2Ttdl7AtsknTMfOC1pDErF8LeF2N8MulaJMnhRUmSpCKwp0uSJKkI7OmSJEkqAkOXJElSERi6JEmSisDQJUmSVASGLkmSpCIwdEmSJBXB/w8r/AYZ0rbWKgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,8))\n",
    "plt.plot(range(1, n_ + 1), cost_);\n",
    "plt.ylabel('SSE');\n",
    "plt.xlabel('Epoch');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-7.57793828e-16, -7.33562757e-01])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}