ML0120EN-1.2-Review-LinearRegressionwithTensorFlow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.bigdatauniversity.com\"><img src=\"https://ibm.box.com/shared/static/qo20b88v1hbjztubt06609ovs85q8fau.png\" width=\"400px\" align=\"center\"></a>\n",
    "\n",
    "<h1 align=\"center\"><font size=\"5\">LINEAR REGRESSION WITH TENSORFLOW</font></h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "<font size=\"3\"><strong>In this notebook we will overview the implementation of Linear Regression with TensorFlow</strong></font>\n",
    "<br>\n",
    "<br>\n",
    "<h2>Table of Contents</h2>\n",
    "<ol>\n",
    " <li><a href=\"#ref1\">Linear Regression</a></li>\n",
    " <li><a href=\"#ref2\">Linear Regression with TensorFlow</a></li>\n",
    "</ol>\n",
    "</div>\n",
    "<br>\n",
    "<br>\n",
    "<p></p>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref1\"></a>\n",
    "<h1>Linear Regression</h1>\n",
    "\n",
    "Defining a linear regression in simple terms, is the approximation of a linear model used to describe the relationship between two or more variables. In a simple linear regression there are two variables, the dependent variable, which can be seen as the \"state\" or \"final goal\" that we study and try to predict, and the independent variables, also known as explanatory variables, which can be seen as the \"causes\" of the \"states\". \n",
    "\n",
    "When more than one independent variable is present the process is called multiple linear regression. <br>\n",
    "When multiple dependent variables are predicted the process is known as multivariate linear regression.\n",
    "\n",
    "The equation of a simple linear model is\n",
    "\n",
    "$$Y = a X + b $$\n",
    "\n",
    "Where Y is the dependent variable and X is the independent variable, and <b>a</b> and <b>b</b> being the parameters we adjust. <b>a</b> is known as \"slope\" or \"gradient\" and <b>b</b> is the \"intercept\". You can interpret this equation as Y being a function of X, or Y being dependent on X.\n",
    "\n",
    "If you plot the model, you will see it is a line, and by adjusting the \"slope\" parameter you will change the angle between the line and the independent variable axis, and the \"intercept parameter\" will affect where it crosses the dependent variable's axis.\n",
    "\n",
    "Let's first import the required packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import pylab as pl\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import matplotlib.patches as mpatches\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10, 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define the independent variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.arange(0.0, 5.0, 0.1)\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##You can adjust the slope and intercept to verify the changes in the graph\n",
    "a = 1\n",
    "b = 0\n",
    "\n",
    "Y= a * X + b \n",
    "\n",
    "plt.plot(X, Y) \n",
    "plt.ylabel('Dependent Variable')\n",
    "plt.xlabel('Indepdendent Variable')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK... but how can we see this concept of linear relations with a more meaningful point of view?\n",
    "\n",
    "Simple linear relations were used to try to describe and quantify many observable physical phenomena, the easiest to understand are speed and distance traveled:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b><pre>\n",
    "\n",
    "$$Distance Traveled = Speed \\times Time + Initial Distance$$\n",
    "\n",
    "$$Speed = Acceleration \\times Time + Initial Speed$$\n",
    "</pre></b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They are also used to describe properties of different materials:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b><pre>\n",
    "\n",
    "$$Force = Deformation \\times Stiffness$$\n",
    "\n",
    "$$Heat Transfered = Temperature Difference \\times Thermal Conductivity$$\n",
    "\n",
    "$$Electrical Tension (Voltage) = Electrical Current \\times Resistance$$\n",
    "\n",
    "$$Mass =  Volume \\times Density$$\n",
    "</pre></b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we perform an experiment and gather the data, or if we already have a dataset and we want to perform a linear regression, what we will do is adjust a simple linear model to the dataset, we adjust the \"slope\" and \"intercept\" parameters to the data the best way possible, because the closer the model comes to describing each ocurrence, the better it will be at representing them.\n",
    "\n",
    "So how is this \"regression\" performed?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref2\"></a>\n",
    "<h1>Linear Regression with TensorFlow</h1>\n",
    "A simple example of a linear function can help us understand the basic mechanism behind TensorFlow.\n",
    "\n",
    "For the first part we will use a sample dataset, and then we'll use TensorFlow to adjust and get the right parameters. We download a dataset that is related to fuel consumption and Carbon dioxide emission of cars. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -O FuelConsumption.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/FuelConsumptionCo2.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Understanding the Data</h2>\n",
    "\n",
    "<h3><code>FuelConsumption.csv</code>:</h3>\n",
    "We have downloaded a fuel consumption dataset, <b><code>FuelConsumption.csv</code></b>, which contains model-specific fuel consumption ratings and estimated carbon dioxide emissions for new light-duty vehicles for retail sale in Canada. <a href=\"http://open.canada.ca/data/en/dataset/98f1a129-f628-4ce4-b24d-6f16bf24dd64\">Dataset source</a>\n",
    "\n",
    "- **MODELYEAR** e.g. 2014\n",
    "- **MAKE** e.g. Acura\n",
    "- **MODEL** e.g. ILX\n",
    "- **VEHICLE CLASS** e.g. SUV\n",
    "- **ENGINE SIZE** e.g. 4.7\n",
    "- **CYLINDERS** e.g 6\n",
    "- **TRANSMISSION** e.g. A6\n",
    "- **FUEL CONSUMPTION in CITY(L/100 km)** e.g. 9.9\n",
    "- **FUEL CONSUMPTION in HWY (L/100 km)** e.g. 8.9\n",
    "- **FUEL CONSUMPTION COMB (L/100 km)** e.g. 9.2\n",
    "- **CO2 EMISSIONS (g/km)** e.g. 182   --> low --> 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"FuelConsumption.csv\")\n",
    "\n",
    "# take a look at the dataset\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets say we want to use linear regression to predict Co2Emission of cars based on their engine size. So, lets define X and Y value for the linear regression, that is, train_x and train_y:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x = np.asanyarray(df[['ENGINESIZE']])\n",
    "train_y = np.asanyarray(df[['CO2EMISSIONS']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we initialize the variables <b>a</b> and <b>b</b>, with any random guess, and then we define the linear function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = tf.Variable(20.0)\n",
    "b = tf.Variable(30.2)\n",
    "y = a * train_x + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are going to define a loss function for our regression, so we can train our model to better fit our data. In a linear regression, we minimize the squared error of the difference between the predicted values(obtained from the equation) and the target values (the data that we have). In other words we want to minimize the square of the predicted values minus the target value. So we define the equation to be minimized as loss.\n",
    "\n",
    "To find value of our loss, we use <b>tf.reduce_mean()</b>. This function finds the mean of a multidimensional tensor, and the result can have a different dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = tf.reduce_mean(tf.square(y - train_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we define the optimizer method. The gradient Descent optimizer takes in parameter: learning rate, which corresponds to the speed with which the optimizer should learn; there are pros and cons for increasing the learning-rate parameter, with a high learning rate the training model converges quickly, but there is a risk that a high learning rate causes instability and the model will not converge. <b>Please feel free to make changes to learning parameter and check its effect</b>. On the other hand decreasing the learning rate might reduce the convergence speed, but it would increase the chance of converging to a solution. You should note that the solution might not be a global optimal solution as there is a chance that the optimizer will get stuck in a local optimal solution. Please review other material for further information on the optimization. Here we will use a simple gradient descent with a learning rate of 0.05: <br>  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.train.GradientDescentOptimizer(0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will define the training method of our graph, what method we will use for minimize the loss? We will use the <b>.minimize()</b> which will minimize the error function of our optimizer, resulting in a better model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = optimizer.minimize(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Don't forget to initialize the variables before executing a graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "sess = tf.Session()\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to start the optimization and run the graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_values = []\n",
    "train_data = []\n",
    "for step in range(100):\n",
    "    _, loss_val, a_val, b_val = sess.run([train, loss, a, b])\n",
    "    loss_values.append(loss_val)\n",
    "    if step % 5 == 0:\n",
    "        print(step, loss_val, a_val, b_val)\n",
    "        train_data.append([a_val, b_val])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets plot the loss values to see how it has changed during the training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(loss_values, 'ro')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets visualize how the coefficient and intercept of line has changed to fit the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cr, cg, cb = (1.0, 1.0, 0.0)\n",
    "for f in train_data:\n",
    "    cb += 1.0 / len(train_data)\n",
    "    cg -= 1.0 / len(train_data)\n",
    "    if cb > 1.0: cb = 1.0\n",
    "    if cg < 0.0: cg = 0.0\n",
    "    [a, b] = f\n",
    "    f_y = np.vectorize(lambda x: a*x + b)(train_x)\n",
    "    line = plt.plot(train_x, f_y)\n",
    "    plt.setp(line, color=(cr,cg,cb))\n",
    "\n",
    "plt.plot(train_x, train_y, 'ro')\n",
    "\n",
    "\n",
    "green_line = mpatches.Patch(color='red', label='Data Points')\n",
    "\n",
    "plt.legend(handles=[green_line])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Want to learn more?\n",
    "\n",
    "Running deep learning programs usually needs a high performance platform. __PowerAI__ speeds up deep learning and AI. Built on IBM’s Power Systems, __PowerAI__ is a scalable software platform that accelerates deep learning and AI with blazing performance for individual users or enterprises. The __PowerAI__ platform supports popular machine learning libraries and dependencies including TensorFlow, Caffe, Torch, and Theano. You can use [PowerAI on IMB Cloud](https://cocl.us/ML0120EN_PAI).\n",
    "\n",
    "Also, you can use __Watson Studio__ to run these notebooks faster with bigger datasets.__Watson Studio__ is IBM’s leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, __Watson Studio__ enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of __Watson Studio__ users today with a free account at [Watson Studio](https://cocl.us/ML0120EN_DSX).This is the end of this lesson. Thank you for reading this notebook, and good luck on your studies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thanks for completing this lesson!\n",
    "\n",
    "If you are familiar with some of these methods and concepts, this tutorial might have been boring for you, but it is important to get used to the TensorFlow mechanics, and feel familiar and comfortable using it, so you can build more complex algorithms in it.\n",
    "\n",
    "Created by <a href=\"https://linkedin.com/in/saeedaghabozorgi\"> Saeed Aghabozorgi </a> , <a href=\"https://ca.linkedin.com/in/rafaelblsilva\"> Rafael Belo Da Silva</a><br>\n",
    "\n",
    "This tutorial was inspired by the documentation of TensorFlow : https://www.tensorflow.org/versions/r0.9/get_started/index.html <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "Copyright &copy; 2018 [Cognitive Class](https://cocl.us/DX0108EN_CC). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}