Programming Assignment 3.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "# Assignment 3\n\nWelcome to Assignment 3. This will be even more fun. Now we will calculate statistical measures. \n\n##\u00a0You only have to pass 4 out of 7 functions\n\nJust make sure you hit the play button on each cell from top to down. There are seven functions you have to implement. Please also make sure than on each change on a function you hit the play button again on the corresponding cell to make it available to the rest of this notebook."
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "This notebook is designed to run in a IBM Watson Studio default runtime (NOT the Watson Studio Apache Spark Runtime as the default runtime with 1 vCPU is free of charge). Therefore, we install Apache Spark in local mode for test purposes only. Please don't use it in production.\n\nIn case you are facing issues, please read the following two documents first:\n\nhttps://github.com/IBM/skillsnetwork/wiki/Environment-Setup\n\nhttps://github.com/IBM/skillsnetwork/wiki/FAQ\n\nThen, please feel free to ask:\n\nhttps://coursera.org/learn/machine-learning-big-data-apache-spark/discussions/all\n\nPlease make sure to follow the guidelines before asking a question:\n\nhttps://github.com/IBM/skillsnetwork/wiki/FAQ#im-feeling-lost-and-confused-please-help-me\n\n\nIf running outside Watson Studio, this should work as well. In case you are running in an Apache Spark context outside Watson Studio, please remove the Apache Spark setup in the first notebook cells."
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "metadata": {},
            "outputs": [],
            "source": "from IPython.display import Markdown, display\ndef printmd(string):\n    display(Markdown('# <span style=\"color:red\">'+string+'</span>'))\n\n\nif ('sc' in locals() or 'sc' in globals()):\n    printmd('<<<<<!!!!! It seems that you are running in a IBM Watson Studio Apache Spark Notebook. Please run it in an IBM Watson Studio Default Runtime (without Apache Spark) !!!!!>>>>>')\n"
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "Collecting pyspark==2.4.5\n  Downloading pyspark-2.4.5.tar.gz (217.8 MB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 217.8 MB 9.7 kB/s  eta 0:00:01   |\u2588\u258f                              | 8.3 MB 12.5 MB/s eta 0:00:17 MB 37.0 MB/s eta 0:00:06     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b          | 147.3 MB 37.3 MB/s eta 0:00:02\n\u001b[?25hCollecting py4j==0.10.7\n  Downloading py4j-0.10.7-py2.py3-none-any.whl (197 kB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 197 kB 12.1 MB/s eta 0:00:01\n\u001b[?25hBuilding wheels for collected packages: pyspark\n  Building wheel for pyspark (setup.py) ... \u001b[?25ldone\n\u001b[?25h  Created wheel for pyspark: filename=pyspark-2.4.5-py2.py3-none-any.whl size=218257927 sha256=b36acce8978c813ba90078012858e40b1d049f897de36b722aeb49d27a6a584f\n  Stored in directory: /tmp/wsuser/.cache/pip/wheels/01/c0/03/1c241c9c482b647d4d99412a98a5c7f87472728ad41ae55e1e\nSuccessfully built pyspark\nInstalling collected packages: py4j, pyspark\nSuccessfully installed py4j-0.10.7 pyspark-2.4.5\n"
                }
            ],
            "source": "!pip install pyspark==2.4.5"
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "metadata": {},
            "outputs": [],
            "source": "try:\n    from pyspark import SparkContext, SparkConf\n    from pyspark.sql import SparkSession\nexcept ImportError as e:\n    printmd('<<<<<!!!!! Please restart your kernel after installing Apache Spark !!!!!>>>>>')"
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "metadata": {},
            "outputs": [],
            "source": "sc = SparkContext.getOrCreate(SparkConf().setMaster(\"local[*]\"))\n\nspark = SparkSession \\\n    .builder \\\n    .getOrCreate()"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "All functions can be implemented using DataFrames, ApacheSparkSQL or RDDs. We are only interested in the result. You are given the reference to the data frame in the \"df\" parameter and in case you want to use SQL just use the \"spark\" parameter which is a reference to the global SparkSession object. Finally if you want to use RDDs just use \"df.rdd\" for obtaining a reference to the underlying RDD object. But we discurage using RDD at this point in time.\n\nLet's start with the first function. Please calculate the minimal temperature for the test data set you have created. We've provided a little skeleton for you in case you want to use SQL. Everything can be implemented using SQL only if you like."
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "metadata": {},
            "outputs": [],
            "source": "def minTemperature(df,spark):\n    #TODO Please enter your code here, you are not required to use the template code below\n    #some reference: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame\n    return spark.sql(\"SELECT MIN(temperature) as mintemp from washing\").first().mintemp"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Please now do the same for the mean of the temperature"
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "metadata": {},
            "outputs": [],
            "source": "def meanTemperature(df,spark):\n    #TODO Please enter your code here, you are not required to use the template code below\n    #some reference: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame\n    return spark.sql(\"SELECT AVG(temperature) as meantemp from washing\").first().meantemp"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Please now do the same for the maximum of the temperature"
        },
        {
            "cell_type": "code",
            "execution_count": 7,
            "metadata": {},
            "outputs": [],
            "source": "def maxTemperature(df,spark):\n    #TODO Please enter your code here, you are not required to use the template code below\n    #some reference: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame\n    return spark.sql(\"SELECT MAX(temperature) as maxtemp from washing\").first().maxtemp"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Please now do the same for the standard deviation of the temperature"
        },
        {
            "cell_type": "code",
            "execution_count": 8,
            "metadata": {},
            "outputs": [],
            "source": "def sdTemperature(df,spark):\n    #TODO Please enter your code here, you are not required to use the template code below\n    #some reference: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame\n    #https://spark.apache.org/docs/2.3.0/api/sql/\n    return spark.sql(\"SELECT STDDEV(temperature) as sdtemp from washing\").first().sdtemp"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Please now do the same for the skew of the temperature. Since the SQL statement for this is a bit more complicated we've provided a skeleton for you. You have to insert custom code at four positions in order to make the function work. Alternatively you can also remove everything and implement if on your own. Note that we are making use of two previously defined functions, so please make sure they are correct. Also note that we are making use of python's string formatting capabilitis where the results of the two function calls to \"meanTemperature\" and \"sdTemperature\" are inserted at the \"%s\" symbols in the SQL string."
        },
        {
            "cell_type": "code",
            "execution_count": 9,
            "metadata": {},
            "outputs": [],
            "source": "def skewTemperature(df,spark):    \n    return spark.sql(\"\"\"\nSELECT \n    (\n        count(temperature)/(count(temperature)-1)/(count(temperature)-2)\n    ) *\n    SUM (\n        POWER(temperature-%s,3)/POWER(%s,3)\n    )\n\nas sktemperature from washing\n                    \"\"\" %(meanTemperature(df,spark),sdTemperature(df,spark))).first().sktemperature"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Kurtosis is the 4th statistical moment, so if you are smart you can make use of the code for skew which is the 3rd statistical moment. Actually only two things are different."
        },
        {
            "cell_type": "code",
            "execution_count": 10,
            "metadata": {},
            "outputs": [],
            "source": "def kurtosisTemperature(df,spark):    \n        return spark.sql(\"\"\"\nSELECT \n    (\n        1/COUNT(temperature)\n    ) *\n    SUM (\n        POWER(temperature-%s,4)/POWER(%s,4)\n    )\nas ktemperature from washing\n                    \"\"\" %(meanTemperature(df,spark),sdTemperature(df,spark))).first().ktemperature\n"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Just a hint. This can be solved easily using SQL as well, but as shown in the lecture also using RDDs."
        },
        {
            "cell_type": "code",
            "execution_count": 11,
            "metadata": {},
            "outputs": [],
            "source": "def correlationTemperatureHardness(df,spark):\n    #TODO Please enter your code here, you are not required to use the template code below\n    #some reference: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame\n    #https://spark.apache.org/docs/2.3.0/api/sql/\n    return spark.sql(\"SELECT corr(temperature,hardness) as temperaturehardness from washing\").first().temperaturehardness"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Now it is time to grab a PARQUET file and create a dataframe out of it. Using SparkSQL you can handle it like a database. "
        },
        {
            "cell_type": "code",
            "execution_count": 12,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "--2020-12-08 12:03:24--  https://github.com/IBM/coursera/blob/master/coursera_ds/washing.parquet?raw=true\nResolving github.com (github.com)... 140.82.113.3\nConnecting to github.com (github.com)|140.82.113.3|:443... connected.\nHTTP request sent, awaiting response... 301 Moved Permanently\nLocation: https://github.com/IBM/skillsnetwork/blob/master/coursera_ds/washing.parquet?raw=true [following]\n--2020-12-08 12:03:24--  https://github.com/IBM/skillsnetwork/blob/master/coursera_ds/washing.parquet?raw=true\nReusing existing connection to github.com:443.\nHTTP request sent, awaiting response... 302 Found\nLocation: https://github.com/IBM/skillsnetwork/raw/master/coursera_ds/washing.parquet [following]\n--2020-12-08 12:03:24--  https://github.com/IBM/skillsnetwork/raw/master/coursera_ds/washing.parquet\nReusing existing connection to github.com:443.\nHTTP request sent, awaiting response... 302 Found\nLocation: https://raw.githubusercontent.com/IBM/skillsnetwork/master/coursera_ds/washing.parquet [following]\n--2020-12-08 12:03:25--  https://raw.githubusercontent.com/IBM/skillsnetwork/master/coursera_ds/washing.parquet\nResolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\nConnecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 112048 (109K) [application/octet-stream]\nSaving to: \u2018washing.parquet?raw=true\u2019\n\nwashing.parquet?raw 100%[===================>] 109.42K  --.-KB/s    in 0.003s  \n\n2020-12-08 12:03:25 (36.6 MB/s) - \u2018washing.parquet?raw=true\u2019 saved [112048/112048]\n\n"
                }
            ],
            "source": "!wget https://github.com/IBM/coursera/blob/master/coursera_ds/washing.parquet?raw=true\n!mv washing.parquet?raw=true washing.parquet"
        },
        {
            "cell_type": "code",
            "execution_count": 13,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "+--------------------+--------------------+-----+--------+----------+---------+--------+-----+-----------+-------------+-------+\n|                 _id|                _rev|count|flowrate|fluidlevel|frequency|hardness|speed|temperature|           ts|voltage|\n+--------------------+--------------------+-----+--------+----------+---------+--------+-----+-----------+-------------+-------+\n|0d86485d0f88d1f9d...|1-57940679fb8a713...|    4|      11|acceptable|     null|      77| null|        100|1547808723923|   null|\n|0d86485d0f88d1f9d...|1-15ff3a0b304d789...|    2|    null|      null|     null|    null| 1046|       null|1547808729917|   null|\n|0d86485d0f88d1f9d...|1-97c2742b68c7b07...|    4|    null|      null|       71|    null| null|       null|1547808731918|    236|\n|0d86485d0f88d1f9d...|1-eefb903dbe45746...|   19|      11|acceptable|     null|      75| null|         86|1547808738999|   null|\n|0d86485d0f88d1f9d...|1-5f68b4c72813c25...|    7|    null|      null|       75|    null| null|       null|1547808740927|    235|\n|0d86485d0f88d1f9d...|1-cd4b6c57ddbe77e...|    5|    null|      null|     null|    null| 1014|       null|1547808744923|   null|\n|0d86485d0f88d1f9d...|1-a35b25b5bf43aaf...|   32|      11|acceptable|     null|      73| null|         84|1547808752028|   null|\n|0d86485d0f88d1f9d...|1-b717f7289a8476d...|   48|      11|acceptable|     null|      79| null|         84|1547808768065|   null|\n|0d86485d0f88d1f9d...|1-c2f1f8fcf178b2f...|   18|    null|      null|       73|    null| null|       null|1547808773944|    228|\n|0d86485d0f88d1f9d...|1-15033dd9eebb4a8...|   59|      11|acceptable|     null|      72| null|         96|1547808779093|   null|\n|0d86485d0f88d1f9d...|1-753dae825f9a6c2...|   62|      11|acceptable|     null|      73| null|         88|1547808782113|   null|\n|0d86485d0f88d1f9d...|1-b168089f44f03f0...|   13|    null|      null|     null|    null| 1097|       null|1547808784940|   null|\n|0d86485d0f88d1f9d...|1-403b687c6be0dea...|   23|    null|      null|       80|    null| null|       null|1547808788955|    236|\n|0d86485d0f88d1f9d...|1-195551e0455a24b...|   72|      11|acceptable|     null|      77| null|         87|1547808792134|   null|\n|0d86485d0f88d1f9d...|1-060a39fc6c2ddee...|   26|    null|      null|       62|    null| null|       null|1547808797959|    233|\n|0d86485d0f88d1f9d...|1-2234514bffee465...|   27|    null|      null|       61|    null| null|       null|1547808800960|    226|\n|0d86485d0f88d1f9d...|1-4265898bb401db0...|   82|      11|acceptable|     null|      79| null|         96|1547808802154|   null|\n|0d86485d0f88d1f9d...|1-2fbf7ca9a0425a0...|   94|      11|acceptable|     null|      73| null|         90|1547808814186|   null|\n|0d86485d0f88d1f9d...|1-203c0ee6d7fbd21...|   97|      11|acceptable|     null|      77| null|         88|1547808817190|   null|\n|0d86485d0f88d1f9d...|1-47e1965db94fcab...|  104|      11|acceptable|     null|      75| null|         80|1547808824198|   null|\n+--------------------+--------------------+-----+--------+----------+---------+--------+-----+-----------+-------------+-------+\nonly showing top 20 rows\n\n"
                }
            ],
            "source": "df = spark.read.parquet('washing.parquet')\ndf.createOrReplaceTempView('washing')\ndf.show()"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Now let's test the functions you've implemented"
        },
        {
            "cell_type": "code",
            "execution_count": 14,
            "metadata": {},
            "outputs": [],
            "source": "min_temperature = 0\nmean_temperature = 0\nmax_temperature = 0\nsd_temperature = 0\nskew_temperature = 0\nkurtosis_temperature = 0\ncorrelation_temperature = 0\n"
        },
        {
            "cell_type": "code",
            "execution_count": 15,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "80\n"
                }
            ],
            "source": "min_temperature = minTemperature(df,spark)\nprint(min_temperature)"
        },
        {
            "cell_type": "code",
            "execution_count": 16,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "90.03800298062593\n"
                }
            ],
            "source": "mean_temperature = meanTemperature(df,spark)\nprint(mean_temperature)"
        },
        {
            "cell_type": "code",
            "execution_count": 17,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "100\n"
                }
            ],
            "source": "max_temperature = maxTemperature(df,spark)\nprint(max_temperature)"
        },
        {
            "cell_type": "code",
            "execution_count": 18,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": "6.1007610586219725\n"
                }
            ],
            "source": "sd_temperature = sdTemperature(df,spark)\nprint(sd_temperature)"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "skew_temperature = skewTemperature(df,spark)\nprint(skew_temperature)"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "kurtosis_temperature = kurtosisTemperature(df,spark)\nprint(kurtosis_temperature)"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "correlation_temperature = correlationTemperatureHardness(df,spark)\nprint(correlation_temperature)"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "Congratulations, you are done, please submit this notebook to the grader. \nWe have to install a little library in order to submit to coursera first.\n\nThen, please provide your email address and obtain a submission token on the grader\u2019s submission page in coursera, then execute the subsequent cells\n\n### Note: We've changed the grader in this assignment and will do so for the others soon since it gives less errors\nThis means you can directly submit your solutions from this notebook"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "!rm -f rklib.py\n!wget https://raw.githubusercontent.com/IBM/coursera/master/rklib.py"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "from rklib import submitAll\nimport json\n\nkey = \"Suy4biHNEeimFQ479R3GjA\"\nemail = \"gerald.suleh@strathmore.edu\"\ntoken = \"UsXt9QPNGYRGAPeY\"\n\n\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": "parts_data = {}\nparts_data[\"FWMEL\"] = json.dumps(min_temperature)\nparts_data[\"3n3TK\"] = json.dumps(mean_temperature)\nparts_data[\"KD3By\"] = json.dumps(max_temperature)\nparts_data[\"06Zie\"] = json.dumps(sd_temperature)\nparts_data[\"Qc8bI\"] = json.dumps(skew_temperature)\nparts_data[\"LoqQi\"] = json.dumps(kurtosis_temperature)\nparts_data[\"ehNGV\"] = json.dumps(correlation_temperature)\n\n\n\nsubmitAll(email, token, key, parts_data)"
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.7",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.7.9"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 1
}