Feature Engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the last assignment for the Coursera course \"Advanced Machine Learning and Signal Processing\"\n",
    "\n",
    "Just execute all cells one after the other and you are done - just note that in the last one you should update your email address (the one you've used for coursera) and obtain a submission token, you get this from the programming assignment directly on coursera.\n",
    "\n",
    "Please fill in the sections labelled with \"###YOUR_CODE_GOES_HERE###\"\n",
    "\n",
    "The purpose of this assignment is to learn how feature engineering boosts model performance. You will apply Discrete Fourier Transformation on the accelerometer sensor time series and therefore transforming the dataset from the time to the frequency domain. \n",
    "\n",
    "After that, you’ll use a classification algorithm of your choice to create a model and submit the new predictions to the grader. Done.\n",
    "\n",
    "Please make sure you run this notebook from an Apache Spark 2.3 notebook.\n",
    "\n",
    "So the first thing we need to ensure is that we are on the latest version of SystemML, which is 1.3.0 (as of 20th March'19) Please use the code block below to check if you are already on 1.3.0 or higher. 1.3 contains a necessary fix, that's we are running against the SNAPSHOT\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for a Spark session to start...\n",
      "Spark Initialization Done! ApplicationId = app-20190902005623-0001\n",
      "KERNEL_ID = 486c4b6d-39c7-46b8-a215-45fa76b6c77b\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'1.3.0-SNAPSHOT'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from systemml import MLContext\n",
    "ml = MLContext(spark)\n",
    "ml.version()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "If you are blow version 1.3.0, or you got the error message \"No module named 'systemml'\"  please execute the next two code blocks and then\n",
    "\n",
    "# PLEASE RESTART THE KERNEL !!!\n",
    "\n",
    "Otherwise your changes won't take effect, just double-check every time you run this notebook if you are on SystemML 1.3\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting https://github.com/IBM/coursera/blob/master/systemml-1.3.0-SNAPSHOT-python.tar.gz?raw=true\n",
      "\u001b[?25l  Downloading https://github.com/IBM/coursera/blob/master/systemml-1.3.0-SNAPSHOT-python.tar.gz?raw=true (9.9MB)\n",
      "\u001b[K    100% |################################| 9.9MB 1.3MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting numpy>=1.8.2 (from systemml==1.3.0)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/75/92/57179ed45307ec6179e344231c47da7f3f3da9e2eee5c8ab506bd279ce4e/numpy-1.17.1-cp36-cp36m-manylinux1_x86_64.whl (20.4MB)\n",
      "\u001b[K    100% |################################| 20.4MB 1.0MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting scipy>=0.15.1 (from systemml==1.3.0)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/29/50/a552a5aff252ae915f522e44642bb49a7b7b31677f9580cfd11bcc869976/scipy-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (25.2MB)\n",
      "\u001b[K    100% |################################| 25.2MB 946kB/s eta 0:00:01\n",
      "\u001b[?25hCollecting pandas (from systemml==1.3.0)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/73/9b/52e228545d14f14bb2a1622e225f38463c8726645165e1cb7dde95bfe6d4/pandas-0.25.1-cp36-cp36m-manylinux1_x86_64.whl (10.5MB)\n",
      "\u001b[K    100% |################################| 10.5MB 1.8MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting scikit-learn (from systemml==1.3.0)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a0/c5/d2238762d780dde84a20b8c761f563fe882b88c5a5fb03c056547c442a19/scikit_learn-0.21.3-cp36-cp36m-manylinux1_x86_64.whl (6.7MB)\n",
      "\u001b[K    100% |################################| 6.7MB 2.3MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting Pillow>=2.0.0 (from systemml==1.3.0)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/14/41/db6dec65ddbc176a59b89485e8cc136a433ed9c6397b6bfe2cd38412051e/Pillow-6.1.0-cp36-cp36m-manylinux1_x86_64.whl (2.1MB)\n",
      "\u001b[K    100% |################################| 2.1MB 4.2MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting python-dateutil>=2.6.1 (from pandas->systemml==1.3.0)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)\n",
      "\u001b[K    100% |################################| 235kB 5.7MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting pytz>=2017.2 (from pandas->systemml==1.3.0)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/87/76/46d697698a143e05f77bec5a526bf4e56a0be61d63425b68f4ba553b51f2/pytz-2019.2-py2.py3-none-any.whl (508kB)\n",
      "\u001b[K    100% |################################| 512kB 5.3MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting joblib>=0.11 (from scikit-learn->systemml==1.3.0)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/cd/c1/50a758e8247561e58cb87305b1e90b171b8c767b15b12a1734001f41d356/joblib-0.13.2-py2.py3-none-any.whl (278kB)\n",
      "\u001b[K    100% |################################| 286kB 3.9MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting six>=1.5 (from python-dateutil>=2.6.1->pandas->systemml==1.3.0)\n",
      "  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl\n",
      "Building wheels for collected packages: systemml\n",
      "  Building wheel for systemml (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /home/spark/shared/.cache/pip/wheels/aa/bf/28/4344dd13abd8b9b6cbd4032baf4b851873d2e2288a65631fd2\n",
      "Successfully built systemml\n",
      "\u001b[31mtensorflow 1.13.1 requires tensorboard<1.14.0,>=1.13.0, which is not installed.\u001b[0m\n",
      "Installing collected packages: numpy, scipy, six, python-dateutil, pytz, pandas, joblib, scikit-learn, Pillow, systemml\n",
      "Successfully installed Pillow-6.1.0 joblib-0.13.2 numpy-1.17.1 pandas-0.25.1 python-dateutil-2.8.0 pytz-2019.2 scikit-learn-0.21.3 scipy-1.3.1 six-1.12.0 systemml-1.3.0\n"
     ]
    }
   ],
   "source": [
    "!pip install https://github.com/IBM/coursera/blob/master/systemml-1.3.0-SNAPSHOT-python.tar.gz?raw=true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Now we need to create two sym links that the newest version is picket up - this is a workaround and will be removed as soon as SystemML 1.3 will be pre-installed on Watson Studio once officially released.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ln -s -f ~/user-libs/python3/systemml/systemml-java/systemml-1.3.0-SNAPSHOT-extra.jar ~/user-libs/spark2/systemml-1.3.0-SNAPSHOT-extra.jar\n",
    "!ln -s -f ~/user-libs/python3/systemml/systemml-java/systemml-1.3.0-SNAPSHOT.jar ~/user-libs/spark2/systemml-1.3.0-SNAPSHOT.jar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Please now restart the kernel and start from the beginning to make sure you've installed SystemML 1.3\n",
    "\n",
    "Let's download the test data since it's so small we don't use COS (IBM Cloud Object Store) here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-09-02 00:56:44--  https://github.com/IBM/coursera/blob/master/coursera_ml/shake.parquet?raw=true\n",
      "Resolving github.com (github.com)... 140.82.114.4\n",
      "Connecting to github.com (github.com)|140.82.114.4|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://github.com/IBM/coursera/raw/master/coursera_ml/shake.parquet [following]\n",
      "--2019-09-02 00:56:44--  https://github.com/IBM/coursera/raw/master/coursera_ml/shake.parquet\n",
      "Reusing existing connection to github.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/IBM/coursera/master/coursera_ml/shake.parquet [following]\n",
      "--2019-09-02 00:56:44--  https://raw.githubusercontent.com/IBM/coursera/master/coursera_ml/shake.parquet\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 74727 (73K) [application/octet-stream]\n",
      "Saving to: 'shake.parquet?raw=true'\n",
      "\n",
      "shake.parquet?raw=t 100%[===================>]  72.98K  --.-KB/s    in 0.006s  \n",
      "\n",
      "2019-09-02 00:56:44 (11.9 MB/s) - 'shake.parquet?raw=true' saved [74727/74727]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://github.com/IBM/coursera/blob/master/coursera_ml/shake.parquet?raw=true\n",
    "!mv shake.parquet?raw=true shake.parquet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it’s time to read the sensor data and create a temporary query table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=spark.read.parquet('shake.parquet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+---------+-----+-----+-----+\n",
      "|CLASS| SENSORID|    X|    Y|    Z|\n",
      "+-----+---------+-----+-----+-----+\n",
      "|    2| qqqqqqqq| 0.12| 0.12| 0.12|\n",
      "|    2|aUniqueID| 0.03| 0.03| 0.03|\n",
      "|    2| qqqqqqqq|-3.84|-3.84|-3.84|\n",
      "|    2| 12345678| -0.1| -0.1| -0.1|\n",
      "|    2| 12345678|-0.15|-0.15|-0.15|\n",
      "|    2| 12345678| 0.47| 0.47| 0.47|\n",
      "|    2| 12345678|-0.06|-0.06|-0.06|\n",
      "|    2| 12345678|-0.09|-0.09|-0.09|\n",
      "|    2| 12345678| 0.21| 0.21| 0.21|\n",
      "|    2| 12345678|-0.08|-0.08|-0.08|\n",
      "|    2| 12345678| 0.44| 0.44| 0.44|\n",
      "|    2|    gholi| 0.76| 0.76| 0.76|\n",
      "|    2|    gholi| 1.62| 1.62| 1.62|\n",
      "|    2|    gholi| 5.81| 5.81| 5.81|\n",
      "|    2| bcbcbcbc| 0.58| 0.58| 0.58|\n",
      "|    2| bcbcbcbc|-8.24|-8.24|-8.24|\n",
      "|    2| bcbcbcbc|-0.45|-0.45|-0.45|\n",
      "|    2| bcbcbcbc| 1.03| 1.03| 1.03|\n",
      "|    2|aUniqueID|-0.05|-0.05|-0.05|\n",
      "|    2| qqqqqqqq|-0.44|-0.44|-0.44|\n",
      "+-----+---------+-----+-----+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pixiedust\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/bc/a8/e84b2ed12ee387589c099734b6f914a520e1fef2733c955982623080e813/pixiedust-1.1.17.tar.gz (197kB)\n",
      "\u001b[K    100% |################################| 204kB 4.4MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting mpld3 (from pixiedust)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/91/95/a52d3a83d0a29ba0d6898f6727e9858fe7a43f6c2ce81a5fe7e05f0f4912/mpld3-0.3.tar.gz (788kB)\n",
      "\u001b[K    100% |################################| 798kB 4.5MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting lxml (from pixiedust)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ec/be/5ab8abdd8663c0386ec2dd595a5bc0e23330a0549b8a91e32f38c20845b6/lxml-4.4.1-cp36-cp36m-manylinux1_x86_64.whl (5.8MB)\n",
      "\u001b[K    100% |################################| 5.8MB 2.3MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting geojson (from pixiedust)\n",
      "  Downloading https://files.pythonhosted.org/packages/e4/8d/9e28e9af95739e6d2d2f8d4bef0b3432da40b7c3588fbad4298c1be09e48/geojson-2.5.0-py2.py3-none-any.whl\n",
      "Collecting astunparse (from pixiedust)\n",
      "  Downloading https://files.pythonhosted.org/packages/2e/37/5dd0dd89b87bb5f0f32a7e775458412c52d78f230ab8d0c65df6aabc4479/astunparse-1.6.2-py2.py3-none-any.whl\n",
      "Collecting markdown (from pixiedust)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/c0/4e/fd492e91abdc2d2fcb70ef453064d980688762079397f779758e055f6575/Markdown-3.1.1-py2.py3-none-any.whl (87kB)\n",
      "\u001b[K    100% |################################| 92kB 4.7MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting colour (from pixiedust)\n",
      "  Downloading https://files.pythonhosted.org/packages/74/46/e81907704ab203206769dee1385dc77e1407576ff8f50a0681d0a6b541be/colour-0.1.5-py2.py3-none-any.whl\n",
      "Collecting requests (from pixiedust)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)\n",
      "\u001b[K    100% |################################| 61kB 3.4MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting wheel<1.0,>=0.23.0 (from astunparse->pixiedust)\n",
      "  Downloading https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl\n",
      "Collecting six<2.0,>=1.6.1 (from astunparse->pixiedust)\n",
      "  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl\n",
      "Collecting setuptools>=36 (from markdown->pixiedust)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b2/86/095d2f7829badc207c893dd4ac767e871f6cd547145df797ea26baea4e2e/setuptools-41.2.0-py2.py3-none-any.whl (576kB)\n",
      "\u001b[K    100% |################################| 583kB 4.9MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting certifi>=2017.4.17 (from requests->pixiedust)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl (157kB)\n",
      "\u001b[K    100% |################################| 163kB 5.9MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting chardet<3.1.0,>=3.0.2 (from requests->pixiedust)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)\n",
      "\u001b[K    100% |################################| 143kB 6.1MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->pixiedust)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)\n",
      "\u001b[K    100% |################################| 153kB 5.7MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting idna<2.9,>=2.5 (from requests->pixiedust)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)\n",
      "\u001b[K    100% |################################| 61kB 3.6MB/s eta 0:00:01\n",
      "\u001b[?25hBuilding wheels for collected packages: pixiedust, mpld3\n",
      "  Building wheel for pixiedust (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /home/spark/shared/.cache/pip/wheels/25/fa/a5/09c1e8f4c91b34c5f7f4ac6e41be81dd0667030a2372546a8d\n",
      "  Building wheel for mpld3 (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /home/spark/shared/.cache/pip/wheels/c0/47/fb/8a64f89aecfe0059830479308ad42d62e898a3e3cefdf6ba28\n",
      "Successfully built pixiedust mpld3\n",
      "\u001b[31mtensorflow 1.13.1 requires tensorboard<1.14.0,>=1.13.0, which is not installed.\u001b[0m\n",
      "\u001b[31mspyder 3.3.3 requires pyqt5<=5.12; python_version >= \"3\", which is not installed.\u001b[0m\n",
      "\u001b[31mibm-cos-sdk-core 2.4.3 has requirement urllib3<1.25,>=1.20, but you'll have urllib3 1.25.3 which is incompatible.\u001b[0m\n",
      "\u001b[31mbotocore 1.12.82 has requirement urllib3<1.25,>=1.20, but you'll have urllib3 1.25.3 which is incompatible.\u001b[0m\n",
      "Installing collected packages: mpld3, lxml, geojson, wheel, six, astunparse, setuptools, markdown, colour, certifi, chardet, urllib3, idna, requests, pixiedust\n",
      "Successfully installed astunparse-1.6.2 certifi-2019.6.16 chardet-3.0.4 colour-0.1.5 geojson-2.5.0 idna-2.8 lxml-4.4.1 markdown-3.1.1 mpld3-0.3 pixiedust-1.1.17 requests-2.22.0 setuptools-41.2.0 six-1.12.0 urllib3-1.25.3 wheel-0.33.6\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3/__pycache__ already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3/six.py already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3/six-1.12.0.dist-info already exists. Specify --upgrade to force replacement.\u001b[0m\n",
      "\u001b[33mTarget directory /home/spark/shared/user-libs/python3/bin already exists. Specify --upgrade to force replacement.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install pixiedust"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pixiedust": {
     "displayParams": {
      "handlerId": "tableView"
     }
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">.pd_warning{display:none;}</style><div class=\"pd_warning\"><em>Hey, there's something awesome here! To see it, open this notebook outside GitHub, in a viewer like Jupyter</em></div>\n",
       "        <div class=\"pd_save is-viewer-good\" style=\"padding-right:10px;text-align: center;line-height:initial !important;font-size: xx-large;font-weight: 500;color: coral;\">\n",
       "            \n",
       "        </div>\n",
       "    <div id=\"chartFigurea62b5dcb\" class=\"pd_save is-viewer-good\" style=\"overflow-x:auto\">\n",
       "            <style type=\"text/css\" class=\"pd_save\">\n",
       "    .df-table-wrapper .panel-heading {\n",
       "      border-radius: 0;\n",
       "      padding: 0px;\n",
       "    }\n",
       "    .df-table-wrapper .panel-heading:hover {\n",
       "      border-color: #008571;\n",
       "    }\n",
       "    .df-table-wrapper .panel-title a {\n",
       "      background-color: #f9f9fb;\n",
       "      color: #333333;\n",
       "      display: block;\n",
       "      outline: none;\n",
       "      padding: 10px 15px;\n",
       "      text-decoration: none;\n",
       "    }\n",
       "    .df-table-wrapper .panel-title a:hover {\n",
       "      background-color: #337ab7;\n",
       "      border-color: #2e6da4;\n",
       "      color: #ffffff;\n",
       "      display: block;\n",
       "      padding: 10px 15px;\n",
       "      text-decoration: none;\n",
       "    }\n",
       "    .df-table-wrapper {\n",
       "      font-size: small;\n",
       "      font-weight: 300;\n",
       "      letter-spacing: 0.5px;\n",
       "      line-height: normal;\n",
       "      height: inherit;\n",
       "      overflow: auto;\n",
       "    }\n",
       "    .df-table-search {\n",
       "      margin: 0 0 20px 0;\n",
       "    }\n",
       "    .df-table-search-count {\n",
       "      display: inline-block;\n",
       "      margin: 0 0 20px 0;\n",
       "    }\n",
       "    .df-table-container {\n",
       "      max-height: 50vh;\n",
       "      max-width: 100%;\n",
       "      overflow-x: auto;\n",
       "      position: relative;\n",
       "    }\n",
       "    .df-table-wrapper table {\n",
       "      border: 0 none #ffffff;\n",
       "      border-collapse: collapse;\n",
       "      margin: 0;\n",
       "      min-width: 100%;\n",
       "      padding: 0;\n",
       "      table-layout: fixed;\n",
       "      height: inherit;\n",
       "      overflow: auto;\n",
       "    }\n",
       "    .df-table-wrapper tr.hidden {\n",
       "      display: none;\n",
       "    }\n",
       "    .df-table-wrapper tr:nth-child(even) {\n",
       "      background-color: #f9f9fb;\n",
       "    }\n",
       "    .df-table-wrapper tr.even {\n",
       "      background-color: #f9f9fb;\n",
       "    }\n",
       "    .df-table-wrapper tr.odd {\n",
       "      background-color: #ffffff;\n",
       "    }\n",
       "    .df-table-wrapper td + td {\n",
       "      border-left: 1px solid #e0e0e0;\n",
       "    }\n",
       "  \n",
       "    .df-table-wrapper thead,\n",
       "    .fixed-header {\n",
       "      font-weight: 600;\n",
       "    }\n",
       "    .df-table-wrapper tr,\n",
       "    .fixed-row {\n",
       "      border: 0 none #ffffff;\n",
       "      margin: 0;\n",
       "      padding: 0;\n",
       "    }\n",
       "    .df-table-wrapper th,\n",
       "    .df-table-wrapper td,\n",
       "    .fixed-cell {\n",
       "      border: 0 none #ffffff;\n",
       "      margin: 0;\n",
       "      min-width: 50px;\n",
       "      padding: 5px 20px 5px 10px;\n",
       "      text-align: left;\n",
       "      word-wrap: break-word;\n",
       "    }\n",
       "    .df-table-wrapper th {\n",
       "      padding-bottom: 0;\n",
       "      padding-top: 0;\n",
       "    }\n",
       "    .df-table-wrapper th div {\n",
       "      max-height: 1px;\n",
       "      visibility: hidden;\n",
       "    }\n",
       "  \n",
       "    .df-schema-field {\n",
       "      margin-left: 10px;\n",
       "    }\n",
       "  \n",
       "    .fixed-header-container {\n",
       "      overflow: hidden;\n",
       "      position: relative;\n",
       "    }\n",
       "    .fixed-header {\n",
       "      border-bottom: 2px solid #000;\n",
       "      display: table;\n",
       "      position: relative;\n",
       "    }\n",
       "    .fixed-row {\n",
       "      display: table-row;\n",
       "    }\n",
       "    .fixed-cell {\n",
       "      display: table-cell;\n",
       "    }\n",
       "  </style>\n",
       "  \n",
       "  \n",
       "  <div class=\"df-table-wrapper df-table-wrapper-a62b5dcb panel-group pd_save\">\n",
       "    <!-- dataframe schema -->\n",
       "    \n",
       "    <div class=\"panel panel-default\">\n",
       "      <div class=\"panel-heading\">\n",
       "        <h4 class=\"panel-title\" style=\"margin: 0px;\">\n",
       "          <a data-toggle=\"collapse\" href=\"#df-schema-a62b5dcb\" data-parent=\"#df-table-wrapper-a62b5dcb\">Schema</a>\n",
       "        </h4>\n",
       "      </div>\n",
       "      <div id=\"df-schema-a62b5dcb\" class=\"panel-collapse collapse\">\n",
       "        <div class=\"panel-body\" style=\"font-family: monospace;\">\n",
       "          <div class=\"df-schema-fields\">\n",
       "            <div>Field types:</div>\n",
       "            \n",
       "              <div class=\"df-schema-field\"><strong>CLASS: </strong> int64</div>\n",
       "            \n",
       "              <div class=\"df-schema-field\"><strong>SENSORID: </strong> object</div>\n",
       "            \n",
       "              <div class=\"df-schema-field\"><strong>X: </strong> float64</div>\n",
       "            \n",
       "              <div class=\"df-schema-field\"><strong>Y: </strong> float64</div>\n",
       "            \n",
       "              <div class=\"df-schema-field\"><strong>Z: </strong> float64</div>\n",
       "            \n",
       "          </div>\n",
       "        </div>\n",
       "      </div>\n",
       "    </div>\n",
       "    \n",
       "    <!-- dataframe table -->\n",
       "    <div class=\"panel panel-default\">\n",
       "      \n",
       "      <div class=\"panel-heading\">\n",
       "        <h4 class=\"panel-title\" style=\"margin: 0px;\">\n",
       "          <a data-toggle=\"collapse\" href=\"#df-table-a62b5dcb\" data-parent=\"#df-table-wrapper-a62b5dcb\"> Table</a>\n",
       "        </h4>\n",
       "      </div>\n",
       "      \n",
       "      <div id=\"df-table-a62b5dcb\" class=\"panel-collapse collapse in\">\n",
       "        <div class=\"panel-body\">\n",
       "          \n",
       "          <input type=\"text\" class=\"df-table-search form-control input-sm\" placeholder=\"Search table\">\n",
       "          \n",
       "          <div>\n",
       "            \n",
       "            <span class=\"df-table-search-count\">Showing 100 of 9883 rows</span>\n",
       "            \n",
       "          </div>\n",
       "          <!-- fixed header for when dataframe table scrolls -->\n",
       "          <div class=\"fixed-header-container\">\n",
       "            <div class=\"fixed-header\">\n",
       "              <div class=\"fixed-row\">\n",
       "                \n",
       "                <div class=\"fixed-cell\">CLASS</div>\n",
       "                \n",
       "                <div class=\"fixed-cell\">SENSORID</div>\n",
       "                \n",
       "                <div class=\"fixed-cell\">X</div>\n",
       "                \n",
       "                <div class=\"fixed-cell\">Y</div>\n",
       "                \n",
       "                <div class=\"fixed-cell\">Z</div>\n",
       "                \n",
       "              </div>\n",
       "            </div>\n",
       "          </div>\n",
       "          <div class=\"df-table-container\">\n",
       "            <table class=\"df-table\">\n",
       "              <thead>\n",
       "                <tr>\n",
       "                  \n",
       "                  <th><div>CLASS</div></th>\n",
       "                  \n",
       "                  <th><div>SENSORID</div></th>\n",
       "                  \n",
       "                  <th><div>X</div></th>\n",
       "                  \n",
       "                  <th><div>Y</div></th>\n",
       "                  \n",
       "                  <th><div>Z</div></th>\n",
       "                  \n",
       "                </tr>\n",
       "              </thead>\n",
       "              <tbody>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345678</td>\n",
       "                  \n",
       "                  <td>-0.78</td>\n",
       "                  \n",
       "                  <td>-0.78</td>\n",
       "                  \n",
       "                  <td>-0.78</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>pj123456</td>\n",
       "                  \n",
       "                  <td>-0.22</td>\n",
       "                  \n",
       "                  <td>-0.22</td>\n",
       "                  \n",
       "                  <td>-0.22</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Bbbbbbb</td>\n",
       "                  \n",
       "                  <td>-0.07</td>\n",
       "                  \n",
       "                  <td>-0.07</td>\n",
       "                  \n",
       "                  <td>-0.07</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>3.07</td>\n",
       "                  \n",
       "                  <td>3.07</td>\n",
       "                  \n",
       "                  <td>3.07</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>3.56</td>\n",
       "                  \n",
       "                  <td>3.56</td>\n",
       "                  \n",
       "                  <td>3.56</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>-0.23</td>\n",
       "                  \n",
       "                  <td>-0.23</td>\n",
       "                  \n",
       "                  <td>-0.23</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Qwertyui</td>\n",
       "                  \n",
       "                  <td>-0.24</td>\n",
       "                  \n",
       "                  <td>-0.24</td>\n",
       "                  \n",
       "                  <td>-0.24</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aUniqueID</td>\n",
       "                  \n",
       "                  <td>1.01</td>\n",
       "                  \n",
       "                  <td>1.01</td>\n",
       "                  \n",
       "                  <td>1.01</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>14.59</td>\n",
       "                  \n",
       "                  <td>14.59</td>\n",
       "                  \n",
       "                  <td>14.59</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>qqqqqqqq</td>\n",
       "                  \n",
       "                  <td>0.45</td>\n",
       "                  \n",
       "                  <td>0.45</td>\n",
       "                  \n",
       "                  <td>0.45</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>-14.74</td>\n",
       "                  \n",
       "                  <td>-14.74</td>\n",
       "                  \n",
       "                  <td>-14.74</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Qwertyui</td>\n",
       "                  \n",
       "                  <td>-0.39</td>\n",
       "                  \n",
       "                  <td>-0.39</td>\n",
       "                  \n",
       "                  <td>-0.39</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>-0.56</td>\n",
       "                  \n",
       "                  <td>-0.56</td>\n",
       "                  \n",
       "                  <td>-0.56</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aUniqueID</td>\n",
       "                  \n",
       "                  <td>-0.38</td>\n",
       "                  \n",
       "                  <td>-0.38</td>\n",
       "                  \n",
       "                  <td>-0.38</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345678</td>\n",
       "                  \n",
       "                  <td>-0.2</td>\n",
       "                  \n",
       "                  <td>-0.2</td>\n",
       "                  \n",
       "                  <td>-0.2</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345</td>\n",
       "                  \n",
       "                  <td>-1.64</td>\n",
       "                  \n",
       "                  <td>-1.64</td>\n",
       "                  \n",
       "                  <td>-1.64</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>-21.44</td>\n",
       "                  \n",
       "                  <td>-21.44</td>\n",
       "                  \n",
       "                  <td>-21.44</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Bbbbbbb</td>\n",
       "                  \n",
       "                  <td>-72.45</td>\n",
       "                  \n",
       "                  <td>-72.45</td>\n",
       "                  \n",
       "                  <td>-72.45</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345678</td>\n",
       "                  \n",
       "                  <td>2.48</td>\n",
       "                  \n",
       "                  <td>2.48</td>\n",
       "                  \n",
       "                  <td>2.48</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>qqqqqqqq</td>\n",
       "                  \n",
       "                  <td>-23.16</td>\n",
       "                  \n",
       "                  <td>-23.16</td>\n",
       "                  \n",
       "                  <td>-23.16</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>qqqqqqqq</td>\n",
       "                  \n",
       "                  <td>-7.53</td>\n",
       "                  \n",
       "                  <td>-7.53</td>\n",
       "                  \n",
       "                  <td>-7.53</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Bbbbbbb</td>\n",
       "                  \n",
       "                  <td>0.1</td>\n",
       "                  \n",
       "                  <td>0.1</td>\n",
       "                  \n",
       "                  <td>0.1</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>33.29</td>\n",
       "                  \n",
       "                  <td>33.29</td>\n",
       "                  \n",
       "                  <td>33.29</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Bbbbbbb</td>\n",
       "                  \n",
       "                  <td>-0.04</td>\n",
       "                  \n",
       "                  <td>-0.04</td>\n",
       "                  \n",
       "                  <td>-0.04</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Qwertyui</td>\n",
       "                  \n",
       "                  <td>0.1</td>\n",
       "                  \n",
       "                  <td>0.1</td>\n",
       "                  \n",
       "                  <td>0.1</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Qwertyui</td>\n",
       "                  \n",
       "                  <td>0.11</td>\n",
       "                  \n",
       "                  <td>0.11</td>\n",
       "                  \n",
       "                  <td>0.11</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Qwertyui</td>\n",
       "                  \n",
       "                  <td>-0.08</td>\n",
       "                  \n",
       "                  <td>-0.08</td>\n",
       "                  \n",
       "                  <td>-0.08</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>qqqqqqqq</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345678</td>\n",
       "                  \n",
       "                  <td>-0.07</td>\n",
       "                  \n",
       "                  <td>-0.07</td>\n",
       "                  \n",
       "                  <td>-0.07</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aUniqueID</td>\n",
       "                  \n",
       "                  <td>-0.14</td>\n",
       "                  \n",
       "                  <td>-0.14</td>\n",
       "                  \n",
       "                  <td>-0.14</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345678</td>\n",
       "                  \n",
       "                  <td>-0.08</td>\n",
       "                  \n",
       "                  <td>-0.08</td>\n",
       "                  \n",
       "                  <td>-0.08</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>-0.97</td>\n",
       "                  \n",
       "                  <td>-0.97</td>\n",
       "                  \n",
       "                  <td>-0.97</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>qqqqqqqq</td>\n",
       "                  \n",
       "                  <td>-0.14</td>\n",
       "                  \n",
       "                  <td>-0.14</td>\n",
       "                  \n",
       "                  <td>-0.14</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>-1.03</td>\n",
       "                  \n",
       "                  <td>-1.03</td>\n",
       "                  \n",
       "                  <td>-1.03</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>4.43</td>\n",
       "                  \n",
       "                  <td>4.43</td>\n",
       "                  \n",
       "                  <td>4.43</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345</td>\n",
       "                  \n",
       "                  <td>2.48</td>\n",
       "                  \n",
       "                  <td>2.48</td>\n",
       "                  \n",
       "                  <td>2.48</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345678</td>\n",
       "                  \n",
       "                  <td>-0.24</td>\n",
       "                  \n",
       "                  <td>-0.24</td>\n",
       "                  \n",
       "                  <td>-0.24</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>-6.11</td>\n",
       "                  \n",
       "                  <td>-6.11</td>\n",
       "                  \n",
       "                  <td>-6.11</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>29.8</td>\n",
       "                  \n",
       "                  <td>29.8</td>\n",
       "                  \n",
       "                  <td>29.8</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>qqqqqqqq</td>\n",
       "                  \n",
       "                  <td>11.11</td>\n",
       "                  \n",
       "                  <td>11.11</td>\n",
       "                  \n",
       "                  <td>11.11</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Bbbbbbb</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aUniqueID</td>\n",
       "                  \n",
       "                  <td>-6.99</td>\n",
       "                  \n",
       "                  <td>-6.99</td>\n",
       "                  \n",
       "                  <td>-6.99</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Bbbbbbb</td>\n",
       "                  \n",
       "                  <td>-0.42</td>\n",
       "                  \n",
       "                  <td>-0.42</td>\n",
       "                  \n",
       "                  <td>-0.42</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>-13.09</td>\n",
       "                  \n",
       "                  <td>-13.09</td>\n",
       "                  \n",
       "                  <td>-13.09</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>-10.8</td>\n",
       "                  \n",
       "                  <td>-10.8</td>\n",
       "                  \n",
       "                  <td>-10.8</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>-1.8</td>\n",
       "                  \n",
       "                  <td>-1.8</td>\n",
       "                  \n",
       "                  <td>-1.8</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>-2.19</td>\n",
       "                  \n",
       "                  <td>-2.19</td>\n",
       "                  \n",
       "                  <td>-2.19</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>-9.39</td>\n",
       "                  \n",
       "                  <td>-9.39</td>\n",
       "                  \n",
       "                  <td>-9.39</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345678</td>\n",
       "                  \n",
       "                  <td>19.11</td>\n",
       "                  \n",
       "                  <td>19.11</td>\n",
       "                  \n",
       "                  <td>19.11</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>-0.05</td>\n",
       "                  \n",
       "                  <td>-0.05</td>\n",
       "                  \n",
       "                  <td>-0.05</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>1.29</td>\n",
       "                  \n",
       "                  <td>1.29</td>\n",
       "                  \n",
       "                  <td>1.29</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Bbbbbbb</td>\n",
       "                  \n",
       "                  <td>0.76</td>\n",
       "                  \n",
       "                  <td>0.76</td>\n",
       "                  \n",
       "                  <td>0.76</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>6.37</td>\n",
       "                  \n",
       "                  <td>6.37</td>\n",
       "                  \n",
       "                  <td>6.37</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>pj123456</td>\n",
       "                  \n",
       "                  <td>-0.03</td>\n",
       "                  \n",
       "                  <td>-0.03</td>\n",
       "                  \n",
       "                  <td>-0.03</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>pj123456</td>\n",
       "                  \n",
       "                  <td>-0.09</td>\n",
       "                  \n",
       "                  <td>-0.09</td>\n",
       "                  \n",
       "                  <td>-0.09</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Qwertyui</td>\n",
       "                  \n",
       "                  <td>0.02</td>\n",
       "                  \n",
       "                  <td>0.02</td>\n",
       "                  \n",
       "                  <td>0.02</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Qwertyui</td>\n",
       "                  \n",
       "                  <td>-0.01</td>\n",
       "                  \n",
       "                  <td>-0.01</td>\n",
       "                  \n",
       "                  <td>-0.01</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345</td>\n",
       "                  \n",
       "                  <td>-0.19</td>\n",
       "                  \n",
       "                  <td>-0.19</td>\n",
       "                  \n",
       "                  <td>-0.19</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Bbbbbbb</td>\n",
       "                  \n",
       "                  <td>0.01</td>\n",
       "                  \n",
       "                  <td>0.01</td>\n",
       "                  \n",
       "                  <td>0.01</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Bbbbbbb</td>\n",
       "                  \n",
       "                  <td>-0.41</td>\n",
       "                  \n",
       "                  <td>-0.41</td>\n",
       "                  \n",
       "                  <td>-0.41</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>30.65</td>\n",
       "                  \n",
       "                  <td>30.65</td>\n",
       "                  \n",
       "                  <td>30.65</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>qqqqqqqq</td>\n",
       "                  \n",
       "                  <td>18.51</td>\n",
       "                  \n",
       "                  <td>18.51</td>\n",
       "                  \n",
       "                  <td>18.51</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>pj123456</td>\n",
       "                  \n",
       "                  <td>-0.07</td>\n",
       "                  \n",
       "                  <td>-0.07</td>\n",
       "                  \n",
       "                  <td>-0.07</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345678</td>\n",
       "                  \n",
       "                  <td>3.0</td>\n",
       "                  \n",
       "                  <td>3.0</td>\n",
       "                  \n",
       "                  <td>3.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345</td>\n",
       "                  \n",
       "                  <td>0.9</td>\n",
       "                  \n",
       "                  <td>0.9</td>\n",
       "                  \n",
       "                  <td>0.9</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>-1.87</td>\n",
       "                  \n",
       "                  <td>-1.87</td>\n",
       "                  \n",
       "                  <td>-1.87</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>-2.03</td>\n",
       "                  \n",
       "                  <td>-2.03</td>\n",
       "                  \n",
       "                  <td>-2.03</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>qqqqqqqq</td>\n",
       "                  \n",
       "                  <td>-0.04</td>\n",
       "                  \n",
       "                  <td>-0.04</td>\n",
       "                  \n",
       "                  <td>-0.04</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>0.22</td>\n",
       "                  \n",
       "                  <td>0.22</td>\n",
       "                  \n",
       "                  <td>0.22</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345</td>\n",
       "                  \n",
       "                  <td>2.14</td>\n",
       "                  \n",
       "                  <td>2.14</td>\n",
       "                  \n",
       "                  <td>2.14</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aUniqueID</td>\n",
       "                  \n",
       "                  <td>-6.55</td>\n",
       "                  \n",
       "                  <td>-6.55</td>\n",
       "                  \n",
       "                  <td>-6.55</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>2.27</td>\n",
       "                  \n",
       "                  <td>2.27</td>\n",
       "                  \n",
       "                  <td>2.27</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>-9.21</td>\n",
       "                  \n",
       "                  <td>-9.21</td>\n",
       "                  \n",
       "                  <td>-9.21</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>bcbcbcbc</td>\n",
       "                  \n",
       "                  <td>-9.09</td>\n",
       "                  \n",
       "                  <td>-9.09</td>\n",
       "                  \n",
       "                  <td>-9.09</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345678</td>\n",
       "                  \n",
       "                  <td>1.67</td>\n",
       "                  \n",
       "                  <td>1.67</td>\n",
       "                  \n",
       "                  <td>1.67</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345678</td>\n",
       "                  \n",
       "                  <td>-10.5</td>\n",
       "                  \n",
       "                  <td>-10.5</td>\n",
       "                  \n",
       "                  <td>-10.5</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345678</td>\n",
       "                  \n",
       "                  <td>6.84</td>\n",
       "                  \n",
       "                  <td>6.84</td>\n",
       "                  \n",
       "                  <td>6.84</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aaaaaaaa</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                  <td>0.0</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345678</td>\n",
       "                  \n",
       "                  <td>-0.12</td>\n",
       "                  \n",
       "                  <td>-0.12</td>\n",
       "                  \n",
       "                  <td>-0.12</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>gholi</td>\n",
       "                  \n",
       "                  <td>1.1</td>\n",
       "                  \n",
       "                  <td>1.1</td>\n",
       "                  \n",
       "                  <td>1.1</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Qwertyui</td>\n",
       "                  \n",
       "                  <td>-0.04</td>\n",
       "                  \n",
       "                  <td>-0.04</td>\n",
       "                  \n",
       "                  <td>-0.04</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>12345</td>\n",
       "                  \n",
       "                  <td>0.14</td>\n",
       "                  \n",
       "                  <td>0.14</td>\n",
       "                  \n",
       "                  <td>0.14</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Qwertyui</td>\n",
       "                  \n",
       "                  <td>0.11</td>\n",
       "                  \n",
       "                  <td>0.11</td>\n",
       "                  \n",
       "                  <td>0.11</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>aUniqueID</td>\n",
       "                  \n",
       "                  <td>-0.05</td>\n",
       "                  \n",
       "                  <td>-0.05</td>\n",
       "                  \n",
       "                  <td>-0.05</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "                <tr>\n",
       "                  \n",
       "                  <td>2</td>\n",
       "                  \n",
       "                  <td>Bbbbbbb</td>\n",
       "                  \n",
       "                  <td>22.03</td>\n",
       "                  \n",
       "                  <td>22.03</td>\n",
       "                  \n",
       "                  <td>22.03</td>\n",
       "                  \n",
       "                </tr>\n",
       "                \n",
       "              </tbody>\n",
       "            </table>\n",
       "          </div>\n",
       "        </div>\n",
       "      </div>\n",
       "    </div>\n",
       "  </div>\n",
       "  \n",
       "  <script class=\"pd_save\">\n",
       "    $(function() {\n",
       "      var tableWrapper = $('.df-table-wrapper-a62b5dcb');\n",
       "      var fixedHeader = $('.fixed-header', tableWrapper);\n",
       "      var tableContainer = $('.df-table-container', tableWrapper);\n",
       "      var table = $('.df-table', tableContainer);\n",
       "      var rows = $('tbody > tr', table);\n",
       "      var total = 100;\n",
       "  \n",
       "      fixedHeader\n",
       "        .css('width', table.width())\n",
       "        .find('.fixed-cell')\n",
       "        .each(function(i, e) {\n",
       "          $(this).css('width', $('.df-table-wrapper-a62b5dcb th:nth-child(' + (i+1) + ')').css('width'));\n",
       "        });\n",
       "  \n",
       "      tableContainer.scroll(function() {\n",
       "        fixedHeader.css({ left: table.position().left });\n",
       "      });\n",
       "  \n",
       "      rows.on(\"click\", function(e){\n",
       "          var txt = e.delegateTarget.innerText;\n",
       "          var splits = txt.split(\"\\t\");\n",
       "          var len = splits.length;\n",
       "          var hdrs = $(fixedHeader).find(\".fixed-cell\");\n",
       "          // Add all cells in the selected row as a map to be consumed by the target as needed\n",
       "          var payload = {type:\"select\", targetDivId: \"\" };\n",
       "          for (var i = 0; i < len; i++) {\n",
       "            payload[hdrs[i].innerHTML] = splits[i];\n",
       "          }\n",
       "  \n",
       "          //simple selection highlighting, client adds \"selected\" class\n",
       "          $(this).addClass(\"selected\").siblings().removeClass(\"selected\");\n",
       "          $(document).trigger('pd_event', payload);\n",
       "      });\n",
       "  \n",
       "      $('.df-table-search', tableWrapper).keyup(function() {\n",
       "        var val = '^(?=.*\\\\b' + $.trim($(this).val()).split(/\\s+/).join('\\\\b)(?=.*\\\\b') + ').*$';\n",
       "        var reg = RegExp(val, 'i');\n",
       "        var index = 0;\n",
       "        \n",
       "        rows.each(function(i, e) {\n",
       "          if (!reg.test($(this).text().replace(/\\s+/g, ' '))) {\n",
       "            $(this).attr('class', 'hidden');\n",
       "          }\n",
       "          else {\n",
       "            $(this).attr('class', (++index % 2 == 0 ? 'even' : 'odd'));\n",
       "          }\n",
       "        });\n",
       "        $('.df-table-search-count', tableWrapper).html('Showing ' + index + ' of ' + total + ' rows');\n",
       "      });\n",
       "    });\n",
       "  \n",
       "    $(\".df-table-wrapper td:contains('http://')\").each(function(){var tc = this.textContent; $(this).wrapInner(\"<a target='_blank' href='\" + tc + \"'></a>\");});\n",
       "    $(\".df-table-wrapper td:contains('https://')\").each(function(){var tc = this.textContent; $(this).wrapInner(\"<a target='_blank' href='\" + tc + \"'></a>\");});\n",
       "  </script>\n",
       "  \n",
       "        </div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pixiedust\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.createOrReplaceTempView(\"df\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ll use Apache SystemML to implement Discrete Fourier Transformation. This way all computation continues to happen on the Apache Spark cluster for advanced scalability and performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from systemml import MLContext, dml\n",
    "ml = MLContext(spark)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you’ve learned from the lecture, implementing Discrete Fourier Transformation in a linear algebra programming language is simple. Apache SystemML DML is such a language and as you can see the implementation is straightforward and doesn’t differ too much from the mathematical definition (Just note that the sum operator has been swapped with a vector dot product using the %*% syntax borrowed from R\n",
    "):\n",
    "\n",
    "<img style=\"float: left;\" src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/1af0a78dc50bbf118ab6bd4c4dcc3c4ff8502223\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "dml_script = '''\n",
    "PI = 3.141592654\n",
    "N = nrow(signal)\n",
    "\n",
    "n = seq(0, N-1, 1)\n",
    "k = seq(0, N-1, 1)\n",
    "\n",
    "M = (n %*% t(k))*(2*PI/N)\n",
    "\n",
    "Xa = cos(M) %*% signal\n",
    "Xb = sin(M) %*% signal\n",
    "\n",
    "DFT = cbind(Xa, Xb)\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it’s time to create a function which takes a single row Apache Spark data frame as argument (the one containing the accelerometer measurement time series for one axis) and returns the Fourier transformation of it. In addition, we are adding an index column for later joining all axis together and renaming the columns to appropriate names. The result of this function is an Apache Spark DataFrame containing the Fourier Transformation of its input in two columns. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import monotonically_increasing_id\n",
    "\n",
    "def dft_systemml(signal,name):\n",
    "    prog = dml(dml_script).input('signal', signal).output('DFT')\n",
    "    \n",
    "    return (\n",
    "\n",
    "    #execute the script inside the SystemML engine running on top of Apache Spark\n",
    "    ml.execute(prog) \n",
    "     \n",
    "         #read result from SystemML execution back as SystemML Matrix\n",
    "        .get('DFT') \n",
    "     \n",
    "         #convert SystemML Matrix to ApacheSpark DataFrame \n",
    "        .toDF() \n",
    "     \n",
    "         #rename default column names\n",
    "        .selectExpr('C1 as %sa' % (name), 'C2 as %sb' % (name)) \n",
    "     \n",
    "         #add unique ID per row for later joining\n",
    "        .withColumn(\"id\", monotonically_increasing_id())\n",
    "    )\n",
    "        \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it’s time to create DataFrames containing for each accelerometer sensor axis and one for each class. This means you’ll get 6 DataFrames. Please implement this using the relational API of DataFrames or SparkSQL. Please use class 1 and 2 and not 0 and 1.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "x0 = spark.sql(\"SELECT X from df where class = 0\")\n",
    "y0 = spark.sql(\"SELECT Y from df where class = 0\")\n",
    "z0 = spark.sql(\"SELECT Z from df where class = 0\")\n",
    "x1 = spark.sql(\"SELECT X from df where class = 1\")\n",
    "y1 = spark.sql(\"SELECT Y from df where class = 1\")\n",
    "z1 = spark.sql(\"SELECT Z from df where class = 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we’ve created this cool DFT function before, we can just call it for each of the 6 DataFrames now. And since the result of this function call is a DataFrame again we can use the pyspark best practice in simply calling methods on it sequentially. So what we are doing is the following:\n",
    "\n",
    "- Calling DFT for each class and accelerometer sensor axis.\n",
    "- Joining them together on the ID column. \n",
    "- Re-adding a column containing the class index.\n",
    "- Stacking both Dataframes for each classes together\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SystemML Statistics:\n",
      "Total execution time:\t\t0.013 sec.\n",
      "Number of executed Spark inst:\t0.\n",
      "\n",
      "\n",
      "SystemML Statistics:\n",
      "Total execution time:\t\t0.000 sec.\n",
      "Number of executed Spark inst:\t0.\n",
      "\n",
      "\n",
      "SystemML Statistics:\n",
      "Total execution time:\t\t0.000 sec.\n",
      "Number of executed Spark inst:\t0.\n",
      "\n",
      "\n",
      "SystemML Statistics:\n",
      "Total execution time:\t\t0.458 sec.\n",
      "Number of executed Spark inst:\t0.\n",
      "\n",
      "\n",
      "SystemML Statistics:\n",
      "Total execution time:\t\t0.459 sec.\n",
      "Number of executed Spark inst:\t0.\n",
      "\n",
      "\n",
      "SystemML Statistics:\n",
      "Total execution time:\t\t0.396 sec.\n",
      "Number of executed Spark inst:\t0.\n",
      "\n",
      "\n",
      "+----------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+-----+\n",
      "|        id|                  xa|                  xb|                  ya|                  yb|                  za|                  zb|class|\n",
      "+----------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+-----+\n",
      "|8589934592|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|         0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|8589934596|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|8589934598|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|8589934593|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|         6|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|         5|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|         1|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|         3|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|8589934594|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|8589934595|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|         2|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|         4|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|8589934597|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|\n",
      "|        26|0.007432298669951747|-0.00394663166701...|0.007432298669951747|-0.00394663166701...|0.007432298669951747|-0.00394663166701...|    1|\n",
      "|        29| 0.02589077158423112|-0.02414578651495463| 0.02589077158423112|-0.02414578651495463| 0.02589077158423112|-0.02414578651495463|    1|\n",
      "|8589934592| 0.02589077121123453|0.024145789283483897| 0.02589077121123453|0.024145789283483897| 0.02589077121123453|0.024145789283483897|    1|\n",
      "|        19|-0.08486126908675795| -0.0218913608396156|-0.08486126908675795| -0.0218913608396156|-0.08486126908675795| -0.0218913608396156|    1|\n",
      "|        54| 0.02941472472264519| 0.08753524593777502| 0.02941472472264519| 0.08753524593777502| 0.02941472472264519| 0.08753524593777502|    1|\n",
      "|         0|-0.00305531695825...|0.013615410059352576|-0.00305531695825...|0.013615410059352576|-0.00305531695825...|0.013615410059352576|    1|\n",
      "+----------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.functions import lit\n",
    "\n",
    "df_class_0 = dft_systemml(x0,'x') \\\n",
    "    .join(dft_systemml(y0,'y'), on=['id'], how='inner') \\\n",
    "    .join(dft_systemml(z0,'z'), on=['id'], how='inner') \\\n",
    "    .withColumn('class', lit(0))\n",
    "    \n",
    "df_class_1 = dft_systemml(x1,'x') \\\n",
    "    .join(dft_systemml(y1,'y'), on=['id'], how='inner') \\\n",
    "    .join(dft_systemml(z1,'z'), on=['id'], how='inner') \\\n",
    "    .withColumn('class', lit(1))\n",
    "\n",
    "df_dft = df_class_0.union(df_class_1)\n",
    "\n",
    "df_dft.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please create a VectorAssembler which consumes the newly created DFT columns and produces a column “features”\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import VectorAssembler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorAssembler = VectorAssembler(\n",
    "    inputCols=[\"xa\", \"xb\", \"ya\", \"yb\", \"za\", \"zb\"],\n",
    "    outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please insatiate a classifier from the SparkML package and assign it to the classifier variable. Make sure to set the “class” column as target.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.classification import GBTClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier = GBTClassifier(labelCol=\"class\", featuresCol=\"features\", maxDepth=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s train and evaluate…\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml import Pipeline\n",
    "pipeline = Pipeline(stages=[vectorAssembler, classifier])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = pipeline.fit(df_dft)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = model.transform(df_dft)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+-----+--------------------+--------------------+--------------------+----------+\n",
      "|        id|                  xa|                  xb|                  ya|                  yb|                  za|                  zb|class|            features|       rawPrediction|         probability|prediction|\n",
      "+----------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+-----+--------------------+--------------------+--------------------+----------+\n",
      "|8589934592|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|         0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|8589934596|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|8589934598|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|8589934593|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|         6|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|         5|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|         1|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|         3|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|8589934594|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|8589934595|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|         2|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|         4|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|8589934597|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|                 0.0|    0|           (6,[],[])|[1.54350200272498...|[0.95635347857270...|       0.0|\n",
      "|        26|0.007432298669951747|-0.00394663166701...|0.007432298669951747|-0.00394663166701...|0.007432298669951747|-0.00394663166701...|    1|[0.00743229866995...|[-1.5435020027249...|[0.04364652142729...|       1.0|\n",
      "|        29| 0.02589077158423112|-0.02414578651495463| 0.02589077158423112|-0.02414578651495463| 0.02589077158423112|-0.02414578651495463|    1|[0.02589077158423...|[-1.5435020027249...|[0.04364652142729...|       1.0|\n",
      "|8589934592| 0.02589077121123453|0.024145789283483897| 0.02589077121123453|0.024145789283483897| 0.02589077121123453|0.024145789283483897|    1|[0.02589077121123...|[-1.5435020027249...|[0.04364652142729...|       1.0|\n",
      "|        19|-0.08486126908675795| -0.0218913608396156|-0.08486126908675795| -0.0218913608396156|-0.08486126908675795| -0.0218913608396156|    1|[-0.0848612690867...|[-1.5435020027249...|[0.04364652142729...|       1.0|\n",
      "|        54| 0.02941472472264519| 0.08753524593777502| 0.02941472472264519| 0.08753524593777502| 0.02941472472264519| 0.08753524593777502|    1|[0.02941472472264...|[-1.5435020027249...|[0.04364652142729...|       1.0|\n",
      "|         0|-0.00305531695825...|0.013615410059352576|-0.00305531695825...|0.013615410059352576|-0.00305531695825...|0.013615410059352576|    1|[-0.0030553169582...|[-1.5435020027249...|[0.04364652142729...|       1.0|\n",
      "+----------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+-----+--------------------+--------------------+--------------------+----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "prediction.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyspark.ml.evaluation import MulticlassClassificationEvaluator\n",
    "binEval = MulticlassClassificationEvaluator().setMetricName(\"accuracy\") .setPredictionCol(\"prediction\").setLabelCol(\"class\")\n",
    "    \n",
    "binEval.evaluate(prediction) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are happy with the result (I’m happy with > 0.8) please submit your solution to the grader by executing the following cells, please don’t forget to obtain an assignment submission token (secret) from the Courera’s graders web page and paste it to the “secret” variable below, including your email address you’ve used for Coursera. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -Rf a2_m4.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = prediction.repartition(1)\n",
    "prediction.write.json('a2_m4.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-09-02 01:11:45--  http://wget/\n",
      "Resolving wget (wget)... failed: Name or service not known.\n",
      "wget: unable to resolve host address 'wget'\n",
      "--2019-09-02 01:11:45--  https://raw.githubusercontent.com/IBM/coursera/master/rklib.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2540 (2.5K) [text/plain]\n",
      "Saving to: 'rklib.py'\n",
      "\n",
      "rklib.py            100%[===================>]   2.48K  --.-KB/s    in 0s      \n",
      "\n",
      "2019-09-02 01:11:45 (56.1 MB/s) - 'rklib.py' saved [2540/2540]\n",
      "\n",
      "FINISHED --2019-09-02 01:11:45--\n",
      "Total wall clock time: 0.2s\n",
      "Downloaded: 1 files, 2.5K in 0s (56.1 MB/s)\n"
     ]
    }
   ],
   "source": [
    "!rm -f rklib.py\n",
    "!wget wget https://raw.githubusercontent.com/IBM/coursera/master/rklib.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rklib import zipit\n",
    "zipit('a2_m4.json.zip','a2_m4.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "!base64 a2_m4.json.zip > a2_m4.json.zip.base64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission successful, please check on the coursera grader page for the status\n",
      "-------------------------\n",
      "{\"elements\":[{\"itemId\":\"B8wXV\",\"id\":\"f_F-qCtuEei_fRLwaVDk3g~B8wXV~1M4_Dc0eEemH3Aqun-E-ng\",\"courseId\":\"f_F-qCtuEei_fRLwaVDk3g\"}],\"paging\":{},\"linked\":{}}\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "from rklib import submit\n",
    "key = \"-fBiYHYDEeiR4QqiFhAvkA\"\n",
    "part = \"IjtJk\"\n",
    "email = \"dennislamcv@gmail.com\"\n",
    "submission_token = \"iUtw6qagq8plYqp4\"\n",
    "\n",
    "with open('a2_m4.json.zip.base64', 'r') as myfile:\n",
    "    data=myfile.read()\n",
    "submit(email, submission_token, key, part, [part], data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 with Spark",
   "language": "python3",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}