DS105-Intermediate-Statistics/DS105-L2-Normality-and-Transformations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "43e37a98",
   "metadata": {},
   "source": [
    "# DS105 Intermediate Statistics : Lesson Two Normality and Transformations\n",
    "## Data Transformations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df79948e",
   "metadata": {},
   "source": [
    "### Table of Contents <a class=\"anchor\" id=\"DS105L2_toc\"></a>\n",
    "\n",
    "* [Table of Contents](#DS105L2_toc)\n",
    "    * [Page 1 - Introduction](#DS105L2_page_1)\n",
    "    * [Page 2 - Normality](#DS105L2_page_2)\n",
    "    * [Page 3 - Data Transformations](#DS105L2_page_3)\n",
    "    * [Page 4 - Transformations in R](#DS105L2_page_4)\n",
    "    * [Page 5 - Transformations in R Activity](#DS105L2_page_5)\n",
    "    * [Page 6 - Transformations in R Activity Solution](#DS105L2_page_6)\n",
    "    * [Page 7 - Transformations in Python](#DS105L2_page_7)\n",
    "    * [Page 8 - Transformations in Python Activity](#DS105L2_page_8)\n",
    "    * [Page 9 - Transformations in Python Activity Solution](#DS105L2_page_9)\n",
    "    * [Page 10 - Key Terms](#DS105L2_page_10)\n",
    "    * [Page 11 - Lesson 2 Hands-On](#DS105L2_page_11)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f67e388d",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 1 - Introduction<a class=\"anchor\" id=\"DS105L2_page_1\"></a>\n",
    "\n",
    "[Back to Top](#DS105L2_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "162a8c3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/388868344\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7ff388251460>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Basic Statistics in Python\n",
    "VimeoVideo('388868344', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22e37d16",
   "metadata": {},
   "source": [
    "The transcript for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO105L02overview.zip)**.\n",
    "\n",
    "# Introduction\n",
    "\n",
    "Up to this point, you have learned that normality is an assumption for many statistical tests, and it will be for many more you still have to learn.  However, you've never learned what to do when the data isn't normal - you've just soldiered on.  You'll now learn how to transform your data so that it becomes normal.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f565fd16",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 2 - Normality<a class=\"anchor\" id=\"DS105L2_page_2\"></a>\n",
    "\n",
    "[Back to Top](#DS105L2_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "17f91e9e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/335019896\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x1053dd2b0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Skew and Kurtosis\n",
    "VimeoVideo('335019896', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e3b4e3a",
   "metadata": {},
   "source": [
    "```c-lms\n",
    "topic: Normality\n",
    "video-id: Skew and Kurtosis\n",
    "video-url-mp4: https://player.vimeo.com/external/335019896.hd.mp4?s=53826cc2933dfccf074c99fa43fade47a7969aff&profile_id=175\n",
    "video-url-mp4-1080: https://player.vimeo.com/external/335019896.hd.mp4?s=53826cc2933dfccf074c99fa43fade47a7969aff&profile_id=175\n",
    "video-url-mp4-720: https://player.vimeo.com/external/335019896.hd.mp4?s=53826cc2933dfccf074c99fa43fade47a7969aff&profile_id=174\n",
    "video-url-mp4-540: https://player.vimeo.com/external/335019896.sd.mp4?s=ecba9491cd7d721398956d63208914a8eb73e990&profile_id=165\n",
    "video-url-mp4-360: https://player.vimeo.com/external/335019896.sd.mp4?s=ecba9491cd7d721398956d63208914a8eb73e990&profile_id=164\n",
    "```\n",
    "\n",
    "The transcript for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO105L02pg2tutorial.zip)**.\n",
    "\n",
    "# Normal Distribution Review\n",
    "\n",
    "As a reminder, the normal distribution is also called the bell curve because it takes on a lovely bell shape.  For instance, look at this one down below: \n",
    "\n",
    "![A curve is plotted on a graph. Three vertical lines are drawn from the curve to meet the x-axis at three points labeled, mu minus sigma, mu, mu plus sigma.](Media/skew1.png)\n",
    "\n",
    "But data comes in many more shapes than that, even if the normal distribution does come up more than you'd think. \n",
    "\n",
    "---\n",
    "# Skew\n",
    "\n",
    "When the bell curve is shifted to the left or right, then this is called *skew*.  A *negatively skewed distribution* has a tail of data on the left, and the bulk of the data on the right.  A *positively skewed distribution* has the tail of data on the right, and the bulk of the data on the left.  \n",
    "\n",
    "![Two graphs labeled negative skew and positive skew. A dashed line meets the x-axis from the peak point.](Media/skew2.png)\n",
    "\n",
    "The data being shifted from the center means that the average will no longer be centered either. In a normal distribution, the mean, median, and mode are pretty much the same.  But if it's negatively skewed, the mean becomes less than the mode and the median, and if it's positively skewed, the mean becomes greater than the mode and the median. \n",
    "\n",
    "![A graph depicts the plot of random variable against the value of function. Three curves are plotted and they are labeled left skew mean is less than mode, normal mean equals mode, and right skew mean exceeds mode.](Media/skew4.png)\n",
    "\n",
    "---\n",
    "# Kurtosis\n",
    "\n",
    "When the bell curve is much taller or shorter than the normal distribution, then it is said to have *kurtosis*.  A *positively kurtotic distribution* is really tall.  The technical term for this is *leptokurtic*. A *negatively kurtotic distribution* is really flat or may even actually dip downwards in the opposite direction. Another word for negatively kurtotic is *platykurtic*.   \n",
    "\n",
    "![A deep inverted curve labeled positive kurtosis leptokurtic and an almost-flat curve labeled negative kurtosis platykurtic.](Media/skew3.png)\n",
    "\n",
    "---\n",
    "\n",
    "# Overview\n",
    "\n",
    "This info is summarized nicely in the figure below: \n",
    "\n",
    "![A table has six graphs in two columns and three rows. The column headings are labeled skewness and kurtosis. The graphs on the skewness column are labeled plus, zero, and minus. The graphs on the kurtosis column are labeled platykurtic, mesokurtic, and leptokurtic.](Media/skew5.png)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05f2d7a2",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 3 - Data Transformations<a class=\"anchor\" id=\"DS105L2_page_3\"></a>\n",
    "\n",
    "[Back to Top](#DS105L2_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c27fe5e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/335019817\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7ff378099130>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Basic Statistics in Python\n",
    "VimeoVideo('335019817', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff8e0313",
   "metadata": {},
   "source": [
    "The transcript for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO105L02pg3tutorial.zip)**.\n",
    "\n",
    "# Transformations\n",
    "\n",
    "Both skew and kurtosis affect many statistical analyses and make them inaccurate, so you need to correct for them if normality is an assumption.  Correcting for skew and kurtosis is called *transformation*.   \n",
    "\n",
    "---\n",
    "\n",
    "## Pros and Cons to Transforming Your Data\n",
    "\n",
    "Pros: \n",
    "\n",
    "* Makes your distribution normal\n",
    "* Helps fix issues with homoscedasticity (even variance) and non-linearity\n",
    "* Improves the accuracy of the results\n",
    "\n",
    "There is only one major con.  In transforming your data, you change the scale, which means that interpretation, especially for regressions, can be much more difficult. \n",
    "\n",
    "---\n",
    "\n",
    "## Common Transformations\n",
    "\n",
    "When you transform data, you change the *power*.  Power is basically: \n",
    "\n",
    "```text\n",
    "1 - slope\n",
    "```\n",
    "\n",
    "The table below shows the most common data transformations and how to do them. \n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Power</th>\n",
    "        <th>Transformation</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>3</td>\n",
    "        <td>Cube</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>2</td>\n",
    "        <td>Square</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>1</td>\n",
    "        <td>No change</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>1/2</td>\n",
    "        <td>Square root</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>0</td>\n",
    "        <td>Logarithm</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>-1/2</td>\n",
    "        <td>Reciprocal square root</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>-1</td>\n",
    "        <td>Reciprocal</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "---\n",
    "\n",
    "### Transforming Negatively Skewed Data\n",
    "\n",
    "The baseline on the chart above is 1.  That would be your data left alone, with no transformation.  Go up from one to a power of two or three, and those transformations will help with negatively skewed data.  The higher the power, the stronger the transformation; so start small and work your way up.  \n",
    "\n",
    "---\n",
    "\n",
    "### Transforming Positively Skewed Data\n",
    "\n",
    "Go down from one, and the next two transformations (power 1/2 and 0) are meant to help with positively skewed data. The smaller the power and the farther away from one, the stronger the transformation. \n",
    "\n",
    "---\n",
    "\n",
    "### Transforming Data with the Reciprocal\n",
    "\n",
    "Lastly, there are two negative value powers, which have you take the reciprocal.  Those are for flipping your data, or reversing the direction it goes in.\n",
    "\n",
    "---\n",
    "\n",
    "### Prescribing Transformations\n",
    "\n",
    "Below is a figure that provides the original, non-parametric data, and the most likely transformation to use.  For instance, if your data looks like the upper left picture, positively skewed but not very kurtotic, try a square root transformation.  \n",
    "\n",
    "![A box labeled transformation on the top of six graphs. The graphs are labeled square root, reflect and square root, logarithm, reflect and logarithm, inverse, and reflect and inverse. The figure has a caption on the bottom that reads, original distributions and common transformations to produce normality.](Media/skew6.png)\n",
    "*From Tabachnick, B.G. & Fidell, L.S. (2007) Using Multivariate Statistics (5th ed). Boston, MA: Pearson.*\n",
    "\n",
    "Although this figure will help guide you, no data distribution is exactly the same, and you may need to try several different transformations before you get it right.  Choose the transformation that most closely approximates the normal distribution. It may not be perfect, but it should at least be better.\n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Always look at a histogram or QQ plot of your data after transformation to see if it did what you intended! </p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17df06a9",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 4 - Transformations in R<a class=\"anchor\" id=\"DS105L2_page_4\"></a>\n",
    "\n",
    "[Back to Top](#DS105L2_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4b20b851",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/335019940\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x7ff38820bc40>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Basic Statistics in Python\n",
    "VimeoVideo('335019940', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "836d6127",
   "metadata": {},
   "source": [
    "The transcript for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO105L02pg4tutorial.zip)**.# Transformations in R\n",
    "\n",
    "Now that you have a basic understanding of what transformations are, you'll do them in R.\n",
    "\n",
    "---\n",
    "## Visualizing Transformations \n",
    "\n",
    "You've already learned how to visually check for normality with ```ggplot```'s ```geom_histogram``` and ```geom_qq```.  But wouldn't it be awesome if a curve was printed on your histogram for you, so you could better assess whether it looked approximately normal? \n",
    "\n",
    "Well, good news! There is. It comes from the ```rcompanion``` library, and is the function ```plotNormalHistogram```.  \n",
    "\n",
    "You'll use some of the datasets you used last lesson to inspect the data for normality.  How about taking a peek at the **[anime data](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/anime.zip)** you were using last hands on? \n",
    "\n",
    "First, you'll need to install and import ```rcompanion```.\n",
    "\n",
    "```{r}\n",
    "library(rcompanion)\n",
    "```\n",
    "\n",
    "Then, you can use the ```plotNormalHistogram``` function on the ```score``` variable: \n",
    "\n",
    "```{r}\n",
    "plotNormalHistogram(anime$score)\n",
    "```\n",
    "\n",
    "![A graph depicts the plot of x values against the frequency. The x-axis ranges from 1940 to 2020 in 5 units and the y-axis ranges from 0 to 1500 in 4 units. A curve is plotted on the bars. The curve and the bars have similar peak points.](Media/skew7.png)\n",
    "\n",
    "Oh good! This one looks to be approximately normally distributed.  No need to transform that. \n",
    "\n",
    "---\n",
    "\n",
    "## Transforming Positively Skewed Data\n",
    "\n",
    "How about ```scored_by```?\n",
    "\n",
    "```{r}\n",
    "plotNormalHistogram(anime$scored_by)\n",
    "```\n",
    "\n",
    "![A graph depicts the plot of x against the frequency. The value of x ranges from 0 e plus 00 to 1 e plus 06 in 6 units. A curve is plotted along the bars. The curve and the bars share almost equal peak points.](Media/skew8.png)\n",
    "\n",
    "No such luck on that one.  It looks to be horribly positively skewed. \n",
    "\n",
    "---\n",
    "\n",
    "### Using sqrt()\n",
    "\n",
    "So start with a square root transformation, by using the function ```sqrt()``` just as a mathematical operation: \n",
    "\n",
    "```{r}\n",
    "anime$scored_bySQRT <- sqrt(anime$scored_by)\n",
    "```\n",
    "\n",
    "Which will produce and image that will look like this:\n",
    "\n",
    "![sqrttransformation](Media/105sqrttransformation.png \"sqrt transformation\")\n",
    "\n",
    "This transforms the ```scored_by``` variable by square rooting it.  It is then stored in a new column in anime, called ```scored_bySQRT``` so you can tell the old and the transformed data apart. \n",
    "\n",
    "Then run the histogram again: \n",
    "\n",
    "```{r}\n",
    "plotNormalHistogram(anime$scored_bySQRT)\n",
    "```\n",
    "\n",
    "\n",
    "\n",
    "It looks better than before, but perhaps it can be better still.  \n",
    "\n",
    "---\n",
    "\n",
    "### Using log()\n",
    "\n",
    "So make a larger transformation - you will now take the log of the ```scored_by``` variable, using the ```log()``` function: \n",
    "\n",
    "```{r}\n",
    "anime$scored_byLOG <- log(anime$scored_by)\n",
    "```\n",
    "\n",
    "Then try to graph: \n",
    "\n",
    "```{r}\n",
    "plotNormalHistogram(anime$scored_byLOG)\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "### Removing Infinite Values\n",
    "\n",
    "You probably got an error here: \n",
    "\n",
    "```Error in seq.default(min(x), max(x), length = length) : 'from' must be a finite number```\n",
    "\n",
    "This is because the log transformation made a few of the numbers so small that R gave up, and just gave them the value of ```-inf```, or infinitely small.  Most functions in R will not work with infinite data, including this one. There is, however, a package and a line of code that can save the day. \n",
    "\n",
    "```{r}\n",
    "library(\"IDPmisc\")\n",
    "anime2 <- NaRV.omit(anime)\n",
    "```\n",
    "\n",
    "The library ```IDPmisc``` contains an ability to omit missing and infinite value data, called ```NaRV.omit()```.  Simply put your dataset name in as the argument.  Then graph again, with the new dataset name: \n",
    "\n",
    "```{r}\n",
    "plotNormalHistogram(anime2$scored_byLOG)\n",
    "```\n",
    "\n",
    "And you are good to go!\n",
    "\n",
    "![A graph depicts the plot of x values against the frequency. The x-axis ranges from 2 to 14 in seven units and the y-axis ranges from 0 to 1000. A curve is plotted on the bars. The curve and the bars have similar peak points.](Media/skew9.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Transforming Negatively Skewed Data\n",
    "\n",
    "You will follow a similar process to transform negatively skewed data, but this time, you will use squaring or cubing to transform your data.  \n",
    "\n",
    "The variable ```aired_from_year``` is negatively skewed: \n",
    "\n",
    "![A graph depicts the plot of x values against the frequency. The x-axis ranges from 1940 to 2020 in 5 units and the y-axis ranges from 0 to 1500 in 4 units. A curve is plotted on the bars. The curve and the bars have similar peak points.](Media/skew10.png)\n",
    "\n",
    "---\n",
    "\n",
    "### Squaring the Variable\n",
    "\n",
    "So you can start by squaring it: \n",
    "\n",
    "```{r}\n",
    "anime$aired_from_yearSQ <- anime$aired_from_year * anime$aired_from_year\n",
    "```\n",
    "\n",
    "Then take a peek at it again: \n",
    "\n",
    "```{r}\n",
    "plotNormalHistogram(anime$aired_from_yearSQ)\n",
    "```\n",
    "\n",
    "![A graph depicts the plot of x values against the frequency. The x-axis ranges from 3750000 to 4050000 in 7 units and the y-axis ranges from 0 to 1500 in 4 units. A curve is plotted on the bars. The curve and the bars have similar peak points.](Media/skew11.png)\n",
    "\n",
    "That looks a bit better, but not amazing.  No where near normally distributed.  \n",
    "\n",
    "---\n",
    "\n",
    "### Cubing the Variable \n",
    "\n",
    "Now try cubing it: \n",
    "\n",
    "```{r}\n",
    "anime$aired_from_yearCUBE <- anime$aired_from_year ^ 3\n",
    "```\n",
    "\n",
    "And take another look:\n",
    "\n",
    "```{r}\n",
    "plotNormalHistogram(anime$aired_from_yearCUBE)\n",
    "```\n",
    "\n",
    "![A graph depicts the plot of x values against the frequency. The x-axis ranges from 7.4 e plus 09 to 8.2 e plus 09 in 5 units and the y-axis ranges from 0 to 1500 in 4 units. A curve is plotted on the bars. The curve and the bars have similar peak points.](Media/skew12.png)\n",
    "\n",
    "Overall, that definitely looks like the best option, so you can settle for that.  Even though it is not perfect, you are out of options.\n",
    "\n",
    "---\n",
    "\n",
    "## Tukey's Ladder of Power Transformation\n",
    "\n",
    "Wouldn't it be wonderful if there was some function that automatically transformed your data correctly, without all this time consuming guess work? Well, there is great news for you! There is.  It's called the ```transformTukey``` function, which performs Tukey's Ladder of Power Transformation, and is available from the ```rcompanion``` package as well. \n",
    "\n",
    "You will try out this new wizardry using the **[cruise ship data](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/cruise_ship.zip)**.  Here is what the original data looks like: \n",
    "\n",
    "```{r}\n",
    "library(rcompanion)\n",
    "plotNormalHistogram(cruise_ship$Tonnage)\n",
    "```\n",
    "\n",
    "![A graph depicts the plot of x values against the frequency. The x-axis ranges from 0 to 10000 in 6 units and the y-axis ranges from 0 to 40 in 5 units. A curve is plotted on the bars. The curve and the bars have similar peak points.](Media/skew13.png)\n",
    "\n",
    "And then you can transform it with Tukey's, where the function ```transformTukey()``` takes the argument ```data$variable``` and where the argument ```plotit=``` takes either ```TRUE``` to add a plot or ```FALSE``` to do without.\n",
    "\n",
    "```\n",
    "cruise_ship$TonnageTUK <- transformTukey(cruise_ship$Tonnage, plotit=FALSE)\n",
    "```\n",
    "\n",
    "Then you can plot again to see if it made a difference: \n",
    "\n",
    "```{r}\n",
    "plotNormalHistogram(cruise_ship$TonnageTUK)\n",
    "```\n",
    "\n",
    "![A graph depicts the plot of x values against the frequency. The x-axis ranges from 0 to 10000 in 6 units and the y-axis ranges from 0 to 40 in 5 units. A curve is plotted on the bars. The curve and the bars have similar peak points.](Media/skew14.png)\n",
    "\n",
    "And it did! Your data is now approximately normal, and it didn't require you to do a lot of trial and error guesswork.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84644b43",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 5 - Transformations in R Activity<a class=\"anchor\" id=\"DS105L2_page_5\"></a>\n",
    "\n",
    "[Back to Top](#DS105L2_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2d27e99",
   "metadata": {},
   "source": [
    "For this Activity, you will transform data in R. `This Hands-On will not be graded`, but you are encouraged to complete it. The best way to become a great data scientist is to practice! Once you have submitted your project, you will be able to access the solution on the next page. Note that the solution will be slightly different from yours, but should look similar.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Do not submit your project until you have completed all requirements, as you will not be able to resubmit.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Requirements\n",
    "\n",
    "Using the **[cruise ship data from last lesson](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/cruise_ship.zip)**, determine whether each continuous variable is positively skewed, negatively skewed, or normally distributed.  Then perform the correct transformations to get as close to the normal distribution as possible for each variable. \n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Be sure to zip and submit your entire directory when finished!</p>\n",
    "    </div>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc7f18e2",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 6 - Transformations in R Activity Solution<a class=\"anchor\" id=\"DS105L2_page_6\"></a>\n",
    "\n",
    "[Back to Top](#DS105L2_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fe7608e",
   "metadata": {},
   "source": [
    "# Solution\n",
    "\n",
    "---\n",
    "\n",
    "## Answers\n",
    "\n",
    "The following variables are positively skewed and can be transformed by taking the square root: \n",
    "\n",
    "* Tonnage\n",
    "* Cabins\n",
    "* passngrs\n",
    "* Crew\n",
    "* PassSpcR\n",
    "* outcab\n",
    "\n",
    "The following variables are negatively skewed: \n",
    "\n",
    "* YearBlt\n",
    "* Length\n",
    "\n",
    "```YearBlt``` should be transformed by cubing, ```Length``` by squaring.\n",
    "\n",
    "---\n",
    "\n",
    "## Code\n",
    "\n",
    "```{r}\n",
    "# Load the appropriate library\n",
    "\n",
    "library(rcompanion)\n",
    "```\n",
    "# Look to see what all the distributions are\n",
    "```{r}\n",
    "plotNormalHistogram(cruise_ship$YearBlt)\n",
    "plotNormalHistogram(cruise_ship$Tonnage)\n",
    "plotNormalHistogram(cruise_ship$passngrs)\n",
    "plotNormalHistogram(cruise_ship$Length)\n",
    "plotNormalHistogram(cruise_ship$Cabins)\n",
    "plotNormalHistogram(cruise_ship$Crew)\n",
    "plotNormalHistogram(cruise_ship$PassSpcR)\n",
    "plotNormalHistogram(cruise_ship$outcab)\n",
    "```\n",
    "# Transform Positively Skewed Variables\n",
    "```{r}\n",
    "cruise_ship$TonnageSQRT <- sqrt(cruise_ship$Tonnage)\n",
    "cruise_ship$passngrsSQRT <- sqrt(cruise_ship$passngrs)\n",
    "cruise_ship$CabinsSQRT <- sqrt(cruise_ship$Cabins)\n",
    "cruise_ship$CrewSQRT <- sqrt(cruise_ship$Crew)\n",
    "cruise_ship$PassSpcRSQRT <- sqrt(cruise_ship$PassSpcR)\n",
    "cruise_ship$outcabSQRT <- sqrt(cruise_ship$outcab)\n",
    "```\n",
    "# See if that fixes the issues\n",
    "```{r}\n",
    "plotNormalHistogram(cruise_ship$TonnageSQRT)\n",
    "plotNormalHistogram(cruise_ship$passngrsSQRT)\n",
    "plotNormalHistogram(cruise_ship$CabinsSQRT)\n",
    "plotNormalHistogram(cruise_ship$CrewSQRT)\n",
    "plotNormalHistogram(cruise_ship$PassSpcRSQRT)\n",
    "plotNormalHistogram(cruise_ship$outcabSQRT)\n",
    "```\n",
    "# Transform Negatively Skewed Variables\n",
    "```{r}\n",
    "cruise_ship$YearBltSQ <- cruise_ship$YearBlt * cruise_ship$YearBlt\n",
    "cruise_ship$LengthSQ <- cruise_ship$Length * cruise_ship$Length\n",
    "```\n",
    "# See if that made them normal\n",
    "```{r}\n",
    "plotNormalHistogram(cruise_ship$YearBltSQ)\n",
    "plotNormalHistogram(cruise_ship$LengthSQ)\n",
    "```\n",
    "# Length looks ok, but YaerBlt could use additional transformation\n",
    "```{r}\n",
    "cruise_ship$YearBltCUBE <- cruise_ship$YearBlt ^3\n",
    "\n",
    "plotNormalHistogram(cruise_ship$YearBltCUBE)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f00dcca",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 7 - Transformations in Python<a class=\"anchor\" id=\"DS105L2_page_7\"></a>\n",
    "\n",
    "[Back to Top](#DS105L2_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "67c8e912",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/335021517\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x112add5e0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Basic Statistics in Python\n",
    "VimeoVideo('335021517', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5192512",
   "metadata": {},
   "source": [
    "The transcript for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO105L02pg7tutorial.zip)**.\n",
    "\n",
    "# Transformations in Python\n",
    "\n",
    "Data transformations can also be done in Python.  You'll use the same **[anime data](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/anime.zip)** that you used to transform data in R.\n",
    "\n",
    "---\n",
    "\n",
    "## Visualizing Transformations\n",
    "\n",
    "```pandas``` has a function called ```.hist()``` that works pretty well for data normalization visualization, because it just ignores missing values.  That means that you don't need to do a whole lot of data cleaning.  \n",
    "\n",
    "Here's how ```.hist()``` works: \n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "anime.aired_from_year.hist()\n",
    "```\n",
    "\n",
    "Just place the data frame name first, then the variable name, and finally call the function ```.hist()```. The only downside to ```.hist()``` is that is does not provide a fitted curve.  \n",
    "\n",
    "![A bar chart represents the year from 1940 to 2020 in 9 units on the x-axis and the numbers from 0 to 3000 on the y-axis. The bars are plotted in an increasing pattern.](Media/skew15.png)\n",
    "\n",
    "If you want a fitted curve, you'll need to pull from ```seaborn```, using the ```distplot()``` function.\n",
    "\n",
    "### [Errata](Errata/DS105-Change-Log.ipynb)\n",
    "\n",
    "```python\n",
    "import seaborn as sns\n",
    "sns.histplot(anime['aired_from_year'])\n",
    "```\n",
    "<a class=\"anchor\" id=\"histplot\"></a>\n",
    "Here is the result: \n",
    "\n",
    "![A graph represents the plot with the representation of aired from year on the x-axis. The x-axis ranges from 1940 to 2020. The y-axis ranges from 0.00 to 0.10 in six units. A curve along with the bars is plotted on the graph.](Media/skew16.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Transforming Positively Skewed Data\n",
    "\n",
    "Take a look at the ```scored_by``` variable: \n",
    "\n",
    "```python\n",
    "anime.scored_by.hist()\n",
    "```\n",
    "\n",
    "![The x-axis on a graph ranges from 0 to 1000000 and the y-axis ranges from 0 to 6000. The tallest bar crosses the point 6000 on the y-axis.](Media/skew17.png)\n",
    "\n",
    "It is quite positively skewed! \n",
    "\n",
    "---\n",
    "\n",
    "### Using np.sqrt()\n",
    "\n",
    "So, try a square root transformation first, using the function from ```numpy``` called ```.sqrt()```: \n",
    "\n",
    "```python\n",
    "anime['scored_bySQRT'] = np.sqrt(anime['scored_by'])\n",
    "```\n",
    "\n",
    "![The x-axis on a graph ranges from 0 to 1000 and the y-axis ranges from 0 to 4000. The tallest bar crosses the point 4000 on the y-axis.](Media/skew18.png)\n",
    "\n",
    "Looking at the above graph, that is better, but still no where close to normal.  \n",
    "\n",
    "---\n",
    "\n",
    "### Using np.log()\n",
    "\n",
    "So, try a log transformation! \n",
    "\n",
    "```python\n",
    "anime['scored_byLOG'] = np.log(anime['scored_by'])\n",
    "```\n",
    "\n",
    "Again, this ```.log()``` function comes from ```numpy```.  You may get a warning when you run it like this one: \n",
    "\n",
    "```\n",
    "RuntimeWarning: divide by zero encountered in log\n",
    "```\n",
    "\n",
    "This tells you that there will be infinite values! \n",
    "\n",
    "---\n",
    "\n",
    "### Dealing with Infinite Data\n",
    "\n",
    "If you try and run the histogram of your log transformed data when you've been given a warning about infinite values, you will get this error: \n",
    "\n",
    "```python\n",
    "ValueError: range parameter must be finite.\n",
    "```\n",
    "\n",
    "But don't panic! There is a fix, and a relatively easy one at that.  Dropping ```na``` values will also get rid of infinite values! Simply call the ```dropna()``` function on your dataset.\n",
    "\n",
    "```python\n",
    "anime.dropna(inplace=True)\n",
    "```\n",
    "\n",
    "And then you are ready to rumble with your histogram once more: \n",
    "\n",
    "```python\n",
    "anime.scored_byLOG.hist()\n",
    "```\n",
    "\n",
    "![A bar chart with eight units on the x-axis and seven units on the y-axis. The x-axis ranges from seven to fourteen and the y-axis ranges from 0 to 70.](Media/skew19.png)\n",
    "\n",
    "That doesn't look too shabby! Not quite normal, but you'll take it. \n",
    "\n",
    "---\n",
    "\n",
    "## Transforming Negatively Skewed Data\n",
    "\n",
    "In order to transform negatively skewed data, you will either square or cube your data. \n",
    "\n",
    "---\n",
    "\n",
    "### Squaring the Variable\n",
    "\n",
    "How about trying to transform the ```aired_from_year``` variable that you looked at earlier? It had a relatively large negative skew to it. So, start by squaring your data. The ```**``` means that you are raising the variable to a power of 2. \n",
    "\n",
    "```python\n",
    "anime['aired_from_yearSQ'] = anime['aired_from_year']**2\n",
    "```\n",
    "\n",
    "Then take a look at the histogram to assess your progress: \n",
    "\n",
    "```python\n",
    "anime.aired_from_yearSQ.hist()\n",
    "```\n",
    "\n",
    "![A bar chart with six units on the x-axis and the y-axis. The x-axis ranges from 3800000 to 4050000 and the y-axis ranges from 0 to 3000. The bars are plotted in an increasing pattern.](Media/skew20.png)\n",
    "\n",
    "---\n",
    "\n",
    "### Cubing the Variable\n",
    "\n",
    "The histogram above still does not look very normal, which means that it is time to try cubing it!\n",
    "\n",
    "```python\n",
    "anime['aired_from_yearCUBE'] = anime['aired_from_year']**3\n",
    "```\n",
    "\n",
    "Then check the histogram once more: \n",
    "\n",
    "```python\n",
    "anime.aired_from_yearCUBE.hist()\n",
    "```\n",
    "\n",
    "![A bar chart with five units on the x-axis and the y-axis. The x-axis ranges from 0 to 8.2 and the y-axis ranges from 0 to 3000. The bars are plotted in an increasing pattern.](Media/skew21.png)\n",
    "\n",
    "This has not made a lot of impact, but it is slightly better than the original, so you most likely want to use the cubed transformation.\n",
    "\n",
    "---\n",
    "\n",
    "## BoxCox Transformation\n",
    "\n",
    "Just like the Tukey's Ladder of Power Transformations in R, you can transform by power in Python as well.  However, the Python version, called ```boxcox()```, has limited functionality. It does not seem to work well for negatively skewed data (so check your results very carefully!), and it does not automatically transform your data, unlike R. It just runs off a power system.\n",
    "\n",
    "In order to use the ```boxcox()``` function, you will need to import a few packages:\n",
    "\n",
    "```python\n",
    "from scipy import stats\n",
    "from scipy.stats import boxcox\n",
    "```\n",
    "\n",
    "Then you can call the function into a new variable, like you have been doing: \n",
    "\n",
    "```python\n",
    "anime['scored_byLOG1'] = boxcox(anime['scored_by'],0)\n",
    "```\n",
    "\n",
    "Use the ```boxcox()``` function, and specify the variable.  Then the last argument is the power value that you learned about at the beginning of this lesson.  A power of 0 is a log transformation.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5fbf088",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 8 - Transformations in Python Activity<a class=\"anchor\" id=\"DS105L2_page_8\"></a>\n",
    "\n",
    "[Back to Top](#DS105L2_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f52b12f",
   "metadata": {},
   "source": [
    "For this Activity, you will transform data in Python. `This Hands-On will not be graded`, but you are encouraged to complete it. The best way to become a great data scientist is to practice! Once you have submitted your project, you will be able to access the solution on the next page. Note that the solution will be slightly different from yours, but should look similar. \n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Do not submit your project until you have completed all requirements, as you will not be able to resubmit.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Requirements\n",
    "\n",
    "Using the **[cruise ship data from last lesson](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/cruise_ship.zip)**, determine whether each continuous variable is positively skewed, negatively skewed, or normally distributed.  Then perform the correct transformations to get as close to the normal distribution as possible for each variable. \n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Be sure to zip and submit your entire directory when finished!</p>\n",
    "    </div>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bd36cb3",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 9 - Transformations in Python Activity Solution<a class=\"anchor\" id=\"DS105L2_page_9\"></a>\n",
    "\n",
    "[Back to Top](#DS105L2_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bde834fd",
   "metadata": {},
   "source": [
    "# Solution\n",
    "\n",
    "Here is a [Jupyter Notebook with the answers](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/Transforming_Data_in_Python_Activity_Solution.zip), so that you can check your work! \n",
    "\n",
    "<div class=\"panel panel-info\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Tip!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>You will need to extract the zip file and save it to your computer, then open it in Jupyter Notebook in order to open the above file.</p>\n",
    "    </div>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "325c7456",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 10 - Key Terms<a class=\"anchor\" id=\"DS105L2_page_10\"></a>\n",
    "\n",
    "[Back to Top](#DS105L2_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eea8a17",
   "metadata": {},
   "source": [
    "# Key Terms\n",
    "\n",
    "Below is a list and short description of the important keywords learned in this lesson. Please read through and go back and review any concepts you do not fully understand. Great Work!\n",
    "\n",
    "---\n",
    "\n",
    "## Key Terms\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Skew</td>\n",
    "        <td>Non-parametric distribution that is not normal horizontally from side to side.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Negative Skew</td>\n",
    "        <td>A distribution that has the tail to the left and the bulk of data to the right.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Positive Skew</td>\n",
    "        <td>A distribution that has the tail to the right and the bulk of the data to the left.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Kurtosis</td>\n",
    "        <td>Non-parametric distribution that is not normally vertically, from top to bottom.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Leptokurtic</td>\n",
    "        <td>A positively kurtotic distribution that looks taller than the normal distribution.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Platykurtic</td>\n",
    "        <td>A negatively kurtotic distribution that looks flatter than the normal distribution.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Mesokurtic</td>\n",
    "        <td>The normal distribution's kurtosis. </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Power</td>\n",
    "        <td>Defined as 1-slope, power is on a ladder system that determines how data can be transformed.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>Reciprocal or Reflection</td>\n",
    "        <td>The exact opposite of your current data; flipped.</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "---\n",
    "\n",
    "## Key R Libraries \n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>rcompanion</td>\n",
    "        <td>Used to test for the normal distribution. </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>IDPmisc</td>\n",
    "        <td>A library used to screen out missing values.</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## Key R Code\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>plotNormalHistogram()</td>\n",
    "        <td>Easily creates a histogram with a best-fit curve to show how it matches the normal distribution.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>sqrt()</td>\n",
    "        <td>Takes the square root of a variable.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>log()</td>\n",
    "        <td>Takes the log of a variable.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>NaRV.omit()</td>\n",
    "        <td>Removes missing and infinite values from a dataset.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>transformTukey</td>\n",
    "        <td>Automatically transforms your data to approximate the normal distribution as best as possible.</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## Key Python Packages\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>boxcox</td>\n",
    "        <td>A package that houses the BoxCox transformation.</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## Key Python Code\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>np.sqrt()</td>\n",
    "        <td>Takes the square root of a variable.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>np.log()</td>\n",
    "        <td>Takes the log of a variable.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>.dropna(inplace=True)</td>\n",
    "        <td>Drops missing values.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>boxcox()</td>\n",
    "        <td>Performs a BoxCox transformation based on the power level you choose.</td>\n",
    "    </tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "513e689f",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 11 - Lesson 2 Hands-On<a class=\"anchor\" id=\"DS105L2_page_11\"></a>\n",
    "\n",
    "[Back to Top](#DS105L2_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "083e5343",
   "metadata": {},
   "source": [
    "For this Hands On, you will be transforming data in both Python and R.  \n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Do not submit your project until you have completed all requirements, as you will not be able to resubmit.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Requirements\n",
    "\n",
    "This hands on uses a dataset about the number of trips done at the Seattle Parks and Recreation department.  It is located **[here](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/Seattle_ParksnRec.zip)**. For each part, assess and transform the requested data, then submit your annotated program files for review.\n",
    "\n",
    "---\n",
    "\n",
    "### Part I: Transforming Data in Python\n",
    "\n",
    "In Python, assess the skew of the distribution and then transform it if necessary for the following variables:  \n",
    "\n",
    "* ```# of trips Winter```\n",
    "* ```# of participants Winter```\n",
    "* ```# of trips Spring```\n",
    "* ```# of participants Spring```\n",
    "* ```# of trips Summer```\n",
    "* ```# of participants Summer```\n",
    "\n",
    "Please make notes about each variable's distribution and the transformation you made in your Python file and submit.\n",
    "\n",
    "---\n",
    "\n",
    "### Part II: Transforming Data in R\n",
    "\n",
    "In R, assess the skew of the distribution and then transform it if necessary for the following variables:  \n",
    "\n",
    "* ```# of trips Fall```\n",
    "* ```# of participants Fall```\n",
    "* ```# of trips per Year```\n",
    "* ```# participants per Year```\n",
    "* ```increase/decrease of prior year```\n",
    "* ```Average # people per trip```\n",
    "\n",
    "Please make notes about each variable's distribution and the transformation you made in your R script and submit.\n",
    "\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Be sure to zip and submit your entire directory when finished!</p>\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3a52e49",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}