Python vs R - the Good, the Bad, and the Ugly.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python vs. R - The Good, The Bad, and the Ugly\n",
    "\n",
    "This is a **Jupyter Notebook** with Python3, but... cool hack: we can also run R in cells in the same Notebook!  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some house keeping\n",
    "\n",
    "#### Virtual Environments\n",
    "\n",
    "I use https://docs.python.org/3/library/venv.html to create virtual environment (python3) on my Mac.  (Python 2 is going away...)\n",
    "\n",
    "#### Jupyter Notebooks\n",
    "\n",
    "I don't use the ananconda installation of jupyter because they don't keep up with all the python libraries.  In install Jupyter Notebook in my Python3 environment\n",
    "\n",
    "Use `pip3 install jupyter`\n",
    "\n",
    "#### Output Jupyter Notebook to PDF\n",
    "\n",
    "You'll need a couple of more things:\n",
    "\n",
    "* http://pandoc.org/installing.html\n",
    "* https://nbconvert.readthedocs.io/en/latest/install.html#installing-tex - need for LaTex\n",
    "* (and some tweaking of PATH)\n",
    "\n",
    "The output will go to your Download folder\n",
    "But it might be easier to \"print\" as PDF\n",
    "\n",
    "#### Jupyter Notebook magic\n",
    "Lots of cool stuff with Jupyter Notebook (old name: iPython) Magic:\n",
    "\n",
    "* https://ipython.readthedocs.io/en/stable/interactive/magics.html\n",
    "\n",
    "\n",
    "#### Many languages are suppored in Jupyter Notebooks\n",
    "\n",
    "* https://github.com/jupyter/jupyter/wiki/Jupyter-kernels\n",
    "\n",
    "\n",
    "#### Running both Python and R in same Notebook\n",
    "\n",
    "You probably don't want to do this in Real Life, but...\n",
    "\n",
    "If you want to use R with Python in Jupyter Notebook, need:  \n",
    "\n",
    "* https://rpy2.github.io/doc/v2.9.x/html/index.html\n",
    "\n",
    "NOTE: you have to be running jupyter notebook under python3, unless you have previously installed rpy2 under python 2\n",
    "\n",
    "On your command line:\n",
    "\n",
    "`pip3 install rpy2 tzlocal`\n",
    "\n",
    "## Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext rpy2.ipython\n",
    "# This is allows the us to run R in cells with %%R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "print(sys.version_info)\n",
    "print(sys.version)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "getRversion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Every Computer Language is a collection of features and compromises\n",
    "* Some features are **Good** (depending)\n",
    "* Some features are **Bad** (depending)\n",
    "* Some features are **Ugly**\n",
    "\n",
    "Which language is better (best?)\n",
    "\n",
    "Depends.  Which tool is better: screw driver or hammer?\n",
    "\n",
    "You're a better handyperson if you know how to use more than one tool!\n",
    "\n",
    "It's a bit like being bilingual.\n",
    "* beware of *false friends*: expressions in different languages that look the same but aren't\n",
    "    * *demand* in French means *ask*; in English it means *ask forcefully*\n",
    "    \n",
    "    \n",
    "\n",
    "## For Data Science, there's an important consideration\n",
    "\n",
    "**Which language supports the library that I need to do my job?**\n",
    "\n",
    "* R has better support for **statistics** (especially more obscure/sophisticated)\n",
    "* Python has better support for **Machine Learning** and many other libraries\n",
    "    * Python is more widely used.\n",
    "    * Python has a fair amount of statistical support, so R may be less useful.\n",
    "\n",
    "\n",
    "## Underlying Philosophy\n",
    "\n",
    "* Python was designed and implemented by \"Computer Scientists\"\n",
    "\n",
    "* R was designed and implemented by \"Statisticians\"\n",
    "    * Joke:  The good news about R is it's a language designed by statisticians.  The bad news about R is..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## R and Python are *Object Oriented* (OO)\n",
    "\n",
    "Object Oriented Programming (OOP), has many advantages:\n",
    "* you can package up different groups of functions together\n",
    "* you create a separate name space, so two different authors can use the same name for a function\n",
    "* You can use objects instead of using dictionaries or R's \"list\" to compartmentalize named values.\n",
    "\n",
    "\n",
    "### Good\n",
    "* Fortunately, OOP in Python was designed and implemented by Computer Scientists.\n",
    "* Python also has `collections.namedtuple` which gives you OOP syntax but lighter weight\n",
    "\n",
    "### Bad\n",
    "* \n",
    "\n",
    "### Ugly\n",
    "* Unfortunately, OOP in R was not designed and implemented by Computer Scientists.  As a result, there are 3 major (and various minor) versions of R's OP. OOP notation not built into language. Gory details here:  https://adv-r.hadley.nz/oo.html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## R and Python are *Free*\n",
    "\n",
    "* Free is **Good**\n",
    "* Free is **Bad** not professionally maintained and designed, bugs!\n",
    "* Free is **Ugly**\n",
    "    * R has inconsistently named libraries, interfaces, names, error messages, and namespace collisions.  Documentation is often lacking.\n",
    "    * In R, you can specify function from a library with this syntax: *LIBRARY*::*FUNCTION*.  Example:  `dplyr::intersect(first, second)`\n",
    "    * with Python, this is less of an issue, because of recommended standards\n",
    "    * with both, you have to worry about consistency/dependency between libraries but there are ways package up your environment with specific versions of the libraries.\n",
    "    * with both, some libraries were re-implemented, so picking a \"best\" library is hard: R has three graphics packages.  I'm not sure how many Python has...  This is because there's no enforcing standards committee for libraries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## R and Python are *Interpreted*\n",
    "\n",
    "Two types of computer languages:\n",
    "* Interpreted\n",
    "    * Faster to get a resulting program, but it will run slower.\n",
    "    * Most programs are run only once!\n",
    "* Compiled\n",
    "    * Slower to get a resulting program, but it will run faster\n",
    "    * can be big savings over time\n",
    "    \n",
    "### Good - Interpreted means experimentation is fast\n",
    "* Most programs don't even run once!\n",
    "\n",
    "### Bad - Interpreted means the code could be impractical to run in production\n",
    "* Often Optimized libraries are reimplemented in C (or other languages) to run faster\n",
    "\n",
    "### Ugly - there are ways to speed up the code but it's not pretty...\n",
    "* Often critical libraries are rewritten to run fast but they can be clumsy to use\n",
    "* Python and R weren't designed to run in parallel, but there are hacks!\n",
    "* Using double FOR loops on matices is SLOW; better to use libraries\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment of Python and R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 1\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R \n",
    "x <- 1  # I prefer this, as it reminds me that I'm using R!\n",
    "x = 1   # allowed\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## R Variable Naming - Ugly\n",
    "\n",
    "In R:\n",
    "* Periods **.** are treated as an allowable character in variable names\n",
    "* variables can't start with an underscore **\\_**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%R\n",
    "# DON'T BE FOOLED INTO THINKING is IS AN OBJECT\n",
    "print(is.null(1))\n",
    "print(is.character(\"a\"))\n",
    "internal_ <- NULL\n",
    "print(is.null(internal_)) # trailing underscore is OK\n",
    "\n",
    "\n",
    "print(is.that)            # that() isn't a member of \"is\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R \n",
    "_internal <- 4   # BAD: leading underscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What Constitutes the Basic \"Operand/Entity\" in Python vs R?\n",
    "\n",
    "In Python it's \"Object\" or built-in type (integer, string, etc.)\n",
    "\n",
    "In R, it's a \"Vector\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 1\n",
    "print(type(a))\n",
    "\n",
    "b = [1, 2, 3]\n",
    "print(type(b))\n",
    "\n",
    "c = (1, 2, 3)\n",
    "print(type(c))  # immutable collection\n",
    "\n",
    "# And lots of other Python aggregate classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "a <- 1\n",
    "print(class(a))\n",
    "\n",
    "b <- c(1, 2, 3)     # R is a bit clumsier creating a \"vector\"\n",
    "print(class(b))\n",
    "\n",
    "# R has a type called \"list\" but it's very different from what Python and Computer Scientists call a \"list\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Operating on Basic Entities in Python vs R\n",
    "\n",
    "* These operations make sense for CS, but not so much to a Statistician"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding two arrays in Python: concatenation\n",
    "v1 = [1,2,3]\n",
    "v2 = [4,5,6]\n",
    "v1+v2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Multiplying an array in Python: replication\n",
    "3*v1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1*v2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "# Adding two arrays in R: parallelization \n",
    "v1 = c(1,2,3)\n",
    "v2 = c(4,5,6)\n",
    "v1+v2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "# Multiplying an array in R: parallelization\n",
    "3*v1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "v1*v2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enter Python numpy\n",
    "\n",
    "Numpy is Python's Numeric library.  It has its own array implementation which is more efficient and allows for more R type syntax.\n",
    "\n",
    "Drawback, you have to remember if an object is numpy or regular python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np  # import the numpy library and allow for the nickname 'np'\n",
    "\n",
    "w1 = np.array( [1, 2, 3] )\n",
    "w2 = np.array( [4, 5, 6] )\n",
    "print(w1+w2)\n",
    "print(3*w1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(w1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert(type(w1)== list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing an Array/Vector in Python vs. R\n",
    "\n",
    "* Computer Scientists\n",
    "    * like to start counting from 0.  (Makes sense from Computer Architecture)\n",
    "    * like to stop counting at N-1\n",
    "        * BUT: in python code, often N shows up with the understanding the end is N-1\n",
    "\n",
    "* Everyone else, including statisticians\n",
    "    * like to start counting from 1.  (Make sense from Prehistoric times).\n",
    "    * like to stop counting at N"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(v1[0])\n",
    "print(v1[2])\n",
    "print(v1[3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1[3] = 4  # Python protects you when you goof, but less \"convenient\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "print(v1[1])\n",
    "print(v1[3])\n",
    "\n",
    "# \n",
    "print(v1[0])  \n",
    "print(v1[4])  # oops!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "v1[5] <- 5   # Convenient but potentially dangerous\n",
    "v1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing Multiple Elements from Array/Vector in Python vs. R\n",
    "\n",
    "Python's numpy array and R's vector can handle a \"list\" of multiple indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w1[[1,2]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "v1[ c(2,3) ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BUT regular Python arrays works differently\n",
    "\n",
    "It needs a \"slice\" or a **List Comprehensions**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# non-numpy array\n",
    "v1[[1,2]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1[1:3]  # slice does NOT include 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List Comprehensions\n",
    "[ v1[i] for i in [1,2]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing an Array/Vector with NEGATIVE indices in Python vs. R\n",
    "\n",
    "Address an element of an array in assembly code or C is usually a bad idea.\n",
    "\n",
    "Idea!  Why not use negative indices to mean something \"useful\"?!!\n",
    "\n",
    "* Computer Scientists are always accessing an element from the end of an array with the algorithms\n",
    "    * Use negative index to represent the end of the array\n",
    "    * less ugly than `v1[len(v1) -1]`\n",
    "* Statisticians are always removing elements of an array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(v1[-1])   # access element 3\n",
    "v1[len(v1) -1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "print(v1[-1])  # drops element 1\n",
    "v1[length(v1)]  # no minus one, since R counts from 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "a = np.array([1,2,3])\n",
    "b = np.array([1,2,3])\n",
    "a+b\n",
    "3*a\n",
    "a[3] = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python Dictionary vs. R List\n",
    "\n",
    "What R calls a \"List\" is different from what computer science calls a \"List\"\n",
    "\n",
    "R's list is more like an ordered dictionary.\n",
    "\n",
    "BUT: R's list, the keys cannot be numbers\n",
    "\n",
    "Syntactically:\n",
    "\n",
    "* Python, a \"list\" (array) and dictionary looks the same.  Python's dictionary can be thought of as an **associative array**, so that syntatically, addressing by key in a dictionary looks the same as addressing a list by index.  Advantage: if you change your mind about using a list vs. dictionary (vs. numpy array), you don't have to change the syntax.\n",
    "    * BUT if **insertion order** is important, use `OrderedDict`\n",
    "    * BUT if **key order** is important, use `sortedcontainers.SortedDict`\n",
    "    \n",
    "\n",
    "in R, accessing elements in vector and \"list\" looks different.  Accessing an element of a \"list\" looks ugly.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Most like R's \"list\"\n",
    "from collections import OrderedDict\n",
    "od = OrderedDict() \n",
    "od['a'] = 1\n",
    "od['b'] = 2\n",
    "od['c'] = 3\n",
    "od['d'] = 4\n",
    "od[5] = 'five'           # key for Python sd can be an integer\n",
    "print(od['b'])           # access by key\n",
    "print(list(od)[1])       # for  iloc to access by index\n",
    "od"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keeping keys in Sorted Order\n",
    "# pip install sortedcontainers\n",
    "# These are similiar\n",
    "# collections.OrderedDict:  \n",
    "# sortedcontainers.SortedDict but SortedDict has iloc() and requires keys be comparable (e.g. can't have mixed int & str)\n",
    "from sortedcontainers  import SortedDict  \n",
    "sd = SortedDict() \n",
    "sd['a'] = 1\n",
    "sd['b'] = 2\n",
    "sd['c'] = 3\n",
    "sd['d'] = 4\n",
    "sd['5'] = 'five'        # key for Python sd cannot be an int if other keys are strings!\n",
    "print(sd['b'])          # access by key\n",
    "print(sd.keys()[1])     # for  iloc to access by index\n",
    "sd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "rlist <- list(\n",
    "    \"a\" = 1,\n",
    "    \"b\" = 2,\n",
    "    \"c\" = 3,\n",
    "    \"d\" = 4,\n",
    "    5 = \"five\"  # ERROR: numerics can't be a key!  This is because ints can be used to index R list\n",
    ")\n",
    "rlist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "rlist <- list(\n",
    "    \"a\" = 1,\n",
    "    \"b\" = 2,\n",
    "    \"c\" = 3,\n",
    "    \"d\" = 4,\n",
    "    \"5\" = \"five\"  # but a number as a string can be a key\n",
    ")\n",
    "rlist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "print(class(rlist))\n",
    "print(class( rlist['a'] ))  # with a single bracket, you get back a list and not a vector\n",
    "rlist['a']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "rlist[['a']]    ## Access single item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "rlist[['a']] = \"hi\"\n",
    "rlist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IMPORTANT: R list can hold heterogeneous collection of data types\n",
    "\n",
    "Python arrays and tuples can hold heterogeneuous collection of types.  While more flexible, the implementation for this is inefficient.\n",
    "\n",
    "R vector's can only be the same type.  CAUTION: If you try to create a R vector of multiple types, R will silently convert the types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "('a', 0, False)  # tuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "['a', 0, False]  # Python list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "c(1,  TRUE)  # WITHOUT WARNING: Here, R vector converts everything to NUMBERS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "c(1, \"string\", TRUE)  # WITHOUT WARNING: Here, R vector converts everything to STRINGS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "# Use R's \"list\" if you want to have a collection of heterogenuous items\n",
    "complex <- list(1, list(1, \"string\", TRUE)) # Nested list, no keys\n",
    "complex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "complex[[2]][2] # get second item from second item (which is a list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Truth?\n",
    "\n",
    "Conditional statements are the work horse of any language but every language implements what is TRUE or FALSE differently!\n",
    "\n",
    "\n",
    "It's hard to remember the implementation details, be careful if argument isn't explicitly a Boolean in Python or R.\n",
    "\n",
    "Sometimes, the error message can be obscure.\n",
    "\n",
    "R has the additional complication of having `NULL` and `NA` as well as `NaN`.  If you accidentally test these values for TRUE/FALSE, you'll get a somewhat mysterious error message.\n",
    "\n",
    "** Use R's built in tests**:\n",
    "* `is.nan()`\n",
    "* `is.null()`\n",
    "* `is.na()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 0:\n",
    "    print(\"TRUE\")\n",
    "else:\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if []:\n",
    "    print(\"TRUE\")\n",
    "else:\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if ():\n",
    "    print(\"TRUE\")\n",
    "else:\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (None):\n",
    "    print(\"TRUE\")\n",
    "else:\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (float('nan')):\n",
    "    print(\"TRUE\")\n",
    "else:\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "if (0)\n",
    "    print(\"TRUE\")\n",
    "else\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "if (c())\n",
    "    print(\"TRUE\")\n",
    "else\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "myNA <- NA\n",
    "if (myNA)\n",
    "    print(\"TRUE\")\n",
    "else\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "if (is.na(myNA))\n",
    "    print(\"TRUE\")\n",
    "else\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "myNULL <- NULL\n",
    "if (myNULL)\n",
    "    print(\"TRUE\")\n",
    "else\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "myNULL <- NULL\n",
    "if (is.null(myNULL))\n",
    "    print(\"TRUE\")\n",
    "else\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "myNaN <- NaN\n",
    "if (myNaN)\n",
    "    print(\"TRUE\")\n",
    "else\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "if (is.nan(myNaN))\n",
    "    print(\"TRUE\")\n",
    "else\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "is.null(sqrt(-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## R's treatment of boolean vectors can be tricky but it is powerful\n",
    "\n",
    "R and Python both have `any()` and `all()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "# Looks like R just checks the first element\n",
    "if (c(0, 1)) \n",
    "    print(\"TRUE\")\n",
    "else\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "if (c(1, 0, 0, 0, 0, 1))\n",
    "    print(\"TRUE\")\n",
    "else\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "if (any(c(0, 0, 1, 0, 0, 0)))\n",
    "    print(\"TRUE\")\n",
    "else\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "if (all(c(0, 0, 1, 0, 0, 0)))\n",
    "    print(\"TRUE\")\n",
    "else\n",
    "    print(\"FALSE\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataFrames\n",
    "\n",
    "The heart of doing Data Science is putting your data into a **DataFrame**.  Superficially, it looks like a matrix or if you are more sophisticated: a table from a spreadsheet.\n",
    "\n",
    "Key: the main design and implementation consideration of DataFrames is to make Data Science and Machine Learning Operations more efficient.\n",
    "* A DataFrame is more restricted than a matrix\n",
    "* IF abused, DF will have unexpected behavior.\n",
    "\n",
    "* A **column** of a DataFrame has the same data type throughout\n",
    "    * Homgenuous Data Type has a more efficient layout in memory\n",
    "* A **row** of a DataFrame represents a heterogenuous collection of data.\n",
    "    * Allows for heterogenuous data for a training set example\n",
    "\n",
    "| ID | Name | Weight | Smokes |   Ethnicity    |\n",
    "|----|------|--------|--------|----------------|\n",
    "| 24 | Sue  |  115.2 | TRUE   | NativeAmerican |\n",
    "| 45 | Sam  | 230.8  | FALSE  | PacificIslander|\n",
    "\n",
    "Columns:\n",
    "* ID is an integer\n",
    "* Name is a string, arbitrary number of these\n",
    "* Weight is a float\n",
    "* Smokes is a boolean\n",
    "* Ethnicity: could be a string OR could be a \"category\"/\"factor\": a predetermined symbol that represents an **unordered value** represented as a small set of ints (more efficient)\n",
    "    * Can't (easily) add to existing set of factors\n",
    "\n",
    "Rows:\n",
    "* is an instance of one training set example\n",
    "\n",
    "### Good\n",
    "* R understood the importance of this and has a clean implementation, build into the language and somewhat easy to use syntax for it.\n",
    "\n",
    "### Bad\n",
    "* Python was designed as a general purpose language.  The idea of a DataFrame came later, but the syntax isn't as consistent\n",
    "    * numpy - homogenuous matrix, not columns - can't(always?) be used for ML\n",
    "    * Pandas - hacked on top of Python data structure\n",
    "    * others? (reinventing the wheel!)\n",
    " \n",
    " Python and R have similar constructs in constructing a DataFrame, about the same in ease of use.\n",
    " There are MANY different ways to build a DataFrame! Check the documentation\n",
    "\n",
    "\n",
    "### H2O\n",
    "* I often use http://h20.ai for my Machine Learning, which has it's own DataFrame.  H2O has Python and R API's\n",
    "\n",
    "#### References\n",
    "https://medium.com/dunder-data/selecting-subsets-of-data-in-pandas-6fcd0170be9c\n",
    "\n",
    "### Usually load data from files\n",
    "* you probably won't do this much..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Create Data per Row\n",
    "data = [[24, 'Sue',115.2, True, 'NativeAmerican'], [45, 'Sam', 230.8, False, 'PacificIslander']]\n",
    "\n",
    "# Create Data per Column\n",
    "data = {'ID':[24, 45 ],\n",
    "        'Name':['Sue', 'Sam'],\n",
    "        'Weight':[115.2, 230.8],\n",
    "        'Smokes':[True, False], \n",
    "        'Ethnicity':pd.Series(['NativeAmerican', 'PacificIslander'], dtype=\"category\")}\n",
    "df = pd.DataFrame(data, columns=['ID', 'Name','Weight', 'Smokes', 'Ethnicity'])\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "df <- data.frame(\n",
    "    # insert by columns, with type of column implied by data\n",
    "    ID = c(24, 45),\n",
    "    Name = as.character(c('Sue', 'Sam')),\n",
    "    Weight = c(115.2, 230.8),\n",
    "    Smokes = c(TRUE, FALSE),\n",
    "    Ethnicity = as.factor(c('NativeAmerican', 'PacificIslander'))\n",
    "     , stringsAsFactors=FALSE  # Default is default.stringsAsFactors() which is usually TRUE\n",
    ")\n",
    "df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: You can give rows symbolic names, like \"First\", but this is something I never do in practice, so enough said."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Did we create what we think we did?\n",
    "\n",
    "Here's how to check the types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "str(df)  # Confusing name!  str = structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "# shocking ugliness in R: \"mode\" has a meaning in statistics, but R ignores that!\n",
    "mode(df)\n",
    "# In R, a DF is implemented as a list of vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "typeof(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "class(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check Column Names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(df)  # not obvious!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "colnames(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Count of Rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.shape)\n",
    "len(df)  # df.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%R\n",
    "print(length(df))  # EASY MISTAKE TO MAKE IN R.  This count of columns!\n",
    "print(dim(df))  # rows, columns\n",
    "nrow(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()  # ignores non-numeric columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Ethnicity\"].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "summary(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "table(df$Ethnicity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Head\n",
    "\n",
    "Look at the first few rows to see if they look OK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "head(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing Data from DataFrames\n",
    "\n",
    "### Good\n",
    "* R is matrix-like notation\n",
    "\n",
    "### Bad\n",
    "* Python's Panda feels arbitrary.  Consult the documentation!  Document your code!\n",
    "\n",
    "### Ugly\n",
    "* Behavior of Data Frames will surprise you in both languages.  It pays to double check to make sure you know what you are actually getting!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(df[\"Name\"]))  # A Column is actually implemented as a \"Panda Series\" Object!\n",
    "df[\"Name\"]  # SINGLE square bracket returns a PANDA SERIES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print( type(df[[\"Name\"]]) )\n",
    "df[[\"Name\"]] # DOUBLE square bracket returns a DATAFRAME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "print( class(df[\"Name\"]) )\n",
    "            (df[\"Name\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "print( class(df[, \"Name\"]) )\n",
    "            (df[, \"Name\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Panda Accessing Columns by an INDEX: `iloc()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Panda Accessing Columns by a NAME: `loc()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[:, \"Name\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## R uses the same matrix-like syntax for both"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "df[, 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "df[, \"Name\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing by Row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "df[1, ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing by Row and Column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[0][\"Smokes\"]  # Just a Boolean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "df[1, \"Smokes\"]  # Not a DataFrame!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[0][[\"Smokes\"]]  # DataFrame of one cell!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "df[c(1), c(\"Smokes\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[0:2][[\"Name\", \"Smokes\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "df[c(1:2), c(\"Name\", \"Smokes\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bool_vector = df[\"Weight\"] < 200\n",
    "bool_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[bool_vector]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%R\n",
    "bool_vector <- (df[\"Weight\"] < 200)\n",
    "print(class(bool_vector))\n",
    "(bool_vector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "print( class(df[bool_vector, ]))\n",
    "             df[bool_vector, ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "print( class(df[bool_vector ]))\n",
    "             df[bool_vector ]  # Probably NOT what you want: mix types get coerced into strings!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "# Better way is to use \"which\"\n",
    "bool_vector <- which(df[\"Weight\"] < 200)\n",
    "print(class(bool_vector))\n",
    "(bool_vector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%R\n",
    "print( class(df[bool_vector, ]))\n",
    "             df[bool_vector, ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "* We've gone over a lot of Language Nits in Python vs. R\n",
    "\n",
    "Bottom line:\n",
    "* **CHECK YOUR CODE CAREFULLY** especially if you're going back and forth between the two languages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphing\n",
    "(Make a separate talk some day....)\n",
    "\n",
    "### Good\n",
    "* R recognized immediately the importance of the Graphics support\n",
    "* Python added this later\n",
    "\n",
    "### Ugly\n",
    "* R has three different plotting packages\n",
    "* Python has multiple ones as well\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}