SubsurfaceDataAnalytics_DataFrame.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "\n",
    "<p align=\"center\">\n",
    "    <img src=\"https://github.com/GeostatsGuy/GeostatsPy/blob/master/TCG_color_logo.png?raw=true\" width=\"220\" height=\"240\" />\n",
    "\n",
    "</p>\n",
    "\n",
    "## Subsurface Data Analytics \n",
    "\n",
    "### Tabular Data Structures / DataFrames in Python \n",
    "\n",
    "#### Michael Pyrcz, Associate Professor, University of Texas at Austin \n",
    "\n",
    "##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)\n",
    "\n",
    "### Exercise: Tabular Data Structures / DataFrames in Python \n",
    "\n",
    "This is a tutorial for / demonstration of **Tabular Data Structures in Python**.  In Python, the common tool for dealing with Tabular Data Structures is the DataFrame from the pandas Python package. \n",
    "\n",
    "* Tabular Data in subsurface data analytics includes any data set with a limited number of samples as oposed to gridded maps that provide exhaustively sampled data.\n",
    "\n",
    "This tutorial includes the methods and operations that would commonly be required for and Geoscientists, Engineers and Data Scientists working with Tabular Data Structures for the purpose of:\n",
    "\n",
    "1. Data Checking and Cleaning\n",
    "2. Data Mining / Inferential Data Analysis\n",
    "3. Data Analytics / Building Predictive Models with Geostatistics and Machine Learning\n",
    "\n",
    "Learning to work with Pandas DataFrames is essential for dealing with tabular data (e.g. well data) in subsurface modeling workflows and for subsurface machine learning.\n",
    "\n",
    "##### Tabular Data Structures\n",
    "\n",
    "In Python we will commonly store our data in two formats, tables and arrays.  For sampled data with typically multiple features $1,\\ldots,m$ over $1,\\ldots,n$ samples we will work with tables.  For exhaustive maps and models usually representing a single feature on a regular grid over $1,\\ldots,n_{i}$ for $i = 1,\\ldots,n_{dim}$ we will work with arrays.\n",
    "\n",
    "pandas package provides a convenient DataFrame object for working with data in a table and numpy package provides a convenient ndarray object for working with gridded data. In the following tutorial we will focus on DataFrames although we will utilize ndarrays a couple of times.  There is another section on Gridded Data Structures that focuses on ndarrays.\n",
    "\n",
    "#### Additional Resources\n",
    "\n",
    "These workflows are based on standard methods with their associated limitations and assumptions. For more information see:\n",
    "\n",
    "* [pandas DataFrames Lecture](https://www.youtube.com/watch?v=cggieFcKdiM&list=PLG19vXLQHvSDUmEOmBoaxGbFAbvaLdfx4&index=8&t=0s)\n",
    "\n",
    "I have provided various workflows for subsurface data analytics, geostatistics and machine learning:\n",
    "\n",
    "* [Python](https://git.io/fh4eX)\n",
    "\n",
    "* [Excel](https://github.com/GeostatsGuy/LectureExercises/blob/master/Lecture7_CI_Hypoth_eg_R.xlsx) \n",
    "* [R](https://github.com/GeostatsGuy/LectureExercises/blob/master/Lecture7_CI_Hypoth_eg.R)  \n",
    "\n",
    "and all of my University of Texas at Austin \n",
    "\n",
    "* [Lectures](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig/featured?view_as=subscriber)\n",
    "\n",
    "#### Workflow Goals\n",
    "\n",
    "Learn the basics for working with Tabular Data Structures in Python. This includes:\n",
    "\n",
    "* Loading tabular data\n",
    "* Visualizing tabular data\n",
    "* Data QC and Cleaning\n",
    "* Interacting with the tabular data\n",
    "\n",
    "#### Objective \n",
    "\n",
    "I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows. \n",
    "\n",
    "The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.    \n",
    "\n",
    "#### Getting Started\n",
    "\n",
    "Here's the steps to get setup in Python with the GeostatsPy package:\n",
    "\n",
    "1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/). \n",
    "2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal. \n",
    "3. In the terminal type: pip install geostatspy. \n",
    "4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality. \n",
    "\n",
    "You will need to copy the data file to your working directory.  They are available here:\n",
    "\n",
    "* Tabular data - 2D_MV_200wells.csv at [here].(https://github.com/GeostatsGuy/GeoDataSets/blob/master/2D_MV_200wells.csv)\n",
    "\n",
    "I have put together various subsurface workflows for data analytics, geostatistics and machine learning. Go [here](https://git.io/fh4eX) for other example workflows and source code. \n",
    "\n",
    "#### Load the required libraries\n",
    "\n",
    "The following code loads the required libraries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os                                                   # to set current working directory \n",
    "import numpy as np                                          # arrays and matrix math\n",
    "import pandas as pd                                         # DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing `python -m pip install [package-name]`. More assistance is available with the respective package docs.  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set the working directory\n",
    "\n",
    "I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see below) data file in this directory.  When we are finished with this tutorial we will write our new dataset back to this directory.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(\"c:/PGE383\")                                       # set the working directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading Data \n",
    "\n",
    "Let's load the provided multivariate, spatial dataset.  '2D_MV_200wells.csv' is available [here](https://github.com/GeostatsGuy/GeoDataSets/blob/master/2D_MV_200wells.csv).  It is a comma delimited file with: \n",
    "\n",
    "* X and Y coordinates ($m$)\n",
    "* facies 1 and 2 (1 is sandstone and 2 interbedded sand and mudstone)\n",
    "* porosity (fraction)\n",
    "* permeability ($mD$)\n",
    "* acoustic impedance ($\\frac{kg}{m^3} \\cdot \\frac{m}{s} \\cdot 10^6$). \n",
    "\n",
    "We load it with the pandas 'read_csv' function into a data frame we called 'df' and then preview it by printing a slice and by utilizing the 'head' DataFrame member function (with a nice and clean format, see below).\n",
    "\n",
    "**Python Tip: using functions from a package** just type the label for the package that we declared at the beginning:\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "```\n",
    "\n",
    "so we can access the pandas function 'read_csv' with the command: \n",
    "\n",
    "```python\n",
    "pd.read_csv()\n",
    "```\n",
    "\n",
    "but read csv has required input parameters. The essential one is the name of the file. For our circumstance all the other default parameters are fine. If you want to see all the possible parameters for this function, just go to the docs [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html).  \n",
    "\n",
    "* The docs are always helpful\n",
    "* There is often a lot of flexibility for Python functions, possible through using various inputs parameters\n",
    "\n",
    "also, the program has an output, a pandas DataFrame loaded from the data.  So we have to specficy the name / variable representing that new object.\n",
    "\n",
    "```python\n",
    "df = pd.read_csv(\"2D_MV_200wells.csv\")  \n",
    "```\n",
    "\n",
    "Let's run this command to load the data and then look at the resulting DataFrame to ensure that we loaded it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>facies_threshold_0.3</th>\n",
       "      <th>porosity</th>\n",
       "      <th>permeability</th>\n",
       "      <th>acoustic_impedance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>565</td>\n",
       "      <td>1485</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1184</td>\n",
       "      <td>6.170</td>\n",
       "      <td>2.009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2585</td>\n",
       "      <td>1185</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1566</td>\n",
       "      <td>6.275</td>\n",
       "      <td>2.864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2065</td>\n",
       "      <td>2865</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1920</td>\n",
       "      <td>92.297</td>\n",
       "      <td>3.524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3575</td>\n",
       "      <td>2655</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1621</td>\n",
       "      <td>9.048</td>\n",
       "      <td>2.157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1835</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1766</td>\n",
       "      <td>7.123</td>\n",
       "      <td>3.979</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      X     Y  facies_threshold_0.3  porosity  permeability  \\\n",
       "0   565  1485                     1    0.1184         6.170   \n",
       "1  2585  1185                     1    0.1566         6.275   \n",
       "2  2065  2865                     2    0.1920        92.297   \n",
       "3  3575  2655                     1    0.1621         9.048   \n",
       "4  1835    35                     1    0.1766         7.123   \n",
       "\n",
       "   acoustic_impedance  \n",
       "0               2.009  \n",
       "1               2.864  \n",
       "2               3.524  \n",
       "3               2.157  \n",
       "4               3.979  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"2D_MV_200wells.csv\")                      # read a .csv file in as a DataFrame\n",
    "#print(df.iloc[0:5,:])                                      # display first 4 samples in the table as a preview\n",
    "df.head()                                                   # we could also use this command for a table preview "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Summary Statistics\n",
    "\n",
    "It is useful to review the summary statistics of our loaded DataFrame.  That can be accomplished with the 'describe' DataFrame member function.  We transpose to switch the axes for ease of visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>facies_threshold_0.3</th>\n",
       "      <th>porosity</th>\n",
       "      <th>permeability</th>\n",
       "      <th>acoustic_impedance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>200.000000</td>\n",
       "      <td>200.00000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>200.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2053.400000</td>\n",
       "      <td>1876.15000</td>\n",
       "      <td>1.330000</td>\n",
       "      <td>0.149300</td>\n",
       "      <td>25.287462</td>\n",
       "      <td>3.000435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1113.524641</td>\n",
       "      <td>1137.58016</td>\n",
       "      <td>0.471393</td>\n",
       "      <td>0.032948</td>\n",
       "      <td>64.470135</td>\n",
       "      <td>0.592201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>25.000000</td>\n",
       "      <td>35.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.050000</td>\n",
       "      <td>0.015820</td>\n",
       "      <td>2.009000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1112.500000</td>\n",
       "      <td>920.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.132175</td>\n",
       "      <td>1.366750</td>\n",
       "      <td>2.483250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>2160.000000</td>\n",
       "      <td>1855.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.150150</td>\n",
       "      <td>4.825500</td>\n",
       "      <td>2.964500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2915.000000</td>\n",
       "      <td>2782.50000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.174200</td>\n",
       "      <td>14.597000</td>\n",
       "      <td>3.527000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>3955.000000</td>\n",
       "      <td>3995.00000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.223200</td>\n",
       "      <td>463.641000</td>\n",
       "      <td>3.984000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 X           Y  facies_threshold_0.3    porosity  \\\n",
       "count   200.000000   200.00000            200.000000  200.000000   \n",
       "mean   2053.400000  1876.15000              1.330000    0.149300   \n",
       "std    1113.524641  1137.58016              0.471393    0.032948   \n",
       "min      25.000000    35.00000              1.000000    0.050000   \n",
       "25%    1112.500000   920.00000              1.000000    0.132175   \n",
       "50%    2160.000000  1855.00000              1.000000    0.150150   \n",
       "75%    2915.000000  2782.50000              2.000000    0.174200   \n",
       "max    3955.000000  3995.00000              2.000000    0.223200   \n",
       "\n",
       "       permeability  acoustic_impedance  \n",
       "count    200.000000          200.000000  \n",
       "mean      25.287462            3.000435  \n",
       "std       64.470135            0.592201  \n",
       "min        0.015820            2.009000  \n",
       "25%        1.366750            2.483250  \n",
       "50%        4.825500            2.964500  \n",
       "75%       14.597000            3.527000  \n",
       "max      463.641000            3.984000  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Rename a Variable / Features\n",
    "\n",
    "Let's rename the facies, permeability and acoustic impedance for convenience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>facies</th>\n",
       "      <th>porosity</th>\n",
       "      <th>perm</th>\n",
       "      <th>ai</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>565</td>\n",
       "      <td>1485</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1184</td>\n",
       "      <td>6.170</td>\n",
       "      <td>2.009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2585</td>\n",
       "      <td>1185</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1566</td>\n",
       "      <td>6.275</td>\n",
       "      <td>2.864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2065</td>\n",
       "      <td>2865</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1920</td>\n",
       "      <td>92.297</td>\n",
       "      <td>3.524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3575</td>\n",
       "      <td>2655</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1621</td>\n",
       "      <td>9.048</td>\n",
       "      <td>2.157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1835</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1766</td>\n",
       "      <td>7.123</td>\n",
       "      <td>3.979</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      X     Y  facies  porosity    perm     ai\n",
       "0   565  1485       1    0.1184   6.170  2.009\n",
       "1  2585  1185       1    0.1566   6.275  2.864\n",
       "2  2065  2865       2    0.1920  92.297  3.524\n",
       "3  3575  2655       1    0.1621   9.048  2.157\n",
       "4  1835    35       1    0.1766   7.123  3.979"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.rename(columns={'facies_threshold_0.3': 'facies','permeability':'perm','acoustic_impedance':'ai'}) # rename columns of the \n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Slicing and Subsets\n",
    "\n",
    "It is straightforward to extract subsets from a DataFrame to make a new DataFrame.  \n",
    "\n",
    "* This is useful for cleaning up data by removing features that are no longer of interest.  \n",
    "\n",
    "If the samples are in random order then the first $n_{s}$ samples are a random sample of size $n_{s}$.  Below we make a new DataFrame, 'df_subset', with the rows 0 to 4 and columns 2 to 6 and the **X and Y coordinates removed**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   facies  porosity    perm     ai\n",
      "0       1    0.1184   6.170  2.009\n",
      "1       1    0.1566   6.275  2.864\n",
      "2       2    0.1920  92.297  3.524\n",
      "3       1    0.1621   9.048  2.157\n",
      "4       1    0.1766   7.123  3.979\n"
     ]
    }
   ],
   "source": [
    "df_subset = df.iloc[0:5,2:7]                                # make a new dataframe with just the first 4 samples and no X,Y\n",
    "print(df_subset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's demonstrate some more complicated slicing options.  We demonstrate two methods:\n",
    "\n",
    "* list the exact indexes that you want\n",
    "\n",
    "```python\n",
    "df_subset2 = df.iloc[[0,2,4,5,10,43],:]      # extract rows 0,2,4,5...,43 for all columns\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       X     Y  facies  porosity    perm     ai\n",
      "0    565  1485       1    0.1184   6.170  2.009\n",
      "2   2065  2865       2    0.1920  92.297  3.524\n",
      "4   1835    35       1    0.1766   7.123  3.979\n",
      "5   3375  2525       1    0.1239   1.468  2.337\n",
      "10  2125  1105       1    0.1369   3.693  3.627\n",
      "43  1785  1045       1    0.1517   2.766  3.562\n",
      "     facies       perm     ai\n",
      "2         2   92.29700  3.524\n",
      "3         1    9.04800  2.157\n",
      "4         1    7.12300  3.979\n",
      "5         1    1.46800  2.337\n",
      "6         1   31.93300  3.491\n",
      "7         2  116.78100  2.187\n",
      "8         1    3.00300  2.048\n",
      "9         1    5.21300  2.251\n",
      "10        1    3.69300  3.627\n",
      "11        1    0.26270  2.860\n",
      "12        1    9.91400  2.742\n",
      "13        1   14.31100  3.045\n",
      "14        1    0.77310  2.323\n",
      "15        2   22.57800  2.711\n",
      "16        2   18.74300  2.583\n",
      "17        1    9.09200  3.801\n",
      "18        2    0.46420  3.771\n",
      "19        1  146.31900  2.341\n",
      "20        1    0.06073  3.872\n",
      "21        1   31.18100  2.316\n",
      "22        1    1.43200  3.081\n",
      "23        1   76.29800  2.175\n",
      "24        1  124.22200  2.451\n",
      "25        1    3.52300  3.173\n",
      "26        2  201.36300  2.877\n",
      "27        1    3.25400  2.484\n",
      "28        1   11.28300  3.697\n",
      "29        2    3.22700  3.801\n",
      "30        1    1.50400  2.221\n",
      "31        1    0.08758  2.533\n",
      "..      ...        ...    ...\n",
      "170       1    7.77100  3.807\n",
      "171       1   30.25800  2.100\n",
      "172       1    4.78400  3.410\n",
      "173       1    2.40300  3.121\n",
      "174       1   13.41600  3.530\n",
      "175       1    2.64800  2.954\n",
      "176       2   61.92900  3.170\n",
      "177       2    0.45250  2.162\n",
      "178       1    5.69200  3.913\n",
      "179       1    3.72200  3.501\n",
      "180       2   55.67800  3.881\n",
      "181       2    4.41500  3.065\n",
      "182       2   14.83700  3.394\n",
      "183       1    9.38200  2.434\n",
      "184       1   29.16500  2.267\n",
      "185       1    0.34410  3.086\n",
      "186       1    3.83000  2.112\n",
      "187       1    0.79830  3.135\n",
      "188       2   14.71400  2.192\n",
      "189       1    2.73000  3.692\n",
      "190       2   88.45900  2.416\n",
      "191       1    1.03800  3.574\n",
      "192       1    6.38600  2.072\n",
      "193       1    0.69850  3.933\n",
      "194       2   23.88300  2.625\n",
      "195       1    9.54200  2.945\n",
      "196       1    6.41200  2.298\n",
      "197       1    2.40400  2.087\n",
      "198       1    0.01582  3.907\n",
      "199       1    0.99000  2.992\n",
      "\n",
      "[198 rows x 3 columns]\n"
     ]
    }
   ],
   "source": [
    "df_subset2 = df.iloc[[0,2,4,5,10,43],:]       # new dataframe with samples 0, 2 ,...,43 and all features\n",
    "print(df_subset2)\n",
    "\n",
    "df_subset3 = df.iloc[2:,[2,4,5]]              # new dataframe with all samples from 2 and features 2,4,5\n",
    "print(df_subset3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adding a Variable / Features\n",
    "\n",
    "It is also easy to add a column to our data frame.  \n",
    "\n",
    "* Note, we assume that the array is in the same order as the DataFrame.  This could be an issue if any rows were removed form either before adding etc.  \n",
    "\n",
    "To demonstrate we make a 1D numpy array of zeros using the 'zeros' function and add it to our DataFrame with the feature name indicated as 'zero'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>facies</th>\n",
       "      <th>porosity</th>\n",
       "      <th>perm</th>\n",
       "      <th>ai</th>\n",
       "      <th>zero</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>565</td>\n",
       "      <td>1485</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1184</td>\n",
       "      <td>6.170</td>\n",
       "      <td>2.009</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2585</td>\n",
       "      <td>1185</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1566</td>\n",
       "      <td>6.275</td>\n",
       "      <td>2.864</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2065</td>\n",
       "      <td>2865</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1920</td>\n",
       "      <td>92.297</td>\n",
       "      <td>3.524</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3575</td>\n",
       "      <td>2655</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1621</td>\n",
       "      <td>9.048</td>\n",
       "      <td>2.157</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1835</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1766</td>\n",
       "      <td>7.123</td>\n",
       "      <td>3.979</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      X     Y  facies  porosity    perm     ai  zero\n",
       "0   565  1485       1    0.1184   6.170  2.009   0.0\n",
       "1  2585  1185       1    0.1566   6.275  2.864   0.0\n",
       "2  2065  2865       2    0.1920  92.297  3.524   0.0\n",
       "3  3575  2655       1    0.1621   9.048  2.157   0.0\n",
       "4  1835    35       1    0.1766   7.123  3.979   0.0"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zeros = np.zeros(200)                                       # make a array of zeros\n",
    "df['zero'] = pd.Series(zeros)                               # add the array to our DataFrame\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also remove unwanted columns without having to subset the DataFrame.  \n",
    "\n",
    "* That's why we just added a column of zeros, I wanted to also demonstrated removing a column.\n",
    "\n",
    "We do this with the 'drop' member function of the DataFrame object. We just have the give the column name and by indicating axis=1 we specify to drop a column instead of a row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>facies</th>\n",
       "      <th>porosity</th>\n",
       "      <th>perm</th>\n",
       "      <th>ai</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>565</td>\n",
       "      <td>1485</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1184</td>\n",
       "      <td>6.170</td>\n",
       "      <td>2.009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2585</td>\n",
       "      <td>1185</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1566</td>\n",
       "      <td>6.275</td>\n",
       "      <td>2.864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2065</td>\n",
       "      <td>2865</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1920</td>\n",
       "      <td>92.297</td>\n",
       "      <td>3.524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3575</td>\n",
       "      <td>2655</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1621</td>\n",
       "      <td>9.048</td>\n",
       "      <td>2.157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1835</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1766</td>\n",
       "      <td>7.123</td>\n",
       "      <td>3.979</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      X     Y  facies  porosity    perm     ai\n",
       "0   565  1485       1    0.1184   6.170  2.009\n",
       "1  2585  1185       1    0.1566   6.275  2.864\n",
       "2  2065  2865       2    0.1920  92.297  3.524\n",
       "3  3575  2655       1    0.1621   9.048  2.157\n",
       "4  1835    35       1    0.1766   7.123  3.979"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.drop('zero',axis=1)                                      # remove the zero column\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Standardizing / Manipulating Variables / Features\n",
    "\n",
    "We may want to make new features by using mathematical operators applied to existing features.\n",
    "\n",
    "* We can use any combinations of features, constants, math and even other data tables\n",
    "\n",
    "For example, we can make a porosity feature that is in percentage instead of fraction (called 'porosity100') or a ratio of permeability divided by porosity (called 'permpor') may be useful for subsequent calculations such as the Lorenz Coefficient.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>facies</th>\n",
       "      <th>porosity</th>\n",
       "      <th>perm</th>\n",
       "      <th>ai</th>\n",
       "      <th>porosity100</th>\n",
       "      <th>permpor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>565</td>\n",
       "      <td>1485</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1184</td>\n",
       "      <td>6.170</td>\n",
       "      <td>2.009</td>\n",
       "      <td>11.84</td>\n",
       "      <td>52.111486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2585</td>\n",
       "      <td>1185</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1566</td>\n",
       "      <td>6.275</td>\n",
       "      <td>2.864</td>\n",
       "      <td>15.66</td>\n",
       "      <td>40.070243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2065</td>\n",
       "      <td>2865</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1920</td>\n",
       "      <td>92.297</td>\n",
       "      <td>3.524</td>\n",
       "      <td>19.20</td>\n",
       "      <td>480.713542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3575</td>\n",
       "      <td>2655</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1621</td>\n",
       "      <td>9.048</td>\n",
       "      <td>2.157</td>\n",
       "      <td>16.21</td>\n",
       "      <td>55.817397</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1835</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1766</td>\n",
       "      <td>7.123</td>\n",
       "      <td>3.979</td>\n",
       "      <td>17.66</td>\n",
       "      <td>40.334088</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      X     Y  facies  porosity    perm     ai  porosity100     permpor\n",
       "0   565  1485       1    0.1184   6.170  2.009        11.84   52.111486\n",
       "1  2585  1185       1    0.1566   6.275  2.864        15.66   40.070243\n",
       "2  2065  2865       2    0.1920  92.297  3.524        19.20  480.713542\n",
       "3  3575  2655       1    0.1621   9.048  2.157        16.21   55.817397\n",
       "4  1835    35       1    0.1766   7.123  3.979        17.66   40.334088"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['porosity100'] = df['porosity']*100                      # add a new column with porosity in percentage\n",
    "df['permpor'] = df['perm']/df['porosity']           # add a new feature with ratio of perm / por \n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Truncating and Categorizing Variables / Features\n",
    "\n",
    "We could also use conditional statements when assigning values to a new feature.  \n",
    "\n",
    "* We could use any condition with any combination of features and variables from any sourc\n",
    "\n",
    "For example, we could have a categorical porosity measure for high and low porosity, called 'tporosity' for truncated porosity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>facies</th>\n",
       "      <th>porosity</th>\n",
       "      <th>perm</th>\n",
       "      <th>ai</th>\n",
       "      <th>porosity100</th>\n",
       "      <th>permpor</th>\n",
       "      <th>tporosity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>565</td>\n",
       "      <td>1485</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1184</td>\n",
       "      <td>6.170</td>\n",
       "      <td>2.009</td>\n",
       "      <td>11.84</td>\n",
       "      <td>52.111486</td>\n",
       "      <td>low</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2585</td>\n",
       "      <td>1185</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1566</td>\n",
       "      <td>6.275</td>\n",
       "      <td>2.864</td>\n",
       "      <td>15.66</td>\n",
       "      <td>40.070243</td>\n",
       "      <td>high</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2065</td>\n",
       "      <td>2865</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1920</td>\n",
       "      <td>92.297</td>\n",
       "      <td>3.524</td>\n",
       "      <td>19.20</td>\n",
       "      <td>480.713542</td>\n",
       "      <td>high</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3575</td>\n",
       "      <td>2655</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1621</td>\n",
       "      <td>9.048</td>\n",
       "      <td>2.157</td>\n",
       "      <td>16.21</td>\n",
       "      <td>55.817397</td>\n",
       "      <td>high</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1835</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1766</td>\n",
       "      <td>7.123</td>\n",
       "      <td>3.979</td>\n",
       "      <td>17.66</td>\n",
       "      <td>40.334088</td>\n",
       "      <td>high</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      X     Y  facies  porosity    perm     ai  porosity100     permpor  \\\n",
       "0   565  1485       1    0.1184   6.170  2.009        11.84   52.111486   \n",
       "1  2585  1185       1    0.1566   6.275  2.864        15.66   40.070243   \n",
       "2  2065  2865       2    0.1920  92.297  3.524        19.20  480.713542   \n",
       "3  3575  2655       1    0.1621   9.048  2.157        16.21   55.817397   \n",
       "4  1835    35       1    0.1766   7.123  3.979        17.66   40.334088   \n",
       "\n",
       "  tporosity  \n",
       "0       low  \n",
       "1      high  \n",
       "2      high  \n",
       "3      high  \n",
       "4      high  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['tporosity'] = np.where(df['porosity']>=0.12, 'high', 'low') # conditional statement assign a new feature\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Truncating to Remove Nonphysical Values\n",
    "\n",
    "Here's an example where we use a conditional statement to assign a very low permeability value (0.0001 mD) for all porosity values below a threshold. Of course, this is for demonstration, in practice a much lower porosity threshold would likely be applied.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>facies</th>\n",
       "      <th>porosity</th>\n",
       "      <th>perm</th>\n",
       "      <th>ai</th>\n",
       "      <th>porosity100</th>\n",
       "      <th>permpor</th>\n",
       "      <th>tporosity</th>\n",
       "      <th>perm_cutoff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>565</td>\n",
       "      <td>1485</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1184</td>\n",
       "      <td>6.170</td>\n",
       "      <td>2.009</td>\n",
       "      <td>11.84</td>\n",
       "      <td>52.111486</td>\n",
       "      <td>low</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2585</td>\n",
       "      <td>1185</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1566</td>\n",
       "      <td>6.275</td>\n",
       "      <td>2.864</td>\n",
       "      <td>15.66</td>\n",
       "      <td>40.070243</td>\n",
       "      <td>high</td>\n",
       "      <td>6.2750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2065</td>\n",
       "      <td>2865</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1920</td>\n",
       "      <td>92.297</td>\n",
       "      <td>3.524</td>\n",
       "      <td>19.20</td>\n",
       "      <td>480.713542</td>\n",
       "      <td>high</td>\n",
       "      <td>92.2970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3575</td>\n",
       "      <td>2655</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1621</td>\n",
       "      <td>9.048</td>\n",
       "      <td>2.157</td>\n",
       "      <td>16.21</td>\n",
       "      <td>55.817397</td>\n",
       "      <td>high</td>\n",
       "      <td>9.0480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1835</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1766</td>\n",
       "      <td>7.123</td>\n",
       "      <td>3.979</td>\n",
       "      <td>17.66</td>\n",
       "      <td>40.334088</td>\n",
       "      <td>high</td>\n",
       "      <td>7.1230</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      X     Y  facies  porosity    perm     ai  porosity100     permpor  \\\n",
       "0   565  1485       1    0.1184   6.170  2.009        11.84   52.111486   \n",
       "1  2585  1185       1    0.1566   6.275  2.864        15.66   40.070243   \n",
       "2  2065  2865       2    0.1920  92.297  3.524        19.20  480.713542   \n",
       "3  3575  2655       1    0.1621   9.048  2.157        16.21   55.817397   \n",
       "4  1835    35       1    0.1766   7.123  3.979        17.66   40.334088   \n",
       "\n",
       "  tporosity  perm_cutoff  \n",
       "0       low       0.0001  \n",
       "1      high       6.2750  \n",
       "2      high      92.2970  \n",
       "3      high       9.0480  \n",
       "4      high       7.1230  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['perm_cutoff'] = np.where(df['porosity']>=0.12, df['perm'],0.0001) # conditional statement assign a new feature\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dealing with Missing Data\n",
    "\n",
    "What about missing or invalid values?  Let's assign a single porosity value to NaN, 'not a number', indicating a missing or eroneous value.  \n",
    "\n",
    "**Python Tip: manipulating DataFrames manually**\n",
    "\n",
    "We us this command to access anyone of the individual records in our data table.\n",
    "\n",
    "```p\n",
    "df.at[irow,'feature name'] \n",
    "```\n",
    "\n",
    "We can get the value or set the value to a new value:\n",
    "\n",
    "```p\n",
    "df.at[irow,'feature name'] = new_value\n",
    "```\n",
    "\n",
    "We will then check for the number of NaN values in our DataFrame.  Then we can search for and display the sample with the NaN porosity value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of null values in our DataFrame =  1\n",
      "      X     Y  facies  porosity   perm     ai  porosity100    permpor  \\\n",
      "1  2585  1185       1       NaN  6.275  2.864        15.66  40.070243   \n",
      "\n",
      "  tporosity  perm_cutoff  \n",
      "1      high        6.275  \n"
     ]
    }
   ],
   "source": [
    "df.at[1,'porosity'] = np.NaN # let's give ourselves a NaN / missing value in our table\n",
    "\n",
    "# Count the number of samples with missing values\n",
    "print('Number of null values in our DataFrame = ', str(df.isnull().sum().sum()))  \n",
    "# Find the row(s) with missing values and look at them\n",
    "nan_rows = df[df['porosity'].isnull()]                      # find the row with missing values\n",
    "print(nan_rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that sample 1 (see the '1' on the left hand side, that is the sample index in the table) has a NaN porosity value.  \n",
    "\n",
    "#### Drop Samples with Missing Values\n",
    "\n",
    "Now we may choose to remove the sample with the NaN.  The 'dropna' DataFrame member function will remove all samples with NaN entries from the entire DataFrame.  By visualizing the index at the left of the DataFrame preview you can confirm that sample 1 is removed (it jumps from 0 to 2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>facies</th>\n",
       "      <th>porosity</th>\n",
       "      <th>perm</th>\n",
       "      <th>ai</th>\n",
       "      <th>porosity100</th>\n",
       "      <th>permpor</th>\n",
       "      <th>tporosity</th>\n",
       "      <th>perm_cutoff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>565</td>\n",
       "      <td>1485</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1184</td>\n",
       "      <td>6.170</td>\n",
       "      <td>2.009</td>\n",
       "      <td>11.84</td>\n",
       "      <td>52.111486</td>\n",
       "      <td>low</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2065</td>\n",
       "      <td>2865</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1920</td>\n",
       "      <td>92.297</td>\n",
       "      <td>3.524</td>\n",
       "      <td>19.20</td>\n",
       "      <td>480.713542</td>\n",
       "      <td>high</td>\n",
       "      <td>92.2970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3575</td>\n",
       "      <td>2655</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1621</td>\n",
       "      <td>9.048</td>\n",
       "      <td>2.157</td>\n",
       "      <td>16.21</td>\n",
       "      <td>55.817397</td>\n",
       "      <td>high</td>\n",
       "      <td>9.0480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1835</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1766</td>\n",
       "      <td>7.123</td>\n",
       "      <td>3.979</td>\n",
       "      <td>17.66</td>\n",
       "      <td>40.334088</td>\n",
       "      <td>high</td>\n",
       "      <td>7.1230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>3375</td>\n",
       "      <td>2525</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1239</td>\n",
       "      <td>1.468</td>\n",
       "      <td>2.337</td>\n",
       "      <td>12.39</td>\n",
       "      <td>11.848265</td>\n",
       "      <td>high</td>\n",
       "      <td>1.4680</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      X     Y  facies  porosity    perm     ai  porosity100     permpor  \\\n",
       "0   565  1485       1    0.1184   6.170  2.009        11.84   52.111486   \n",
       "2  2065  2865       2    0.1920  92.297  3.524        19.20  480.713542   \n",
       "3  3575  2655       1    0.1621   9.048  2.157        16.21   55.817397   \n",
       "4  1835    35       1    0.1766   7.123  3.979        17.66   40.334088   \n",
       "5  3375  2525       1    0.1239   1.468  2.337        12.39   11.848265   \n",
       "\n",
       "  tporosity  perm_cutoff  \n",
       "0       low       0.0001  \n",
       "2      high      92.2970  \n",
       "3      high       9.0480  \n",
       "4      high       7.1230  \n",
       "5      high       1.4680  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.dropna()                                            # drop any rows (samples) with atleast one missing value        \n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Searching with Tabular Data\n",
    "\n",
    "One could extract samples into a new DataFrame with multiple criteria.  This is shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>facies</th>\n",
       "      <th>porosity</th>\n",
       "      <th>perm</th>\n",
       "      <th>ai</th>\n",
       "      <th>porosity100</th>\n",
       "      <th>permpor</th>\n",
       "      <th>tporosity</th>\n",
       "      <th>perm_cutoff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2065</td>\n",
       "      <td>2865</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1920</td>\n",
       "      <td>92.297</td>\n",
       "      <td>3.524</td>\n",
       "      <td>19.20</td>\n",
       "      <td>480.713542</td>\n",
       "      <td>high</td>\n",
       "      <td>92.297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2295</td>\n",
       "      <td>1325</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1790</td>\n",
       "      <td>31.933</td>\n",
       "      <td>3.491</td>\n",
       "      <td>17.90</td>\n",
       "      <td>178.396648</td>\n",
       "      <td>high</td>\n",
       "      <td>31.933</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3715</td>\n",
       "      <td>3045</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1914</td>\n",
       "      <td>116.781</td>\n",
       "      <td>2.187</td>\n",
       "      <td>19.14</td>\n",
       "      <td>610.141066</td>\n",
       "      <td>high</td>\n",
       "      <td>116.781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>545</td>\n",
       "      <td>3765</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1817</td>\n",
       "      <td>14.311</td>\n",
       "      <td>3.045</td>\n",
       "      <td>18.17</td>\n",
       "      <td>78.761695</td>\n",
       "      <td>high</td>\n",
       "      <td>14.311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1385</td>\n",
       "      <td>2415</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1774</td>\n",
       "      <td>22.578</td>\n",
       "      <td>2.711</td>\n",
       "      <td>17.74</td>\n",
       "      <td>127.271702</td>\n",
       "      <td>high</td>\n",
       "      <td>22.578</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       X     Y  facies  porosity     perm     ai  porosity100     permpor  \\\n",
       "2   2065  2865       2    0.1920   92.297  3.524        19.20  480.713542   \n",
       "6   2295  1325       1    0.1790   31.933  3.491        17.90  178.396648   \n",
       "7   3715  3045       2    0.1914  116.781  2.187        19.14  610.141066   \n",
       "13   545  3765       1    0.1817   14.311  3.045        18.17   78.761695   \n",
       "15  1385  2415       2    0.1774   22.578  2.711        17.74  127.271702   \n",
       "\n",
       "   tporosity  perm_cutoff  \n",
       "2       high       92.297  \n",
       "6       high       31.933  \n",
       "7       high      116.781  \n",
       "13      high       14.311  \n",
       "15      high       22.578  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_extract = df.loc[(df['porosity'] > 0.12) & (df['perm'] > 10.0)] # extract with multiple conditions to a new table\n",
    "df_extract.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Making Data Frames\n",
    "\n",
    "We already covered the idea of making a DataFrame by loading data from a file. \n",
    "\n",
    "It is also possible to build a brandnew DataFrame from a set of 1D arrays.  \n",
    "\n",
    "* Note, they must have the same size and be sorted consistently.  \n",
    "\n",
    "We will extract 'porosity' and 'perm' features as arrays.\n",
    "\n",
    "**Python Tip: extracting data from DataFrames**\n",
    "\n",
    "We can extract the data for a single feature with this command:\n",
    "\n",
    "```python\n",
    "1D_series = df['feature_name']\n",
    "```\n",
    "\n",
    "The 'series' retains information about the feature that was included in the DataFrame including the name and the indexing. This is fine, but some methods don't work with series so we can also extract the data as a 1D ndarray.  By adding the '.values' the series is converted to a 1D array.\n",
    "\n",
    "```python\n",
    "1D_ndarray = df['feature_name'].values\n",
    "```\n",
    "\n",
    "We then use the pandas DataFrame command to make a new DataFrame with each 1D array and the column names specified as 'porosity' and 'permeabilty'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>porosity</th>\n",
       "      <th>permeability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.1184</td>\n",
       "      <td>6.170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.1920</td>\n",
       "      <td>92.297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.1621</td>\n",
       "      <td>9.048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.1766</td>\n",
       "      <td>7.123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.1239</td>\n",
       "      <td>1.468</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   porosity  permeability\n",
       "0    0.1184         6.170\n",
       "1    0.1920        92.297\n",
       "2    0.1621         9.048\n",
       "3    0.1766         7.123\n",
       "4    0.1239         1.468"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "por = df['porosity'].values                                 # extract porosity column as vector\n",
    "perm = df['perm'].values                                    # extract permeability column as vector\n",
    "df_new = pd.DataFrame({'porosity': por, 'permeability': perm}) # make a new DataFrame from the vectors\n",
    "df_new.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Information About Our Tabular Data\n",
    "\n",
    "We can reach in and retrieve the actual raw information in the DataFrame including the column names and actual values as an numpy array.  \n",
    "\n",
    "* We can't edit them like this, but we can access and use this information.  \n",
    "\n",
    "This includs:\n",
    "\n",
    "1. 'index' with information about the index (i.e. index from start to stop with step)\n",
    "2. 'columns' with the names of the features \n",
    "3. 'values' with the data table entries as a 2D array.  \n",
    "\n",
    "Let's look at these components of our DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Int64Index([  0,   2,   3,   4,   5,   6,   7,   8,   9,  10,\n",
      "            ...\n",
      "            190, 191, 192, 193, 194, 195, 196, 197, 198, 199],\n",
      "           dtype='int64', length=199)\n",
      "Index(['X', 'Y', 'facies', 'porosity', 'perm', 'ai', 'porosity100', 'permpor',\n",
      "       'tporosity', 'perm_cutoff'],\n",
      "      dtype='object')\n",
      "[[565 1485 1 ... 52.111486486486484 'low' 0.0001]\n",
      " [2065 2865 2 ... 480.71354166666674 'high' 92.29700000000001]\n",
      " [3575 2655 1 ... 55.81739666872301 'high' 9.048]\n",
      " ...\n",
      " [375 1705 1 ... 18.198334595003786 'high' 2.404]\n",
      " [3795 535 1 ... 0.25968483256730135 'low' 0.0001]\n",
      " [3455 1645 1 ... 6.578073089700997 'high' 0.99]]\n"
     ]
    }
   ],
   "source": [
    "print(df.index)                                             # get information about the index\n",
    "print(df.columns)                                           # get the list of feature names\n",
    "print(df.values)                                            # get the 2D array with all the table data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a method for getting a list of the DataFrame feature names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['X',\n",
       " 'Y',\n",
       " 'facies',\n",
       " 'porosity',\n",
       " 'perm',\n",
       " 'ai',\n",
       " 'porosity100',\n",
       " 'permpor',\n",
       " 'tporosity',\n",
       " 'perm_cutoff']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(df)                                                    # get a list with the feature names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More Precise Information from Tabular Data\n",
    "\n",
    "Let's interact with the DataFrame more surgically, one feature and sample at a time.  Here we retrieve the 4th column feature name and the porosity value for sample \\#1.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "porosity\n",
      "Porosity value for sample number 1 is 0.192.\n"
     ]
    }
   ],
   "source": [
    "col2_name = df.columns[3]                                   # get the name of the 4th feature (porosity)\n",
    "print(col2_name)                                          \n",
    "por1 = df.values[1,3]                                       # get the value for sample 1 of the 4th feature (porosity)\n",
    "print('Porosity value for sample number 1 is ' + str(por1) + '.') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also manually change values.  \n",
    "\n",
    "* We can use the 'at' pandas DataFrame member function to get and set manually individual records.  \n",
    "\n",
    "We look up the porosity value for sample 1 and then we use the 'at' again DataFrame member function to change the value to 0.1000.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The value of porosity for sample 2 is 0.1.\n",
      "The value of porosity for sample 2 is now 0.1000.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "      <th>facies</th>\n",
       "      <th>porosity</th>\n",
       "      <th>perm</th>\n",
       "      <th>ai</th>\n",
       "      <th>porosity100</th>\n",
       "      <th>permpor</th>\n",
       "      <th>tporosity</th>\n",
       "      <th>perm_cutoff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>565</td>\n",
       "      <td>1485</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1184</td>\n",
       "      <td>6.170</td>\n",
       "      <td>2.009</td>\n",
       "      <td>11.84</td>\n",
       "      <td>52.111486</td>\n",
       "      <td>low</td>\n",
       "      <td>0.0001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2065</td>\n",
       "      <td>2865</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1000</td>\n",
       "      <td>92.297</td>\n",
       "      <td>3.524</td>\n",
       "      <td>19.20</td>\n",
       "      <td>480.713542</td>\n",
       "      <td>high</td>\n",
       "      <td>92.2970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3575</td>\n",
       "      <td>2655</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1621</td>\n",
       "      <td>9.048</td>\n",
       "      <td>2.157</td>\n",
       "      <td>16.21</td>\n",
       "      <td>55.817397</td>\n",
       "      <td>high</td>\n",
       "      <td>9.0480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1835</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1766</td>\n",
       "      <td>7.123</td>\n",
       "      <td>3.979</td>\n",
       "      <td>17.66</td>\n",
       "      <td>40.334088</td>\n",
       "      <td>high</td>\n",
       "      <td>7.1230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>3375</td>\n",
       "      <td>2525</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1239</td>\n",
       "      <td>1.468</td>\n",
       "      <td>2.337</td>\n",
       "      <td>12.39</td>\n",
       "      <td>11.848265</td>\n",
       "      <td>high</td>\n",
       "      <td>1.4680</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      X     Y  facies  porosity    perm     ai  porosity100     permpor  \\\n",
       "0   565  1485       1    0.1184   6.170  2.009        11.84   52.111486   \n",
       "2  2065  2865       2    0.1000  92.297  3.524        19.20  480.713542   \n",
       "3  3575  2655       1    0.1621   9.048  2.157        16.21   55.817397   \n",
       "4  1835    35       1    0.1766   7.123  3.979        17.66   40.334088   \n",
       "5  3375  2525       1    0.1239   1.468  2.337        12.39   11.848265   \n",
       "\n",
       "  tporosity  perm_cutoff  \n",
       "0       low       0.0001  \n",
       "2      high      92.2970  \n",
       "3      high       9.0480  \n",
       "4      high       7.1230  \n",
       "5      high       1.4680  "
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "por = df.at[2,'porosity']                               # get the value for sample 1 of the porosity feature\n",
    "print('The value of porosity for sample 2 is ' + str(por) + '.')\n",
    "df.at[2,'porosity'] = 0.10                              # set the value for sample 1 of the porosity feature\n",
    "print('The value of porosity for sample 2 is now 0.1000.')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Writing the Tabular Data to a File\n",
    "\n",
    "It may be useful to write the DataFrame out for storage or curation and / or to be utilize with another platform (even R or Excel!).  It is easy to write the DataFrame back to a comma delimited file.  We have the 'to_csv' DataFrame member function to accomplish this.  The file will write to the working directory (another reason we set that at the beginning).  Go to that folder and open this new file with TextPad, Excel or any other program that opens .txt files to check it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"2D_MV_200wells_out.csv\")                      # write out the df DataFrame to a comma delimited file "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More Exercises\n",
    "\n",
    "There are so many more exercises and tests that one could attempt to gain experience with the pandas package, DataFrames objects in Python. I'll end here for brevity, but I invite you to continue. Check out the docs at https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html.  I'm always happy to discuss,\n",
    "\n",
    "*Michael*\n",
    "\n",
    "#### The Author:\n",
    "\n",
    "### Michael Pyrcz, Associate Professor, University of Texas at Austin \n",
    "*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*\n",
    "\n",
    "With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development. \n",
    "\n",
    "For more about Michael check out these links:\n",
    "\n",
    "#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)\n",
    "\n",
    "#### Want to Work Together?\n",
    "\n",
    "I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.\n",
    "\n",
    "* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you! \n",
    "\n",
    "* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!\n",
    "\n",
    "* I can be reached at mpyrcz@austin.utexas.edu.\n",
    "\n",
    "I'm always happy to discuss,\n",
    "\n",
    "*Michael*\n",
    "\n",
    "Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}