bonus-4-projects.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Mini-Projects\n",
    "\n",
    "I have designed a set of mini-projects that should help you recap what you've learned throughout this tutorial. Whether you work through them in order, or just work through a select few, is your choice.\n",
    "\n",
    "If you're feeling up to the challenge, feel free to also explore the data in-depth in this notebook.\n",
    "\n",
    "Otherwise, you'll get enough practice for this tutorial by following the exercise prompts below. The easy ones have a lot of specific instructions; the more difficult ones will leave a lot of decisions/choices in your hands. Be sure to try both!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Project 1: Schema & Value Checking (Easier)\n",
    "\n",
    "In this mini-project, we will explore through a dataset collected as part of a study of smartphone sanitization methods. The data are found in the file `data/smartphone_sanitization_corrupted.csv`. It should be clear that I have intentionally named it as such!\n",
    "\n",
    "1. The data provider did not write the YAML specification file for the data. Write the YAML spec for the columns that should be present. Then, write the test for the columns as a defensive check. **(recap)**\n",
    "1. For each row, the column `site` should be only one of three values: `phone`, `junction`, and `case`. Write a test that checks that the `site` column has only these three values.\n",
    "1. The `colonies_` and `morphologies_` columns cannot contain negative values. Write a test that checks for this.\n",
    "1. The `treatment` column can only be one of eight values: `ethanol`, `phonesoap`, `bleachwipe`, `quatricide`, `kimwipe`, `FBM_2`, `CB30` and `cellblaster`.\n",
    "\n",
    "Once you're done with this, try making a plot of the pre-colony and post-colony counts in a Jupyter notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Project 2: Schema & Value Checking (Challenging)\n",
    "\n",
    "In this mini-project, we will explore through a dataset of Financial Aid applicants to a conference. The dataset has been anonymized, and all personally-identifiable data have been removed. The data are found in the file `data/finaid-applications.csv`, and the metadata spec file can be found in `data/metadata_finaid.yml`.\n",
    "\n",
    "1. The data provider has provided a `metadata_finaid.yml` file. Check that the columns are correct. There might be a bit of a challenge here: there is extra nested metadata, but the schema is guaranteed to be stable.\n",
    "1. The data are a bit corrupted. Do some forensics to figure out how the data are corrupted. If you're stuck, you can reference the [`corrupt_data_changes.md`](./corrupt_data_changes.md) file for the answer, but it's better if you can figure it out. Once you've done that, write tests to check for file integrity.\n",
    "\n",
    "Once you're done with this, treat yourself to a data analysis task: plot the proportions of all \"international students\", i.e. students who are not living in their country of citizenship."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Project 3: File Integrity Checking\n",
    "\n",
    "In this mini-project, we will continue the file integrity checks by figuring out how to automate routine testing tasks.\n",
    "\n",
    "- Write a function that extends the code from Notebook 3 (File Integrity Checks), such that it records the hash of every data file (only CSV files) under the `data/` directory. \n",
    "- If you've refactored a big chunk of code into functions, write tests for each of those functions.\n",
    "- Write a test for data file integrity. The test function should test that the hash of every file in the database is consistent with its current hash on disk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 4: Meta-testing - Missing Value Checks (Challenging)\n",
    "\n",
    "In this mini-project, we will build some tools to automatically spit out meta-level tests. \n",
    "\n",
    "- Write a function that automatically produces `missingno.matrix()` plots for every CSV file in the `data/` directory.\n",
    "- Write a function that automatically reports every column in every CSV file that contains missing data, using the `pandas_summary` package.\n",
    "- Write a function that tests that every CSV file under `data/` has its own corresponding `metadata` YAML spec file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Projct 5: Property-Based Testing\n",
    "\n",
    "In this mini-project, we will practice the use of property-based tests. (TBD)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Environment (conda_datatest)",
   "language": "python",
   "name": "conda_datatest"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}