Skip to content

Latest commit

 

History

History

docs

readme.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <meta http-equiv="Content-Style-Type" content="text/css" />
  <meta name="generator" content="pandoc" />
  <title></title>
  <style type="text/css">code{white-space: pre;}</style>
  <style type="text/css">
div.sourceCode { overflow-x: auto; }
table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode {
  margin: 0; padding: 0; vertical-align: baseline; border: none; }
table.sourceCode { width: 100%; line-height: 100%; }
td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; color: #aaaaaa; border-right: 1px solid #aaaaaa; }
td.sourceCode { padding-left: 5px; }
code > span.kw { color: #007020; font-weight: bold; } /* Keyword */
code > span.dt { color: #902000; } /* DataType */
code > span.dv { color: #40a070; } /* DecVal */
code > span.bn { color: #40a070; } /* BaseN */
code > span.fl { color: #40a070; } /* Float */
code > span.ch { color: #4070a0; } /* Char */
code > span.st { color: #4070a0; } /* String */
code > span.co { color: #60a0b0; font-style: italic; } /* Comment */
code > span.ot { color: #007020; } /* Other */
code > span.al { color: #ff0000; font-weight: bold; } /* Alert */
code > span.fu { color: #06287e; } /* Function */
code > span.er { color: #ff0000; font-weight: bold; } /* Error */
code > span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
code > span.cn { color: #880000; } /* Constant */
code > span.sc { color: #4070a0; } /* SpecialChar */
code > span.vs { color: #4070a0; } /* VerbatimString */
code > span.ss { color: #bb6688; } /* SpecialString */
code > span.im { } /* Import */
code > span.va { color: #19177c; } /* Variable */
code > span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code > span.op { color: #666666; } /* Operator */
code > span.bu { } /* BuiltIn */
code > span.ex { } /* Extension */
code > span.pp { color: #bc7a00; } /* Preprocessor */
code > span.at { color: #7d9029; } /* Attribute */
code > span.do { color: #ba2121; font-style: italic; } /* Documentation */
code > span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code > span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code > span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
  </style>
  <link rel="stylesheet" href="book.css" type="text/css" />
</head>
<body>
<h1 id="best-testing-practices-for-data-science">Best Testing Practices for Data Science</h1>
<p>A short tutorial for data scientists on how to write tests for your code and your data. Before the tutorial, please read through this README file, for it contains a lot of useful information that will help you best prepare for the tutorial.</p>
<h2 id="how-to-use-this-repository">How to use this repository</h2>
<p>The tutorial notes are typed up in Jupyter notebooks, and static HTML versions are available under the <a href="./docs/"><code>docs</code></a> folder. For the non-bonus material, I suggest working through the notes in order. With the exception of the Projects, the bonus material can be tackled in any order. During the tutorial, be sure to have the HTML versions open.</p>
<h2 id="pre-requisite-knowledge">Pre-Requisite Knowledge</h2>
<p>I am assuming you are of the following type of coder:</p>
<ul>
<li>You are a data analytics type, who knows how to read/write CSV files with Pandas, and do basic data manipulation (slicing, indexing rows + columns, using the <code>.apply()</code> function).</li>
<li>You are not necessarily a seasoned software developer who has experience running tests.</li>
<li>You are comfortable with operating in the Terminal environment.</li>
<li>You have some rudimentary knowledge of <code>numpy</code>, particularly the the <code>array.min()</code>, <code>array.max()</code>, <code>array.mean()</code>, <code>array.std()</code>, and <code>numpy.allclose(a1, a2)</code> function calls.</li>
</ul>
<p>In order to prepare for the tutorial, there are some pieces of Python syntax that will come in handy to know: - the context manager syntax (<code>with ....</code>), - assertions (<code>assert conditions1 == condition2</code>), - file I/O (<code>with open(....) as f:...</code>), - list/dict/tuple comprehensions (<code>[a for a in container if condition(a)]</code>), - checking types &amp; attributes (<code>isinstance(obj, type) or hasattr(obj, attr)</code>).</p>
<h2 id="feedback">Feedback</h2>
<p>If you've taken a version of this tutorial, please leave feedback <a href="https://ericma1.typeform.com/to/Ua0LBs">here</a>. I use the suggestions in there to adjust the tutorial content and make it better. The changes are always released publicly on GitHub, so everybody benefits!</p>
<h1 id="environment-setup">Environment Setup</h1>
<h2 id="conda-setup"><code>conda</code> setup</h2>
<p>This installation route should work cross-platform. I recommend using the <a href="https://www.continuum.io/downloads">Anaconda distribution</a> of Python because it is a good way to bootstrap your data science environment.</p>
<p>To get setup, create a <code>conda</code> environment based on the provided <a href="./environment.yml"><code>environment.yml</code></a> spec file. Run the following commands in your bash terminal.</p>
<div class="sourceCode"><pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="fu">bash</span> conda-setup.sh</code></pre></div>
<h2 id="pip-setup"><code>pip</code> setup</h2>
<p>The alternative way is to use a virtualenv environment:</p>
<div class="sourceCode"><pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="fu">bash</span> venv-setup.sh
$ <span class="bu">source</span> datatest/bin/activate</code></pre></div>
<p>Alternatively, you can <code>pip</code> install each of the dependencies listed in the <code>environment.yml</code> file. (The <code>requirements.txt</code> file may be less eagerly maintained than the <code>environment.yml</code> file, given the <code>conda</code>-biases that I have.)</p>
<h2 id="manual-setup">Manual Setup</h2>
<p>If you prefer having more control over your installation process, <code>conda</code> or <code>pip</code> install the dependencies listed in the <code>environment.yml</code> file.</p>
<h2 id="checks">Checks</h2>
<p>To check whether the environment is correctly setup, run the <code>checkenv.py</code> script:</p>
<div class="sourceCode"><pre class="sourceCode bash"><code class="sourceCode bash">$ <span class="ex">python</span> checkenv.py</code></pre></div>
<p>It should print to your terminal, <code>All packages found; environment checks passed.</code>. Otherwise, <code>conda</code> or <code>pip</code> install the necessary packages stated (they will show up one by one).</p>
<h1 id="authors">Authors</h1>
<ul>
<li><a href="http://www.ericmjl.com">Eric J. Ma</a></li>
</ul>
<h1 id="contributors">Contributors</h1>
<p>Special thanks goes to individuals who have contributed in ways big and small to the improvement of the material.</p>
<ul>
<li>Renee Chu</li>
<li>Matt Bachmann: <span class="citation">@Bachmann1234</span></li>
<li>Hugo Bowne-Anderson: <span class="citation">@hugobowne</span></li>
<li>Boston Python tutorial attendees:
<ul>
<li><span class="citation">@races1986</span></li>
<li>Thao Nguyen: <span class="citation">@ThaoNguyen15</span></li>
<li><span class="citation">@ChrisMuir</span></li>
</ul></li>
</ul>
<h1 id="data-credits">Data Credits</h1>
<ul>
<li><a href="https://www.divvybikes.com/data">Divvy Data Set</a></li>
<li><a href="https://data.boston.gov/">Analyze Boston</a></li>
<li>Mia T. Lieberman for the sanitization dataset.</li>
</ul>
</body>
</html>