interface-walkthrough.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Writing and Using a Bespoke Model Interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import emat\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import gzip\n",
    "import asyncio\n",
    "from emat.util.show_dir import show_dir, show_file_contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is meant to illustrate the use of TMIP-EMAT's\n",
    "various modes of operation.  It provides an illustration of how to use \n",
    "TMIP-EMAT and the demo interface to run the command line version\n",
    "of the [Road Test](https://tmip-emat.github.io/source/emat.examples/RoadTest/road_test_yaml.html) \n",
    "model. A similar approach can be developed to run\n",
    "any transportation model that can be run from the command line, including\n",
    "for proprietary modeling tools that are typically run from a graphical\n",
    "user interface (GUI) but that provide command line access also."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example notebook, we will activate some logging features.  The \n",
    "same logging utility is written directly into the EMAT and the\n",
    "[core_files_demo.py](https://github.com/tmip-emat/tmip-emat-bespoke-demo/blob/main/core_files_demo.py) module. This will give us a view of what's happening\n",
    "inside the code as it runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "from emat.util.loggers import log_to_stderr\n",
    "log = log_to_stderr(logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connecting to the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The interface for this model is located in the `core_files_demo.py`\n",
    "module, which we will import into this notebook.  This file is extensively\n",
    "documented in comments, and is a great starting point for new users\n",
    "who want to write an interface for a new bespoke travel demand model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import core_files_demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Within this module, you will find a definition for the \n",
    "`RoadTestFileModel` class.\n",
    "\n",
    "We initialize an instance of the model interface object.\n",
    "If you look at the module code, you'll note the `__init__` function\n",
    "does a number of things, including creating a temporary directory\n",
    "to work in, copying the needed files into this temporary directory,\n",
    "loading the scope, and creating a SQLite database to work within.\n",
    "For your implementation, you might or might not do any of these steps.\n",
    "In particular, you'll probably want to use a database that is\n",
    "not in a temporary location, so that the results will be available\n",
    "after this notebook is closed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:03.84] MainProcess/WARNING: changing cwd to /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc\n",
      "[00:03.88] MainProcess/INFO: running script emat_db_init.sql\n",
      "[00:03.89] MainProcess/INFO: running script meta_model.sql\n",
      "[00:03.90] MainProcess/INFO: found no experiments with missing run_id's\n",
      "[00:03.90] MainProcess/INFO: running script emat_db_init_views.sql\n"
     ]
    }
   ],
   "source": [
    "fx = core_files_demo.RoadTestFileModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have loaded the `RoadTestFileModel` class, we have\n",
    "a number of files available in the \"master_directory\" that \n",
    "was created as that temporary directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tmpygcky5rc/\n",
      "├── road-test-colleague.sqlitedb\n",
      "├── road-test-demo.db\n",
      "├── road-test-files/\n",
      "│   ├── demo-inputs-l.yml\n",
      "│   └── demo-inputs-x.yml.template\n",
      "├── road-test-model-config.yml\n",
      "└── road-test-scope.yml\n"
     ]
    }
   ],
   "source": [
    "show_dir(fx.master_directory.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding Directories\n",
    "\n",
    "The TMIP-EMAT interface design for files-based bespoke models uses\n",
    "pointers for several directories to control the operation of the \n",
    "model.\n",
    "\n",
    "- **local_directory**  \n",
    "    This is the working directory for this instance of TMIP-EMAT,\n",
    "    not that for the core model itself. Typically it can be Python's \n",
    "    usual current working directory, accessible via `os.getcwd()`.\n",
    "    In this directory typically you'll have a TMIP-EMAT model \n",
    "    configuration *yaml* file, a scope definition *yaml* file, and\n",
    "    a sub-directory containing the files needed to run the core model\n",
    "    itself.\n",
    "\n",
    "- **model_path**  \n",
    "    The relative path from the `local_directory` to the directory where\n",
    "    the core model files are located.  When the core model itself is actually\n",
    "    run, this should be to the \"current working directory\" for that run.\n",
    "    The `model_path` must be given in the model config *yaml* file.\n",
    "\n",
    "- **rel_output_path**  \n",
    "    The relative path from the `model_path` to the directory where\n",
    "    the core model output files are located. The default value of this \n",
    "    path is \"./Outputs\" but this can be overridden by setting \n",
    "    `rel_output_path` in the model config *yaml* file. If the outputs\n",
    "    are comingled with other input files in the core model directory,\n",
    "    this can be set to \".\" (just a dot).\n",
    "\n",
    "- **archive_path**  \n",
    "    The path where model archive directories can be found. This path\n",
    "    must be given in the model config *yaml* file. It can be given as\n",
    "    an absolute path, or a relative path. If it is a relative path, \n",
    "    it should be relative to the `local_directory`.\n",
    "    \n",
    "These directories, especially the ones other than the `local_directory`,\n",
    "are defined in a model configuration *yaml* file. This makes it easy to\n",
    "change the directory pointers when moving TMIP-EMAT between different\n",
    "machines that may have different file system structures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single Run Operation for Development and Debugging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we take on the task of running this model in exploratory mode, we'll\n",
    "want to make sure that our interface code is working correctly. To check each\n",
    "of the components of the interface (setup, run, post-process, load-measures,\n",
    "and archive), we can run each individually in sequence, and inspect the results\n",
    "to make sure they are correct."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### setup\n",
    "\n",
    "This method is the place where the core model *set up* takes place,\n",
    "including creating or modifying files as necessary to prepare\n",
    "for a core model run.  When running experiments, this method\n",
    "is called once for each core model experiment, where each experiment\n",
    "is defined by a set of particular values for both the exogenous\n",
    "uncertainties and the policy levers.  These values are passed to\n",
    "the experiment only here, and not in the `run` method itself.\n",
    "This facilitates debugging, as the `setup` method can be used \n",
    "without the `run` method, as we do here. This allows us to manually\n",
    "inspect the prepared files and ensure they are correct before\n",
    "actually running a potentially expensive model.\n",
    "\n",
    "Each input exogenous uncertainty or policy lever can potentially\n",
    "be used to manipulate multiple different aspects of the underlying\n",
    "core model.  For example, a policy lever that includes a number of\n",
    "discrete future network \"build\" options might trigger the replacement\n",
    "of multiple related network definition files.  Or, a single uncertainty\n",
    "relating to the cost of fuel might scale both a parameter linked to\n",
    "the modeled per-mile cost of operating an automobile and the\n",
    "modeled total cost of fuel used by transit services."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this demo model, running the core model itself in files mode \n",
    "requires two configuration files to be available, one for levers and\n",
    "another for uncertainties.  These two files are provided in the demo\n",
    "in two ways: as a runnable base file (for the levers) and as a template\n",
    "file (for the uncertainties)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The levers file is a *ready-to-use* file (for this demo, in YAML format,\n",
    "although your model may use a different file format for input files).\n",
    "It has default values pre-coded into the file, and to modify this \n",
    "file for use by EMAT the `setup` method needs to parse and edit this\n",
    "file to swap out the default values for new ones in each experiment.\n",
    "This can be done using regular expressions (as in this demo), or any other method you\n",
    "like to edit the file appropriately.  The advantage of this approach\n",
    "is that the base file is ready to use with the core model as-is, facilitating\n",
    "the use of this file outside the EMAT context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---\n",
      "# This file defines lever values for the files-based\n",
      "# Road Test example.  It is intentionally a complex way\n",
      "# to implement this Python-based model, designed to\n",
      "# demonstrate how to use a files-based model called\n",
      "# from the command line.\n",
      "expand_capacity: 10\n",
      "amortization_period: 30\n",
      "interest_rate_lock: False\n",
      "debt_type: GO Bond\n",
      "lane_width: 10\n",
      "mandatory_unused_lever: 42\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "show_file_contents(fx.master_directory.name, 'road-test-files', 'demo-inputs-l.yml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By contrast, the uncertainties file is in a *template* format. The\n",
    "values of the parameters that will be manipulated by EMAT for each \n",
    "experiment are not given by default values, but instead \n",
    "each value to be set is indicated in the file by a unique token that is easy to\n",
    "search and replace, and definitely not something that appear in any script otherwise.\n",
    "This approach makes the text-substitution code that is used in this module much\n",
    "simpler and less prone to bugs.  But there is a small downside of this approach:\n",
    "every parameter must definitely be replaced in this process, as the template file\n",
    "is unusable outside the EMAT context, and also every unique token needs to be replaced. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---\n",
      "# This file defines uncertainty values for the files-based\n",
      "# Road Test example.  It is intentionally a complex way\n",
      "# to implement this Python-based model, designed to\n",
      "# demonstrate how to use a files-based model called\n",
      "# from the command line.\n",
      "alpha: __EMAT_PROVIDES_VALUE__ALPHA__\n",
      "beta: __EMAT_PROVIDES_VALUE__BETA__\n",
      "input_flow: __EMAT_PROVIDES_VALUE__INPUT_FLOW__\n",
      "value_of_time: __EMAT_PROVIDES_VALUE__VALUE_OF_TIME__\n",
      "labor_unit_cost_expansion: __EMAT_PROVIDES_VALUE__LABOR_UNIT_COST_EXPANSION__\n",
      "materials_unit_cost_expansion: __EMAT_PROVIDES_VALUE__MATERIALS_UNIT_COST_EXPANSION__\n",
      "interest_rate: __EMAT_PROVIDES_VALUE__INTEREST_RATE__\n",
      "yield_curve: __EMAT_PROVIDES_VALUE__YIELD_CURVE__\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "show_file_contents(fx.master_directory.name, 'road-test-files', 'demo-inputs-x.yml.template')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regardless of which file management system you use, the `setup` method\n",
    "is the place to make edits to these input files and write them into \n",
    "your working directory.  To do so,\n",
    "the `setup` method takes one argument: a dictionary containing key-value\n",
    "pairs that assign a particular value to each input (exogenous uncertainty \n",
    "or policy lever) that is defined in the model scope.  The keys must match \n",
    "exactly with the names of the parameters given in the scope. \n",
    "\n",
    "If you have written your `setup` method to call the super-class `setup`,\n",
    "you will find that if you give keys as input that are not defined in\n",
    "the scope, you'll get a KeyError."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:03.94] MainProcess/ERROR: SETUP ERROR: 'name_not_in_scope' not found in scope parameters\n",
      "[00:03.94] MainProcess/ERROR: KeyError(\"'name_not_in_scope' not found in scope parameters\")\n"
     ]
    }
   ],
   "source": [
    "bad_params = {\n",
    "    'name_not_in_scope': 'is_a_problem',\n",
    "}\n",
    "\n",
    "try:\n",
    "    fx.setup(bad_params)\n",
    "except KeyError as error:\n",
    "    log.error(repr(error))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the other hand, your custom model may or may not allow you to leave out\n",
    "some parameters.  It is up to you to decide how to handle missing values, \n",
    "either by setting them at their default values or raising an error. In \n",
    "normal operation, parameters typically won't be left out from the design\n",
    "of experiments, so it is not usually important to monitor this carefully.\n",
    "\n",
    "In our example module's `setup`, all of the uncertainty values must be given,\n",
    "because the template file would be unusable otherwise. But the policy levers \n",
    "can be omitted, and if so they are left at their default values in the \n",
    "original file.  Note that the default values in that file are not strictly\n",
    "consistent with the default values in the scope file, and TMIP-EMAT does \n",
    "nothing on its own to address this discrepancy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:03.97] MainProcess/INFO: RoadTestFileModel SETUP RUNID-c7c8bb08-6428-11eb-8c2b-acde48001122\n",
      "[00:03.97] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 1 RUNID-c7c8bb08-6428-11eb-8c2b-acde48001122\n"
     ]
    }
   ],
   "source": [
    "params = {\n",
    "    'expand_capacity': 75,\n",
    "    'amortization_period': 25,\n",
    "    'debt_type': \"Paygo\",\n",
    "    'alpha': 0.1234,\n",
    "    'beta': 4.0,\n",
    "    'input_flow': 100,\n",
    "    'value_of_time': 0.075,\n",
    "    'unit_cost_expansion': 100,\n",
    "    'interest_rate': 0.035,\n",
    "    'yield_curve': 0.01,\n",
    "} # interest_rate_lock is missing, that's ok\n",
    "\n",
    "fx.setup(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running `setup` successfully, we will have overwritten the \n",
    "\"demo-inputs-l.yml\" file with new values, and written a new \n",
    "\"demo-inputs-x.yml\" file into the model working directory with those\n",
    "values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tmpygcky5rc/\n",
      "├── _emat_experiment_id_.yml\n",
      "├── _emat_parameters_.yml\n",
      "├── archive/\n",
      "│   └── scp_EMAT Road Test/\n",
      "│       └── exp_001_c7c8bb08-6428-11eb-8c2b-acde48001122/\n",
      "│           └── _emat_start_.log\n",
      "├── road-test-colleague.sqlitedb\n",
      "├── road-test-demo.db\n",
      "├── road-test-files/\n",
      "│   ├── demo-inputs-l.yml\n",
      "│   ├── demo-inputs-x.yml\n",
      "│   └── demo-inputs-x.yml.template\n",
      "├── road-test-model-config.yml\n",
      "└── road-test-scope.yml\n"
     ]
    }
   ],
   "source": [
    "show_dir(fx.local_directory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---\n",
      "# This file defines lever values for the files-based\n",
      "# Road Test example.  It is intentionally a complex way\n",
      "# to implement this Python-based model, designed to\n",
      "# demonstrate how to use a files-based model called\n",
      "# from the command line.\n",
      "expand_capacity: 75\n",
      "amortization_period: 25\n",
      "interest_rate_lock: False\n",
      "debt_type: Paygo  \n",
      "lane_width: 10\n",
      "mandatory_unused_lever: 42\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "show_file_contents(fx.local_directory, 'road-test-files', 'demo-inputs-l.yml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---\n",
      "# This file defines uncertainty values for the files-based\n",
      "# Road Test example.  It is intentionally a complex way\n",
      "# to implement this Python-based model, designed to\n",
      "# demonstrate how to use a files-based model called\n",
      "# from the command line.\n",
      "alpha: 0.1234\n",
      "beta: 4.0\n",
      "input_flow: 100\n",
      "value_of_time: 0.075\n",
      "labor_unit_cost_expansion: 60.0\n",
      "materials_unit_cost_expansion: 40.0\n",
      "interest_rate: 0.035\n",
      "yield_curve: 0.01\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "show_file_contents(fx.local_directory, 'road-test-files', 'demo-inputs-x.yml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `run` method is the place where the core model run takes place.\n",
    "Note that this method takes no arguments; all the input\n",
    "exogenous uncertainties and policy levers are delivered to the\n",
    "core model in the `setup` method, which will be executed prior\n",
    "to calling this method. This facilitates debugging, as the `setup`\n",
    "method can be used without the `run` method as we did above, allowing\n",
    "us to manually inspect the prepared files and ensure they\n",
    "are correct before actually running a potentially expensive model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:06.49] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-c7c8bb08-6428-11eb-8c2b-acde48001122\n"
     ]
    }
   ],
   "source": [
    "fx.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `RoadTestFileModel` class includes a custom `last_run_logs` method,\n",
    "which displays both the \"stdout\" and \"stderr\" logs generated by the \n",
    "model executable during the most recent call to the `run` method.\n",
    "We can use this method for debugging purposes, to identify why the \n",
    "core model crashes (if it does crash).  In this first test it did not\n",
    "crash, and the logs look good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== STDOUT ===\n",
      "[2021-01-31 18:59:50,158] emat.RoadTest.INFO: running emat-road-test-demo\n",
      "[2021-01-31 18:59:50,164] emat.RoadTest.INFO: emat-road-test-demo completed without errors\n",
      "\n",
      "=== END OF LOG ===\n"
     ]
    }
   ],
   "source": [
    "fx.last_run_logs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### post-process\n",
    "\n",
    "There is an (optional) `post_process` step that is separate from the `run` step.\n",
    "\n",
    "Post-processing differs from the main model run in two important ways:\n",
    "\n",
    "- It can be run to efficiently generate a subset of performance measures.\n",
    "- It can be run based on archived model main-run core model results.\n",
    "\n",
    "Both features are designed to support workflows where new performance \n",
    "measures are added to the exploratory scope after the main model run(s)\n",
    "are completed. By allowing the `post_process` method to be run only for a \n",
    "subset of measures, we can avoid replicating possibly expensive \n",
    "post-processing steps when we have already completed them, or when they\n",
    "are not needed for a particular application.  \n",
    "\n",
    "For example, consider an exploratory modeling activity where the scope \n",
    "at the time of the initial model run experiments was focused on highway\n",
    "measures, and transit usage was not explored extensively, and no \n",
    "network assignment was done for transit trips when the experiments were\n",
    "initially run.  By creating a post-process step to run the transit \n",
    "network assignment, we can apply that step to existing archived results,\n",
    "as well as have it run automatically for future model experients\n",
    "where transit usage is under study, but continue to omit it for future \n",
    "model experients where we do not need it.\n",
    "\n",
    "An optional `measure_names` argument allows the post-processor to\n",
    "identify which measures need additional computational effort to generate,\n",
    "and to skip excluded measures that are not currently of interest, or\n",
    "which have already been computed and do not need to be computed again.\n",
    "\n",
    "The post processing is isolated from the main model run to allow it to\n",
    "be run later using archived model results.  When executed directly \n",
    "after a core model run, it will operate on the results of the model\n",
    "stored in the local working directory.  However, it can also be\n",
    "used with an optional `output_path` argument, which can be pointed at\n",
    "a model archive directory instead of the local working directory.\n",
    "\n",
    "A consequence of this (and an intentional limitation) is that the \n",
    "`post_process` method should only use files from the set of files \n",
    "that are or will be archived from the core model run, and not attempt\n",
    "to use other non-persistent temporary or intermediate files that \n",
    "will not be archived."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:06.52] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-c7c8bb08-6428-11eb-8c2b-acde48001122\n"
     ]
    }
   ],
   "source": [
    "fx.post_process()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, the model's output performance measures should be available in one\n",
    "or more output files that can be read in the next step.  For this example, the\n",
    "results are written to two separate files: 'output_1.csv.gz' and 'output.yaml'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "build_travel_time: 60.78943107038733\n",
      "no_build_travel_time: 67.404\n",
      "time_savings: 6.614568929612666\n",
      "\n"
     ]
    }
   ],
   "source": [
    "show_file_contents(fx.local_directory, 'road-test-files', \"Outputs\", \"output.yaml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note in this example, some of the values in the `output_1.csv.gz` file\n",
    "are intentionally manipulated in a contrived manner, so that there is \n",
    "some work for the post-processor to do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ",value_of_time_savings,present_cost_expansion,cost_of_capacity_expansion,net_benefits\n",
      "exp,1.0508604102769246,,1.3651577800056909,\n",
      "plain,49.60926697209499,7500.0,311.2700117047018,-261.6607447326066\n",
      "\n"
     ]
    }
   ],
   "source": [
    "show_file_contents(fx.local_directory, 'road-test-files', \"Outputs\", \"output_1.csv.gz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### load-measures\n",
    "\n",
    "The `load_measures` method is the place to actually reach into\n",
    "files in the core model's run results and extract performance\n",
    "measures, returning a dictionary of key-value pairs for the \n",
    "various performance measures. It takes an optional list giving a \n",
    "subset of performance measures to load, and like the `post_process` \n",
    "method also can be pointed at an archive location instead of loading \n",
    "measures from the local working directory (which is the default).\n",
    "The `load_measures` method should not do any post-processing\n",
    "of results (i.e. it should read from but not write to the model\n",
    "outputs directory)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'value_of_time_savings': 49.60926697209499,\n",
       " 'present_cost_expansion': 7500.0,\n",
       " 'cost_of_capacity_expansion': 311.2700117047018,\n",
       " 'net_benefits': -261.6607447326066,\n",
       " 'build_travel_time': 60.78943107038733,\n",
       " 'no_build_travel_time': 67.404,\n",
       " 'time_savings': 6.614568929612666}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fx.load_measures()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may note that the implementation of `RoadTestFileModel` in the `core_files_demo` module\n",
    "does not actually include a `load_measures` method itself, but instead inherits this method\n",
    "from the `FilesCoreModel` superclass. The instructions on how to actually find the relevant\n",
    "performance measures for this file are instead loaded into table parsers, which are defined\n",
    "in the `RoadTestFileModel.__init__` constructor.  There are [details and illustrations\n",
    "of how to write and use parsers in the file parsing examples page of the TMIP-EMAT documentation.](https://tmip-emat.github.io/source/emat.models/table_parse_example.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### archive\n",
    "\n",
    "The `archive` method copies the relevant model output files to an archive location for \n",
    "longer term storage.  The particular archive location is based on the experiment id\n",
    "for a particular experiment, and can be customized if desired by overloading the \n",
    "`get_experiment_archive_path` method.  This customization is not done in this demo,\n",
    "so the default location is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_001_c7c8bb08-6428-11eb-8c2b-acde48001122'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fx.get_experiment_archive_path(parameters=params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually running the `archive` method should copy any relevant output files\n",
    "from the `model_path` of the current active model into a subdirectory of `archive_path`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:06.55] MainProcess/INFO: RoadTestFileModel ARCHIVE\n",
      " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n",
      "   to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_001_c7c8bb08-6428-11eb-8c2b-acde48001122\n"
     ]
    }
   ],
   "source": [
    "fx.archive(params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tmpygcky5rc/\n",
      "├── _emat_experiment_id_.yml\n",
      "├── _emat_parameters_.yml\n",
      "├── archive/\n",
      "│   └── scp_EMAT Road Test/\n",
      "│       └── exp_001_c7c8bb08-6428-11eb-8c2b-acde48001122/\n",
      "│           ├── _emat_start_.log\n",
      "│           ├── demo-inputs-l.yml\n",
      "│           ├── demo-inputs-x.yml\n",
      "│           ├── demo-inputs-x.yml.template\n",
      "│           ├── emat-road-test.log\n",
      "│           ├── output.csv.gz\n",
      "│           ├── output.yaml\n",
      "│           └── Outputs/\n",
      "│               ├── output.yaml\n",
      "│               └── output_1.csv.gz\n",
      "├── road-test-colleague.sqlitedb\n",
      "├── road-test-demo.db\n",
      "├── road-test-files/\n",
      "│   ├── demo-inputs-l.yml\n",
      "│   ├── demo-inputs-x.yml\n",
      "│   ├── demo-inputs-x.yml.template\n",
      "│   ├── emat-road-test.log\n",
      "│   ├── output.csv.gz\n",
      "│   ├── output.yaml\n",
      "│   └── Outputs/\n",
      "│       ├── output.yaml\n",
      "│       └── output_1.csv.gz\n",
      "├── road-test-model-config.yml\n",
      "└── road-test-scope.yml\n"
     ]
    }
   ],
   "source": [
    "show_dir(fx.local_directory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is permissible, but not required, to simply copy the entire contents of the \n",
    "former to the latter, as is done in this example. However, if the current active model\n",
    "directory has a lot of boilerplate files that don't change with the inputs, or\n",
    "if it becomes full of intermediate or temporary files that definitely will never\n",
    "be used to compute performance measures, it can be advisable to selectively copy\n",
    "only relevant files. In that case, those files and whatever related sub-directory\n",
    "tree structure exists in the current active model should be replicated within the\n",
    "experiments archive directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normal Operation for Running Multiple Experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this demo, we'll create a design of experiments with only 8 experiments.\n",
    "The `design_experiments` method of the `RoadTestFileModel` object is not defined\n",
    "in the custom `core_files_demo` written for this model, but rather is a generic\n",
    "function provide by the TMIP-EMAT main library.\n",
    "Real applications will typically use a larger number of experiments, but this small number\n",
    "is sufficient to demonstrate the operation of the tools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>amortization_period</th>\n",
       "      <th>beta</th>\n",
       "      <th>debt_type</th>\n",
       "      <th>expand_capacity</th>\n",
       "      <th>input_flow</th>\n",
       "      <th>interest_rate</th>\n",
       "      <th>interest_rate_lock</th>\n",
       "      <th>unit_cost_expansion</th>\n",
       "      <th>value_of_time</th>\n",
       "      <th>yield_curve</th>\n",
       "      <th>free_flow_time</th>\n",
       "      <th>initial_capacity</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>experiment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.134750</td>\n",
       "      <td>36</td>\n",
       "      <td>4.642370</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>24.901018</td>\n",
       "      <td>85</td>\n",
       "      <td>0.039219</td>\n",
       "      <td>False</td>\n",
       "      <td>143.519760</td>\n",
       "      <td>0.017005</td>\n",
       "      <td>0.003280</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.115907</td>\n",
       "      <td>50</td>\n",
       "      <td>5.242315</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>11.985022</td>\n",
       "      <td>114</td>\n",
       "      <td>0.025997</td>\n",
       "      <td>True</td>\n",
       "      <td>107.836739</td>\n",
       "      <td>0.145378</td>\n",
       "      <td>0.009659</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.178456</td>\n",
       "      <td>30</td>\n",
       "      <td>3.510139</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>72.399552</td>\n",
       "      <td>121</td>\n",
       "      <td>0.029066</td>\n",
       "      <td>True</td>\n",
       "      <td>127.838010</td>\n",
       "      <td>0.067821</td>\n",
       "      <td>-0.000839</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.110023</td>\n",
       "      <td>44</td>\n",
       "      <td>4.887030</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>28.565637</td>\n",
       "      <td>127</td>\n",
       "      <td>0.034673</td>\n",
       "      <td>True</td>\n",
       "      <td>125.799829</td>\n",
       "      <td>0.083593</td>\n",
       "      <td>0.002714</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.161977</td>\n",
       "      <td>24</td>\n",
       "      <td>3.865644</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>82.811459</td>\n",
       "      <td>135</td>\n",
       "      <td>0.028634</td>\n",
       "      <td>True</td>\n",
       "      <td>118.989447</td>\n",
       "      <td>0.054968</td>\n",
       "      <td>0.006745</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.173449</td>\n",
       "      <td>21</td>\n",
       "      <td>4.094118</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>43.476172</td>\n",
       "      <td>142</td>\n",
       "      <td>0.033847</td>\n",
       "      <td>False</td>\n",
       "      <td>133.480778</td>\n",
       "      <td>0.184904</td>\n",
       "      <td>0.017674</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.141973</td>\n",
       "      <td>19</td>\n",
       "      <td>5.331978</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>52.445940</td>\n",
       "      <td>98</td>\n",
       "      <td>0.031519</td>\n",
       "      <td>False</td>\n",
       "      <td>99.317350</td>\n",
       "      <td>0.115228</td>\n",
       "      <td>0.014752</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.193762</td>\n",
       "      <td>40</td>\n",
       "      <td>4.453382</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>92.278968</td>\n",
       "      <td>95</td>\n",
       "      <td>0.037907</td>\n",
       "      <td>False</td>\n",
       "      <td>103.543415</td>\n",
       "      <td>0.092517</td>\n",
       "      <td>0.014360</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               alpha  amortization_period      beta debt_type  \\\n",
       "experiment                                                      \n",
       "2           0.134750                   36  4.642370  Rev Bond   \n",
       "3           0.115907                   50  5.242315  Rev Bond   \n",
       "4           0.178456                   30  3.510139     Paygo   \n",
       "5           0.110023                   44  4.887030     Paygo   \n",
       "6           0.161977                   24  3.865644   GO Bond   \n",
       "7           0.173449                   21  4.094118     Paygo   \n",
       "8           0.141973                   19  5.331978   GO Bond   \n",
       "9           0.193762                   40  4.453382  Rev Bond   \n",
       "\n",
       "            expand_capacity  input_flow  interest_rate  interest_rate_lock  \\\n",
       "experiment                                                                   \n",
       "2                 24.901018          85       0.039219               False   \n",
       "3                 11.985022         114       0.025997                True   \n",
       "4                 72.399552         121       0.029066                True   \n",
       "5                 28.565637         127       0.034673                True   \n",
       "6                 82.811459         135       0.028634                True   \n",
       "7                 43.476172         142       0.033847               False   \n",
       "8                 52.445940          98       0.031519               False   \n",
       "9                 92.278968          95       0.037907               False   \n",
       "\n",
       "            unit_cost_expansion  value_of_time  yield_curve  free_flow_time  \\\n",
       "experiment                                                                    \n",
       "2                    143.519760       0.017005     0.003280              60   \n",
       "3                    107.836739       0.145378     0.009659              60   \n",
       "4                    127.838010       0.067821    -0.000839              60   \n",
       "5                    125.799829       0.083593     0.002714              60   \n",
       "6                    118.989447       0.054968     0.006745              60   \n",
       "7                    133.480778       0.184904     0.017674              60   \n",
       "8                     99.317350       0.115228     0.014752              60   \n",
       "9                    103.543415       0.092517     0.014360              60   \n",
       "\n",
       "            initial_capacity  \n",
       "experiment                    \n",
       "2                        100  \n",
       "3                        100  \n",
       "4                        100  \n",
       "5                        100  \n",
       "6                        100  \n",
       "7                        100  \n",
       "8                        100  \n",
       "9                        100  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "design1 = fx.design_experiments(design_name='lhs_1', n_samples=8)\n",
    "design1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `run_experiments` command will automatically run the model once for each experiment in the named design.\n",
    "The demo command line version of the road test model is (intentionally) a little bit slow, so will take a few\n",
    "seconds to conduct these eight model experiment runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:06.66] MainProcess/INFO: performing 8 scenarios/policies * 1 model(s) = 8 experiments\n",
      "[00:06.67] MainProcess/INFO: performing experiments sequentially\n",
      "[00:06.67] MainProcess/INFO: RoadTestFileModel SETUP RUNID-c9660fe2-6428-11eb-8c2b-acde48001122\n",
      "[00:06.68] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 2 RUNID-c9660fe2-6428-11eb-8c2b-acde48001122\n",
      "[00:09.14] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-c9660fe2-6428-11eb-8c2b-acde48001122\n",
      "[00:09.15] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-c9660fe2-6428-11eb-8c2b-acde48001122\n",
      "[00:09.16] MainProcess/INFO: RoadTestFileModel ARCHIVE\n",
      " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n",
      "   to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_002_c9660fe2-6428-11eb-8c2b-acde48001122\n",
      "[00:09.17] MainProcess/INFO: RAN EXPERIMENT IN 2.50 SECONDS\n",
      "[00:09.17] MainProcess/INFO: 1 cases completed\n",
      "[00:09.18] MainProcess/INFO: RoadTestFileModel SETUP RUNID-cae3c1f2-6428-11eb-8c2b-acde48001122\n",
      "[00:09.18] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 3 RUNID-cae3c1f2-6428-11eb-8c2b-acde48001122\n",
      "[00:11.56] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-cae3c1f2-6428-11eb-8c2b-acde48001122\n",
      "[00:11.57] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-cae3c1f2-6428-11eb-8c2b-acde48001122\n",
      "[00:11.58] MainProcess/INFO: RoadTestFileModel ARCHIVE\n",
      " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n",
      "   to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_003_cae3c1f2-6428-11eb-8c2b-acde48001122\n",
      "[00:11.59] MainProcess/INFO: RAN EXPERIMENT IN 2.42 SECONDS\n",
      "[00:11.59] MainProcess/INFO: 2 cases completed\n",
      "[00:11.60] MainProcess/INFO: RoadTestFileModel SETUP RUNID-cc552f1c-6428-11eb-8c2b-acde48001122\n",
      "[00:11.60] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 4 RUNID-cc552f1c-6428-11eb-8c2b-acde48001122\n",
      "[00:14.00] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-cc552f1c-6428-11eb-8c2b-acde48001122\n",
      "[00:14.01] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-cc552f1c-6428-11eb-8c2b-acde48001122\n",
      "[00:14.02] MainProcess/INFO: RoadTestFileModel ARCHIVE\n",
      " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n",
      "   to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_004_cc552f1c-6428-11eb-8c2b-acde48001122\n",
      "[00:14.03] MainProcess/INFO: RAN EXPERIMENT IN 2.44 SECONDS\n",
      "[00:14.03] MainProcess/INFO: 3 cases completed\n",
      "[00:14.04] MainProcess/INFO: RoadTestFileModel SETUP RUNID-cdc944c8-6428-11eb-8c2b-acde48001122\n",
      "[00:14.04] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 5 RUNID-cdc944c8-6428-11eb-8c2b-acde48001122\n",
      "[00:16.46] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-cdc944c8-6428-11eb-8c2b-acde48001122\n",
      "[00:16.47] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-cdc944c8-6428-11eb-8c2b-acde48001122\n",
      "[00:16.48] MainProcess/INFO: RoadTestFileModel ARCHIVE\n",
      " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n",
      "   to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_005_cdc944c8-6428-11eb-8c2b-acde48001122\n",
      "[00:16.48] MainProcess/INFO: RAN EXPERIMENT IN 2.45 SECONDS\n",
      "[00:16.48] MainProcess/INFO: 4 cases completed\n",
      "[00:16.49] MainProcess/INFO: RoadTestFileModel SETUP RUNID-cf3fd362-6428-11eb-8c2b-acde48001122\n",
      "[00:16.50] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 6 RUNID-cf3fd362-6428-11eb-8c2b-acde48001122\n",
      "[00:18.99] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-cf3fd362-6428-11eb-8c2b-acde48001122\n",
      "[00:19.00] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-cf3fd362-6428-11eb-8c2b-acde48001122\n",
      "[00:19.01] MainProcess/INFO: RoadTestFileModel ARCHIVE\n",
      " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n",
      "   to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_006_cf3fd362-6428-11eb-8c2b-acde48001122\n",
      "[00:19.02] MainProcess/INFO: RAN EXPERIMENT IN 2.53 SECONDS\n",
      "[00:19.02] MainProcess/INFO: 5 cases completed\n",
      "[00:19.02] MainProcess/INFO: RoadTestFileModel SETUP RUNID-d0c23324-6428-11eb-8c2b-acde48001122\n",
      "[00:19.03] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 7 RUNID-d0c23324-6428-11eb-8c2b-acde48001122\n",
      "[00:21.37] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-d0c23324-6428-11eb-8c2b-acde48001122\n",
      "[00:21.39] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-d0c23324-6428-11eb-8c2b-acde48001122\n",
      "[00:21.40] MainProcess/INFO: RoadTestFileModel ARCHIVE\n",
      " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n",
      "   to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_007_d0c23324-6428-11eb-8c2b-acde48001122\n",
      "[00:21.40] MainProcess/INFO: RAN EXPERIMENT IN 2.38 SECONDS\n",
      "[00:21.40] MainProcess/INFO: 6 cases completed\n",
      "[00:21.41] MainProcess/INFO: RoadTestFileModel SETUP RUNID-d22e7272-6428-11eb-8c2b-acde48001122\n",
      "[00:21.42] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 8 RUNID-d22e7272-6428-11eb-8c2b-acde48001122\n",
      "[00:23.88] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-d22e7272-6428-11eb-8c2b-acde48001122\n",
      "[00:23.89] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-d22e7272-6428-11eb-8c2b-acde48001122\n",
      "[00:23.91] MainProcess/INFO: RoadTestFileModel ARCHIVE\n",
      " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n",
      "   to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_008_d22e7272-6428-11eb-8c2b-acde48001122\n",
      "[00:23.91] MainProcess/INFO: RAN EXPERIMENT IN 2.51 SECONDS\n",
      "[00:23.91] MainProcess/INFO: 7 cases completed\n",
      "[00:23.92] MainProcess/INFO: RoadTestFileModel SETUP RUNID-d3ad349e-6428-11eb-8c2b-acde48001122\n",
      "[00:23.93] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 9 RUNID-d3ad349e-6428-11eb-8c2b-acde48001122\n",
      "[00:26.40] MainProcess/ERROR: ERROR in run_core_model run 9: Command '['emat-road-test-demo', '--uncs', 'demo-inputs-x.yml', '--levers', 'demo-inputs-l.yml']' returned non-zero exit status 247.\n",
      "[00:26.40] MainProcess/ERROR: run_core_model ABORT 9\n",
      "[00:26.40] MainProcess/INFO: RAN EXPERIMENT IN 2.49 SECONDS\n",
      "[00:26.41] MainProcess/INFO: 8 cases completed\n",
      "[00:26.41] MainProcess/INFO: experiments finished\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>beta</th>\n",
       "      <th>input_flow</th>\n",
       "      <th>value_of_time</th>\n",
       "      <th>unit_cost_expansion</th>\n",
       "      <th>interest_rate</th>\n",
       "      <th>yield_curve</th>\n",
       "      <th>expand_capacity</th>\n",
       "      <th>amortization_period</th>\n",
       "      <th>debt_type</th>\n",
       "      <th>interest_rate_lock</th>\n",
       "      <th>free_flow_time</th>\n",
       "      <th>initial_capacity</th>\n",
       "      <th>no_build_travel_time</th>\n",
       "      <th>build_travel_time</th>\n",
       "      <th>time_savings</th>\n",
       "      <th>value_of_time_savings</th>\n",
       "      <th>net_benefits</th>\n",
       "      <th>cost_of_capacity_expansion</th>\n",
       "      <th>present_cost_expansion</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>experiment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.134750</td>\n",
       "      <td>4.642370</td>\n",
       "      <td>85</td>\n",
       "      <td>0.017005</td>\n",
       "      <td>143.519760</td>\n",
       "      <td>0.039219</td>\n",
       "      <td>0.003280</td>\n",
       "      <td>24.901018</td>\n",
       "      <td>36</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>63.802031</td>\n",
       "      <td>61.354318</td>\n",
       "      <td>2.447713</td>\n",
       "      <td>3.537924</td>\n",
       "      <td>-175.134011</td>\n",
       "      <td>178.671935</td>\n",
       "      <td>3573.788162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.115907</td>\n",
       "      <td>5.242315</td>\n",
       "      <td>114</td>\n",
       "      <td>0.145378</td>\n",
       "      <td>107.836739</td>\n",
       "      <td>0.025997</td>\n",
       "      <td>0.009659</td>\n",
       "      <td>11.985022</td>\n",
       "      <td>50</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>73.822149</td>\n",
       "      <td>67.635961</td>\n",
       "      <td>6.186188</td>\n",
       "      <td>102.524606</td>\n",
       "      <td>42.361931</td>\n",
       "      <td>60.162675</td>\n",
       "      <td>1292.425695</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.178456</td>\n",
       "      <td>3.510139</td>\n",
       "      <td>121</td>\n",
       "      <td>0.067821</td>\n",
       "      <td>127.838010</td>\n",
       "      <td>0.029066</td>\n",
       "      <td>-0.000839</td>\n",
       "      <td>72.399552</td>\n",
       "      <td>30</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>80.905959</td>\n",
       "      <td>63.090263</td>\n",
       "      <td>17.815696</td>\n",
       "      <td>146.201784</td>\n",
       "      <td>-183.229667</td>\n",
       "      <td>329.431451</td>\n",
       "      <td>9255.414627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.110023</td>\n",
       "      <td>4.887030</td>\n",
       "      <td>127</td>\n",
       "      <td>0.083593</td>\n",
       "      <td>125.799829</td>\n",
       "      <td>0.034673</td>\n",
       "      <td>0.002714</td>\n",
       "      <td>28.565637</td>\n",
       "      <td>44</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>81.228935</td>\n",
       "      <td>66.217725</td>\n",
       "      <td>15.011210</td>\n",
       "      <td>159.363387</td>\n",
       "      <td>61.567385</td>\n",
       "      <td>97.796001</td>\n",
       "      <td>3593.552248</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.161977</td>\n",
       "      <td>3.865644</td>\n",
       "      <td>135</td>\n",
       "      <td>0.054968</td>\n",
       "      <td>118.989447</td>\n",
       "      <td>0.028634</td>\n",
       "      <td>0.006745</td>\n",
       "      <td>82.811459</td>\n",
       "      <td>24</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>91.004725</td>\n",
       "      <td>63.010343</td>\n",
       "      <td>27.994382</td>\n",
       "      <td>207.737074</td>\n",
       "      <td>-374.545620</td>\n",
       "      <td>582.282695</td>\n",
       "      <td>9853.689681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.173449</td>\n",
       "      <td>4.094118</td>\n",
       "      <td>142</td>\n",
       "      <td>0.184904</td>\n",
       "      <td>133.480778</td>\n",
       "      <td>0.033847</td>\n",
       "      <td>0.017674</td>\n",
       "      <td>43.476172</td>\n",
       "      <td>21</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>103.733064</td>\n",
       "      <td>69.975507</td>\n",
       "      <td>33.757557</td>\n",
       "      <td>886.351136</td>\n",
       "      <td>604.765873</td>\n",
       "      <td>281.585263</td>\n",
       "      <td>5803.233337</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.141973</td>\n",
       "      <td>5.331978</td>\n",
       "      <td>98</td>\n",
       "      <td>0.115228</td>\n",
       "      <td>99.317350</td>\n",
       "      <td>0.031519</td>\n",
       "      <td>0.014752</td>\n",
       "      <td>52.445940</td>\n",
       "      <td>19</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>67.648456</td>\n",
       "      <td>60.807614</td>\n",
       "      <td>6.840842</td>\n",
       "      <td>77.249238</td>\n",
       "      <td>-282.042005</td>\n",
       "      <td>359.291243</td>\n",
       "      <td>5208.791736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.193762</td>\n",
       "      <td>4.453382</td>\n",
       "      <td>95</td>\n",
       "      <td>0.092517</td>\n",
       "      <td>103.543415</td>\n",
       "      <td>0.037907</td>\n",
       "      <td>0.014360</td>\n",
       "      <td>92.278968</td>\n",
       "      <td>40</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               alpha      beta  input_flow  value_of_time  \\\n",
       "experiment                                                  \n",
       "2           0.134750  4.642370          85       0.017005   \n",
       "3           0.115907  5.242315         114       0.145378   \n",
       "4           0.178456  3.510139         121       0.067821   \n",
       "5           0.110023  4.887030         127       0.083593   \n",
       "6           0.161977  3.865644         135       0.054968   \n",
       "7           0.173449  4.094118         142       0.184904   \n",
       "8           0.141973  5.331978          98       0.115228   \n",
       "9           0.193762  4.453382          95       0.092517   \n",
       "\n",
       "            unit_cost_expansion  interest_rate  yield_curve  expand_capacity  \\\n",
       "experiment                                                                     \n",
       "2                    143.519760       0.039219     0.003280        24.901018   \n",
       "3                    107.836739       0.025997     0.009659        11.985022   \n",
       "4                    127.838010       0.029066    -0.000839        72.399552   \n",
       "5                    125.799829       0.034673     0.002714        28.565637   \n",
       "6                    118.989447       0.028634     0.006745        82.811459   \n",
       "7                    133.480778       0.033847     0.017674        43.476172   \n",
       "8                     99.317350       0.031519     0.014752        52.445940   \n",
       "9                    103.543415       0.037907     0.014360        92.278968   \n",
       "\n",
       "            amortization_period debt_type  interest_rate_lock  free_flow_time  \\\n",
       "experiment                                                                      \n",
       "2                            36  Rev Bond               False              60   \n",
       "3                            50  Rev Bond                True              60   \n",
       "4                            30     Paygo                True              60   \n",
       "5                            44     Paygo                True              60   \n",
       "6                            24   GO Bond                True              60   \n",
       "7                            21     Paygo               False              60   \n",
       "8                            19   GO Bond               False              60   \n",
       "9                            40  Rev Bond               False              60   \n",
       "\n",
       "            initial_capacity  no_build_travel_time  build_travel_time  \\\n",
       "experiment                                                              \n",
       "2                        100             63.802031          61.354318   \n",
       "3                        100             73.822149          67.635961   \n",
       "4                        100             80.905959          63.090263   \n",
       "5                        100             81.228935          66.217725   \n",
       "6                        100             91.004725          63.010343   \n",
       "7                        100            103.733064          69.975507   \n",
       "8                        100             67.648456          60.807614   \n",
       "9                        100                   NaN                NaN   \n",
       "\n",
       "            time_savings  value_of_time_savings  net_benefits  \\\n",
       "experiment                                                      \n",
       "2               2.447713               3.537924   -175.134011   \n",
       "3               6.186188             102.524606     42.361931   \n",
       "4              17.815696             146.201784   -183.229667   \n",
       "5              15.011210             159.363387     61.567385   \n",
       "6              27.994382             207.737074   -374.545620   \n",
       "7              33.757557             886.351136    604.765873   \n",
       "8               6.840842              77.249238   -282.042005   \n",
       "9                    NaN                    NaN           NaN   \n",
       "\n",
       "            cost_of_capacity_expansion  present_cost_expansion  \n",
       "experiment                                                      \n",
       "2                           178.671935             3573.788162  \n",
       "3                            60.162675             1292.425695  \n",
       "4                           329.431451             9255.414627  \n",
       "5                            97.796001             3593.552248  \n",
       "6                           582.282695             9853.689681  \n",
       "7                           281.585263             5803.233337  \n",
       "8                           359.291243             5208.791736  \n",
       "9                                  NaN                     NaN  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fx.run_experiments(design_name='lhs_1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Re-running Failed Experiments\n",
    "\n",
    "If you pay attention to the logged output, you might notice that one of the \n",
    "experiments (the last one) failed.  We can see `NaN` values in the outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>free_flow_time</th>\n",
       "      <th>initial_capacity</th>\n",
       "      <th>alpha</th>\n",
       "      <th>beta</th>\n",
       "      <th>input_flow</th>\n",
       "      <th>value_of_time</th>\n",
       "      <th>unit_cost_expansion</th>\n",
       "      <th>interest_rate</th>\n",
       "      <th>yield_curve</th>\n",
       "      <th>expand_capacity</th>\n",
       "      <th>amortization_period</th>\n",
       "      <th>debt_type</th>\n",
       "      <th>interest_rate_lock</th>\n",
       "      <th>no_build_travel_time</th>\n",
       "      <th>build_travel_time</th>\n",
       "      <th>time_savings</th>\n",
       "      <th>value_of_time_savings</th>\n",
       "      <th>net_benefits</th>\n",
       "      <th>cost_of_capacity_expansion</th>\n",
       "      <th>present_cost_expansion</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>experiment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.134750</td>\n",
       "      <td>4.642370</td>\n",
       "      <td>85</td>\n",
       "      <td>0.017005</td>\n",
       "      <td>143.519760</td>\n",
       "      <td>0.039219</td>\n",
       "      <td>0.003280</td>\n",
       "      <td>24.901018</td>\n",
       "      <td>36</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>63.802031</td>\n",
       "      <td>61.354318</td>\n",
       "      <td>2.447713</td>\n",
       "      <td>3.537924</td>\n",
       "      <td>-175.134011</td>\n",
       "      <td>178.671935</td>\n",
       "      <td>3573.788162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.115907</td>\n",
       "      <td>5.242315</td>\n",
       "      <td>114</td>\n",
       "      <td>0.145378</td>\n",
       "      <td>107.836739</td>\n",
       "      <td>0.025997</td>\n",
       "      <td>0.009659</td>\n",
       "      <td>11.985022</td>\n",
       "      <td>50</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>73.822149</td>\n",
       "      <td>67.635961</td>\n",
       "      <td>6.186188</td>\n",
       "      <td>102.524606</td>\n",
       "      <td>42.361931</td>\n",
       "      <td>60.162675</td>\n",
       "      <td>1292.425695</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.178456</td>\n",
       "      <td>3.510139</td>\n",
       "      <td>121</td>\n",
       "      <td>0.067821</td>\n",
       "      <td>127.838010</td>\n",
       "      <td>0.029066</td>\n",
       "      <td>-0.000839</td>\n",
       "      <td>72.399552</td>\n",
       "      <td>30</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>80.905959</td>\n",
       "      <td>63.090263</td>\n",
       "      <td>17.815696</td>\n",
       "      <td>146.201784</td>\n",
       "      <td>-183.229667</td>\n",
       "      <td>329.431451</td>\n",
       "      <td>9255.414627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.110023</td>\n",
       "      <td>4.887030</td>\n",
       "      <td>127</td>\n",
       "      <td>0.083593</td>\n",
       "      <td>125.799829</td>\n",
       "      <td>0.034673</td>\n",
       "      <td>0.002714</td>\n",
       "      <td>28.565637</td>\n",
       "      <td>44</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>81.228935</td>\n",
       "      <td>66.217725</td>\n",
       "      <td>15.011210</td>\n",
       "      <td>159.363387</td>\n",
       "      <td>61.567385</td>\n",
       "      <td>97.796001</td>\n",
       "      <td>3593.552248</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.161977</td>\n",
       "      <td>3.865644</td>\n",
       "      <td>135</td>\n",
       "      <td>0.054968</td>\n",
       "      <td>118.989447</td>\n",
       "      <td>0.028634</td>\n",
       "      <td>0.006745</td>\n",
       "      <td>82.811459</td>\n",
       "      <td>24</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>91.004725</td>\n",
       "      <td>63.010343</td>\n",
       "      <td>27.994382</td>\n",
       "      <td>207.737074</td>\n",
       "      <td>-374.545620</td>\n",
       "      <td>582.282695</td>\n",
       "      <td>9853.689681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.173449</td>\n",
       "      <td>4.094118</td>\n",
       "      <td>142</td>\n",
       "      <td>0.184904</td>\n",
       "      <td>133.480778</td>\n",
       "      <td>0.033847</td>\n",
       "      <td>0.017674</td>\n",
       "      <td>43.476172</td>\n",
       "      <td>21</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>103.733064</td>\n",
       "      <td>69.975507</td>\n",
       "      <td>33.757557</td>\n",
       "      <td>886.351136</td>\n",
       "      <td>604.765873</td>\n",
       "      <td>281.585263</td>\n",
       "      <td>5803.233337</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.141973</td>\n",
       "      <td>5.331978</td>\n",
       "      <td>98</td>\n",
       "      <td>0.115228</td>\n",
       "      <td>99.317350</td>\n",
       "      <td>0.031519</td>\n",
       "      <td>0.014752</td>\n",
       "      <td>52.445940</td>\n",
       "      <td>19</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>67.648456</td>\n",
       "      <td>60.807614</td>\n",
       "      <td>6.840842</td>\n",
       "      <td>77.249238</td>\n",
       "      <td>-282.042005</td>\n",
       "      <td>359.291243</td>\n",
       "      <td>5208.791736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.193762</td>\n",
       "      <td>4.453382</td>\n",
       "      <td>95</td>\n",
       "      <td>0.092517</td>\n",
       "      <td>103.543415</td>\n",
       "      <td>0.037907</td>\n",
       "      <td>0.014360</td>\n",
       "      <td>92.278968</td>\n",
       "      <td>40</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            free_flow_time  initial_capacity     alpha      beta  input_flow  \\\n",
       "experiment                                                                     \n",
       "2                       60               100  0.134750  4.642370          85   \n",
       "3                       60               100  0.115907  5.242315         114   \n",
       "4                       60               100  0.178456  3.510139         121   \n",
       "5                       60               100  0.110023  4.887030         127   \n",
       "6                       60               100  0.161977  3.865644         135   \n",
       "7                       60               100  0.173449  4.094118         142   \n",
       "8                       60               100  0.141973  5.331978          98   \n",
       "9                       60               100  0.193762  4.453382          95   \n",
       "\n",
       "            value_of_time  unit_cost_expansion  interest_rate  yield_curve  \\\n",
       "experiment                                                                   \n",
       "2                0.017005           143.519760       0.039219     0.003280   \n",
       "3                0.145378           107.836739       0.025997     0.009659   \n",
       "4                0.067821           127.838010       0.029066    -0.000839   \n",
       "5                0.083593           125.799829       0.034673     0.002714   \n",
       "6                0.054968           118.989447       0.028634     0.006745   \n",
       "7                0.184904           133.480778       0.033847     0.017674   \n",
       "8                0.115228            99.317350       0.031519     0.014752   \n",
       "9                0.092517           103.543415       0.037907     0.014360   \n",
       "\n",
       "            expand_capacity  amortization_period debt_type  \\\n",
       "experiment                                                   \n",
       "2                 24.901018                   36  Rev Bond   \n",
       "3                 11.985022                   50  Rev Bond   \n",
       "4                 72.399552                   30     Paygo   \n",
       "5                 28.565637                   44     Paygo   \n",
       "6                 82.811459                   24   GO Bond   \n",
       "7                 43.476172                   21     Paygo   \n",
       "8                 52.445940                   19   GO Bond   \n",
       "9                 92.278968                   40  Rev Bond   \n",
       "\n",
       "            interest_rate_lock  no_build_travel_time  build_travel_time  \\\n",
       "experiment                                                                \n",
       "2                        False             63.802031          61.354318   \n",
       "3                         True             73.822149          67.635961   \n",
       "4                         True             80.905959          63.090263   \n",
       "5                         True             81.228935          66.217725   \n",
       "6                         True             91.004725          63.010343   \n",
       "7                        False            103.733064          69.975507   \n",
       "8                        False             67.648456          60.807614   \n",
       "9                        False                   NaN                NaN   \n",
       "\n",
       "            time_savings  value_of_time_savings  net_benefits  \\\n",
       "experiment                                                      \n",
       "2               2.447713               3.537924   -175.134011   \n",
       "3               6.186188             102.524606     42.361931   \n",
       "4              17.815696             146.201784   -183.229667   \n",
       "5              15.011210             159.363387     61.567385   \n",
       "6              27.994382             207.737074   -374.545620   \n",
       "7              33.757557             886.351136    604.765873   \n",
       "8               6.840842              77.249238   -282.042005   \n",
       "9                    NaN                    NaN           NaN   \n",
       "\n",
       "            cost_of_capacity_expansion  present_cost_expansion  \n",
       "experiment                                                      \n",
       "2                           178.671935             3573.788162  \n",
       "3                            60.162675             1292.425695  \n",
       "4                           329.431451             9255.414627  \n",
       "5                            97.796001             3593.552248  \n",
       "6                           582.282695             9853.689681  \n",
       "7                           281.585263             5803.233337  \n",
       "8                           359.291243             5208.791736  \n",
       "9                                  NaN                     NaN  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = fx.db.read_experiment_all(fx.scope, 'lhs_1')\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can collect the id's of the failed experiments programmatically. To collect all the experiments that are\n",
    "missing any performance measure output, we can do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([9], dtype='int64', name='experiment')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fails = results.isna().any(axis=1)\n",
    "failed_experiment_ids = fails.index[fails]\n",
    "failed_experiment_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When there is an error (thrown as a `subprocess.CalledProcessError`)\n",
    "during the execution of a `FilesCoreModel`, the output from stdout\n",
    "and stderr are written to log files in the archive location, instead\n",
    "of having the legit model outputs written there.\n",
    "\n",
    "We can see the log output by reading in the log file, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2021-01-31 19:00:10,065] emat.RoadTest.INFO: running emat-road-test-demo\n",
      "[2021-01-31 19:00:10,068] emat.RoadTest.ERROR: Random crash, ha ha!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "error_log = os.path.join(\n",
    "    fx.get_experiment_archive_path(9), \n",
    "    'error.stdout.log'\n",
    ")\n",
    "with open(error_log, 'r') as stdout:\n",
    "    error_log_content = stdout.read()\n",
    "    \n",
    "print(error_log_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see the log file is explicitly taunting us about \n",
    "randomly crashing the model run.  That's fine -- we wanted to\n",
    "crash the execution randomly to show what to do in this event, cause it happens \n",
    "sometimes.  Maybe a disk filled up, or there is an intermittent\n",
    "license problem that causes a failure one in a while.  If that's the\n",
    "case and we can fix it just by re-running, awesome!\n",
    "\n",
    "We can load just the failed experiments to try them again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>free_flow_time</th>\n",
       "      <th>initial_capacity</th>\n",
       "      <th>alpha</th>\n",
       "      <th>beta</th>\n",
       "      <th>input_flow</th>\n",
       "      <th>value_of_time</th>\n",
       "      <th>unit_cost_expansion</th>\n",
       "      <th>interest_rate</th>\n",
       "      <th>yield_curve</th>\n",
       "      <th>expand_capacity</th>\n",
       "      <th>amortization_period</th>\n",
       "      <th>debt_type</th>\n",
       "      <th>interest_rate_lock</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>experiment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.193762</td>\n",
       "      <td>4.453382</td>\n",
       "      <td>95</td>\n",
       "      <td>0.092517</td>\n",
       "      <td>103.543415</td>\n",
       "      <td>0.037907</td>\n",
       "      <td>0.01436</td>\n",
       "      <td>92.278968</td>\n",
       "      <td>40</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            free_flow_time  initial_capacity     alpha      beta  input_flow  \\\n",
       "experiment                                                                     \n",
       "9                       60               100  0.193762  4.453382          95   \n",
       "\n",
       "            value_of_time  unit_cost_expansion  interest_rate  yield_curve  \\\n",
       "experiment                                                                   \n",
       "9                0.092517           103.543415       0.037907      0.01436   \n",
       "\n",
       "            expand_capacity  amortization_period debt_type  interest_rate_lock  \n",
       "experiment                                                                      \n",
       "9                 92.278968                   40  Rev Bond               False  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "failed_experiments = fx.read_experiment_parameters(experiment_ids=failed_experiment_ids)\n",
    "failed_experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, there is a \"short circuit\" process that will\n",
    "prevent re-running a core model experiment, instead the performance measure results\n",
    "will simply be loaded from the database, which is typically much faster than\n",
    "actually running the core model.  But, if the performance measures stored in the\n",
    "database are junk, we will not want to trigger the short circuit system, and\n",
    "actually run the full core model again.  To do so, we can disable the\n",
    "short circuit like and re-run the failed experiment.  If it failed because of a transient error, \n",
    "e.g. a disk space problem that's been fixed, then perhaps we can simply re-run the model\n",
    "and it will work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:26.53] MainProcess/INFO: performing 1 scenarios/policies * 1 model(s) = 1 experiments\n",
      "[00:26.54] MainProcess/INFO: performing experiments sequentially\n",
      "[00:26.54] MainProcess/INFO: RoadTestFileModel SETUP RUNID-d53da118-6428-11eb-8c2b-acde48001122\n",
      "[00:26.55] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 9 RUNID-d53da118-6428-11eb-8c2b-acde48001122\n",
      "[00:28.93] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-d53da118-6428-11eb-8c2b-acde48001122\n",
      "[00:28.94] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-d53da118-6428-11eb-8c2b-acde48001122\n",
      "[00:28.95] MainProcess/INFO: RoadTestFileModel ARCHIVE\n",
      " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/road-test-files\n",
      "   to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmpygcky5rc/archive/scp_EMAT Road Test/exp_009_d53da118-6428-11eb-8c2b-acde48001122\n",
      "[00:28.96] MainProcess/INFO: RAN EXPERIMENT IN 2.42 SECONDS\n",
      "[00:28.96] MainProcess/INFO: 1 cases completed\n",
      "[00:28.96] MainProcess/INFO: experiments finished\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>beta</th>\n",
       "      <th>input_flow</th>\n",
       "      <th>value_of_time</th>\n",
       "      <th>unit_cost_expansion</th>\n",
       "      <th>interest_rate</th>\n",
       "      <th>yield_curve</th>\n",
       "      <th>expand_capacity</th>\n",
       "      <th>amortization_period</th>\n",
       "      <th>debt_type</th>\n",
       "      <th>interest_rate_lock</th>\n",
       "      <th>free_flow_time</th>\n",
       "      <th>initial_capacity</th>\n",
       "      <th>no_build_travel_time</th>\n",
       "      <th>build_travel_time</th>\n",
       "      <th>time_savings</th>\n",
       "      <th>value_of_time_savings</th>\n",
       "      <th>net_benefits</th>\n",
       "      <th>cost_of_capacity_expansion</th>\n",
       "      <th>present_cost_expansion</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>experiment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.193762</td>\n",
       "      <td>4.453382</td>\n",
       "      <td>95</td>\n",
       "      <td>0.092517</td>\n",
       "      <td>103.543415</td>\n",
       "      <td>0.037907</td>\n",
       "      <td>0.01436</td>\n",
       "      <td>92.278968</td>\n",
       "      <td>40</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>69.251572</td>\n",
       "      <td>60.503221</td>\n",
       "      <td>8.748351</td>\n",
       "      <td>76.890662</td>\n",
       "      <td>-385.524839</td>\n",
       "      <td>462.4155</td>\n",
       "      <td>9554.87952</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               alpha      beta  input_flow  value_of_time  \\\n",
       "experiment                                                  \n",
       "9           0.193762  4.453382          95       0.092517   \n",
       "\n",
       "            unit_cost_expansion  interest_rate  yield_curve  expand_capacity  \\\n",
       "experiment                                                                     \n",
       "9                    103.543415       0.037907      0.01436        92.278968   \n",
       "\n",
       "            amortization_period debt_type  interest_rate_lock  free_flow_time  \\\n",
       "experiment                                                                      \n",
       "9                            40  Rev Bond               False              60   \n",
       "\n",
       "            initial_capacity  no_build_travel_time  build_travel_time  \\\n",
       "experiment                                                              \n",
       "9                        100             69.251572          60.503221   \n",
       "\n",
       "            time_savings  value_of_time_savings  net_benefits  \\\n",
       "experiment                                                      \n",
       "9               8.748351              76.890662   -385.524839   \n",
       "\n",
       "            cost_of_capacity_expansion  present_cost_expansion  \n",
       "experiment                                                      \n",
       "9                             462.4155              9554.87952  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fx.run_experiments(failed_experiments, allow_short_circuit=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much better!  Now we can see we have a more complete set of outputs, without the NaN's.  Hooray!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>free_flow_time</th>\n",
       "      <th>initial_capacity</th>\n",
       "      <th>alpha</th>\n",
       "      <th>beta</th>\n",
       "      <th>input_flow</th>\n",
       "      <th>value_of_time</th>\n",
       "      <th>unit_cost_expansion</th>\n",
       "      <th>interest_rate</th>\n",
       "      <th>yield_curve</th>\n",
       "      <th>expand_capacity</th>\n",
       "      <th>amortization_period</th>\n",
       "      <th>debt_type</th>\n",
       "      <th>interest_rate_lock</th>\n",
       "      <th>no_build_travel_time</th>\n",
       "      <th>build_travel_time</th>\n",
       "      <th>time_savings</th>\n",
       "      <th>value_of_time_savings</th>\n",
       "      <th>net_benefits</th>\n",
       "      <th>cost_of_capacity_expansion</th>\n",
       "      <th>present_cost_expansion</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>experiment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.134750</td>\n",
       "      <td>4.642370</td>\n",
       "      <td>85</td>\n",
       "      <td>0.017005</td>\n",
       "      <td>143.519760</td>\n",
       "      <td>0.039219</td>\n",
       "      <td>0.003280</td>\n",
       "      <td>24.901018</td>\n",
       "      <td>36</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>63.802031</td>\n",
       "      <td>61.354318</td>\n",
       "      <td>2.447713</td>\n",
       "      <td>3.537924</td>\n",
       "      <td>-175.134011</td>\n",
       "      <td>178.671935</td>\n",
       "      <td>3573.788162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.115907</td>\n",
       "      <td>5.242315</td>\n",
       "      <td>114</td>\n",
       "      <td>0.145378</td>\n",
       "      <td>107.836739</td>\n",
       "      <td>0.025997</td>\n",
       "      <td>0.009659</td>\n",
       "      <td>11.985022</td>\n",
       "      <td>50</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>73.822149</td>\n",
       "      <td>67.635961</td>\n",
       "      <td>6.186188</td>\n",
       "      <td>102.524606</td>\n",
       "      <td>42.361931</td>\n",
       "      <td>60.162675</td>\n",
       "      <td>1292.425695</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.178456</td>\n",
       "      <td>3.510139</td>\n",
       "      <td>121</td>\n",
       "      <td>0.067821</td>\n",
       "      <td>127.838010</td>\n",
       "      <td>0.029066</td>\n",
       "      <td>-0.000839</td>\n",
       "      <td>72.399552</td>\n",
       "      <td>30</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>80.905959</td>\n",
       "      <td>63.090263</td>\n",
       "      <td>17.815696</td>\n",
       "      <td>146.201784</td>\n",
       "      <td>-183.229667</td>\n",
       "      <td>329.431451</td>\n",
       "      <td>9255.414627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.110023</td>\n",
       "      <td>4.887030</td>\n",
       "      <td>127</td>\n",
       "      <td>0.083593</td>\n",
       "      <td>125.799829</td>\n",
       "      <td>0.034673</td>\n",
       "      <td>0.002714</td>\n",
       "      <td>28.565637</td>\n",
       "      <td>44</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>81.228935</td>\n",
       "      <td>66.217725</td>\n",
       "      <td>15.011210</td>\n",
       "      <td>159.363387</td>\n",
       "      <td>61.567385</td>\n",
       "      <td>97.796001</td>\n",
       "      <td>3593.552248</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.161977</td>\n",
       "      <td>3.865644</td>\n",
       "      <td>135</td>\n",
       "      <td>0.054968</td>\n",
       "      <td>118.989447</td>\n",
       "      <td>0.028634</td>\n",
       "      <td>0.006745</td>\n",
       "      <td>82.811459</td>\n",
       "      <td>24</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>91.004725</td>\n",
       "      <td>63.010343</td>\n",
       "      <td>27.994382</td>\n",
       "      <td>207.737074</td>\n",
       "      <td>-374.545620</td>\n",
       "      <td>582.282695</td>\n",
       "      <td>9853.689681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.173449</td>\n",
       "      <td>4.094118</td>\n",
       "      <td>142</td>\n",
       "      <td>0.184904</td>\n",
       "      <td>133.480778</td>\n",
       "      <td>0.033847</td>\n",
       "      <td>0.017674</td>\n",
       "      <td>43.476172</td>\n",
       "      <td>21</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>103.733064</td>\n",
       "      <td>69.975507</td>\n",
       "      <td>33.757557</td>\n",
       "      <td>886.351136</td>\n",
       "      <td>604.765873</td>\n",
       "      <td>281.585263</td>\n",
       "      <td>5803.233337</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.141973</td>\n",
       "      <td>5.331978</td>\n",
       "      <td>98</td>\n",
       "      <td>0.115228</td>\n",
       "      <td>99.317350</td>\n",
       "      <td>0.031519</td>\n",
       "      <td>0.014752</td>\n",
       "      <td>52.445940</td>\n",
       "      <td>19</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>67.648456</td>\n",
       "      <td>60.807614</td>\n",
       "      <td>6.840842</td>\n",
       "      <td>77.249238</td>\n",
       "      <td>-282.042005</td>\n",
       "      <td>359.291243</td>\n",
       "      <td>5208.791736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.193762</td>\n",
       "      <td>4.453382</td>\n",
       "      <td>95</td>\n",
       "      <td>0.092517</td>\n",
       "      <td>103.543415</td>\n",
       "      <td>0.037907</td>\n",
       "      <td>0.014360</td>\n",
       "      <td>92.278968</td>\n",
       "      <td>40</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>69.251572</td>\n",
       "      <td>60.503221</td>\n",
       "      <td>8.748351</td>\n",
       "      <td>76.890662</td>\n",
       "      <td>-385.524839</td>\n",
       "      <td>462.415500</td>\n",
       "      <td>9554.879520</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            free_flow_time  initial_capacity     alpha      beta  input_flow  \\\n",
       "experiment                                                                     \n",
       "2                       60               100  0.134750  4.642370          85   \n",
       "3                       60               100  0.115907  5.242315         114   \n",
       "4                       60               100  0.178456  3.510139         121   \n",
       "5                       60               100  0.110023  4.887030         127   \n",
       "6                       60               100  0.161977  3.865644         135   \n",
       "7                       60               100  0.173449  4.094118         142   \n",
       "8                       60               100  0.141973  5.331978          98   \n",
       "9                       60               100  0.193762  4.453382          95   \n",
       "\n",
       "            value_of_time  unit_cost_expansion  interest_rate  yield_curve  \\\n",
       "experiment                                                                   \n",
       "2                0.017005           143.519760       0.039219     0.003280   \n",
       "3                0.145378           107.836739       0.025997     0.009659   \n",
       "4                0.067821           127.838010       0.029066    -0.000839   \n",
       "5                0.083593           125.799829       0.034673     0.002714   \n",
       "6                0.054968           118.989447       0.028634     0.006745   \n",
       "7                0.184904           133.480778       0.033847     0.017674   \n",
       "8                0.115228            99.317350       0.031519     0.014752   \n",
       "9                0.092517           103.543415       0.037907     0.014360   \n",
       "\n",
       "            expand_capacity  amortization_period debt_type  \\\n",
       "experiment                                                   \n",
       "2                 24.901018                   36  Rev Bond   \n",
       "3                 11.985022                   50  Rev Bond   \n",
       "4                 72.399552                   30     Paygo   \n",
       "5                 28.565637                   44     Paygo   \n",
       "6                 82.811459                   24   GO Bond   \n",
       "7                 43.476172                   21     Paygo   \n",
       "8                 52.445940                   19   GO Bond   \n",
       "9                 92.278968                   40  Rev Bond   \n",
       "\n",
       "            interest_rate_lock  no_build_travel_time  build_travel_time  \\\n",
       "experiment                                                                \n",
       "2                        False             63.802031          61.354318   \n",
       "3                         True             73.822149          67.635961   \n",
       "4                         True             80.905959          63.090263   \n",
       "5                         True             81.228935          66.217725   \n",
       "6                         True             91.004725          63.010343   \n",
       "7                        False            103.733064          69.975507   \n",
       "8                        False             67.648456          60.807614   \n",
       "9                        False             69.251572          60.503221   \n",
       "\n",
       "            time_savings  value_of_time_savings  net_benefits  \\\n",
       "experiment                                                      \n",
       "2               2.447713               3.537924   -175.134011   \n",
       "3               6.186188             102.524606     42.361931   \n",
       "4              17.815696             146.201784   -183.229667   \n",
       "5              15.011210             159.363387     61.567385   \n",
       "6              27.994382             207.737074   -374.545620   \n",
       "7              33.757557             886.351136    604.765873   \n",
       "8               6.840842              77.249238   -282.042005   \n",
       "9               8.748351              76.890662   -385.524839   \n",
       "\n",
       "            cost_of_capacity_expansion  present_cost_expansion  \n",
       "experiment                                                      \n",
       "2                           178.671935             3573.788162  \n",
       "3                            60.162675             1292.425695  \n",
       "4                           329.431451             9255.414627  \n",
       "5                            97.796001             3593.552248  \n",
       "6                           582.282695             9853.689681  \n",
       "7                           281.585263             5803.233337  \n",
       "8                           359.291243             5208.791736  \n",
       "9                           462.415500             9554.879520  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = fx.db.read_experiment_all(scope_name=fx.scope.name, design_name='lhs_1')\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiprocessing for Running Multiple Experiments\n",
    "\n",
    "The examples above are all single-process demonstrations of using TMIP-EMAT to run core model\n",
    "experiments.  If your core model itself is multi-threaded or otherwise is designed to make \n",
    "full use of your multi-core CPU, or if a single core model run will otherwise max out some\n",
    "computational resource (e.g. RAM, disk space) then single process operation should be sufficient.\n",
    "\n",
    "If, on the other hand, your core model is such that you can run multiple independent instances of\n",
    "the model side-by-side on the same machine, then you could benefit from a multiprocessing \n",
    "approach.  This can be accomplished by splitting a design of experiments over several\n",
    "processes that you start manually, or by using an automatic multiprocessing library such as \n",
    "`dask.distributed`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running a Subset of Experiments Manually\n",
    "\n",
    "Suppose, for example, you wanted to distribute the workload of running experiments over several processes,\n",
    "or even over several computers. If each process has file system access to the same TMIP-EMAT database of\n",
    "experiments, we can orchestrate these experiments in parallel by manually splitting up the processes.\n",
    "\n",
    "To begin with, we'll have one process create a complete design of experiments, and save it to the \n",
    "database (which happens automatically here)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "design2 = fx.design_experiments(design_name='lhs_2', n_samples=8, random_seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can create set up a copy of the same model in a different process, even on a different\n",
    "machine, as long as we point back to the same original database file. This implies the different\n",
    "process has access to the file system where the original file is stored. It is valuable to\n",
    "read and write to the same database file, not just a copy of the file, as this will obviate the need\n",
    "to sync the experimental data manually afterwards.  In this demo, we'll \n",
    "just create a new directory to work in, but we'll point to the database in the original directory.\n",
    "Instead of allowing our model to implicitly create a new database file in the new directory, we'll\n",
    "instantiate a SQLiteDB object pointing to the original database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:29.10] MainProcess/INFO: running script emat_db_init.sql\n",
      "[00:29.10] MainProcess/INFO: running script meta_model.sql\n",
      "[00:29.11] MainProcess/INFO: found no experiments with missing run_id's\n",
      "[00:29.11] MainProcess/INFO: running script emat_db_init_views.sql\n"
     ]
    }
   ],
   "source": [
    "database_filename = fx.db.database_path\n",
    "db2 = emat.SQLiteDB(database_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, `db2` is a `emat.SQLiteDB` object, which wraps a *new* connection to the *original* database.\n",
    "Then, we'll pass that `db2` explicitly to the new `RoadTestFileModel` constructor, which will\n",
    "create a complete copy of our model (other than the database) in a new directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:29.12] MainProcess/WARNING: changing cwd to /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmph_ksfgon\n"
     ]
    }
   ],
   "source": [
    "fx2 = core_files_demo.RoadTestFileModel(db=db2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run a particular slice of a design of experiments, we need to load the experimental design first, \n",
    "and then pass that slice to the `run_experiments` function, instead of just giving the `design_name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "design2 = fx.read_experiment_parameters('lhs_2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For splitting the work across a number of similarly capable processes or machines,\n",
    "the double-colon slice is convenient.  If, for example, you are splitting the work\n",
    "over 4 computers, you can run each with slices `0::4`, `1::4`, `2::4`, and `3::4`.\n",
    "This slices in skip-step manner, so slice below will run every 4th experiment\n",
    "from the design, starting with experiment index 0 (i.e. the first one).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:29.19] MainProcess/INFO: performing 2 scenarios/policies * 1 model(s) = 2 experiments\n",
      "[00:29.21] MainProcess/INFO: performing experiments sequentially\n",
      "[00:29.21] MainProcess/INFO: RoadTestFileModel SETUP RUNID-d6d5244c-6428-11eb-8c2b-acde48001122\n",
      "[00:29.22] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 10 RUNID-d6d5244c-6428-11eb-8c2b-acde48001122\n",
      "[00:31.69] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-d6d5244c-6428-11eb-8c2b-acde48001122\n",
      "[00:31.70] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-d6d5244c-6428-11eb-8c2b-acde48001122\n",
      "[00:31.72] MainProcess/INFO: RoadTestFileModel ARCHIVE\n",
      " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmph_ksfgon/road-test-files\n",
      "   to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmph_ksfgon/archive/scp_EMAT Road Test/exp_010_d6d5244c-6428-11eb-8c2b-acde48001122\n",
      "[00:31.72] MainProcess/INFO: RAN EXPERIMENT IN 2.51 SECONDS\n",
      "[00:31.72] MainProcess/INFO: 1 cases completed\n",
      "[00:31.73] MainProcess/INFO: RoadTestFileModel SETUP RUNID-d85582c6-6428-11eb-8c2b-acde48001122\n",
      "[00:31.74] MainProcess/INFO: RoadTestFileModel SETUP complete experiment_id 14 RUNID-d85582c6-6428-11eb-8c2b-acde48001122\n",
      "[00:34.21] MainProcess/INFO: RoadTestFileModel RUN complete RUNID-d85582c6-6428-11eb-8c2b-acde48001122\n",
      "[00:34.22] MainProcess/INFO: RoadTestFileModel POST-PROCESS complete RUNID-d85582c6-6428-11eb-8c2b-acde48001122\n",
      "[00:34.23] MainProcess/INFO: RoadTestFileModel ARCHIVE\n",
      " from: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmph_ksfgon/road-test-files\n",
      "   to: /var/folders/js/bk_dt9015j79_f6bxnc44dsr0000gp/T/tmph_ksfgon/archive/scp_EMAT Road Test/exp_014_d85582c6-6428-11eb-8c2b-acde48001122\n",
      "[00:34.24] MainProcess/INFO: RAN EXPERIMENT IN 2.51 SECONDS\n",
      "[00:34.24] MainProcess/INFO: 2 cases completed\n",
      "[00:34.24] MainProcess/INFO: experiments finished\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>beta</th>\n",
       "      <th>input_flow</th>\n",
       "      <th>value_of_time</th>\n",
       "      <th>unit_cost_expansion</th>\n",
       "      <th>interest_rate</th>\n",
       "      <th>yield_curve</th>\n",
       "      <th>expand_capacity</th>\n",
       "      <th>amortization_period</th>\n",
       "      <th>debt_type</th>\n",
       "      <th>interest_rate_lock</th>\n",
       "      <th>free_flow_time</th>\n",
       "      <th>initial_capacity</th>\n",
       "      <th>no_build_travel_time</th>\n",
       "      <th>build_travel_time</th>\n",
       "      <th>time_savings</th>\n",
       "      <th>value_of_time_savings</th>\n",
       "      <th>net_benefits</th>\n",
       "      <th>cost_of_capacity_expansion</th>\n",
       "      <th>present_cost_expansion</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>experiment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.114450</td>\n",
       "      <td>3.648104</td>\n",
       "      <td>100</td>\n",
       "      <td>0.034746</td>\n",
       "      <td>120.196432</td>\n",
       "      <td>0.028321</td>\n",
       "      <td>0.001514</td>\n",
       "      <td>5.621927</td>\n",
       "      <td>34</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>66.867014</td>\n",
       "      <td>65.624845</td>\n",
       "      <td>1.242169</td>\n",
       "      <td>4.316063</td>\n",
       "      <td>-17.502784</td>\n",
       "      <td>21.818848</td>\n",
       "      <td>675.735529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.132514</td>\n",
       "      <td>4.016263</td>\n",
       "      <td>133</td>\n",
       "      <td>0.107612</td>\n",
       "      <td>131.922290</td>\n",
       "      <td>0.035993</td>\n",
       "      <td>0.015811</td>\n",
       "      <td>54.081760</td>\n",
       "      <td>22</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>84.993873</td>\n",
       "      <td>64.403269</td>\n",
       "      <td>20.590604</td>\n",
       "      <td>294.700221</td>\n",
       "      <td>-37.110353</td>\n",
       "      <td>331.810575</td>\n",
       "      <td>7134.589600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               alpha      beta  input_flow  value_of_time  \\\n",
       "experiment                                                  \n",
       "10          0.114450  3.648104         100       0.034746   \n",
       "14          0.132514  4.016263         133       0.107612   \n",
       "\n",
       "            unit_cost_expansion  interest_rate  yield_curve  expand_capacity  \\\n",
       "experiment                                                                     \n",
       "10                   120.196432       0.028321     0.001514         5.621927   \n",
       "14                   131.922290       0.035993     0.015811        54.081760   \n",
       "\n",
       "            amortization_period debt_type  interest_rate_lock  free_flow_time  \\\n",
       "experiment                                                                      \n",
       "10                           34     Paygo                True              60   \n",
       "14                           22     Paygo               False              60   \n",
       "\n",
       "            initial_capacity  no_build_travel_time  build_travel_time  \\\n",
       "experiment                                                              \n",
       "10                       100             66.867014          65.624845   \n",
       "14                       100             84.993873          64.403269   \n",
       "\n",
       "            time_savings  value_of_time_savings  net_benefits  \\\n",
       "experiment                                                      \n",
       "10              1.242169               4.316063    -17.502784   \n",
       "14             20.590604             294.700221    -37.110353   \n",
       "\n",
       "            cost_of_capacity_expansion  present_cost_expansion  \n",
       "experiment                                                      \n",
       "10                           21.818848              675.735529  \n",
       "14                          331.810575             7134.589600  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fx2.run_experiments(design2.iloc[0::4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we have linked the second model instance back to the same database, after\n",
    "these experiments have finished we can access the results from the original `fx`\n",
    "instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>no_build_travel_time</th>\n",
       "      <th>build_travel_time</th>\n",
       "      <th>time_savings</th>\n",
       "      <th>value_of_time_savings</th>\n",
       "      <th>net_benefits</th>\n",
       "      <th>cost_of_capacity_expansion</th>\n",
       "      <th>present_cost_expansion</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>experiment</th>\n",
       "      <th>run</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <th>d6d5244c-6428-11eb-8c2b-acde48001122</th>\n",
       "      <td>66.867014</td>\n",
       "      <td>65.624845</td>\n",
       "      <td>1.242169</td>\n",
       "      <td>4.316063</td>\n",
       "      <td>-17.502784</td>\n",
       "      <td>21.818848</td>\n",
       "      <td>675.735529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <th>d85582c6-6428-11eb-8c2b-acde48001122</th>\n",
       "      <td>84.993873</td>\n",
       "      <td>64.403269</td>\n",
       "      <td>20.590604</td>\n",
       "      <td>294.700221</td>\n",
       "      <td>-37.110353</td>\n",
       "      <td>331.810575</td>\n",
       "      <td>7134.589600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 no_build_travel_time  \\\n",
       "experiment run                                                          \n",
       "10         d6d5244c-6428-11eb-8c2b-acde48001122             66.867014   \n",
       "14         d85582c6-6428-11eb-8c2b-acde48001122             84.993873   \n",
       "\n",
       "                                                 build_travel_time  \\\n",
       "experiment run                                                       \n",
       "10         d6d5244c-6428-11eb-8c2b-acde48001122          65.624845   \n",
       "14         d85582c6-6428-11eb-8c2b-acde48001122          64.403269   \n",
       "\n",
       "                                                 time_savings  \\\n",
       "experiment run                                                  \n",
       "10         d6d5244c-6428-11eb-8c2b-acde48001122      1.242169   \n",
       "14         d85582c6-6428-11eb-8c2b-acde48001122     20.590604   \n",
       "\n",
       "                                                 value_of_time_savings  \\\n",
       "experiment run                                                           \n",
       "10         d6d5244c-6428-11eb-8c2b-acde48001122               4.316063   \n",
       "14         d85582c6-6428-11eb-8c2b-acde48001122             294.700221   \n",
       "\n",
       "                                                 net_benefits  \\\n",
       "experiment run                                                  \n",
       "10         d6d5244c-6428-11eb-8c2b-acde48001122    -17.502784   \n",
       "14         d85582c6-6428-11eb-8c2b-acde48001122    -37.110353   \n",
       "\n",
       "                                                 cost_of_capacity_expansion  \\\n",
       "experiment run                                                                \n",
       "10         d6d5244c-6428-11eb-8c2b-acde48001122                   21.818848   \n",
       "14         d85582c6-6428-11eb-8c2b-acde48001122                  331.810575   \n",
       "\n",
       "                                                 present_cost_expansion  \n",
       "experiment run                                                           \n",
       "10         d6d5244c-6428-11eb-8c2b-acde48001122              675.735529  \n",
       "14         d85582c6-6428-11eb-8c2b-acde48001122             7134.589600  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fx.read_experiment_measures('lhs_2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to note that for this manual multiprocessing technique to work, where\n",
    "different processes run the model simultaneously, each process must be in a seperate \n",
    "Python instance (e.g. in seperate Jupyter notebooks, not in the same notebook as shown here)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Automatic Multiprocessing for Running Multiple Experiments\n",
    "\n",
    "The examples above are all essentially single-process demonstrations of using TMIP-EMAT to run core model\n",
    "experiments, either by running all in one single process, or by having a user manually instantiate a number \n",
    "of single processes.  If your core model itself is multi-threaded or otherwise is designed to make \n",
    "full use of your multi-core CPU, or if a single core model run will otherwise max out some\n",
    "computational resource (e.g. RAM, disk space) then single process operation should be sufficient.\n",
    "\n",
    "If, on the other hand, your model is such that you can run multiple independent instances of\n",
    "the model side-by-side on the same machine, but you don't want to manage the process of manually, \n",
    "then you could benefit from a multiprocessing approach that uses the `dask.distributed` library.  To\n",
    "demonstrate this, we'll create yet another small design of experiments to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>amortization_period</th>\n",
       "      <th>beta</th>\n",
       "      <th>debt_type</th>\n",
       "      <th>expand_capacity</th>\n",
       "      <th>input_flow</th>\n",
       "      <th>interest_rate</th>\n",
       "      <th>interest_rate_lock</th>\n",
       "      <th>unit_cost_expansion</th>\n",
       "      <th>value_of_time</th>\n",
       "      <th>yield_curve</th>\n",
       "      <th>free_flow_time</th>\n",
       "      <th>initial_capacity</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>experiment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.151341</td>\n",
       "      <td>48</td>\n",
       "      <td>3.676012</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>34.248792</td>\n",
       "      <td>139</td>\n",
       "      <td>0.037682</td>\n",
       "      <td>False</td>\n",
       "      <td>128.627672</td>\n",
       "      <td>0.004817</td>\n",
       "      <td>0.012079</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.118509</td>\n",
       "      <td>41</td>\n",
       "      <td>5.309555</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>76.473048</td>\n",
       "      <td>89</td>\n",
       "      <td>0.035629</td>\n",
       "      <td>True</td>\n",
       "      <td>125.994370</td>\n",
       "      <td>0.091859</td>\n",
       "      <td>0.005987</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.174355</td>\n",
       "      <td>18</td>\n",
       "      <td>4.727608</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>94.255893</td>\n",
       "      <td>131</td>\n",
       "      <td>0.030304</td>\n",
       "      <td>False</td>\n",
       "      <td>139.257007</td>\n",
       "      <td>0.076668</td>\n",
       "      <td>0.016896</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0.188622</td>\n",
       "      <td>26</td>\n",
       "      <td>3.703274</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>96.919294</td>\n",
       "      <td>125</td>\n",
       "      <td>0.036478</td>\n",
       "      <td>False</td>\n",
       "      <td>121.126672</td>\n",
       "      <td>0.110190</td>\n",
       "      <td>0.016048</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.160916</td>\n",
       "      <td>33</td>\n",
       "      <td>4.447292</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>64.247841</td>\n",
       "      <td>149</td>\n",
       "      <td>0.034941</td>\n",
       "      <td>False</td>\n",
       "      <td>104.498216</td>\n",
       "      <td>0.040406</td>\n",
       "      <td>0.010364</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0.148182</td>\n",
       "      <td>24</td>\n",
       "      <td>5.015381</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>83.286755</td>\n",
       "      <td>128</td>\n",
       "      <td>0.027313</td>\n",
       "      <td>False</td>\n",
       "      <td>144.602977</td>\n",
       "      <td>0.061343</td>\n",
       "      <td>0.014137</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0.145859</td>\n",
       "      <td>43</td>\n",
       "      <td>3.943408</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>61.408748</td>\n",
       "      <td>117</td>\n",
       "      <td>0.032835</td>\n",
       "      <td>True</td>\n",
       "      <td>107.081150</td>\n",
       "      <td>0.150078</td>\n",
       "      <td>0.004032</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0.109392</td>\n",
       "      <td>32</td>\n",
       "      <td>5.465265</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>29.100532</td>\n",
       "      <td>98</td>\n",
       "      <td>0.038449</td>\n",
       "      <td>False</td>\n",
       "      <td>142.411065</td>\n",
       "      <td>0.057756</td>\n",
       "      <td>0.001003</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0.154192</td>\n",
       "      <td>37</td>\n",
       "      <td>4.918319</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>55.383965</td>\n",
       "      <td>145</td>\n",
       "      <td>0.038841</td>\n",
       "      <td>True</td>\n",
       "      <td>130.162175</td>\n",
       "      <td>0.119689</td>\n",
       "      <td>0.001579</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0.106310</td>\n",
       "      <td>41</td>\n",
       "      <td>4.276815</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>87.269134</td>\n",
       "      <td>106</td>\n",
       "      <td>0.036746</td>\n",
       "      <td>True</td>\n",
       "      <td>113.206445</td>\n",
       "      <td>0.093281</td>\n",
       "      <td>0.019623</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>0.159061</td>\n",
       "      <td>47</td>\n",
       "      <td>3.630430</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>52.479431</td>\n",
       "      <td>102</td>\n",
       "      <td>0.029875</td>\n",
       "      <td>False</td>\n",
       "      <td>135.066324</td>\n",
       "      <td>0.075284</td>\n",
       "      <td>0.015455</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0.190069</td>\n",
       "      <td>45</td>\n",
       "      <td>5.262368</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>67.884939</td>\n",
       "      <td>101</td>\n",
       "      <td>0.026620</td>\n",
       "      <td>True</td>\n",
       "      <td>118.147867</td>\n",
       "      <td>0.125009</td>\n",
       "      <td>0.013234</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>0.123809</td>\n",
       "      <td>28</td>\n",
       "      <td>4.504676</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>39.255873</td>\n",
       "      <td>124</td>\n",
       "      <td>0.037005</td>\n",
       "      <td>True</td>\n",
       "      <td>121.924833</td>\n",
       "      <td>0.029906</td>\n",
       "      <td>0.017495</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>0.141084</td>\n",
       "      <td>43</td>\n",
       "      <td>4.570515</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>42.644409</td>\n",
       "      <td>85</td>\n",
       "      <td>0.028042</td>\n",
       "      <td>False</td>\n",
       "      <td>101.824130</td>\n",
       "      <td>0.214848</td>\n",
       "      <td>0.005303</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>0.116030</td>\n",
       "      <td>50</td>\n",
       "      <td>4.772191</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>78.781012</td>\n",
       "      <td>120</td>\n",
       "      <td>0.025253</td>\n",
       "      <td>False</td>\n",
       "      <td>119.915395</td>\n",
       "      <td>0.024683</td>\n",
       "      <td>0.011550</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>0.177997</td>\n",
       "      <td>22</td>\n",
       "      <td>4.097314</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>92.535470</td>\n",
       "      <td>138</td>\n",
       "      <td>0.039246</td>\n",
       "      <td>True</td>\n",
       "      <td>108.782051</td>\n",
       "      <td>0.142603</td>\n",
       "      <td>0.004617</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>0.121416</td>\n",
       "      <td>28</td>\n",
       "      <td>5.136051</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>48.793510</td>\n",
       "      <td>112</td>\n",
       "      <td>0.029319</td>\n",
       "      <td>False</td>\n",
       "      <td>115.523335</td>\n",
       "      <td>0.113898</td>\n",
       "      <td>0.000314</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>0.168046</td>\n",
       "      <td>47</td>\n",
       "      <td>3.822497</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>12.183780</td>\n",
       "      <td>135</td>\n",
       "      <td>0.030787</td>\n",
       "      <td>False</td>\n",
       "      <td>127.210823</td>\n",
       "      <td>0.131430</td>\n",
       "      <td>0.006618</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>0.132624</td>\n",
       "      <td>30</td>\n",
       "      <td>3.978002</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>70.001596</td>\n",
       "      <td>105</td>\n",
       "      <td>0.025537</td>\n",
       "      <td>False</td>\n",
       "      <td>98.871279</td>\n",
       "      <td>0.065724</td>\n",
       "      <td>0.007461</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>0.139045</td>\n",
       "      <td>25</td>\n",
       "      <td>4.317356</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>85.104877</td>\n",
       "      <td>148</td>\n",
       "      <td>0.035198</td>\n",
       "      <td>True</td>\n",
       "      <td>134.932609</td>\n",
       "      <td>0.045659</td>\n",
       "      <td>-0.002330</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>0.196744</td>\n",
       "      <td>20</td>\n",
       "      <td>4.412799</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>3.928463</td>\n",
       "      <td>91</td>\n",
       "      <td>0.026193</td>\n",
       "      <td>False</td>\n",
       "      <td>124.260667</td>\n",
       "      <td>0.082519</td>\n",
       "      <td>0.003482</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>0.164386</td>\n",
       "      <td>18</td>\n",
       "      <td>3.559705</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>2.563779</td>\n",
       "      <td>114</td>\n",
       "      <td>0.027669</td>\n",
       "      <td>True</td>\n",
       "      <td>105.599016</td>\n",
       "      <td>0.069089</td>\n",
       "      <td>0.018237</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>0.171876</td>\n",
       "      <td>37</td>\n",
       "      <td>5.189379</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>43.416751</td>\n",
       "      <td>134</td>\n",
       "      <td>0.033763</td>\n",
       "      <td>True</td>\n",
       "      <td>110.240305</td>\n",
       "      <td>0.102707</td>\n",
       "      <td>0.009666</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>0.102494</td>\n",
       "      <td>36</td>\n",
       "      <td>4.637645</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>20.219566</td>\n",
       "      <td>90</td>\n",
       "      <td>0.028766</td>\n",
       "      <td>False</td>\n",
       "      <td>141.028735</td>\n",
       "      <td>0.036429</td>\n",
       "      <td>0.013430</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>0.128699</td>\n",
       "      <td>39</td>\n",
       "      <td>4.224275</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>31.087547</td>\n",
       "      <td>82</td>\n",
       "      <td>0.033269</td>\n",
       "      <td>True</td>\n",
       "      <td>132.576526</td>\n",
       "      <td>0.135245</td>\n",
       "      <td>0.009204</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>0.194011</td>\n",
       "      <td>17</td>\n",
       "      <td>4.119391</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>25.279767</td>\n",
       "      <td>83</td>\n",
       "      <td>0.031359</td>\n",
       "      <td>True</td>\n",
       "      <td>137.502268</td>\n",
       "      <td>0.053824</td>\n",
       "      <td>0.008654</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>0.112179</td>\n",
       "      <td>35</td>\n",
       "      <td>5.082739</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>57.001926</td>\n",
       "      <td>119</td>\n",
       "      <td>0.039705</td>\n",
       "      <td>True</td>\n",
       "      <td>101.590247</td>\n",
       "      <td>0.087633</td>\n",
       "      <td>-0.001666</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>0.185542</td>\n",
       "      <td>21</td>\n",
       "      <td>4.843869</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>18.339744</td>\n",
       "      <td>143</td>\n",
       "      <td>0.034300</td>\n",
       "      <td>True</td>\n",
       "      <td>96.190623</td>\n",
       "      <td>0.169604</td>\n",
       "      <td>0.002675</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>0.182904</td>\n",
       "      <td>31</td>\n",
       "      <td>3.879664</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>9.336654</td>\n",
       "      <td>96</td>\n",
       "      <td>0.032018</td>\n",
       "      <td>False</td>\n",
       "      <td>113.617685</td>\n",
       "      <td>0.100206</td>\n",
       "      <td>-0.000695</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>0.136037</td>\n",
       "      <td>15</td>\n",
       "      <td>5.422761</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>13.567899</td>\n",
       "      <td>108</td>\n",
       "      <td>0.031707</td>\n",
       "      <td>True</td>\n",
       "      <td>97.768721</td>\n",
       "      <td>0.160056</td>\n",
       "      <td>0.018624</td>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               alpha  amortization_period      beta debt_type  \\\n",
       "experiment                                                      \n",
       "18          0.151341                   48  3.676012  Rev Bond   \n",
       "19          0.118509                   41  5.309555     Paygo   \n",
       "20          0.174355                   18  4.727608  Rev Bond   \n",
       "21          0.188622                   26  3.703274   GO Bond   \n",
       "22          0.160916                   33  4.447292   GO Bond   \n",
       "23          0.148182                   24  5.015381     Paygo   \n",
       "24          0.145859                   43  3.943408  Rev Bond   \n",
       "25          0.109392                   32  5.465265     Paygo   \n",
       "26          0.154192                   37  4.918319   GO Bond   \n",
       "27          0.106310                   41  4.276815   GO Bond   \n",
       "28          0.159061                   47  3.630430   GO Bond   \n",
       "29          0.190069                   45  5.262368     Paygo   \n",
       "30          0.123809                   28  4.504676  Rev Bond   \n",
       "31          0.141084                   43  4.570515   GO Bond   \n",
       "32          0.116030                   50  4.772191     Paygo   \n",
       "33          0.177997                   22  4.097314   GO Bond   \n",
       "34          0.121416                   28  5.136051  Rev Bond   \n",
       "35          0.168046                   47  3.822497     Paygo   \n",
       "36          0.132624                   30  3.978002     Paygo   \n",
       "37          0.139045                   25  4.317356  Rev Bond   \n",
       "38          0.196744                   20  4.412799     Paygo   \n",
       "39          0.164386                   18  3.559705  Rev Bond   \n",
       "40          0.171876                   37  5.189379  Rev Bond   \n",
       "41          0.102494                   36  4.637645   GO Bond   \n",
       "42          0.128699                   39  4.224275  Rev Bond   \n",
       "43          0.194011                   17  4.119391  Rev Bond   \n",
       "44          0.112179                   35  5.082739     Paygo   \n",
       "45          0.185542                   21  4.843869     Paygo   \n",
       "46          0.182904                   31  3.879664   GO Bond   \n",
       "47          0.136037                   15  5.422761   GO Bond   \n",
       "\n",
       "            expand_capacity  input_flow  interest_rate  interest_rate_lock  \\\n",
       "experiment                                                                   \n",
       "18                34.248792         139       0.037682               False   \n",
       "19                76.473048          89       0.035629                True   \n",
       "20                94.255893         131       0.030304               False   \n",
       "21                96.919294         125       0.036478               False   \n",
       "22                64.247841         149       0.034941               False   \n",
       "23                83.286755         128       0.027313               False   \n",
       "24                61.408748         117       0.032835                True   \n",
       "25                29.100532          98       0.038449               False   \n",
       "26                55.383965         145       0.038841                True   \n",
       "27                87.269134         106       0.036746                True   \n",
       "28                52.479431         102       0.029875               False   \n",
       "29                67.884939         101       0.026620                True   \n",
       "30                39.255873         124       0.037005                True   \n",
       "31                42.644409          85       0.028042               False   \n",
       "32                78.781012         120       0.025253               False   \n",
       "33                92.535470         138       0.039246                True   \n",
       "34                48.793510         112       0.029319               False   \n",
       "35                12.183780         135       0.030787               False   \n",
       "36                70.001596         105       0.025537               False   \n",
       "37                85.104877         148       0.035198                True   \n",
       "38                 3.928463          91       0.026193               False   \n",
       "39                 2.563779         114       0.027669                True   \n",
       "40                43.416751         134       0.033763                True   \n",
       "41                20.219566          90       0.028766               False   \n",
       "42                31.087547          82       0.033269                True   \n",
       "43                25.279767          83       0.031359                True   \n",
       "44                57.001926         119       0.039705                True   \n",
       "45                18.339744         143       0.034300                True   \n",
       "46                 9.336654          96       0.032018               False   \n",
       "47                13.567899         108       0.031707                True   \n",
       "\n",
       "            unit_cost_expansion  value_of_time  yield_curve  free_flow_time  \\\n",
       "experiment                                                                    \n",
       "18                   128.627672       0.004817     0.012079              60   \n",
       "19                   125.994370       0.091859     0.005987              60   \n",
       "20                   139.257007       0.076668     0.016896              60   \n",
       "21                   121.126672       0.110190     0.016048              60   \n",
       "22                   104.498216       0.040406     0.010364              60   \n",
       "23                   144.602977       0.061343     0.014137              60   \n",
       "24                   107.081150       0.150078     0.004032              60   \n",
       "25                   142.411065       0.057756     0.001003              60   \n",
       "26                   130.162175       0.119689     0.001579              60   \n",
       "27                   113.206445       0.093281     0.019623              60   \n",
       "28                   135.066324       0.075284     0.015455              60   \n",
       "29                   118.147867       0.125009     0.013234              60   \n",
       "30                   121.924833       0.029906     0.017495              60   \n",
       "31                   101.824130       0.214848     0.005303              60   \n",
       "32                   119.915395       0.024683     0.011550              60   \n",
       "33                   108.782051       0.142603     0.004617              60   \n",
       "34                   115.523335       0.113898     0.000314              60   \n",
       "35                   127.210823       0.131430     0.006618              60   \n",
       "36                    98.871279       0.065724     0.007461              60   \n",
       "37                   134.932609       0.045659    -0.002330              60   \n",
       "38                   124.260667       0.082519     0.003482              60   \n",
       "39                   105.599016       0.069089     0.018237              60   \n",
       "40                   110.240305       0.102707     0.009666              60   \n",
       "41                   141.028735       0.036429     0.013430              60   \n",
       "42                   132.576526       0.135245     0.009204              60   \n",
       "43                   137.502268       0.053824     0.008654              60   \n",
       "44                   101.590247       0.087633    -0.001666              60   \n",
       "45                    96.190623       0.169604     0.002675              60   \n",
       "46                   113.617685       0.100206    -0.000695              60   \n",
       "47                    97.768721       0.160056     0.018624              60   \n",
       "\n",
       "            initial_capacity  \n",
       "experiment                    \n",
       "18                       100  \n",
       "19                       100  \n",
       "20                       100  \n",
       "21                       100  \n",
       "22                       100  \n",
       "23                       100  \n",
       "24                       100  \n",
       "25                       100  \n",
       "26                       100  \n",
       "27                       100  \n",
       "28                       100  \n",
       "29                       100  \n",
       "30                       100  \n",
       "31                       100  \n",
       "32                       100  \n",
       "33                       100  \n",
       "34                       100  \n",
       "35                       100  \n",
       "36                       100  \n",
       "37                       100  \n",
       "38                       100  \n",
       "39                       100  \n",
       "40                       100  \n",
       "41                       100  \n",
       "42                       100  \n",
       "43                       100  \n",
       "44                       100  \n",
       "45                       100  \n",
       "46                       100  \n",
       "47                       100  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "design3 = fx.design_experiments(design_name='lhs_3', n_samples=30, random_seed=3)\n",
    "design3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The demo module is set up to facilitate distributed multiprocessing. During the `setup`\n",
    "step, the code detects if it is being run in a distributed \"worker\" environment instead of\n",
    "in a normal Python environment.  If the \"worker\" environment is detected, then a copy\n",
    "of the entire files-based model is made into the worker's local workspace, and the model\n",
    "is run there instead of in the master workspace.  This allows each worker to edit the files\n",
    "independently and simultaneously, without disturbing other parallel workers.\n",
    "\n",
    "With this small modification, we are ready to run this demo model in parallel subprocesses.\n",
    "to do, we will use the `async_experiments` method.  Leveraging the `asyncio` Python interface\n",
    "will allow us to run multiple core models in parallel in the background, and monitor the\n",
    "progress interactively from within a Jupyter notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:34.36] MainProcess/INFO: asynchronous_experiments(max_n_workers=2)\n"
     ]
    }
   ],
   "source": [
    "background = fx.async_experiments(\n",
    "    design=design3,\n",
    "    max_n_workers=2,\n",
    "    stagger_start=5,\n",
    "    batch_size=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'30 runs: 30 pending'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "background.progress() # Initially everything is pending"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:34.37] MainProcess/INFO: AsyncExperimentalDesign.run start\n",
      "[00:34.37] MainProcess/INFO: initializing default DistributedEvaluator.client\n",
      "[00:34.37] MainProcess/INFO:   max_n_workers=2, actual n_workers=2\n",
      "[00:34.37] MainProcess/INFO:   n_workers=2\n",
      "[00:35.31] MainProcess/INFO: completed initializing default DistributedEvaluator.client\n",
      "[00:37.49] MainProcess/INFO: AsyncExperimentalDesign.run dispatching experiments\n",
      "[00:37.50] MainProcess/INFO: performing 30 scenarios/policies * 1 model(s) = 30 experiments\n",
      "[00:37.52] MainProcess/INFO: experiments in asynchronous evaluator\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'30 runs: 2 done, 27 pending, 1 queued'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "await asyncio.sleep(15)\n",
    "background.progress()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After 15 seconds, only 3 runs have executed, as the `stagger_start`\n",
    "allows only one run to start every 5 seconds.  Un-started runs remain in the \n",
    "\"pending\" status.  We can see the status of all runs in `status`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "experiment\n",
       "18       done\n",
       "19       done\n",
       "20     queued\n",
       "21    pending\n",
       "22    pending\n",
       "23    pending\n",
       "24    pending\n",
       "25    pending\n",
       "26    pending\n",
       "27    pending\n",
       "28    pending\n",
       "29    pending\n",
       "30    pending\n",
       "31    pending\n",
       "32    pending\n",
       "33    pending\n",
       "34    pending\n",
       "35    pending\n",
       "36    pending\n",
       "37    pending\n",
       "38    pending\n",
       "39    pending\n",
       "40    pending\n",
       "41    pending\n",
       "42    pending\n",
       "43    pending\n",
       "44    pending\n",
       "45    pending\n",
       "46    pending\n",
       "47    pending\n",
       "dtype: object"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "background.status() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For fast running models, may find that `stagger_start` is not needed.\n",
    "Large and slow models, especially ones that begin with a massive file-copy\n",
    "operation on a hard disk, may benefit from staggering the task start times,\n",
    "so the first run has a clear shot at finishing disk actions promptly and\n",
    "moving to data processing as soon as possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we are dissatisfied with this pace for this small and fast model,\n",
    "the `stagger_start` can be changed dynamically while runs are going.\n",
    "If we set it to zero, all remaining pending runs will be queued \n",
    "immediately (or, within a second or so).  Queued runs will begin \n",
    "as soon as a worker process is available to handle them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:49.53] MainProcess/INFO: AsyncExperimentalDesign.run dispatching task complete\n",
      "[00:50.09] MainProcess/INFO: 3 cases completed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "experiment\n",
       "18      done\n",
       "19      done\n",
       "20      done\n",
       "21    queued\n",
       "22    queued\n",
       "23    queued\n",
       "24    queued\n",
       "25    queued\n",
       "26    queued\n",
       "27    queued\n",
       "28    queued\n",
       "29    queued\n",
       "30    queued\n",
       "31    queued\n",
       "32    queued\n",
       "33    queued\n",
       "34    queued\n",
       "35    queued\n",
       "36    queued\n",
       "37    queued\n",
       "38    queued\n",
       "39    queued\n",
       "40    queued\n",
       "41    queued\n",
       "42    queued\n",
       "43    queued\n",
       "44    queued\n",
       "45    queued\n",
       "46    queued\n",
       "47    queued\n",
       "dtype: object"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "background.stagger_start = 0\n",
    "await asyncio.sleep(1)\n",
    "background.status()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[00:55.05] MainProcess/INFO: 6 cases completed\n",
      "[00:58.24] MainProcess/INFO: 9 cases completed\n",
      "[01:02.97] MainProcess/INFO: 12 cases completed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'30 runs: 13 done, 17 queued'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "await asyncio.sleep(15)\n",
    "background.progress() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After 15 more seconds, only about 10 more runs are complete, as each run takes around 3 seconds,\n",
    "and we only have two workers processing the runs.\n",
    "\n",
    "The `current_results` method allows us to view the run results that are available currently.\n",
    "You might notice that runs are not necessarily dispatched in the order they appear\n",
    "in the table, but we'll get to all of them eventually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>free_flow_time</th>\n",
       "      <th>initial_capacity</th>\n",
       "      <th>alpha</th>\n",
       "      <th>beta</th>\n",
       "      <th>input_flow</th>\n",
       "      <th>value_of_time</th>\n",
       "      <th>unit_cost_expansion</th>\n",
       "      <th>interest_rate</th>\n",
       "      <th>yield_curve</th>\n",
       "      <th>expand_capacity</th>\n",
       "      <th>amortization_period</th>\n",
       "      <th>debt_type</th>\n",
       "      <th>interest_rate_lock</th>\n",
       "      <th>no_build_travel_time</th>\n",
       "      <th>build_travel_time</th>\n",
       "      <th>time_savings</th>\n",
       "      <th>value_of_time_savings</th>\n",
       "      <th>net_benefits</th>\n",
       "      <th>cost_of_capacity_expansion</th>\n",
       "      <th>present_cost_expansion</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>experiment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.151341</td>\n",
       "      <td>3.676012</td>\n",
       "      <td>139</td>\n",
       "      <td>0.004817</td>\n",
       "      <td>128.627672</td>\n",
       "      <td>0.037682</td>\n",
       "      <td>0.012079</td>\n",
       "      <td>34.248792</td>\n",
       "      <td>48</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>90.467198</td>\n",
       "      <td>70.318877</td>\n",
       "      <td>20.148321</td>\n",
       "      <td>13.491094</td>\n",
       "      <td>-192.374479</td>\n",
       "      <td>205.865574</td>\n",
       "      <td>4405.342341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.118509</td>\n",
       "      <td>5.309555</td>\n",
       "      <td>89</td>\n",
       "      <td>0.091859</td>\n",
       "      <td>125.994370</td>\n",
       "      <td>0.035629</td>\n",
       "      <td>0.005987</td>\n",
       "      <td>76.473048</td>\n",
       "      <td>41</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>63.829895</td>\n",
       "      <td>60.187687</td>\n",
       "      <td>3.642208</td>\n",
       "      <td>29.776845</td>\n",
       "      <td>-243.694139</td>\n",
       "      <td>273.470984</td>\n",
       "      <td>9635.173464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.174355</td>\n",
       "      <td>4.727608</td>\n",
       "      <td>131</td>\n",
       "      <td>0.076668</td>\n",
       "      <td>139.257007</td>\n",
       "      <td>0.030304</td>\n",
       "      <td>0.016896</td>\n",
       "      <td>94.255893</td>\n",
       "      <td>18</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>97.497064</td>\n",
       "      <td>61.624340</td>\n",
       "      <td>35.872724</td>\n",
       "      <td>360.289988</td>\n",
       "      <td>-600.998696</td>\n",
       "      <td>961.288684</td>\n",
       "      <td>13125.793539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.188622</td>\n",
       "      <td>3.703274</td>\n",
       "      <td>125</td>\n",
       "      <td>0.110190</td>\n",
       "      <td>121.126672</td>\n",
       "      <td>0.036478</td>\n",
       "      <td>0.016048</td>\n",
       "      <td>96.919294</td>\n",
       "      <td>26</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.160916</td>\n",
       "      <td>4.447292</td>\n",
       "      <td>149</td>\n",
       "      <td>0.040406</td>\n",
       "      <td>104.498216</td>\n",
       "      <td>0.034941</td>\n",
       "      <td>0.010364</td>\n",
       "      <td>64.247841</td>\n",
       "      <td>33</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.148182</td>\n",
       "      <td>5.015381</td>\n",
       "      <td>128</td>\n",
       "      <td>0.061343</td>\n",
       "      <td>144.602977</td>\n",
       "      <td>0.027313</td>\n",
       "      <td>0.014137</td>\n",
       "      <td>83.286755</td>\n",
       "      <td>24</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.145859</td>\n",
       "      <td>3.943408</td>\n",
       "      <td>117</td>\n",
       "      <td>0.150078</td>\n",
       "      <td>107.081150</td>\n",
       "      <td>0.032835</td>\n",
       "      <td>0.004032</td>\n",
       "      <td>61.408748</td>\n",
       "      <td>43</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.109392</td>\n",
       "      <td>5.465265</td>\n",
       "      <td>98</td>\n",
       "      <td>0.057756</td>\n",
       "      <td>142.411065</td>\n",
       "      <td>0.038449</td>\n",
       "      <td>0.001003</td>\n",
       "      <td>29.100532</td>\n",
       "      <td>32</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.154192</td>\n",
       "      <td>4.918319</td>\n",
       "      <td>145</td>\n",
       "      <td>0.119689</td>\n",
       "      <td>130.162175</td>\n",
       "      <td>0.038841</td>\n",
       "      <td>0.001579</td>\n",
       "      <td>55.383965</td>\n",
       "      <td>37</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.106310</td>\n",
       "      <td>4.276815</td>\n",
       "      <td>106</td>\n",
       "      <td>0.093281</td>\n",
       "      <td>113.206445</td>\n",
       "      <td>0.036746</td>\n",
       "      <td>0.019623</td>\n",
       "      <td>87.269134</td>\n",
       "      <td>41</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.159061</td>\n",
       "      <td>3.630430</td>\n",
       "      <td>102</td>\n",
       "      <td>0.075284</td>\n",
       "      <td>135.066324</td>\n",
       "      <td>0.029875</td>\n",
       "      <td>0.015455</td>\n",
       "      <td>52.479431</td>\n",
       "      <td>47</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.190069</td>\n",
       "      <td>5.262368</td>\n",
       "      <td>101</td>\n",
       "      <td>0.125009</td>\n",
       "      <td>118.147867</td>\n",
       "      <td>0.026620</td>\n",
       "      <td>0.013234</td>\n",
       "      <td>67.884939</td>\n",
       "      <td>45</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.123809</td>\n",
       "      <td>4.504676</td>\n",
       "      <td>124</td>\n",
       "      <td>0.029906</td>\n",
       "      <td>121.924833</td>\n",
       "      <td>0.037005</td>\n",
       "      <td>0.017495</td>\n",
       "      <td>39.255873</td>\n",
       "      <td>28</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.141084</td>\n",
       "      <td>4.570515</td>\n",
       "      <td>85</td>\n",
       "      <td>0.214848</td>\n",
       "      <td>101.824130</td>\n",
       "      <td>0.028042</td>\n",
       "      <td>0.005303</td>\n",
       "      <td>42.644409</td>\n",
       "      <td>43</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.116030</td>\n",
       "      <td>4.772191</td>\n",
       "      <td>120</td>\n",
       "      <td>0.024683</td>\n",
       "      <td>119.915395</td>\n",
       "      <td>0.025253</td>\n",
       "      <td>0.011550</td>\n",
       "      <td>78.781012</td>\n",
       "      <td>50</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            free_flow_time  initial_capacity     alpha      beta  input_flow  \\\n",
       "experiment                                                                     \n",
       "18                      60               100  0.151341  3.676012         139   \n",
       "19                      60               100  0.118509  5.309555          89   \n",
       "20                      60               100  0.174355  4.727608         131   \n",
       "21                      60               100  0.188622  3.703274         125   \n",
       "22                      60               100  0.160916  4.447292         149   \n",
       "23                      60               100  0.148182  5.015381         128   \n",
       "24                      60               100  0.145859  3.943408         117   \n",
       "25                      60               100  0.109392  5.465265          98   \n",
       "26                      60               100  0.154192  4.918319         145   \n",
       "27                      60               100  0.106310  4.276815         106   \n",
       "28                      60               100  0.159061  3.630430         102   \n",
       "29                      60               100  0.190069  5.262368         101   \n",
       "30                      60               100  0.123809  4.504676         124   \n",
       "31                      60               100  0.141084  4.570515          85   \n",
       "32                      60               100  0.116030  4.772191         120   \n",
       "\n",
       "            value_of_time  unit_cost_expansion  interest_rate  yield_curve  \\\n",
       "experiment                                                                   \n",
       "18               0.004817           128.627672       0.037682     0.012079   \n",
       "19               0.091859           125.994370       0.035629     0.005987   \n",
       "20               0.076668           139.257007       0.030304     0.016896   \n",
       "21               0.110190           121.126672       0.036478     0.016048   \n",
       "22               0.040406           104.498216       0.034941     0.010364   \n",
       "23               0.061343           144.602977       0.027313     0.014137   \n",
       "24               0.150078           107.081150       0.032835     0.004032   \n",
       "25               0.057756           142.411065       0.038449     0.001003   \n",
       "26               0.119689           130.162175       0.038841     0.001579   \n",
       "27               0.093281           113.206445       0.036746     0.019623   \n",
       "28               0.075284           135.066324       0.029875     0.015455   \n",
       "29               0.125009           118.147867       0.026620     0.013234   \n",
       "30               0.029906           121.924833       0.037005     0.017495   \n",
       "31               0.214848           101.824130       0.028042     0.005303   \n",
       "32               0.024683           119.915395       0.025253     0.011550   \n",
       "\n",
       "            expand_capacity  amortization_period debt_type  \\\n",
       "experiment                                                   \n",
       "18                34.248792                   48  Rev Bond   \n",
       "19                76.473048                   41     Paygo   \n",
       "20                94.255893                   18  Rev Bond   \n",
       "21                96.919294                   26   GO Bond   \n",
       "22                64.247841                   33   GO Bond   \n",
       "23                83.286755                   24     Paygo   \n",
       "24                61.408748                   43  Rev Bond   \n",
       "25                29.100532                   32     Paygo   \n",
       "26                55.383965                   37   GO Bond   \n",
       "27                87.269134                   41   GO Bond   \n",
       "28                52.479431                   47   GO Bond   \n",
       "29                67.884939                   45     Paygo   \n",
       "30                39.255873                   28  Rev Bond   \n",
       "31                42.644409                   43   GO Bond   \n",
       "32                78.781012                   50     Paygo   \n",
       "\n",
       "            interest_rate_lock  no_build_travel_time  build_travel_time  \\\n",
       "experiment                                                                \n",
       "18                       False             90.467198          70.318877   \n",
       "19                        True             63.829895          60.187687   \n",
       "20                       False             97.497064          61.624340   \n",
       "21                       False                   NaN                NaN   \n",
       "22                       False                   NaN                NaN   \n",
       "23                       False                   NaN                NaN   \n",
       "24                        True                   NaN                NaN   \n",
       "25                       False                   NaN                NaN   \n",
       "26                        True                   NaN                NaN   \n",
       "27                        True                   NaN                NaN   \n",
       "28                       False                   NaN                NaN   \n",
       "29                        True                   NaN                NaN   \n",
       "30                        True                   NaN                NaN   \n",
       "31                       False                   NaN                NaN   \n",
       "32                       False                   NaN                NaN   \n",
       "\n",
       "            time_savings  value_of_time_savings  net_benefits  \\\n",
       "experiment                                                      \n",
       "18             20.148321              13.491094   -192.374479   \n",
       "19              3.642208              29.776845   -243.694139   \n",
       "20             35.872724             360.289988   -600.998696   \n",
       "21                   NaN                    NaN           NaN   \n",
       "22                   NaN                    NaN           NaN   \n",
       "23                   NaN                    NaN           NaN   \n",
       "24                   NaN                    NaN           NaN   \n",
       "25                   NaN                    NaN           NaN   \n",
       "26                   NaN                    NaN           NaN   \n",
       "27                   NaN                    NaN           NaN   \n",
       "28                   NaN                    NaN           NaN   \n",
       "29                   NaN                    NaN           NaN   \n",
       "30                   NaN                    NaN           NaN   \n",
       "31                   NaN                    NaN           NaN   \n",
       "32                   NaN                    NaN           NaN   \n",
       "\n",
       "            cost_of_capacity_expansion  present_cost_expansion  \n",
       "experiment                                                      \n",
       "18                          205.865574             4405.342341  \n",
       "19                          273.470984             9635.173464  \n",
       "20                          961.288684            13125.793539  \n",
       "21                                 NaN                     NaN  \n",
       "22                                 NaN                     NaN  \n",
       "23                                 NaN                     NaN  \n",
       "24                                 NaN                     NaN  \n",
       "25                                 NaN                     NaN  \n",
       "26                                 NaN                     NaN  \n",
       "27                                 NaN                     NaN  \n",
       "28                                 NaN                     NaN  \n",
       "29                                 NaN                     NaN  \n",
       "30                                 NaN                     NaN  \n",
       "31                                 NaN                     NaN  \n",
       "32                                 NaN                     NaN  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "background.current_results().head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to simply block the Python interpreter until the runs are done,\n",
    "we can do so by `await`ing the `final_results`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df86a0e2623b41e1a6b2a1d1ca075067",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "IntProgress(value=13, max=30)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[01:06.21] MainProcess/INFO: 15 cases completed\n",
      "[01:11.06] MainProcess/INFO: 18 cases completed\n",
      "[01:14.23] MainProcess/INFO: 21 cases completed\n",
      "[01:19.02] MainProcess/INFO: 24 cases completed\n",
      "[01:22.12] MainProcess/INFO: 27 cases completed\n",
      "[01:26.65] MainProcess/INFO: 30 cases completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>free_flow_time</th>\n",
       "      <th>initial_capacity</th>\n",
       "      <th>alpha</th>\n",
       "      <th>beta</th>\n",
       "      <th>input_flow</th>\n",
       "      <th>value_of_time</th>\n",
       "      <th>unit_cost_expansion</th>\n",
       "      <th>interest_rate</th>\n",
       "      <th>yield_curve</th>\n",
       "      <th>expand_capacity</th>\n",
       "      <th>amortization_period</th>\n",
       "      <th>debt_type</th>\n",
       "      <th>interest_rate_lock</th>\n",
       "      <th>no_build_travel_time</th>\n",
       "      <th>build_travel_time</th>\n",
       "      <th>time_savings</th>\n",
       "      <th>value_of_time_savings</th>\n",
       "      <th>net_benefits</th>\n",
       "      <th>cost_of_capacity_expansion</th>\n",
       "      <th>present_cost_expansion</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>experiment</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.151341</td>\n",
       "      <td>3.676012</td>\n",
       "      <td>139</td>\n",
       "      <td>0.004817</td>\n",
       "      <td>128.627672</td>\n",
       "      <td>0.037682</td>\n",
       "      <td>0.012079</td>\n",
       "      <td>34.248792</td>\n",
       "      <td>48</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>90.467198</td>\n",
       "      <td>70.318877</td>\n",
       "      <td>20.148321</td>\n",
       "      <td>13.491094</td>\n",
       "      <td>-192.374479</td>\n",
       "      <td>205.865574</td>\n",
       "      <td>4405.342341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.118509</td>\n",
       "      <td>5.309555</td>\n",
       "      <td>89</td>\n",
       "      <td>0.091859</td>\n",
       "      <td>125.994370</td>\n",
       "      <td>0.035629</td>\n",
       "      <td>0.005987</td>\n",
       "      <td>76.473048</td>\n",
       "      <td>41</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>63.829895</td>\n",
       "      <td>60.187687</td>\n",
       "      <td>3.642208</td>\n",
       "      <td>29.776845</td>\n",
       "      <td>-243.694139</td>\n",
       "      <td>273.470984</td>\n",
       "      <td>9635.173464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.174355</td>\n",
       "      <td>4.727608</td>\n",
       "      <td>131</td>\n",
       "      <td>0.076668</td>\n",
       "      <td>139.257007</td>\n",
       "      <td>0.030304</td>\n",
       "      <td>0.016896</td>\n",
       "      <td>94.255893</td>\n",
       "      <td>18</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>97.497064</td>\n",
       "      <td>61.624340</td>\n",
       "      <td>35.872724</td>\n",
       "      <td>360.289988</td>\n",
       "      <td>-600.998696</td>\n",
       "      <td>961.288684</td>\n",
       "      <td>13125.793539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.188622</td>\n",
       "      <td>3.703274</td>\n",
       "      <td>125</td>\n",
       "      <td>0.110190</td>\n",
       "      <td>121.126672</td>\n",
       "      <td>0.036478</td>\n",
       "      <td>0.016048</td>\n",
       "      <td>96.919294</td>\n",
       "      <td>26</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>85.859973</td>\n",
       "      <td>62.102800</td>\n",
       "      <td>23.757173</td>\n",
       "      <td>327.226070</td>\n",
       "      <td>-334.329873</td>\n",
       "      <td>661.555942</td>\n",
       "      <td>11739.511583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.160916</td>\n",
       "      <td>4.447292</td>\n",
       "      <td>149</td>\n",
       "      <td>0.040406</td>\n",
       "      <td>104.498216</td>\n",
       "      <td>0.034941</td>\n",
       "      <td>0.010364</td>\n",
       "      <td>64.247841</td>\n",
       "      <td>33</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>116.880236</td>\n",
       "      <td>66.259960</td>\n",
       "      <td>50.620276</td>\n",
       "      <td>304.758181</td>\n",
       "      <td>-30.577074</td>\n",
       "      <td>335.335254</td>\n",
       "      <td>6713.784698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.148182</td>\n",
       "      <td>5.015381</td>\n",
       "      <td>128</td>\n",
       "      <td>0.061343</td>\n",
       "      <td>144.602977</td>\n",
       "      <td>0.027313</td>\n",
       "      <td>0.014137</td>\n",
       "      <td>83.286755</td>\n",
       "      <td>24</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>90.665184</td>\n",
       "      <td>61.468734</td>\n",
       "      <td>29.196450</td>\n",
       "      <td>229.247199</td>\n",
       "      <td>-288.854311</td>\n",
       "      <td>518.101510</td>\n",
       "      <td>12043.512651</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.145859</td>\n",
       "      <td>3.943408</td>\n",
       "      <td>117</td>\n",
       "      <td>0.150078</td>\n",
       "      <td>107.081150</td>\n",
       "      <td>0.032835</td>\n",
       "      <td>0.004032</td>\n",
       "      <td>61.408748</td>\n",
       "      <td>43</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>76.254321</td>\n",
       "      <td>62.460523</td>\n",
       "      <td>13.793797</td>\n",
       "      <td>242.207489</td>\n",
       "      <td>-70.587693</td>\n",
       "      <td>312.795183</td>\n",
       "      <td>6575.719357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.109392</td>\n",
       "      <td>5.465265</td>\n",
       "      <td>98</td>\n",
       "      <td>0.057756</td>\n",
       "      <td>142.411065</td>\n",
       "      <td>0.038449</td>\n",
       "      <td>0.001003</td>\n",
       "      <td>29.100532</td>\n",
       "      <td>32</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>65.877391</td>\n",
       "      <td>61.455236</td>\n",
       "      <td>4.422155</td>\n",
       "      <td>25.029593</td>\n",
       "      <td>-115.116663</td>\n",
       "      <td>140.146256</td>\n",
       "      <td>4144.237736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.154192</td>\n",
       "      <td>4.918319</td>\n",
       "      <td>145</td>\n",
       "      <td>0.119689</td>\n",
       "      <td>130.162175</td>\n",
       "      <td>0.038841</td>\n",
       "      <td>0.001579</td>\n",
       "      <td>55.383965</td>\n",
       "      <td>37</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>117.527065</td>\n",
       "      <td>66.583783</td>\n",
       "      <td>50.943282</td>\n",
       "      <td>884.112291</td>\n",
       "      <td>539.879198</td>\n",
       "      <td>344.233094</td>\n",
       "      <td>7208.897323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.106310</td>\n",
       "      <td>4.276815</td>\n",
       "      <td>106</td>\n",
       "      <td>0.093281</td>\n",
       "      <td>113.206445</td>\n",
       "      <td>0.036746</td>\n",
       "      <td>0.019623</td>\n",
       "      <td>87.269134</td>\n",
       "      <td>41</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>68.183795</td>\n",
       "      <td>60.559329</td>\n",
       "      <td>7.624466</td>\n",
       "      <td>75.389314</td>\n",
       "      <td>-381.495063</td>\n",
       "      <td>456.884376</td>\n",
       "      <td>9879.428427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.159061</td>\n",
       "      <td>3.630430</td>\n",
       "      <td>102</td>\n",
       "      <td>0.075284</td>\n",
       "      <td>135.066324</td>\n",
       "      <td>0.029875</td>\n",
       "      <td>0.015455</td>\n",
       "      <td>52.479431</td>\n",
       "      <td>47</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>70.255064</td>\n",
       "      <td>62.217189</td>\n",
       "      <td>8.037875</td>\n",
       "      <td>61.722934</td>\n",
       "      <td>-256.677328</td>\n",
       "      <td>318.400262</td>\n",
       "      <td>7088.203858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.190069</td>\n",
       "      <td>5.262368</td>\n",
       "      <td>101</td>\n",
       "      <td>0.125009</td>\n",
       "      <td>118.147867</td>\n",
       "      <td>0.026620</td>\n",
       "      <td>0.013234</td>\n",
       "      <td>67.884939</td>\n",
       "      <td>45</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>72.017180</td>\n",
       "      <td>60.786517</td>\n",
       "      <td>11.230663</td>\n",
       "      <td>141.797161</td>\n",
       "      <td>-73.766382</td>\n",
       "      <td>215.563543</td>\n",
       "      <td>8020.460747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.123809</td>\n",
       "      <td>4.504676</td>\n",
       "      <td>124</td>\n",
       "      <td>0.029906</td>\n",
       "      <td>121.924833</td>\n",
       "      <td>0.037005</td>\n",
       "      <td>0.017495</td>\n",
       "      <td>39.255873</td>\n",
       "      <td>28</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>79.576635</td>\n",
       "      <td>64.404583</td>\n",
       "      <td>15.172052</td>\n",
       "      <td>56.263770</td>\n",
       "      <td>-210.531699</td>\n",
       "      <td>266.795469</td>\n",
       "      <td>4786.265822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.141084</td>\n",
       "      <td>4.570515</td>\n",
       "      <td>85</td>\n",
       "      <td>0.214848</td>\n",
       "      <td>101.824130</td>\n",
       "      <td>0.028042</td>\n",
       "      <td>0.005303</td>\n",
       "      <td>42.644409</td>\n",
       "      <td>43</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>64.027525</td>\n",
       "      <td>60.794355</td>\n",
       "      <td>3.233170</td>\n",
       "      <td>59.044512</td>\n",
       "      <td>-139.377225</td>\n",
       "      <td>198.421737</td>\n",
       "      <td>4342.229844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.116030</td>\n",
       "      <td>4.772191</td>\n",
       "      <td>120</td>\n",
       "      <td>0.024683</td>\n",
       "      <td>119.915395</td>\n",
       "      <td>0.025253</td>\n",
       "      <td>0.011550</td>\n",
       "      <td>78.781012</td>\n",
       "      <td>50</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>76.618381</td>\n",
       "      <td>61.038635</td>\n",
       "      <td>15.579747</td>\n",
       "      <td>46.147516</td>\n",
       "      <td>-194.872685</td>\n",
       "      <td>241.020201</td>\n",
       "      <td>9447.056124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.177997</td>\n",
       "      <td>4.097314</td>\n",
       "      <td>138</td>\n",
       "      <td>0.142603</td>\n",
       "      <td>108.782051</td>\n",
       "      <td>0.039246</td>\n",
       "      <td>0.004617</td>\n",
       "      <td>92.535470</td>\n",
       "      <td>22</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>99.966081</td>\n",
       "      <td>62.728737</td>\n",
       "      <td>37.237343</td>\n",
       "      <td>732.799774</td>\n",
       "      <td>104.331664</td>\n",
       "      <td>628.468110</td>\n",
       "      <td>10066.198296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.121416</td>\n",
       "      <td>5.136051</td>\n",
       "      <td>112</td>\n",
       "      <td>0.113898</td>\n",
       "      <td>115.523335</td>\n",
       "      <td>0.029319</td>\n",
       "      <td>0.000314</td>\n",
       "      <td>48.793510</td>\n",
       "      <td>28</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>73.038039</td>\n",
       "      <td>61.693605</td>\n",
       "      <td>11.344433</td>\n",
       "      <td>144.715681</td>\n",
       "      <td>-169.489549</td>\n",
       "      <td>314.205230</td>\n",
       "      <td>5636.788951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.168046</td>\n",
       "      <td>3.822497</td>\n",
       "      <td>135</td>\n",
       "      <td>0.131430</td>\n",
       "      <td>127.210823</td>\n",
       "      <td>0.030787</td>\n",
       "      <td>0.006618</td>\n",
       "      <td>12.183780</td>\n",
       "      <td>47</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>91.752596</td>\n",
       "      <td>80.460756</td>\n",
       "      <td>11.291840</td>\n",
       "      <td>200.351591</td>\n",
       "      <td>159.635389</td>\n",
       "      <td>40.716201</td>\n",
       "      <td>1549.908642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.132624</td>\n",
       "      <td>3.978002</td>\n",
       "      <td>105</td>\n",
       "      <td>0.065724</td>\n",
       "      <td>98.871279</td>\n",
       "      <td>0.025537</td>\n",
       "      <td>0.007461</td>\n",
       "      <td>70.001596</td>\n",
       "      <td>30</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>69.661928</td>\n",
       "      <td>61.170365</td>\n",
       "      <td>8.491563</td>\n",
       "      <td>58.600606</td>\n",
       "      <td>-187.746392</td>\n",
       "      <td>246.346998</td>\n",
       "      <td>6921.147330</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.139045</td>\n",
       "      <td>4.317356</td>\n",
       "      <td>148</td>\n",
       "      <td>0.045659</td>\n",
       "      <td>134.932609</td>\n",
       "      <td>0.035198</td>\n",
       "      <td>-0.002330</td>\n",
       "      <td>85.104877</td>\n",
       "      <td>25</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>105.330218</td>\n",
       "      <td>63.175774</td>\n",
       "      <td>42.154444</td>\n",
       "      <td>284.861228</td>\n",
       "      <td>-395.777779</td>\n",
       "      <td>680.639007</td>\n",
       "      <td>11483.423089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.196744</td>\n",
       "      <td>4.412799</td>\n",
       "      <td>91</td>\n",
       "      <td>0.082519</td>\n",
       "      <td>124.260667</td>\n",
       "      <td>0.026193</td>\n",
       "      <td>0.003482</td>\n",
       "      <td>3.928463</td>\n",
       "      <td>20</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>False</td>\n",
       "      <td>67.785922</td>\n",
       "      <td>66.568468</td>\n",
       "      <td>1.217455</td>\n",
       "      <td>9.142197</td>\n",
       "      <td>-15.633244</td>\n",
       "      <td>24.775440</td>\n",
       "      <td>488.153405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.164386</td>\n",
       "      <td>3.559705</td>\n",
       "      <td>114</td>\n",
       "      <td>0.069089</td>\n",
       "      <td>105.599016</td>\n",
       "      <td>0.027669</td>\n",
       "      <td>0.018237</td>\n",
       "      <td>2.563779</td>\n",
       "      <td>18</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>75.724624</td>\n",
       "      <td>74.369604</td>\n",
       "      <td>1.355019</td>\n",
       "      <td>10.672342</td>\n",
       "      <td>-9.155192</td>\n",
       "      <td>19.827534</td>\n",
       "      <td>270.732536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.171876</td>\n",
       "      <td>5.189379</td>\n",
       "      <td>134</td>\n",
       "      <td>0.102707</td>\n",
       "      <td>110.240305</td>\n",
       "      <td>0.033763</td>\n",
       "      <td>0.009666</td>\n",
       "      <td>43.416751</td>\n",
       "      <td>37</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>107.093529</td>\n",
       "      <td>67.249450</td>\n",
       "      <td>39.844079</td>\n",
       "      <td>548.362942</td>\n",
       "      <td>311.279196</td>\n",
       "      <td>237.083746</td>\n",
       "      <td>4786.275850</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.102494</td>\n",
       "      <td>4.637645</td>\n",
       "      <td>90</td>\n",
       "      <td>0.036429</td>\n",
       "      <td>141.028735</td>\n",
       "      <td>0.028766</td>\n",
       "      <td>0.013430</td>\n",
       "      <td>20.219566</td>\n",
       "      <td>36</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>63.772612</td>\n",
       "      <td>61.606001</td>\n",
       "      <td>2.166611</td>\n",
       "      <td>7.103389</td>\n",
       "      <td>-130.417635</td>\n",
       "      <td>137.521024</td>\n",
       "      <td>2851.539859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.128699</td>\n",
       "      <td>4.224275</td>\n",
       "      <td>82</td>\n",
       "      <td>0.135245</td>\n",
       "      <td>132.576526</td>\n",
       "      <td>0.033269</td>\n",
       "      <td>0.009204</td>\n",
       "      <td>31.087547</td>\n",
       "      <td>39</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>63.339271</td>\n",
       "      <td>61.064241</td>\n",
       "      <td>2.275030</td>\n",
       "      <td>25.230374</td>\n",
       "      <td>-175.639567</td>\n",
       "      <td>200.869942</td>\n",
       "      <td>4121.478933</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.194011</td>\n",
       "      <td>4.119391</td>\n",
       "      <td>83</td>\n",
       "      <td>0.053824</td>\n",
       "      <td>137.502268</td>\n",
       "      <td>0.031359</td>\n",
       "      <td>0.008654</td>\n",
       "      <td>25.279767</td>\n",
       "      <td>17</td>\n",
       "      <td>Rev Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>65.402926</td>\n",
       "      <td>62.135104</td>\n",
       "      <td>3.267821</td>\n",
       "      <td>14.598542</td>\n",
       "      <td>-250.630628</td>\n",
       "      <td>265.229169</td>\n",
       "      <td>3476.025296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.112179</td>\n",
       "      <td>5.082739</td>\n",
       "      <td>119</td>\n",
       "      <td>0.087633</td>\n",
       "      <td>101.590247</td>\n",
       "      <td>0.039705</td>\n",
       "      <td>-0.001666</td>\n",
       "      <td>57.001926</td>\n",
       "      <td>35</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>76.294739</td>\n",
       "      <td>61.645559</td>\n",
       "      <td>14.649179</td>\n",
       "      <td>152.766854</td>\n",
       "      <td>-30.252812</td>\n",
       "      <td>183.019666</td>\n",
       "      <td>5790.839697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.185542</td>\n",
       "      <td>4.843869</td>\n",
       "      <td>143</td>\n",
       "      <td>0.169604</td>\n",
       "      <td>96.190623</td>\n",
       "      <td>0.034300</td>\n",
       "      <td>0.002675</td>\n",
       "      <td>18.339744</td>\n",
       "      <td>21</td>\n",
       "      <td>Paygo</td>\n",
       "      <td>True</td>\n",
       "      <td>122.953743</td>\n",
       "      <td>87.847511</td>\n",
       "      <td>35.106232</td>\n",
       "      <td>851.444399</td>\n",
       "      <td>765.845948</td>\n",
       "      <td>85.598451</td>\n",
       "      <td>1764.111444</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.182904</td>\n",
       "      <td>3.879664</td>\n",
       "      <td>96</td>\n",
       "      <td>0.100206</td>\n",
       "      <td>113.617685</td>\n",
       "      <td>0.032018</td>\n",
       "      <td>-0.000695</td>\n",
       "      <td>9.336654</td>\n",
       "      <td>31</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>False</td>\n",
       "      <td>69.366811</td>\n",
       "      <td>66.625117</td>\n",
       "      <td>2.741694</td>\n",
       "      <td>26.374431</td>\n",
       "      <td>-28.133075</td>\n",
       "      <td>54.507506</td>\n",
       "      <td>1060.809021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>60</td>\n",
       "      <td>100</td>\n",
       "      <td>0.136037</td>\n",
       "      <td>5.422761</td>\n",
       "      <td>108</td>\n",
       "      <td>0.160056</td>\n",
       "      <td>97.768721</td>\n",
       "      <td>0.031707</td>\n",
       "      <td>0.018624</td>\n",
       "      <td>13.567899</td>\n",
       "      <td>15</td>\n",
       "      <td>GO Bond</td>\n",
       "      <td>True</td>\n",
       "      <td>72.389640</td>\n",
       "      <td>66.214709</td>\n",
       "      <td>6.174931</td>\n",
       "      <td>106.740422</td>\n",
       "      <td>-2.377699</td>\n",
       "      <td>109.118121</td>\n",
       "      <td>1326.516148</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            free_flow_time  initial_capacity     alpha      beta  input_flow  \\\n",
       "experiment                                                                     \n",
       "18                      60               100  0.151341  3.676012         139   \n",
       "19                      60               100  0.118509  5.309555          89   \n",
       "20                      60               100  0.174355  4.727608         131   \n",
       "21                      60               100  0.188622  3.703274         125   \n",
       "22                      60               100  0.160916  4.447292         149   \n",
       "23                      60               100  0.148182  5.015381         128   \n",
       "24                      60               100  0.145859  3.943408         117   \n",
       "25                      60               100  0.109392  5.465265          98   \n",
       "26                      60               100  0.154192  4.918319         145   \n",
       "27                      60               100  0.106310  4.276815         106   \n",
       "28                      60               100  0.159061  3.630430         102   \n",
       "29                      60               100  0.190069  5.262368         101   \n",
       "30                      60               100  0.123809  4.504676         124   \n",
       "31                      60               100  0.141084  4.570515          85   \n",
       "32                      60               100  0.116030  4.772191         120   \n",
       "33                      60               100  0.177997  4.097314         138   \n",
       "34                      60               100  0.121416  5.136051         112   \n",
       "35                      60               100  0.168046  3.822497         135   \n",
       "36                      60               100  0.132624  3.978002         105   \n",
       "37                      60               100  0.139045  4.317356         148   \n",
       "38                      60               100  0.196744  4.412799          91   \n",
       "39                      60               100  0.164386  3.559705         114   \n",
       "40                      60               100  0.171876  5.189379         134   \n",
       "41                      60               100  0.102494  4.637645          90   \n",
       "42                      60               100  0.128699  4.224275          82   \n",
       "43                      60               100  0.194011  4.119391          83   \n",
       "44                      60               100  0.112179  5.082739         119   \n",
       "45                      60               100  0.185542  4.843869         143   \n",
       "46                      60               100  0.182904  3.879664          96   \n",
       "47                      60               100  0.136037  5.422761         108   \n",
       "\n",
       "            value_of_time  unit_cost_expansion  interest_rate  yield_curve  \\\n",
       "experiment                                                                   \n",
       "18               0.004817           128.627672       0.037682     0.012079   \n",
       "19               0.091859           125.994370       0.035629     0.005987   \n",
       "20               0.076668           139.257007       0.030304     0.016896   \n",
       "21               0.110190           121.126672       0.036478     0.016048   \n",
       "22               0.040406           104.498216       0.034941     0.010364   \n",
       "23               0.061343           144.602977       0.027313     0.014137   \n",
       "24               0.150078           107.081150       0.032835     0.004032   \n",
       "25               0.057756           142.411065       0.038449     0.001003   \n",
       "26               0.119689           130.162175       0.038841     0.001579   \n",
       "27               0.093281           113.206445       0.036746     0.019623   \n",
       "28               0.075284           135.066324       0.029875     0.015455   \n",
       "29               0.125009           118.147867       0.026620     0.013234   \n",
       "30               0.029906           121.924833       0.037005     0.017495   \n",
       "31               0.214848           101.824130       0.028042     0.005303   \n",
       "32               0.024683           119.915395       0.025253     0.011550   \n",
       "33               0.142603           108.782051       0.039246     0.004617   \n",
       "34               0.113898           115.523335       0.029319     0.000314   \n",
       "35               0.131430           127.210823       0.030787     0.006618   \n",
       "36               0.065724            98.871279       0.025537     0.007461   \n",
       "37               0.045659           134.932609       0.035198    -0.002330   \n",
       "38               0.082519           124.260667       0.026193     0.003482   \n",
       "39               0.069089           105.599016       0.027669     0.018237   \n",
       "40               0.102707           110.240305       0.033763     0.009666   \n",
       "41               0.036429           141.028735       0.028766     0.013430   \n",
       "42               0.135245           132.576526       0.033269     0.009204   \n",
       "43               0.053824           137.502268       0.031359     0.008654   \n",
       "44               0.087633           101.590247       0.039705    -0.001666   \n",
       "45               0.169604            96.190623       0.034300     0.002675   \n",
       "46               0.100206           113.617685       0.032018    -0.000695   \n",
       "47               0.160056            97.768721       0.031707     0.018624   \n",
       "\n",
       "            expand_capacity  amortization_period debt_type  \\\n",
       "experiment                                                   \n",
       "18                34.248792                   48  Rev Bond   \n",
       "19                76.473048                   41     Paygo   \n",
       "20                94.255893                   18  Rev Bond   \n",
       "21                96.919294                   26   GO Bond   \n",
       "22                64.247841                   33   GO Bond   \n",
       "23                83.286755                   24     Paygo   \n",
       "24                61.408748                   43  Rev Bond   \n",
       "25                29.100532                   32     Paygo   \n",
       "26                55.383965                   37   GO Bond   \n",
       "27                87.269134                   41   GO Bond   \n",
       "28                52.479431                   47   GO Bond   \n",
       "29                67.884939                   45     Paygo   \n",
       "30                39.255873                   28  Rev Bond   \n",
       "31                42.644409                   43   GO Bond   \n",
       "32                78.781012                   50     Paygo   \n",
       "33                92.535470                   22   GO Bond   \n",
       "34                48.793510                   28  Rev Bond   \n",
       "35                12.183780                   47     Paygo   \n",
       "36                70.001596                   30     Paygo   \n",
       "37                85.104877                   25  Rev Bond   \n",
       "38                 3.928463                   20     Paygo   \n",
       "39                 2.563779                   18  Rev Bond   \n",
       "40                43.416751                   37  Rev Bond   \n",
       "41                20.219566                   36   GO Bond   \n",
       "42                31.087547                   39  Rev Bond   \n",
       "43                25.279767                   17  Rev Bond   \n",
       "44                57.001926                   35     Paygo   \n",
       "45                18.339744                   21     Paygo   \n",
       "46                 9.336654                   31   GO Bond   \n",
       "47                13.567899                   15   GO Bond   \n",
       "\n",
       "            interest_rate_lock  no_build_travel_time  build_travel_time  \\\n",
       "experiment                                                                \n",
       "18                       False             90.467198          70.318877   \n",
       "19                        True             63.829895          60.187687   \n",
       "20                       False             97.497064          61.624340   \n",
       "21                       False             85.859973          62.102800   \n",
       "22                       False            116.880236          66.259960   \n",
       "23                       False             90.665184          61.468734   \n",
       "24                        True             76.254321          62.460523   \n",
       "25                       False             65.877391          61.455236   \n",
       "26                        True            117.527065          66.583783   \n",
       "27                        True             68.183795          60.559329   \n",
       "28                       False             70.255064          62.217189   \n",
       "29                        True             72.017180          60.786517   \n",
       "30                        True             79.576635          64.404583   \n",
       "31                       False             64.027525          60.794355   \n",
       "32                       False             76.618381          61.038635   \n",
       "33                        True             99.966081          62.728737   \n",
       "34                       False             73.038039          61.693605   \n",
       "35                       False             91.752596          80.460756   \n",
       "36                       False             69.661928          61.170365   \n",
       "37                        True            105.330218          63.175774   \n",
       "38                       False             67.785922          66.568468   \n",
       "39                        True             75.724624          74.369604   \n",
       "40                        True            107.093529          67.249450   \n",
       "41                       False             63.772612          61.606001   \n",
       "42                        True             63.339271          61.064241   \n",
       "43                        True             65.402926          62.135104   \n",
       "44                        True             76.294739          61.645559   \n",
       "45                        True            122.953743          87.847511   \n",
       "46                       False             69.366811          66.625117   \n",
       "47                        True             72.389640          66.214709   \n",
       "\n",
       "            time_savings  value_of_time_savings  net_benefits  \\\n",
       "experiment                                                      \n",
       "18             20.148321              13.491094   -192.374479   \n",
       "19              3.642208              29.776845   -243.694139   \n",
       "20             35.872724             360.289988   -600.998696   \n",
       "21             23.757173             327.226070   -334.329873   \n",
       "22             50.620276             304.758181    -30.577074   \n",
       "23             29.196450             229.247199   -288.854311   \n",
       "24             13.793797             242.207489    -70.587693   \n",
       "25              4.422155              25.029593   -115.116663   \n",
       "26             50.943282             884.112291    539.879198   \n",
       "27              7.624466              75.389314   -381.495063   \n",
       "28              8.037875              61.722934   -256.677328   \n",
       "29             11.230663             141.797161    -73.766382   \n",
       "30             15.172052              56.263770   -210.531699   \n",
       "31              3.233170              59.044512   -139.377225   \n",
       "32             15.579747              46.147516   -194.872685   \n",
       "33             37.237343             732.799774    104.331664   \n",
       "34             11.344433             144.715681   -169.489549   \n",
       "35             11.291840             200.351591    159.635389   \n",
       "36              8.491563              58.600606   -187.746392   \n",
       "37             42.154444             284.861228   -395.777779   \n",
       "38              1.217455               9.142197    -15.633244   \n",
       "39              1.355019              10.672342     -9.155192   \n",
       "40             39.844079             548.362942    311.279196   \n",
       "41              2.166611               7.103389   -130.417635   \n",
       "42              2.275030              25.230374   -175.639567   \n",
       "43              3.267821              14.598542   -250.630628   \n",
       "44             14.649179             152.766854    -30.252812   \n",
       "45             35.106232             851.444399    765.845948   \n",
       "46              2.741694              26.374431    -28.133075   \n",
       "47              6.174931             106.740422     -2.377699   \n",
       "\n",
       "            cost_of_capacity_expansion  present_cost_expansion  \n",
       "experiment                                                      \n",
       "18                          205.865574             4405.342341  \n",
       "19                          273.470984             9635.173464  \n",
       "20                          961.288684            13125.793539  \n",
       "21                          661.555942            11739.511583  \n",
       "22                          335.335254             6713.784698  \n",
       "23                          518.101510            12043.512651  \n",
       "24                          312.795183             6575.719357  \n",
       "25                          140.146256             4144.237736  \n",
       "26                          344.233094             7208.897323  \n",
       "27                          456.884376             9879.428427  \n",
       "28                          318.400262             7088.203858  \n",
       "29                          215.563543             8020.460747  \n",
       "30                          266.795469             4786.265822  \n",
       "31                          198.421737             4342.229844  \n",
       "32                          241.020201             9447.056124  \n",
       "33                          628.468110            10066.198296  \n",
       "34                          314.205230             5636.788951  \n",
       "35                           40.716201             1549.908642  \n",
       "36                          246.346998             6921.147330  \n",
       "37                          680.639007            11483.423089  \n",
       "38                           24.775440              488.153405  \n",
       "39                           19.827534              270.732536  \n",
       "40                          237.083746             4786.275850  \n",
       "41                          137.521024             2851.539859  \n",
       "42                          200.869942             4121.478933  \n",
       "43                          265.229169             3476.025296  \n",
       "44                          183.019666             5790.839697  \n",
       "45                           85.598451             1764.111444  \n",
       "46                           54.507506             1060.809021  \n",
       "47                          109.118121             1326.516148  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "await background.final_results()"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,py:percent"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}