11_NLP_ErrorAnalysis2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Error Analysis (2)\n",
    "\n",
    "From the previous notebook, you have seen that pyConTextNLP helped to improve the precision by excluding irrelevant annotations based on the modifiers. \n",
    "\n",
    "This notebook will continue our previous error analyses by including both false negative errors and false positive ones, through step by step demonstration of how to locate and reduce the errors.\n",
    "\n",
    "## 1. Locate the errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's import some packages\n",
    "import os\n",
    "import pyConTextNLP\n",
    "from pyConTextNLP import pyConText\n",
    "import sklearn.metrics\n",
    "import pandas as pd\n",
    "import networkx as nx\n",
    "import radnlp.view as rview\n",
    "\n",
    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
    "from IPython.display import display, HTML, Image\n",
    "import ipywidgets\n",
    "# And also our utilities for this class\n",
    "\n",
    "from nlp_pneumonia_utils import read_doc_annotations\n",
    "from nlp_pneumonia_utils import mark_document_with_html\n",
    "from DocumentClassifier import DocumentClassifier\n",
    "from visual import view_pycontext_output\n",
    "from visual import snippets_markup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember yesterday, in [06_NLP_ErrorAnalysis1](06_NLP_ErrorAnalysis1.ipynb#cell1), we created a function called *\"list_false_negatives.\"* Now we will extend this function to return both **false negative** and **false positive** document names at the same time: *\"list_errors.\"* Additionally, we will also integrate the *\"calculate_prediction_metrics\"* function inside *\"list_errors\"*, so that we can get the metrics without re-run the pyConText over the documents again.\n",
    "<br/><br/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def list_errors(gold_docs, prediction_function,\n",
    "                print_prediction_metrics=False):\n",
    "    fn_docs = []\n",
    "    fp_docs = []\n",
    "    gold_labels = [x.positive_label for x in gold_docs.values()]\n",
    "    pred_labels = []\n",
    "    for doc_name, gold_doc in gold_docs.items():\n",
    "        gold_label = gold_doc.positive_label;\n",
    "        pred_label = prediction_function(gold_doc.text, doc_name)\n",
    "        pred_labels.append(pred_label)\n",
    "        #       Differentiate false positive and false negative error\n",
    "        if gold_label == 0 and pred_label == 1:\n",
    "            fp_docs.append(doc_name)\n",
    "        elif gold_label == 1 and pred_label == 0:\n",
    "            fn_docs.append(doc_name)\n",
    "    if (print_prediction_metrics):\n",
    "        precision = sklearn.metrics.precision_score(gold_labels, pred_labels)\n",
    "        recall = sklearn.metrics.recall_score(gold_labels, pred_labels)\n",
    "        f1 = sklearn.metrics.f1_score(gold_labels, pred_labels)\n",
    "        # Let's use Pandas to make a confusion matrix for us\n",
    "        se1 = pd.Series(gold_labels, name='Actual');\n",
    "        se2 = pd.Series(pred_labels, name='Predicted')\n",
    "        confusion_matrix_df = pd.crosstab(pd.Series(gold_labels, name='Actual'),\n",
    "                                          pd.Series(pred_labels, name='Predicted'))\n",
    "\n",
    "        print('Precision : {0}'.format(precision))\n",
    "        print('Recall :    {0}'.format(recall))\n",
    "        print('F1:         {0}'.format(f1))\n",
    "\n",
    "        print('\\nConfusion Matrix : ')\n",
    "        print(confusion_matrix_df)\n",
    "    return fn_docs, fp_docs  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now we restore what we got from [10_NLP_DocumentClassification.ipynb](10_NLP_DocumentClassification.ipynb):<br/><br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading annotations from file : data/training_v2.zip\n",
      "Opening local file : data/training_v2.zip\n"
     ]
    }
   ],
   "source": [
    "#Read in the training documents and annotations\n",
    "annotated_doc_map = read_doc_annotations('data/training_v2.zip')\n",
    "\n",
    "#Here we initiate our DocumentClassifier directly through rule files:\n",
    "#Change the file names if you use different files \n",
    "pos_doc_type='PNEUMONIA_DOC_YES'\n",
    "TARGETS_FILE_PATH = 'KB/pneumonia_targets.yml'\n",
    "MODIFIERS_FILE_PATH = 'KB/pneumonia_modifiers.yml'\n",
    "FEATURE_INFERENCER_FILE_PATH = 'KB/featurer_inferences.csv'\n",
    "DOC_INFERENCER_FILE_PATH = 'KB/doc_inferences.csv'\n",
    "# clear just in case files/regular expressions have been updated\n",
    "classifier = DocumentClassifier(TARGETS_FILE_PATH, MODIFIERS_FILE_PATH,\n",
    "                               FEATURE_INFERENCER_FILE_PATH, DOC_INFERENCER_FILE_PATH,\n",
    "                               {pos_doc_type})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision : 0.5185185185185185\n",
      "Recall :    0.8235294117647058\n",
      "F1:         0.6363636363636364\n",
      "\n",
      "Confusion Matrix : \n",
      "Predicted   0   1\n",
      "Actual           \n",
      "0          10  26\n",
      "1           6  28\n"
     ]
    }
   ],
   "source": [
    "# Process the corpus using docClassifier to return errors\n",
    "current_false_negatives,current_false_positives=list_errors(annotated_doc_map, classifier.predict,True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DocumentClassifier also has a wrapped-up function to do this evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start to evaluate against reference standards...\n",
      "Precision : 0.519\n",
      "Recall :    0.824\n",
      "F1:         0.636\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>28</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>26</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted   1   0\n",
       "Actual           \n",
       "1          28   6\n",
       "0          26  10"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "current_false_negatives, current_false_positives, measurements,confusion_matrix_df = classifier.eval(annotated_doc_map)\n",
    "print(measurements)\n",
    "display(confusion_matrix_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Display errors\n",
    "Now we put everything together to display errors. Let's try false negative first:<br/><br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### * Display false negatives\n",
    "Now we can display the **false negatives** with expert annotations.<br/><br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html><div style=\"background-color:#f9f9f9;padding-left:10px;width: 877px; \"><table width=100% ><col style=\"width:25%\"><col style=\"width:75%\"></div><tr><th style=\"text-align:center\">document name</th><th style=\"text-align:center\">Snippets</th></table></div><div style=\"background-color:#f9f9f9;padding:10px; width: 900px; height: 400px; overflow-y: scroll;\"><table width=100% ><col style=\"width:25%\"><col style=\"width:75%\"><tr><td style=\"text-align:left\">subject_id_150_hadm_id_12121</td><td></td></tr><tr><td></td><td style=\"text-align:left\">es are\n",
       "     unremarkable.\n",
       "     \n",
       "     IMPRESSION:  <span style=\"color: blue;\">Small focal opacity in right upper lobe</span> and right paratracheal\n",
       "     opacity.  In the sett</td></tr><tr><td></td><td style=\"text-align:left\">CHEST PA AND LATERAL:  The heart size is normal.  <span style=\"color: blue;\">There is an area of\n",
       "     increased opacity</span> lateral to the right paratracheal stripe.  In the</td></tr><tr><td></td><td style=\"text-align:left\">pacity lateral to the right paratracheal stripe.  <span style=\"color: blue;\">In the right\n",
       "     upper lobe, there is a small focal opacity.</span>  The lungs are otherwise clear.\n",
       "     There are no</td></tr><tr><td style=\"text-align:left\">subject_id_5472_hadm_id_11987</td><td></td></tr><tr><td></td><td style=\"text-align:left\">id SVC. \n",
       "     There is no apparent pneumothorax.  <span style=\"color: blue;\">A right IJ line, NGT, and ETT are\n",
       "     unchanged as are the parenchymal changes in the lungs</span> compared to the earlier\n",
       "     chest x-ray this mor</td></tr><tr><td style=\"text-align:left\">subject_id_7027_hadm_id_33117</td><td></td></tr><tr><td></td><td style=\"text-align:left\">ossibility of free\n",
       "     intraperitoneal air.\n",
       "     <span style=\"color: blue;\">2) Left lower lobe atelectasis/consolidation.</span>\n",
       "     3) Moderate gastric distention with multiple</td></tr><tr><td></td><td style=\"text-align:left\">stinal and hilar contours are\n",
       "     unremarkable.  <span style=\"color: blue;\">There is patchy opacity at the left lower lobe representing\n",
       "     either atelectasis or consolidation</span>.  No definite free air is identified,\n",
       "     howeve</td></tr><tr><td style=\"text-align:left\">subject_id_7272_hadm_id_19098</td><td></td></tr><tr><td></td><td style=\"text-align:left\">rt failure with bilateral pleural effusions.\n",
       "     <span style=\"color: blue;\">Collapse and/or consolidation at the bases bilaterally.\n",
       "</span>\n",
       "</td></tr><tr><td></td><td style=\"text-align:left\">rall heart size is difficult to assess.  There is <span style=\"color: blue;\">dense retrocardiac\n",
       "     opacity, possibly secondary to collapse and/or consolidation</span> in the left lower\n",
       "     lobe.  There is also a rig</td></tr><tr><td></td><td style=\"text-align:left\">ion in the left lower\n",
       "     lobe.  There is also a <span style=\"color: blue;\">right lower lobe and middle lobe opacity consistent\n",
       "     with collapse and/or consolidation</span>.\n",
       "     \n",
       "     IMPRESSION: Persistent left heart fai</td></tr><tr><td style=\"text-align:left\">subject_id_7525_hadm_id_19141</td><td></td></tr><tr><td></td><td style=\"text-align:left\">n distal superior vena cava, unchanged.  There is <span style=\"color: blue;\">marked\n",
       "     improvement of the bilateral consolidations</span>, especially on the right. The NG\n",
       "     tube tip is</td></tr><tr><td style=\"text-align:left\">subject_id_9082_hadm_id_29395</td><td></td></tr><tr><td></td><td style=\"text-align:left\">tient with seizure.\n",
       "     \n",
       "     Low lung volumes.  <span style=\"color: blue;\">Bilateral basilar opacities,</span> considerably larger at the\n",
       "     left base than at</td></tr><tr><td></td><td style=\"text-align:left\">hyroid.\n",
       "     \n",
       "     IMPRESSION:  Lung volumes with <span style=\"color: blue;\">bilateral basilar opacities.</span>\n",
       "     \n",
       "     Question substernal thyroid enlargemen</td></tr></table></div></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fn_docs=dict((k, v) for k, v in annotated_doc_map.items() if k in current_false_negatives)\n",
    "display(HTML(snippets_markup(fn_docs)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### * Display false positives\n",
    "Then we can display the **false positives** with pyConText markups.<br/><br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "22a4ec421ab7440f8f8d0983ff9ccbfd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=0, description='i', max=25), Output()), _dom_classes=('widget-interact',…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fp_docs = dict((k,v) for k, v in classifier.saved_markups_map.items() if k in current_false_positives)\n",
    "view_pycontext_output(fp_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### * Rethink the causes of the false negatives\n",
    "Does the false negatives are all caused by missed keywords?<br/><br/>\n",
    "You may want to review these pyConText markups in **false negative** documents:<br/><br/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No documents to view.\n"
     ]
    }
   ],
   "source": [
    "fn_docs = dict((k,v) for k, v in classifier.saved_markups_map.items() if k in current_false_negatives)\n",
    "view_pycontext_output(fn_docs)\n",
    "# The variable: \"saved_markups_map\" in \"docClassifier\" only saves the documents that have at least one annotation. \n",
    "# Thus, the false negatives caused by missing keywords (no annotations) will not be saved in it,\n",
    "# and only pyConText caused false negatives will be displayed here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Now what?<br/><br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 4. Quiz\n",
    "Let's see if you are ready."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html><body>When you doing error analyses, if you see following pyConText graph plot: <img src=\"images/Selection_008.png\"> <br/> and following snippet:<img src=\"images/Selection_009.png\">, what can you conclude?<br/>\n",
       "<ul>\n",
       "  <li><b>A</b>: This is a false positive error</li>\n",
       "  <li><b>B</b>: This is a false negative error.</li>\n",
       "  <li><b>C</b>: This is correct.</li>\n",
       "  <li><b>D</b>: Not enough information to make a conclusion.</li>\n",
       "</ul>  \n",
       "</body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "977fd3fee8544ce0b25582a39cb041f3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "RadioButtons(description='Choose the best answer:', layout=Layout(width='600px'), options=('A', 'B', 'C', 'D')…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "712dd4de423e42f2bbb27df919dd1977",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Button(description='Submit', style=ButtonStyle())"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from quiz_utils import error_analyses_7\n",
    "error_analyses_7()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html><body>In the question above, what may be the cause?<br/>\n",
       "<ul>\n",
       "  <li><b>A</b>: \"AMBIVALENT_EXISTENCE\" is excluded by DocumentClassifier</li>\n",
       "  <li><b>B</b>: \"INDICATION\" is excluded by pyConText.</li>\n",
       "  <li><b>C</b>: \"pneumonia\" is colored orange.</li>\n",
       "  <li><b>D</b>: \"man\" should not be a \"DEFINITE_NEGATED_EXISTENCE\" clue.</li>\n",
       "</ul>  \n",
       "</body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a37f9a4ff0284ba6a1e52d41db33a43a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "RadioButtons(description='Choose the best answer:', layout=Layout(width='600px'), options=('A', 'B', 'C', 'D')…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ae0d2e99cfac454d91b221e235c83507",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Button(description='Submit', style=ButtonStyle())"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from quiz_utils import error_analyses_8\n",
    "error_analyses_8()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html><body>In the question above, what should we do next?<br/>\n",
       "<ul>\n",
       "  <li><b>A</b>: delete the rule that conclude \"AMBIVALENT_EXISTENCE\" as negative answer in our DocumentClassifier rule file</li>\n",
       "  <li><b>B</b>: add \"man\" in our target rule file.</li>\n",
       "  <li><b>C</b>: add \"pneumonia\" in our target rule file.</li>\n",
       "  <li><b>D</b>: remove the rule like \"man&nbsp;&nbsp;&nbsp;&nbsp;DEFINITE_NEGATED_EXISTENCE&nbsp;....\" in our modifier rule file.</li>\n",
       "</ul>  \n",
       "</body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2391d3c571c348c8a4f10dd59a823260",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "RadioButtons(description='Choose the best answer:', layout=Layout(width='600px'), options=('A', 'B', 'C', 'D')…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d04b7760c6964a139de089d3d8f98cfb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Button(description='Submit', style=ButtonStyle())"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from quiz_utils import error_analyses_9\n",
    "error_analyses_9()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/><br/>This material presented as part of the DeCART Data Science for the Health Science Summer Program at the University of Utah in 2019.<br/>\n",
    "Presenters : Dr. Wendy Chapman, Kelly Peterson, Alec Chapman, Jianlin Shi <br> Acknowledgement: Many thanks to Olga Patterson because part of the materials are adopted from his previous work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}