9-Decision Tree Classifiers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tree Classifiers\n",
    "\n",
    "### Decision Tree Construction Algorithm\n",
    "\n",
    "#### We build a tree top-down in a greedy manner.\n",
    "\n",
    "    function BuildDT(X, Y, threshold)\n",
    "        if (all the labels (Y) are the same\n",
    "            assign as leaf node with label set to class of Y\n",
    "            return tree\n",
    "        elseif (threshold is the minimum number of examples to keep)\n",
    "            assign as leaf node with label set to most commin class\n",
    "            return tree\n",
    "        else\n",
    "            let f be the feature that is best to split on\n",
    "            let leftChildBranch = BuildDT(data (X) where f=True, labels (Y) with f=True)\n",
    "            let rightChildBranch = BuildDT(data (X) where f=False, labels (Y) with f=False)\n",
    "            return tree\n",
    "            \n",
    "### Determining best split feature by evaluating node purity\n",
    "\n",
    "<img style=\"float: l;\" src=\"./images/Purity.png\">\n",
    " \n",
    "$Entropy(t) = - \\displaystyle \\sum_{i=0}^{c-1} p(\\space i \\space | \\space t \\space) \\space log_2 \\space p(\\space i \\space | \\space t \\space) \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space Gini(t) = 1 - \\displaystyle \\sum_{i=0}^{c-1} [p(\\space i \\space | \\space t \\space)]^2 \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space Classification \\space error(t) = 1 - \\max_i [p(\\space i \\space | \\space t \\space)]$\n",
    "\n",
    "#### 1) Determine which feature is better to split on ?\n",
    "<img style=\"float: left;\" src=\"./images/Gini.png\">  $ A.N1 = 1 - (4/7)^2 + (3/7)^2 = 0.4898 \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space$\n",
    "                                           $ A.N2 = 1 - (2/5)^2 + (3/5)^2 = 0.4800 \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space \\space$\n",
    "                                           $A = (7/12) x 0.4898 + (5/12) X 0.4800 = 0.486$ \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "                                           \n",
    "### Environment Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import nltk\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle as pkl\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "from nltk.corpus import stopwords\n",
    "%matplotlib inline \n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Syntatic NLP Processing\n",
    "\n",
    "#### We will define some Python functions that will perform some syntatic work on our corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\", 'year', 'old', 'man', 'woman', 'ap', 'am', 'pm', 'portable', 'pa', 'lat', 'admitting', 'diagnosis', 'lateral']\n"
     ]
    }
   ],
   "source": [
    "def tokenize(text):\n",
    "    tokens = [word for sent in nltk.sent_tokenize(text) for word in nltk.word_tokenize(sent)]\n",
    "    filtered_tokens = [ token for token in tokens if re.search('(^[a-zA-Z]+$)', token) ]\n",
    "    return filtered_tokens\n",
    "\n",
    "cachedStopWords = stopwords.words(\"english\")\n",
    "noisywords = ['year', 'old', 'man', 'woman', 'ap', 'am', 'pm', 'portable', 'pa', 'lat', 'admitting', 'diagnosis', 'lateral']\n",
    "cachedStopWords.extend(noisywords)\n",
    "print(cachedStopWords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieving our Corpus\n",
    "\n",
    "#### Let's pull in our corpus that we had serialized out to disk.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[**2996-12-2**] 10:25 AM\\n CT CHEST W/O CONTRA...</td>\n",
       "      <td>PNA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[**3201-9-21**] 4:50 PM\\n CT CHEST W/CONTRAST ...</td>\n",
       "      <td>PNA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[**3299-6-23**] 5:06 PM\\n CT CHEST W/CONTRAST ...</td>\n",
       "      <td>PNA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[**3186-6-14**] 2:54 PM\\n CT CHEST W/CONTRAST ...</td>\n",
       "      <td>PNA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[**2500-1-17**] 9:41 PM\\n CT CHEST W/O CONTRAS...</td>\n",
       "      <td>PNA</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text label\n",
       "0  [**2996-12-2**] 10:25 AM\\n CT CHEST W/O CONTRA...   PNA\n",
       "1  [**3201-9-21**] 4:50 PM\\n CT CHEST W/CONTRAST ...   PNA\n",
       "2  [**3299-6-23**] 5:06 PM\\n CT CHEST W/CONTRAST ...   PNA\n",
       "3  [**3186-6-14**] 2:54 PM\\n CT CHEST W/CONTRAST ...   PNA\n",
       "4  [**2500-1-17**] 9:41 PM\\n CT CHEST W/O CONTRAS...   PNA"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file = open('differential-corpus.pkl','rb')\n",
    "corpus = pkl.load(file)\n",
    "file.close()\n",
    "corpusList = list(corpus['text'].tolist())\n",
    "labels = list(corpus['label'].tolist())\n",
    "corpus.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[**2996-12-2**] 10:25 AM\n",
      " CT CHEST W/O CONTRAST                                           Clip # [**Clip Number (Radiology) 38620**]\n",
      " Reason: please eval for loculated pleural effusion on the right and\n",
      " Field of view: 30\n",
      " ______________________________________________________________________________\n",
      " UNDERLYING MEDICAL CONDITION:\n",
      "  73 year old woman with h/o RUL subtotal resection for lung CA (3/'[**95**]) as well\n",
      "  as a LLL superior segment resection  for Tb and now presents w/ a RLL PNA and\n",
      "  parapneumonic effusion concerning for empyema\n",
      " REASON FOR THIS EXAMINATION:\n",
      "  please eval for loculated pleural effusion on the right and place a pigtail\n",
      "  catheter (send specimen for Gm stain C+S, AFB smear, cytology, Fungal culture)\n",
      "  please do ASAP as pt may require to go to the OR later this afternoon\n",
      " ______________________________________________________________________________\n",
      "                                 FINAL REPORT\n",
      " INDICATION:  [** Location **]US HISTORY OF SUBTOTAL RESECTION FOR LUNG CANCER IN [**Month (only) **]\n",
      " [**2995**] AND PRIOR LEFT LOWER LOBE SUPERIOR SEGMENT RESECTION FOR TUBERCULOSIS,\n",
      " WHO CURRENTLY REPRESENTS A RIGHT LOWER LOBE PNEUMONIA AND PERIPNEUMONIC\n",
      " EFFUSION.  PLEASE ASSESS PLEURAL EFFUSION FOR EVIDENCE OF EMPYEMA.\n",
      "\n",
      " No prior chest CT scans are available for comparison.  Procurement of [** Location **]us\n",
      " chest CT should most helpful in delineating stability of postoperative\n",
      " abnormalities in this patient with history of prior surgeries and prior lung\n",
      " cancer.\n",
      "\n",
      " Helical CT of the thorax was performed without intravenous or oral contrast\n",
      " administration.  Images were acquired with 5 mm collimation and reconstructed\n",
      " at 5 mm intervals.\n",
      "\n",
      " Assessment of the soft tissue structures of the thorax reveals numerous\n",
      " surgical clips in the right perihilar and paramediastinal area.  There is a\n",
      " very small dependent left pleural effusion present and there is a moderate-\n",
      " sized right pleural effusion which has significantly loculated components,\n",
      " predominantly anteriorly and laterally in the lower right hemithorax. Although\n",
      " intravenous contrast was not administered, precluding assessment for\n",
      " enhancement of the pleural surface, there does not appear to be a slightly\n",
      " pleural rind surrounding the loculated components of the pleural fluid.  In\n",
      " addition to the predominantly anteriorly and lateral loculated components,\n",
      " note is made of an additional small posterior loculated extending into the\n",
      " azygoesophageal recess.  The Hounsfield units measurements of the pleural\n",
      " effusion are within the fluid range.\n",
      "\n",
      " The heart is not significantly enlarged and no pericardial effusion is\n",
      " evident.  There are several small mediastinal nodes which are less than 1 cm\n",
      " in greatest short axis dimension and do not meet strict CT criteria for\n",
      " abnormal enlargement.  Assessment of the hilar is limited due to postoperative\n",
      " change and lack of intravenous contrast, but there is a suggestion of right\n",
      " hilar lymph node enlargement.\n",
      "\n",
      " Imaging of the upper portion of the abdomen reveals no significant abnormality\n",
      " in the upper portion of the liver and spleen, but these organs are not\n",
      " completely imaged on this study.  The adrenal glands were not included on the\n",
      " exam.\n",
      "                                                             (Over)\n",
      "\n",
      " [**2996-12-2**] 10:25 AM\n",
      " CT CHEST W/O CONTRAST                                           Clip # [**Clip Number (Radiology) 38620**]\n",
      " Reason: please eval for loculated pleural effusion on the right and\n",
      " Field of view: 30\n",
      " ______________________________________________________________________________\n",
      "                                 FINAL REPORT\n",
      " (Cont)\n",
      "\n",
      " Assessment of the lungs reveals extensive linear opacities at the lung apices,\n",
      " probably reflecting scarring.  Additional sharply geographically marginated\n",
      " areas of increased opacity in the right paramediastinal region with some\n",
      " traction bronchiectasis is noted and may reflect post-radiation change.  A\n",
      " small wedge-shaped ground-glass opacity is seen in the left upper lobe\n",
      " peripherally.  Within the right lung base, there is an area of ground-glass\n",
      " opacification as well as thickened septal lines adjacent to the areas of\n",
      " loculated pleural effusion.  There are also some patchy consolidation\n",
      " peripherally in the right lower lobe, adjacent to the area of loculated\n",
      " pleural fluid.  Multiple band-like linear opacities are seen in the left lung\n",
      " base.  Additionally, there are two adjacent small lung nodules in the left\n",
      " lobe laterally abutting with one of them abutting the pleural surface\n",
      " measuring up to 7 mm in diameter.  One of these contains a band like area of\n",
      " calcium.  No dominant masses are evident.\n",
      "\n",
      " Assessment of the airways reveals some narrowing of the bronchus intermedius\n",
      " which occurs during an area of respiratory excursion and reflect a component\n",
      " of bronchomalacia.  No obstructing airway lesions are identified.  Note is\n",
      " made of increased opacities adjacent to the superior segment region of left\n",
      " lower lobe with pleural and parenchymal components and some calcification\n",
      " probably reflecting postoperative change.  Finally, a slightly nodular opacity\n",
      " is seen within a linear band like opacification at the extreme left lung base.\n",
      "\n",
      " Assessment of the skeletal structures of the thorax reveals deformities of\n",
      " right lateral ribs probably reflecting [** Location **]us surgery.  No suspicious lytic\n",
      " or blastic lesions are identified.\n",
      "\n",
      " IMPRESSION:\n",
      "\n",
      " 1.  Moderate-sized multiloculated right pleural effusion, which contains a\n",
      " slightly thickened pleural rind, concerning for empyema in this patient with\n",
      " history of pneumonia.\n",
      "\n",
      " 2.  Patchy peripheral consolidation in right lower lobe as well as more\n",
      " diffuse ground-glass opacity, in keeping with history of pneumonia.\n",
      "\n",
      " 3.  Right hilar lymph node enlargement, as well as slightly enlarged nodes in\n",
      " the azygoesophageal recess, which may be reactive in this patient with right\n",
      " lower lobe pneumonia.  Follow up CT after treatment of acute infectious\n",
      " process would be helpful to ensure resolution.\n",
      "\n",
      " 4.  Postoperative and probable post-radiation changes in the right lung\n",
      " related to history of partially resected lung cancer.  It is difficult to\n",
      " evaluate for residual disease in the absence of older films for comparison and\n",
      " in the setting of acute infection.\n",
      "                                                             (Over)\n",
      "\n",
      " [**2996-12-2**] 10:25 AM\n",
      " CT CHEST W/O CONTRAST                                           Clip # [**Clip Number (Radiology) 38620**]\n",
      " Reason: please eval for loculated pleural effusion on the right and\n",
      " Field of view: 30\n",
      " ______________________________________________________________________________\n",
      "                                 FINAL REPORT\n",
      " (Cont)\n",
      "\n",
      " 5.  Peripheral ground-glass opacity in left upper lobe, possibly due to\n",
      " additional site of acute infection, but nonspecific.\n",
      "\n",
      " 6.  Two small left lower lung nodules, one of which contains a component of\n",
      " calcification.  These are difficult to assess due to respiratory motion.  At\n",
      " the time of follow up CT, thin section images of these nodules are suggested\n",
      " to better assess for calcium and to help distinguish between benign granulomas\n",
      " and foci of metastatic disease. Granulomatous disease is certainly possible\n",
      " given history of prior left lower lobe surgery for TB.\n",
      "\n",
      " 7.  Emphysema.\n",
      "\n",
      " 8.  Apparent postoperative scarring related to surgery in superior segment\n",
      " left lower lobe region.\n",
      "\n",
      " ADDENDUM:\n",
      "\n",
      " Additional clinical history has just been provided that the patient's\n",
      " partially resected right upper lobe lung cancer was for a bronchoalveolar cell\n",
      " subtype.  With this additional history in mind, the areas of consolidation and\n",
      " ground-glass in the right lower lobe as well as the additional areas of\n",
      " ground-glass opacity of the left upper could potentially be due to\n",
      " bronchoalveolar cell carcinoma.  As pneumonia may also produce the same\n",
      " appearance, comparison to older CT scans and  careful follow up after\n",
      " antibiotic therapy would be helpful in helping to distinguish between these\n",
      " entities.\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(corpusList[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate Document-Term Frequency Counts\n",
    "\n",
    "#### In this step we tokenize our text and remove stop words in addition to generating our frequency counts.\n",
    "\n",
    "#### 1) How many documents are we working with and how many features (unigrams & bigrams)?\n",
    "\n",
    "#### 2) Can you figure out what max_df and min_df is doing to our feature count?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda3\\lib\\site-packages\\sklearn\\feature_extraction\\text.py:300: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['could', 'might', 'must', 'need', 'sha', 'wo', 'would'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1500, 920)\n",
      "\n",
      "['abdomen', 'abdomen pelvis', 'abdominal', 'abnormalities', 'abnormality', 'acquired', 'acute', 'addition', 'additional', 'additionally', 'adenopathy', 'adjacent', 'administered', 'administration', 'administration cc', 'administration iv', 'admitted', 'adrenal', 'adrenal glands', 'air', 'air bronchograms', 'airspace', 'airways', 'airways patent', 'allowing', 'along', 'also', 'also noted', 'although', 'alveolar', 'amount', 'amt', 'amt final', 'amt underlying', 'angle', 'anterior', 'aorta', 'aortic', 'apex', 'apical', 'apices', 'appear', 'appearance', 'appears', 'appropriate', 'approximately', 'approximately cm', 'arch', 'area', 'areas', 'areas consolidation', 'arterial', 'arteries', 'artery', 'artery calcifications', 'ascites', 'aspect', 'aspiration', 'assess', 'assessment', 'associated', 'asthma', 'atelectasis', 'atelectasis left', 'atelectasis right', 'atherosclerotic', 'atrial', 'atrium', 'attenuation', 'available', 'axial', 'axial images', 'axillary', 'axillary lymphadenopathy', 'axis', 'background', 'base', 'bases', 'basilar', 'bibasilar', 'bibasilar atelectasis', 'bilateral', 'bilateral lower', 'bilateral pleural', 'bilaterally', 'bladder', 'blastic', 'blunting', 'body', 'bone', 'bone windows', 'bones', 'bowel', 'breath', 'bronchi', 'bronchial', 'bronchograms', 'bronchus', 'c', 'c recons', 'ca', 'cabg', 'cad', 'calcification', 'calcifications', 'calcified', 'caliber', 'cancer', 'cardiac', 'cardiac mediastinal', 'cardiac silhouette', 'cardiomediastinal', 'cardiomediastinal silhouette', 'cardiomegaly', 'carina', 'catheter', 'cc', 'cc optiray', 'central', 'central venous', 'centrilobular', 'centrilobular emphysema', 'change', 'change final', 'changed', 'changes', 'chest clip', 'chest compared', 'chest comparison', 'chest contrast', 'chest ct', 'chest history', 'chest indication', 'chest obtained', 'chest pain', 'chest performed', 'chest radiograph', 'chest radiographs', 'chest single', 'chest tube', 'chest w', 'chest wall', 'chest without', 'chf', 'chf final', 'chronic', 'clear', 'clinical', 'clinical history', 'clinical indication', 'clinically', 'clips', 'cm', 'cm carina', 'collapse', 'collimation', 'colon', 'compared', 'compared location', 'compared prior', 'comparison', 'comparison chest', 'comparison ct', 'comparison findings', 'comparison made', 'comparison prior', 'comparison technique', 'comparisons', 'compatible', 'complete', 'component', 'compression', 'concern', 'concerning', 'condition chf', 'condition copd', 'condition severe', 'condition sob', 'confluent', 'congestion', 'congestive', 'congestive heart', 'considered', 'consistent', 'consolidation', 'consolidation left', 'consolidation right', 'consolidations', 'cont', 'contiguous', 'contiguous axial', 'continued', 'contour', 'contours', 'contours stable', 'contours unchanged', 'contraindications', 'contraindications iv', 'contrast', 'contrast clip', 'contrast ct', 'contrast final', 'contrast optiray', 'contrast wet', 'copd', 'copd exacerbation', 'copd flare', 'coronal', 'coronal sagittal', 'coronary', 'coronary arteries', 'coronary artery', 'correlation', 'costophrenic', 'costophrenic angle', 'cough', 'could', 'could represent', 'criteria', 'ct', 'ct abdomen', 'ct chest', 'ct non', 'ct pelvis', 'cta', 'cta chest', 'cxr', 'cyst', 'dated', 'day', 'days', 'decrease', 'decreased', 'defects', 'definite', 'degenerative', 'degenerative changes', 'degree', 'demonstrate', 'demonstrated', 'demonstrates', 'dense', 'density', 'dependent', 'described', 'diameter', 'diaphragm', 'differential', 'difficult', 'diffuse', 'dilatation', 'discussed', 'discussed name', 'disease', 'distal', 'distress', 'distress reason', 'distribution', 'due', 'dyspnea', 'earlier', 'edema', 'edema final', 'effusion', 'effusion final', 'effusion left', 'effusion pneumothorax', 'effusion right', 'effusion small', 'effusion underlying', 'effusions', 'effusions final', 'effusions right', 'either', 'elevated', 'embolism', 'embolus', 'emphysema', 'emphysematous', 'emphysematous changes', 'endotracheal', 'endotracheal tube', 'enlarged', 'enlarged mediastinal', 'enlargement', 'esophagus', 'et', 'et tube', 'etiology', 'ett', 'eval', 'evaluate', 'evaluated', 'evaluation', 'evidence', 'evidence pneumothorax', 'evidence pulmonary', 'exacerbation', 'exacerbation underlying', 'exam', 'examination assess', 'examination eval', 'examination evaluate', 'examination please', 'exclude', 'excluded', 'extending', 'extensive', 'extent', 'failure', 'failure underlying', 'female', 'fever', 'fevers', 'field', 'field view', 'filling', 'filling defects', 'film', 'films', 'findings', 'findings discussed', 'findings impression', 'first', 'first name', 'fissure', 'flare', 'fluid', 'focal', 'focal consolidation', 'foci', 'foley', 'foley catheter', 'follow', 'following', 'followup', 'fracture', 'fractures', 'free', 'free air', 'free fluid', 'frontal', 'gallbladder', 'given', 'glands', 'glass', 'glass opacities', 'great', 'great vessels', 'greater', 'greater left', 'greater right', 'grossly', 'ground', 'ground glass', 'heart', 'heart enlarged', 'heart failure', 'heart normal', 'heart size', 'helical', 'hemidiaphragm', 'hemithorax', 'hernia', 'high', 'hilar', 'hilar contours', 'hilar lymph', 'hilar lymphadenopathy', 'history', 'history copd', 'hours', 'however', 'hx', 'hyperinflated', 'hypertension', 'hypoxia', 'hypoxia reason', 'identified', 'identified impression', 'ij', 'image', 'imaged', 'images', 'images chest', 'images obtained', 'imaging', 'impression evidence', 'impression interval', 'impression pfi', 'improved', 'improvement', 'improving', 'including', 'incompletely', 'increase', 'increased', 'increased size', 'increasing', 'indication', 'indication history', 'indication male', 'indication shortness', 'indications', 'infection', 'infectious', 'infectious process', 'infiltrate', 'infiltrate final', 'infiltrates', 'initial', 'initial pre', 'inlet', 'internal', 'internal jugular', 'interstitial', 'interstitial edema', 'interval', 'interval change', 'interval improvement', 'interval increase', 'intravenous', 'intravenous contrast', 'intubated', 'intubation', 'involving', 'ionic', 'ionic contrast', 'iv', 'iv contrast', 'jugular', 'junction', 'kidney', 'kidneys', 'known', 'l', 'large', 'largest', 'last', 'last name', 'layering', 'leads', 'left', 'left base', 'left costophrenic', 'left effusion', 'left greater', 'left kidney', 'left lower', 'left lung', 'left pleural', 'left retrocardiac', 'left subclavian', 'left upper', 'lesion', 'lesions', 'lesions identified', 'less', 'level', 'likely', 'likely due', 'likely representing', 'limited', 'limits', 'line', 'line placement', 'linear', 'lines', 'lingula', 'liver', 'liver spleen', 'lll', 'lobe', 'lobe atelectasis', 'lobe consolidation', 'lobe opacity', 'lobe pneumonia', 'lobe right', 'lobes', 'located', 'location', 'location us', 'location usly', 'loculated', 'loops', 'loss', 'low', 'lower', 'lower lobe', 'lower lobes', 'lower lung', 'lung', 'lung apices', 'lung base', 'lung bases', 'lungs', 'lymph', 'lymph node', 'lymph nodes', 'lymphadenopathy', 'lytic', 'lytic blastic', 'lytic sclerotic', 'made', 'made prior', 'main', 'main pulmonary', 'male', 'malignancy', 'marked', 'mass', 'may', 'may represent', 'mdct', 'measures', 'measuring', 'measuring mm', 'median', 'median sternotomy', 'mediastinal', 'mediastinal contours', 'mediastinal hilar', 'mediastinal lymph', 'mediastinum', 'mid', 'middle', 'middle lobe', 'mild', 'mildly', 'minimal', 'mitral', 'mitral valve', 'mm', 'moderate', 'motion', 'multidetector', 'multifocal', 'multifocal pneumonia', 'multiplanar', 'multiplanar reformatted', 'multiple', 'name', 'name stitle', 'name un', 'nasogastric', 'nasogastric tube', 'new', 'ng', 'ng tube', 'node', 'nodes', 'nodular', 'nodule', 'nodules', 'non', 'non ionic', 'none', 'nonionic', 'normal', 'normal caliber', 'normal limits', 'normal size', 'note', 'note made', 'noted', 'obstruction', 'obtained', 'obtained chest', 'one', 'opacification', 'opacities', 'opacity', 'opacity left', 'opacity right', 'optiray', 'optiray amt', 'oral', 'oral contrast', 'osseous', 'osseous structures', 'otherwise', 'overall', 'overlying', 'pacemaker', 'pain', 'pancreas', 'paratracheal', 'parenchyma', 'parenchymal', 'partially', 'particularly', 'patchy', 'patchy opacities', 'patent', 'pathologically', 'pathologically enlarged', 'patient', 'patient status', 'pattern', 'pe', 'pe contraindications', 'pelvic', 'pelvis', 'pelvis contrast', 'performed', 'performed without', 'pericardial', 'pericardial effusion', 'perihilar', 'peripheral', 'persistent', 'pfi', 'physician', 'physician initial', 'picc', 'picc line', 'place', 'placement', 'placement final', 'please', 'please assess', 'please eval', 'please evaluate', 'pleural', 'pleural effusion', 'pleural effusions', 'pleural thickening', 'pna', 'pneumonia', 'pneumonia final', 'pneumonia underlying', 'pneumothorax', 'pneumothorax identified', 'portion', 'portions', 'position', 'possible', 'possibly', 'post', 'posterior', 'posteriorly', 'pre', 'pre number', 'predominantly', 'presence', 'present', 'prevascular', 'prior', 'prior chest', 'prior study', 'probable', 'probably', 'process', 'progression', 'prominence', 'prominent', 'provisional', 'provisional findings', 'proximal', 'ptx', 'pulm', 'pulmonary', 'pulmonary arterial', 'pulmonary arteries', 'pulmonary artery', 'pulmonary edema', 'pulmonary embolism', 'pulmonary vascular', 'pulmonary vascularity', 'pulmonary vasculature', 'question', 'r', 'radiograph', 'radiograph chest', 'radiographs', 'radiology ct', 'reactive', 'read', 'reason assess', 'reason eval', 'reason evaluate', 'reason please', 'recent', 'recommended', 'recons', 'recons clip', 'recons ct', 'redistribution', 'reflect', 'reformations', 'reformats', 'reformatted', 'reformatted images', 'region', 'regions', 'related', 'remain', 'remains', 'removed', 'renal', 'renal failure', 'replacement', 'report chest', 'report clinical', 'report cont', 'report ct', 'report history', 'report indication', 'report indications', 'represent', 'representing', 'represents', 'residual', 'resolution', 'resp', 'resp distress', 'respiratory', 'respiratory distress', 'respiratory failure', 'retrocardiac', 'retroperitoneal', 'reviewed', 'rib', 'rib fractures', 'right', 'right atrium', 'right base', 'right greater', 'right hilar', 'right ij', 'right internal', 'right left', 'right lower', 'right lung', 'right middle', 'right pleural', 'right sided', 'right upper', 'rll', 'rule', 'sagittal', 'satisfactory', 'satisfactory position', 'scan', 'scarring', 'scattered', 'sclerotic', 'secondary', 'secretions', 'seen', 'seen left', 'seen right', 'seen within', 'segment', 'segmental', 'septal', 'series', 'setting', 'several', 'severe', 'severe emphysema', 'short', 'shortness', 'shortness breath', 'show', 'side', 'sided', 'sided pleural', 'significant', 'significant change', 'significantly', 'significantly changed', 'silhouette', 'similar', 'simple', 'since', 'since prior', 'single', 'single view', 'size', 'size normal', 'slight', 'slightly', 'small', 'small amount', 'small bilateral', 'small bowel', 'small left', 'small pleural', 'small right', 'smaller', 'sob', 'sob reason', 'soft', 'soft tissue', 'somewhat', 'space', 'spine', 'spleen', 'stable', 'stable appearance', 'status', 'status post', 'sternotomy', 'stitle', 'stomach', 'structures', 'structures unremarkable', 'studies', 'study', 'subcarinal', 'subclavian', 'subcutaneous', 'subdiaphragmatic', 'subsegmental', 'suggest', 'suggesting', 'suggestive', 'superimposed', 'superior', 'supine', 'surgical', 'suspicious', 'suspicious lytic', 'svc', 'tailored', 'technique', 'technique axial', 'technique contiguous', 'technique mdct', 'terminates', 'terminating', 'thickening', 'thoracic', 'thoracic aorta', 'thoracic inlet', 'thoracic spine', 'though', 'throughout', 'thyroid', 'time', 'tiny', 'tip', 'tissue', 'tissues', 'today', 'trace', 'trachea', 'tracheostomy', 'tracheostomy tube', 'tube', 'tube placement', 'tube tip', 'tubes', 'two', 'un', 'unchanged', 'unchanged impression', 'unchanged position', 'unremarkable', 'unremarkable impression', 'upper', 'upper abdomen', 'upper lobe', 'upper lobes', 'upper lung', 'upper zone', 'upright', 'upright chest', 'us', 'us study', 'usly', 'valve', 'valve replacement', 'vascular', 'vascular congestion', 'vascularity', 'vasculature', 'vein', 'venous', 'venous catheter', 'venous line', 'ventricular', 'versus', 'vessels', 'view', 'view chest', 'view contrast', 'view final', 'view underlying', 'views', 'visualized', 'volume', 'vs', 'w', 'w c', 'wall', 'wall thickening', 'well', 'wet', 'wet read', 'windows', 'wires', 'within', 'within left', 'within normal', 'within right', 'without', 'without contrast', 'without evidence', 'without intravenous', 'without iv', 'worse', 'worsening', 'would', 'x', 'x cm', 'yo', 'zone', 'zone redistribution']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#cv = CountVectorizer(lowercase=True, max_df=0.90, max_features=None, min_df=0.033,\n",
    "#                     ngram_range=(1, 2), preprocessor=None, stop_words=cachedStopWords,\n",
    "#                     strip_accents=None, tokenizer=tokenize, vocabulary=None)\n",
    "cv = TfidfVectorizer(lowercase=True, max_df=0.90, max_features=None, min_df=0.033,\n",
    "                     ngram_range=(1, 2), preprocessor=None, stop_words=cachedStopWords,\n",
    "                     strip_accents=None, tokenizer=tokenize, vocabulary=None)\n",
    "X = cv.fit_transform(corpusList)\n",
    "print(X.shape)\n",
    "print()\n",
    "lexicon = cv.get_feature_names()\n",
    "print (lexicon)\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Construct our Classes\n",
    "\n",
    "#### We need to assign a class for each classification. We typically assign numeric values to classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus['label'] = corpus['label'].map({'COPD': 0, 'CHF': 0, 'PNA': 1})\n",
    "corpus.head()\n",
    "Y = corpus['label'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's Run It!\n",
    "\n",
    "#### We will generate models and evaluate the modes using 10-fold cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model  0 : DT K = 1\n",
      "   Fold 1 accuracy: 83.33 %\n",
      "   Fold 2 accuracy: 83.33 %\n",
      "   Fold 3 accuracy: 84.00 %\n",
      "   Fold 4 accuracy: 84.67 %\n",
      "   Fold 5 accuracy: 85.33 %\n",
      "   Fold 6 accuracy: 82.00 %\n",
      "   Fold 7 accuracy: 80.67 %\n",
      "   Fold 8 accuracy: 85.33 %\n",
      "   Fold 9 accuracy: 82.67 %\n",
      "   Fold 10 accuracy: 80.00 %\n",
      "     Overall test accuracy: 83.13 %\n",
      "     Overall training accuracy: 100.00 %\n",
      "model  1 : DT K = 3\n",
      "   Fold 1 accuracy: 82.67 %\n",
      "   Fold 2 accuracy: 82.00 %\n",
      "   Fold 3 accuracy: 88.00 %\n",
      "   Fold 4 accuracy: 85.33 %\n",
      "   Fold 5 accuracy: 78.67 %\n",
      "   Fold 6 accuracy: 78.00 %\n",
      "   Fold 7 accuracy: 82.67 %\n",
      "   Fold 8 accuracy: 84.67 %\n",
      "   Fold 9 accuracy: 85.33 %\n",
      "   Fold 10 accuracy: 82.67 %\n",
      "     Overall test accuracy: 83.00 %\n",
      "     Overall training accuracy: 98.33 %\n",
      "model  2 : DT K = 5\n",
      "   Fold 1 accuracy: 82.00 %\n",
      "   Fold 2 accuracy: 83.33 %\n",
      "   Fold 3 accuracy: 80.00 %\n",
      "   Fold 4 accuracy: 78.00 %\n",
      "   Fold 5 accuracy: 90.00 %\n",
      "   Fold 6 accuracy: 86.00 %\n",
      "   Fold 7 accuracy: 85.33 %\n",
      "   Fold 8 accuracy: 79.33 %\n",
      "   Fold 9 accuracy: 85.33 %\n",
      "   Fold 10 accuracy: 80.67 %\n",
      "     Overall test accuracy: 83.00 %\n",
      "     Overall training accuracy: 95.87 %\n",
      "model  3 : DT K = 10\n",
      "   Fold 1 accuracy: 88.67 %\n",
      "   Fold 2 accuracy: 80.67 %\n",
      "   Fold 3 accuracy: 81.33 %\n",
      "   Fold 4 accuracy: 86.67 %\n",
      "   Fold 5 accuracy: 84.67 %\n",
      "   Fold 6 accuracy: 79.33 %\n",
      "   Fold 7 accuracy: 86.67 %\n",
      "   Fold 8 accuracy: 84.00 %\n",
      "   Fold 9 accuracy: 84.00 %\n",
      "   Fold 10 accuracy: 80.67 %\n",
      "     Overall test accuracy: 83.67 %\n",
      "     Overall training accuracy: 92.47 %\n",
      "model  4 : DT K = 15\n",
      "   Fold 1 accuracy: 86.00 %\n",
      "   Fold 2 accuracy: 81.33 %\n",
      "   Fold 3 accuracy: 78.67 %\n",
      "   Fold 4 accuracy: 86.67 %\n",
      "   Fold 5 accuracy: 82.67 %\n",
      "   Fold 6 accuracy: 87.33 %\n",
      "   Fold 7 accuracy: 82.67 %\n",
      "   Fold 8 accuracy: 84.00 %\n",
      "   Fold 9 accuracy: 83.33 %\n",
      "   Fold 10 accuracy: 85.33 %\n",
      "     Overall test accuracy: 83.80 %\n",
      "     Overall training accuracy: 90.93 %\n",
      "model  5 : DT K = 20\n",
      "   Fold 1 accuracy: 88.00 %\n",
      "   Fold 2 accuracy: 88.00 %\n",
      "   Fold 3 accuracy: 81.33 %\n",
      "   Fold 4 accuracy: 85.33 %\n",
      "   Fold 5 accuracy: 80.67 %\n",
      "   Fold 6 accuracy: 83.33 %\n",
      "   Fold 7 accuracy: 84.00 %\n",
      "   Fold 8 accuracy: 82.00 %\n",
      "   Fold 9 accuracy: 81.33 %\n",
      "   Fold 10 accuracy: 84.67 %\n",
      "     Overall test accuracy: 83.87 %\n",
      "     Overall training accuracy: 89.47 %\n",
      "model  6 : DT K = 40\n",
      "   Fold 1 accuracy: 86.67 %\n",
      "   Fold 2 accuracy: 80.67 %\n",
      "   Fold 3 accuracy: 83.33 %\n",
      "   Fold 4 accuracy: 80.00 %\n",
      "   Fold 5 accuracy: 86.67 %\n",
      "   Fold 6 accuracy: 84.67 %\n",
      "   Fold 7 accuracy: 76.00 %\n",
      "   Fold 8 accuracy: 84.00 %\n",
      "   Fold 9 accuracy: 82.67 %\n",
      "   Fold 10 accuracy: 84.67 %\n",
      "     Overall test accuracy: 82.93 %\n",
      "     Overall training accuracy: 87.53 %\n"
     ]
    }
   ],
   "source": [
    "results = []\n",
    "cont = []\n",
    "kf = KFold(n_splits=10, shuffle=True)\n",
    "K = [1, 3, 5, 10, 15, 20, 40]\n",
    "for k in range(len(K)):\n",
    "    fold = 1\n",
    "    truth = []\n",
    "    dt_prediction = []\n",
    "    print(\"model \", k, \": DT K = \" + str(K[k]))\n",
    "    test_count = 0\n",
    "    dt = DecisionTreeClassifier(random_state=0, min_samples_leaf=K[k], criterion=\"entropy\")\n",
    "    for train_idx, test_idx in kf.split(X):\n",
    "        trainX = X[train_idx]\n",
    "        trainY = Y[train_idx]\n",
    "        testX = X[test_idx]\n",
    "        testY = Y[test_idx]\n",
    "        truth.append(testY)\n",
    "        dt.fit(trainX, trainY)\n",
    "        Y_hat = dt.predict(testX)\n",
    "        dt_prediction.append(Y_hat)\n",
    "        print(\"   Fold %d accuracy: %.2f %%\" % (fold, ((np.sum(Y_hat == testY)/len(testY)) * 100.0)))                        \n",
    "        fold += 1\n",
    "    truth = np.concatenate(truth, axis=0)    \n",
    "    dt_prediction = np.concatenate(dt_prediction, axis=0)\n",
    "    test_results = np.sum(dt_prediction == truth)/len(truth)\n",
    "    print(\"     Overall test accuracy: %.2f %%\" % (test_results * 100))  \n",
    "    dt = dt.fit(X, Y)\n",
    "    Y_hat = dt.predict(X)\n",
    "    train_results = np.sum(Y_hat == Y)/len(Y)\n",
    "    print(\"     Overall training accuracy: %.2f %%\" % (train_results * 100.0))  \n",
    "    results.append([train_results, test_results])   \n",
    "    cont.append([truth, dt_prediction])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3XecVNX9//HXextLWVyaVAELFrAgrGDv3VjRCNYYDJqvGruiYgl2fyaxRkMUu2ILkSQao0YlsYOioAQBGyug9I6w7Of3x7m7DMPs7myZvbvs5/l43MfOrfO5l2E+c8659xyZGc4551xNZcUdgHPOucbNE4lzzrla8UTinHOuVjyROOecqxVPJM4552rFE4lzzrla8UTinNuIpF9I+m+a2z4q6aZMx+QaLk8kjYSkbyStkrRM0mJJ70o6V1JWtP4VScujaa2kNQnzD1Zy3P0lmaQr6u9sXGUkvSXp7IT5/SUtkjQ4xbY9o3+/j5OWt48+A9/UQ8gVihLSuuhzuFTSJEk/izOmdCRc15y4Y2kMPJE0LkebWQHQA7gNuBJ4GMDMjjCzVmbWCngKuKNs3szOreSYZwILo7/1qrH9J40jXkmHAn8FfmlmYyrZtKWkHRPmTwG+zmhw6Xsv+lwWEj6vz0lqW50DKGg031eN7bNdW43mH8atZ2ZLzGwccDJwZtIXSNoktQBOBM4DekkqSlq/d1TyWSxplqRfRMubS/qdpG8lLZH032jZ/pKKk47xjaSDo9c3SHpB0pOSlgK/kDRA0nvRe8yRdJ+kvIT9+0h6TdJCST9IulpSJ0krJbVL2K6/pHmScpPev0tUkmubsGxXSfPLtpX0S0lTo1/9r0rqkbCtSTpP0nRgevSF9gdJP0bn/lnZ9U9RkiivHqpsv0r+fX4GPAecYmZjK9sWeIINfwycATyedLwdohgXS/pc0jEJ69pJGheVGj4Etk7ad/uEf4dpkn5eRTwbMbNSYDTQHNhKUhtJf4/+3RZFr7slvOdbkm6W9A6wMtrnrOjfapmkrySdk7D9/pKKJV0RXec5ko6TdKSkL6PYr07YPkvScEkzJS2QlJjgxkd/FyuUpvaI9kn7s1Ld69OYeSJpxMzsQ6AY2KeGhxgELAeeB14lfPkAIKk78ApwL9AB6AtMilbfCfQH9gTaAlcApWm+57HAC4Rfp08B64CLgfbAHsBBwP9FMRQArwP/BLoA2wBvmNlc4C0g8cvsNGCMma1NfDMzmw28F51rmVOAF8xsraTjgKuBE6Lz/A/wTFLMxwEDgd7AocC+wLbROZwMLEjjvKu739HAk8CJZvZyGsd/EhgsKVvSDkAB8EHZyihp/g34F7A5cAHwlKTtok3uB1YDnYFfRlPZvi2B14Cno32HAH+U1CeNuMop/Eo/m/CZm074/nmEUMLuDqwC7kva7XRgWHQ+3wI/Aj8DWgNnAX+Q1C9h+05APtAVuA74M+Gz0Z/w/+Q6SVtF2/6G8G+7H+HztSi6DhD+rQAKo1L9ezX4rDQdZuZTI5iAb4CDUyx/H7gmadmjwE1pHPN14K7o9RBgHpAbzV8FjE2xTxbhP/wuKdbtDxRXFDdwAzC+ipguKnvfKKZPKtjuZOCd6HU2MBcYUMG2ZwP/jl4LmAXsG82/AgxNOr+VQI9o3oADE9YfCHwJ7A5kJb3PW8DZCfO/AP5b1X4p4n0LWAp8CDSvYtueUYw50b/nYYRqz2uAg4Fvou32ia5RVsK+z0T/JtnAWmD7hHW3JMR+MvCfpPf9E3B9VZ+36BqUAIuB+dHndaPPcbRtX2BR0nUYWcX5/xW4MOHztwrIjuYLomszMGH7icBx0eupwEEJ6zpH1yEn8bomrK/WZ6UpTV4iafy6Eto4qkXSFsABhFIBwEuEX3JHRfNbADNT7No+2i7VunTMSopj26hKY25U3XVL9B6VxVAWb+/o1+UhwBILJbRUXgD2kNSF8EvTCL8mIfwavjuq7llMuJYiXNeNYjazfxN+Nd8P/CBplKTWVZ10Dfa7FvgJ+KukZlUdP/I44Yt7CKGEkqgLMMtC9VKZbwnn2YHw5TkraV2ZHsDAsmsUXadTCb/+0/G+mRWaWXsz293MXodQtSrpTwpVpEsJ1UmFkrIT9k3+vBwh6f2ommoxcCTrPy8AC8xsXfR6VfT3h4T1q4BWCec1NuGcphJKyB0rOI9qfVaaEk8kjZik3Qgf4rRu00xyOuHf/2+S5gJfERJEWfXWLJLqySPzCVUgqdatAFokxJdN+JJKlNzd9APA/4BeZtaaUHWgKmLAzFYT2g9Ojc7liVTbRdsuJlTp/JxQrfWMRT8ho/c4J/qiK5uam9m7FcVsZveYWX+gD6Gq6vJU50/SF20l+6WygvAluRnwgpLafirwIuGHwFdm9m3SutnAFtqwwbo78D2hJFpCSNyJ68rMAt5OukatzOzXacRUmUuB7Qglhtasr05Swjbl1z5KqC8SqlY7mlkh8HLS9tUxCzgi6bzyzex7Nv6clm1frc9KU+GJpBGS1DpqiB0DPGlmk2twmDOA3xKqE8qmQcBRCo3YTwEHS/q5pJyoMbavrW8w/b1CQ3a2pD2i/+RfAvmSjoq++EYAVf2aLiBU4yyXtD2Q+OX0d6CTpIskNZNUIGlgwvqyX+DHsPEv8GRPR+c8KHpd5kHgqrL6fkmbSTqpooNI2k3SwOj8VhCSatkv4EnACdEv7W2AoWnul5KZLQMOJ5Qmnk76pZ5q+xWEKrSzU6z+IHrfKyTlStqf0A4zJvoF/xfghij23mzYcP93YFtJp0f75kbns0Nl8aShgFBCWBw1cl9fxfZ5hM/TPKBE0hGEtqeaehC4uazBXFIHScdG6+YR2v22Sto+7c9KU+KJpHH5m6RlhF9G1wC/JzQ4Vouk3Ql1wPeb2dyEaRwwAxhiZt8RfhFfSijCTwJ2iQ5xGTAZ+Chadzuh7n0JoaH8IcIv3RWEmwEqcxmhlLCM0DD6bNmK6Iv0EMIX3lxCA+0BCevfIfxn/9jMvqnifcYBvYAfzOzThGOMjeIfE1WvTAGOqOQ4raM4FxGqfxYQfiED/AFYQ6hKeYz11YZV7VehqDR1CKEE87iquAXWzCaY2UbVgWa2hpBwjyCUKv8InGFm/4s2OZ9Q5TOX0ObxSMK+ywhf2IMJJZu5hGuWbpVbRe4i3MFV1nbyz8o2juL4DaEkuojwuRlXi/e/O9r/X9H/q/cJDeWY2UrgZuCdqCpr9xp8VpoMrS/hO9f4SPo38LSZPRR3LM41VZ5IXKMVtRG9BmwR/Vp1zsXAq7ZcoyTpMcLtrhd5EnEuXl4icc45VyteInHOOVcrTaJjsfbt21vPnj3jDsM55xqViRMnzjez5GfBNtIkEknPnj2ZMGFC3GE451yjIin5wdaUvGrLOedcrXgicc45VyueSJxzztVKk2gjcc7Fa+3atRQXF7N69eq4Q3Ep5Ofn061bN3Jz0+kbdGOeSJxzGVdcXExBQQE9e/ZEqmlnvS4TzIwFCxZQXFzMlltuWaNjZLRqS9JohSEvp1SwXpLukTRDYejRfgnrzpQ0PZrOTFjeX9LkaJ975J9K5xq81atX065dO08iDZAk2rVrV6vSYqbbSB4ldINdkSMIPbL2Igyn+QBAQpfSA4EBwPWS2kT7PBBtW7ZfZcd3zjUQnkQartr+22Q0kZjZeCofve9Y4HEL3ieMjtaZMFzoa2a20MwWETrmOzxa19rM3osGJnqcMEZyRvzjH/Dww5k6unPObRrivmurKxsOTVkcLatseXGK5RuRNEzSBEkT5s2bV+3AzODBB+Hcc2H8+Grv7pxrQBYsWEDfvn3p27cvnTp1omvXruXza9asSesYZ511FtOmTat0m/vvv5+nnnqq0m3Stffee7PddtuVx3nyySfXyXEzIe7G9lTlKavB8o0Xmo0CRgEUFRVVu2dKCZ54AgYOhBNPhAkToHv3qvdzzjU87dq1Y9KkSQDccMMNtGrVissuu2yDbcwMMyMrK/Xv60ceeSTl8kTnnXde7YNN8Oyzz9K3b98K15eUlJCTk1PhfLr71VbcJZJiNhwnuhthBLbKlndLsTwjCgvhr3+F1avh+ONh5cpMvZNzLg4zZsxgxx135Nxzz6Vfv37MmTOHYcOGUVRURJ8+fRg5cmT5tnvvvTeTJk2ipKSEwsJChg8fzi677MIee+zBjz/+CMCIESO46667yrcfPnw4AwYMYLvttuPdd8PQ7itWrGDQoEHssssuDBkyhKKiovIkl47TTjuNSy+9lAMOOICrr76aESNGcM4553DIIYdw1llnsWrVKs4880x22mkn+vXrx/ioSuWhhx5i8ODB/OxnP+OII+p2YMe4SyTjgPMljSE0rC8xszmSXgVuSWhgPxS4yswWSloWDRX7AWEM7nszGeAOO8BTT8Gxx8KvfgVPPhlKK865mrnoIqjG92Za+vaF6Pu72r744gseeeQRHnzwQQBuu+022rZtS0lJCQcccAAnnngivXv33mCfJUuWsN9++3HbbbdxySWXMHr0aIYPH77Rsc2MDz/8kHHjxjFy5Ej++c9/cu+999KpUydefPFFPv30U/r167fRfmVOPvlkmjdvDsDhhx/ObbfdBsDMmTN54403yMrKYsSIEXzyySeMHz+e/Px8br/9dvLy8pg8eTKff/45Rx55JNOnTwfgvffeY9KkSbRp06bC96yJjCYSSc8A+wPtJRUT7sTKBTCzB4GXCeOCzwBWEo0/HiWMGwljggOMNLOyRvtfE+4Gaw68Ek0ZdfTRMHIkXHst7LorJJWInXON2NZbb81uu+1WPv/MM8/w8MMPU1JSwuzZs/niiy82SiTNmzcv/1Xfv39//vOf/6Q89gknnFC+zTfffAPAf//7X6688koAdtllF/r06VNhbBVVbZ100kkbVMEde+yx5Ofnlx//8ssvB6BPnz506dKFGTNmAHDooYfWeRKBDCcSMxtSxXoDUlYqmtloYHSK5ROAHeskwGq45prwK+rKK2HnneHQQ+s7Auc2DTUtOWRKy5Yty19Pnz6du+++mw8//JDCwkJOO+20lM9X5OXllb/Ozs6mpKQk5bGbNWu20TZ1MZhgYszJ85UdP3m/uhJ3G0mjIcGjj0KfPjB4MMycGXdEzrm6tnTpUgoKCmjdujVz5szh1VdfrfP32HvvvXnuuecAmDx5Ml988UWdHn/fffctv3Ns6tSpzJkzh2222aZO3yNZ3G0kjUqrVqHxfbfdQpvJe+9BQUHcUTnn6kq/fv3o3bs3O+64I1tttRV77bVXnb/HBRdcwBlnnMHOO+9Mv3792HHHHdlss81SbpvYRtKxY8e0EtsFF1zAOeecw0477URubi6PP/74BiWoTGgSY7YXFRVZXQ5s9frrcNhhIZm88AJUcLegcy4ydepUdthhh7jDaBBKSkooKSkhPz+f6dOnc+ihhzJ9+vQ6vR23JlL9G0maaGZFVe3rJZIaOPhguPNOuOQSuOkmuO66uCNyzjUWy5cv56CDDqKkpAQz409/+lPsSaS2Gnf0MbroIvjkE7j+ethll1A6cc65qhQWFjJx4sS4w6hTXilTQxL86U9QVASnnQZ13F7mnHONhieSWmjeHMaOhZYtQ4lk0aK4I3LOufrniaSWunWDv/wFvv0WTjkF1q2LOyLnnKtfnkjqwJ57wv33wz//CVdfHXc0zjlXvzyR1JFf/Qp+/Wu44w545pm4o3HOJaqLbuQBRo8ezdy5c8vn0+laPh0lJSVkZ2eXx9S3b1/+3//7f7U+bn3xu7bq0F13wZQpMHQobL996JfLORe/dLqRT8fo0aPp168fnTp1AtLrWj5dBQUFVfYC3FC6jU/mJZI6lJcXHlBs3x6OOw6inqWdcw3YY489xoABA+jbty//93//R2lpKSUlJZx++unstNNO7Ljjjtxzzz08++yzTJo0iZNPPrm8JJNO1/LTp09n4MCBDBgwgGuvvZbCwsJqxdetWzduvPFG9tprL8aOHcvee+/NNddcw7777st9993H119/zQEHHMDOO+/MIYccQnFxGPsvubv5TPISSR3bfPNwJ9fee8NJJ4Wn4HNz447KuYbjon9exKS5dduPfN9Ofbnr8Or3BjllyhTGjh3Lu+++S05ODsOGDWPMmDFsvfXWzJ8/n8mTJwOwePFiCgsLuffee7nvvvtS9shbUdfyF1xwAZdddhknnXQS9913X4WxLFu2bIPjjhgxghNPPBEInS2+8847ANx9990sXbq0fJyRI444grPPPptTTz2VUaNGcdFFF/HCCy8AG3Y3n0leIsmA/v3DWO/jx0MjquZ0rsl5/fXX+eijjygqKqJv3768/fbbzJw5k2222YZp06Zx4YUX8uqrr1bYF1ai5K7ly7qN/+CDDxg0aBAAp5xySoX7l1VtlU1lSQTYaJjdwYMHl7/+4IMPyufPOOOMDbq0T+5uPlO8RJIhp5wCzz8Pt90WGuI7dIg7IucahpqUHDLFzPjlL3/JjTfeuNG6zz77jFdeeYV77rmHF198kVGjRlV6rHS7lq+JyrqNr85+meIlkgy69dYwPG+Kz6hzrgE4+OCDee6555g/fz4Q7u767rvvmDdvHmbGSSedxG9/+1s+/vhjIJQali1bVq33GDBgAGPHjgVgzJgxdXsCwO67717eLf2TTz7JvvvuW+fvURUvkWTQ9tuH0sgDD8BvfgMZHhLAOVdNO+20E9dffz0HH3wwpaWl5Obm8uCDD5Kdnc3QoUMxMyRx++23A+F237PPPpvmzZvz4YcfpvUe99xzD6effjq33347Rx55ZIXVZMltJEcddRQ333xzlce/7777GDp0KLfeeisdO3as0zvJ0uXdyGfY3LkhgRx5JEQ/GpxrcppyN/IrVqygRYsWSOLJJ59k7NixvPjii3GHtRHvRr4B69QJLr8cbrgBPvgABg6MOyLnXH366KOPuOiiiygtLaVNmzaxlBgyzRNJPbj00lC9dcUV8NZboedg51zTsP/++1f5oGFjl9HGdkmHS5omaYak4SnW95D0hqTPJL0lqVu0/ABJkxKm1ZKOi9Y9KunrhHUb39DdwLRqFUok48fD3/8edzTOxaMpVKM3VrX9t8lYIpGUDdwPHAH0BoZI6p202Z3A42a2MzASuBXAzN40s75m1hc4EFgJ/Cthv8vL1ptZo0j1Q4fCdtvBlVdCHd4V6FyjkJ+fz4IFCzyZNEBmxoIFC8jPz6/xMTJZtTUAmGFmXwFIGgMcCyQOAdUbuDh6/Sbw1xTHORF4xcxWZjDWjMvNDc+UHH88PPJIuJvLuaaiW7duFBcXM2/evLhDcSnk5+fTrVu3Gu+fyUTSFZiVMF8MJDc1fwoMAu4GjgcKJLUzswUJ2wwGfp+0382SrgPeAIab2U/Jby5pGDAMoHv37rU5jzpz7LGw115hjPdTTgkDYjnXFOTm5rLlllvGHYbLkEy2kaRqUk4u114G7CfpE2A/4HugvOJHUmdgJ+DVhH2uArYHdgPaAlemenMzG2VmRWZW1KGBPFYuhS5T5s6F3yenRueca6QymUiKgS0S5rsBsxM3MLPZZnaCme0KXBMtW5Kwyc+BsWa2NmGfORb8BDxCqEJrNPbYA044IYxb8sMPcUfjnHO1l8lE8hHQS9KWkvIIVVTjEjeQ1F5SWQxXAaOTjjEEeCZpn87RXwHHAVMyEHtG3XorrFoFI0fGHYlzztVexhKJmZUA5xOqpaYCz5nZ55JGSjom2mx/YJqkL4GOQHl/AJJ6Eko0bycd+ilJk4HJQHvgpkydQ6Zsuy2ccw6MGgVffhl3NM45VzveRUpMfvghdJ1y2GFhMCznnGto0u0ixXv/jUnHjuFJ9xdfhPfeizsa55yrOU8kMbrkkvV9cTWBgqFzbhPliSRGLVvCb38L77wDL70UdzTOOVcznkhi9stfhnFLrrwS1q6tenvnnGtoPJHELCcHbr893L318MNxR+Occ9XniaQBOPpo2Gef0EPw8uVxR+Occ9XjiaQBKOs65Ycf4He/izsa55yrHk8kDcTAgXDSSev74nLOucbCE0kDcsst8NNP4U4u55xrLDyRNCDbbAPnngt//jP8739xR+Occ+nxRNLAXHsttGgBV10VdyTOOZceTyQNzOabh2dK/vpX+O9/447GOeeq5omkAbr4YujSBS69FNasiTsa55yrnCeSBqhFC7jzTvjwQxg82J94d841bJ5IGqghQ+Duu2Hs2PDak4lzrqHyRNKA/eY3YWz3F1+EU0+FkpKq93HOufqWE3cArnIXXxy6mL/0UsjKgiefDP1zOedcQ+FfSY3AJZfAunVhIKysLHj8cU8mzrmGw7+OGonLL4fSUhg+PCSTxx6D7Oy4o3LOOU8kjcqVV4ZkcvXVIZk88ognE+dc/DLa2C7pcEnTJM2QNDzF+h6S3pD0maS3JHVLWLdO0qRoGpewfEtJH0iaLulZSXmZPIeG5qqr4MYb4YknYOjQUOXlnHNxylgikZQN3A8cAfQGhkjqnbTZncDjZrYzMBK4NWHdKjPrG03HJCy/HfiDmfUCFgFDM3UODdWIEaFjx8ceg1/9KpRSnHMuLpkskQwAZpjZV2a2BhgDHJu0TW/gjej1mynWb0CSgAOBF6JFjwHH1VnEjch118H114fqrWHDPJk45+KTyUTSFZiVMF8cLUv0KTAoen08UCCpXTSfL2mCpPcllSWLdsBiMyt7oiLVMQGQNCzaf8K8efNqey4N0vXXh9LJww+HXoM9mTjn4pDJxnalWGZJ85cB90n6BTAe+B4oSxLdzWy2pK2Af0uaDCxN45hhodkoYBRAUVFRym0aOwlGjgwJ5JZbQgP8H/8Y/jrnXH3JZCIpBrZImO8GzE7cwMxmAycASGoFDDKzJQnrMLOvJL0F7Aq8CBRKyolKJRsds6mR4KabQjK57baQRO6/Pyx3zrn6kMnfrh8BvaK7rPKAwcC4xA0ktZdUFsNVwOhoeRtJzcq2AfYCvjAzI7SlnBjtcybwUgbPoVGQQonk8svhgQfgggvC0/DOOVcfMlYiMbMSSecDrwLZwGgz+1zSSGCCmY0D9gdulWSEqq3zot13AP4kqZSQ7G4zsy+idVcCYyTdBHwCPJypc2hMJLj99lAy+d3vwvMld93lJRPnXObJmsBP16KiIpswYULcYdSLsn65/vAHuPDC8NeTiXOuJiRNNLOiqrbzJ9s3MVIokZSWhm7os7PD2CaeTJxzmeKJZBMkhZLIunWhG/qsLLjjDk8mzrnM8ESyiZLgnntCyeTOO0Myue02TybOubrniWQTJsF994VkcscdoZrr5ps9mTjn6pYnkk2cFJ4rKS2FW28NJZMbb/Rk4pyrO55ImoCsrPB8SWlpKJFkZ4dOH51zri54ImkisrLgT38KyWTkyDB//fVxR+Wc2xR4ImlCsrLgz38OyeSGG8L8tdfGHZVzrrHzRNLEZGXBQw+FZHLddWH+mmvijso515h5ImmCsrNh9OiQTEaMCPPDNxq/0jnn0uOJpInKzoZHHw3J5KqrQsnkiivijso51xh5ImnCsrPDcL2lpXDllSGZXHZZ3FE55xobTyRNXE4OPPFESCaXXx6Sy8UXxx2Vc64x8UTiyMmBJ58MyeSSS0LJ5MIL447KOddYeCJxAOTmwtNPh2Ry0UUhmVxwQdxROecagypHSJR0vqQ29RGMi1duLowZA8cdB7/5TehaxTnnqpLOULudgI8kPSfpcMl7adqU5ebCs8/CMcfA+eeHrlWcc64yVSYSMxsB9CIMafsLYLqkWyRtneHYXEzy8uD55+Hoo+H//i90reKccxVJp0SChfF450ZTCdAGeEHSHRmMzcWoLJkcdRSce27oWsU551JJp43kN5ImAncA7wA7mdmvgf7AoAzH52LUrBm88AIccQQMGxaehnfOuWTplEjaAyeY2WFm9ryZrQUws1LgZ5XtGLWpTJM0Q9JGnXBI6iHpDUmfSXpLUrdoeV9J70n6PFp3csI+j0r6WtKkaOpbrTN21ZKfD3/5Cxx2GJx9NjzySNwROecamnQSycvAwrIZSQWSBgKY2dSKdpKUDdwPHAH0BoZI6p202Z3A42a2MzASuDVavhI4w8z6AIcDd0kqTNjvcjPrG02T0jgHVwv5+fDXv8LBB8PQoeFpeOecK5NOInkAWJ4wvyJaVpUBwAwz+8rM1gBjgGOTtukNvBG9frNsvZl9aWbTo9ezgR+BDmm8p8uQ/Hx46SU46CA466zQT5dzzkF6iURRYztQXqWVzoOMXYFZCfPF0bJEn7K+neV4oEBSuw3eXBoA5AEzExbfHFV5/UFSs5RBS8MkTZA0Yd68eWmE66rSvHlIJgccEJLJnnuGaq916+KOzDkXp3QSyVdRg3tuNF0IfJXGfqmeN7Gk+cuA/SR9AuwHfE+4KywcQOoMPAGcFSUwgKuA7YHdgLbAlane3MxGmVmRmRV16OCFmbrSogX84x9w770wdy4MGgTbbx+eN1m5Mu7onHNxSCeRnAvsSfiSLwYGAsPS2K8Y2CJhvhswO3EDM5ttZieY2a7ANdGyJQCSWgP/AEaY2fsJ+8yx4CfgEUIVmqtH+fnhYcXp0+G556BNm/C8SY8eYeRFLwA617Sk80Dij2Y22Mw2N7OOZnaKmf2YxrE/AnpJ2lJSHjAYGJe4gaT2kspiuAoYHS3PA8YSGuKfT9qnc/RXwHHAlDRicRmQnQ0nnQQffABvvw177AG//S107w6//nVINM65TV86z5HkSzpP0h8ljS6bqtrPzEqA84FXganAc2b2uaSRko6JNtsfmCbpS6AjcHO0/OfAvsAvUtzm+5SkycBkwq3JN6V/ui4TJNh3Xxg3Dr74Ak49NTxzst12cMIJ8N57cUfonMskJbSjp95Aeh74H3AK4RbdU4GpZtZoOhovKiqyCRMmxB1GkzJ3Ltx3H/zxj7BoUWiYv/zy0O1Kdnbc0Tnn0iFpopkVVbVdOm0k25jZtcAKM3sMOArYqbYBuk1bp05w003w3Xdw990wezYcfzzssEPou2vVqrgjdM7VlXQSydro72JJOwKbAT0zFpHbpLRqFbqknz49dFHfunXou6tHDxg5EubPjztC51wfyEMVAAAaaElEQVRtpZNIRkXjkYwgNJZ/Adye0ajcJicnB04+GT76CN58EwYMgOuvDw3z550HM2bEHaFzrqYqTSTRHVVLzWyRmY03s62iu7e8Y3FXIxLsvz/8/e8wZQoMGQIPPQTbbgsnngjvv1/lIZxzDUyliSR6CPD8eorFNTF9+sDDD8M338Dw4fDGG+EW4n32CU/Ql5ZWeQjnXAOQTtXWa5Iuk7SFpLZlU8Yjc01G585wyy2hYf6uu2DWrDDcb+/eYRyU1avjjtA5V5l0EskvgfOA8cDEaPJ7aV2dKyiACy8M7SXPPAMtW4ZxUHr0CHeALVgQd4TOuVTSebJ9yxTTVvURnGuacnJg8GCYMAH+/W/o3x+uvTY0zF9wAXyVTk9vzrl6k84DiWekWm5mj2ckogzwBxIbvylT4He/g6eeCr0NDxoEl10W7v5yzmVGXT6QuFvCtA9wA3BMZTs4V9d23DGMzvjNN+EJ+X/9CwYOhP32g7/9zRvmnYtTOlVbFyRMvwJ2JYwP4ly969IFbrstNMj//vfw9ddwzDHhDrCHHvKGeefikE6JJNlKoFddB+JcdRQUwMUXw8yZoborPx9+9Svo2RNuvtmfmHeuPqXT++/fJI2Lpr8D04CXMh+ac1XLzYVTToGPP4bXX4e+fWHECNhii5BYpvggA85lXDqN7fslzJYA35pZcUajqmPe2N60fP453HMPPPFE6BzyoIPCbcVHHQVZNSmDO9dE1WVj+3fAB2b2tpm9AyyQ1LOW8TmXMX36hB6GZ82CW2+FadNCO8q224YEs2xZ3BE6t2lJJ5E8DyTeE7MuWuZcg9auXeh65auv4NlnYfPNQ8mka9fQvuLPozhXN9JJJDlmtqZsJnrtd225RiM3F37+c3j33TAs8NFHh0G3ttkmdMXy5ptQRQ2vc64S6SSSeQlD4yLpWMDviXGN0oAB4S6vb7+Fq6+Gd96BAw8MjfSjR/vtw87VRDqJ5FzgaknfSfoOuBI4J7NhOZdZXbqsH8HxoYdCiWTo0HC317XXhhEdnXPpSeeBxJlmtjvQG+hjZnuamQ9D5DYJzZuHBPLpp6Eb+z33DM+h9OgBp50WBuJyzlUunedIbpFUaGbLzWyZpDaSbkrn4JIOlzRN0gxJw1Os7yHpDUmfSXpLUreEdWdKmh5NZyYs7y9pcnTMeyQp3ZN1riJSqOJ66aUwLPB558G4caEqbM894bnnoKQk7iida5jSqdo6wswWl82Y2SLgyKp2kpQN3A8cQSjNDJHUO2mzO4HHzWxnYCRwa7RvW+B6YCAwALg+Gu4X4AFgGOHp+l7A4Wmcg3Np23rrMC5KcXH4++OPYZjgLbeE22+HhQvjjtC5hiWdRJItqVnZjKTmQLNKti8zAJhhZl9Fd3qNAY5N2qY38Eb0+s2E9YcBr5nZwihxvQYcLqkz0NrM3rPwJOXjwHFpxOJctbVuHW4XnjYtlFS23TbcTtytG5x7LnzxRdwROtcwpJNIngTekDRU0lDCl/pjaezXFZiVMF8cLUv0KTAoen08UCCpXSX7do1eV3ZMACQNkzRB0oR58+alEa5zqWVnhwca33gjtKWccgo8+mh48PGww+Dll733Yde0pdPYfgdwE7ADoQTxT6BHGsdO1XaRfLf+ZcB+kj4B9gO+J3TDUtG+6RyzLO5RZlZkZkUdOnRII1znqrbzzuEur1mzwl1fkyeHrld22AHuvx+WL487QufqX7o9D80lPN0+CDgImJrGPsXAFgnz3YANbqo0s9lmdoKZ7QpcEy1bUsm+xdHrCo/pXH3o0AGuuSaMj/LUU7DZZnD++aHa67LLwnLnmooKE4mkbSVdJ2kqcB+hqklmdoCZ3ZfGsT8CeknaUlIeMBgYl/Qe7SWVxXAVMDp6/SpwaHSHWBvgUOBVM5sDLJO0e3S31hl4T8QuRnl5oarrgw/Ck/OHHx4a6LfeOoziOH68PzXvNn2VlUj+Ryh9HG1me5vZvYR+ttJiZiXA+YSkMBV4zsw+lzQy4Un5/YFpkr4EOgI3R/suBG4kJKOPgJHRMoBfAw8BM4CZwCvpxuRcpkiwxx4wZkwYbOuKK0LXK/vtF8acf+wx+OmnuKN0LjMq7EZe0vGEUsSehHaRMcBDZrZl/YVXN7wbeReHlSvhySfh7rvDHV6bbw5nnQW77BL6+erVCwoL447SuYql2418OuORtCTcYjsEOJBwx9ZYM/tXXQRaHzyRuDiZhUG37r473OGV+F+uffuQUMoSS9m0zTah3cW5ONVZIkk6aFvgJOBkMzuwFvHVK08krqFYtSoMDzx9ephmzFj/+vvvN9y2Q4cNE0vi69at44nfNS0ZSSSNlScS1xisXFlxkknuRHLzzStOMgUF8cTvNj3pJpKc+gjGOVe1Fi1gp53ClGzFitRJ5tVXw8ORiTp2rDjJtGpVL6fimhhPJM41Ai1bhochd95543XLl6dOMq+8AnPnbrhtp04bt8WU/W3Zsn7OxW16PJE418i1ahXuBNtll43XLVuWOsn84x/www8bbtu5c8VJpkWL+jkX1zh5InFuE1ZQEEZ/7Nt343VLl26YZMoSzd/+Fno8TtS168bVZL16hQcvPck4TyTONVGtW8Ouu4Yp2ZIlqZPMSy9Bch+o3bpVnGSaN6+fc3Hx8kTinNvIZptBv35hSrZkyYZ3lJUlmbFjYf78DbfdYovUz8hsvTXk59fPubjM80TinKuWzTYL3b7077/xusWLN751ecYMePFFWLBg/XZSSDKp7izbaitPMo2NJxLnXJ0pLITddgtTskWLUieZ55/fcNRJCbp3T/3E/5ZbQrN0htVz9coTiXOuXrRpAwMGhCnZwoUbJ5np00MnmIsXr98uK6vyJJOXV3/n49bzROKci13btjBwYJiSLViQOsk8/XRorymTlQU9eqROMj17epLJJE8kzrkGrV27MO2++4bLzdYnmeRE8/774fbmMtnZ65NMcqLp2RNyc+v1lDY5nkicc42SFHpPbt8+jAWTyCzcQZYqybz7bnhQs0x2dkgmFSWZHP+WrJJfIufcJkcKvSd36AB77rnhOrPwLEyqJPPf/4YuZ8rk5GyYZBITTY8enmTK+GVwzjUpUug9efPNYa+9NlxnFp7qT5Vkxo8PnWeWyckJDfypkkz37k0ryTShU3XOucpJoffkjh1h7703XGcW+idLlWTefnvDJJObW3mSyc6u3/PKNE8kzjmXBin0ntypE+yzz4brzEJPy6mSzJtvhrFmyuTlhYcuUz3xv8UWjTPJeCJxzrlakkLvyZ07w777brjODObM2bhLmenT4Y03wqiZZfLyQvcxFSWZrKz6Pa90ZTSRSDocuBvIBh4ys9uS1ncnjAFfGG0z3MxelnQqcHnCpjsD/cxskqS3gM5A2eU/1MyS+ip1zrmGQYIuXcK0334bristrTjJvPYarF69fttmzSpOMt26xZtkMjbUrqRs4EvgEKAY+AgYYmZfJGwzCvjEzB6Q1Bt42cx6Jh1nJ+AlM9sqmn8LuMzM0h4714fadc41NqWlYYjlVElmxgz46af12+bnhySTqu+yrl1rnmQawlC7A4AZZvZVFNAY4Fjgi4RtDGgdvd4MSBqZGoAhwDMZjNM55xqcrKxQ0ujWDQ44YMN1paVQXLxxv2VffhlGxkxMMpMmpR70rC5lMpF0BWYlzBcDyR0g3AD8S9IFQEvg4BTHOZmQgBI9Imkd8CJwk6UoVkkaBgwD6N69e03id865Bqmsz7Hu3eHAAzdct27dhkmmV696iCeDx1aKZclf+EOAR82sG3Ak8ISk8pgkDQRWmtmUhH1ONbOdgH2i6fRUb25mo8ysyMyKOnToUJvzcM65RqOsO5iDDoJzz62fESwzmUiKgS0S5ruxcdXVUOA5ADN7D8gH2iesH0xStZaZfR/9XQY8TahCc845F5NMJpKPgF6StpSUR0gK45K2+Q44CEDSDoREMi+azwJOAsaUbSwpR1L76HUu8DNgCs4552KTsTYSMyuRdD7wKuHW3tFm9rmkkcAEMxsHXAr8WdLFhGqvXyS0d+wLFJc11keaAa9GSSQbeB34c6bOwTnnXNUydvtvQ+K3/zrnXPWle/tvA31O0jnnXGPhicQ551yteCJxzjlXK55InHPO1YonEuecc7XiicQ551yteCJxzjlXK55InHPO1YonEuecc7XiicQ551yteCJxzjlXK55InHPO1YonEuecc7XiicQ551yteCJxzjlXK55InHPO1YonEuecc7XiicQ551yteCJxzjlXK55InHPO1UpGE4mkwyVNkzRD0vAU67tLelPSJ5I+k3RktLynpFWSJkXTgwn79Jc0OTrmPZKUyXNwzjlXuYwlEknZwP3AEUBvYIik3kmbjQCeM7NdgcHAHxPWzTSzvtF0bsLyB4BhQK9oOjxT5+Ccc65qmSyRDABmmNlXZrYGGAMcm7SNAa2j15sBsys7oKTOQGsze8/MDHgcOK5uw3bOOVcdmUwkXYFZCfPF0bJENwCnSSoGXgYuSFi3ZVTl9bakfRKOWVzFMQGQNEzSBEkT5s2bV4vTcM45V5lMJpJUbReWND8EeNTMugFHAk9IygLmAN2jKq9LgKcltU7zmGGh2SgzKzKzog4dOtT4JJxzzlUuJ4PHLga2SJjvxsZVV0OJ2jjM7D1J+UB7M/sR+ClaPlHSTGDb6Jjdqjimc865epTJEslHQC9JW0rKIzSmj0va5jvgIABJOwD5wDxJHaLGeiRtRWhU/8rM5gDLJO0e3a11BvBSBs/BOedcFTJWIjGzEknnA68C2cBoM/tc0khggpmNAy4F/izpYkIV1S/MzCTtC4yUVAKsA841s4XRoX8NPAo0B16JJuecczFRuPlp01ZUVGQTJkyIOwznnGtUJE00s6KqtvMn251zztWKJxLnnHO14onEOedcrXgicc45VyueSJxzztWKJxLnnHO1kskn251rUMyMktIS1pauZe26tWn/XbNuTbX3WVsa7Ze0TBKFzQpp27xt+dSmeZv1r/Pb0KZ5G3Ky/L+mazz80+piU1Jawqq1q1i5diUr1q5g5dqVlU4r1iRtU5J6/aqSVRV+udcHIfKy88jNziU3K3eDv2bG4tWLWfLTkkqP0bpZ6/LEskHSSZhPTEBl61rktsCH6HH1zROJq5Y169bw+Y+fM33h9A2+2KtKBKnWr1m3ptrvn5uVS4vcFrTIbUHLvJblr1vktqCwdSEtcluQn5NPXlbqL/LcrNwKv+Qr+puXnZf2trlZuWRnZVd5HiWlJSxZvYSFqxaWT4tWL1r/etUiFq5ev27Kj1PKX1eWEPOy81InneSElFQKKswvTCtu51LxROIq9FPJT0z5cQoT50xk4uyJfDz3Yz774bMKE0B+Tn74gs/d8Au+VV4rNm+5+foEkLQ+1ZScJFrktqB5TnNys3Pr+SpkRk5WDu1atKNdi3bV2s/MWLF2RUg0KRJQchKatXQWn/3wGQtXLWTZmmWVHrswv7BGpaDmuc1rcyncJsATiQNgdclqPvvhs5Aw5nzMxDkTmfLjlPJfv23y29Cvcz8uGngR/Tr3o3eH3rRu1nr9l3xuc7Lk925kmiRa5bWiVV4rtthsi6p3SLB23VoWrV5UeRJKmP92ybfl266zdRUeNz8nv9qloLbN29K6WWv/zGwiPJE0QSvXruTTuZ+WJ4yJcyby+Y+fl39ZtGvejv5d+nPpHpfSv0t/+nfuT8/Cnl733sjlZueyecvN2bzl5tXaz8xYtmbZBgmosiT09eKv+XjOxyxctZAVa1dUeNwsZaVVCkpVFdcsp1ltL4erQ55INnHL1yxn0txJ65PG7IlMnT+VUisFoEOLDvTv0p+jtz2a/p37069zP7pv1t2ThisnidbNWtO6WWt6FPao1r4/lfy0USmosiQ0c9FMFq1axKLVi8o/o6m0yG1Ro1JQQV6Bf7YzwBPJJmTpT0uZNHcSE2eHUsbHcz7mf/P/h0WDSHZq1Yn+nfszaIdB9Ovcj/5d+tO1oKv/x3IZ0yynGZ1adaJTq07V2q/USln609L1bT4V3ZQQvZ6+cHr5tqtKVlV43GxllyeX6paCNpX2uUzwRFKJbxZ/Q05WDh1bdmxwH6Ilq5eUlzLK/n654Mvy9V0LutK/S39O7nNyefVU54LOMUbsXPrKqr0K8wuhTfX2XbV2VXmCqao9aN7KeUxbMI2FqxayZPWS8h9dqbTKa1Vx0qnkhoSWuS03+R9rnkgqcf7L5/OP6f9AiPYt2tOloAudCzrTuVXn8LpVZzoXrH/dqVWnjNTdLly1kI/nfLxB9dTMRTPL13ffrDv9Ovfj9J1PL6+e6tiqY53H4Vxj0Dy3Oc1zm9OloEu19ltXuo4lP62/JbuqJDR1/tTy+cpuZc/Jyqk46VRSCirML2w0D6b6wFaV+M+3/2Hq/KnMWTaH2ctmM2f5HOYsD69/WP5DyjtZ2jVvV55sOrXqRLPsmieWhasX8smcT/h68dfly3oW9qR/5/7lCaNf5350aNmhxu/hnKsdM2NVyaqUCaiqO+OW/rS00mOXPZha3VJQ85zmdVIKSndgK08kNbSudB3zV85fn2BSJJu5y+dSUlpS4/domduSXTvvSr9OoT2jX+d+tG3etg7PwjkXp5LSEhavXlyjJFTZd0vig6kvDX6JbdpuU6P40k0kjaPc1ABlZ2XTsVVHOrbqyK7sGnc4zrlGKCcrh/Yt2tO+Rftq7Vf2YOpGD6KmSDoFeQUZin49TyTOOdfIJD6Y2n2z7nGHk9lu5CUdLmmapBmShqdY313Sm5I+kfSZpCOj5YdImihpcvT3wIR93oqOOSmaqvd0lXPOuTqVsRKJpGzgfuAQoBj4SNI4M/siYbMRwHNm9oCk3sDLQE9gPnC0mc2WtCPwKtA1Yb9TzaxuGz2cc87VSCZLJAOAGWb2lZmtAcYAxyZtY0Dr6PVmwGwAM/vEzGZHyz8H8iV5nwjOOdcAZTKRdAVmJcwXs2GpAuAG4DRJxYTSyAUpjjMI+MTMfkpY9khUrXWtKrjHTdIwSRMkTZg3b16NT8I551zlMplIUn3BJ99rPAR41My6AUcCT0jruwOV1Ae4HTgnYZ9TzWwnYJ9oOj3Vm5vZKDMrMrOiDh38OQvnnMuUTCaSYiCxn+tuRFVXCYYCzwGY2XtAPtAeQFI3YCxwhpmVP8ZtZt9Hf5cBTxOq0JxzzsUkk4nkI6CXpC0l5QGDgXFJ23wHHAQgaQdCIpknqRD4B3CVmb1TtrGkHElliSYX+BkwJYPn4JxzrgoZSyRmVgKcT7jjairh7qzPJY2UdEy02aXAryR9CjwD/MLCo/bnA9sA1ybd5tsMeFXSZ8Ak4Hvgz5k6B+ecc1VrEl2kSJoHfFvB6vaE240bIo+tZjy2mvHYamZTjq2HmVXZyNwkEkllJE1Ipy+ZOHhsNeOx1YzHVjMeW4afbHfOObfp80TinHOuVjyRwKi4A6iEx1YzHlvNeGw10+Rja/JtJM4552rHSyTOOedqxROJc865WmnSiaSq8VLiJOmbaDyWSZJi7TJf0mhJP0qakrCsraTXJE2P/rZpQLHdIOn7hIdZj4wpti2i8XamSvpc0oXR8tivXSWxxX7tJOVL+lDSp1Fsv42Wbynpg+i6PRv1mNFQYntU0tcJ161vfceWEGN2NMbT36P5zF83M2uSE5ANzAS2AvKAT4HecceVEN83QPu444hi2RfoB0xJWHYHMDx6PRy4vQHFdgNwWQO4bp2BftHrAuBLoHdDuHaVxBb7tSN0+Noqep0LfADsTuiXb3C0/EHg1w0otkeBE+P+zEVxXULoh/Dv0XzGr1tTLpGkM16KA8xsPLAwafGxwGPR68eA4+o1qEgFsTUIZjbHzD6OXi8jdBXUlQZw7SqJLXYWLI9mc6PJgAOBF6LlcV23imJrEKLObo8CHormRT1ct6acSNIZLyVOBvwrGmp4WNzBpNDRzOZA+FICGtqQx+crDN88Oq5qt0SSegK7En7BNqhrlxQbNIBrF1XPTAJ+BF4j1B4sttCHH8T4/zU5NjMru243R9ftD4pvIL67gCuA0mi+HfVw3ZpyIklnvJQ47WVm/YAjgPMk7Rt3QI3IA8DWQF9gDvC7OIOR1Ap4EbjIzJbGGUuyFLE1iGtnZuvMrC9h+IkBwA6pNqvfqKI3TYpNYTjwq4Dtgd2AtsCV9R2XpJ8BP5rZxMTFKTat8+vWlBNJOuOlxMaioYbN7EfCuCwNbdyVHyR1Boj+/hhzPOXM7IfoP3spoXfo2K5dNNzBi8BTZvaXaHGDuHapYmtI1y6KZzHwFqEdolBSTrQq9v+vCbEdHlUVmoWRXB8hnuu2F3CMpG8IVfUHEkooGb9uTTmRpDNeSiwktZRUUPYaOJSGN+7KOODM6PWZwEsxxrKBsi/pyPHEdO2i+umHgalm9vuEVbFfu4piawjXTlIHhTGJkNQcOJjQhvMmcGK0WVzXLVVs/0v4YSBCG0S9Xzczu8rMuplZT8L32b/N7FTq47rFfYdBnBNheN8vCfWv18QdT0JcWxHuIvsU+Dzu2AhjxcwB1hJKckMJda9vANOjv20bUGxPAJOBzwhf2p1jim1vQjVC2fg5k6LPXOzXrpLYYr92wM7AJ1EMU4DrouVbAR8CM4DngWYNKLZ/R9dtCvAk0Z1dcU3A/qy/ayvj1827SHHOOVcrTblqyznnXB3wROKcc65WPJE455yrFU8kzjnnasUTiXPOuVrxROJcDCQtT3h9ZNQza/c4Y3KupnKq3sQ5lymSDgLuBQ41s+/ijse5mvBE4lxMJO1D6IbkSDObGXc8ztWUP5DoXAwkrQWWAfub2Wdxx+NcbXgbiXPxWAu8S+jSxblGzROJc/EoBX4O7Cbp6riDca42vI3EuZiY2cpoDIn/SPrBzB6OOybnasITiXMxMrOFkg4Hxkuab2YNpjt+59Llje3OOedqxdtInHPO1YonEuecc7XiicQ551yteCJxzjlXK55InHPO1YonEuecc7XiicQ551yt/H/M7Ym6i8QeWwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tracc = [item[0] for item in results]\n",
    "tstacc = [item[1] for item in results]\n",
    "\n",
    "trainAcc = plt.plot(K, tracc, '-b', label='Training Error')\n",
    "testACC = plt.plot(K, tstacc, '-g', label='Testing Error')\n",
    "plt.title(\"DT Accuracy versus K Model Parameter\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.xlabel(\"K\")\n",
    "plt.legend(loc=1)\n",
    "plt.show(block=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Contingency Tables\n",
    "\n",
    "#### Let's look at the contingency tables\n",
    "\n",
    "#### 1) Can you calculate the Sensitivity, Specificity, PPV, NPV? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter Best Model: 5\n",
      "\n",
      "Decision Tree\n",
      "       Other  PNA  Total\n",
      "Other    890  132   1022\n",
      "PNA      110  368    478\n",
      "Total   1000  500   1500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model = input(\"Enter Best Model: \")\n",
    "best = cont[int(model)]\n",
    "dt_ct = pd.crosstab(best[1], best[0], margins=True)\n",
    "dt_ct.columns = [\"Other\", \"PNA\", \"Total\"]\n",
    "dt_ct.index = [\"Other\", \"PNA\", \"Total\"]\n",
    "print()\n",
    "print(\"Decision Tree\")\n",
    "print(dt_ct)\n",
    "print()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation Statistics\n",
    "\n",
    "#### We will calculate some test statistics for our classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree: Sensitivity: 0.73600 Specificity: 0.89000 PPV: 0.76987 NPV: 0.87084 Accuracy: 0.83867\n"
     ]
    }
   ],
   "source": [
    "Sens = dt_ct.iloc[1][1]/dt_ct.iloc[2][1]\n",
    "Spec = dt_ct.iloc[0][0]/dt_ct.iloc[2][0]\n",
    "PPV = dt_ct.iloc[1][1]/dt_ct.iloc[1][2]\n",
    "NPV = dt_ct.iloc[0][0]/dt_ct.iloc[0][2]\n",
    "ACC = (dt_ct.iloc[0][0] + dt_ct.iloc[1][1]) / dt_ct.iloc[2][2]\n",
    "print(\"Decision Tree: Sensitivity: %.5f Specificity: %.5f PPV: %.5f NPV: %.5f Accuracy: %.5f\" % (Sens, Spec, PPV, NPV, ACC))                                                                                      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### ROC Curve\n",
    "\n",
    "#### 1) So which classifier do you think is better?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzs3Xd4VGX2wPHvIaFDQi8JJVQhiaIQQUQExAVhsYsiLjYU0LXr/uwIrLqAIApYQEURuywouqzY110bJoAISImYQCjSUoGElPP7496EEFImkMlkJufzPHlyZ+bOnXNT5sx933vPEVXFGGOMAajh6wCMMcZUHZYUjDHGFLCkYIwxpoAlBWOMMQUsKRhjjClgScEYY0wBSwrGGGMKWFIwAUdEEkTksIhkiMhuEXlNRBoUWedsEflSRNJFJFVEPhKRyCLrhIjIMyKyzd1WvHu7WeXukTGVx5KCCVQXqmoD4HTgDODB/AdEpC/wKfAhEAZ0AH4GvhWRju46tYAvgCjgAiAEOBvYD/T2VtAiEuytbRvjCUsKJqCp6m5gBU5yyDcdeF1Vn1XVdFU9oKqPAD8Ak9x1rgXaAZeq6gZVzVPVPar6d1VdXtxriUiUiHwmIgdE5A8Reci9/zURebzQegNFJKnQ7QQRuV9E1gIHReQREVlcZNvPishsdzlURF4RkV0iskNEHheRoJP8URkDWFIwAU5E2gDDgHj3dj2cT/zvF7P6e8Cf3OXzgU9UNcPD12kIfA58gnP00RnnSMNTVwN/BhoBi4DhIhLibjsIuBJ4y113IZDjvsYZwBDgpnK8ljElsqRgAtUHIpIObAf2AI+59zfB+bvfVcxzdgH58wVNS1inJCOA3ao6U1Uz3SOQH8vx/Nmqul1VD6tqIrAKuMR97DzgkKr+ICItcZLcXap6UFX3ALOAUeV4LWNKZEnBBKpLVLUhMBDoxtE3+2QgD2hdzHNaA/vc5f0lrFOStsBvJxSpY3uR22/hHD0AjOboUUJ7oCawS0RSRCQFmAe0OInXNqaAJQUT0FT1P8BrwAz39kHge2BkMatfydEhn8+BoSJS38OX2g50KuGxg0C9QrdbFRdqkdvvAwPd4a9LOZoUtgNZQDNVbeR+hahqlIdxGlMqSwqmOngG+JOI5E82PwBcJyJ3iEhDEWnsTgT3BSa76yzCeQP+p4h0E5EaItJURB4SkeHFvMbHQCsRuUtEarvb7eM+tgZnjqCJiLQC7iorYFXdC3wNvAr8rqq/uvfvwjlzaqZ7ymwNEekkIgNO4OdizHEsKZiA577Bvg486t7+HzAUuAxn3iARZ8L2HFXd4q6ThTPZvBH4DEgDVuIMQx03V6Cq6TiT1BcCu4EtwCD34UU4p7wm4Lyhv+th6G+5MbxV5P5rgVrABpzhsMWUb6jLmBKJNdkxxhiTz44UjDHGFLCkYIwxpoAlBWOMMQUsKRhjjCngd8W3mjVrphEREb4Owxhj/EpcXNw+VW1e1np+lxQiIiKIjY31dRjGGONXRCTRk/Vs+MgYY0wBSwrGGGMKWFIwxhhTwO/mFIqTnZ1NUlISmZmZvg7FAHXq1KFNmzbUrFnT16EYY8opIJJCUlISDRs2JCIiAhHxdTjVmqqyf/9+kpKS6NChg6/DMcaUk9eGj0RkgYjsEZF1JTwuIjLbbYa+VkR6nuhrZWZm0rRpU0sIVYCI0LRpUztqM8ZPeXNO4TWchuclGQZ0cb/GAS+czItZQqg67HdhjP/yWlJQ1W+AA6WscjFO83RV1R+ARiJi5X+NMcalquxOzeTj1YlMXrySdTtSvf6avpxTCOfYFoRJ7n3H9cUVkXE4RxO0a9euUoIrr6CgIE499VSys7MJDg7muuuu46677qJGjfLn3YkTJ3Luuedy/vnnF/v4iy++SL169bj22mvLtd0VK1Zw//33AxAfH094eDh169bltNNO4/XXXy93nMaYipOXpyQeOMT6nams35nGuh2pbNiZxv6DR5wVNI8OYc2JDg/1ahy+TArFjTEU29xBVecD8wFiYmKqZAOIunXrsmbNGgD27NnD6NGjSU1NZfLkyWU883hTpkwp9fEJEyacUIxDhw5l6NChAAwcOJAZM2YQExNz3Ho5OTkEBwfEOQjGVEnZuXnE78k45s1/w640MrJyAKgZJHRsWo/a+7dw4L//pnnNLJ5/4iGGnu39kzd8+Z+fhNPsPF8bYKePYqlQLVq0YP78+Zx55plMmjSJvLw8HnjgAb7++muysrL461//yvjx4wGYPn06ixYtokaNGgwbNoypU6dy/fXXM2LECK644goeeOABli1bRnBwMEOGDGHGjBlMmjSJBg0acN9997FmzRomTJjAoUOH6NSpEwsWLKBx48YMHDiQPn368NVXX5GSksIrr7xC//79S4z55Zdf5vPPPycjI4OsrCw+++wzpk6dypIlS8jMzOSKK65g4sSJACxcuJDnnnuOI0eOcPbZZzN37twTOiIypjo4fCSXX3ensX5nGhvco4CNu9M5kpMHQN2aQXRv3ZDLeoYTFRZCVFgoHZvVI+aMHmzatIn77ruPSZMmUbdu3UqJ15dJYRlwm4i8A/QBUt3+sydl8kfr2bAz7aSDKywyLITHLixfX/SOHTuSl5fHnj17+PDDDwkNDeWnn34iKyuLfv36MWTIEDZu3MgHH3zAjz/+SL169Thw4NgpmAMHDrB06VI2btyIiJCSknLc61x77bXMmTOHAQMGMHHiRCZPnswzzzwDOJ/4V65cyfLly5k8eTKff/55qTF///33rFmzhsaNG7N8+XK2bdvGjz/+iKoyfPhwvvvuO0JCQli6dCnfffcdwcHBjBs3jnfeeYfRo0eX6+djTCBKPZTN+l3OJ/91O5wE8NveDPLc8Y3QujWJDg/h+rMjChJAh2b1CarhDJzs37+fJk1CEBGeeOIJ2rZtW+zRvDd5LSmIyNvAQKCZiCQBjwE1AVT1RWA5MByIBw4BN3grFl/Jb3X66aefsnbtWhYvXgxAamoqW7Zs4fPPP+eGG26gXr16ADRp0uSY54eEhFCnTh1uuukm/vznPzNixIhjHk9NTSUlJYUBA5ye7ddddx0jR44sePyyyy4DoFevXiQkJJQZ75AhQ2jcuHFBzP/+978544wzAMjIyGDz5s2kpKTw008/FfyhHj58mLZt25a4TWMC1Z60zILhn/U701i/K5XtBw4XPN4qpA5RYSEMO7W1mwBCCG9Ut9iz81SVN998kzvvvJOpU6dy8803c+mll1bm7hTwWlJQ1avLeFyBv1b065b3E723bN26laCgIFq0aIGqMmfOnILx/HyffPJJqadvBgcHs3LlSr744gveeecd5s6dy5dffulxDLVr1wacSfCcnJwy169fv37BsqryyCOPMHbs2GPWmTVrFjfeeCN///vfPY7DGH+mqmw7cMh54y+YBE5jX0ZWwToRTetxWptGXN27HVFhoUSFhdCsQW2Ptr99+3YmTJjA8uXLOeuss+jXr5+3dsUjNpvoBXv37mXChAncdtttiAhDhw7lhRde4LzzzqNmzZps3ryZ8PBwhgwZwpQpUxg9enTB8FHho4WMjAwOHTrE8OHDOeuss+jcufMxrxMaGkrjxo3573//S//+/Vm0aFHBUcPJGjp0KI8//jijRo2ifv36JCUlUadOHc4//3yuuOIK7rzzTpo1a8b+/fs5ePBglT0rzJjyyMnN47e9B1m/M5V1O5wksGFXGumZzoeqoBpClxYNGNC1OVFhIUSHh9K9dUMa1jmxki5vv/0248ePJzc3l2eeeYbbbruNoKCgitylcrOkUEEOHz7M6aefXnBK6pgxY7jnnnsAuOmmm0hISKBnz56oKs2bN+eDDz7gggsuYM2aNcTExFCrVi2GDx/Ok08+WbDN9PR0Lr74YjIzM1FVZs2addzrLly4sGCiuWPHjrz66qsVsj/Dhw9n48aNnHXWWQA0bNiQt956i1NPPZXHHnuM888/n7y8PGrWrMmLL75oScH4nczsXDbtTmed++l//c40Nu5KI8udAK5TswbdWoVwUY8wosOdT/9dWzakTs2Ke9Nu3Lgxffr0Yf78+VWmLIzkj3v7i5iYGC3aZOfXX3+le/fuPorIFMd+J6YqScvMZoP7xr9+Zyrrd6QRvzeDXHcGuGGdYOeTf1goUeH5ZwDVJzioYs+qy8nJYdasWRw5coSHH34YcIanKqMKgIjEqWqZs9Z2pGCMCSh707MKxv7zvyfuP1TweIuGtYkKC+FPkS2JdhNAm8bFTwBXpJ9//pmxY8cSFxfHlVdeWZAMqlpZGEsKxhi/pKokJR8ulACcM4H2pB+dAG7XpB5RYSFcGdOWSPcMoBYN61RqnFlZWTz++ONMnTqVJk2a8P7773P55ZdXuWSQL2CSQmUdgpmy+duQpKn6cvOUrXszCj79r9vhXAGcejgbgBoCnVs04JzOzdw3/1Aiw0IIrev7nh5btmxh2rRpjB49mqeffpqmTZv6OqRSBURSqFOnDvv377fy2VVAfj+FOnUq99OYCRxZObls3p3hvPnnXwG8K53D2bkA1AquQfdWDRl+auuC4Z9urSp2AvhkZWRk8OGHH3LNNdcQHR3Nxo0b6dixo6/D8khAJIU2bdqQlJTE3r17fR2K4WjnNWPKkpGV404AHx0C2vJHOjn5E8C1g+keFuKe/x9CVHgInZo3oGYFTwBXpM8++4xx48aRmJhIz5496d69u98kBAiQpFCzZs0qczqXMaZ4+zOyjo7973RKQSTsP0j+aGOzBrWICgtl0CnNiQoLJTo8hLaN61Gjhn8c/ScnJ3PfffexYMECunbtyn/+8x+/PAMvIJKCMabqUFV2pmayfkcq6woVgduVerQbX5vGdYkKC+HSM8ILhoBaNKztt8O/ubm59OvXj82bN/Pggw8yceJEvx1CtaRgjDlhuXnK7/ucK4A3FDoKSDl0dAK4Y/MG9O7QxLkGICyEyLAQGtWr5ePIK8a+ffto0qQJQUFBPPnkk7Rr146ePU+4s3CVYEnBGOORIzl5bP4j3akA6n76/3VXGoeOuBPAQTU4pVVDLohq5Y7/OxPA9WoF3tuMqrJo0SLuuusupk6dyrhx47jkkkt8HVaFCLzfljHmpB3MymHj7rSC+j/rd6ax+Y90snOdCYD6tYKIdM//zy8B3blFA2oFV90J4IqSmJjI+PHjWbFiBWeffTbnnnuur0OqUJYUjKnmkg8eOXr+v/v9931HJ4Cb1K9FVFgIY8/pWFACOqJpfb+ZAK5Ib7zxBrfccktB5eNbb7014BpMWVIwpppQVXanZbJ+x9Hhnw0709iRcrQHQFhoHaLCQ7moR1jBGUCtQur47QRwRWvevDn9+vVj3rx5tG/f3tfheEVAFMQzxhwrvwl8QQOYncc2gReBDs3qF9T+j3avAG5SPzAmgCtKdnY2M2fOJDs7m0cffRTw3+oJVhDPmGoiOzePLX9kHFME7tdd6cc0ge/SoiGDu7coSALdW4dQv7b9+5dm9erVjB07ltWrVzNq1KgqW8CuotlfhTF+pHAT+PXuUcCm3ekcyT3aBD4yLOSYJvBdWjagdnDVKQFR1WVmZjJlyhSmT59Os2bN+Oc//1nQ2rY6sKRgTBWVeij7mE//63amsbVQE/hG9WoSFRbC9f2KbwJvTkx8fDwzZszg2muvZebMmQV9y6uLUpOCiJwJ/AXoD7QGDgPrgH8Bb6lqutcjNCbAqSp78nsAFJoETko+tgl8dHgIw90m8NHhoYSF2gRwRcnIyGDp0qWMGTOG6OhoNm3aVG1L55SYFETkY2A/8CEwE9gD1AG6AoOAf4nIdFX9uDICNSYQFG4Cf3QS+Ngm8B2a1adH20aM7tOu4Crgph42gTflt2LFCsaNG8f27duJiYmhe/fu1TYhQOlHCmNV9Y8i92UCK92vaSLSwmuRGePncnLziN+bwfodR8s//LozjXR3Aji4htDZbQKfX//nZJrAm/LZv38/99xzD6+//jrdunXjv//9r18WsKtoJSaF/IQgIhOAt1U1tZh19ngxNmP8RmZ2Lht3px9tALMzlY27049rAn/xGWEFZwBVdBN447n8Anbx8fE8/PDDPPLII35bwK6ieTLRHAGsEpEfgQWq+rl3QzKmastvAr9ux9EicIWbwIfUCSYqLJQxZ7UnKty5BqCDF5rAm/Lbu3cvTZs2JSgoiGnTptG+fXtOP/10X4dVpXh08ZqI1ACGATcAPYC3cRJEglejK4ZdvGYq0570zIIrf/PnALYdOL4JfHR4aMEZQJXRBN6Uj6ry2muvcc899zB16lTGjx/v65AqXYVevKaqeSKSACQAp+KcifShiCxX1QdPJlBjqoLCTeALF4Er2gQ+OjyEq870XRN4U34JCQmMGzeOzz77jP79+zNo0CBfh1SllZkURORW4HogDXgFeFhVs9yjh3jAkoLxK/lN4Ne5p4DmXweQlulMAAfVEDo3P9oEPjrcKQERYhPAfmfRokXccsstiAjPP/8848ePD7gCdhXNkyOFNsAoVd1a+E736OEi74RlTMXIzM5l8x/pRy8A25HGxt1pZGY7E8D5TeBH9AgrGP6pak3gzYlr2bIl5557Li+++CLt2rXzdTh+wZOkEFY0IYjIa6p6vaqu81JcxpRbemY2v+5KP2YIKH5PxjFN4CPDQhjdu33BPECn5jYBHEiys7OZPn06ubm5TJw4kSFDhjBkyBBfh+VXPEkKpxW+4Q4bnemdcIzxzL6CJvCpBXWAEvYfnQDObwJ/XrcWBZPA/tQE3pTfqlWruPHGG/n5558ZPXq031Yz9bXSrmi+H3gAaCgiB/LvBhRnbsEYr1NVdqQcLrjyN78I3O6045vAX96zDVEB0ATelM/hw4eZPHkyM2bMoHnz5ixdujRgWmP6QomnpIrzHxUE/AMnOQCgqrkeb1zkAuBZdzsvq+rUIo+3AxYCjdx1HlDV5aVt005JDVyFm8AXPgoo3AS+U/MGBWP/gdYE3pyY9evXc8YZZ3Dttdfy1FNPVbsCdp6qiFNSO6vqFhFZBEQV2jAAqrq2jACCgOeAPwFJwE8iskxVNxRa7RHgPVV9QUQigeU4F8uZAJeVk3tMD4B1O5weAIezi2kC7w7/dG8VQt1aNgFsIC0tjSVLlnD99dcTFRXFli1bArYTWmUrLSk8AIzFeWMvSoGyulX3BuLzJ6lF5B3gYqBwUlAgxF0OBXZ6ELPxMwezcvh1V9oxReC27Dm+CfxVZ7Y9pgdATZsANsVYvnw5EyZMYMeOHfTp04fu3btbQqhApdU+Gut+73+C2w4Hthe6nQT0KbLOJOBTEbkdqA+cX9yGRGQcMA6w08qquPwm8OsKDQEV1wT+3K4dC84Aat/EJoBN2fbt28fdd9/NG2+8QWRkJN9++60VsPMCTy5eW4VT1uI9VU0sx7aL+y8vOoFxNfCaqs4Ukb7AIhGJVtW8Y56kOh+YD86cQjliMF6iquxKzTzm/P8NO1PZmXp0Aji8UV0iw0K4uIfbBcyawJsTlF/AbuvWrUycOJGHHnqI2rWtnLg3eHJK6kjgKmCZiBwC3gXeV9UdZTwvCWhb6HYbjh8eGgtcAKCq34tIHaAZTu8GU0Xk5SkJ+w8WHAHkF4E7UKQJfExEk2MmgRtbE3hzkv744w+aN29OUFAQM2bMoH379px22mllP9GcMI8K4hWsLNIdeAi4WlXL6toWDGwGBgM7gJ+A0aq6vtA6/wbeVdXX3G1/AYRrKUHZ2Ufeld8E/uibv/P94BFnArhmkNC1ZcOCN//o8BC6tbIm8KZiqSoLFizg3nvvZerUqUyYMMHXIfm9Ci2IJyJtgCtxjhiCgYfLeo6q5ojIbcAKnNNNF6jqehGZAsSq6jLgXuAlEbkbZ2jp+tISgqlYh47k8OuudDbknwG0M5XNuzMKmsDXqxVE99YhXN6rDdFhTv2fri0bUivYJoCN92zdupWbb76ZL7/8kgEDBnD++cVONRovKfNIQUS+BRoC7+N8qt9cGYGVxI4UTkzhJvD5k8BFm8BHFzr335rAG19YuHAht956K0FBQTz11FPcfPPNVsCuglTkkcJ4q3HkP/KbwB/t/+tMAu9IOdoEvnVoHaLCnCbw0WEhRFkTeFNFhIWFcd555/HCCy/Qpk0bX4dTLZV2RfPVqvq2iNxR3OOqOturkZXAjhSOyss72gR+/c5U1u10zgDal3GkYJ0Ozeo75Z/D8pvAWBN4U3UcOXKEqVOnkpeXx6RJk3wdTkCriCOF/GvFmxfzmI37V7LCTeDzh3+KawI/8JQWBZPA1gTeVGU//fQTN954I+vWrWPMmDFWwK6KKO3itefdxX+p6g+FHxORs7waVTWXmZ1bcAVw/lHAxt3pHCnUBL5766NN4KPdK4CtB4DxB4cOHWLixInMmjWL1q1bs2zZMi688EJfh2VcnswpPA/0LHLfc0Cvig+n+kk9nF1w6md+Avht78HjmsBf17d9wfn/HZs3sAlg47d+//135syZw80338y0adMIDQ31dUimkNJKZ/cG+gLNi8wrhAA2JnGC1mxP4X9b9hYcBRRtAh8dHsrQqFbWBN4ElNTUVJYsWcINN9xAVFQU8fHxtG3btuwnmkpX2pFCfZyri4M5dl4hHecqZ1NOh4/kcuW87zmSk0f7pkebwOcngOYNbQLYBJ5//etfjB8/nl27dtG3b1+6detmCaEKK21O4SvgKxF5tWg7TnNifk5K4UhOHvPG9GJoVCtfh2OMV+3du5e77rqLt956i+joaJYsWUK3bt18HZYpQ2nDRzNV9V5gpogcd7aRql7m1cgCUFxiMgB9OjTxcSTGeFdubi7nnHMOv//+O5MnT+aBBx6gVi2rheUPShs+etf9PrcyAqkOYhMO0LlFA+sUZgLW7t27adGiBUFBQcycOZOIiAiio6N9HZYphxKvH1fVle73L/K/gFhgl7tsyiEvT4lLTCamvbUKNIEnLy+PefPm0bVrV+bNmwfAiBEjLCH4oTKLiojIFyISIiKNgV+At0TkKe+HFlji92aQlplDL0sKJsDEx8czePBgJkyYwJlnnsnQoUN9HZI5CZ5UmmqiqmnAZcBCVT0dsN96OeXPJ8RE2HyCCRyvvvoqp556KqtWreKll17i888/p2PHjr4Oy5wET5JCsIg0xzkN9SMvxxOwYhOSaVq/FhFN6/k6FGMqTLt27Rg6dCgbNmzgpptusmtqAoAnVzQ/AfwH+J+qrhSRjsDv3g0r8MQlHqBX+8b2T2P8WlZWFv/4xz/Iy8tjypQpDB48mMGDB/s6LFOByjxSUNV3VDVSVce5t7eq6sXeDy1w7E3PImH/IZtPMH7txx9/pFevXkyePJlt27Zh/bACU5lHCiLSDLgRiCi8fn6SMGU7Op9gScH4n4MHD/Loo4/yzDPPEB4ezscff8yf//xnX4dlvMST4aMPgR+A/wG53g0nMMUlHqBWcA2iw63wl/E/iYmJPP/880yYMIGpU6cSEhLi65CMF3mSFOq7VzabExSbmMxp4aHUDrbS1sY/pKSksHjxYm666SYiIyOJj4+3TmjVhCdnH/1bRIZ4PZIAlZmdy7odqfSyoSPjJz788EMiIyOZMGECGzduBLCEUI14khQmAJ+ISIaIHBCRZBE54O3AAsXapFSyc5WY9nZ9gqna9uzZw6hRo7jkkkto3rw5P/zwgxWwq4Y8GT5q5vUoAlhsopM/7cwjU5Xl5ubSr18/tm3bxuOPP87//d//UbOmtU2pjspMCqqaKyKjgI6q+qSItAFaAnFejy4AxCUk07F5fZrUtyJ4purZuXMnrVq1IigoiGeffZaIiAgiIyN9HZbxIU9qH80FBgFj3LsOAS96M6hAoarEbbMieKbqycvL44UXXqBbt268+KLz7zx8+HBLCMajOYWzVXU8kAmgqgcA+9jrgd/2HiTlULbNJ5gqZfPmzQwaNIhbb72VPn36MGzYMF+HZKoQT5JCtojUABRARJoCeV6NKkDE5c8n2JlHpop45ZVX6NGjB2vXrmXBggV8+umndOjQwddhmSrEk6TwHPBPoLmITMa5iG2aV6MKELEJyTSuV5OOzer7OhRjAIiIiGDYsGFs2LCBG264wWpxmeN4MtH8uojEAee7d41U1XXeDSswxCUmWxE841NZWVn8/e9/B+Dxxx+3AnamTCUeKYhIHREJAlDV9cC/cIaNrFi6B/ZnZLF130F62XyC8ZHvvvuO008/nSeeeIJdu3ZZATvjkdKGj1YAnQBEpBOwEogE7hGRJyohNr9mRfCMr2RkZHDnnXdyzjnncOjQIT755BNeeeUVO2I1HiktKTRR1c3u8nXAO6p6C07XtYs82biIXCAim0QkXkQeKGGdK0Vkg4isF5G3yhV9FRaXmEytoBqcakXwTCXbtm0b8+bN469//Svr1q2z9pimXEqbUyh8rHkeMBNAVbNEpMyzj9yhp+eAPwFJwE8iskxVNxRapwvwINBPVZNFpMUJ7EOVFJuYTHR4CHVqWhE8433Jycm8//77jBs3jsjISLZu3UpYWJivwzJ+qLQjhfUiMlVEbge6Ap8CiEgo4MlxaG8g3m3KcwR4ByjanOdm4DlVTQZQ1T3l3YGqKDM7l1+SUq0fs6kUS5cuJTIykltvvZVNmzYBWEIwJ6y0pHATkAF0Ay5Q1YPu/dHA0x5sOxzYXuh2kntfYV2BriLyrYj8ICIXFLchERknIrEiErt3714PXtq31u1I5UhuntU7Ml61e/duRo4cyWWXXUarVq1YuXIlp5xyiq/DMn6uxOEjNwk8Xsz93wLferDt4o4mip7+EAx0AQYCbYD/iki0qqYUec35wHyAmJiYKn8KRaw7yWxJwXhLbm4u/fv3Z/v27Tz55JPcd999VsDOVIgSk4KIfADMAz5T1Zwij7XHmXxOUtUFJWwiCWhb6HYbYGcx6/ygqtnA7yKyCSdJ/FSuvahi4hKT6dCsPs0a1PZ1KCbAJCUlERYWRlBQELNnz6ZDhw5W3tpUqNKGj/6KM0m8WUS+F5FlIvKpiMQDrwLrS0kI4LyxdxGRDiJSCxgFLCuyzgc4xfbye0F3Bbae4L5UCarKKveiNWPvn5wCAAAgAElEQVQqSl5eHnPmzKFbt2688MILAAwbNswSgqlwpQ0f7QDuwbkuoTPQGjgMbFLV9LI2rKo5InIbzvUOQcACVV0vIlOAWFVd5j42REQ24PR//puq7j/pvfKh3/cdZP/BI5YUTIXZuHEjN910E99++y1Dhw5lxIgRvg7JBDBPmuygqvFAfHk3rqrLgeVF7ptYaFlxE095t11V5c8nWLlsUxFefvllbrvtNurVq8fChQsZM2aMXYRmvMqjpGA8F5eQTGjdmnRq3sDXoZgA0KlTJy688ELmzp1Ly5YtfR2OqQYsKVSw2MQD9GrfmBo17NOcKb/MzEymTJkCwJNPPsmgQYMYNGiQj6My1YknpbMRkVruvIIpRfLBI/y296DNJ5gT8u2333L66afzj3/8g71791oBO+MTnrTj/DPwC/CZe/t0EVnq7cD8UZzNJ5gTkJ6ezu23307//v3JyspixYoVvPTSSzZ3YHzCkyOFKUAfIAVAVdcAdtRQjNjEZGoGCT3aNvJ1KMaPJCUl8fLLL3P77bfzyy+/MGTIEF+HZKoxT+YUslU1pcinFjuuLUZc4gGiwkKtCJ4p0/79+3nvvfe45ZZb6N69O1u3bqV169a+DssYj44UfhWRK4Ea7oVozwA/eDkuv5OVk8vPSak2dGRKpaosXryYyMhI7rjjjoICdpYQTFXhSVK4DeiF03VtCZAJ3OnNoPzR+p1pHMnJs6Y6pkS7du3i8ssvZ+TIkbRt25bY2FgrYGeqHE+Gj4aq6v3A/fl3iMhlOAnCuOIS8ovgWblsc7z8AnY7duxg+vTp3H333QQH2xnhpurx5K/yEY5PAA8Xc1+1Fpt4gPZN69G8oRXBM0dt376d8PBwgoKCeO655+jQoQNdu3b1dVjGlKjE4SMRGSois4BwEXm60NfLOENJxqWqxCUm06udDR0ZR25uLrNnzz6mgN3QoUMtIZgqr7QjhT3AOpw5hPWF7k8Hiu23XF0l7j/Evowj9LL5BAP8+uuvjB07lu+//55hw4Zx4YUX+jokYzxWWpXU1cBqEXlTVTMrMSa/c7QIns0nVHfz58/n9ttvp2HDhixatIhrrrnGLkIzfsWTOYVwEXkCiATq5N+pqnYc7IpLPEBInWC6tLAieNVdly5duPTSS5k9ezYtWrTwdTjGlJsnSeE1nLacM4BhwA3YnMIxYhOS6WlF8Kqlw4cPM2nSJESEqVOnWgE74/c8uU6hnqquAFDV31T1EdxuaQZSDh1hy54Mu2itGvrmm2/o0aMH06dPJzU11QrYmYDgSVLIEmdQ9DcRmSAiFwJ2XOxatc2uT6hu0tLSuPXWWxkwYAC5ubl88cUXvPDCCzZ3YAKCJ0nhbqABcAfQD7gZuNGbQfmT2IRkgmsIp1sRvGpj586dvPbaa9xzzz2sXbuW8847z9chGVNhypxTUNUf3cV0YAyAiLTxZlD+JC4xmaiwEOrWsiJ4gWzfvn2899573HrrrXTr1o3ff//dOqGZgFTqkYKInCkil4hIM/d2lIi8jhXEAyA7N4+fk1Js6CiAqSrvvvsukZGR3HXXXWzevBnAEoIJWKVd0fwP4E3gGuATEXkY+Ar4GbDTUXGK4GVmWxG8QLVz504uueQSRo0aRfv27YmLi7Mrkk3AK2346GKgh6oeFpEmwE739qbKCa3qi004AGDtNwNQbm4u5557Ljt27GDGjBnceeedVsDOVAul/ZVnquphAFU9ICIbLSEcKy4xmTaN69IypE7ZKxu/kJiYSJs2bQgKCuL555+nY8eOdO5sjQZN9VHanEJHEVnifi0FIgrdrvYVUlWV2MRkuz4hQOTm5vL000/TvXv3ggJ2Q4YMsYRgqp3SjhQuL3J7rjcD8TfbDxxmb3oWvSJsktnfrVu3jrFjx7Jy5UpGjBjBJZdc4uuQjPGZ0grifVGZgfib2ERnPsGOFPzbiy++yB133EFoaChvvfUWo0aNsovQTLXmycVrphixick0rB1M15YNfR2KOQH5JSm6d+/OyJEj2bBhA1dffbUlBFPt2ekUJyguIZkz2jcmyIrg+ZVDhw4xceJEgoKCmDZtGgMGDGDAgAG+DsuYKsPjIwURsT6TrtTD2Wzek25DR37m66+/5rTTTmPmzJlkZGRYATtjilFmUhCR3iLyC7DFvd1DROZ4PbIqbNW2ZFRtPsFfpKamMn78+IKS1l9++SXPPfecDRUZUwxPjhRmAyOA/QCq+jPVvHT2qsRkgmoIp7ezInj+YNeuXbzxxhvcd999rF271vodGFMKT5JCDVVNLHJfricbF5ELRGSTiMSLSIl9nUXkChFREYnxZLu+FpuQTGTrEOrVsimZqmrv3r3MmeMc0Hbr1o2EhASeeuop6tWr5+PIjKnaPEkK20WkN6AiEiQidwGby3qSiAQBz+F0a4sErhaRyGLWa4hTlvvHoo9VRdm5eazZnmKlLaooVeWtt96ie/fu3HvvvQUF7Jo3b+7jyIzxD54khVuAe4B2wB/AWe59ZekNxKvqVlU9AryDU0+pqL8D04FMjyL2sV93pXE4O9eSQhW0fft2LrzwQq655ho6d+7M6tWrrYCdMeXkyfhHjqqOOoFthwPbC91OAvoUXkFEzgDaqurHInJfSRsSkXHAOIB27dqdQCgVJzbB6bRmlVGrlpycHAYOHMju3buZNWsWt99+O0FB1uPCmPLyJCn8JCKbgHeBJaqa7uG2izu1o+AcQBGpAcwCri9rQ6o6H5gPEBMT49PzCOMSkwlvVJfWoXV9GYZxJSQk0LZtW4KDg5k3bx4dO3akY8eOvg7LGL9V5vCRqnYCHgd6Ab+IyAci4smRQxLQttDtNjjlt/M1BKKBr0UkAWdYallVnmx2iuAdsKGjKiAnJ4cZM2bQvXt3nn/+eQDOP/98SwjGnCSPLl5T1e9U9Q6gJ5CG03ynLD8BXUSkg4jUAkYBywptM1VVm6lqhKpG4HRzu0hVY8u7E5UlKfkwf6Rl2dCRj61du5a+ffvyt7/9jaFDh3L55UVrNxpjTpQnF681EJFrROQjYCWwFzi7rOepag5wG7AC+BV4T1XXi8gUEbnoJOP2ibhEZz7BjhR85/nnn6dXr14kJiby7rvvsnTpUsLCwnwdljEBw5M5hXXAR8B0Vf1veTauqsuB5UXum1jCugPLs21fiE08QIPawXRrFeLrUKodVUVEiI6OZtSoUcyaNYtmzZr5OixjAo4nSaGjquZ5PRI/EJuQzBntGlkRvEp08OBBHnnkEYKDg3nqqac499xzOffcc30dljEBq8ThIxGZ6S7+s3DHteraeS09M5tNf6Tb0FEl+uKLLzj11FN55plnyMrKsgJ2xlSC0o4U3nW/W8c1YPW2FLcInnVa87aUlBTuu+8+XnnlFbp06cI333xD//79fR2WMdVCiUcKqrrSXeyuql8U/gK6V054VUdsYjI1BCuCVwn++OMP3nnnHe6//35+/vlnSwjGVCJPTkm9sZj7xlZ0IFVdXOIBurUKoUFtK4LnDX/88QfPPvssAKeccgoJCQlMnTqVunXtIkFjKlOJ73AichXOtQUdiswhNARSvB1YVZKTm8fqbSlc0auNr0MJOKrKm2++yZ133klGRgbDhw+nS5cudmaRMT5S2sfelTg9FNrgVDvNlw6s9mZQVc3G3ekcOmJF8Cratm3bmDBhAv/+97/p27dvwRyCMcZ3SkwKqvo78DvweeWFUzXFJhwAICbCJpkrSn4Buz179jB79mxuvfVWK2BnTBVQ2vDRf1R1gIgkU6iQHU6hO1XVavMOGZuYTOvQOoQ3svHtk7V161bat29PcHAwL730Ep06dSIiIsLXYRljXKVNNOf3LGwGNC/0lX+72ohLTLaho5OUk5PDtGnTiIyM5LnnnNHIwYMHW0Iwpoop7ZTU/KuY2wJBqpoL9AXGA/UrIbYqYUfKYXalZhJjSeGErVmzhj59+vDAAw8wfPhwRo4c6euQjDEl8OSU1A9wWnF2Al7HuUbhLa9GVYXYfMLJmTt3LmeeeSY7duxg8eLFLFmyhNatW/s6LGNMCTxJCnmqmg1cBjyjqrfjdFWrFlYlJlOvVhDdWjX0dSh+Jb8kxWmnncY111zDhg0brMS1MX7Ao3acIjISGANc4t5X03shVS2xiU4RvOAgj1pPVHsZGRk8/PDD1KxZkxkzZlgBO2P8jKdXNA/CKZ29VUQ6AG97N6yqISMrh193pdHL6h155NNPPyU6Opo5c+aQnZ1tBeyM8UOetONcB9wBxIpIN2C7qj7h9ciqgDXbUshTa6pTluTkZG644QaGDh1KnTp1+Oabb3j22WcRsRLjxvgbTzqv9QfigVeABcBmEenn7cCqgtjEA4jAGVYEr1R79uxh8eLFPPjgg6xZs4ZzzjnH1yEZY06QJ3MKs4DhqroBQES6A4uAGG8GVhXEJSZzSsuGhNSpNlMoHtu9ezdvv/02d999d0EBu6ZNm/o6LGPMSfJkTqFWfkIAUNVfgVreC6lqyM1TVm9LISbCho4KU1UWLlxIZGQkDz74IFu2bAGwhGBMgPAkKawSkXkico779QLVoCDext1pZGTlWFOdQhISErjgggu4/vrriYyMZM2aNVbAzpgA48nw0QScieb/w6l79A0wx5tBVQVxicmATTLny8nJYdCgQezbt4/nnnuOCRMmUKOGnaZrTKApNSmIyKlAJ2Cpqk6vnJCqhtiEZFqG1KZN4+pdBC8+Pp4OHToQHBzMggUL6NixI+3bt/d1WMYYLynxo56IPIRT4uIa4DMRKa4DW8CKS0wmpn2TantaZXZ2Nk8++SRRUVEFBewGDRpkCcGYAFfakcI1wGmqelBEmgPLcU5JDXi7Ug+zI+UwY8/p4OtQfGLVqlWMHTuWNWvWMHLkSK666ipfh2SMqSSlDQpnqepBAFXdW8a6ASV/PqE6nnk0e/Zsevfuze7du1myZAnvvfceLVu29HVYxphKUtqRQsdCvZkF6FS4V7OqXubVyHwoNiGZujWD6N46xNehVBpVRUQ444wzuPbaa5k5cyaNG1e/pGhMdVdaUiha0nKuNwOpSuISkzm9bSNqVoMieOnp6Tz44IPUrl2bmTNn0r9/f/r37+/rsIwxPlJaj+YvKjOQquJgVg4bdqVxy4BOvg7F6z755BPGjx/P9u3bueuuuwqOFowx1VfgfxQup5+3p5Cbp/QK4PmE/fv3c9111zFs2DDq16/Pt99+y9NPP20JwRhjSaGo2MRkRKBnu8BOCkuXLuXRRx9l9erV9O3b19chGWOqCI+TgojULu/GReQCEdkkIvEi8kAxj98jIhtEZK2IfCEiPj8JPjYxma4tGhJaN7CK4O3atYsZM2agqnTt2pXExESmTJlC7drl/rUaYwKYJ6Wze4vIL8AW93YPESmzzIWIBAHPAcOASOBqEYksstpqIEZVTwMWAz69ajo3T1mdmBxQQ0eqyoIFC+jevTuPPvoo8fHxAHZmkTGmWJ4cKcwGRgD7AVT1Z5xObGXpDcSr6lZVPQK8A1xceAVV/UpVD7k3fwDaeBq4N2z+I530rBxiAqTe0e+//86QIUMYO3YsPXr04Oeff7YCdsaYUnlSEK+GqiYWmYTM9eB54cD2QreTgD6lrD8W+HdxD4jIOGAcQLt27Tx46RMTm3/RWgBURs3JyeG8885j//79vPDCC4wbN84K2BljyuRJUtguIr0BdYeEbgc2e/C84k5lKbZpr4j8Badpz4DiHlfV+cB8gJiYGK81/o1LOEDzhrVp28R/i+Bt2bKFjh07EhwczKuvvkqnTp1o27atr8MyxvgJTz463gLcA7QD/gDOcu8rSxJQ+N2oDbCz6Eoicj7wMHCRqmZ5sF2viduWTEz7xn55amZ2djaPP/440dHRzJ3rXGc4cOBASwjGmHIp80hBVfcAo05g2z8BXUSkA7DD3cbowiuIyBnAPOAC93V8Zk9aJtsPHOa6vhG+DOOExMbGMnbsWNauXcuoUaO4+uqrfR2SMcZPlZkUROQlihn2UdVxpT1PVXNE5DZgBRAELFDV9SIyBYhV1WXAU0AD4H330/k2Vb2o/Ltx8grmEyL8az7h2Wef5Z577qFVq1Z8+OGHXHSRT358xpgA4cmcwueFlusAl3LsBHKJVHU5TsntwvdNLLR8vifbqQyxCcnUDq5BpJ8UwcsvSRETE8PYsWOZPn06jRo18nVYxhg/58nw0buFb4vIIuAzr0XkI3GJB+jRthG1gqv2GTppaWncf//91KlTh1mzZtGvXz/69evn67CMMQHiRN4BOwA+v/K4Ih0+ksv6nWlV/vqE5cuXExUVxfz58wkODkbVaydiGWOqKU/mFJI5OqdQAzgAHFeywp+t2Z5CTp5W2aY6+/bt46677uLNN98kKiqKxYsX06dPaZd8GGPMiSk1KYgz+9sD5+whgDwNwI+ncYkHgKpbBC85OZmPPvqIxx57jIceeohatWr5OiRjTIAqNSmoqorIUlXtVVkB+UJsYjJdWjSgUb2q82a7Y8cO3nzzTf72t7/RpUsXEhMTbSLZGON1nswprBSRnl6PxEfy8pRViclVZuhIVXnppZeIjIxk0qRJ/PbbbwCWEIwxlaLEpCAi+UcR5+Akhk0iskpEVovIqsoJz/u27MkgLTOHXlWg3tFvv/3G4MGDGTduHD179mTt2rV07tzZ12EZY6qR0oaPVgI9gUsqKRafiCsogufbI4WcnBwGDx7MgQMHmDdvHjfddJMVsDPGVLrSkoIAqOpvlRSLT8QmHqBZg1q0b1rPJ6+/adMmOnXqRHBwMAsXLqRTp060aePTCuLGmGqstKTQXETuKelBVX3aC/FUurjEZHr5oAjekSNH+Mc//sETTzzBU089xZ133smAAcUWiTXGmEpTWlIIwqlL5H8lQz20Nz2LxP2H+Eufyr0Wb+XKlYwdO5Z169YxevRorrnmmkp9fWOMKUlpSWGXqk6ptEh8oOD6hEqcT3jmmWe49957ad26NR999BEjRoyotNc2xpiylDaTGbBHCPliE5KpFVyD6HDvF8HLv+avd+/e3Hzzzaxfv94SgjGmyintSGFwpUXhI7GJyfRoE0rt4CCvvUZqair/93//R926dXnmmWc4++yzOfvss732esYYczJKPFJQ1QOVGUhly8zOZf3OVK9en/DRRx8RGRnJyy+/TO3ata2AnTGmyqu2J8L/vD2F7Fz1yvUJe/fuZfTo0Vx00UU0bdqUH374gWnTpvllm09jTPVSbZNCfqe1Xl5ICqmpqSxfvpzJkycTGxvLmWeeWeGvYYwx3uBJ57WAFJeYTKfm9Wlcv2KK4G3fvp033niDBx54gM6dO5OYmEhoaGiFbNsYYypLtTxSyMtT4hKTiamA+YS8vDxefPFFoqKiePzxxwsK2FlCMMb4o2qZFLbuyyD1cDa9TrIy6pYtWzjvvPO45ZZb6N27N7/88osVsDPG+LVqOXwUm3DyRfBycnL405/+REpKCq+88go33HCDTSQbY/xe9UwKick0qV+LDs3ql/u5v/76K126dCE4OJhFixbRqVMnwsLCvBClMcZUvmo5fBSXmEzPduUrgpeVlcVjjz3Gaaedxty5cwHo37+/JQRjTECpdkcK+zKy+H3fQa46s63Hz/nhhx8YO3YsGzZsYMyYMYwZM8aLERpjjO9UuyOF8jbVmTlzJmeffTbp6eksX76c119/naZNm3ozRGOM8ZlqmRRqBdUgOrz0U0bz8vIA6Nu3LxMmTGDdunUMGzasMkI0xhifqXbDR7EJBzi1TSh1ahZfBC8lJYV7772XevXqMWfOHCtgZ4ypVqrVkUJmdi7rdqSVOHT0wQcfEBkZycKFC2nYsKEVsDPGVDvVKin8siOVI7l5x9U72rNnD1deeSWXXnopLVu2ZOXKlTz55JN23YExptqpVkkh/6K1okkhLS2Nzz77jCeeeIKVK1fSs2dPX4RnjDE+V63mFOISk+nYrD5NG9Rm27ZtLFq0iIceeojOnTuzbds2GjZs6OsQjTHGp7x6pCAiF4jIJhGJF5EHinm8toi86z7+o4hEeCsWVWXVtmR6tm/E888/T1RUFE8++WRBATtLCMYY48WkICJBwHPAMCASuFpEIousNhZIVtXOwCxgmrfi2brvIAcOHuHLd1/mr3/9K3379mX9+vVWwM4YYwrx5pFCbyBeVbeq6hHgHeDiIutcDCx0lxcDg8VLs7srt+4DIDHuK1599VVWrFhBRESEN17KGGP8ljfnFMKB7YVuJwF9SlpHVXNEJBVoCuwrvJKIjAPGAbRr1+6EgmnaoA69WgYz+9vPCLd6RcYYUyxvJoXiPvEXPfHfk3VQ1fnAfICYmJgTunhgSFQrhkS1OpGnGmNMteHN4aMkoHDVuTbAzpLWEZFgIBQ44MWYjDHGlMKbSeEnoIuIdBCRWsAoYFmRdZYB17nLVwBfql1GbIwxPuO14SN3juA2YAUQBCxQ1fUiMgWIVdVlwCvAIhGJxzlCGOWteIwxxpTNqxevqepyYHmR+yYWWs4ERnozBmOMMZ6rVmUujDHGlM6SgjHGmAKWFIwxxhSwpGCMMaaA+NsZoCKyF0g8wac3o8jV0tWA7XP1YPtcPZzMPrdX1eZlreR3SeFkiEisqsb4Oo7KZPtcPdg+Vw+Vsc82fGSMMaaAJQVjjDEFqltSmO/rAHzA9rl6sH2uHry+z9VqTsEYY0zpqtuRgjHGmFJYUjDGGFMgIJOCiFwgIptEJF5EHijm8doi8q77+I8iElH5UVYsD/b5HhHZICJrReQLEWnvizgrUln7XGi9K0RERcTvT1/0ZJ9F5Er3d71eRN6q7Bgrmgd/2+1E5CsRWe3+fQ/3RZwVRUQWiMgeEVlXwuMiIrPdn8daEelZoQGoakB94ZTp/g3oCNQCfgYii6xzK/CiuzwKeNfXcVfCPg8C6rnLt1SHfXbXawh8A/wAxPg67kr4PXcBVgON3dstfB13JezzfOAWdzkSSPB13Ce5z+cCPYF1JTw+HPg3TufKs4AfK/L1A/FIoTcQr6pbVfUI8A5wcZF1LgYWusuLgcEiUlxrUH9R5j6r6leqesi9+QNOJzx/5snvGeDvwHQgszKD8xJP9vlm4DlVTQZQ1T2VHGNF82SfFQhxl0M5vsOjX1HVbyi9A+XFwOvq+AFoJCKtK+r1AzEphAPbC91Ocu8rdh1VzQFSgaaVEp13eLLPhY3F+aThz8rcZxE5A2irqh9XZmBe5MnvuSvQVUS+FZEfROSCSovOOzzZ50nAX0QkCad/y+2VE5rPlPf/vVy82mTHR4r7xF/0vFtP1vEnHu+PiPwFiAEGeDUi7yt1n0WkBjALuL6yAqoEnvyeg3GGkAbiHA3+V0SiVTXFy7F5iyf7fDXwmqrOFJG+ON0co1U1z/vh+YRX378C8UghCWhb6HYbjj+cLFhHRIJxDjlLO1yr6jzZZ0TkfOBh4CJVzaqk2LylrH1uCEQDX4tIAs7Y6zI/n2z29G/7Q1XNVtXfgU04ScJfebLPY4H3AFT1e6AOTuG4QOXR//uJCsSk8BPQRUQ6iEgtnInkZUXWWQZc5y5fAXyp7gyOnypzn92hlHk4CcHfx5mhjH1W1VRVbaaqEaoagTOPcpGqxvom3Arhyd/2BzgnFSAizXCGk7ZWapQVy5N93gYMBhCR7jhJYW+lRlm5lgHXumchnQWkququitp4wA0fqWqOiNwGrMA5c2GBqq4XkSlArKouA17BOcSMxzlCGOW7iE+eh/v8FNAAeN+dU9+mqhf5LOiT5OE+BxQP93kFMERENgC5wN9Udb/voj45Hu7zvcBLInI3zjDK9f78IU9E3sYZ/mvmzpM8BtQEUNUXceZNhgPxwCHghgp9fT/+2RljjKlggTh8ZIwx5gRZUjDGGFPAkoIxxpgClhSMMcYUsKRgjDGmgCWFakxEckVkTaGviFLWjSipamNlE5EYEZntLg8UkbMLPTZBRK6txFhOP5GqnCLSWkQ+dpebulU+M0Rk7gnG8bBbFXWt+7vscyLbKWX7y0Wkkbt8h4j8KiJvishFpVWoddf/zv0eISKjPXitESIyuWIiN+Vlp6RWYyKSoaoNPFw3AvhYVaO9GlQ5icgkIENVZ3jxNYLdGlnFPXY9TvXV28q5zaeA/6nqhyJSHzgD5wrs6BPYVl/gaWCgqma5F63VUlWvFIYTkY3AMPeK6fI8byBwn6qOKGM9AVYB/QoVcTSVxI4UzDHcT3P/FZFV7tfZxawTJSIr3U+ka0Wki3v/XwrdP09Egop5boKITHPXWykind3724vT5yG/30M79/6RIrJORH4WkW/c+waKyMduopoA3O2+Zn8RmSQi94lIdxFZWWS/1rrLvUTkPyISJyIrpJgKkyLymog8LSJfAdNEpLeIfCdOzf7vROQU9wrbKcBV7utfJSL1xamH/5O7bnGVWwEuBz4BUNWDqvo/TrySa2tgX37pElXdl58QSvl5NxeRf7px/iQi/dz7G4jIqyLyi/u7uLzQdpqJyIs4ZayXicjdInJ9/tGNiLQUkaXu7+rn/L8dEclw45wK9Hd/Vne7f2enF/qZfysip7kXnn0NlJo8jJd4uza4fVXdL5wrXte4X0vd++oBddzlLjhXjQJE4NZ3B+YA17jLtYC6QHfgI6Cme//zwLXFvGYC8LC7fC3O0Qfuc69zl28EPnCXfwHC3eVG7veBhZ43CefTJ0Vvu/vV0V2+H3gE58rQ74Dm7v1X4VwlWzTO14CPgSD3dggQ7C6fD/zTXb4emFvoeU8Cf8mPF9gM1C+y7Q5AXDGvecy2yvF7bODu62b35z7Ag5/3W8A57nI74Fd3eRrwTKHnNy60nWbFLBfEDLwL3OUuBwGh7nJG0d+be/u6/NfCKccRW+ixa4A5vv4fqY5fAVfmwpTLYVU9vch9NYG57ie4XJx/1qK+Bx4WkTbAEpCjoOgAAAOnSURBVFXdIiKDgV7AT87RP3WBkmosvV3o+yx3uS9wmbu8CKcHAsC3wGsi8h6wpDw7h1Mk7UqcT6hXuV+n4AzTfObGGQSUVDfmfVXNdZdDgYXuUZHilh0oxhDgIhG5z71dB/dNt9A6ranA2jyqmiEivYD+OHWP3hWRB1T1NXeV4n7e5wORcrSNSIiINHTvLyj7om5fBg+dh5N4cH9uqWWs/z7wqIj8DeeDwGuFHtsDhJXjtU0FsaRgirob+APogTO8eNyQhqq+JSI/An8GVojITTjlfBeq6oMevIaWsHzcOqo6QZxJ0z8DawoPN3jgXZxaT0ucTekWETkVWK+qfT14/sFCy38HvlLVS91hq69LeI4Al6vqplK2exgnWXjM/RnMc29O1CK1ndw34a9xqsL+gvMp/LX8hwuv6n6vAfRV1cNFXkeopDLyqnpIRD7DaRpzJU5J93x1cH5OppLZnIIpKhTYpU4t+jE4n6SPISIdga2qOhunYuNpwBfAFSLSwl2niZTcB/qqQt+/d5e/4+gn1GuA/7nb6aSqP6rqRGAfx5YMBkjHKZN9HFX9Dedo51GcBAFOKenm4kzOIiI1RSSqhDgLCwV2uMvXl/L6K4Db3TfX/Oq0RW3GGY7zmPszON39KloB95T8eR3X6UBiodvF/bw/BQomtAsl26L3Ny5HmF/gtHpFRIJEJKTI48X9rl4GZgM/qWrh8vVdgSpxtlt1Y0nBFPU8cJ2I/IDzj3mwmHWuAtaJyBqgG05rwA04Y/afuhO6n+EMkxSntnukcSfOkQnAHcD/t3e/KhEFURzHv79gs5gEg1m2ui+w+AQmi4I+gdhtCqI+hE0wGcRsMhgW3EW2CQajSbCIcAxn9qrXFVZQr7C/T71/ZphwD2fO3JmN8uxauQZwWIqeN+RZy73au86A5WGheURbJ8Aqb/vtP5Pbpe9L6pFz8Z+K6SMcAHuSLvkYKC/IaZhrSStkRjEF9Eufd+oviogn4HZY9IUs5JIriNYl3UtqjdGnoWlyamtQxq9F1laGvhrvdikmD8iCPcAuMKNS3Kdswz2mTaBTMpUuUA+2feClFKG3ACKiCzwCR7V7O8D5N9q2H+IlqfanysevHREPTfelSZKWgcWI2P7ldu74x+MtaY6c9loo2SmSZoHjiFhqsm+TypmCWQMi4pRcxTOxlD8ZXpGro94fnTlPnpFgDXCmYGZmFWcKZmZWcVAwM7OKg4KZmVUcFMzMrOKgYGZmlVeuuR0AtFDNDgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC =  0.813\n"
     ]
    }
   ],
   "source": [
    "dt_fpr, dt_tpr, dt_thresholds = roc_curve(best[0], best[1], pos_label=1)\n",
    "roc_auc = auc(dt_fpr, dt_tpr)\n",
    "plt.figure(1)\n",
    "plt.plot([0, 1], [0, 1], 'k--')\n",
    "plt.plot(dt_fpr, dt_tpr, label='DecisionTree')\n",
    "plt.xlabel('False positive rate (1 - Specificity)')\n",
    "plt.ylabel('True Positive Rate (Sensitivity)')\n",
    "plt.title('ROC curve')\n",
    "plt.legend(loc='best')\n",
    "plt.show()\n",
    "print(\"AUC = \", roc_auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "widgets": {
   "state": {},
   "version": "1.1.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}