21_Introduction_to_Annotation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "3c16a729-423b-4978-8d30-39a2ffff3161"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What is the purpose of annotation?\n",
    "1. Create a training or development set for NLP \n",
    "2. Test NLP against a \"reference standard\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "55dacc19-5f92-4f4f-afc8-cea14796618c"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Annotation workflow\n",
    "![image.png](./images/Slide09.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "ce94fc5c-289e-4e61-a050-287ea4010b63"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Markables – elements to be annotated\n",
    "Annotation type = label to be assigned to a segment of text\n",
    "\n",
    "Relationship = link between instances of annotations\n",
    "\n",
    "Attributes = features of annotation types and relationships\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "d0392878-4cb6-42de-bee8-3fccb5c28181"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "## Annotation Schema – collection of all markables for a project\n",
    "\n",
    "![image.png](./images/Selection_064.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "472f845b-e5ae-48b0-a43d-a3d3ebe2ede1"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Concept definition\n",
    "Diagnosis, lab test, action, event…\n",
    "## Variable definition – Values that the concepts can have\n",
    "Diagnosis: explicitly mentioned or inferred \n",
    "\n",
    "Lab test: exact numeric value or range or direction\n",
    "\n",
    "Action: planned or occurred\n",
    "\n",
    "Event: explicitly mentioned or inferred\n",
    "## Level of the annotation \n",
    "patient, event, document, or phrase\n",
    "## Attribute definition – additional labels to labels\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "971a9f77-ffa7-4a68-8961-eb83013befd9"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![image.png](./images/attributes.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "078b37fd-a5a3-415f-962b-a8a1ad84d90d"
    }
   },
   "source": [
    "## Document level annotation is what our clients care about, but to build the NLP system we need to be able to identify mentions/markables in the text that demonstrate evidence for pneumonia \n",
    "We had an expert annotate some reports at the document level and mark evidence in the reports so that we can learn how she determines when a document is positive for pneumonia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "8d1d3751-dc60-409c-9103-91d0defe6381"
    }
   },
   "source": [
    "## Here is an article Wendy published as part of her dissertation looking at the finding-disease continuum of pneumonia in chest x-ray reports\n",
    "![image.png](images/pub.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "a77793e1-f049-4944-8e64-7ed36a5c5188"
    }
   },
   "source": [
    "## Fill in the cells of the table with variables you want to identify as mentions (evidence in the text)\n",
    "| Level        | Source           | Concept definition |  Variable  | Attribute |\n",
    "| ------------ |-------------| -----|-----|-------|\n",
    "| document     | chest x-ray reports | evidence of pneumonia descdribed in the report| PNEUMONIA_DOC_YES |  |\n",
    "| document     | chest x-ray reports | no evidence of pneumonia descdribed in the report| PNEUMONIA_DOC_NO |  |\n",
    "| mention      | chest x-ray reports | mention of pneumonia | PNEUMONIA | asserted, negated, uncertain |\n",
    "| mention      | chest x-ray reports | mention of consolidation | CONSOLIDATION | asserted, negated, uncertain |\n",
    "| mention      | chest x-ray reports | mention of localized infiltrate | LOCAL_INFILTRATE | asserted, negated, uncertain |\n",
    "\n",
    "To make the annotation task easier, at mention level, we only annotate asserted mentions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "7d2c0411-5ad7-47f9-86b6-d060520b4d53"
    }
   },
   "source": [
    "## If you are going to annotate mentions, you have to think about the span\n",
    "![image.png](./images/annotool.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "a71f92ed-a8cd-4632-b89e-301b9e808a71"
    }
   },
   "source": [
    "## All of your instructions should go into a document that is your annotation guideline \n",
    "### Use these to train your annotators - it is a living document during the training phase until you freeze it\n",
    "![image.png](./images/guideline.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "10fce140-7321-44de-a14e-453516a38275"
    }
   },
   "source": [
    "Here's a paper to help you think through what to include or not include when marking symptoms and diseases in clinical text: \n",
    "[chapman_jbi_2007_annotation_schema.pdf](./chapman_jbi_2007_annotation_schema.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "d7eec1a2-28a1-4627-a1a5-c85ff17c6311"
    }
   },
   "source": [
    "<br/><br/>This material presented as part of the DeCART Data Science for the Health Science Summer Program at the University of Utah in 2019.<br/>\n",
    "Presenters : Dr. Wendy Chapman, Kelly Peterson, Alec Chapman, Jianlin Shi <br> Acknowledgement: Many thanks to Olga Patterson because part of the materials are adopted from his previous work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "687bfa7e-a06e-498b-99b9-65e7323a33ce"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "nbpresent": {
   "slides": {
    "13c05bfb-f984-409e-a3e9-a1b0df84d7c9": {
     "id": "13c05bfb-f984-409e-a3e9-a1b0df84d7c9",
     "prev": "25479455-d679-4cda-8a5c-4cc570209ded",
     "regions": {
      "cc531130-4757-4f10-a868-af05cc45403f": {
       "attrs": {
        "height": 1,
        "width": 1,
        "x": 0,
        "y": 0
       },
       "content": {
        "cell": "55dacc19-5f92-4f4f-afc8-cea14796618c",
        "part": "source"
       },
       "id": "cc531130-4757-4f10-a868-af05cc45403f"
      }
     }
    },
    "25479455-d679-4cda-8a5c-4cc570209ded": {
     "id": "25479455-d679-4cda-8a5c-4cc570209ded",
     "prev": null,
     "regions": {
      "c8a33824-5905-42e5-8bf9-264f471269d5": {
       "attrs": {
        "height": 1,
        "width": 1,
        "x": 0,
        "y": 0
       },
       "content": {
        "cell": "3c16a729-423b-4978-8d30-39a2ffff3161",
        "part": "source"
       },
       "id": "c8a33824-5905-42e5-8bf9-264f471269d5"
      }
     }
    },
    "62d2fc02-69fd-4a5b-a6eb-f04936bddedb": {
     "id": "62d2fc02-69fd-4a5b-a6eb-f04936bddedb",
     "prev": "13c05bfb-f984-409e-a3e9-a1b0df84d7c9",
     "regions": {
      "90f7a027-03ab-4719-b15c-963b604b2641": {
       "attrs": {
        "height": 1,
        "width": 1,
        "x": 0,
        "y": 0
       },
       "content": {
        "cell": "ce94fc5c-289e-4e61-a050-287ea4010b63",
        "part": "source"
       },
       "id": "90f7a027-03ab-4719-b15c-963b604b2641"
      }
     }
    },
    "7075772f-c9d3-4d84-8e0f-87c2b7085b33": {
     "id": "7075772f-c9d3-4d84-8e0f-87c2b7085b33",
     "prev": "62d2fc02-69fd-4a5b-a6eb-f04936bddedb",
     "regions": {
      "07aa95ab-6eea-4970-8afd-d7a95a26db5d": {
       "attrs": {
        "height": 1,
        "width": 1,
        "x": 0,
        "y": 0
       },
       "content": {
        "cell": "d0392878-4cb6-42de-bee8-3fccb5c28181",
        "part": "source"
       },
       "id": "07aa95ab-6eea-4970-8afd-d7a95a26db5d"
      }
     }
    },
    "8b82da2c-d006-47d3-ad03-dd28b4db081b": {
     "id": "8b82da2c-d006-47d3-ad03-dd28b4db081b",
     "prev": "a3b865df-ea14-4888-ad1e-19c045b0e4ff",
     "regions": {
      "6b0d972d-cc11-437a-883b-2b2c827ced5d": {
       "attrs": {
        "height": 1,
        "width": 1,
        "x": 0,
        "y": 0
       },
       "content": {
        "cell": "971a9f77-ffa7-4a68-8961-eb83013befd9",
        "part": "source"
       },
       "id": "6b0d972d-cc11-437a-883b-2b2c827ced5d"
      }
     }
    },
    "a3b865df-ea14-4888-ad1e-19c045b0e4ff": {
     "id": "a3b865df-ea14-4888-ad1e-19c045b0e4ff",
     "prev": "7075772f-c9d3-4d84-8e0f-87c2b7085b33",
     "regions": {
      "ab9a3f02-fa3e-4d78-a430-b08909e76228": {
       "attrs": {
        "height": 1,
        "width": 1,
        "x": 0,
        "y": 0
       },
       "content": {
        "cell": "472f845b-e5ae-48b0-a43d-a3d3ebe2ede1",
        "part": "source"
       },
       "id": "ab9a3f02-fa3e-4d78-a430-b08909e76228"
      }
     }
    }
   },
   "themes": {}
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "384px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}