Session1_Introduction to Natural Language Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8511b908",
   "metadata": {},
   "source": [
    "# Natural Language Processing using Python\n",
    "#### This  4 hour workshop will introduce the audience to the field of Natural Language Processing,  we will cover the fundamental tasks in NLP and walk through the Python code for some of the important steps in an NLP project.\n",
    "\n",
    "#### Author: Dr. Nimrita Koul, Associate Professor, Machine Learning\n",
    "#### Who this talk is suitable for: Data analysts, students and anyone interested in using Natural Language Processing\n",
    "#### Prerequisites: Python Programming, basic familiarity with concepts in data science and machine learning\n",
    "#### Duration: 4 Sessions of 1 hour each\n",
    "#### Time:  Starting 9th March 2023, Every Thursday,  8.00 pm to 9.00 pm Indian Standard Time\n",
    "#### Github URL of this Jupyter Notebook: https://github.com/NimritaKoul/NLP_WWC2023/blob/main/Session1_Introduction%20to%20Natural%20Language%20Processing.ipynb\n",
    "\n",
    "## https://tinyurl.com/tjsf6wbk\n",
    "\n",
    "\n",
    "\n",
    "### Contents: \n",
    "\n",
    "#### Session 1:\n",
    "1. Introduction to Natural Language Processing, components, task, applications. \n",
    "2. Python Tools and Libraries for NLP - with focus on NLTK and SpaCy \n",
    "\n",
    "#### Session 2:\n",
    "1. Text Preprocessing - Cleaning, normalization - tokenization, stopword removal, case conversion, stemming, lemmatization, POS tagging, Named Entity Recognition\n",
    "2. Text Representation - Bag of word models, TF-IDF, Word Embeddings\n",
    "\n",
    "#### Session 3:\n",
    "1. Text Classification - Text classification models Naive Bayes, Decision Tree, RNN\n",
    "2. Transformers\n",
    "\n",
    "#### Session 4 :  Case Study\n",
    "1. Sentiment Analysis\n",
    "2. Chatbot "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a622aca2",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "1. https://www.ibm.com/in-en/topics/natural-language-processing\n",
    "\n",
    "2. https://python-course.eu/machine-learning/natural-language-processing-with-python.php\n",
    "\n",
    "3. https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1184/syllabus.html\n",
    "\n",
    "4. https://www.nltk.org/book\n",
    "\n",
    "5. https://www.analyticsvidhya.com/blog/2021/07/nltk-a-beginners-hands-on-guide-to-natural-language-processing/\n",
    "\n",
    "6. https://web.stanford.edu/~gentzkow/research/text-as-data.pdf\n",
    "\n",
    "7. https://github.com/n-kostadinov/sentiment-analysis\n",
    "\n",
    "8. https://notebook.community/DryingPole/sentiment/notebooks/Final%20Project%20Process%20Book\n",
    "\n",
    "9. https://nbviewer.org/url/norvig.com/ipython/How%20to%20Do%20Things%20with%20Words.ipynb\n",
    "\n",
    "10. https://towardsdatascience.com/named-entity-recognition-with-nltk-and-spacy-8c4a7d88e7da\n",
    "\n",
    "11. https://spacy.io/usage/spacy-101\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d1e06f6",
   "metadata": {},
   "source": [
    "## Session 1 : 9 March 2023\n",
    "\n",
    "### On the Agenda:\n",
    "\n",
    "1. Introduction to Natural Language Processing \n",
    "2. Getting started with Python Libraries NLTK and SpaCy\n",
    "\n",
    "# Natural Language Processing (NLP) \n",
    "\n",
    "#### A Natural Language is a language like Hindi, English, German, French, Spanish etc. that is used by \"Humans\" to communicate among themselves. \n",
    "\n",
    "#### Spoken and written text in human languages is the most abundant type of data on the Internet and it is unstructured. \n",
    "\n",
    "<img src = 'dataeveryminute.png' width = 900>\n",
    "<center>Source: https://dailyinfographic.com/how-much-data-is-generated-every-minute</center>\n",
    "\n",
    "#### This data comes from: \n",
    "\n",
    "- Social media interaction of users: twitter, facebook, linkedin, youtube, reddit, instagram etc.\n",
    "- Large collections of unstructured electronic text for research purposes\n",
    "- Data provided for open access e.g., google books, project gutenberg, wikidata, data at government sites\n",
    "- Library databases: data provided by libraries of academic universities\n",
    "etc. \n",
    "\n",
    "\n",
    "#### The unstructured text data holds valuable information. Application of ML models to this data has lead to development of applications and tools for improving the way we do work. Such applications are used in the domains of governance, finance, social media, customer engagement, politics, health care, entertainment. (Alexa, Siri, Google Translate, ChatGPT)\n",
    "\n",
    "\n",
    "## Natural Language Processing (NLP) is a branch of AI that aims at building computer systems that can understand natural language and respond in a natural language.\n",
    "\n",
    "- <b>Human language is filled with ambiguities</b> To \"understand\" written or spoken text in a natural language means to recognize the meaning, the intent and the sentiment of it.NLP uses statistical models, machine learning, deep learning to understand text. \n",
    "\n",
    "\n",
    "\n",
    "<img src = 'Ambiguities1.png' width=800 height = 600>\n",
    "<center>Source: https://fidelandrada-writing.medium.com/ambiguity-f14fc53b4721</center>\n",
    "\n",
    " \n",
    " \n",
    "<img src = 'ambiguity3a.jpg' width = 400 height = 400>\n",
    "<center>Source:https://www.pinterest.com/pin/siri-call-an-ambulance--13088655139792945/</center>\n",
    "\n",
    "### These ambiguities in natural language make task of understanding natural language very hard. A number of NLP tasks are required to convert the text data into a meaningful form. These involves tasks like:\n",
    "\n",
    "<img src = 'postagsnltk.png'>\n",
    "<center>Source: https://www.analyticsvidhya.com</center>\n",
    "\n",
    "\n",
    "### Text Preprocessing\n",
    "1.<b>Tokenization </b> Splitting a large chunk of text into constituents - sentences, words, punctuations.\n",
    "\n",
    "2.<b>Normalization </b> Case conversion, Removing the non-useful words (stopwords like the, a, an etc.), bringing words in their stem forms, replacing words by their most used form (lemma).\n",
    "\n",
    "### Converting Text to Numerical Vectors\n",
    "3.<b>Vector Embedding</b> Converting words and sentences into numeric vectors for use with computational algorithms. \n",
    "\n",
    "### Lexical Tasks\n",
    "\n",
    "4.<b>Part of speech tagging </b>or grammatical tagging, Identifying grammatical class (Part of speech) of the word e.g., nouns, verbs etc. based on its use and context. E.g., to correctly tag  ‘make’ as a verb in ‘I can make a paper plane,’ and as a noun in ‘What make of car do you own?’\n",
    "\n",
    "5.<b>Identifying relationships</b> Recognising relationships among words in the text, e.g., subject - object relationships.\n",
    "\n",
    "6.<b>Word sense disambiguation</b> Recognising the intended meaning of a word in context. E.g., 'make' in ‘make the grade’ (achieve) vs. 'make a bet' (place).\n",
    "\n",
    "7.<b>Named entity recognition </b> Is the task of identifying names of people, geographic locations, organisations etc. from text.  \n",
    "\n",
    "8.<b>Co-reference resolution </b> is the task of identifying if and when two words refer to the same entity. It includes pro-noun resolution, identifying idiom or metaphor in text. \n",
    "\n",
    "\n",
    "### Higher Level Tasks\n",
    "9.<b>Sentiment analysis</b> Identification of subjective attitudes, emotions etc. \n",
    "\n",
    "10.<b>Text Classification</b> Categorising text into separate classes based on presence of certain words.\n",
    "\n",
    "11.<b>Speech recognition </b> It is the task of converting voice data into text data. Different speeds, punctuation, emphasis, intonation, accents, incorrect grammar slurrying words together makes speech recognition very challenging.\n",
    "\n",
    "12.<b>Recognizing Textual Entailment:</b> Textual entailment or Natural Language Inference (NLI) is the task of recognizing whether one piece of text can be inferred from another piece of text.\n",
    "\n",
    "13.<b>Natural language generation </b>is the task of putting structured information into human language.\n",
    "\n",
    "14.<b>Language Modelling</b> A language model generates a probability distribution over sequences of words in one or many languages.  We train language models on the text corpora on there languages. Once trained, these models can generate natural langauge text accurately. (GPT3!!)\n",
    "\n",
    "## NLP use cases\n",
    "\n",
    "1.<b>Cybersecurity</b> Spam detection involves using text classification to identify spam email. Presence of certain words and attributes of text like overuse of financial terms, wrong grammar, threats, inappropriate urgency etc. \n",
    "\n",
    "2.<b>Machine translation:</b> Translation of text in one language to another while preserving the intended meaning. \n",
    "\n",
    "3.<b>Conversational AI/Smart Virtual agents and chatbots: </b> These softwares first use speech recognition to capture the voice command from a human user, then they use natural language generation technologies to generate response. Chatbots have proved to be very effective as customer service chatbots for organisations for the tasks like answering FAQs, booking appointments, tickets etc. \n",
    "\n",
    "4 <b>Question-answering systems:</b> These can respond to well-formed and not so well formed questions. \n",
    "\n",
    "5.<b>Social media sentiment analysis:</b> Such systems help discover hidden insights, intents, attitudes, emotions from data customers put of social media forums, e-commerce sites etc. \n",
    "\n",
    "6.<b>Text summarization:</b> These systems parse huge volumes of text and create summaries of them. They employ semantic reasoning and natural language generation. \n",
    "\n",
    "7.<b>Generating Captions for Images.</b>\n",
    "\n",
    "8.<b>Language Models:</b> Used for machine translation, OCR, text generation, (GPT, LaMDA etc.)\n",
    "\n",
    "9.<b>Autocorrect</b> Correct the spellings or grammar of text as it is being types.\n",
    "\n",
    "10.<b>Autocomplete:</b> Suggest sentence completion\n",
    "\n",
    "11.<b>Semantic Search:</b> Context aware search, useful for e-commerce\n",
    "\n",
    "12.<b>Healthcare:</b> Convert hand written prescriptions into text or voice, clinical documentation, clinical trial matching, computational phenotyping, etc.\n",
    "\n",
    "13.<b>Finance and Insurance:</b> Stock Analysis, credit scoring, insurance claims management, financial reporting and auditing\n",
    "\n",
    "14.<b>HR:</b> Resume screening and evaluation, interview assessment (sentiment analysis, document summarization), video transcription, employee sentiment analysis.\n",
    "\n",
    "15.<b>Market Intelligence:</b> Applies webscraping, sentiment analysis to detect market trends and optimize current market strategy for a company."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67e97b3c",
   "metadata": {},
   "source": [
    "\n",
    "## Machine Learning Models for NLP\n",
    "\n",
    "#### Machine Learning Algorithms used for NLP\n",
    "\n",
    "1. Naive Bayes Classifier\n",
    "\n",
    "2. Decision Tree\n",
    "\n",
    "3. Support Vector Machines etc. \n",
    "\n",
    "#### Deep Learning Architectures for NLP\n",
    "\n",
    "1. RNN\n",
    "\n",
    "2. CNN\n",
    "\n",
    "3. LSTM\n",
    "\n",
    "4. Transformers\n",
    "\n",
    "#### Prominent Deep Learning Models for NLP\n",
    "\n",
    "1. BERT: Pretrained Deep Bidirectional Transformers\n",
    "\n",
    "2. GPT2 and GPT3: Unsupervised Multitask Learners, Few-Shot Learners\n",
    "\n",
    "3. XLNet: Generalized Autoregressive Pretraining \n",
    "\n",
    "4. T5: Transfer Learning with a Unified Text-to-Text Transformer\n",
    "\n",
    "5. PaLM: Language Modeling with Pathways\n",
    "\n",
    "#### Python has some very powerful libraries for NLP\n",
    "\n",
    "1. NLTK - Natural Language Toolkit - NLTK is a powerful, open source library for research and development in NLP. It has more than 50 text corpora and lexical resources built into it. It supports text tokenization, part of speech tagging, stemming, lemmatization,named entity extraction, segmentation, classification, semantic reasoning.  \n",
    "\n",
    "2. SpaCY - SpaCy is also an open source python library for production level use for building real world projects. It has builtin support for multi-trained transformers like BERT and pre-trained NLP pipelines for more than 17 languages. It is very fast and provides following functionality - tokenization in more than 49 languages, part of speech tagging, segmentation, lemmatization, named entity recognition, text classification.\n",
    "\n",
    "3. TextBlob - TextBlob is an open-source library built on top of NLTK. It provides an easy interface and supports tasks like sentiment analysis, phrase extraction, parsing, Part-of-speech tagging, N-grams,Spelling correction, Tokenization\n",
    "Classification, Noun phrase extraction.\n",
    "\n",
    "4. Gensim - GenSim supports algorithms like Hierarhical Dirichlet Process (HDP), Random Projections, latent Dirichlet Allocation (LDA), latent Semantic Analysis or word2vec deep learning. It is very fast and optimizes memory useage. \n",
    "\n",
    "5. PolyGlot - PolyGlot supports many languages and is built on SpaCy and NumPy libraries. It supports tokenization in 165 languages, language detection for 196 languages, named entity recognition, POS tagging, sentiment analysis, word embeddings for 137 languages, morphological analysis, transliteration in 69 languages.  \n",
    "\n",
    "6. sklearn - Standard Machine Learning library in Python\n",
    "\n",
    "We will look at the functionality of NLTK and SpaCy in some detail."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78f7f326",
   "metadata": {},
   "source": [
    "## Natural Language Toolkit (NLTK)\n",
    "\n",
    "- For this series of sessions on NLP, familiarity with Python Programming (Python 3) is assumed. For installing libraries for NLP, you should have Python already installed on your computer. Anaconda Distribution of Python is the preferred Python for this series. You can have Python download from https://www.python.org/downloads/. You can download and install Anaconda Python from here: https://www.anaconda.com/products/distribution. \n",
    "\n",
    "- In case you need to refresh your Python Programming skills, please refer these tutorials:\n",
    "1. https://www.w3schools.com/python/ \n",
    "2. https://docs.python.org/3/tutorial/\n",
    "\n",
    "#### Download and Install\n",
    "using pip:\n",
    "<code> pip install nltk</code>\n",
    "\n",
    "For Ananconda using conda:\n",
    "<code> conda install nltk</code>\n",
    "\n",
    "#### NLTK Data Download\n",
    "NLTK comes with many corpora, toy grammars, trained models, etc. (https://www.nltk.org/nltk_data/)\n",
    "After installing NLTK, we should install data using NLTK's data downloader:\n",
    "\n",
    "<code>\n",
    "import nltk\n",
    "nltk.download()\n",
    "</code>\n",
    "\n",
    "A new window should open, showing the NLTK Downloader. You can choose which corpora you wish to download. You can also download all of them.\n",
    "\n",
    "### Language processing tasks and corresponding NLTK modules with examples of functionality.\n",
    "\n",
    "|Language processing task\t|NLTK modules\t|Functionality|\n",
    "|-|-|-|\n",
    "|Accessing corpora|\tcorpus|\tstandardized interfaces to corpora and lexicons|\n",
    "|String processing|\ttokenize, stem\t| tokenizers, sentence tokenizers, stemmers|\n",
    "|Collocation discovery|\tcollocations|\tt-test, chi-squared, point-wise mutual information|\n",
    "|Part-of-speech tagging|\ttag\t| n-gram, backoff, Brill, HMM, TnT|\n",
    "|Machine learning|\tclassify, cluster, tbl\t|decision tree, maximum entropy, naive Bayes, EM, k-means|\n",
    "|Chunking|\tchunk|\tregular expression, n-gram, named-entity|\n",
    "|Parsing|\tparse, ccg|\tchart, feature-based, unification, probabilistic, dependency|\n",
    "|Semantic interpretation|\tsem, inference|\tlambda calculus, first-order logic, model checking|\n",
    "|Evaluation metrics|\tmetrics\t|precision, recall, agreement coefficients|\n",
    "|Probability and estimation|\tprobability\t|frequency distributions, smoothed probability distributions|\n",
    "|Applications|\tapp, chat\t|graphical concordancer, parsers, WordNet browser, chatbots|\n",
    "|Linguistic fieldwork|\ttoolbox|\tmanipulate data in SIL Toolbox format|\n",
    "\n",
    "We will see a demonstration of following functionality of NLTK here:\n",
    "1. Accessing Corpora\n",
    "2. Tokenization\n",
    "3. Stemming \n",
    "4. Lemmatization\n",
    "5. POS Tagging\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a55ccc0",
   "metadata": {},
   "source": [
    "### NLTK includes a diverse set of corpora which can be read using the nltk.corpus package. Each corpus is accessed by means of a “corpus reader” object from nltk.corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "79e03211",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ca01',\n",
       " 'ca02',\n",
       " 'ca03',\n",
       " 'ca04',\n",
       " 'ca05',\n",
       " 'ca06',\n",
       " 'ca07',\n",
       " 'ca08',\n",
       " 'ca09',\n",
       " 'ca10',\n",
       " 'ca11',\n",
       " 'ca12',\n",
       " 'ca13',\n",
       " 'ca14',\n",
       " 'ca15',\n",
       " 'ca16',\n",
       " 'ca17',\n",
       " 'ca18',\n",
       " 'ca19',\n",
       " 'ca20',\n",
       " 'ca21',\n",
       " 'ca22',\n",
       " 'ca23',\n",
       " 'ca24',\n",
       " 'ca25',\n",
       " 'ca26',\n",
       " 'ca27',\n",
       " 'ca28',\n",
       " 'ca29',\n",
       " 'ca30',\n",
       " 'ca31',\n",
       " 'ca32',\n",
       " 'ca33',\n",
       " 'ca34',\n",
       " 'ca35',\n",
       " 'ca36',\n",
       " 'ca37',\n",
       " 'ca38',\n",
       " 'ca39',\n",
       " 'ca40',\n",
       " 'ca41',\n",
       " 'ca42',\n",
       " 'ca43',\n",
       " 'ca44',\n",
       " 'cb01',\n",
       " 'cb02',\n",
       " 'cb03',\n",
       " 'cb04',\n",
       " 'cb05',\n",
       " 'cb06',\n",
       " 'cb07',\n",
       " 'cb08',\n",
       " 'cb09',\n",
       " 'cb10',\n",
       " 'cb11',\n",
       " 'cb12',\n",
       " 'cb13',\n",
       " 'cb14',\n",
       " 'cb15',\n",
       " 'cb16',\n",
       " 'cb17',\n",
       " 'cb18',\n",
       " 'cb19',\n",
       " 'cb20',\n",
       " 'cb21',\n",
       " 'cb22',\n",
       " 'cb23',\n",
       " 'cb24',\n",
       " 'cb25',\n",
       " 'cb26',\n",
       " 'cb27',\n",
       " 'cc01',\n",
       " 'cc02',\n",
       " 'cc03',\n",
       " 'cc04',\n",
       " 'cc05',\n",
       " 'cc06',\n",
       " 'cc07',\n",
       " 'cc08',\n",
       " 'cc09',\n",
       " 'cc10',\n",
       " 'cc11',\n",
       " 'cc12',\n",
       " 'cc13',\n",
       " 'cc14',\n",
       " 'cc15',\n",
       " 'cc16',\n",
       " 'cc17',\n",
       " 'cd01',\n",
       " 'cd02',\n",
       " 'cd03',\n",
       " 'cd04',\n",
       " 'cd05',\n",
       " 'cd06',\n",
       " 'cd07',\n",
       " 'cd08',\n",
       " 'cd09',\n",
       " 'cd10',\n",
       " 'cd11',\n",
       " 'cd12',\n",
       " 'cd13',\n",
       " 'cd14',\n",
       " 'cd15',\n",
       " 'cd16',\n",
       " 'cd17',\n",
       " 'ce01',\n",
       " 'ce02',\n",
       " 'ce03',\n",
       " 'ce04',\n",
       " 'ce05',\n",
       " 'ce06',\n",
       " 'ce07',\n",
       " 'ce08',\n",
       " 'ce09',\n",
       " 'ce10',\n",
       " 'ce11',\n",
       " 'ce12',\n",
       " 'ce13',\n",
       " 'ce14',\n",
       " 'ce15',\n",
       " 'ce16',\n",
       " 'ce17',\n",
       " 'ce18',\n",
       " 'ce19',\n",
       " 'ce20',\n",
       " 'ce21',\n",
       " 'ce22',\n",
       " 'ce23',\n",
       " 'ce24',\n",
       " 'ce25',\n",
       " 'ce26',\n",
       " 'ce27',\n",
       " 'ce28',\n",
       " 'ce29',\n",
       " 'ce30',\n",
       " 'ce31',\n",
       " 'ce32',\n",
       " 'ce33',\n",
       " 'ce34',\n",
       " 'ce35',\n",
       " 'ce36',\n",
       " 'cf01',\n",
       " 'cf02',\n",
       " 'cf03',\n",
       " 'cf04',\n",
       " 'cf05',\n",
       " 'cf06',\n",
       " 'cf07',\n",
       " 'cf08',\n",
       " 'cf09',\n",
       " 'cf10',\n",
       " 'cf11',\n",
       " 'cf12',\n",
       " 'cf13',\n",
       " 'cf14',\n",
       " 'cf15',\n",
       " 'cf16',\n",
       " 'cf17',\n",
       " 'cf18',\n",
       " 'cf19',\n",
       " 'cf20',\n",
       " 'cf21',\n",
       " 'cf22',\n",
       " 'cf23',\n",
       " 'cf24',\n",
       " 'cf25',\n",
       " 'cf26',\n",
       " 'cf27',\n",
       " 'cf28',\n",
       " 'cf29',\n",
       " 'cf30',\n",
       " 'cf31',\n",
       " 'cf32',\n",
       " 'cf33',\n",
       " 'cf34',\n",
       " 'cf35',\n",
       " 'cf36',\n",
       " 'cf37',\n",
       " 'cf38',\n",
       " 'cf39',\n",
       " 'cf40',\n",
       " 'cf41',\n",
       " 'cf42',\n",
       " 'cf43',\n",
       " 'cf44',\n",
       " 'cf45',\n",
       " 'cf46',\n",
       " 'cf47',\n",
       " 'cf48',\n",
       " 'cg01',\n",
       " 'cg02',\n",
       " 'cg03',\n",
       " 'cg04',\n",
       " 'cg05',\n",
       " 'cg06',\n",
       " 'cg07',\n",
       " 'cg08',\n",
       " 'cg09',\n",
       " 'cg10',\n",
       " 'cg11',\n",
       " 'cg12',\n",
       " 'cg13',\n",
       " 'cg14',\n",
       " 'cg15',\n",
       " 'cg16',\n",
       " 'cg17',\n",
       " 'cg18',\n",
       " 'cg19',\n",
       " 'cg20',\n",
       " 'cg21',\n",
       " 'cg22',\n",
       " 'cg23',\n",
       " 'cg24',\n",
       " 'cg25',\n",
       " 'cg26',\n",
       " 'cg27',\n",
       " 'cg28',\n",
       " 'cg29',\n",
       " 'cg30',\n",
       " 'cg31',\n",
       " 'cg32',\n",
       " 'cg33',\n",
       " 'cg34',\n",
       " 'cg35',\n",
       " 'cg36',\n",
       " 'cg37',\n",
       " 'cg38',\n",
       " 'cg39',\n",
       " 'cg40',\n",
       " 'cg41',\n",
       " 'cg42',\n",
       " 'cg43',\n",
       " 'cg44',\n",
       " 'cg45',\n",
       " 'cg46',\n",
       " 'cg47',\n",
       " 'cg48',\n",
       " 'cg49',\n",
       " 'cg50',\n",
       " 'cg51',\n",
       " 'cg52',\n",
       " 'cg53',\n",
       " 'cg54',\n",
       " 'cg55',\n",
       " 'cg56',\n",
       " 'cg57',\n",
       " 'cg58',\n",
       " 'cg59',\n",
       " 'cg60',\n",
       " 'cg61',\n",
       " 'cg62',\n",
       " 'cg63',\n",
       " 'cg64',\n",
       " 'cg65',\n",
       " 'cg66',\n",
       " 'cg67',\n",
       " 'cg68',\n",
       " 'cg69',\n",
       " 'cg70',\n",
       " 'cg71',\n",
       " 'cg72',\n",
       " 'cg73',\n",
       " 'cg74',\n",
       " 'cg75',\n",
       " 'ch01',\n",
       " 'ch02',\n",
       " 'ch03',\n",
       " 'ch04',\n",
       " 'ch05',\n",
       " 'ch06',\n",
       " 'ch07',\n",
       " 'ch08',\n",
       " 'ch09',\n",
       " 'ch10',\n",
       " 'ch11',\n",
       " 'ch12',\n",
       " 'ch13',\n",
       " 'ch14',\n",
       " 'ch15',\n",
       " 'ch16',\n",
       " 'ch17',\n",
       " 'ch18',\n",
       " 'ch19',\n",
       " 'ch20',\n",
       " 'ch21',\n",
       " 'ch22',\n",
       " 'ch23',\n",
       " 'ch24',\n",
       " 'ch25',\n",
       " 'ch26',\n",
       " 'ch27',\n",
       " 'ch28',\n",
       " 'ch29',\n",
       " 'ch30',\n",
       " 'cj01',\n",
       " 'cj02',\n",
       " 'cj03',\n",
       " 'cj04',\n",
       " 'cj05',\n",
       " 'cj06',\n",
       " 'cj07',\n",
       " 'cj08',\n",
       " 'cj09',\n",
       " 'cj10',\n",
       " 'cj11',\n",
       " 'cj12',\n",
       " 'cj13',\n",
       " 'cj14',\n",
       " 'cj15',\n",
       " 'cj16',\n",
       " 'cj17',\n",
       " 'cj18',\n",
       " 'cj19',\n",
       " 'cj20',\n",
       " 'cj21',\n",
       " 'cj22',\n",
       " 'cj23',\n",
       " 'cj24',\n",
       " 'cj25',\n",
       " 'cj26',\n",
       " 'cj27',\n",
       " 'cj28',\n",
       " 'cj29',\n",
       " 'cj30',\n",
       " 'cj31',\n",
       " 'cj32',\n",
       " 'cj33',\n",
       " 'cj34',\n",
       " 'cj35',\n",
       " 'cj36',\n",
       " 'cj37',\n",
       " 'cj38',\n",
       " 'cj39',\n",
       " 'cj40',\n",
       " 'cj41',\n",
       " 'cj42',\n",
       " 'cj43',\n",
       " 'cj44',\n",
       " 'cj45',\n",
       " 'cj46',\n",
       " 'cj47',\n",
       " 'cj48',\n",
       " 'cj49',\n",
       " 'cj50',\n",
       " 'cj51',\n",
       " 'cj52',\n",
       " 'cj53',\n",
       " 'cj54',\n",
       " 'cj55',\n",
       " 'cj56',\n",
       " 'cj57',\n",
       " 'cj58',\n",
       " 'cj59',\n",
       " 'cj60',\n",
       " 'cj61',\n",
       " 'cj62',\n",
       " 'cj63',\n",
       " 'cj64',\n",
       " 'cj65',\n",
       " 'cj66',\n",
       " 'cj67',\n",
       " 'cj68',\n",
       " 'cj69',\n",
       " 'cj70',\n",
       " 'cj71',\n",
       " 'cj72',\n",
       " 'cj73',\n",
       " 'cj74',\n",
       " 'cj75',\n",
       " 'cj76',\n",
       " 'cj77',\n",
       " 'cj78',\n",
       " 'cj79',\n",
       " 'cj80',\n",
       " 'ck01',\n",
       " 'ck02',\n",
       " 'ck03',\n",
       " 'ck04',\n",
       " 'ck05',\n",
       " 'ck06',\n",
       " 'ck07',\n",
       " 'ck08',\n",
       " 'ck09',\n",
       " 'ck10',\n",
       " 'ck11',\n",
       " 'ck12',\n",
       " 'ck13',\n",
       " 'ck14',\n",
       " 'ck15',\n",
       " 'ck16',\n",
       " 'ck17',\n",
       " 'ck18',\n",
       " 'ck19',\n",
       " 'ck20',\n",
       " 'ck21',\n",
       " 'ck22',\n",
       " 'ck23',\n",
       " 'ck24',\n",
       " 'ck25',\n",
       " 'ck26',\n",
       " 'ck27',\n",
       " 'ck28',\n",
       " 'ck29',\n",
       " 'cl01',\n",
       " 'cl02',\n",
       " 'cl03',\n",
       " 'cl04',\n",
       " 'cl05',\n",
       " 'cl06',\n",
       " 'cl07',\n",
       " 'cl08',\n",
       " 'cl09',\n",
       " 'cl10',\n",
       " 'cl11',\n",
       " 'cl12',\n",
       " 'cl13',\n",
       " 'cl14',\n",
       " 'cl15',\n",
       " 'cl16',\n",
       " 'cl17',\n",
       " 'cl18',\n",
       " 'cl19',\n",
       " 'cl20',\n",
       " 'cl21',\n",
       " 'cl22',\n",
       " 'cl23',\n",
       " 'cl24',\n",
       " 'cm01',\n",
       " 'cm02',\n",
       " 'cm03',\n",
       " 'cm04',\n",
       " 'cm05',\n",
       " 'cm06',\n",
       " 'cn01',\n",
       " 'cn02',\n",
       " 'cn03',\n",
       " 'cn04',\n",
       " 'cn05',\n",
       " 'cn06',\n",
       " 'cn07',\n",
       " 'cn08',\n",
       " 'cn09',\n",
       " 'cn10',\n",
       " 'cn11',\n",
       " 'cn12',\n",
       " 'cn13',\n",
       " 'cn14',\n",
       " 'cn15',\n",
       " 'cn16',\n",
       " 'cn17',\n",
       " 'cn18',\n",
       " 'cn19',\n",
       " 'cn20',\n",
       " 'cn21',\n",
       " 'cn22',\n",
       " 'cn23',\n",
       " 'cn24',\n",
       " 'cn25',\n",
       " 'cn26',\n",
       " 'cn27',\n",
       " 'cn28',\n",
       " 'cn29',\n",
       " 'cp01',\n",
       " 'cp02',\n",
       " 'cp03',\n",
       " 'cp04',\n",
       " 'cp05',\n",
       " 'cp06',\n",
       " 'cp07',\n",
       " 'cp08',\n",
       " 'cp09',\n",
       " 'cp10',\n",
       " 'cp11',\n",
       " 'cp12',\n",
       " 'cp13',\n",
       " 'cp14',\n",
       " 'cp15',\n",
       " 'cp16',\n",
       " 'cp17',\n",
       " 'cp18',\n",
       " 'cp19',\n",
       " 'cp20',\n",
       " 'cp21',\n",
       " 'cp22',\n",
       " 'cp23',\n",
       " 'cp24',\n",
       " 'cp25',\n",
       " 'cp26',\n",
       " 'cp27',\n",
       " 'cp28',\n",
       " 'cp29',\n",
       " 'cr01',\n",
       " 'cr02',\n",
       " 'cr03',\n",
       " 'cr04',\n",
       " 'cr05',\n",
       " 'cr06',\n",
       " 'cr07',\n",
       " 'cr08',\n",
       " 'cr09']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Builtin corpora in NLTK (https://www.nltk.org/howto/corpus.html)\n",
    "import nltk.corpus\n",
    "from nltk.corpus import brown\n",
    "brown.fileids()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48552856",
   "metadata": {},
   "source": [
    "Most corpora consist of a set of files, each containing a document (or other pieces of text). A list of identifiers for these files is accessed via the fileids() method of the corpus reader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e2fd2f77",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['wsj_0001.mrg',\n",
       " 'wsj_0002.mrg',\n",
       " 'wsj_0003.mrg',\n",
       " 'wsj_0004.mrg',\n",
       " 'wsj_0005.mrg',\n",
       " 'wsj_0006.mrg',\n",
       " 'wsj_0007.mrg',\n",
       " 'wsj_0008.mrg',\n",
       " 'wsj_0009.mrg',\n",
       " 'wsj_0010.mrg',\n",
       " 'wsj_0011.mrg',\n",
       " 'wsj_0012.mrg',\n",
       " 'wsj_0013.mrg',\n",
       " 'wsj_0014.mrg',\n",
       " 'wsj_0015.mrg',\n",
       " 'wsj_0016.mrg',\n",
       " 'wsj_0017.mrg',\n",
       " 'wsj_0018.mrg',\n",
       " 'wsj_0019.mrg',\n",
       " 'wsj_0020.mrg',\n",
       " 'wsj_0021.mrg',\n",
       " 'wsj_0022.mrg',\n",
       " 'wsj_0023.mrg',\n",
       " 'wsj_0024.mrg',\n",
       " 'wsj_0025.mrg',\n",
       " 'wsj_0026.mrg',\n",
       " 'wsj_0027.mrg',\n",
       " 'wsj_0028.mrg',\n",
       " 'wsj_0029.mrg',\n",
       " 'wsj_0030.mrg',\n",
       " 'wsj_0031.mrg',\n",
       " 'wsj_0032.mrg',\n",
       " 'wsj_0033.mrg',\n",
       " 'wsj_0034.mrg',\n",
       " 'wsj_0035.mrg',\n",
       " 'wsj_0036.mrg',\n",
       " 'wsj_0037.mrg',\n",
       " 'wsj_0038.mrg',\n",
       " 'wsj_0039.mrg',\n",
       " 'wsj_0040.mrg',\n",
       " 'wsj_0041.mrg',\n",
       " 'wsj_0042.mrg',\n",
       " 'wsj_0043.mrg',\n",
       " 'wsj_0044.mrg',\n",
       " 'wsj_0045.mrg',\n",
       " 'wsj_0046.mrg',\n",
       " 'wsj_0047.mrg',\n",
       " 'wsj_0048.mrg',\n",
       " 'wsj_0049.mrg',\n",
       " 'wsj_0050.mrg',\n",
       " 'wsj_0051.mrg',\n",
       " 'wsj_0052.mrg',\n",
       " 'wsj_0053.mrg',\n",
       " 'wsj_0054.mrg',\n",
       " 'wsj_0055.mrg',\n",
       " 'wsj_0056.mrg',\n",
       " 'wsj_0057.mrg',\n",
       " 'wsj_0058.mrg',\n",
       " 'wsj_0059.mrg',\n",
       " 'wsj_0060.mrg',\n",
       " 'wsj_0061.mrg',\n",
       " 'wsj_0062.mrg',\n",
       " 'wsj_0063.mrg',\n",
       " 'wsj_0064.mrg',\n",
       " 'wsj_0065.mrg',\n",
       " 'wsj_0066.mrg',\n",
       " 'wsj_0067.mrg',\n",
       " 'wsj_0068.mrg',\n",
       " 'wsj_0069.mrg',\n",
       " 'wsj_0070.mrg',\n",
       " 'wsj_0071.mrg',\n",
       " 'wsj_0072.mrg',\n",
       " 'wsj_0073.mrg',\n",
       " 'wsj_0074.mrg',\n",
       " 'wsj_0075.mrg',\n",
       " 'wsj_0076.mrg',\n",
       " 'wsj_0077.mrg',\n",
       " 'wsj_0078.mrg',\n",
       " 'wsj_0079.mrg',\n",
       " 'wsj_0080.mrg',\n",
       " 'wsj_0081.mrg',\n",
       " 'wsj_0082.mrg',\n",
       " 'wsj_0083.mrg',\n",
       " 'wsj_0084.mrg',\n",
       " 'wsj_0085.mrg',\n",
       " 'wsj_0086.mrg',\n",
       " 'wsj_0087.mrg',\n",
       " 'wsj_0088.mrg',\n",
       " 'wsj_0089.mrg',\n",
       " 'wsj_0090.mrg',\n",
       " 'wsj_0091.mrg',\n",
       " 'wsj_0092.mrg',\n",
       " 'wsj_0093.mrg',\n",
       " 'wsj_0094.mrg',\n",
       " 'wsj_0095.mrg',\n",
       " 'wsj_0096.mrg',\n",
       " 'wsj_0097.mrg',\n",
       " 'wsj_0098.mrg',\n",
       " 'wsj_0099.mrg',\n",
       " 'wsj_0100.mrg',\n",
       " 'wsj_0101.mrg',\n",
       " 'wsj_0102.mrg',\n",
       " 'wsj_0103.mrg',\n",
       " 'wsj_0104.mrg',\n",
       " 'wsj_0105.mrg',\n",
       " 'wsj_0106.mrg',\n",
       " 'wsj_0107.mrg',\n",
       " 'wsj_0108.mrg',\n",
       " 'wsj_0109.mrg',\n",
       " 'wsj_0110.mrg',\n",
       " 'wsj_0111.mrg',\n",
       " 'wsj_0112.mrg',\n",
       " 'wsj_0113.mrg',\n",
       " 'wsj_0114.mrg',\n",
       " 'wsj_0115.mrg',\n",
       " 'wsj_0116.mrg',\n",
       " 'wsj_0117.mrg',\n",
       " 'wsj_0118.mrg',\n",
       " 'wsj_0119.mrg',\n",
       " 'wsj_0120.mrg',\n",
       " 'wsj_0121.mrg',\n",
       " 'wsj_0122.mrg',\n",
       " 'wsj_0123.mrg',\n",
       " 'wsj_0124.mrg',\n",
       " 'wsj_0125.mrg',\n",
       " 'wsj_0126.mrg',\n",
       " 'wsj_0127.mrg',\n",
       " 'wsj_0128.mrg',\n",
       " 'wsj_0129.mrg',\n",
       " 'wsj_0130.mrg',\n",
       " 'wsj_0131.mrg',\n",
       " 'wsj_0132.mrg',\n",
       " 'wsj_0133.mrg',\n",
       " 'wsj_0134.mrg',\n",
       " 'wsj_0135.mrg',\n",
       " 'wsj_0136.mrg',\n",
       " 'wsj_0137.mrg',\n",
       " 'wsj_0138.mrg',\n",
       " 'wsj_0139.mrg',\n",
       " 'wsj_0140.mrg',\n",
       " 'wsj_0141.mrg',\n",
       " 'wsj_0142.mrg',\n",
       " 'wsj_0143.mrg',\n",
       " 'wsj_0144.mrg',\n",
       " 'wsj_0145.mrg',\n",
       " 'wsj_0146.mrg',\n",
       " 'wsj_0147.mrg',\n",
       " 'wsj_0148.mrg',\n",
       " 'wsj_0149.mrg',\n",
       " 'wsj_0150.mrg',\n",
       " 'wsj_0151.mrg',\n",
       " 'wsj_0152.mrg',\n",
       " 'wsj_0153.mrg',\n",
       " 'wsj_0154.mrg',\n",
       " 'wsj_0155.mrg',\n",
       " 'wsj_0156.mrg',\n",
       " 'wsj_0157.mrg',\n",
       " 'wsj_0158.mrg',\n",
       " 'wsj_0159.mrg',\n",
       " 'wsj_0160.mrg',\n",
       " 'wsj_0161.mrg',\n",
       " 'wsj_0162.mrg',\n",
       " 'wsj_0163.mrg',\n",
       " 'wsj_0164.mrg',\n",
       " 'wsj_0165.mrg',\n",
       " 'wsj_0166.mrg',\n",
       " 'wsj_0167.mrg',\n",
       " 'wsj_0168.mrg',\n",
       " 'wsj_0169.mrg',\n",
       " 'wsj_0170.mrg',\n",
       " 'wsj_0171.mrg',\n",
       " 'wsj_0172.mrg',\n",
       " 'wsj_0173.mrg',\n",
       " 'wsj_0174.mrg',\n",
       " 'wsj_0175.mrg',\n",
       " 'wsj_0176.mrg',\n",
       " 'wsj_0177.mrg',\n",
       " 'wsj_0178.mrg',\n",
       " 'wsj_0179.mrg',\n",
       " 'wsj_0180.mrg',\n",
       " 'wsj_0181.mrg',\n",
       " 'wsj_0182.mrg',\n",
       " 'wsj_0183.mrg',\n",
       " 'wsj_0184.mrg',\n",
       " 'wsj_0185.mrg',\n",
       " 'wsj_0186.mrg',\n",
       " 'wsj_0187.mrg',\n",
       " 'wsj_0188.mrg',\n",
       " 'wsj_0189.mrg',\n",
       " 'wsj_0190.mrg',\n",
       " 'wsj_0191.mrg',\n",
       " 'wsj_0192.mrg',\n",
       " 'wsj_0193.mrg',\n",
       " 'wsj_0194.mrg',\n",
       " 'wsj_0195.mrg',\n",
       " 'wsj_0196.mrg',\n",
       " 'wsj_0197.mrg',\n",
       " 'wsj_0198.mrg',\n",
       " 'wsj_0199.mrg']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.corpus.treebank.fileids()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c08c4b32",
   "metadata": {},
   "source": [
    "Each corpus reader provides a variety of methods to read data from the corpus, depending on the format of the corpus. For example, plaintext corpora support methods to read the corpus as raw text, a list of words, a list of sentences, or a list of paragraphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "e7af1af1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Fellow-Citizens of the Senate and of the House of Representatives:\\n\\nAmong the vicissitudes incident to life no event could have filled me with greater anxieties than that of which the notification was transmitted by your order, and received on the 14th day of the present month. On the one hand, I was summoned by my Country, whose voice I can never hear but with veneration and love, from a retreat which I had chosen with the fondest predilection, and, in my flattering hopes, with an immutable decision, as the asylum of my declining years -- a retreat which was rendered every day more necessary as well as more dear to me by the addition of habit to inclination, and of frequent interruptions in my health to the gradual waste committed on it by time. On the other hand, the magnitude and difficulty of the trust to which the voice of my country called me, being sufficient to awaken in the wisest and most experienced of her citizens a distrustful scrutiny into his qualifications, could not but overwhelm with despondence one who (inheriting inferior endowments from nature and unpracticed in the duties of civil administration) ought to be peculiarly conscious of his own deficiencies. In this conflict of emotions all I dare aver is that it has been my faithful study to collect my duty from a just appreciation of every circumstance by which it might be affected. All I dare hope is that if, in executing this task, I have been too much swayed by a grateful remembrance of former instances, or by an affectionate sensibility to this transcendent proof of the confidence of my fellow citizens, and have thence too little consulted my incapacity as well as disinclination for the weighty and untried cares before me, my error will be palliated by the motives which mislead me, and its consequences be judged by my country with some share of the partiality in which they originated.\\n\\nSuch being the impressions under which I have, in obedience to the public summons, repaired to the present station, it would be peculiarly improper to omit in this first official act my fervent supplications to that Almighty Being who rules over the universe, who presides in the councils of nations, and whose providential aids can supply every human defect, that His benediction may consecrate to the liberties and happiness of the people of the United States a Government instituted by themselves for these essential purposes, and may enable every instrument employed in its administration to execute with success the functions allotted to his charge. In tendering this homage to the Great Author of every public and private good, I assure myself that it expresses your sentiments not less than my own, nor those of my fellow citizens at large less than either. No people can be bound to acknowledge and adore the Invisible Hand which conducts the affairs of men more than those of the United States. Every step by which they have advanced to the character of an independent nation seems to have been distinguished by some token of providential agency; and in the important revolution just accomplished in the system of their united government the tranquil deliberations and voluntary consent of so many distinct communities from which the event has resulted can not be compared with the means by which most governments have been established without some return of pious gratitude, along with an humble anticipation of the future blessings which the past seem to presage. These reflections, arising out of the present crisis, have forced themselves too strongly on my mind to be suppressed. You will join with me, I trust, in thinking that there are none under the influence of which the proceedings of a new and free government can more auspiciously commence.\\n\\nBy the article establishing the executive department it is made the duty of the President \"to recommend to your consideration such measures as he shall judge necessary and expedient.\" The circumstances under which I now meet you will acquit me from entering into that subject further than to refer to the great constitutional charter under which you are assembled, and which, in defining your powers, designates the objects to which your attention is to be given. It will be more consistent with those circumstances, and far more congenial with the feelings which actuate me, to substitute, in place of a recommendation of particular measures, the tribute that is due to the talents, the rectitude, and the patriotism which adorn the characters selected to devise and adopt them. In these honorable qualifications I behold the surest pledges that as on one side no local prejudices or attachments, no separate views nor party animosities, will misdirect the comprehensive and equal eye which ought to watch over this great assemblage of communities and interests, so, on another, that the foundation of our national policy will be laid in the pure and immutable principles of private morality, and the preeminence of free government be exemplified by all the attributes which can win the affections of its citizens and command the respect of the world. I dwell on this prospect with every satisfaction which an ardent love for my country can inspire, since there is no truth more thoroughly established than that there exists in the economy and course of nature an indissoluble union between virtue and happiness; between duty and advantage; between the genuine maxims of an honest and magnanimous policy and the solid rewards of public prosperity and felicity; since we ought to be no less persuaded that the propitious smiles of Heaven can never be expected on a nation that disregards the eternal rules of order and right which Heaven itself has ordained; and since the preservation of the sacred fire of liberty and the destiny of the republican model of government are justly considered, perhaps, as deeply, as finally, staked on the experiment entrusted to the hands of the American people.\\n\\nBesides the ordinary objects submitted to your care, it will remain with your judgment to decide how far an exercise of the occasional power delegated by the fifth article of the Constitution is rendered expedient at the present juncture by the nature of objections which have been urged against the system, or by the degree of inquietude which has given birth to them. Instead of undertaking particular recommendations on this subject, in which I could be guided by no lights derived from official opportunities, I shall again give way to my entire confidence in your discernment and pursuit of the public good; for I assure myself that whilst you carefully avoid every alteration which might endanger the benefits of an united and effective government, or which ought to await the future lessons of experience, a reverence for the characteristic rights of freemen and a regard for the public harmony will sufficiently influence your deliberations on the question how far the former can be impregnably fortified or the latter be safely and advantageously promoted.\\n\\nTo the foregoing observations I have one to add, which will be most properly addressed to the House of Representatives. It concerns myself, and will therefore be as brief as possible. When I was first honored with a call into the service of my country, then on the eve of an arduous struggle for its liberties, the light in which I contemplated my duty required that I should renounce every pecuniary compensation. From this resolution I have in no instance departed; and being still under the impressions which produced it, I must decline as inapplicable to myself any share in the personal emoluments which may be indispensably included in a permanent provision for the executive department, and must accordingly pray that the pecuniary estimates for the station in which I am placed may during my continuance in it be limited to such actual expenditures as the public good may be thought to require.\\n\\nHaving thus imparted to you my sentiments as they have been awakened by the occasion which brings us together, I shall take my present leave; but not without resorting once more to the benign Parent of the Human Race in humble supplication that, since He has been pleased to favor the American people with opportunities for deliberating in perfect tranquillity, and dispositions for deciding with unparalleled unanimity on a form of government for the security of their union and the advancement of their happiness, so His divine blessing may be equally conspicuous in the enlarged views, the temperate consultations, and the wise measures on which the success of this Government must depend. \\n'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import inaugural\n",
    "inaugural.raw('1789-Washington.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "dec90b15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Fellow', '-', 'Citizens', 'of', 'the', 'Senate', ...]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inaugural.words('1789-Washington.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "9f88765a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Fellow', '-', 'Citizens', 'of', 'the', 'Senate', 'and', 'of', 'the', 'House', 'of', 'Representatives', ':'], ['Among', 'the', 'vicissitudes', 'incident', 'to', 'life', 'no', 'event', 'could', 'have', 'filled', 'me', 'with', 'greater', 'anxieties', 'than', 'that', 'of', 'which', 'the', 'notification', 'was', 'transmitted', 'by', 'your', 'order', ',', 'and', 'received', 'on', 'the', '14th', 'day', 'of', 'the', 'present', 'month', '.'], ...]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inaugural.sents('1789-Washington.txt')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "926314aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[['Fellow', '-', 'Citizens', 'of', 'the', 'Senate', 'and', 'of', 'the', 'House', 'of', 'Representatives', ':']], [['Among', 'the', 'vicissitudes', 'incident', 'to', 'life', 'no', 'event', 'could', 'have', 'filled', 'me', 'with', 'greater', 'anxieties', 'than', 'that', 'of', 'which', 'the', 'notification', 'was', 'transmitted', 'by', 'your', 'order', ',', 'and', 'received', 'on', 'the', '14th', 'day', 'of', 'the', 'present', 'month', '.'], ['On', 'the', 'one', 'hand', ',', 'I', 'was', 'summoned', 'by', 'my', 'Country', ',', 'whose', 'voice', 'I', 'can', 'never', 'hear', 'but', 'with', 'veneration', 'and', 'love', ',', 'from', 'a', 'retreat', 'which', 'I', 'had', 'chosen', 'with', 'the', 'fondest', 'predilection', ',', 'and', ',', 'in', 'my', 'flattering', 'hopes', ',', 'with', 'an', 'immutable', 'decision', ',', 'as', 'the', 'asylum', 'of', 'my', 'declining', 'years', '--', 'a', 'retreat', 'which', 'was', 'rendered', 'every', 'day', 'more', 'necessary', 'as', 'well', 'as', 'more', 'dear', 'to', 'me', 'by', 'the', 'addition', 'of', 'habit', 'to', 'inclination', ',', 'and', 'of', 'frequent', 'interruptions', 'in', 'my', 'health', 'to', 'the', 'gradual', 'waste', 'committed', 'on', 'it', 'by', 'time', '.'], ['On', 'the', 'other', 'hand', ',', 'the', 'magnitude', 'and', 'difficulty', 'of', 'the', 'trust', 'to', 'which', 'the', 'voice', 'of', 'my', 'country', 'called', 'me', ',', 'being', 'sufficient', 'to', 'awaken', 'in', 'the', 'wisest', 'and', 'most', 'experienced', 'of', 'her', 'citizens', 'a', 'distrustful', 'scrutiny', 'into', 'his', 'qualifications', ',', 'could', 'not', 'but', 'overwhelm', 'with', 'despondence', 'one', 'who', '(', 'inheriting', 'inferior', 'endowments', 'from', 'nature', 'and', 'unpracticed', 'in', 'the', 'duties', 'of', 'civil', 'administration', ')', 'ought', 'to', 'be', 'peculiarly', 'conscious', 'of', 'his', 'own', 'deficiencies', '.'], ['In', 'this', 'conflict', 'of', 'emotions', 'all', 'I', 'dare', 'aver', 'is', 'that', 'it', 'has', 'been', 'my', 'faithful', 'study', 'to', 'collect', 'my', 'duty', 'from', 'a', 'just', 'appreciation', 'of', 'every', 'circumstance', 'by', 'which', 'it', 'might', 'be', 'affected', '.'], ['All', 'I', 'dare', 'hope', 'is', 'that', 'if', ',', 'in', 'executing', 'this', 'task', ',', 'I', 'have', 'been', 'too', 'much', 'swayed', 'by', 'a', 'grateful', 'remembrance', 'of', 'former', 'instances', ',', 'or', 'by', 'an', 'affectionate', 'sensibility', 'to', 'this', 'transcendent', 'proof', 'of', 'the', 'confidence', 'of', 'my', 'fellow', 'citizens', ',', 'and', 'have', 'thence', 'too', 'little', 'consulted', 'my', 'incapacity', 'as', 'well', 'as', 'disinclination', 'for', 'the', 'weighty', 'and', 'untried', 'cares', 'before', 'me', ',', 'my', 'error', 'will', 'be', 'palliated', 'by', 'the', 'motives', 'which', 'mislead', 'me', ',', 'and', 'its', 'consequences', 'be', 'judged', 'by', 'my', 'country', 'with', 'some', 'share', 'of', 'the', 'partiality', 'in', 'which', 'they', 'originated', '.']], ...]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inaugural.paras('1789-Washington.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ce862224",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['মহিষের', 'সন্তান', ':', 'তোড়া', 'উপজাতি', '৷', ...]\n",
      "[('মহিষের', 'NN'), ('সন্তান', 'NN'), (':', 'SYM'), ...]\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import indian\n",
    "print(indian.words()) \n",
    "print(indian.tagged_words()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8a5e4b33",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[('The', 'DET'), ('Fulton', 'NOUN'), ('County', 'NOUN'), ('Grand', 'ADJ'), ('Jury', 'NOUN'), ('said', 'VERB'), ('Friday', 'NOUN'), ('an', 'DET'), ('investigation', 'NOUN'), ('of', 'ADP'), (\"Atlanta's\", 'NOUN'), ('recent', 'ADJ'), ('primary', 'NOUN'), ('election', 'NOUN'), ('produced', 'VERB'), ('``', '.'), ('no', 'DET'), ('evidence', 'NOUN'), (\"''\", '.'), ('that', 'ADP'), ('any', 'DET'), ('irregularities', 'NOUN'), ('took', 'VERB'), ('place', 'NOUN'), ('.', '.')], [('The', 'DET'), ('jury', 'NOUN'), ('further', 'ADV'), ('said', 'VERB'), ('in', 'ADP'), ('term-end', 'NOUN'), ('presentments', 'NOUN'), ('that', 'ADP'), ('the', 'DET'), ('City', 'NOUN'), ('Executive', 'ADJ'), ('Committee', 'NOUN'), (',', '.'), ('which', 'DET'), ('had', 'VERB'), ('over-all', 'ADJ'), ('charge', 'NOUN'), ('of', 'ADP'), ('the', 'DET'), ('election', 'NOUN'), (',', '.'), ('``', '.'), ('deserves', 'VERB'), ('the', 'DET'), ('praise', 'NOUN'), ('and', 'CONJ'), ('thanks', 'NOUN'), ('of', 'ADP'), ('the', 'DET'), ('City', 'NOUN'), ('of', 'ADP'), ('Atlanta', 'NOUN'), (\"''\", '.'), ('for', 'ADP'), ('the', 'DET'), ('manner', 'NOUN'), ('in', 'ADP'), ('which', 'DET'), ('the', 'DET'), ('election', 'NOUN'), ('was', 'VERB'), ('conducted', 'VERB'), ('.', '.')], ...]\n"
     ]
    }
   ],
   "source": [
    "print(brown.tagged_sents(tagset='universal'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39c2e48c",
   "metadata": {},
   "source": [
    "#### Word Lists and Lexicons\n",
    "The NLTK data package also includes a number of lexicons and word lists. These are accessed just like text corpora. The following examples illustrate the use of the wordlist corpora:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e24dab0c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['en', 'en-basic']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import words\n",
    "words.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "24dea83c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['A', 'a', 'aa', 'aal', 'aalii', 'aam', 'Aani', 'aardvark', 'aardwolf', 'Aaron', 'Aaronic', 'Aaronical', 'Aaronite', 'Aaronitic', 'Aaru', 'Ab', 'aba', 'Ababdeh', 'Ababua', 'abac', 'abaca', 'abacate', 'abacay', 'abacinate', 'abacination', 'abaciscus', 'abacist', 'aback', 'abactinal', 'abactinally', 'abaction', 'abactor', 'abaculus', 'abacus', 'Abadite', 'abaff', 'abaft', 'abaisance', 'abaiser', 'abaissed', 'abalienate', 'abalienation', 'abalone', 'Abama', 'abampere', 'abandon', 'abandonable', 'abandoned', 'abandonedly', 'abandonee', 'abandoner', 'abandonment', 'Abanic', 'Abantes', 'abaptiston', 'Abarambo', 'Abaris', 'abarthrosis', 'abarticular', 'abarticulation', 'abas', 'abase', 'abased', 'abasedly', 'abasedness', 'abasement', 'abaser', 'Abasgi', 'abash', 'abashed', 'abashedly', 'abashedness', 'abashless', 'abashlessly', 'abashment', 'abasia', 'abasic', 'abask', 'Abassin', 'abastardize', 'abatable', 'abate', 'abatement', 'abater', 'abatis', 'abatised', 'abaton', 'abator', 'abattoir', 'Abatua', 'abature', 'abave', 'abaxial', 'abaxile', 'abaze', 'abb', 'Abba', 'abbacomes', 'abbacy', 'Abbadide']\n",
      "A\n",
      "a\n",
      "aa\n",
      "aal\n",
      "aalii\n",
      "aam\n",
      "Aani\n",
      "aardvark\n",
      "aardwolf\n",
      "Aaron\n",
      "Aaronic\n",
      "Aaronical\n",
      "Aaronite\n",
      "Aaronitic\n",
      "Aaru\n",
      "Ab\n",
      "aba\n",
      "A\n"
     ]
    }
   ],
   "source": [
    "print(words.words()[:100])\n",
    "print(words.raw()[:100])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "b5c72dba",
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Stopwords : Words that add little or nothing to the meaning of the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "65986b02",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['arabic',\n",
       " 'azerbaijani',\n",
       " 'basque',\n",
       " 'bengali',\n",
       " 'catalan',\n",
       " 'chinese',\n",
       " 'danish',\n",
       " 'dutch',\n",
       " 'english',\n",
       " 'finnish',\n",
       " 'french',\n",
       " 'german',\n",
       " 'greek',\n",
       " 'hebrew',\n",
       " 'hinglish',\n",
       " 'hungarian',\n",
       " 'indonesian',\n",
       " 'italian',\n",
       " 'kazakh',\n",
       " 'nepali',\n",
       " 'norwegian',\n",
       " 'portuguese',\n",
       " 'romanian',\n",
       " 'russian',\n",
       " 'slovene',\n",
       " 'spanish',\n",
       " 'swedish',\n",
       " 'tajik',\n",
       " 'turkish']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords \n",
    "stopwords.fileids() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7268df1a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['female.txt', 'male.txt']"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import names\n",
    "names.fileids()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a8db4a17",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Aamir',\n",
       " 'Aaron',\n",
       " 'Abbey',\n",
       " 'Abbie',\n",
       " 'Abbot',\n",
       " 'Abbott',\n",
       " 'Abby',\n",
       " 'Abdel',\n",
       " 'Abdul',\n",
       " 'Abdulkarim',\n",
       " 'Abdullah',\n",
       " 'Abe',\n",
       " 'Abel',\n",
       " 'Abelard',\n",
       " 'Abner',\n",
       " 'Abraham',\n",
       " 'Abram',\n",
       " 'Ace',\n",
       " 'Adair',\n",
       " 'Adam',\n",
       " 'Adams',\n",
       " 'Addie',\n",
       " 'Adger',\n",
       " 'Aditya',\n",
       " 'Adlai',\n",
       " 'Adnan',\n",
       " 'Adolf',\n",
       " 'Adolfo',\n",
       " 'Adolph',\n",
       " 'Adolphe',\n",
       " 'Adolpho',\n",
       " 'Adolphus',\n",
       " 'Adrian',\n",
       " 'Adrick',\n",
       " 'Adrien',\n",
       " 'Agamemnon',\n",
       " 'Aguinaldo',\n",
       " 'Aguste',\n",
       " 'Agustin',\n",
       " 'Aharon',\n",
       " 'Ahmad',\n",
       " 'Ahmed',\n",
       " 'Ahmet',\n",
       " 'Ajai',\n",
       " 'Ajay',\n",
       " 'Al',\n",
       " 'Alaa',\n",
       " 'Alain',\n",
       " 'Alan',\n",
       " 'Alasdair',\n",
       " 'Alastair',\n",
       " 'Albatros',\n",
       " 'Albert',\n",
       " 'Alberto',\n",
       " 'Albrecht',\n",
       " 'Alden',\n",
       " 'Aldis',\n",
       " 'Aldo',\n",
       " 'Aldric',\n",
       " 'Aldrich',\n",
       " 'Aldus',\n",
       " 'Aldwin',\n",
       " 'Alec',\n",
       " 'Aleck',\n",
       " 'Alejandro',\n",
       " 'Aleks',\n",
       " 'Aleksandrs',\n",
       " 'Alessandro',\n",
       " 'Alex',\n",
       " 'Alexander',\n",
       " 'Alexei',\n",
       " 'Alexis',\n",
       " 'Alf',\n",
       " 'Alfie',\n",
       " 'Alfonse',\n",
       " 'Alfonso',\n",
       " 'Alfonzo',\n",
       " 'Alford',\n",
       " 'Alfred',\n",
       " 'Alfredo',\n",
       " 'Algernon',\n",
       " 'Ali',\n",
       " 'Alic',\n",
       " 'Alister',\n",
       " 'Alix',\n",
       " 'Allah',\n",
       " 'Allan',\n",
       " 'Allen',\n",
       " 'Alley',\n",
       " 'Allie',\n",
       " 'Allin',\n",
       " 'Allyn',\n",
       " 'Alonso',\n",
       " 'Alonzo',\n",
       " 'Aloysius',\n",
       " 'Alphonse',\n",
       " 'Alphonso',\n",
       " 'Alston',\n",
       " 'Alton',\n",
       " 'Alvin',\n",
       " 'Alwin',\n",
       " 'Amadeus',\n",
       " 'Ambros',\n",
       " 'Ambrose',\n",
       " 'Ambrosi',\n",
       " 'Ambrosio',\n",
       " 'Ambrosius',\n",
       " 'Amery',\n",
       " 'Amory',\n",
       " 'Amos',\n",
       " 'Anatol',\n",
       " 'Anatole',\n",
       " 'Anatollo',\n",
       " 'Anatoly',\n",
       " 'Anders',\n",
       " 'Andie',\n",
       " 'Andonis',\n",
       " 'Andre',\n",
       " 'Andrea',\n",
       " 'Andreas',\n",
       " 'Andrej',\n",
       " 'Andres',\n",
       " 'Andrew',\n",
       " 'Andrey',\n",
       " 'Andri',\n",
       " 'Andros',\n",
       " 'Andrus',\n",
       " 'Andrzej',\n",
       " 'Andy',\n",
       " 'Angel',\n",
       " 'Angelico',\n",
       " 'Angelo',\n",
       " 'Angie',\n",
       " 'Angus',\n",
       " 'Ansel',\n",
       " 'Ansell',\n",
       " 'Anselm',\n",
       " 'Anson',\n",
       " 'Anthony',\n",
       " 'Antin',\n",
       " 'Antoine',\n",
       " 'Anton',\n",
       " 'Antone',\n",
       " 'Antoni',\n",
       " 'Antonin',\n",
       " 'Antonino',\n",
       " 'Antonio',\n",
       " 'Antonius',\n",
       " 'Antony',\n",
       " 'Anurag',\n",
       " 'Apollo',\n",
       " 'Apostolos',\n",
       " 'Aram',\n",
       " 'Archibald',\n",
       " 'Archibold',\n",
       " 'Archie',\n",
       " 'Archon',\n",
       " 'Archy',\n",
       " 'Arel',\n",
       " 'Ari',\n",
       " 'Arie',\n",
       " 'Ariel',\n",
       " 'Aristotle',\n",
       " 'Arlo',\n",
       " 'Armand',\n",
       " 'Armando',\n",
       " 'Armond',\n",
       " 'Armstrong',\n",
       " 'Arne',\n",
       " 'Arnie',\n",
       " 'Arnold',\n",
       " 'Arnoldo',\n",
       " 'Aron',\n",
       " 'Arron',\n",
       " 'Art',\n",
       " 'Arther',\n",
       " 'Arthur',\n",
       " 'Artie',\n",
       " 'Artur',\n",
       " 'Arturo',\n",
       " 'Arvie',\n",
       " 'Arvin',\n",
       " 'Arvind',\n",
       " 'Arvy',\n",
       " 'Ash',\n",
       " 'Ashby',\n",
       " 'Ashish',\n",
       " 'Ashley',\n",
       " 'Ashton',\n",
       " 'Aub',\n",
       " 'Aube',\n",
       " 'Aubert',\n",
       " 'Aubrey',\n",
       " 'Augie',\n",
       " 'August',\n",
       " 'Augustin',\n",
       " 'Augustine',\n",
       " 'Augusto',\n",
       " 'Augustus',\n",
       " 'Austen',\n",
       " 'Austin',\n",
       " 'Ave',\n",
       " 'Averell',\n",
       " 'Averil',\n",
       " 'Averill',\n",
       " 'Avery',\n",
       " 'Avi',\n",
       " 'Avraham',\n",
       " 'Avram',\n",
       " 'Avrom',\n",
       " 'Axel',\n",
       " 'Aylmer',\n",
       " 'Aziz',\n",
       " 'Bailey',\n",
       " 'Bailie',\n",
       " 'Baillie',\n",
       " 'Baily',\n",
       " 'Baird',\n",
       " 'Baldwin',\n",
       " 'Bancroft',\n",
       " 'Barbabas',\n",
       " 'Barclay',\n",
       " 'Bard',\n",
       " 'Barde',\n",
       " 'Barn',\n",
       " 'Barnabas',\n",
       " 'Barnabe',\n",
       " 'Barnaby',\n",
       " 'Barnard',\n",
       " 'Barnebas',\n",
       " 'Barnett',\n",
       " 'Barney',\n",
       " 'Barnie',\n",
       " 'Barny',\n",
       " 'Baron',\n",
       " 'Barr',\n",
       " 'Barret',\n",
       " 'Barrett',\n",
       " 'Barri',\n",
       " 'Barrie',\n",
       " 'Barris',\n",
       " 'Barron',\n",
       " 'Barry',\n",
       " 'Bart',\n",
       " 'Bartel',\n",
       " 'Barth',\n",
       " 'Barthel',\n",
       " 'Bartholemy',\n",
       " 'Bartholomeo',\n",
       " 'Bartholomeus',\n",
       " 'Bartholomew',\n",
       " 'Bartie',\n",
       " 'Bartlet',\n",
       " 'Bartlett',\n",
       " 'Bartolemo',\n",
       " 'Bartolomei',\n",
       " 'Bartolomeo',\n",
       " 'Barton',\n",
       " 'Barty',\n",
       " 'Bary',\n",
       " 'Basil',\n",
       " 'Batholomew',\n",
       " 'Baxter',\n",
       " 'Bay',\n",
       " 'Bayard',\n",
       " 'Beale',\n",
       " 'Bealle',\n",
       " 'Bear',\n",
       " 'Bearnard',\n",
       " 'Beau',\n",
       " 'Beaufort',\n",
       " 'Beauregard',\n",
       " 'Beck',\n",
       " 'Bela',\n",
       " 'Ben',\n",
       " 'Benedict',\n",
       " 'Bengt',\n",
       " 'Benito',\n",
       " 'Benjamen',\n",
       " 'Benjamin',\n",
       " 'Benji',\n",
       " 'Benjie',\n",
       " 'Benjy',\n",
       " 'Benn',\n",
       " 'Bennet',\n",
       " 'Bennett',\n",
       " 'Bennie',\n",
       " 'Benny',\n",
       " 'Benson',\n",
       " 'Bentley',\n",
       " 'Benton',\n",
       " 'Beowulf',\n",
       " 'Berchtold',\n",
       " 'Berk',\n",
       " 'Berke',\n",
       " 'Berkeley',\n",
       " 'Berkie',\n",
       " 'Berkley',\n",
       " 'Bernard',\n",
       " 'Bernardo',\n",
       " 'Bernd',\n",
       " 'Bernhard',\n",
       " 'Bernie',\n",
       " 'Bert',\n",
       " 'Bertie',\n",
       " 'Bertram',\n",
       " 'Bertrand',\n",
       " 'Bharat',\n",
       " 'Biff',\n",
       " 'Bill',\n",
       " 'Billie',\n",
       " 'Billy',\n",
       " 'Bing',\n",
       " 'Binky',\n",
       " 'Bishop',\n",
       " 'Bjorn',\n",
       " 'Bjorne',\n",
       " 'Blaine',\n",
       " 'Blair',\n",
       " 'Blake',\n",
       " 'Blare',\n",
       " 'Blayne',\n",
       " 'Bo',\n",
       " 'Bob',\n",
       " 'Bobbie',\n",
       " 'Bobby',\n",
       " 'Bogart',\n",
       " 'Bogdan',\n",
       " 'Boniface',\n",
       " 'Boris',\n",
       " 'Boyce',\n",
       " 'Boyd',\n",
       " 'Brad',\n",
       " 'Braden',\n",
       " 'Bradford',\n",
       " 'Bradley',\n",
       " 'Bradly',\n",
       " 'Brady',\n",
       " 'Brandon',\n",
       " 'Brandy',\n",
       " 'Brant',\n",
       " 'Brendan',\n",
       " 'Brent',\n",
       " 'Bret',\n",
       " 'Brett',\n",
       " 'Brewer',\n",
       " 'Brewster',\n",
       " 'Brian',\n",
       " 'Brice',\n",
       " 'Briggs',\n",
       " 'Brinkley',\n",
       " 'Britt',\n",
       " 'Brock',\n",
       " 'Broddie',\n",
       " 'Broddy',\n",
       " 'Broderic',\n",
       " 'Broderick',\n",
       " 'Brodie',\n",
       " 'Brody',\n",
       " 'Bronson',\n",
       " 'Brook',\n",
       " 'Brooke',\n",
       " 'Brooks',\n",
       " 'Bruce',\n",
       " 'Bruno',\n",
       " 'Bryan',\n",
       " 'Bryant',\n",
       " 'Bryce',\n",
       " 'Bryn',\n",
       " 'Bryon',\n",
       " 'Bubba',\n",
       " 'Buck',\n",
       " 'Bucky',\n",
       " 'Bud',\n",
       " 'Buddy',\n",
       " 'Burgess',\n",
       " 'Burke',\n",
       " 'Burl',\n",
       " 'Burnaby',\n",
       " 'Burt',\n",
       " 'Burton',\n",
       " 'Buster',\n",
       " 'Butch',\n",
       " 'Butler',\n",
       " 'Byram',\n",
       " 'Byron',\n",
       " 'Caesar',\n",
       " 'Cain',\n",
       " 'Cal',\n",
       " 'Caldwell',\n",
       " 'Caleb',\n",
       " 'Calhoun',\n",
       " 'Calvin',\n",
       " 'Cam',\n",
       " 'Cameron',\n",
       " 'Cammy',\n",
       " 'Carey',\n",
       " 'Carl',\n",
       " 'Carleigh',\n",
       " 'Carlie',\n",
       " 'Carlin',\n",
       " 'Carlo',\n",
       " 'Carlos',\n",
       " 'Carlton',\n",
       " 'Carlyle',\n",
       " 'Carmine',\n",
       " 'Carroll',\n",
       " 'Carson',\n",
       " 'Carsten',\n",
       " 'Carter',\n",
       " 'Cary',\n",
       " 'Caryl',\n",
       " 'Case',\n",
       " 'Casey',\n",
       " 'Caspar',\n",
       " 'Casper',\n",
       " 'Cass',\n",
       " 'Cat',\n",
       " 'Cecil',\n",
       " 'Cesar',\n",
       " 'Chad',\n",
       " 'Chadd',\n",
       " 'Chaddie',\n",
       " 'Chaddy',\n",
       " 'Chadwick',\n",
       " 'Chaim',\n",
       " 'Chalmers',\n",
       " 'Chan',\n",
       " 'Chance',\n",
       " 'Chancey',\n",
       " 'Chanderjit',\n",
       " 'Chandler',\n",
       " 'Chane',\n",
       " 'Chariot',\n",
       " 'Charles',\n",
       " 'Charleton',\n",
       " 'Charley',\n",
       " 'Charlie',\n",
       " 'Charlton',\n",
       " 'Chas',\n",
       " 'Chase',\n",
       " 'Chaunce',\n",
       " 'Chauncey',\n",
       " 'Che',\n",
       " 'Chelton',\n",
       " 'Chen',\n",
       " 'Chester',\n",
       " 'Cheston',\n",
       " 'Chet',\n",
       " 'Chev',\n",
       " 'Chevalier',\n",
       " 'Chevy',\n",
       " 'Chip',\n",
       " 'Chris',\n",
       " 'Chrissy',\n",
       " 'Christ',\n",
       " 'Christian',\n",
       " 'Christiano',\n",
       " 'Christie',\n",
       " 'Christof',\n",
       " 'Christofer',\n",
       " 'Christoph',\n",
       " 'Christophe',\n",
       " 'Christopher',\n",
       " 'Christorpher',\n",
       " 'Christos',\n",
       " 'Christy',\n",
       " 'Chrisy',\n",
       " 'Chuck',\n",
       " 'Churchill',\n",
       " 'Clair',\n",
       " 'Claire',\n",
       " 'Clancy',\n",
       " 'Clarance',\n",
       " 'Clare',\n",
       " 'Clarence',\n",
       " 'Clark',\n",
       " 'Clarke',\n",
       " 'Claude',\n",
       " 'Claudio',\n",
       " 'Claudius',\n",
       " 'Claus',\n",
       " 'Clay',\n",
       " 'Clayborn',\n",
       " 'Clayborne',\n",
       " 'Claybourne',\n",
       " 'Clayton',\n",
       " 'Cleland',\n",
       " 'Clem',\n",
       " 'Clemens',\n",
       " 'Clement',\n",
       " 'Clemente',\n",
       " 'Clemmie',\n",
       " 'Cletus',\n",
       " 'Cleveland',\n",
       " 'Cliff',\n",
       " 'Clifford',\n",
       " 'Clifton',\n",
       " 'Clint',\n",
       " 'Clinten',\n",
       " 'Clinton',\n",
       " 'Clive',\n",
       " 'Clyde',\n",
       " 'Cob',\n",
       " 'Cobb',\n",
       " 'Cobbie',\n",
       " 'Cobby',\n",
       " 'Cody',\n",
       " 'Colbert',\n",
       " 'Cole',\n",
       " 'Coleman',\n",
       " 'Colin',\n",
       " 'Collin',\n",
       " 'Collins',\n",
       " 'Conan',\n",
       " 'Connie',\n",
       " 'Connolly',\n",
       " 'Connor',\n",
       " 'Conrad',\n",
       " 'Conroy',\n",
       " 'Constantin',\n",
       " 'Constantine',\n",
       " 'Constantinos',\n",
       " 'Conway',\n",
       " 'Cooper',\n",
       " 'Corbin',\n",
       " 'Corby',\n",
       " 'Corey',\n",
       " 'Corky',\n",
       " 'Cornelius',\n",
       " 'Cornellis',\n",
       " 'Corrie',\n",
       " 'Cortese',\n",
       " 'Corwin',\n",
       " 'Cory',\n",
       " 'Cosmo',\n",
       " 'Costa',\n",
       " 'Courtney',\n",
       " 'Craig',\n",
       " 'Crawford',\n",
       " 'Creighton',\n",
       " 'Cris',\n",
       " 'Cristopher',\n",
       " 'Curt',\n",
       " 'Curtice',\n",
       " 'Curtis',\n",
       " 'Cy',\n",
       " 'Cyril',\n",
       " 'Cyrill',\n",
       " 'Cyrille',\n",
       " 'Cyrillus',\n",
       " 'Cyrus',\n",
       " 'Dabney',\n",
       " 'Daffy',\n",
       " 'Dale',\n",
       " 'Dallas',\n",
       " 'Dalton',\n",
       " 'Damian',\n",
       " 'Damien',\n",
       " 'Damon',\n",
       " 'Dan',\n",
       " 'Dana',\n",
       " 'Dane',\n",
       " 'Dani',\n",
       " 'Danie',\n",
       " 'Daniel',\n",
       " 'Dannie',\n",
       " 'Danny',\n",
       " 'Dante',\n",
       " 'Darby',\n",
       " 'Darcy',\n",
       " 'Daren',\n",
       " 'Darian',\n",
       " 'Darien',\n",
       " 'Darin',\n",
       " 'Dario',\n",
       " 'Darius',\n",
       " 'Darrel',\n",
       " 'Darrell',\n",
       " 'Darren',\n",
       " 'Darrick',\n",
       " 'Darrin',\n",
       " 'Darryl',\n",
       " 'Darth',\n",
       " 'Darwin',\n",
       " 'Daryl',\n",
       " 'Daryle',\n",
       " 'Dave',\n",
       " 'Davey',\n",
       " 'David',\n",
       " 'Davidde',\n",
       " 'Davide',\n",
       " 'Davidson',\n",
       " 'Davie',\n",
       " 'Davin',\n",
       " 'Davis',\n",
       " 'Davon',\n",
       " 'Davoud',\n",
       " 'Davy',\n",
       " 'Dawson',\n",
       " 'Dean',\n",
       " 'Deane',\n",
       " 'Del',\n",
       " 'Delbert',\n",
       " 'Dell',\n",
       " 'Delmar',\n",
       " 'Demetre',\n",
       " 'Demetri',\n",
       " 'Demetris',\n",
       " 'Demetrius',\n",
       " 'Demosthenis',\n",
       " 'Denis',\n",
       " 'Dennie',\n",
       " 'Dennis',\n",
       " 'Denny',\n",
       " 'Derby',\n",
       " 'Derek',\n",
       " 'Derick',\n",
       " 'Derk',\n",
       " 'Derrek',\n",
       " 'Derrick',\n",
       " 'Derrin',\n",
       " 'Derrol',\n",
       " 'Derron',\n",
       " 'Deryl',\n",
       " 'Desmond',\n",
       " 'Desmund',\n",
       " 'Devin',\n",
       " 'Devon',\n",
       " 'Dewey',\n",
       " 'Dewitt',\n",
       " 'Dexter',\n",
       " 'Dick',\n",
       " 'Dickey',\n",
       " 'Dickie',\n",
       " 'Diego',\n",
       " 'Dieter',\n",
       " 'Dietrich',\n",
       " 'Dillon',\n",
       " 'Dimitri',\n",
       " 'Dimitrios',\n",
       " 'Dimitris',\n",
       " 'Dimitrou',\n",
       " 'Dimitry',\n",
       " 'Dino',\n",
       " 'Dion',\n",
       " 'Dionis',\n",
       " 'Dionysus',\n",
       " 'Dirk',\n",
       " 'Dmitri',\n",
       " 'Dom',\n",
       " 'Domenic',\n",
       " 'Domenico',\n",
       " 'Dominic',\n",
       " 'Dominick',\n",
       " 'Dominique',\n",
       " 'Don',\n",
       " 'Donal',\n",
       " 'Donald',\n",
       " 'Donn',\n",
       " 'Donnie',\n",
       " 'Donny',\n",
       " 'Donovan',\n",
       " 'Dorian',\n",
       " 'Dory',\n",
       " 'Doug',\n",
       " 'Douggie',\n",
       " 'Dougie',\n",
       " 'Douglas',\n",
       " 'Douglass',\n",
       " 'Douglis',\n",
       " 'Dov',\n",
       " 'Doyle',\n",
       " 'Drake',\n",
       " 'Drew',\n",
       " 'Dru',\n",
       " 'Dryke',\n",
       " 'Duane',\n",
       " 'Dudley',\n",
       " 'Duffie',\n",
       " 'Duffy',\n",
       " 'Dugan',\n",
       " 'Duke',\n",
       " 'Dunc',\n",
       " 'Duncan',\n",
       " 'Dunstan',\n",
       " 'Durand',\n",
       " 'Durant',\n",
       " 'Durante',\n",
       " 'Durward',\n",
       " 'Dustin',\n",
       " 'Dwain',\n",
       " 'Dwaine',\n",
       " 'Dwane',\n",
       " 'Dwayne',\n",
       " 'Dwight',\n",
       " 'Dylan',\n",
       " 'Dyson',\n",
       " 'Earl',\n",
       " 'Earle',\n",
       " 'Easton',\n",
       " 'Eben',\n",
       " 'Ebeneser',\n",
       " 'Ebenezer',\n",
       " 'Eberhard',\n",
       " 'Ed',\n",
       " 'Eddie',\n",
       " 'Eddy',\n",
       " 'Edgar',\n",
       " 'Edgardo',\n",
       " 'Edie',\n",
       " 'Edmond',\n",
       " 'Edmund',\n",
       " 'Edouard',\n",
       " 'Edsel',\n",
       " 'Eduard',\n",
       " 'Eduardo',\n",
       " 'Edward',\n",
       " 'Edwin',\n",
       " 'Efram',\n",
       " 'Egbert',\n",
       " 'Ehud',\n",
       " 'Elbert',\n",
       " 'Elden',\n",
       " 'Eldon',\n",
       " 'Eli',\n",
       " 'Elias',\n",
       " 'Elihu',\n",
       " 'Elijah',\n",
       " 'Eliot',\n",
       " 'Eliott',\n",
       " 'Elisha',\n",
       " 'Elliot',\n",
       " 'Elliott',\n",
       " 'Ellis',\n",
       " 'Ellsworth',\n",
       " 'Ellwood',\n",
       " 'Elmer',\n",
       " 'Elmore',\n",
       " 'Elnar',\n",
       " 'Elric',\n",
       " 'Elroy',\n",
       " 'Elton',\n",
       " 'Elvin',\n",
       " 'Elvis',\n",
       " 'Elwin',\n",
       " 'Elwood',\n",
       " 'Elwyn',\n",
       " 'Ely',\n",
       " 'Emanuel',\n",
       " 'Emerson',\n",
       " 'Emery',\n",
       " 'Emil',\n",
       " 'Emile',\n",
       " 'Emilio',\n",
       " 'Emmanuel',\n",
       " 'Emmery',\n",
       " 'Emmet',\n",
       " 'Emmett',\n",
       " 'Emmit',\n",
       " 'Emmott',\n",
       " 'Emmy',\n",
       " 'Emory',\n",
       " 'Ender',\n",
       " 'Engelbart',\n",
       " 'Engelbert',\n",
       " 'Englebart',\n",
       " 'Englebert',\n",
       " 'Enoch',\n",
       " 'Enrico',\n",
       " 'Enrique',\n",
       " 'Ephraim',\n",
       " 'Ephram',\n",
       " 'Ephrayim',\n",
       " 'Ephrem',\n",
       " 'Er',\n",
       " 'Erasmus',\n",
       " 'Erastus',\n",
       " 'Erek',\n",
       " 'Erhard',\n",
       " 'Erhart',\n",
       " 'Eric',\n",
       " 'Erich',\n",
       " 'Erick',\n",
       " 'Erik',\n",
       " 'Erin',\n",
       " 'Erl',\n",
       " 'Ernest',\n",
       " 'Ernesto',\n",
       " 'Ernie',\n",
       " 'Ernst',\n",
       " 'Erny',\n",
       " 'Errol',\n",
       " 'Ervin',\n",
       " 'Erwin',\n",
       " 'Esau',\n",
       " 'Esme',\n",
       " 'Esteban',\n",
       " 'Ethan',\n",
       " 'Ethelbert',\n",
       " 'Ethelred',\n",
       " 'Etienne',\n",
       " 'Euclid',\n",
       " 'Eugen',\n",
       " 'Eugene',\n",
       " 'Eustace',\n",
       " 'Ev',\n",
       " 'Evan',\n",
       " 'Evelyn',\n",
       " 'Everard',\n",
       " 'Everett',\n",
       " 'Ewan',\n",
       " 'Ewart',\n",
       " 'Ez',\n",
       " 'Ezechiel',\n",
       " 'Ezekiel',\n",
       " 'Ezra',\n",
       " 'Fabian',\n",
       " 'Fabio',\n",
       " 'Fairfax',\n",
       " 'Farley',\n",
       " 'Fazeel',\n",
       " 'Federico',\n",
       " 'Felice',\n",
       " 'Felicio',\n",
       " 'Felipe',\n",
       " 'Felix',\n",
       " 'Ferd',\n",
       " 'Ferdie',\n",
       " 'Ferdinand',\n",
       " 'Ferdy',\n",
       " 'Fergus',\n",
       " 'Ferguson',\n",
       " 'Ferinand',\n",
       " 'Fernando',\n",
       " 'Fidel',\n",
       " 'Filbert',\n",
       " 'Filip',\n",
       " 'Filipe',\n",
       " 'Filmore',\n",
       " 'Finley',\n",
       " 'Finn',\n",
       " 'Fitz',\n",
       " 'Fitzgerald',\n",
       " 'Flem',\n",
       " 'Fleming',\n",
       " 'Flemming',\n",
       " 'Fletch',\n",
       " 'Fletcher',\n",
       " 'Flin',\n",
       " 'Flinn',\n",
       " 'Flint',\n",
       " 'Flipper',\n",
       " 'Florian',\n",
       " 'Floyd',\n",
       " 'Flynn',\n",
       " 'Fons',\n",
       " 'Fonsie',\n",
       " 'Fonz',\n",
       " 'Fonzie',\n",
       " 'Forbes',\n",
       " 'Ford',\n",
       " 'Forest',\n",
       " 'Forester',\n",
       " 'Forrest',\n",
       " 'Forrester',\n",
       " 'Forster',\n",
       " 'Foster',\n",
       " 'Fowler',\n",
       " 'Fox',\n",
       " 'Fran',\n",
       " 'Francesco',\n",
       " 'Francis',\n",
       " 'Francisco',\n",
       " 'Francois',\n",
       " 'Frank',\n",
       " 'Frankie',\n",
       " 'Franklin',\n",
       " 'Franklyn',\n",
       " 'Franky',\n",
       " 'Frans',\n",
       " 'Franz',\n",
       " 'Fraser',\n",
       " 'Frazier',\n",
       " 'Fred',\n",
       " 'Freddie',\n",
       " 'Freddy',\n",
       " 'Frederic',\n",
       " 'Frederich',\n",
       " 'Frederick',\n",
       " 'Frederico',\n",
       " 'Frederik',\n",
       " 'Fredric',\n",
       " 'Fredrick',\n",
       " 'Freeman',\n",
       " 'Freemon',\n",
       " 'Fremont',\n",
       " 'French',\n",
       " 'Friedric',\n",
       " 'Friedrich',\n",
       " 'Friedrick',\n",
       " 'Fritz',\n",
       " 'Fulton',\n",
       " 'Fyodor',\n",
       " 'Gabe',\n",
       " 'Gabriel',\n",
       " 'Gabriele',\n",
       " 'Gabriell',\n",
       " 'Gabriello',\n",
       " 'Gail',\n",
       " 'Gale',\n",
       " 'Galen',\n",
       " 'Gallagher',\n",
       " 'Gamaliel',\n",
       " 'Garcia',\n",
       " 'Garcon',\n",
       " 'Gardener',\n",
       " 'Gardiner',\n",
       " 'Gardner',\n",
       " 'Garey',\n",
       " 'Garfield',\n",
       " 'Garfinkel',\n",
       " 'Garold',\n",
       " 'Garp',\n",
       " 'Garret',\n",
       " 'Garrett',\n",
       " 'Garrot',\n",
       " 'Garrott',\n",
       " 'Garry',\n",
       " 'Garth',\n",
       " 'Garv',\n",
       " 'Garvey',\n",
       " 'Garvin',\n",
       " 'Garvy',\n",
       " 'Garwin',\n",
       " 'Garwood',\n",
       " 'Gary',\n",
       " 'Gaspar',\n",
       " 'Gasper',\n",
       " 'Gaston',\n",
       " 'Gav',\n",
       " 'Gaven',\n",
       " 'Gavin',\n",
       " 'Gavriel',\n",
       " 'Gay',\n",
       " 'Gayle',\n",
       " 'Gearard',\n",
       " 'Gene',\n",
       " 'Geo',\n",
       " 'Geof',\n",
       " 'Geoff',\n",
       " 'Geoffrey',\n",
       " 'Geoffry',\n",
       " 'Georg',\n",
       " 'George',\n",
       " 'Georges',\n",
       " 'Georgia',\n",
       " 'Georgie',\n",
       " 'Georgy',\n",
       " 'Gerald',\n",
       " 'Geraldo',\n",
       " 'Gerard',\n",
       " 'Gere',\n",
       " 'Gerhard',\n",
       " 'Gerhardt',\n",
       " 'Geri',\n",
       " 'Germaine',\n",
       " 'Gerold',\n",
       " 'Gerome',\n",
       " 'Gerrard',\n",
       " 'Gerri',\n",
       " 'Gerrit',\n",
       " 'Gerry',\n",
       " 'Gershom',\n",
       " 'Gershon',\n",
       " 'Giacomo',\n",
       " 'Gian',\n",
       " 'Giancarlo',\n",
       " 'Giavani',\n",
       " 'Gibb',\n",
       " 'Gideon',\n",
       " 'Giff',\n",
       " 'Giffard',\n",
       " 'Giffer',\n",
       " 'Giffie',\n",
       " 'Gifford',\n",
       " 'Giffy',\n",
       " 'Gil',\n",
       " 'Gilbert',\n",
       " 'Gilberto',\n",
       " 'Gilburt',\n",
       " 'Giles',\n",
       " 'Gill',\n",
       " 'Gilles',\n",
       " 'Ginger',\n",
       " 'Gino',\n",
       " 'Giordano',\n",
       " 'Giorgi',\n",
       " 'Giorgio',\n",
       " 'Giovanne',\n",
       " ...]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names.words('male.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "612b354b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Abagael',\n",
       " 'Abagail',\n",
       " 'Abbe',\n",
       " 'Abbey',\n",
       " 'Abbi',\n",
       " 'Abbie',\n",
       " 'Abby',\n",
       " 'Abigael',\n",
       " 'Abigail',\n",
       " 'Abigale',\n",
       " 'Abra',\n",
       " 'Acacia',\n",
       " 'Ada',\n",
       " 'Adah',\n",
       " 'Adaline',\n",
       " 'Adara',\n",
       " 'Addie',\n",
       " 'Addis',\n",
       " 'Adel',\n",
       " 'Adela',\n",
       " 'Adelaide',\n",
       " 'Adele',\n",
       " 'Adelice',\n",
       " 'Adelina',\n",
       " 'Adelind',\n",
       " 'Adeline',\n",
       " 'Adella',\n",
       " 'Adelle',\n",
       " 'Adena',\n",
       " 'Adey',\n",
       " 'Adi',\n",
       " 'Adiana',\n",
       " 'Adina',\n",
       " 'Adora',\n",
       " 'Adore',\n",
       " 'Adoree',\n",
       " 'Adorne',\n",
       " 'Adrea',\n",
       " 'Adria',\n",
       " 'Adriaens',\n",
       " 'Adrian',\n",
       " 'Adriana',\n",
       " 'Adriane',\n",
       " 'Adrianna',\n",
       " 'Adrianne',\n",
       " 'Adrien',\n",
       " 'Adriena',\n",
       " 'Adrienne',\n",
       " 'Aeriel',\n",
       " 'Aeriela',\n",
       " 'Aeriell',\n",
       " 'Ag',\n",
       " 'Agace',\n",
       " 'Agata',\n",
       " 'Agatha',\n",
       " 'Agathe',\n",
       " 'Aggi',\n",
       " 'Aggie',\n",
       " 'Aggy',\n",
       " 'Agna',\n",
       " 'Agnella',\n",
       " 'Agnes',\n",
       " 'Agnese',\n",
       " 'Agnesse',\n",
       " 'Agneta',\n",
       " 'Agnola',\n",
       " 'Agretha',\n",
       " 'Aida',\n",
       " 'Aidan',\n",
       " 'Aigneis',\n",
       " 'Aila',\n",
       " 'Aile',\n",
       " 'Ailee',\n",
       " 'Aileen',\n",
       " 'Ailene',\n",
       " 'Ailey',\n",
       " 'Aili',\n",
       " 'Ailina',\n",
       " 'Ailyn',\n",
       " 'Aime',\n",
       " 'Aimee',\n",
       " 'Aimil',\n",
       " 'Aina',\n",
       " 'Aindrea',\n",
       " 'Ainslee',\n",
       " 'Ainsley',\n",
       " 'Ainslie',\n",
       " 'Ajay',\n",
       " 'Alaine',\n",
       " 'Alameda',\n",
       " 'Alana',\n",
       " 'Alanah',\n",
       " 'Alane',\n",
       " 'Alanna',\n",
       " 'Alayne',\n",
       " 'Alberta',\n",
       " 'Albertina',\n",
       " 'Albertine',\n",
       " 'Albina',\n",
       " 'Alecia',\n",
       " 'Aleda',\n",
       " 'Aleece',\n",
       " 'Aleecia',\n",
       " 'Aleen',\n",
       " 'Alejandra',\n",
       " 'Alejandrina',\n",
       " 'Alena',\n",
       " 'Alene',\n",
       " 'Alessandra',\n",
       " 'Aleta',\n",
       " 'Alethea',\n",
       " 'Alex',\n",
       " 'Alexa',\n",
       " 'Alexandra',\n",
       " 'Alexandrina',\n",
       " 'Alexi',\n",
       " 'Alexia',\n",
       " 'Alexina',\n",
       " 'Alexine',\n",
       " 'Alexis',\n",
       " 'Alfie',\n",
       " 'Alfreda',\n",
       " 'Ali',\n",
       " 'Alia',\n",
       " 'Alica',\n",
       " 'Alice',\n",
       " 'Alicea',\n",
       " 'Alicia',\n",
       " 'Alida',\n",
       " 'Alidia',\n",
       " 'Alina',\n",
       " 'Aline',\n",
       " 'Alis',\n",
       " 'Alisa',\n",
       " 'Alisha',\n",
       " 'Alison',\n",
       " 'Alissa',\n",
       " 'Alisun',\n",
       " 'Alix',\n",
       " 'Aliza',\n",
       " 'Alla',\n",
       " 'Alleen',\n",
       " 'Allegra',\n",
       " 'Allene',\n",
       " 'Alli',\n",
       " 'Allianora',\n",
       " 'Allie',\n",
       " 'Allina',\n",
       " 'Allis',\n",
       " 'Allison',\n",
       " 'Allissa',\n",
       " 'Allsun',\n",
       " 'Ally',\n",
       " 'Allyce',\n",
       " 'Allyn',\n",
       " 'Allys',\n",
       " 'Allyson',\n",
       " 'Alma',\n",
       " 'Almeda',\n",
       " 'Almeria',\n",
       " 'Almeta',\n",
       " 'Almira',\n",
       " 'Almire',\n",
       " 'Aloise',\n",
       " 'Aloisia',\n",
       " 'Aloysia',\n",
       " 'Alpa',\n",
       " 'Alta',\n",
       " 'Althea',\n",
       " 'Alvera',\n",
       " 'Alvina',\n",
       " 'Alvinia',\n",
       " 'Alvira',\n",
       " 'Alyce',\n",
       " 'Alyda',\n",
       " 'Alys',\n",
       " 'Alysa',\n",
       " 'Alyse',\n",
       " 'Alysia',\n",
       " 'Alyson',\n",
       " 'Alyss',\n",
       " 'Alyssa',\n",
       " 'Amabel',\n",
       " 'Amabelle',\n",
       " 'Amalea',\n",
       " 'Amalee',\n",
       " 'Amaleta',\n",
       " 'Amalia',\n",
       " 'Amalie',\n",
       " 'Amalita',\n",
       " 'Amalle',\n",
       " 'Amanda',\n",
       " 'Amandi',\n",
       " 'Amandie',\n",
       " 'Amandy',\n",
       " 'Amara',\n",
       " 'Amargo',\n",
       " 'Amata',\n",
       " 'Amber',\n",
       " 'Amberly',\n",
       " 'Ambrosia',\n",
       " 'Ambur',\n",
       " 'Ame',\n",
       " 'Amelia',\n",
       " 'Amelie',\n",
       " 'Amelina',\n",
       " 'Ameline',\n",
       " 'Amelita',\n",
       " 'Ami',\n",
       " 'Amie',\n",
       " 'Amity',\n",
       " 'Ammamaria',\n",
       " 'Amy',\n",
       " 'Ana',\n",
       " 'Anabel',\n",
       " 'Anabella',\n",
       " 'Anabelle',\n",
       " 'Anais',\n",
       " 'Analiese',\n",
       " 'Analise',\n",
       " 'Anallese',\n",
       " 'Anallise',\n",
       " 'Anastasia',\n",
       " 'Anastasie',\n",
       " 'Anastassia',\n",
       " 'Anatola',\n",
       " 'Andee',\n",
       " 'Andi',\n",
       " 'Andie',\n",
       " 'Andra',\n",
       " 'Andrea',\n",
       " 'Andreana',\n",
       " 'Andree',\n",
       " 'Andrei',\n",
       " 'Andria',\n",
       " 'Andriana',\n",
       " 'Andriette',\n",
       " 'Andromache',\n",
       " 'Andromeda',\n",
       " 'Andy',\n",
       " 'Anestassia',\n",
       " 'Anet',\n",
       " 'Anett',\n",
       " 'Anetta',\n",
       " 'Anette',\n",
       " 'Ange',\n",
       " 'Angel',\n",
       " 'Angela',\n",
       " 'Angele',\n",
       " 'Angelia',\n",
       " 'Angelica',\n",
       " 'Angelika',\n",
       " 'Angelina',\n",
       " 'Angeline',\n",
       " 'Angelique',\n",
       " 'Angelita',\n",
       " 'Angelle',\n",
       " 'Angie',\n",
       " 'Angil',\n",
       " 'Angy',\n",
       " 'Ania',\n",
       " 'Anica',\n",
       " 'Anissa',\n",
       " 'Anita',\n",
       " 'Anitra',\n",
       " 'Anja',\n",
       " 'Anjanette',\n",
       " 'Anjela',\n",
       " 'Ann',\n",
       " 'Ann-Mari',\n",
       " 'Ann-Marie',\n",
       " 'Anna',\n",
       " 'Anna-Diana',\n",
       " 'Anna-Diane',\n",
       " 'Anna-Maria',\n",
       " 'Annabal',\n",
       " 'Annabel',\n",
       " 'Annabela',\n",
       " 'Annabell',\n",
       " 'Annabella',\n",
       " 'Annabelle',\n",
       " 'Annadiana',\n",
       " 'Annadiane',\n",
       " 'Annalee',\n",
       " 'Annalena',\n",
       " 'Annaliese',\n",
       " 'Annalisa',\n",
       " 'Annalise',\n",
       " 'Annalyse',\n",
       " 'Annamari',\n",
       " 'Annamaria',\n",
       " 'Annamarie',\n",
       " 'Anne',\n",
       " 'Anne-Corinne',\n",
       " 'Anne-Mar',\n",
       " 'Anne-Marie',\n",
       " 'Annecorinne',\n",
       " 'Anneliese',\n",
       " 'Annelise',\n",
       " 'Annemarie',\n",
       " 'Annetta',\n",
       " 'Annette',\n",
       " 'Anni',\n",
       " 'Annice',\n",
       " 'Annie',\n",
       " 'Annissa',\n",
       " 'Annmaria',\n",
       " 'Annmarie',\n",
       " 'Annnora',\n",
       " 'Annora',\n",
       " 'Anny',\n",
       " 'Anselma',\n",
       " 'Ansley',\n",
       " 'Anstice',\n",
       " 'Anthe',\n",
       " 'Anthea',\n",
       " 'Anthia',\n",
       " 'Antoinette',\n",
       " 'Antonella',\n",
       " 'Antonetta',\n",
       " 'Antonia',\n",
       " 'Antonie',\n",
       " 'Antonietta',\n",
       " 'Antonina',\n",
       " 'Anya',\n",
       " 'Aphrodite',\n",
       " 'Appolonia',\n",
       " 'April',\n",
       " 'Aprilette',\n",
       " 'Ara',\n",
       " 'Arabel',\n",
       " 'Arabela',\n",
       " 'Arabele',\n",
       " 'Arabella',\n",
       " 'Arabelle',\n",
       " 'Arda',\n",
       " 'Ardath',\n",
       " 'Ardeen',\n",
       " 'Ardelia',\n",
       " 'Ardelis',\n",
       " 'Ardella',\n",
       " 'Ardelle',\n",
       " 'Arden',\n",
       " 'Ardene',\n",
       " 'Ardenia',\n",
       " 'Ardine',\n",
       " 'Ardis',\n",
       " 'Ardith',\n",
       " 'Ardra',\n",
       " 'Ardyce',\n",
       " 'Ardys',\n",
       " 'Ardyth',\n",
       " 'Aretha',\n",
       " 'Ariadne',\n",
       " 'Ariana',\n",
       " 'Arianne',\n",
       " 'Aridatha',\n",
       " 'Ariel',\n",
       " 'Ariela',\n",
       " 'Ariella',\n",
       " 'Arielle',\n",
       " 'Arlana',\n",
       " 'Arlee',\n",
       " 'Arleen',\n",
       " 'Arlen',\n",
       " 'Arlena',\n",
       " 'Arlene',\n",
       " 'Arleta',\n",
       " 'Arlette',\n",
       " 'Arleyne',\n",
       " 'Arlie',\n",
       " 'Arliene',\n",
       " 'Arlina',\n",
       " 'Arlinda',\n",
       " 'Arline',\n",
       " 'Arly',\n",
       " 'Arlyn',\n",
       " 'Arlyne',\n",
       " 'Aryn',\n",
       " 'Ashely',\n",
       " 'Ashlee',\n",
       " 'Ashleigh',\n",
       " 'Ashlen',\n",
       " 'Ashley',\n",
       " 'Ashli',\n",
       " 'Ashlie',\n",
       " 'Ashly',\n",
       " 'Asia',\n",
       " 'Astra',\n",
       " 'Astrid',\n",
       " 'Astrix',\n",
       " 'Atalanta',\n",
       " 'Athena',\n",
       " 'Athene',\n",
       " 'Atlanta',\n",
       " 'Atlante',\n",
       " 'Auberta',\n",
       " 'Aubine',\n",
       " 'Aubree',\n",
       " 'Aubrette',\n",
       " 'Aubrey',\n",
       " 'Aubrie',\n",
       " 'Aubry',\n",
       " 'Audi',\n",
       " 'Audie',\n",
       " 'Audra',\n",
       " 'Audre',\n",
       " 'Audrey',\n",
       " 'Audrie',\n",
       " 'Audry',\n",
       " 'Audrye',\n",
       " 'Audy',\n",
       " 'Augusta',\n",
       " 'Auguste',\n",
       " 'Augustina',\n",
       " 'Augustine',\n",
       " 'Aura',\n",
       " 'Aurea',\n",
       " 'Aurel',\n",
       " 'Aurelea',\n",
       " 'Aurelia',\n",
       " 'Aurelie',\n",
       " 'Auria',\n",
       " 'Aurie',\n",
       " 'Aurilia',\n",
       " 'Aurlie',\n",
       " 'Auroora',\n",
       " 'Aurora',\n",
       " 'Aurore',\n",
       " 'Austin',\n",
       " 'Austina',\n",
       " 'Austine',\n",
       " 'Ava',\n",
       " 'Aveline',\n",
       " 'Averil',\n",
       " 'Averyl',\n",
       " 'Avie',\n",
       " 'Avis',\n",
       " 'Aviva',\n",
       " 'Avivah',\n",
       " 'Avril',\n",
       " 'Avrit',\n",
       " 'Ayn',\n",
       " 'Bab',\n",
       " 'Babara',\n",
       " 'Babette',\n",
       " 'Babita',\n",
       " 'Babs',\n",
       " 'Bambi',\n",
       " 'Bambie',\n",
       " 'Bamby',\n",
       " 'Barb',\n",
       " 'Barbabra',\n",
       " 'Barbara',\n",
       " 'Barbara-Anne',\n",
       " 'Barbaraanne',\n",
       " 'Barbe',\n",
       " 'Barbee',\n",
       " 'Barbette',\n",
       " 'Barbey',\n",
       " 'Barbi',\n",
       " 'Barbie',\n",
       " 'Barbra',\n",
       " 'Barby',\n",
       " 'Bari',\n",
       " 'Barrie',\n",
       " 'Barry',\n",
       " 'Basia',\n",
       " 'Bathsheba',\n",
       " 'Batsheva',\n",
       " 'Bea',\n",
       " 'Beatrice',\n",
       " 'Beatrisa',\n",
       " 'Beatrix',\n",
       " 'Beatriz',\n",
       " 'Beau',\n",
       " 'Bebe',\n",
       " 'Becca',\n",
       " 'Becka',\n",
       " 'Becki',\n",
       " 'Beckie',\n",
       " 'Becky',\n",
       " 'Bee',\n",
       " 'Beilul',\n",
       " 'Beitris',\n",
       " 'Bekki',\n",
       " 'Bel',\n",
       " 'Belia',\n",
       " 'Belicia',\n",
       " 'Belinda',\n",
       " 'Belita',\n",
       " 'Bell',\n",
       " 'Bella',\n",
       " 'Bellamy',\n",
       " 'Bellanca',\n",
       " 'Belle',\n",
       " 'Bellina',\n",
       " 'Belva',\n",
       " 'Belvia',\n",
       " 'Bendite',\n",
       " 'Benedetta',\n",
       " 'Benedicta',\n",
       " 'Benedikta',\n",
       " 'Benetta',\n",
       " 'Benita',\n",
       " 'Benni',\n",
       " 'Bennie',\n",
       " 'Benny',\n",
       " 'Benoite',\n",
       " 'Berenice',\n",
       " 'Beret',\n",
       " 'Berget',\n",
       " 'Berna',\n",
       " 'Bernadene',\n",
       " 'Bernadette',\n",
       " 'Bernadina',\n",
       " 'Bernadine',\n",
       " 'Bernardina',\n",
       " 'Bernardine',\n",
       " 'Bernelle',\n",
       " 'Bernete',\n",
       " 'Bernetta',\n",
       " 'Bernette',\n",
       " 'Berni',\n",
       " 'Bernice',\n",
       " 'Bernie',\n",
       " 'Bernita',\n",
       " 'Berny',\n",
       " 'Berri',\n",
       " 'Berrie',\n",
       " 'Berry',\n",
       " 'Bert',\n",
       " 'Berta',\n",
       " 'Berte',\n",
       " 'Bertha',\n",
       " 'Berthe',\n",
       " 'Berti',\n",
       " 'Bertie',\n",
       " 'Bertina',\n",
       " 'Bertine',\n",
       " 'Berty',\n",
       " 'Beryl',\n",
       " 'Beryle',\n",
       " 'Bess',\n",
       " 'Bessie',\n",
       " 'Bessy',\n",
       " 'Beth',\n",
       " 'Bethanne',\n",
       " 'Bethany',\n",
       " 'Bethena',\n",
       " 'Bethina',\n",
       " 'Betsey',\n",
       " 'Betsy',\n",
       " 'Betta',\n",
       " 'Bette',\n",
       " 'Bette-Ann',\n",
       " 'Betteann',\n",
       " 'Betteanne',\n",
       " 'Betti',\n",
       " 'Bettie',\n",
       " 'Bettina',\n",
       " 'Bettine',\n",
       " 'Betty',\n",
       " 'Bettye',\n",
       " 'Beulah',\n",
       " 'Bev',\n",
       " 'Beverie',\n",
       " 'Beverlee',\n",
       " 'Beverlie',\n",
       " 'Beverly',\n",
       " 'Bevvy',\n",
       " 'Bianca',\n",
       " 'Bianka',\n",
       " 'Biddy',\n",
       " 'Bidget',\n",
       " 'Bill',\n",
       " 'Billi',\n",
       " 'Billie',\n",
       " 'Billy',\n",
       " 'Binni',\n",
       " 'Binnie',\n",
       " 'Binny',\n",
       " 'Bird',\n",
       " 'Birdie',\n",
       " 'Birgit',\n",
       " 'Birgitta',\n",
       " 'Blair',\n",
       " 'Blaire',\n",
       " 'Blake',\n",
       " 'Blakelee',\n",
       " 'Blakeley',\n",
       " 'Blanca',\n",
       " 'Blanch',\n",
       " 'Blancha',\n",
       " 'Blanche',\n",
       " 'Blinni',\n",
       " 'Blinnie',\n",
       " 'Blinny',\n",
       " 'Bliss',\n",
       " 'Blisse',\n",
       " 'Blithe',\n",
       " 'Blondell',\n",
       " 'Blondelle',\n",
       " 'Blondie',\n",
       " 'Blondy',\n",
       " 'Blythe',\n",
       " 'Bo',\n",
       " 'Bobbette',\n",
       " 'Bobbi',\n",
       " 'Bobbie',\n",
       " 'Bobby',\n",
       " 'Bobette',\n",
       " 'Bobina',\n",
       " 'Bobine',\n",
       " 'Bobinette',\n",
       " 'Bonita',\n",
       " 'Bonnee',\n",
       " 'Bonni',\n",
       " 'Bonnie',\n",
       " 'Bonny',\n",
       " 'Brana',\n",
       " 'Brandais',\n",
       " 'Brande',\n",
       " 'Brandea',\n",
       " 'Brandi',\n",
       " 'Brandice',\n",
       " 'Brandie',\n",
       " 'Brandise',\n",
       " 'Brandy',\n",
       " 'Brea',\n",
       " 'Breanne',\n",
       " 'Brear',\n",
       " 'Bree',\n",
       " 'Breena',\n",
       " 'Bren',\n",
       " 'Brena',\n",
       " 'Brenda',\n",
       " 'Brenn',\n",
       " 'Brenna',\n",
       " 'Brett',\n",
       " 'Bria',\n",
       " 'Briana',\n",
       " 'Brianna',\n",
       " 'Brianne',\n",
       " 'Bride',\n",
       " 'Bridget',\n",
       " 'Bridgett',\n",
       " 'Bridgette',\n",
       " 'Bridie',\n",
       " 'Brier',\n",
       " 'Brietta',\n",
       " 'Brigid',\n",
       " 'Brigida',\n",
       " 'Brigit',\n",
       " 'Brigitta',\n",
       " 'Brigitte',\n",
       " 'Brina',\n",
       " 'Briney',\n",
       " 'Briny',\n",
       " 'Brit',\n",
       " 'Brita',\n",
       " 'Britaney',\n",
       " 'Britani',\n",
       " 'Briteny',\n",
       " 'Britney',\n",
       " 'Britni',\n",
       " 'Britt',\n",
       " 'Britta',\n",
       " 'Brittan',\n",
       " 'Brittany',\n",
       " 'Britte',\n",
       " 'Brittney',\n",
       " 'Brook',\n",
       " 'Brooke',\n",
       " 'Brooks',\n",
       " 'Brunella',\n",
       " 'Brunhilda',\n",
       " 'Brunhilde',\n",
       " 'Bryana',\n",
       " 'Bryn',\n",
       " 'Bryna',\n",
       " 'Brynn',\n",
       " 'Brynna',\n",
       " 'Brynne',\n",
       " 'Buffy',\n",
       " 'Bunni',\n",
       " 'Bunnie',\n",
       " 'Bunny',\n",
       " 'Burta',\n",
       " 'Cabrina',\n",
       " 'Cacilia',\n",
       " 'Cacilie',\n",
       " 'Caitlin',\n",
       " 'Caitrin',\n",
       " 'Cal',\n",
       " 'Calida',\n",
       " 'Calla',\n",
       " 'Calley',\n",
       " 'Calli',\n",
       " 'Callida',\n",
       " 'Callie',\n",
       " 'Cally',\n",
       " 'Calypso',\n",
       " 'Cam',\n",
       " 'Camala',\n",
       " 'Camel',\n",
       " 'Camella',\n",
       " 'Camellia',\n",
       " 'Cameo',\n",
       " 'Cami',\n",
       " 'Camila',\n",
       " 'Camile',\n",
       " 'Camilla',\n",
       " 'Camille',\n",
       " 'Cammi',\n",
       " 'Cammie',\n",
       " 'Cammy',\n",
       " 'Canada',\n",
       " 'Candace',\n",
       " 'Candi',\n",
       " 'Candice',\n",
       " 'Candida',\n",
       " 'Candide',\n",
       " 'Candie',\n",
       " 'Candis',\n",
       " 'Candra',\n",
       " 'Candy',\n",
       " 'Cappella',\n",
       " 'Caprice',\n",
       " 'Cara',\n",
       " 'Caralie',\n",
       " 'Caren',\n",
       " 'Carena',\n",
       " 'Caresa',\n",
       " 'Caressa',\n",
       " 'Caresse',\n",
       " 'Carey',\n",
       " 'Cari',\n",
       " 'Caria',\n",
       " 'Carie',\n",
       " 'Caril',\n",
       " 'Carilyn',\n",
       " 'Carin',\n",
       " 'Carina',\n",
       " 'Carine',\n",
       " 'Cariotta',\n",
       " 'Carissa',\n",
       " 'Carita',\n",
       " 'Caritta',\n",
       " 'Carla',\n",
       " 'Carlee',\n",
       " 'Carleen',\n",
       " 'Carlen',\n",
       " 'Carlena',\n",
       " 'Carlene',\n",
       " 'Carley',\n",
       " 'Carli',\n",
       " 'Carlie',\n",
       " 'Carlin',\n",
       " 'Carlina',\n",
       " 'Carline',\n",
       " 'Carlisle',\n",
       " 'Carlita',\n",
       " 'Carlota',\n",
       " 'Carlotta',\n",
       " 'Carly',\n",
       " 'Carlye',\n",
       " 'Carlyn',\n",
       " 'Carlynn',\n",
       " 'Carlynne',\n",
       " 'Carma',\n",
       " 'Carmel',\n",
       " 'Carmela',\n",
       " 'Carmelia',\n",
       " 'Carmelina',\n",
       " 'Carmelita',\n",
       " 'Carmella',\n",
       " 'Carmelle',\n",
       " 'Carmen',\n",
       " 'Carmina',\n",
       " 'Carmine',\n",
       " 'Carmita',\n",
       " 'Carmon',\n",
       " 'Caro',\n",
       " 'Carol',\n",
       " 'Carol-Jean',\n",
       " 'Carola',\n",
       " 'Carolan',\n",
       " 'Carolann',\n",
       " 'Carole',\n",
       " 'Carolee',\n",
       " 'Caroleen',\n",
       " 'Carolie',\n",
       " 'Carolin',\n",
       " 'Carolina',\n",
       " 'Caroline',\n",
       " 'Caroljean',\n",
       " 'Carolyn',\n",
       " 'Carolyne',\n",
       " 'Carolynn',\n",
       " 'Caron',\n",
       " 'Carree',\n",
       " 'Carri',\n",
       " 'Carrie',\n",
       " 'Carrissa',\n",
       " 'Carrol',\n",
       " 'Carroll',\n",
       " 'Carry',\n",
       " 'Cary',\n",
       " 'Caryl',\n",
       " 'Caryn',\n",
       " 'Casandra',\n",
       " 'Casey',\n",
       " 'Casi',\n",
       " 'Casia',\n",
       " 'Casie',\n",
       " 'Cass',\n",
       " 'Cassandra',\n",
       " 'Cassandre',\n",
       " 'Cassandry',\n",
       " 'Cassaundra',\n",
       " 'Cassey',\n",
       " 'Cassi',\n",
       " 'Cassie',\n",
       " 'Cassondra',\n",
       " 'Cassy',\n",
       " 'Cat',\n",
       " 'Catarina',\n",
       " 'Cate',\n",
       " 'Caterina',\n",
       " 'Catha',\n",
       " 'Catharina',\n",
       " 'Catharine',\n",
       " 'Cathe',\n",
       " 'Cathee',\n",
       " 'Catherin',\n",
       " 'Catherina',\n",
       " 'Catherine',\n",
       " 'Cathi',\n",
       " 'Cathie',\n",
       " 'Cathleen',\n",
       " 'Cathlene',\n",
       " 'Cathrin',\n",
       " 'Cathrine',\n",
       " 'Cathryn',\n",
       " 'Cathy',\n",
       " 'Cathyleen',\n",
       " 'Cati',\n",
       " 'Catie',\n",
       " 'Catina',\n",
       " 'Catlaina',\n",
       " 'Catlee',\n",
       " 'Catlin',\n",
       " 'Catrina',\n",
       " 'Catriona',\n",
       " 'Caty',\n",
       " 'Cayla',\n",
       " 'Cecelia',\n",
       " 'Cecil',\n",
       " 'Cecile',\n",
       " 'Ceciley',\n",
       " 'Cecilia',\n",
       " 'Cecilla',\n",
       " 'Cecily',\n",
       " 'Ceil',\n",
       " 'Cele',\n",
       " 'Celene',\n",
       " 'Celesta',\n",
       " 'Celeste',\n",
       " 'Celestia',\n",
       " 'Celestina',\n",
       " 'Celestine',\n",
       " 'Celestyn',\n",
       " 'Celestyna',\n",
       " 'Celia',\n",
       " 'Celie',\n",
       " 'Celina',\n",
       " 'Celinda',\n",
       " 'Celine',\n",
       " 'Celinka',\n",
       " 'Celisse',\n",
       " 'Celle',\n",
       " 'Cesya',\n",
       " 'Chad',\n",
       " 'Chanda',\n",
       " 'Chandal',\n",
       " 'Chandra',\n",
       " 'Channa',\n",
       " 'Chantal',\n",
       " 'Chantalle',\n",
       " 'Charil',\n",
       " 'Charin',\n",
       " 'Charis',\n",
       " 'Charissa',\n",
       " 'Charisse',\n",
       " 'Charita',\n",
       " 'Charity',\n",
       " 'Charla',\n",
       " 'Charlean',\n",
       " 'Charleen',\n",
       " 'Charlena',\n",
       " 'Charlene',\n",
       " 'Charline',\n",
       " 'Charlot',\n",
       " 'Charlott',\n",
       " 'Charlotta',\n",
       " 'Charlotte',\n",
       " 'Charmain',\n",
       " 'Charmaine',\n",
       " 'Charmane',\n",
       " 'Charmian',\n",
       " 'Charmine',\n",
       " 'Charmion',\n",
       " 'Charo',\n",
       " 'Charyl',\n",
       " 'Chastity',\n",
       " 'Chelsae',\n",
       " 'Chelsea',\n",
       " 'Chelsey',\n",
       " 'Chelsie',\n",
       " 'Chelsy',\n",
       " 'Cher',\n",
       " 'Chere',\n",
       " 'Cherey',\n",
       " 'Cheri',\n",
       " 'Cherianne',\n",
       " 'Cherice',\n",
       " 'Cherida',\n",
       " 'Cherie',\n",
       " 'Cherilyn',\n",
       " 'Cherilynn',\n",
       " 'Cherin',\n",
       " 'Cherise',\n",
       " 'Cherish',\n",
       " 'Cherlyn',\n",
       " 'Cherri',\n",
       " 'Cherrita',\n",
       " 'Cherry',\n",
       " 'Chery',\n",
       " 'Cherye',\n",
       " 'Cheryl',\n",
       " 'Cheslie',\n",
       " 'Chiarra',\n",
       " 'Chickie',\n",
       " 'Chicky',\n",
       " 'Chiquita',\n",
       " 'Chloe',\n",
       " 'Chloette',\n",
       " 'Chloris',\n",
       " 'Chris',\n",
       " 'Chriss',\n",
       " 'Chrissa',\n",
       " 'Chrissie',\n",
       " 'Chrissy',\n",
       " 'Christa',\n",
       " 'Christabel',\n",
       " 'Christabella',\n",
       " 'Christabelle',\n",
       " 'Christal',\n",
       " 'Christalle',\n",
       " 'Christan',\n",
       " 'Christean',\n",
       " 'Christel',\n",
       " 'Christen',\n",
       " 'Christi',\n",
       " 'Christian',\n",
       " 'Christiana',\n",
       " 'Christiane',\n",
       " 'Christie',\n",
       " 'Christin',\n",
       " 'Christina',\n",
       " 'Christine',\n",
       " 'Christy',\n",
       " 'Christyna',\n",
       " 'Chrysa',\n",
       " 'Chrysler',\n",
       " 'Chrystal',\n",
       " 'Chryste',\n",
       " 'Chrystel',\n",
       " 'Ciara',\n",
       " 'Cicely',\n",
       " 'Cicily',\n",
       " 'Ciel',\n",
       " 'Cilka',\n",
       " 'Cinda',\n",
       " 'Cindee',\n",
       " 'Cindelyn',\n",
       " 'Cinderella',\n",
       " 'Cindi',\n",
       " 'Cindie',\n",
       " 'Cindra',\n",
       " 'Cindy',\n",
       " 'Cinnamon',\n",
       " 'Cissie',\n",
       " 'Cissy',\n",
       " 'Clair',\n",
       " 'Claire',\n",
       " 'Clara',\n",
       " 'Clarabelle',\n",
       " 'Clare',\n",
       " ...]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names.words('female.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb00e22b",
   "metadata": {},
   "source": [
    "#### Corpus vs lexicon\n",
    "\n",
    "Corpus is large collection of text data (written or spoken) in particular language. A corpus may include additional information about the words like their POS tags or parse tree of sentences etc. \n",
    "\n",
    "Lexicon is the entire set of a language’s lexemes (vocabulary). Many lexicons contains a core token (lexeme), its Noun form, adjective form, related verb, related adverb etc., its Synonym, antonym etc. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b14864a",
   "metadata": {},
   "source": [
    "##### opinion_lexicon\n",
    "A list of positive and negative opinion words or sentiment words for English."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "d5eae099",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2-faced',\n",
       " '2-faces',\n",
       " 'abnormal',\n",
       " 'abolish',\n",
       " 'abominable',\n",
       " 'abominably',\n",
       " 'abominate',\n",
       " 'abomination',\n",
       " 'abort',\n",
       " 'aborted']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import opinion_lexicon\n",
    "opinion_lexicon.words()[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "ce7d832b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2-faced', '2-faces', 'abnormal', 'abolish', 'abominable']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "opinion_lexicon.negative()[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c5547e7",
   "metadata": {},
   "source": [
    "##### product_reviews_1 and product_reviews_2\n",
    "These two datasets respectively contain annotated customer reviews of 5 and 9 products from amazon.com."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "201a3ee4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i',\n",
       " 'recently',\n",
       " 'purchased',\n",
       " 'the',\n",
       " 'canon',\n",
       " 'powershot',\n",
       " 'g3',\n",
       " 'and',\n",
       " 'am',\n",
       " 'extremely',\n",
       " 'satisfied',\n",
       " 'with',\n",
       " 'the',\n",
       " 'purchase',\n",
       " '.']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import product_reviews_1\n",
    "camera_reviews = product_reviews_1.reviews('Canon_G3.txt')\n",
    "review = camera_reviews[0]\n",
    "review.sents()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "2138b2e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('canon powershot g3', '+3'),\n",
       " ('use', '+2'),\n",
       " ('picture', '+2'),\n",
       " ('picture quality', '+1'),\n",
       " ('picture quality', '+1'),\n",
       " ('camera', '+2'),\n",
       " ('use', '+2'),\n",
       " ('feature', '+1'),\n",
       " ('picture quality', '+3'),\n",
       " ('use', '+1'),\n",
       " ('option', '+1')]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review.features()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3a63ba7",
   "metadata": {},
   "source": [
    "### Simple NLP Tasks in NLTK:   Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "e523adad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I', 'will', 'walk', '500', 'miles', 'and', 'I', 'would', 'walk', '500', 'more', ',', 'just', 'to', 'be', 'the', 'man', 'who', 'walks', 'a', 'thousand', 'miles', 'to', 'fall', 'down', 'at', 'your', 'door', '!']\n",
      "['I will walk 500 miles and I would walk 500 more, just to be the man who walks a thousand miles to fall down at your door!']\n"
     ]
    }
   ],
   "source": [
    "# Tokenization\n",
    "from nltk import word_tokenize, sent_tokenize\n",
    "sent = \"I will walk 500 miles and I would walk 500 more, just to be the man who walks a thousand miles to fall down at your door!\"\n",
    "print(word_tokenize(sent))\n",
    "print(sent_tokenize(sent))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b52fcc0f",
   "metadata": {},
   "source": [
    "### Stopwords removal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "243768ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the unclean version: ['I', 'will', 'pick', 'you', 'up', 'at', '5.00', 'pm', '.', 'We', 'will', 'go', 'for', 'a', 'walk']\n",
      "This is the cleaned version: ['I', 'pick', '5.00', 'pm', '.', 'We', 'go', 'walk']\n"
     ]
    }
   ],
   "source": [
    "#Stopwords removal\n",
    "from nltk.corpus import stopwords        # the corpus module is an extremely useful one. \n",
    "sent = \"I will pick you up at 5.00 pm. We will go for a walk\"                                         \n",
    "stop_words = stopwords.words('english')  # this is the full list of all stop-words stored in nltk\n",
    "token = nltk.word_tokenize(sent)\n",
    "cleaned_token = []\n",
    "for word in token:\n",
    "    if word not in stop_words:\n",
    "        cleaned_token.append(word)\n",
    "print(\"This is the unclean version:\", token)\n",
    "print(\"This is the cleaned version:\", cleaned_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bd39052",
   "metadata": {},
   "source": [
    "###  Stemming and Lemmatization\n",
    "\n",
    "Stemming is the process of removing the suffixes from a word. \n",
    "Lemmatization converts a word into its base form (lemma). \n",
    "\n",
    "The difference between stemming and lemmatization is that lemmatization involves morphological analysis and considers the context.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7200b87",
   "metadata": {},
   "source": [
    "#### Stemming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "46d37a4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "feet\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem import PorterStemmer\n",
    "stemmer = PorterStemmer()\n",
    "print(stemmer.stem(\"feet\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "93ec6845",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['play', 'play', 'play', 'play', 'playful', 'play']\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem import PorterStemmer\n",
    "stemmer = PorterStemmer()\n",
    "words = ['play', 'playing', 'plays', 'played','playfullness', 'playful']\n",
    "stemmed = [stemmer.stem(word) for word in words]\n",
    "print(stemmed)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb94e41d",
   "metadata": {},
   "source": [
    "You can also train your own using unsupervised clustering or supervised classification ML models.\n",
    "\n",
    "#### Lemmatization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5d5e4c95",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "foot\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "print(lemmatizer.lemmatize(\"feet\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c501e7d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens: ['I', 'have', 'two', 'hands', 'a', 'two', 'feet']\n",
      "Lemmas: ['I', 'have', 'two', 'hand', 'a', 'two', 'foot']\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "sent = \"I have two hands a two feet\"\n",
    "tokens = nltk.word_tokenize(sent)\n",
    "print('Tokens:', tokens)\n",
    "\n",
    "lemmed = [WordNetLemmatizer().lemmatize(t) for  t in tokens] \n",
    "print('Lemmas:', lemmed)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a6ee726",
   "metadata": {},
   "source": [
    "### Part of Speech Tagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "de95a10a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('The', 'DT'), ('pos_tag', 'NN'), ('(', '('), (')', ')'), ('method', 'NN'), ('takes', 'VBZ'), ('list', 'NN'), ('tokenized', 'JJ'), ('words', 'NNS'), (',', ','), ('tags', 'NNS'), ('corresponding', 'VBG'), ('Parts', 'NNS'), ('Speech', 'NN')]\n"
     ]
    }
   ],
   "source": [
    "# POS tagging\n",
    "from nltk import pos_tag \n",
    "from nltk.corpus import stopwords \n",
    "\n",
    "stop_words = stopwords.words('english')\n",
    "\n",
    "sentence = \"The pos_tag() method takes in a list of tokenized words, and tags each of them with a corresponding Parts of Speech\"\n",
    "tokens = nltk.word_tokenize(sentence)\n",
    "\n",
    "cleaned_token = []\n",
    "for word in tokens:\n",
    "    if word not in stop_words:\n",
    "        cleaned_token.append(word)\n",
    "tagged = pos_tag(cleaned_token)                 \n",
    "print(tagged)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0c8bff5",
   "metadata": {},
   "source": [
    "The pos_tag() method takes in a list of tokenized words, and tags each of them with a corresponding Parts of Speech identifier into tuples. For example, VB refers to ‘verb’, NNS refers to ‘plural nouns’, DT refers to a ‘determiner’. \n",
    "\n",
    "### Named Entity Recognition : \n",
    "\n",
    "NER is the NLP task of locating and classifying named entities into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. It helps answer questions like this: \n",
    "- Which companies were mentioned in the report?\n",
    "- Does the tweet speak about a specific person?\n",
    "- What places, which companies are mentioned in the news article?\n",
    "- Which product is being spoken about?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "348b24e3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg baseProfile=\"full\" height=\"120px\" preserveAspectRatio=\"xMidYMid meet\" style=\"font-family: times, serif; font-weight:normal; font-style: normal; font-size: 16px;\" version=\"1.1\" viewBox=\"0,0,544.0,120.0\" width=\"544px\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:ev=\"http://www.w3.org/2001/xml-events\" xmlns:xlink=\"http://www.w3.org/1999/xlink\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">S</text></svg><svg width=\"7.35294%\" x=\"0%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">I</text></svg><svg width=\"100%\" x=\"0%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">PRP</text></svg></svg><line stroke=\"black\" x1=\"50%\" x2=\"50%\" y1=\"1.2em\" y2=\"3em\" /></svg><line stroke=\"black\" x1=\"50%\" x2=\"3.67647%\" y1=\"1.2em\" y2=\"3em\" /><svg width=\"11.7647%\" x=\"7.35294%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">played</text></svg><svg width=\"100%\" x=\"0%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">VBD</text></svg></svg><line stroke=\"black\" x1=\"50%\" x2=\"50%\" y1=\"1.2em\" y2=\"3em\" /></svg><line stroke=\"black\" x1=\"50%\" x2=\"13.2353%\" y1=\"1.2em\" y2=\"3em\" /><svg width=\"8.82353%\" x=\"19.1176%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">play</text></svg><svg width=\"100%\" x=\"0%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">NN</text></svg></svg><line stroke=\"black\" x1=\"50%\" x2=\"50%\" y1=\"1.2em\" y2=\"3em\" /></svg><line stroke=\"black\" x1=\"50%\" x2=\"23.5294%\" y1=\"1.2em\" y2=\"3em\" /><svg width=\"16.1765%\" x=\"27.9412%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">playfully</text></svg><svg width=\"100%\" x=\"0%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">RB</text></svg></svg><line stroke=\"black\" x1=\"50%\" x2=\"50%\" y1=\"1.2em\" y2=\"3em\" /></svg><line stroke=\"black\" x1=\"50%\" x2=\"36.0294%\" y1=\"1.2em\" y2=\"3em\" /><svg width=\"13.2353%\" x=\"44.1176%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">players</text></svg><svg width=\"100%\" x=\"0%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">NNS</text></svg></svg><line stroke=\"black\" x1=\"50%\" x2=\"50%\" y1=\"1.2em\" y2=\"3em\" /></svg><line stroke=\"black\" x1=\"50%\" x2=\"50.7353%\" y1=\"1.2em\" y2=\"3em\" /><svg width=\"13.2353%\" x=\"57.3529%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">playing</text></svg><svg width=\"100%\" x=\"0%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">VBG</text></svg></svg><line stroke=\"black\" x1=\"50%\" x2=\"50%\" y1=\"1.2em\" y2=\"3em\" /></svg><line stroke=\"black\" x1=\"50%\" x2=\"63.9706%\" y1=\"1.2em\" y2=\"3em\" /><svg width=\"8.82353%\" x=\"70.5882%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">play</text></svg><svg width=\"100%\" x=\"0%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">NN</text></svg></svg><line stroke=\"black\" x1=\"50%\" x2=\"50%\" y1=\"1.2em\" y2=\"3em\" /></svg><line stroke=\"black\" x1=\"50%\" x2=\"75%\" y1=\"1.2em\" y2=\"3em\" /><svg width=\"20.5882%\" x=\"79.4118%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">playfullness</text></svg><svg width=\"100%\" x=\"0%\" y=\"3em\"><defs /><svg width=\"100%\" x=\"0\" y=\"0em\"><defs /><text text-anchor=\"middle\" x=\"50%\" y=\"1em\">NN</text></svg></svg><line stroke=\"black\" x1=\"50%\" x2=\"50%\" y1=\"1.2em\" y2=\"3em\" /></svg><line stroke=\"black\" x1=\"50%\" x2=\"89.7059%\" y1=\"1.2em\" y2=\"3em\" /></svg>"
      ],
      "text/plain": [
       "Tree('S', [('I', 'PRP'), ('played', 'VBD'), ('play', 'NN'), ('playfully', 'RB'), ('players', 'NNS'), ('playing', 'VBG'), ('play', 'NN'), ('playfullness', 'NN')])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entities = nltk.chunk.ne_chunk(tagged)\n",
    "entities"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6672115",
   "metadata": {},
   "source": [
    "### WordNet Corpus Reader\n",
    "WordNet is An NLTK interface for WordNet. WordNet is a lexical database of English. WordNet uses Synsets to store words. A synset is a set of synonyms that share a common meaning. Using synsets, it helps find conceptual relationships between words such. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9a6ab8b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Synset('dog.n.01'),\n",
       " Synset('frump.n.01'),\n",
       " Synset('dog.n.03'),\n",
       " Synset('cad.n.01'),\n",
       " Synset('frank.n.02'),\n",
       " Synset('pawl.n.01'),\n",
       " Synset('andiron.n.01'),\n",
       " Synset('chase.v.01')]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import wordnet as wn\n",
    "wn.synsets('dog')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "313cdbef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Synset('chase.v.01')]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wn.synsets('dog', pos=wn.VERB)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3544b59f",
   "metadata": {},
   "source": [
    "##### The WordNet corpus reader gives access to the Open Multilingual WordNet, using ISO-639 language codes. These languages are not loaded by default, but only lazily, when needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0917db33",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['als',\n",
       " 'arb',\n",
       " 'bul',\n",
       " 'cat',\n",
       " 'cmn',\n",
       " 'dan',\n",
       " 'ell',\n",
       " 'eng',\n",
       " 'eus',\n",
       " 'fin',\n",
       " 'fra',\n",
       " 'glg',\n",
       " 'heb',\n",
       " 'hrv',\n",
       " 'ind',\n",
       " 'isl',\n",
       " 'ita',\n",
       " 'ita_iwn',\n",
       " 'jpn',\n",
       " 'lit',\n",
       " 'nld',\n",
       " 'nno',\n",
       " 'nob',\n",
       " 'pol',\n",
       " 'por',\n",
       " 'ron',\n",
       " 'slk',\n",
       " 'slv',\n",
       " 'spa',\n",
       " 'swe',\n",
       " 'tha',\n",
       " 'zsm']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(wn.langs())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50fdaaf6",
   "metadata": {},
   "source": [
    "#### nltk moview reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fa98cd4b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['neg/cv000_29416.txt',\n",
       " 'neg/cv001_19502.txt',\n",
       " 'neg/cv002_17424.txt',\n",
       " 'neg/cv003_12683.txt',\n",
       " 'neg/cv004_12641.txt']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "import string\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.corpus import movie_reviews\n",
    "movie_reviews.fileids()[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c97f0936",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['pos/cv995_21821.txt',\n",
       " 'pos/cv996_11592.txt',\n",
       " 'pos/cv997_5046.txt',\n",
       " 'pos/cv998_14111.txt',\n",
       " 'pos/cv999_13106.txt']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_reviews.fileids()[-5:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3bfc3f9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "neg_files = movie_reviews.fileids('neg')\n",
    "pos_files = movie_reviews.fileids('pos')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a20de2f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['films', 'adapted', 'from', 'comic', 'books', 'have', ...]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_reviews.words(fileids=pos_files[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fdf23f6",
   "metadata": {},
   "source": [
    "#### Building a classifier for movie reviews using NLTK Naive Bayes Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cd6b5f53",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "import string\n",
    "#from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.corpus import movie_reviews\n",
    "\n",
    "neg_files = movie_reviews.fileids('neg')\n",
    "pos_files = movie_reviews.fileids('pos')\n",
    "\n",
    "\n",
    "def feature_extraction(words):\n",
    "    stopwordsandpunct = nltk.corpus.stopwords.words(\"english\") + list(string.punctuation)\n",
    "    return { word:'present' for word in words if not word in stopwordsandpunct}\n",
    "\n",
    "neg_words = [(feature_extraction(movie_reviews.words(fileids=[f])), 'neg') for f in neg_files]\n",
    "pos_words = [(feature_extraction(movie_reviews.words(fileids=[f])), 'pos') for f in pos_files]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f98d5a7a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[({'plot': 'present',\n",
       "   'two': 'present',\n",
       "   'teen': 'present',\n",
       "   'couples': 'present',\n",
       "   'go': 'present',\n",
       "   'church': 'present',\n",
       "   'party': 'present',\n",
       "   'drink': 'present',\n",
       "   'drive': 'present',\n",
       "   'get': 'present',\n",
       "   'accident': 'present',\n",
       "   'one': 'present',\n",
       "   'guys': 'present',\n",
       "   'dies': 'present',\n",
       "   'girlfriend': 'present',\n",
       "   'continues': 'present',\n",
       "   'see': 'present',\n",
       "   'life': 'present',\n",
       "   'nightmares': 'present',\n",
       "   'deal': 'present',\n",
       "   'watch': 'present',\n",
       "   'movie': 'present',\n",
       "   'sorta': 'present',\n",
       "   'find': 'present',\n",
       "   'critique': 'present',\n",
       "   'mind': 'present',\n",
       "   'fuck': 'present',\n",
       "   'generation': 'present',\n",
       "   'touches': 'present',\n",
       "   'cool': 'present',\n",
       "   'idea': 'present',\n",
       "   'presents': 'present',\n",
       "   'bad': 'present',\n",
       "   'package': 'present',\n",
       "   'makes': 'present',\n",
       "   'review': 'present',\n",
       "   'even': 'present',\n",
       "   'harder': 'present',\n",
       "   'write': 'present',\n",
       "   'since': 'present',\n",
       "   'generally': 'present',\n",
       "   'applaud': 'present',\n",
       "   'films': 'present',\n",
       "   'attempt': 'present',\n",
       "   'break': 'present',\n",
       "   'mold': 'present',\n",
       "   'mess': 'present',\n",
       "   'head': 'present',\n",
       "   'lost': 'present',\n",
       "   'highway': 'present',\n",
       "   'memento': 'present',\n",
       "   'good': 'present',\n",
       "   'ways': 'present',\n",
       "   'making': 'present',\n",
       "   'types': 'present',\n",
       "   'folks': 'present',\n",
       "   'snag': 'present',\n",
       "   'correctly': 'present',\n",
       "   'seem': 'present',\n",
       "   'taken': 'present',\n",
       "   'pretty': 'present',\n",
       "   'neat': 'present',\n",
       "   'concept': 'present',\n",
       "   'executed': 'present',\n",
       "   'terribly': 'present',\n",
       "   'problems': 'present',\n",
       "   'well': 'present',\n",
       "   'main': 'present',\n",
       "   'problem': 'present',\n",
       "   'simply': 'present',\n",
       "   'jumbled': 'present',\n",
       "   'starts': 'present',\n",
       "   'normal': 'present',\n",
       "   'downshifts': 'present',\n",
       "   'fantasy': 'present',\n",
       "   'world': 'present',\n",
       "   'audience': 'present',\n",
       "   'member': 'present',\n",
       "   'going': 'present',\n",
       "   'dreams': 'present',\n",
       "   'characters': 'present',\n",
       "   'coming': 'present',\n",
       "   'back': 'present',\n",
       "   'dead': 'present',\n",
       "   'others': 'present',\n",
       "   'look': 'present',\n",
       "   'like': 'present',\n",
       "   'strange': 'present',\n",
       "   'apparitions': 'present',\n",
       "   'disappearances': 'present',\n",
       "   'looooot': 'present',\n",
       "   'chase': 'present',\n",
       "   'scenes': 'present',\n",
       "   'tons': 'present',\n",
       "   'weird': 'present',\n",
       "   'things': 'present',\n",
       "   'happen': 'present',\n",
       "   'explained': 'present',\n",
       "   'personally': 'present',\n",
       "   'trying': 'present',\n",
       "   'unravel': 'present',\n",
       "   'film': 'present',\n",
       "   'every': 'present',\n",
       "   'give': 'present',\n",
       "   'clue': 'present',\n",
       "   'kind': 'present',\n",
       "   'fed': 'present',\n",
       "   'biggest': 'present',\n",
       "   'obviously': 'present',\n",
       "   'got': 'present',\n",
       "   'big': 'present',\n",
       "   'secret': 'present',\n",
       "   'hide': 'present',\n",
       "   'seems': 'present',\n",
       "   'want': 'present',\n",
       "   'completely': 'present',\n",
       "   'final': 'present',\n",
       "   'five': 'present',\n",
       "   'minutes': 'present',\n",
       "   'make': 'present',\n",
       "   'entertaining': 'present',\n",
       "   'thrilling': 'present',\n",
       "   'engaging': 'present',\n",
       "   'meantime': 'present',\n",
       "   'really': 'present',\n",
       "   'sad': 'present',\n",
       "   'part': 'present',\n",
       "   'arrow': 'present',\n",
       "   'dig': 'present',\n",
       "   'flicks': 'present',\n",
       "   'actually': 'present',\n",
       "   'figured': 'present',\n",
       "   'half': 'present',\n",
       "   'way': 'present',\n",
       "   'point': 'present',\n",
       "   'strangeness': 'present',\n",
       "   'start': 'present',\n",
       "   'little': 'present',\n",
       "   'bit': 'present',\n",
       "   'sense': 'present',\n",
       "   'still': 'present',\n",
       "   'guess': 'present',\n",
       "   'bottom': 'present',\n",
       "   'line': 'present',\n",
       "   'movies': 'present',\n",
       "   'always': 'present',\n",
       "   'sure': 'present',\n",
       "   'given': 'present',\n",
       "   'password': 'present',\n",
       "   'enter': 'present',\n",
       "   'understanding': 'present',\n",
       "   'mean': 'present',\n",
       "   'showing': 'present',\n",
       "   'melissa': 'present',\n",
       "   'sagemiller': 'present',\n",
       "   'running': 'present',\n",
       "   'away': 'present',\n",
       "   'visions': 'present',\n",
       "   '20': 'present',\n",
       "   'throughout': 'present',\n",
       "   'plain': 'present',\n",
       "   'lazy': 'present',\n",
       "   'okay': 'present',\n",
       "   'people': 'present',\n",
       "   'chasing': 'present',\n",
       "   'know': 'present',\n",
       "   'need': 'present',\n",
       "   'giving': 'present',\n",
       "   'us': 'present',\n",
       "   'different': 'present',\n",
       "   'offering': 'present',\n",
       "   'insight': 'present',\n",
       "   'apparently': 'present',\n",
       "   'studio': 'present',\n",
       "   'took': 'present',\n",
       "   'director': 'present',\n",
       "   'chopped': 'present',\n",
       "   'shows': 'present',\n",
       "   'might': 'present',\n",
       "   'decent': 'present',\n",
       "   'somewhere': 'present',\n",
       "   'suits': 'present',\n",
       "   'decided': 'present',\n",
       "   'turning': 'present',\n",
       "   'music': 'present',\n",
       "   'video': 'present',\n",
       "   'edge': 'present',\n",
       "   'would': 'present',\n",
       "   'actors': 'present',\n",
       "   'although': 'present',\n",
       "   'wes': 'present',\n",
       "   'bentley': 'present',\n",
       "   'seemed': 'present',\n",
       "   'playing': 'present',\n",
       "   'exact': 'present',\n",
       "   'character': 'present',\n",
       "   'american': 'present',\n",
       "   'beauty': 'present',\n",
       "   'new': 'present',\n",
       "   'neighborhood': 'present',\n",
       "   'kudos': 'present',\n",
       "   'holds': 'present',\n",
       "   'entire': 'present',\n",
       "   'feeling': 'present',\n",
       "   'unraveling': 'present',\n",
       "   'overall': 'present',\n",
       "   'stick': 'present',\n",
       "   'entertain': 'present',\n",
       "   'confusing': 'present',\n",
       "   'rarely': 'present',\n",
       "   'excites': 'present',\n",
       "   'feels': 'present',\n",
       "   'redundant': 'present',\n",
       "   'runtime': 'present',\n",
       "   'despite': 'present',\n",
       "   'ending': 'present',\n",
       "   'explanation': 'present',\n",
       "   'craziness': 'present',\n",
       "   'came': 'present',\n",
       "   'oh': 'present',\n",
       "   'horror': 'present',\n",
       "   'slasher': 'present',\n",
       "   'flick': 'present',\n",
       "   'packaged': 'present',\n",
       "   'someone': 'present',\n",
       "   'assuming': 'present',\n",
       "   'genre': 'present',\n",
       "   'hot': 'present',\n",
       "   'kids': 'present',\n",
       "   'also': 'present',\n",
       "   'wrapped': 'present',\n",
       "   'production': 'present',\n",
       "   'years': 'present',\n",
       "   'ago': 'present',\n",
       "   'sitting': 'present',\n",
       "   'shelves': 'present',\n",
       "   'ever': 'present',\n",
       "   'whatever': 'present',\n",
       "   'skip': 'present',\n",
       "   'joblo': 'present',\n",
       "   'nightmare': 'present',\n",
       "   'elm': 'present',\n",
       "   'street': 'present',\n",
       "   '3': 'present',\n",
       "   '7': 'present',\n",
       "   '10': 'present',\n",
       "   'blair': 'present',\n",
       "   'witch': 'present',\n",
       "   '2': 'present',\n",
       "   'crow': 'present',\n",
       "   '9': 'present',\n",
       "   'salvation': 'present',\n",
       "   '4': 'present',\n",
       "   'stir': 'present',\n",
       "   'echoes': 'present',\n",
       "   '8': 'present'},\n",
       "  'neg'),\n",
       " ({'happy': 'present',\n",
       "   'bastard': 'present',\n",
       "   'quick': 'present',\n",
       "   'movie': 'present',\n",
       "   'review': 'present',\n",
       "   'damn': 'present',\n",
       "   'y2k': 'present',\n",
       "   'bug': 'present',\n",
       "   'got': 'present',\n",
       "   'head': 'present',\n",
       "   'start': 'present',\n",
       "   'starring': 'present',\n",
       "   'jamie': 'present',\n",
       "   'lee': 'present',\n",
       "   'curtis': 'present',\n",
       "   'another': 'present',\n",
       "   'baldwin': 'present',\n",
       "   'brother': 'present',\n",
       "   'william': 'present',\n",
       "   'time': 'present',\n",
       "   'story': 'present',\n",
       "   'regarding': 'present',\n",
       "   'crew': 'present',\n",
       "   'tugboat': 'present',\n",
       "   'comes': 'present',\n",
       "   'across': 'present',\n",
       "   'deserted': 'present',\n",
       "   'russian': 'present',\n",
       "   'tech': 'present',\n",
       "   'ship': 'present',\n",
       "   'strangeness': 'present',\n",
       "   'kick': 'present',\n",
       "   'power': 'present',\n",
       "   'back': 'present',\n",
       "   'little': 'present',\n",
       "   'know': 'present',\n",
       "   'within': 'present',\n",
       "   'going': 'present',\n",
       "   'gore': 'present',\n",
       "   'bringing': 'present',\n",
       "   'action': 'present',\n",
       "   'sequences': 'present',\n",
       "   'virus': 'present',\n",
       "   'still': 'present',\n",
       "   'feels': 'present',\n",
       "   'empty': 'present',\n",
       "   'like': 'present',\n",
       "   'flash': 'present',\n",
       "   'substance': 'present',\n",
       "   'really': 'present',\n",
       "   'middle': 'present',\n",
       "   'nowhere': 'present',\n",
       "   'origin': 'present',\n",
       "   'took': 'present',\n",
       "   'big': 'present',\n",
       "   'pink': 'present',\n",
       "   'flashy': 'present',\n",
       "   'thing': 'present',\n",
       "   'hit': 'present',\n",
       "   'mir': 'present',\n",
       "   'course': 'present',\n",
       "   'donald': 'present',\n",
       "   'sutherland': 'present',\n",
       "   'stumbling': 'present',\n",
       "   'around': 'present',\n",
       "   'drunkenly': 'present',\n",
       "   'throughout': 'present',\n",
       "   'hey': 'present',\n",
       "   'let': 'present',\n",
       "   'chase': 'present',\n",
       "   'people': 'present',\n",
       "   'robots': 'present',\n",
       "   'acting': 'present',\n",
       "   'average': 'present',\n",
       "   'even': 'present',\n",
       "   'likes': 'present',\n",
       "   'likely': 'present',\n",
       "   'get': 'present',\n",
       "   'work': 'present',\n",
       "   'halloween': 'present',\n",
       "   'h20': 'present',\n",
       "   'wasted': 'present',\n",
       "   'well': 'present',\n",
       "   'real': 'present',\n",
       "   'star': 'present',\n",
       "   'stan': 'present',\n",
       "   'winston': 'present',\n",
       "   'robot': 'present',\n",
       "   'design': 'present',\n",
       "   'schnazzy': 'present',\n",
       "   'cgi': 'present',\n",
       "   'occasional': 'present',\n",
       "   'good': 'present',\n",
       "   'shot': 'present',\n",
       "   'picking': 'present',\n",
       "   'someone': 'present',\n",
       "   'brain': 'present',\n",
       "   'body': 'present',\n",
       "   'parts': 'present',\n",
       "   'turn': 'present',\n",
       "   'otherwise': 'present',\n",
       "   'pretty': 'present',\n",
       "   'much': 'present',\n",
       "   'sunken': 'present'},\n",
       "  'neg'),\n",
       " ({'movies': 'present',\n",
       "   'like': 'present',\n",
       "   'make': 'present',\n",
       "   'jaded': 'present',\n",
       "   'movie': 'present',\n",
       "   'viewer': 'present',\n",
       "   'thankful': 'present',\n",
       "   'invention': 'present',\n",
       "   'timex': 'present',\n",
       "   'indiglo': 'present',\n",
       "   'watch': 'present',\n",
       "   'based': 'present',\n",
       "   'late': 'present',\n",
       "   '1960': 'present',\n",
       "   'television': 'present',\n",
       "   'show': 'present',\n",
       "   'name': 'present',\n",
       "   'mod': 'present',\n",
       "   'squad': 'present',\n",
       "   'tells': 'present',\n",
       "   'tale': 'present',\n",
       "   'three': 'present',\n",
       "   'reformed': 'present',\n",
       "   'criminals': 'present',\n",
       "   'employ': 'present',\n",
       "   'police': 'present',\n",
       "   'go': 'present',\n",
       "   'undercover': 'present',\n",
       "   'however': 'present',\n",
       "   'things': 'present',\n",
       "   'wrong': 'present',\n",
       "   'evidence': 'present',\n",
       "   'gets': 'present',\n",
       "   'stolen': 'present',\n",
       "   'immediately': 'present',\n",
       "   'suspicion': 'present',\n",
       "   'course': 'present',\n",
       "   'ads': 'present',\n",
       "   'seem': 'present',\n",
       "   'much': 'present',\n",
       "   'quick': 'present',\n",
       "   'cuts': 'present',\n",
       "   'cool': 'present',\n",
       "   'music': 'present',\n",
       "   'claire': 'present',\n",
       "   'dane': 'present',\n",
       "   'nice': 'present',\n",
       "   'hair': 'present',\n",
       "   'cute': 'present',\n",
       "   'outfits': 'present',\n",
       "   'car': 'present',\n",
       "   'chases': 'present',\n",
       "   'stuff': 'present',\n",
       "   'blowing': 'present',\n",
       "   'sounds': 'present',\n",
       "   'first': 'present',\n",
       "   'fifteen': 'present',\n",
       "   'minutes': 'present',\n",
       "   'quickly': 'present',\n",
       "   'becomes': 'present',\n",
       "   'apparent': 'present',\n",
       "   'certainly': 'present',\n",
       "   'slick': 'present',\n",
       "   'looking': 'present',\n",
       "   'production': 'present',\n",
       "   'complete': 'present',\n",
       "   'costumes': 'present',\n",
       "   'simply': 'present',\n",
       "   'enough': 'present',\n",
       "   'film': 'present',\n",
       "   'best': 'present',\n",
       "   'described': 'present',\n",
       "   'cross': 'present',\n",
       "   'hour': 'present',\n",
       "   'long': 'present',\n",
       "   'cop': 'present',\n",
       "   'video': 'present',\n",
       "   'stretched': 'present',\n",
       "   'span': 'present',\n",
       "   'half': 'present',\n",
       "   'comes': 'present',\n",
       "   'every': 'present',\n",
       "   'single': 'present',\n",
       "   'clich': 'present',\n",
       "   'really': 'present',\n",
       "   'matter': 'present',\n",
       "   'plot': 'present',\n",
       "   'elements': 'present',\n",
       "   'recycled': 'present',\n",
       "   'everything': 'present',\n",
       "   'already': 'present',\n",
       "   'seen': 'present',\n",
       "   'characters': 'present',\n",
       "   'acting': 'present',\n",
       "   'nothing': 'present',\n",
       "   'spectacular': 'present',\n",
       "   'sometimes': 'present',\n",
       "   'even': 'present',\n",
       "   'bordering': 'present',\n",
       "   'wooden': 'present',\n",
       "   'danes': 'present',\n",
       "   'omar': 'present',\n",
       "   'epps': 'present',\n",
       "   'deliver': 'present',\n",
       "   'lines': 'present',\n",
       "   'bored': 'present',\n",
       "   'transfers': 'present',\n",
       "   'onto': 'present',\n",
       "   'audience': 'present',\n",
       "   'one': 'present',\n",
       "   'escape': 'present',\n",
       "   'relatively': 'present',\n",
       "   'unscathed': 'present',\n",
       "   'giovanni': 'present',\n",
       "   'ribisi': 'present',\n",
       "   'plays': 'present',\n",
       "   'resident': 'present',\n",
       "   'crazy': 'present',\n",
       "   'man': 'present',\n",
       "   'ultimately': 'present',\n",
       "   'thing': 'present',\n",
       "   'worth': 'present',\n",
       "   'watching': 'present',\n",
       "   'unfortunately': 'present',\n",
       "   'save': 'present',\n",
       "   'convoluted': 'present',\n",
       "   'mess': 'present',\n",
       "   'apart': 'present',\n",
       "   'occupying': 'present',\n",
       "   'screen': 'present',\n",
       "   'time': 'present',\n",
       "   'young': 'present',\n",
       "   'cast': 'present',\n",
       "   'clothes': 'present',\n",
       "   'hip': 'present',\n",
       "   'soundtrack': 'present',\n",
       "   'appears': 'present',\n",
       "   'geared': 'present',\n",
       "   'towards': 'present',\n",
       "   'teenage': 'present',\n",
       "   'mindset': 'present',\n",
       "   'despite': 'present',\n",
       "   'american': 'present',\n",
       "   'r': 'present',\n",
       "   'rating': 'present',\n",
       "   'content': 'present',\n",
       "   'justify': 'present',\n",
       "   'way': 'present',\n",
       "   'juvenile': 'present',\n",
       "   'older': 'present',\n",
       "   'information': 'present',\n",
       "   'literally': 'present',\n",
       "   'spoon': 'present',\n",
       "   'fed': 'present',\n",
       "   'would': 'present',\n",
       "   'hard': 'present',\n",
       "   'us': 'present',\n",
       "   'instead': 'present',\n",
       "   'telling': 'present',\n",
       "   'dialogue': 'present',\n",
       "   'poorly': 'present',\n",
       "   'written': 'present',\n",
       "   'extremely': 'present',\n",
       "   'predictable': 'present',\n",
       "   'progresses': 'present',\n",
       "   'likely': 'present',\n",
       "   'care': 'present',\n",
       "   'heroes': 'present',\n",
       "   'jeopardy': 'present',\n",
       "   'know': 'present',\n",
       "   'basing': 'present',\n",
       "   'nobody': 'present',\n",
       "   'remembers': 'present',\n",
       "   'questionable': 'present',\n",
       "   'wisdom': 'present',\n",
       "   'especially': 'present',\n",
       "   'considers': 'present',\n",
       "   'target': 'present',\n",
       "   'fact': 'present',\n",
       "   'number': 'present',\n",
       "   'memorable': 'present',\n",
       "   'films': 'present',\n",
       "   'shows': 'present',\n",
       "   'counted': 'present',\n",
       "   'hand': 'present',\n",
       "   'missing': 'present',\n",
       "   'finger': 'present',\n",
       "   'two': 'present',\n",
       "   'times': 'present',\n",
       "   'checked': 'present',\n",
       "   'six': 'present',\n",
       "   'clear': 'present',\n",
       "   'indication': 'present',\n",
       "   'attempt': 'present',\n",
       "   'cash': 'present',\n",
       "   'spending': 'present',\n",
       "   'dollar': 'present',\n",
       "   'judging': 'present',\n",
       "   'rash': 'present',\n",
       "   'awful': 'present',\n",
       "   'teen': 'present',\n",
       "   'flicks': 'present',\n",
       "   'seeing': 'present',\n",
       "   'avoid': 'present',\n",
       "   'costs': 'present'},\n",
       "  'neg'),\n",
       " ({'quest': 'present',\n",
       "   'camelot': 'present',\n",
       "   'warner': 'present',\n",
       "   'bros': 'present',\n",
       "   'first': 'present',\n",
       "   'feature': 'present',\n",
       "   'length': 'present',\n",
       "   'fully': 'present',\n",
       "   'animated': 'present',\n",
       "   'attempt': 'present',\n",
       "   'steal': 'present',\n",
       "   'clout': 'present',\n",
       "   'disney': 'present',\n",
       "   'cartoon': 'present',\n",
       "   'empire': 'present',\n",
       "   'mouse': 'present',\n",
       "   'reason': 'present',\n",
       "   'worried': 'present',\n",
       "   'recent': 'present',\n",
       "   'challenger': 'present',\n",
       "   'throne': 'present',\n",
       "   'last': 'present',\n",
       "   'fall': 'present',\n",
       "   'promising': 'present',\n",
       "   'flawed': 'present',\n",
       "   '20th': 'present',\n",
       "   'century': 'present',\n",
       "   'fox': 'present',\n",
       "   'production': 'present',\n",
       "   'anastasia': 'present',\n",
       "   'hercules': 'present',\n",
       "   'lively': 'present',\n",
       "   'cast': 'present',\n",
       "   'colorful': 'present',\n",
       "   'palate': 'present',\n",
       "   'beat': 'present',\n",
       "   'hands': 'present',\n",
       "   'came': 'present',\n",
       "   'time': 'present',\n",
       "   'crown': 'present',\n",
       "   '1997': 'present',\n",
       "   'best': 'present',\n",
       "   'piece': 'present',\n",
       "   'animation': 'present',\n",
       "   'year': 'present',\n",
       "   'contest': 'present',\n",
       "   'pretty': 'present',\n",
       "   'much': 'present',\n",
       "   'dead': 'present',\n",
       "   'arrival': 'present',\n",
       "   'even': 'present',\n",
       "   'magic': 'present',\n",
       "   'kingdom': 'present',\n",
       "   'mediocre': 'present',\n",
       "   '--': 'present',\n",
       "   'pocahontas': 'present',\n",
       "   'keeping': 'present',\n",
       "   'score': 'present',\n",
       "   'nearly': 'present',\n",
       "   'dull': 'present',\n",
       "   'story': 'present',\n",
       "   'revolves': 'present',\n",
       "   'around': 'present',\n",
       "   'adventures': 'present',\n",
       "   'free': 'present',\n",
       "   'spirited': 'present',\n",
       "   'kayley': 'present',\n",
       "   'voiced': 'present',\n",
       "   'jessalyn': 'present',\n",
       "   'gilsig': 'present',\n",
       "   'early': 'present',\n",
       "   'teen': 'present',\n",
       "   'daughter': 'present',\n",
       "   'belated': 'present',\n",
       "   'knight': 'present',\n",
       "   'king': 'present',\n",
       "   'arthur': 'present',\n",
       "   'round': 'present',\n",
       "   'table': 'present',\n",
       "   'dream': 'present',\n",
       "   'follow': 'present',\n",
       "   'father': 'present',\n",
       "   'footsteps': 'present',\n",
       "   'gets': 'present',\n",
       "   'chance': 'present',\n",
       "   'evil': 'present',\n",
       "   'warlord': 'present',\n",
       "   'ruber': 'present',\n",
       "   'gary': 'present',\n",
       "   'oldman': 'present',\n",
       "   'ex': 'present',\n",
       "   'member': 'present',\n",
       "   'gone': 'present',\n",
       "   'bad': 'present',\n",
       "   'steals': 'present',\n",
       "   'magical': 'present',\n",
       "   'sword': 'present',\n",
       "   'excalibur': 'present',\n",
       "   'accidentally': 'present',\n",
       "   'loses': 'present',\n",
       "   'dangerous': 'present',\n",
       "   'booby': 'present',\n",
       "   'trapped': 'present',\n",
       "   'forest': 'present',\n",
       "   'help': 'present',\n",
       "   'hunky': 'present',\n",
       "   'blind': 'present',\n",
       "   'timberland': 'present',\n",
       "   'dweller': 'present',\n",
       "   'garrett': 'present',\n",
       "   'carey': 'present',\n",
       "   'elwes': 'present',\n",
       "   'two': 'present',\n",
       "   'headed': 'present',\n",
       "   'dragon': 'present',\n",
       "   'eric': 'present',\n",
       "   'idle': 'present',\n",
       "   'rickles': 'present',\n",
       "   'always': 'present',\n",
       "   'arguing': 'present',\n",
       "   'might': 'present',\n",
       "   'able': 'present',\n",
       "   'break': 'present',\n",
       "   'medieval': 'present',\n",
       "   'sexist': 'present',\n",
       "   'mold': 'present',\n",
       "   'prove': 'present',\n",
       "   'worth': 'present',\n",
       "   'fighter': 'present',\n",
       "   'side': 'present',\n",
       "   'missing': 'present',\n",
       "   'pure': 'present',\n",
       "   'showmanship': 'present',\n",
       "   'essential': 'present',\n",
       "   'element': 'present',\n",
       "   'ever': 'present',\n",
       "   'expected': 'present',\n",
       "   'climb': 'present',\n",
       "   'high': 'present',\n",
       "   'ranks': 'present',\n",
       "   'nothing': 'present',\n",
       "   'differentiates': 'present',\n",
       "   'something': 'present',\n",
       "   'see': 'present',\n",
       "   'given': 'present',\n",
       "   'saturday': 'present',\n",
       "   'morning': 'present',\n",
       "   'subpar': 'present',\n",
       "   'instantly': 'present',\n",
       "   'forgettable': 'present',\n",
       "   'songs': 'present',\n",
       "   'poorly': 'present',\n",
       "   'integrated': 'present',\n",
       "   'computerized': 'present',\n",
       "   'footage': 'present',\n",
       "   'compare': 'present',\n",
       "   'run': 'present',\n",
       "   'angry': 'present',\n",
       "   'ogre': 'present',\n",
       "   'herc': 'present',\n",
       "   'battle': 'present',\n",
       "   'hydra': 'present',\n",
       "   'rest': 'present',\n",
       "   'case': 'present',\n",
       "   'characters': 'present',\n",
       "   'stink': 'present',\n",
       "   'none': 'present',\n",
       "   'remotely': 'present',\n",
       "   'interesting': 'present',\n",
       "   'film': 'present',\n",
       "   'becomes': 'present',\n",
       "   'race': 'present',\n",
       "   'one': 'present',\n",
       "   'bland': 'present',\n",
       "   'others': 'present',\n",
       "   'end': 'present',\n",
       "   'tie': 'present',\n",
       "   'win': 'present',\n",
       "   'comedy': 'present',\n",
       "   'shtick': 'present',\n",
       "   'awfully': 'present',\n",
       "   'cloying': 'present',\n",
       "   'least': 'present',\n",
       "   'shows': 'present',\n",
       "   'signs': 'present',\n",
       "   'pulse': 'present',\n",
       "   'fans': 'present',\n",
       "   \"-'\": 'present',\n",
       "   '90s': 'present',\n",
       "   'tgif': 'present',\n",
       "   'television': 'present',\n",
       "   'line': 'present',\n",
       "   'thrilled': 'present',\n",
       "   'find': 'present',\n",
       "   'jaleel': 'present',\n",
       "   'urkel': 'present',\n",
       "   'white': 'present',\n",
       "   'bronson': 'present',\n",
       "   'balki': 'present',\n",
       "   'pinchot': 'present',\n",
       "   'sharing': 'present',\n",
       "   'scenes': 'present',\n",
       "   'nicely': 'present',\n",
       "   'realized': 'present',\n",
       "   'though': 'present',\n",
       "   'loss': 'present',\n",
       "   'recall': 'present',\n",
       "   'enough': 'present',\n",
       "   'specific': 'present',\n",
       "   'actors': 'present',\n",
       "   'providing': 'present',\n",
       "   'voice': 'present',\n",
       "   'talent': 'present',\n",
       "   'enthusiastic': 'present',\n",
       "   'paired': 'present',\n",
       "   'singers': 'present',\n",
       "   'sound': 'present',\n",
       "   'thing': 'present',\n",
       "   'like': 'present',\n",
       "   'big': 'present',\n",
       "   'musical': 'present',\n",
       "   'moments': 'present',\n",
       "   'jane': 'present',\n",
       "   'seymour': 'present',\n",
       "   'celine': 'present',\n",
       "   'dion': 'present',\n",
       "   'must': 'present',\n",
       "   'strain': 'present',\n",
       "   'mess': 'present',\n",
       "   'good': 'present',\n",
       "   'aside': 'present',\n",
       "   'fact': 'present',\n",
       "   'children': 'present',\n",
       "   'probably': 'present',\n",
       "   'bored': 'present',\n",
       "   'watching': 'present',\n",
       "   'adults': 'present',\n",
       "   'grievous': 'present',\n",
       "   'error': 'present',\n",
       "   'complete': 'present',\n",
       "   'lack': 'present',\n",
       "   'personality': 'present',\n",
       "   'learn': 'present',\n",
       "   'goes': 'present',\n",
       "   'long': 'present',\n",
       "   'way': 'present'},\n",
       "  'neg')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neg_words[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d1476043",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[({'films': 'present',\n",
       "   'adapted': 'present',\n",
       "   'comic': 'present',\n",
       "   'books': 'present',\n",
       "   'plenty': 'present',\n",
       "   'success': 'present',\n",
       "   'whether': 'present',\n",
       "   'superheroes': 'present',\n",
       "   'batman': 'present',\n",
       "   'superman': 'present',\n",
       "   'spawn': 'present',\n",
       "   'geared': 'present',\n",
       "   'toward': 'present',\n",
       "   'kids': 'present',\n",
       "   'casper': 'present',\n",
       "   'arthouse': 'present',\n",
       "   'crowd': 'present',\n",
       "   'ghost': 'present',\n",
       "   'world': 'present',\n",
       "   'never': 'present',\n",
       "   'really': 'present',\n",
       "   'book': 'present',\n",
       "   'like': 'present',\n",
       "   'hell': 'present',\n",
       "   'starters': 'present',\n",
       "   'created': 'present',\n",
       "   'alan': 'present',\n",
       "   'moore': 'present',\n",
       "   'eddie': 'present',\n",
       "   'campbell': 'present',\n",
       "   'brought': 'present',\n",
       "   'medium': 'present',\n",
       "   'whole': 'present',\n",
       "   'new': 'present',\n",
       "   'level': 'present',\n",
       "   'mid': 'present',\n",
       "   '80s': 'present',\n",
       "   '12': 'present',\n",
       "   'part': 'present',\n",
       "   'series': 'present',\n",
       "   'called': 'present',\n",
       "   'watchmen': 'present',\n",
       "   'say': 'present',\n",
       "   'thoroughly': 'present',\n",
       "   'researched': 'present',\n",
       "   'subject': 'present',\n",
       "   'jack': 'present',\n",
       "   'ripper': 'present',\n",
       "   'would': 'present',\n",
       "   'saying': 'present',\n",
       "   'michael': 'present',\n",
       "   'jackson': 'present',\n",
       "   'starting': 'present',\n",
       "   'look': 'present',\n",
       "   'little': 'present',\n",
       "   'odd': 'present',\n",
       "   'graphic': 'present',\n",
       "   'novel': 'present',\n",
       "   '500': 'present',\n",
       "   'pages': 'present',\n",
       "   'long': 'present',\n",
       "   'includes': 'present',\n",
       "   'nearly': 'present',\n",
       "   '30': 'present',\n",
       "   'consist': 'present',\n",
       "   'nothing': 'present',\n",
       "   'footnotes': 'present',\n",
       "   'words': 'present',\n",
       "   'dismiss': 'present',\n",
       "   'film': 'present',\n",
       "   'source': 'present',\n",
       "   'get': 'present',\n",
       "   'past': 'present',\n",
       "   'thing': 'present',\n",
       "   'might': 'present',\n",
       "   'find': 'present',\n",
       "   'another': 'present',\n",
       "   'stumbling': 'present',\n",
       "   'block': 'present',\n",
       "   'directors': 'present',\n",
       "   'albert': 'present',\n",
       "   'allen': 'present',\n",
       "   'hughes': 'present',\n",
       "   'getting': 'present',\n",
       "   'brothers': 'present',\n",
       "   'direct': 'present',\n",
       "   'seems': 'present',\n",
       "   'almost': 'present',\n",
       "   'ludicrous': 'present',\n",
       "   'casting': 'present',\n",
       "   'carrot': 'present',\n",
       "   'top': 'present',\n",
       "   'well': 'present',\n",
       "   'anything': 'present',\n",
       "   'riddle': 'present',\n",
       "   'better': 'present',\n",
       "   'set': 'present',\n",
       "   'ghetto': 'present',\n",
       "   'features': 'present',\n",
       "   'violent': 'present',\n",
       "   'street': 'present',\n",
       "   'crime': 'present',\n",
       "   'mad': 'present',\n",
       "   'geniuses': 'present',\n",
       "   'behind': 'present',\n",
       "   'menace': 'present',\n",
       "   'ii': 'present',\n",
       "   'society': 'present',\n",
       "   'question': 'present',\n",
       "   'course': 'present',\n",
       "   'whitechapel': 'present',\n",
       "   '1888': 'present',\n",
       "   'london': 'present',\n",
       "   'east': 'present',\n",
       "   'end': 'present',\n",
       "   'filthy': 'present',\n",
       "   'sooty': 'present',\n",
       "   'place': 'present',\n",
       "   'whores': 'present',\n",
       "   'unfortunates': 'present',\n",
       "   'nervous': 'present',\n",
       "   'mysterious': 'present',\n",
       "   'psychopath': 'present',\n",
       "   'carving': 'present',\n",
       "   'profession': 'present',\n",
       "   'surgical': 'present',\n",
       "   'precision': 'present',\n",
       "   'first': 'present',\n",
       "   'stiff': 'present',\n",
       "   'turns': 'present',\n",
       "   'copper': 'present',\n",
       "   'peter': 'present',\n",
       "   'godley': 'present',\n",
       "   'robbie': 'present',\n",
       "   'coltrane': 'present',\n",
       "   'enough': 'present',\n",
       "   'calls': 'present',\n",
       "   'inspector': 'present',\n",
       "   'frederick': 'present',\n",
       "   'abberline': 'present',\n",
       "   'johnny': 'present',\n",
       "   'depp': 'present',\n",
       "   'blow': 'present',\n",
       "   'crack': 'present',\n",
       "   'case': 'present',\n",
       "   'widower': 'present',\n",
       "   'prophetic': 'present',\n",
       "   'dreams': 'present',\n",
       "   'unsuccessfully': 'present',\n",
       "   'tries': 'present',\n",
       "   'quell': 'present',\n",
       "   'copious': 'present',\n",
       "   'amounts': 'present',\n",
       "   'absinthe': 'present',\n",
       "   'opium': 'present',\n",
       "   'upon': 'present',\n",
       "   'arriving': 'present',\n",
       "   'befriends': 'present',\n",
       "   'unfortunate': 'present',\n",
       "   'named': 'present',\n",
       "   'mary': 'present',\n",
       "   'kelly': 'present',\n",
       "   'heather': 'present',\n",
       "   'graham': 'present',\n",
       "   'proceeds': 'present',\n",
       "   'investigate': 'present',\n",
       "   'horribly': 'present',\n",
       "   'gruesome': 'present',\n",
       "   'crimes': 'present',\n",
       "   'even': 'present',\n",
       "   'police': 'present',\n",
       "   'surgeon': 'present',\n",
       "   'stomach': 'present',\n",
       "   'think': 'present',\n",
       "   'anyone': 'present',\n",
       "   'needs': 'present',\n",
       "   'briefed': 'present',\n",
       "   'go': 'present',\n",
       "   'particulars': 'present',\n",
       "   'unique': 'present',\n",
       "   'interesting': 'present',\n",
       "   'theory': 'present',\n",
       "   'identity': 'present',\n",
       "   'killer': 'present',\n",
       "   'reasons': 'present',\n",
       "   'chooses': 'present',\n",
       "   'slay': 'present',\n",
       "   'bother': 'present',\n",
       "   'cloaking': 'present',\n",
       "   'screenwriters': 'present',\n",
       "   'terry': 'present',\n",
       "   'hayes': 'present',\n",
       "   'vertical': 'present',\n",
       "   'limit': 'present',\n",
       "   'rafael': 'present',\n",
       "   'yglesias': 'present',\n",
       "   'les': 'present',\n",
       "   'mis': 'present',\n",
       "   'rables': 'present',\n",
       "   'good': 'present',\n",
       "   'job': 'present',\n",
       "   'keeping': 'present',\n",
       "   'hidden': 'present',\n",
       "   'viewers': 'present',\n",
       "   'funny': 'present',\n",
       "   'watch': 'present',\n",
       "   'locals': 'present',\n",
       "   'blindly': 'present',\n",
       "   'point': 'present',\n",
       "   'finger': 'present',\n",
       "   'blame': 'present',\n",
       "   'jews': 'present',\n",
       "   'indians': 'present',\n",
       "   'englishman': 'present',\n",
       "   'could': 'present',\n",
       "   'capable': 'present',\n",
       "   'committing': 'present',\n",
       "   'ghastly': 'present',\n",
       "   'acts': 'present',\n",
       "   'ending': 'present',\n",
       "   'whistling': 'present',\n",
       "   'stonecutters': 'present',\n",
       "   'song': 'present',\n",
       "   'simpsons': 'present',\n",
       "   'days': 'present',\n",
       "   'holds': 'present',\n",
       "   'back': 'present',\n",
       "   'electric': 'present',\n",
       "   'car': 'present',\n",
       "   'made': 'present',\n",
       "   'steve': 'present',\n",
       "   'guttenberg': 'present',\n",
       "   'star': 'present',\n",
       "   'worry': 'present',\n",
       "   'make': 'present',\n",
       "   'sense': 'present',\n",
       "   'see': 'present',\n",
       "   'onto': 'present',\n",
       "   'appearance': 'present',\n",
       "   'certainly': 'present',\n",
       "   'dark': 'present',\n",
       "   'bleak': 'present',\n",
       "   'surprising': 'present',\n",
       "   'much': 'present',\n",
       "   'looks': 'present',\n",
       "   'tim': 'present',\n",
       "   'burton': 'present',\n",
       "   'planet': 'present',\n",
       "   'apes': 'present',\n",
       "   'times': 'present',\n",
       "   'sleepy': 'present',\n",
       "   'hollow': 'present',\n",
       "   '2': 'present',\n",
       "   'print': 'present',\n",
       "   'saw': 'present',\n",
       "   'completely': 'present',\n",
       "   'finished': 'present',\n",
       "   'color': 'present',\n",
       "   'music': 'present',\n",
       "   'finalized': 'present',\n",
       "   'comments': 'present',\n",
       "   'marilyn': 'present',\n",
       "   'manson': 'present',\n",
       "   'cinematographer': 'present',\n",
       "   'deming': 'present',\n",
       "   'word': 'present',\n",
       "   'ably': 'present',\n",
       "   'captures': 'present',\n",
       "   'dreariness': 'present',\n",
       "   'victorian': 'present',\n",
       "   'era': 'present',\n",
       "   'helped': 'present',\n",
       "   'flashy': 'present',\n",
       "   'killing': 'present',\n",
       "   'scenes': 'present',\n",
       "   'remind': 'present',\n",
       "   'crazy': 'present',\n",
       "   'flashbacks': 'present',\n",
       "   'twin': 'present',\n",
       "   'peaks': 'present',\n",
       "   'though': 'present',\n",
       "   'violence': 'present',\n",
       "   'pales': 'present',\n",
       "   'comparison': 'present',\n",
       "   'black': 'present',\n",
       "   'white': 'present',\n",
       "   'oscar': 'present',\n",
       "   'winner': 'present',\n",
       "   'martin': 'present',\n",
       "   'childs': 'present',\n",
       "   'shakespeare': 'present',\n",
       "   'love': 'present',\n",
       "   'production': 'present',\n",
       "   'design': 'present',\n",
       "   'original': 'present',\n",
       "   'prague': 'present',\n",
       "   'surroundings': 'present',\n",
       "   'one': 'present',\n",
       "   'creepy': 'present',\n",
       "   'acting': 'present',\n",
       "   'solid': 'present',\n",
       "   'dreamy': 'present',\n",
       "   'turning': 'present',\n",
       "   'typically': 'present',\n",
       "   'strong': 'present',\n",
       "   'performance': 'present',\n",
       "   'deftly': 'present',\n",
       "   'handling': 'present',\n",
       "   'british': 'present',\n",
       "   'accent': 'present',\n",
       "   'ians': 'present',\n",
       "   'holm': 'present',\n",
       "   'joe': 'present',\n",
       "   'gould': 'present',\n",
       "   'secret': 'present',\n",
       "   'richardson': 'present',\n",
       "   '102': 'present',\n",
       "   'dalmatians': 'present',\n",
       "   'log': 'present',\n",
       "   'great': 'present',\n",
       "   'supporting': 'present',\n",
       "   'roles': 'present',\n",
       "   'big': 'present',\n",
       "   'surprise': 'present',\n",
       "   'cringed': 'present',\n",
       "   'time': 'present',\n",
       "   'opened': 'present',\n",
       "   'mouth': 'present',\n",
       "   'imagining': 'present',\n",
       "   'attempt': 'present',\n",
       "   'irish': 'present',\n",
       "   'actually': 'present',\n",
       "   'half': 'present',\n",
       "   'bad': 'present',\n",
       "   'however': 'present',\n",
       "   '00': 'present',\n",
       "   'r': 'present',\n",
       "   'gore': 'present',\n",
       "   'sexuality': 'present',\n",
       "   'language': 'present',\n",
       "   'drug': 'present',\n",
       "   'content': 'present'},\n",
       "  'pos'),\n",
       " ({'every': 'present',\n",
       "   'movie': 'present',\n",
       "   'comes': 'present',\n",
       "   'along': 'present',\n",
       "   'suspect': 'present',\n",
       "   'studio': 'present',\n",
       "   'indication': 'present',\n",
       "   'stinker': 'present',\n",
       "   'everybody': 'present',\n",
       "   'surprise': 'present',\n",
       "   'perhaps': 'present',\n",
       "   'even': 'present',\n",
       "   'film': 'present',\n",
       "   'becomes': 'present',\n",
       "   'critical': 'present',\n",
       "   'darling': 'present',\n",
       "   'mtv': 'present',\n",
       "   'films': 'present',\n",
       "   '_election': 'present',\n",
       "   'high': 'present',\n",
       "   'school': 'present',\n",
       "   'comedy': 'present',\n",
       "   'starring': 'present',\n",
       "   'matthew': 'present',\n",
       "   'broderick': 'present',\n",
       "   'reese': 'present',\n",
       "   'witherspoon': 'present',\n",
       "   'current': 'present',\n",
       "   'example': 'present',\n",
       "   'anybody': 'present',\n",
       "   'know': 'present',\n",
       "   'existed': 'present',\n",
       "   'week': 'present',\n",
       "   'opened': 'present',\n",
       "   'plot': 'present',\n",
       "   'deceptively': 'present',\n",
       "   'simple': 'present',\n",
       "   'george': 'present',\n",
       "   'washington': 'present',\n",
       "   'carver': 'present',\n",
       "   'student': 'present',\n",
       "   'elections': 'present',\n",
       "   'tracy': 'present',\n",
       "   'flick': 'present',\n",
       "   'achiever': 'present',\n",
       "   'hand': 'present',\n",
       "   'raised': 'present',\n",
       "   'nearly': 'present',\n",
       "   'question': 'present',\n",
       "   'way': 'present',\n",
       "   'mr': 'present',\n",
       "   'sick': 'present',\n",
       "   'megalomaniac': 'present',\n",
       "   'encourages': 'present',\n",
       "   'paul': 'present',\n",
       "   'popular': 'present',\n",
       "   'slow': 'present',\n",
       "   'jock': 'present',\n",
       "   'run': 'present',\n",
       "   'nihilistic': 'present',\n",
       "   'sister': 'present',\n",
       "   'jumps': 'present',\n",
       "   'race': 'present',\n",
       "   'well': 'present',\n",
       "   'personal': 'present',\n",
       "   'reasons': 'present',\n",
       "   'dark': 'present',\n",
       "   'side': 'present',\n",
       "   'sleeper': 'present',\n",
       "   'success': 'present',\n",
       "   'expectations': 'present',\n",
       "   'low': 'present',\n",
       "   'going': 'present',\n",
       "   'fact': 'present',\n",
       "   'quality': 'present',\n",
       "   'stuff': 'present',\n",
       "   'made': 'present',\n",
       "   'reviews': 'present',\n",
       "   'enthusiastic': 'present',\n",
       "   'right': 'present',\n",
       "   'help': 'present',\n",
       "   'baggage': 'present',\n",
       "   'glowing': 'present',\n",
       "   'contrast': 'present',\n",
       "   'negative': 'present',\n",
       "   'reviewers': 'present',\n",
       "   'likely': 'present',\n",
       "   'good': 'present',\n",
       "   'live': 'present',\n",
       "   'hype': 'present',\n",
       "   'makes': 'present',\n",
       "   '_election_': 'present',\n",
       "   'disappointing': 'present',\n",
       "   'contains': 'present',\n",
       "   'significant': 'present',\n",
       "   'details': 'present',\n",
       "   'lifted': 'present',\n",
       "   'directly': 'present',\n",
       "   '_rushmore_': 'present',\n",
       "   'released': 'present',\n",
       "   'months': 'present',\n",
       "   'earlier': 'present',\n",
       "   'similarities': 'present',\n",
       "   'staggering': 'present',\n",
       "   'president': 'present',\n",
       "   'extraordinary': 'present',\n",
       "   'number': 'present',\n",
       "   'clubs': 'present',\n",
       "   'involved': 'present',\n",
       "   'play': 'present',\n",
       "   'max': 'present',\n",
       "   'fischer': 'present',\n",
       "   'tension': 'present',\n",
       "   'potential': 'present',\n",
       "   'relationship': 'present',\n",
       "   'teacher': 'present',\n",
       "   'single': 'present',\n",
       "   'parent': 'present',\n",
       "   'home': 'present',\n",
       "   'contributed': 'present',\n",
       "   'drive': 'present',\n",
       "   'male': 'present',\n",
       "   'bumbling': 'present',\n",
       "   'adult': 'present',\n",
       "   'pursues': 'present',\n",
       "   'extramarital': 'present',\n",
       "   'affair': 'present',\n",
       "   'gets': 'present',\n",
       "   'caught': 'present',\n",
       "   'whole': 'present',\n",
       "   'life': 'present',\n",
       "   'ruined': 'present',\n",
       "   'bee': 'present',\n",
       "   'sting': 'present',\n",
       "   'bill': 'present',\n",
       "   'murray': 'present',\n",
       "   'several': 'present',\n",
       "   'stings': 'present',\n",
       "   'happened': 'present',\n",
       "   'individual': 'present',\n",
       "   'screenplay': 'present',\n",
       "   'novel': 'present',\n",
       "   'contain': 'present',\n",
       "   'many': 'present',\n",
       "   'points': 'present',\n",
       "   'yet': 'present',\n",
       "   'probably': 'present',\n",
       "   'aware': 'present',\n",
       "   'two': 'present',\n",
       "   'different': 'present',\n",
       "   'studios': 'present',\n",
       "   'genre': 'present',\n",
       "   'geeks': 'present',\n",
       "   'revenge': 'present',\n",
       "   'fully': 'present',\n",
       "   'formed': 'present',\n",
       "   'strengths': 'present',\n",
       "   'rely': 'present',\n",
       "   'upon': 'present',\n",
       "   'fantastic': 'present',\n",
       "   'performances': 'present',\n",
       "   'newcomer': 'present',\n",
       "   'jessica': 'present',\n",
       "   'campbell': 'present',\n",
       "   'anti': 'present',\n",
       "   'social': 'present',\n",
       "   'tammy': 'present',\n",
       "   'playing': 'present',\n",
       "   'rooney': 'present',\n",
       "   'role': 'present',\n",
       "   '_ferris': 'present',\n",
       "   'bueller_': 'present',\n",
       "   'seems': 'present',\n",
       "   'fun': 'present',\n",
       "   'since': 'present',\n",
       "   'revelation': 'present',\n",
       "   'early': 'present',\n",
       "   'year': 'present',\n",
       "   'teenagers': 'present',\n",
       "   'little': 'present',\n",
       "   'clout': 'present',\n",
       "   'money': 'present',\n",
       "   'deserves': 'present',\n",
       "   'oscar': 'present',\n",
       "   'nomination': 'present',\n",
       "   'character': 'present',\n",
       "   'like': 'present',\n",
       "   'speech': 'present',\n",
       "   'gymnasium': 'present',\n",
       "   'one': 'present',\n",
       "   'thing': 'present',\n",
       "   'bothering': 'present',\n",
       "   'seen': 'present',\n",
       "   'amount': 'present',\n",
       "   'sexuality': 'present',\n",
       "   'suppose': 'present',\n",
       "   'coming': 'present',\n",
       "   'expect': 'present',\n",
       "   'less': 'present',\n",
       "   'starts': 'present',\n",
       "   'light': 'present',\n",
       "   'airy': 'present',\n",
       "   'sitcom': 'present',\n",
       "   'screws': 'present',\n",
       "   'tighten': 'present',\n",
       "   'tensions': 'present',\n",
       "   'mount': 'present',\n",
       "   'alexander': 'present',\n",
       "   'payne': 'present',\n",
       "   'decides': 'present',\n",
       "   'add': 'present',\n",
       "   'elements': 'present',\n",
       "   'frankly': 'present',\n",
       "   'distract': 'present',\n",
       "   'story': 'present',\n",
       "   'bad': 'present',\n",
       "   'enough': 'present',\n",
       "   'determination': 'present',\n",
       "   'win': 'present',\n",
       "   'costs': 'present',\n",
       "   'throw': 'present',\n",
       "   'logical': 'present',\n",
       "   'reason': 'present',\n",
       "   'lot': 'present',\n",
       "   'tonal': 'present',\n",
       "   'nosedive': 'present',\n",
       "   'takes': 'present',\n",
       "   'explicitly': 'present',\n",
       "   'sex': 'present',\n",
       "   'driven': 'present',\n",
       "   'mark': 'present',\n",
       "   'disappointment': 'present'},\n",
       "  'pos'),\n",
       " ({'got': 'present',\n",
       "   'mail': 'present',\n",
       "   'works': 'present',\n",
       "   'alot': 'present',\n",
       "   'better': 'present',\n",
       "   'deserves': 'present',\n",
       "   'order': 'present',\n",
       "   'make': 'present',\n",
       "   'film': 'present',\n",
       "   'success': 'present',\n",
       "   'cast': 'present',\n",
       "   'two': 'present',\n",
       "   'extremely': 'present',\n",
       "   'popular': 'present',\n",
       "   'attractive': 'present',\n",
       "   'stars': 'present',\n",
       "   'share': 'present',\n",
       "   'screen': 'present',\n",
       "   'hours': 'present',\n",
       "   'collect': 'present',\n",
       "   'profits': 'present',\n",
       "   'real': 'present',\n",
       "   'acting': 'present',\n",
       "   'involved': 'present',\n",
       "   'original': 'present',\n",
       "   'inventive': 'present',\n",
       "   'bone': 'present',\n",
       "   'body': 'present',\n",
       "   'basically': 'present',\n",
       "   'complete': 'present',\n",
       "   'shoot': 'present',\n",
       "   'shop': 'present',\n",
       "   'around': 'present',\n",
       "   'corner': 'present',\n",
       "   'adding': 'present',\n",
       "   'modern': 'present',\n",
       "   'twists': 'present',\n",
       "   'essentially': 'present',\n",
       "   'goes': 'present',\n",
       "   'defies': 'present',\n",
       "   'concepts': 'present',\n",
       "   'good': 'present',\n",
       "   'contemporary': 'present',\n",
       "   'filmmaking': 'present',\n",
       "   'overly': 'present',\n",
       "   'sentimental': 'present',\n",
       "   'times': 'present',\n",
       "   'terribly': 'present',\n",
       "   'mushy': 'present',\n",
       "   'mention': 'present',\n",
       "   'manipulative': 'present',\n",
       "   'oh': 'present',\n",
       "   'enjoyable': 'present',\n",
       "   'manipulation': 'present',\n",
       "   'must': 'present',\n",
       "   'something': 'present',\n",
       "   'casting': 'present',\n",
       "   'makes': 'present',\n",
       "   'movie': 'present',\n",
       "   'work': 'present',\n",
       "   'well': 'present',\n",
       "   'absolutely': 'present',\n",
       "   'hated': 'present',\n",
       "   'previous': 'present',\n",
       "   'ryan': 'present',\n",
       "   'hanks': 'present',\n",
       "   'teaming': 'present',\n",
       "   'sleepless': 'present',\n",
       "   'seattle': 'present',\n",
       "   'directing': 'present',\n",
       "   'films': 'present',\n",
       "   'helmed': 'present',\n",
       "   'woman': 'present',\n",
       "   'quite': 'present',\n",
       "   'yet': 'present',\n",
       "   'figured': 'present',\n",
       "   'liked': 'present',\n",
       "   'much': 'present',\n",
       "   'really': 'present',\n",
       "   'important': 'present',\n",
       "   'like': 'present',\n",
       "   'even': 'present',\n",
       "   'question': 'present',\n",
       "   'storyline': 'present',\n",
       "   'cliched': 'present',\n",
       "   'come': 'present',\n",
       "   'tom': 'present',\n",
       "   'plays': 'present',\n",
       "   'joe': 'present',\n",
       "   'fox': 'present',\n",
       "   'insanely': 'present',\n",
       "   'likeable': 'present',\n",
       "   'owner': 'present',\n",
       "   'discount': 'present',\n",
       "   'book': 'present',\n",
       "   'chain': 'present',\n",
       "   'meg': 'present',\n",
       "   'kathleen': 'present',\n",
       "   'kelley': 'present',\n",
       "   'proprietor': 'present',\n",
       "   'family': 'present',\n",
       "   'run': 'present',\n",
       "   'children': 'present',\n",
       "   'called': 'present',\n",
       "   'nice': 'present',\n",
       "   'homage': 'present',\n",
       "   'soon': 'present',\n",
       "   'become': 'present',\n",
       "   'bitter': 'present',\n",
       "   'rivals': 'present',\n",
       "   'new': 'present',\n",
       "   'books': 'present',\n",
       "   'store': 'present',\n",
       "   'opening': 'present',\n",
       "   'right': 'present',\n",
       "   'across': 'present',\n",
       "   'block': 'present',\n",
       "   'small': 'present',\n",
       "   'business': 'present',\n",
       "   'little': 'present',\n",
       "   'know': 'present',\n",
       "   'already': 'present',\n",
       "   'love': 'present',\n",
       "   'internet': 'present',\n",
       "   'neither': 'present',\n",
       "   'party': 'present',\n",
       "   'knows': 'present',\n",
       "   'person': 'present',\n",
       "   'true': 'present',\n",
       "   'identity': 'present',\n",
       "   'rest': 'present',\n",
       "   'story': 'present',\n",
       "   'serve': 'present',\n",
       "   'mere': 'present',\n",
       "   'backdrop': 'present',\n",
       "   'sure': 'present',\n",
       "   'mildly': 'present',\n",
       "   'interesting': 'present',\n",
       "   'subplots': 'present',\n",
       "   'fail': 'present',\n",
       "   'comparison': 'present',\n",
       "   'utter': 'present',\n",
       "   'cuteness': 'present',\n",
       "   'main': 'present',\n",
       "   'relationship': 'present',\n",
       "   'course': 'present',\n",
       "   'leads': 'present',\n",
       "   'predictable': 'present',\n",
       "   'climax': 'present',\n",
       "   'foreseeable': 'present',\n",
       "   'ending': 'present',\n",
       "   'damn': 'present',\n",
       "   'cute': 'present',\n",
       "   'done': 'present',\n",
       "   'doubt': 'present',\n",
       "   'entire': 'present',\n",
       "   'year': 'present',\n",
       "   'contains': 'present',\n",
       "   'scene': 'present',\n",
       "   'evokes': 'present',\n",
       "   'pure': 'present',\n",
       "   'joy': 'present',\n",
       "   'part': 'present',\n",
       "   'discovers': 'present',\n",
       "   'online': 'present',\n",
       "   'filled': 'present',\n",
       "   'lack': 'present',\n",
       "   'word': 'present',\n",
       "   'happiness': 'present',\n",
       "   'first': 'present',\n",
       "   'time': 'present',\n",
       "   'actually': 'present',\n",
       "   'left': 'present',\n",
       "   'theater': 'present',\n",
       "   'smiling': 'present'},\n",
       "  'pos'),\n",
       " ({'jaws': 'present',\n",
       "   'rare': 'present',\n",
       "   'film': 'present',\n",
       "   'grabs': 'present',\n",
       "   'attention': 'present',\n",
       "   'shows': 'present',\n",
       "   'single': 'present',\n",
       "   'image': 'present',\n",
       "   'screen': 'present',\n",
       "   'movie': 'present',\n",
       "   'opens': 'present',\n",
       "   'blackness': 'present',\n",
       "   'distant': 'present',\n",
       "   'alien': 'present',\n",
       "   'like': 'present',\n",
       "   'underwater': 'present',\n",
       "   'sounds': 'present',\n",
       "   'comes': 'present',\n",
       "   'first': 'present',\n",
       "   'ominous': 'present',\n",
       "   'bars': 'present',\n",
       "   'composer': 'present',\n",
       "   'john': 'present',\n",
       "   'williams': 'present',\n",
       "   'infamous': 'present',\n",
       "   'score': 'present',\n",
       "   'dah': 'present',\n",
       "   'dum': 'present',\n",
       "   'director': 'present',\n",
       "   'steven': 'present',\n",
       "   'spielberg': 'present',\n",
       "   'wastes': 'present',\n",
       "   'time': 'present',\n",
       "   'taking': 'present',\n",
       "   'us': 'present',\n",
       "   'water': 'present',\n",
       "   'midnight': 'present',\n",
       "   'swim': 'present',\n",
       "   'beautiful': 'present',\n",
       "   'girl': 'present',\n",
       "   'turns': 'present',\n",
       "   'deadly': 'present',\n",
       "   'right': 'present',\n",
       "   'away': 'present',\n",
       "   'lets': 'present',\n",
       "   'know': 'present',\n",
       "   'vulnerable': 'present',\n",
       "   'floating': 'present',\n",
       "   'ocean': 'present',\n",
       "   'attacked': 'present',\n",
       "   'never': 'present',\n",
       "   'relinquishes': 'present',\n",
       "   'grip': 'present',\n",
       "   'perhaps': 'present',\n",
       "   'outstanding': 'present',\n",
       "   'builds': 'present',\n",
       "   'works': 'present',\n",
       "   'theatrical': 'present',\n",
       "   'production': 'present',\n",
       "   'act': 'present',\n",
       "   'second': 'present',\n",
       "   'unlike': 'present',\n",
       "   'many': 'present',\n",
       "   'modern': 'present',\n",
       "   'filmmakers': 'present',\n",
       "   'great': 'present',\n",
       "   'deal': 'present',\n",
       "   'restraint': 'present',\n",
       "   'refuses': 'present',\n",
       "   'show': 'present',\n",
       "   'shark': 'present',\n",
       "   'middle': 'present',\n",
       "   'merely': 'present',\n",
       "   'suggests': 'present',\n",
       "   'presence': 'present',\n",
       "   'creepy': 'present',\n",
       "   'subjective': 'present',\n",
       "   'shots': 'present',\n",
       "   'music': 'present',\n",
       "   'building': 'present',\n",
       "   'tension': 'present',\n",
       "   'bit': 'present',\n",
       "   'climax': 'present',\n",
       "   'arrival': 'present',\n",
       "   'truly': 'present',\n",
       "   'terrifying': 'present',\n",
       "   'let': 'present',\n",
       "   'get': 'present',\n",
       "   'bored': 'present',\n",
       "   'imagery': 'present',\n",
       "   'police': 'present',\n",
       "   'chief': 'present',\n",
       "   'martin': 'present',\n",
       "   'brody': 'present',\n",
       "   'roy': 'present',\n",
       "   'scheider': 'present',\n",
       "   'new': 'present',\n",
       "   'york': 'present',\n",
       "   'cop': 'present',\n",
       "   'taken': 'present',\n",
       "   'easy': 'present',\n",
       "   'peaceful': 'present',\n",
       "   'job': 'present',\n",
       "   'running': 'present',\n",
       "   'station': 'present',\n",
       "   'amity': 'present',\n",
       "   'island': 'present',\n",
       "   'fictitious': 'present',\n",
       "   'england': 'present',\n",
       "   'resort': 'present',\n",
       "   'town': 'present',\n",
       "   'murder': 'present',\n",
       "   'gun': 'present',\n",
       "   'fired': 'present',\n",
       "   '25': 'present',\n",
       "   'years': 'present',\n",
       "   'shaken': 'present',\n",
       "   'several': 'present',\n",
       "   'vicious': 'present',\n",
       "   'white': 'present',\n",
       "   'attacks': 'present',\n",
       "   'fourth': 'present',\n",
       "   'july': 'present',\n",
       "   'mayor': 'present',\n",
       "   'larry': 'present',\n",
       "   'vaughn': 'present',\n",
       "   'murray': 'present',\n",
       "   'hamilton': 'present',\n",
       "   'want': 'present',\n",
       "   'shut': 'present',\n",
       "   'beaches': 'present',\n",
       "   'reliant': 'present',\n",
       "   'summer': 'present',\n",
       "   'tourist': 'present',\n",
       "   'money': 'present',\n",
       "   'joined': 'present',\n",
       "   'matt': 'present',\n",
       "   'hooper': 'present',\n",
       "   'richard': 'present',\n",
       "   'dreyfuss': 'present',\n",
       "   'young': 'present',\n",
       "   'ambitious': 'present',\n",
       "   'expert': 'present',\n",
       "   'marine': 'present',\n",
       "   'institute': 'present',\n",
       "   'fascinated': 'present',\n",
       "   'determined': 'present',\n",
       "   'help': 'present',\n",
       "   'stop': 'present',\n",
       "   '--': 'present',\n",
       "   'knowledge': 'present',\n",
       "   'exact': 'present',\n",
       "   'workings': 'present',\n",
       "   'perfect': 'present',\n",
       "   'engine': 'present',\n",
       "   'eating': 'present',\n",
       "   'machine': 'present',\n",
       "   'make': 'present',\n",
       "   'much': 'present',\n",
       "   'finally': 'present',\n",
       "   'relents': 'present',\n",
       "   'join': 'present',\n",
       "   'crusty': 'present',\n",
       "   'old': 'present',\n",
       "   'killer': 'present',\n",
       "   'named': 'present',\n",
       "   'quint': 'present',\n",
       "   'robert': 'present',\n",
       "   'shaw': 'present',\n",
       "   'decrepit': 'present',\n",
       "   'boat': 'present',\n",
       "   'orca': 'present',\n",
       "   'search': 'present',\n",
       "   'entire': 'present',\n",
       "   'takes': 'present',\n",
       "   'place': 'present',\n",
       "   'three': 'present',\n",
       "   'men': 'present',\n",
       "   'hunt': 'present',\n",
       "   'inevitably': 'present',\n",
       "   'hunted': 'present',\n",
       "   'thriller': 'present',\n",
       "   'keen': 'present',\n",
       "   'sense': 'present',\n",
       "   'humor': 'present',\n",
       "   'incredible': 'present',\n",
       "   'pacing': 'present',\n",
       "   'horror': 'present',\n",
       "   'ten': 'present',\n",
       "   'movies': 'present',\n",
       "   'rolled': 'present',\n",
       "   'one': 'present',\n",
       "   'wonder': 'present',\n",
       "   'took': 'present',\n",
       "   'america': 'present',\n",
       "   'storm': 'present',\n",
       "   '1975': 'present',\n",
       "   'enough': 'present',\n",
       "   'crown': 'present',\n",
       "   'box': 'present',\n",
       "   'office': 'present',\n",
       "   'champ': 'present',\n",
       "   'unceremoniously': 'present',\n",
       "   'dethroned': 'present',\n",
       "   '1977': 'present',\n",
       "   'star': 'present',\n",
       "   'wars': 'present',\n",
       "   'even': 'present',\n",
       "   'today': 'present',\n",
       "   'fascination': 'present',\n",
       "   'par': 'present',\n",
       "   'hitchcock': 'present',\n",
       "   'psycho': 'present',\n",
       "   'seems': 'present',\n",
       "   'age': 'present',\n",
       "   'although': 'present',\n",
       "   'grand': 'present',\n",
       "   'technology': 'present',\n",
       "   'exists': 'present',\n",
       "   'makes': 'present',\n",
       "   'technical': 'present',\n",
       "   'sequences': 'present',\n",
       "   'including': 'present',\n",
       "   'mechanical': 'present',\n",
       "   'sharks': 'present',\n",
       "   'obsolete': 'present',\n",
       "   'none': 'present',\n",
       "   'could': 'present',\n",
       "   'improve': 'present',\n",
       "   'would': 'present',\n",
       "   'lead': 'present',\n",
       "   'overkill': 'present',\n",
       "   'limitations': 'present',\n",
       "   'faced': 'present',\n",
       "   'may': 'present',\n",
       "   'actually': 'present',\n",
       "   'produced': 'present',\n",
       "   'better': 'present',\n",
       "   'forced': 'present',\n",
       "   'rely': 'present',\n",
       "   'traditional': 'present',\n",
       "   'cinematic': 'present',\n",
       "   'elements': 'present',\n",
       "   'characterization': 'present',\n",
       "   'sharp': 'present',\n",
       "   'editing': 'present',\n",
       "   'creative': 'present',\n",
       "   'photography': 'present',\n",
       "   'instead': 'present',\n",
       "   'simply': 'present',\n",
       "   'dousing': 'present',\n",
       "   'audience': 'present',\n",
       "   'digital': 'present',\n",
       "   'effects': 'present',\n",
       "   'known': 'present',\n",
       "   'actors': 'present',\n",
       "   'made': 'present',\n",
       "   'draw': 'present',\n",
       "   'redford': 'present',\n",
       "   'paul': 'present',\n",
       "   'newman': 'present',\n",
       "   'nevertheless': 'present',\n",
       "   'guaranteed': 'present',\n",
       "   'successful': 'present',\n",
       "   'careers': 'present',\n",
       "   'gave': 'present',\n",
       "   'performance': 'present',\n",
       "   'refused': 'present',\n",
       "   'overshadowed': 'present',\n",
       "   'hits': 'present',\n",
       "   'notes': 'present',\n",
       "   'sympathetic': 'present',\n",
       "   'husband': 'present',\n",
       "   'father': 'present',\n",
       "   'caught': 'present',\n",
       "   'political': 'present',\n",
       "   'quagmire': 'present',\n",
       "   'going': 'present',\n",
       "   'warns': 'present',\n",
       "   'previously': 'present',\n",
       "   'seen': 'present',\n",
       "   'american': 'present',\n",
       "   'graffiti': 'present',\n",
       "   '1973': 'present',\n",
       "   'apprenticeship': 'present',\n",
       "   'duddy': 'present',\n",
       "   'kravitz': 'present',\n",
       "   '1974': 'present',\n",
       "   'gives': 'present',\n",
       "   'surprisingly': 'present',\n",
       "   'mature': 'present',\n",
       "   'complex': 'present',\n",
       "   'someone': 'present',\n",
       "   'literally': 'present',\n",
       "   'played': 'present',\n",
       "   'kids': 'present',\n",
       "   'teenagers': 'present',\n",
       "   'however': 'present',\n",
       "   'gnarled': 'present',\n",
       "   'captain': 'present',\n",
       "   'ahab': 'present',\n",
       "   'sorely': 'present',\n",
       "   'overlooked': 'present',\n",
       "   'academy': 'present',\n",
       "   'awards': 'present',\n",
       "   'bordering': 'present',\n",
       "   'parody': 'present',\n",
       "   'plays': 'present',\n",
       "   'grizzled': 'present',\n",
       "   'loner': 'present',\n",
       "   'whose': 'present',\n",
       "   'machismo': 'present',\n",
       "   'borders': 'present',\n",
       "   'masochism': 'present',\n",
       "   'slightly': 'present',\n",
       "   'deranged': 'present',\n",
       "   'almost': 'present',\n",
       "   'caricature': 'present',\n",
       "   'scene': 'present',\n",
       "   'late': 'present',\n",
       "   'deck': 'present',\n",
       "   'comparing': 'present',\n",
       "   'scars': 'present',\n",
       "   'drawn': 'present',\n",
       "   'telling': 'present',\n",
       "   'story': 'present',\n",
       "   'experiences': 'present',\n",
       "   'aboard': 'present',\n",
       "   'u': 'present',\n",
       "   'indianapolis': 'present',\n",
       "   'navy': 'present',\n",
       "   'ship': 'present',\n",
       "   'world': 'present',\n",
       "   'war': 'present',\n",
       "   'ii': 'present',\n",
       "   'sunk': 'present',\n",
       "   'japanese': 'present',\n",
       "   'tale': 'present',\n",
       "   'week': 'present',\n",
       "   '1': 'present',\n",
       "   '000': 'present',\n",
       "   'swarms': 'present',\n",
       "   'slowly': 'present',\n",
       "   'devoured': 'present',\n",
       "   'hair': 'present',\n",
       "   'raising': 'present',\n",
       "   'anything': 'present',\n",
       "   'put': 'present',\n",
       "   'delivers': 'present',\n",
       "   'long': 'present',\n",
       "   'take': 'present',\n",
       "   'best': 'present',\n",
       "   'acting': 'present',\n",
       "   'course': 'present',\n",
       "   'leave': 'present',\n",
       "   'black': 'present',\n",
       "   'eyes': 'present',\n",
       "   'endless': 'present',\n",
       "   'rows': 'present',\n",
       "   'teeth': 'present',\n",
       "   'insatiable': 'present',\n",
       "   'urge': 'present',\n",
       "   'eat': 'present',\n",
       "   'basically': 'present',\n",
       "   'epitome': 'present',\n",
       "   'mankind': 'present',\n",
       "   'fears': 'present',\n",
       "   'unknown': 'present',\n",
       "   'threatening': 'present',\n",
       "   'nature': 'present',\n",
       "   'nemesis': 'present',\n",
       "   'real': 'present',\n",
       "   'survived': 'present',\n",
       "   'sinch': 'present',\n",
       "   'dinosaurs': 'present',\n",
       "   'whites': 'present',\n",
       "   'exist': 'present',\n",
       "   'large': 'present',\n",
       "   'threat': 'present',\n",
       "   'every': 'present',\n",
       "   'feel': 'present',\n",
       "   'queasy': 'present',\n",
       "   'see': 'present',\n",
       "   'look': 'present',\n",
       "   'bunch': 'present',\n",
       "   'writihing': 'present',\n",
       "   'dangling': 'present',\n",
       "   'completely': 'present',\n",
       "   'unprotected': 'present',\n",
       "   'legs': 'present',\n",
       "   'ready': 'present',\n",
       "   'chomped': 'present',\n",
       "   'combination': 'present',\n",
       "   'actual': 'present',\n",
       "   'footage': 'present',\n",
       "   'five': 'present',\n",
       "   'different': 'present',\n",
       "   'nicknamed': 'present',\n",
       "   'bruce': 'present',\n",
       "   'crew': 'present',\n",
       "   'built': 'present',\n",
       "   'shot': 'present',\n",
       "   'angles': 'present',\n",
       "   'forgotten': 'present',\n",
       "   'sort': 'present',\n",
       "   'precursor': 'present',\n",
       "   'waterworld': 'present',\n",
       "   '1995': 'present',\n",
       "   'soggy': 'present',\n",
       "   'cost': 'present',\n",
       "   'overruns': 'present',\n",
       "   'universal': 'present',\n",
       "   'studios': 'present',\n",
       "   'worried': 'present',\n",
       "   'bomb': 'present',\n",
       "   'overcame': 'present',\n",
       "   'obstacles': 'present',\n",
       "   'delivered': 'present',\n",
       "   'finest': 'present',\n",
       "   'primal': 'present',\n",
       "   'scare': 'present',\n",
       "   'thrillers': 'present',\n",
       "   'ever': 'present',\n",
       "   'come': 'present',\n",
       "   'hollywood': 'present'},\n",
       "  'pos'),\n",
       " ({'moviemaking': 'present',\n",
       "   'lot': 'present',\n",
       "   'like': 'present',\n",
       "   'general': 'present',\n",
       "   'manager': 'present',\n",
       "   'nfl': 'present',\n",
       "   'team': 'present',\n",
       "   'post': 'present',\n",
       "   'salary': 'present',\n",
       "   'cap': 'present',\n",
       "   'era': 'present',\n",
       "   '--': 'present',\n",
       "   'got': 'present',\n",
       "   'know': 'present',\n",
       "   'allocate': 'present',\n",
       "   'resources': 'present',\n",
       "   'every': 'present',\n",
       "   'dollar': 'present',\n",
       "   'spent': 'present',\n",
       "   'free': 'present',\n",
       "   'agent': 'present',\n",
       "   'defensive': 'present',\n",
       "   'tackle': 'present',\n",
       "   'one': 'present',\n",
       "   'less': 'present',\n",
       "   'spend': 'present',\n",
       "   'linebackers': 'present',\n",
       "   'safeties': 'present',\n",
       "   'centers': 'present',\n",
       "   'leads': 'present',\n",
       "   'teams': 'present',\n",
       "   'detroit': 'present',\n",
       "   'lions': 'present',\n",
       "   'boast': 'present',\n",
       "   'superstar': 'present',\n",
       "   'running': 'present',\n",
       "   'back': 'present',\n",
       "   'huge': 'present',\n",
       "   'contract': 'present',\n",
       "   'field': 'present',\n",
       "   'five': 'present',\n",
       "   'guys': 'present',\n",
       "   'named': 'present',\n",
       "   'herb': 'present',\n",
       "   'block': 'present',\n",
       "   'movies': 'present',\n",
       "   'end': 'present',\n",
       "   'films': 'present',\n",
       "   'spawn': 'present',\n",
       "   'special': 'present',\n",
       "   'effects': 'present',\n",
       "   'budget': 'present',\n",
       "   'enough': 'present',\n",
       "   'money': 'present',\n",
       "   'hire': 'present',\n",
       "   'recognizable': 'present',\n",
       "   'actors': 'present',\n",
       "   'jackie': 'present',\n",
       "   'chan': 'present',\n",
       "   'barry': 'present',\n",
       "   'sanders': 'present',\n",
       "   'spins': 'present',\n",
       "   'darts': 'present',\n",
       "   'across': 'present',\n",
       "   'screen': 'present',\n",
       "   'cutting': 'present',\n",
       "   'line': 'present',\n",
       "   'watching': 'present',\n",
       "   'operation': 'present',\n",
       "   'condor': 'present',\n",
       "   'drives': 'present',\n",
       "   'motorcycle': 'present',\n",
       "   'crowded': 'present',\n",
       "   'streets': 'present',\n",
       "   'madrid': 'present',\n",
       "   'fleeing': 'present',\n",
       "   'armada': 'present',\n",
       "   'pursuers': 'present',\n",
       "   'identical': 'present',\n",
       "   'black': 'present',\n",
       "   'compact': 'present',\n",
       "   'cars': 'present',\n",
       "   'reminiscent': 'present',\n",
       "   'daylight': 'present',\n",
       "   'chicago': 'present',\n",
       "   'bears': 'present',\n",
       "   'hot': 'present',\n",
       "   'pursuit': 'present',\n",
       "   'except': 'present',\n",
       "   'worry': 'present',\n",
       "   'rescuing': 'present',\n",
       "   'runaway': 'present',\n",
       "   'baby': 'present',\n",
       "   'carriages': 'present',\n",
       "   'star': 'present',\n",
       "   'anybody': 'present',\n",
       "   'almost': 'present',\n",
       "   'cent': 'present',\n",
       "   'invested': 'present',\n",
       "   'movie': 'present',\n",
       "   'goes': 'present',\n",
       "   'stunts': 'present',\n",
       "   'rest': 'present',\n",
       "   'pay': 'present',\n",
       "   'hospital': 'present',\n",
       "   'bills': 'present',\n",
       "   'leaves': 'present',\n",
       "   '75': 'present',\n",
       "   'cents': 'present',\n",
       "   'things': 'present',\n",
       "   'directors': 'present',\n",
       "   'directs': 'present',\n",
       "   'scripts': 'present',\n",
       "   'dubbing': 'present',\n",
       "   'supporting': 'present',\n",
       "   'characters': 'present',\n",
       "   'mention': 'present',\n",
       "   'hideous': 'present',\n",
       "   'title': 'present',\n",
       "   'sequence': 'present',\n",
       "   'also': 'present',\n",
       "   'explains': 'present',\n",
       "   'shot': 'present',\n",
       "   'odd': 'present',\n",
       "   'places': 'present',\n",
       "   'morocco': 'present',\n",
       "   'spain': 'present',\n",
       "   'first': 'present',\n",
       "   'release': 'present',\n",
       "   'country': 'present',\n",
       "   'rumble': 'present',\n",
       "   'bronx': 'present',\n",
       "   'supposedly': 'present',\n",
       "   'set': 'present',\n",
       "   'new': 'present',\n",
       "   'york': 'present',\n",
       "   'filmed': 'present',\n",
       "   'vancouver': 'present',\n",
       "   'chase': 'present',\n",
       "   'scenes': 'present',\n",
       "   'canadian': 'present',\n",
       "   'rockies': 'present',\n",
       "   'clearly': 'present',\n",
       "   'visible': 'present',\n",
       "   'heck': 'present',\n",
       "   'even': 'present',\n",
       "   'haircut': 'present',\n",
       "   'looks': 'present',\n",
       "   'much': 'present',\n",
       "   'personal': 'present',\n",
       "   'hairstylist': 'present',\n",
       "   'plays': 'present',\n",
       "   'character': 'present',\n",
       "   'always': 'present',\n",
       "   'played': 'present',\n",
       "   'mixture': 'present',\n",
       "   'bruce': 'present',\n",
       "   'lee': 'present',\n",
       "   'tim': 'present',\n",
       "   'allen': 'present',\n",
       "   'master': 'present',\n",
       "   'kung': 'present',\n",
       "   'fu': 'present',\n",
       "   'slapstick': 'present',\n",
       "   'sent': 'present',\n",
       "   'un': 'present',\n",
       "   'retrieve': 'present',\n",
       "   'cache': 'present',\n",
       "   'lost': 'present',\n",
       "   'nazi': 'present',\n",
       "   'gold': 'present',\n",
       "   'north': 'present',\n",
       "   'african': 'present',\n",
       "   'desert': 'present',\n",
       "   'chased': 'present',\n",
       "   'horde': 'present',\n",
       "   'neo': 'present',\n",
       "   'sympathizers': 'present',\n",
       "   'two': 'present',\n",
       "   'stereotypical': 'present',\n",
       "   'arabs': 'present',\n",
       "   'political': 'present',\n",
       "   'correctness': 'present',\n",
       "   'joined': 'present',\n",
       "   'three': 'present',\n",
       "   'women': 'present',\n",
       "   'little': 'present',\n",
       "   'scream': 'present',\n",
       "   'save': 'present',\n",
       "   'us': 'present',\n",
       "   'misuse': 'present',\n",
       "   'firearms': 'present',\n",
       "   'villain': 'present',\n",
       "   'old': 'present',\n",
       "   'whose': 'present',\n",
       "   'legs': 'present',\n",
       "   'broken': 'present',\n",
       "   'secret': 'present',\n",
       "   'base': 'present',\n",
       "   'carried': 'present',\n",
       "   'everywhere': 'present',\n",
       "   'pathetic': 'present',\n",
       "   'evil': 'present',\n",
       "   'en': 'present',\n",
       "   'route': 'present',\n",
       "   'extended': 'present',\n",
       "   'scene': 'present',\n",
       "   'hilarious': 'present',\n",
       "   'fight': 'present',\n",
       "   'moroccan': 'present',\n",
       "   'version': 'present',\n",
       "   'motel': 'present',\n",
       "   '6': 'present',\n",
       "   'nazis': 'present',\n",
       "   'confrontations': 'present',\n",
       "   'savage': 'present',\n",
       "   'natives': 'present',\n",
       "   'long': 'present',\n",
       "   'chop': 'present',\n",
       "   'socky': 'present',\n",
       "   'followed': 'present',\n",
       "   'film': 'present',\n",
       "   'centerpiece': 'present',\n",
       "   'wind': 'present',\n",
       "   'tunnel': 'present',\n",
       "   'better': 'present',\n",
       "   'face': 'present',\n",
       "   'well': 'present',\n",
       "   'choreographed': 'present',\n",
       "   'sequences': 'present',\n",
       "   'giant': 'present',\n",
       "   'kevlar': 'present',\n",
       "   'hamster': 'present',\n",
       "   'balls': 'present',\n",
       "   'smashed': 'present',\n",
       "   'crates': 'present',\n",
       "   'bananas': 'present',\n",
       "   'scorpions': 'present',\n",
       "   'ignore': 'present',\n",
       "   'gaping': 'present',\n",
       "   'holes': 'present',\n",
       "   'plot': 'present',\n",
       "   'exactly': 'present',\n",
       "   'escape': 'present',\n",
       "   'take': 'present',\n",
       "   'key': 'present',\n",
       "   'production': 'present',\n",
       "   'values': 'present',\n",
       "   'japanese': 'present',\n",
       "   'girl': 'present',\n",
       "   'hitchhiking': 'present',\n",
       "   'sahara': 'present',\n",
       "   'go': 'present',\n",
       "   'see': 'present',\n",
       "   'pretentions': 'present',\n",
       "   'raiders': 'present',\n",
       "   'ark': 'present',\n",
       "   'knockoff': 'present',\n",
       "   'wonders': 'present',\n",
       "   'could': 'present',\n",
       "   'franchise': 'present',\n",
       "   'blocking': 'present',\n",
       "   'lawrence': 'present',\n",
       "   'kazdan': 'present',\n",
       "   'screenplay': 'present',\n",
       "   'john': 'present',\n",
       "   'williams': 'present',\n",
       "   'score': 'present',\n",
       "   'spielberg': 'present',\n",
       "   'directing': 'present',\n",
       "   'george': 'present',\n",
       "   'lucas': 'present',\n",
       "   'producing': 'present',\n",
       "   'might': 'present',\n",
       "   'however': 'present',\n",
       "   'something': 'present',\n",
       "   'talent': 'present',\n",
       "   'mainstream': 'present',\n",
       "   'hollywood': 'present',\n",
       "   'ought': 'present',\n",
       "   'utilize': 'present'},\n",
       "  'pos'),\n",
       " ({'june': 'present',\n",
       "   '30': 'present',\n",
       "   '1960': 'present',\n",
       "   'self': 'present',\n",
       "   'taught': 'present',\n",
       "   'idealistic': 'present',\n",
       "   'yet': 'present',\n",
       "   'pragmatic': 'present',\n",
       "   'young': 'present',\n",
       "   'man': 'present',\n",
       "   'became': 'present',\n",
       "   'age': 'present',\n",
       "   '36': 'present',\n",
       "   'first': 'present',\n",
       "   'head': 'present',\n",
       "   'government': 'present',\n",
       "   'newly': 'present',\n",
       "   'independent': 'present',\n",
       "   'african': 'present',\n",
       "   'state': 'present',\n",
       "   'formerly': 'present',\n",
       "   'belgian': 'present',\n",
       "   'congo': 'present',\n",
       "   'two': 'present',\n",
       "   'months': 'present',\n",
       "   'later': 'present',\n",
       "   'ousted': 'present',\n",
       "   'powerful': 'present',\n",
       "   'position': 'present',\n",
       "   'hunted': 'present',\n",
       "   'troops': 'present',\n",
       "   'captured': 'present',\n",
       "   'brutally': 'present',\n",
       "   'murdered': 'present',\n",
       "   'along': 'present',\n",
       "   'aides': 'present',\n",
       "   'little': 'present',\n",
       "   'known': 'present',\n",
       "   'story': 'present',\n",
       "   'meteoric': 'present',\n",
       "   'rise': 'present',\n",
       "   'fall': 'present',\n",
       "   'told': 'present',\n",
       "   'international': 'present',\n",
       "   'filmmaker': 'present',\n",
       "   'raoul': 'present',\n",
       "   'peck': 'present',\n",
       "   'lumumba': 'present',\n",
       "   'patrice': 'present',\n",
       "   'eriq': 'present',\n",
       "   'ebouaney': 'present',\n",
       "   'previously': 'present',\n",
       "   'helmer': 'present',\n",
       "   '1991': 'present',\n",
       "   'award': 'present',\n",
       "   'winning': 'present',\n",
       "   'documentary': 'present',\n",
       "   'death': 'present',\n",
       "   'prophet': 'present',\n",
       "   'virtually': 'present',\n",
       "   'guaranteeing': 'present',\n",
       "   'new': 'present',\n",
       "   'fictional': 'present',\n",
       "   'account': 'present',\n",
       "   'patriot': 'present',\n",
       "   'remains': 'present',\n",
       "   'true': 'present',\n",
       "   'subject': 'present',\n",
       "   'co': 'present',\n",
       "   'writer': 'present',\n",
       "   'pascal': 'present',\n",
       "   'bonitzer': 'present',\n",
       "   'begins': 'present',\n",
       "   'end': 'present',\n",
       "   'political': 'present',\n",
       "   'leader': 'present',\n",
       "   'watch': 'present',\n",
       "   'white': 'present',\n",
       "   'men': 'present',\n",
       "   'perform': 'present',\n",
       "   'gruesome': 'present',\n",
       "   'task': 'present',\n",
       "   'dismembering': 'present',\n",
       "   'bodies': 'present',\n",
       "   'three': 'present',\n",
       "   'black': 'present',\n",
       "   'images': 'present',\n",
       "   'hatchets': 'present',\n",
       "   'saws': 'present',\n",
       "   'fast': 'present',\n",
       "   'emptying': 'present',\n",
       "   'whiskey': 'present',\n",
       "   'bottles': 'present',\n",
       "   'accompany': 'present',\n",
       "   'grisly': 'present',\n",
       "   'image': 'present',\n",
       "   'jump': 'present',\n",
       "   'back': 'present',\n",
       "   'years': 'present',\n",
       "   'meeting': 'present',\n",
       "   'among': 'present',\n",
       "   'leaders': 'present',\n",
       "   'owned': 'present',\n",
       "   'third': 'present',\n",
       "   'class': 'present',\n",
       "   'postal': 'present',\n",
       "   'worker': 'present',\n",
       "   'speaks': 'present',\n",
       "   'mind': 'present',\n",
       "   'heads': 'present',\n",
       "   'tribes': 'present',\n",
       "   'proclaiming': 'present',\n",
       "   'tribal': 'present',\n",
       "   'regional': 'present',\n",
       "   'national': 'present',\n",
       "   'small': 'present',\n",
       "   'mobile': 'present',\n",
       "   'party': 'present',\n",
       "   'congolese': 'present',\n",
       "   'movement': 'present',\n",
       "   'mnc': 'present',\n",
       "   'gaining': 'present',\n",
       "   'prominence': 'present',\n",
       "   'leaves': 'present',\n",
       "   'clerical': 'present',\n",
       "   'job': 'present',\n",
       "   'sell': 'present',\n",
       "   'beer': 'present',\n",
       "   'get': 'present',\n",
       "   'face': 'present',\n",
       "   'bustling': 'present',\n",
       "   'capital': 'present',\n",
       "   'stanleyville': 'present',\n",
       "   'time': 'present',\n",
       "   'colonial': 'present',\n",
       "   'empires': 'present',\n",
       "   'falling': 'present',\n",
       "   'around': 'present',\n",
       "   'world': 'present',\n",
       "   'right': 'present',\n",
       "   'place': 'present',\n",
       "   'savvy': 'present',\n",
       "   'chess': 'present',\n",
       "   'like': 'present',\n",
       "   'manipulation': 'present',\n",
       "   'achieves': 'present',\n",
       "   'leadership': 'present',\n",
       "   'date': 'present',\n",
       "   'independence': 'present',\n",
       "   'approaches': 'present',\n",
       "   'tactically': 'present',\n",
       "   'positions': 'present',\n",
       "   'nation': 'present',\n",
       "   'prime': 'present',\n",
       "   'minister': 'present',\n",
       "   'defense': 'present',\n",
       "   'supporting': 'present',\n",
       "   'presidency': 'present',\n",
       "   'joseph': 'present',\n",
       "   'kasa': 'present',\n",
       "   'vubu': 'present',\n",
       "   'maka': 'present',\n",
       "   'kotto': 'present',\n",
       "   'coalition': 'present',\n",
       "   'created': 'present',\n",
       "   'soon': 'present',\n",
       "   'starts': 'present',\n",
       "   'apart': 'present',\n",
       "   'former': 'present',\n",
       "   'masters': 'present',\n",
       "   'continue': 'present',\n",
       "   'exert': 'present',\n",
       "   'influence': 'present',\n",
       "   'struggling': 'present',\n",
       "   'strive': 'present',\n",
       "   'maintain': 'present',\n",
       "   'economic': 'present',\n",
       "   'hold': 'present',\n",
       "   'country': 'present',\n",
       "   'vast': 'present',\n",
       "   'natural': 'present',\n",
       "   'resources': 'present',\n",
       "   'copper': 'present',\n",
       "   'diamonds': 'present',\n",
       "   'gold': 'present',\n",
       "   'seek': 'present',\n",
       "   'help': 'present',\n",
       "   'us': 'present',\n",
       "   'knowing': 'present',\n",
       "   'would': 'present',\n",
       "   'try': 'present',\n",
       "   'create': 'present',\n",
       "   'de': 'present',\n",
       "   'facto': 'present',\n",
       "   'american': 'present',\n",
       "   'control': 'present',\n",
       "   'fledgling': 'present',\n",
       "   'initial': 'present',\n",
       "   'investigation': 'present',\n",
       "   'soviet': 'present',\n",
       "   'assistance': 'present',\n",
       "   'immediately': 'present',\n",
       "   'tags': 'present',\n",
       "   'communist': 'present',\n",
       "   'integrity': 'present',\n",
       "   'overshadowed': 'present',\n",
       "   'cold': 'present',\n",
       "   'war': 'present',\n",
       "   'threat': 'present',\n",
       "   'russian': 'present',\n",
       "   'domination': 'present',\n",
       "   'situation': 'present',\n",
       "   'goes': 'present',\n",
       "   'bad': 'present',\n",
       "   'worse': 'present',\n",
       "   'army': 'present',\n",
       "   'mutinies': 'present',\n",
       "   'remaining': 'present',\n",
       "   'whites': 'present',\n",
       "   'begin': 'present',\n",
       "   'evacuate': 'present',\n",
       "   'arm': 'present',\n",
       "   'violently': 'present',\n",
       "   'intervene': 'present',\n",
       "   'lucrative': 'present',\n",
       "   'katanga': 'present',\n",
       "   'province': 'present',\n",
       "   'succeeds': 'present',\n",
       "   'rival': 'present',\n",
       "   'moise': 'present',\n",
       "   'tschombe': 'present',\n",
       "   'nzonzi': 'present',\n",
       "   'refused': 'present',\n",
       "   'access': 'present',\n",
       "   'returns': 'present',\n",
       "   'conference': 'present',\n",
       "   'abroad': 'present',\n",
       "   'tumultuous': 'present',\n",
       "   'period': 'present',\n",
       "   'modern': 'present',\n",
       "   'history': 'present',\n",
       "   'saw': 'present',\n",
       "   'score': 'present',\n",
       "   'nations': 'present',\n",
       "   'sometimes': 'present',\n",
       "   'odious': 'present',\n",
       "   'colonialists': 'present',\n",
       "   'ruled': 'present',\n",
       "   'much': 'present',\n",
       "   'european': 'present',\n",
       "   'seats': 'present',\n",
       "   'power': 'present',\n",
       "   'centuries': 'present',\n",
       "   'focuses': 'present',\n",
       "   'familiar': 'present',\n",
       "   'material': 'present',\n",
       "   'strives': 'present',\n",
       "   'give': 'present',\n",
       "   'honest': 'present',\n",
       "   'portrayal': 'present',\n",
       "   'friend': 'present',\n",
       "   'foes': 'present',\n",
       "   'movements': 'present',\n",
       "   'gripped': 'present',\n",
       "   'africa': 'present',\n",
       "   '50': 'present',\n",
       "   '60': 'present',\n",
       "   'takes': 'present',\n",
       "   'many': 'present',\n",
       "   'including': 'present',\n",
       "   'nigeria': 'present',\n",
       "   'somalia': 'present',\n",
       "   'born': 'present',\n",
       "   'varying': 'present',\n",
       "   'degrees': 'present',\n",
       "   'success': 'present',\n",
       "   'failure': 'present',\n",
       "   'usually': 'present',\n",
       "   'dependent': 'present',\n",
       "   'upon': 'present',\n",
       "   'colonized': 'present',\n",
       "   'better': 'present',\n",
       "   'others': 'present',\n",
       "   'effort': 'present',\n",
       "   'involved': 'present',\n",
       "   'quite': 'present',\n",
       "   'ambitious': 'present',\n",
       "   'crew': 'present',\n",
       "   'behind': 'present',\n",
       "   'camera': 'present',\n",
       "   'bring': 'present',\n",
       "   'life': 'present',\n",
       "   'slice': 'present',\n",
       "   'might': 'present',\n",
       "   'gone': 'present',\n",
       "   'unexplored': 'present',\n",
       "   'decades': 'present',\n",
       "   'production': 'present',\n",
       "   'values': 'present',\n",
       "   'rate': 'present',\n",
       "   'must': 'present',\n",
       "   'standards': 'present',\n",
       "   'budget': 'present',\n",
       "   'feel': 'present',\n",
       "   'realistic': 'present',\n",
       "   'settings': 'present',\n",
       "   'nicely': 'present',\n",
       "   'maintained': 'present',\n",
       "   'traveled': 'present',\n",
       "   'zimbabwe': 'present',\n",
       "   'mozambique': 'present',\n",
       "   'screenplay': 'present',\n",
       "   'covers': 'present',\n",
       "   'lot': 'present',\n",
       "   'ground': 'present',\n",
       "   'yeoman': 'present',\n",
       "   'work': 'present',\n",
       "   'providing': 'present',\n",
       "   'great': 'present',\n",
       "   'deal': 'present',\n",
       "   'detailed': 'present',\n",
       "   'trying': 'present',\n",
       "   'justice': 'present',\n",
       "   'side': 'present',\n",
       "   'things': 'present',\n",
       "   'evenly': 'present',\n",
       "   'linear': 'present',\n",
       "   'straightforward': 'present',\n",
       "   'manner': 'present',\n",
       "   'teaches': 'present',\n",
       "   'preaches': 'present',\n",
       "   'concentrates': 'present',\n",
       "   'good': 'present',\n",
       "   'deeds': 'present',\n",
       "   'bit': 'present',\n",
       "   'stalwart': 'present',\n",
       "   'saint': 'present',\n",
       "   'embellish': 'present',\n",
       "   'larger': 'present',\n",
       "   'persona': 'present',\n",
       "   'family': 'present',\n",
       "   'handled': 'present',\n",
       "   'several': 'present',\n",
       "   'perfunctory': 'present',\n",
       "   'brief': 'present',\n",
       "   'interludes': 'present',\n",
       "   'show': 'present',\n",
       "   'talking': 'present',\n",
       "   'one': 'present',\n",
       "   'children': 'present',\n",
       "   'embracing': 'present',\n",
       "   'wife': 'present',\n",
       "   'lamenting': 'present',\n",
       "   'child': 'present',\n",
       "   'know': 'present',\n",
       "   'intent': 'present',\n",
       "   'flesh': 'present',\n",
       "   'short': 'present',\n",
       "   'shrift': 'present',\n",
       "   'given': 'present',\n",
       "   'lopsided': 'present',\n",
       "   'high': 'present',\n",
       "   'marks': 'present',\n",
       "   'go': 'present',\n",
       "   'title': 'present',\n",
       "   'character': 'present',\n",
       "   'actor': 'present',\n",
       "   'gives': 'present',\n",
       "   'convincing': 'present',\n",
       "   'charismatic': 'present',\n",
       "   'performance': 'present',\n",
       "   'multifaceted': 'present',\n",
       "   'politically': 'present',\n",
       "   'deft': 'present',\n",
       "   'people': 'present',\n",
       "   'force': 'present',\n",
       "   'driving': 'present',\n",
       "   'ambitions': 'present',\n",
       "   'docudrama': 'present',\n",
       "   'tradition': 'present',\n",
       "   'cast': 'present',\n",
       "   'outshine': 'present',\n",
       "   'star': 'present',\n",
       "   'complementing': 'present',\n",
       "   'efforts': 'present',\n",
       "   'instead': 'present',\n",
       "   'solid': 'present',\n",
       "   'interesting': 'present',\n",
       "   'educational': 'present',\n",
       "   'appeal': 'present',\n",
       "   'film': 'present',\n",
       "   'buffs': 'present',\n",
       "   'politicos': 'present',\n",
       "   'intelligence': 'present',\n",
       "   'telling': 'present',\n",
       "   'anything': 'present',\n",
       "   'seen': 'present',\n",
       "   'hollywood': 'present',\n",
       "   'b': 'present'},\n",
       "  'pos'),\n",
       " ({'apparently': 'present',\n",
       "   'director': 'present',\n",
       "   'tony': 'present',\n",
       "   'kaye': 'present',\n",
       "   'major': 'present',\n",
       "   'battle': 'present',\n",
       "   'new': 'present',\n",
       "   'line': 'present',\n",
       "   'regarding': 'present',\n",
       "   'film': 'present',\n",
       "   'american': 'present',\n",
       "   'history': 'present',\n",
       "   'x': 'present',\n",
       "   'know': 'present',\n",
       "   'details': 'present',\n",
       "   'fight': 'present',\n",
       "   'seems': 'present',\n",
       "   'happy': 'present',\n",
       "   'final': 'present',\n",
       "   'product': 'present',\n",
       "   'nearly': 'present',\n",
       "   'removed': 'present',\n",
       "   'name': 'present',\n",
       "   'credits': 'present',\n",
       "   'altogether': 'present',\n",
       "   'heard': 'present',\n",
       "   'kind': 'present',\n",
       "   'thing': 'present',\n",
       "   'happening': 'present',\n",
       "   'makes': 'present',\n",
       "   'wonder': 'present',\n",
       "   'much': 'present',\n",
       "   'input': 'present',\n",
       "   'studio': 'present',\n",
       "   'films': 'present',\n",
       "   'produce': 'present',\n",
       "   'found': 'present',\n",
       "   'extremely': 'present',\n",
       "   'good': 'present',\n",
       "   'focused': 'present',\n",
       "   'look': 'present',\n",
       "   'touchy': 'present',\n",
       "   'subject': 'present',\n",
       "   'racism': 'present',\n",
       "   'powerful': 'present',\n",
       "   'charismatic': 'present',\n",
       "   'performance': 'present',\n",
       "   'edward': 'present',\n",
       "   'norton': 'present',\n",
       "   'hard': 'present',\n",
       "   'believe': 'present',\n",
       "   'two': 'present',\n",
       "   'years': 'present',\n",
       "   'since': 'present',\n",
       "   'fantastic': 'present',\n",
       "   'role': 'present',\n",
       "   'primal': 'present',\n",
       "   'fear': 'present',\n",
       "   'starring': 'present',\n",
       "   'making': 'present',\n",
       "   'star': 'present',\n",
       "   'one': 'present',\n",
       "   'performers': 'present',\n",
       "   'becomes': 'present',\n",
       "   'character': 'present',\n",
       "   'work': 'present',\n",
       "   'best': 'present',\n",
       "   'performances': 'present',\n",
       "   'year': 'present',\n",
       "   'plays': 'present',\n",
       "   'young': 'present',\n",
       "   'man': 'present',\n",
       "   'named': 'present',\n",
       "   'derek': 'present',\n",
       "   'vinyard': 'present',\n",
       "   'skinhead': 'present',\n",
       "   'living': 'present',\n",
       "   'venice': 'present',\n",
       "   'beach': 'present',\n",
       "   'brother': 'present',\n",
       "   'danny': 'present',\n",
       "   'furlong': 'present',\n",
       "   'mother': 'present',\n",
       "   'beverly': 'present',\n",
       "   'angelo': 'present',\n",
       "   'sister': 'present',\n",
       "   'davin': 'present',\n",
       "   'jennifer': 'present',\n",
       "   'lien': 'present',\n",
       "   'opens': 'present',\n",
       "   'flashback': 'present',\n",
       "   'brutally': 'present',\n",
       "   'kills': 'present',\n",
       "   'black': 'present',\n",
       "   'men': 'present',\n",
       "   'vandalizing': 'present',\n",
       "   'car': 'present',\n",
       "   'find': 'present',\n",
       "   'lands': 'present',\n",
       "   'prison': 'present',\n",
       "   'point': 'present',\n",
       "   'seen': 'present',\n",
       "   'eyes': 'present',\n",
       "   'present': 'present',\n",
       "   'time': 'present',\n",
       "   'high': 'present',\n",
       "   'school': 'present',\n",
       "   'eager': 'present',\n",
       "   'follow': 'present',\n",
       "   'footsteps': 'present',\n",
       "   'told': 'present',\n",
       "   'see': 'present',\n",
       "   'path': 'present',\n",
       "   'leads': 'present',\n",
       "   'adoption': 'present',\n",
       "   'white': 'present',\n",
       "   'supremacy': 'present',\n",
       "   'released': 'present',\n",
       "   'served': 'present',\n",
       "   'three': 'present',\n",
       "   'finds': 'present',\n",
       "   'full': 'present',\n",
       "   'blown': 'present',\n",
       "   'however': 'present',\n",
       "   'given': 'present',\n",
       "   'violence': 'present',\n",
       "   'tries': 'present',\n",
       "   'get': 'present',\n",
       "   'understand': 'present',\n",
       "   'comes': 'present',\n",
       "   'bad': 'present',\n",
       "   'things': 'present',\n",
       "   'interesting': 'present',\n",
       "   'stupid': 'present',\n",
       "   'thoughtless': 'present',\n",
       "   'people': 'present',\n",
       "   '--': 'present',\n",
       "   'intelligent': 'present',\n",
       "   'articulate': 'present',\n",
       "   'voice': 'present',\n",
       "   'beliefs': 'present',\n",
       "   'disturbingly': 'present',\n",
       "   'straightforward': 'present',\n",
       "   'terms': 'present',\n",
       "   'make': 'present',\n",
       "   'controversial': 'present',\n",
       "   'movie': 'present',\n",
       "   'preach': 'present',\n",
       "   'right': 'present',\n",
       "   'note': 'present',\n",
       "   'material': 'present',\n",
       "   'mainstream': 'present',\n",
       "   'redemption': 'present',\n",
       "   'phase': 'present',\n",
       "   'main': 'present',\n",
       "   'may': 'present',\n",
       "   'think': 'present',\n",
       "   'way': 'present',\n",
       "   'sympathetic': 'present',\n",
       "   'partially': 'present',\n",
       "   'disagree': 'present',\n",
       "   'although': 'present',\n",
       "   'advocate': 'present',\n",
       "   'presents': 'present',\n",
       "   'loud': 'present',\n",
       "   'obnoxious': 'present',\n",
       "   'also': 'present',\n",
       "   'smart': 'present',\n",
       "   'reasons': 'present',\n",
       "   'believable': 'present',\n",
       "   'father': 'present',\n",
       "   'arbitrarily': 'present',\n",
       "   'killed': 'present',\n",
       "   'group': 'present',\n",
       "   'clear': 'present',\n",
       "   'passionate': 'present',\n",
       "   'punk': 'present',\n",
       "   'looking': 'present',\n",
       "   'excuse': 'present',\n",
       "   'beat': 'present',\n",
       "   'course': 'present',\n",
       "   'helps': 'present',\n",
       "   'actor': 'present',\n",
       "   'talented': 'present',\n",
       "   'play': 'present',\n",
       "   'part': 'present',\n",
       "   'astonishing': 'present',\n",
       "   'frightening': 'present',\n",
       "   'looks': 'present',\n",
       "   'shaved': 'present',\n",
       "   'head': 'present',\n",
       "   'swastika': 'present',\n",
       "   'chest': 'present',\n",
       "   'addition': 'present',\n",
       "   'getting': 'present',\n",
       "   'perfect': 'present',\n",
       "   'requires': 'present',\n",
       "   'intelligence': 'present',\n",
       "   'depth': 'present',\n",
       "   'whole': 'present',\n",
       "   'lot': 'present',\n",
       "   'shouting': 'present',\n",
       "   'ease': 'present',\n",
       "   'even': 'present',\n",
       "   'meanest': 'present',\n",
       "   'likable': 'present',\n",
       "   'quality': 'present',\n",
       "   'gutsy': 'present',\n",
       "   'approach': 'present',\n",
       "   'telling': 'present',\n",
       "   'story': 'present',\n",
       "   'adds': 'present',\n",
       "   'subplot': 'present',\n",
       "   'principal': 'present',\n",
       "   'avery': 'present',\n",
       "   'brooks': 'present',\n",
       "   'obsessed': 'present',\n",
       "   'purging': 'present',\n",
       "   'hatred': 'present',\n",
       "   'terrific': 'present',\n",
       "   'standouts': 'present',\n",
       "   'visually': 'present',\n",
       "   'indulges': 'present',\n",
       "   'artistic': 'present',\n",
       "   'choices': 'present',\n",
       "   'nicely': 'present',\n",
       "   'lots': 'present',\n",
       "   'slow': 'present',\n",
       "   'motion': 'present',\n",
       "   'strange': 'present',\n",
       "   'camera': 'present',\n",
       "   'angles': 'present',\n",
       "   'add': 'present',\n",
       "   'moody': 'present',\n",
       "   'atmosphere': 'present',\n",
       "   'like': 'present',\n",
       "   'movies': 'present',\n",
       "   'lately': 'present',\n",
       "   'skims': 'present',\n",
       "   'past': 'present',\n",
       "   'greatness': 'present',\n",
       "   'last': 'present',\n",
       "   'minutes': 'present',\n",
       "   'climactic': 'present',\n",
       "   'scene': 'present',\n",
       "   'moving': 'present',\n",
       "   'picture': 'present',\n",
       "   'ends': 'present',\n",
       "   'pretentious': 'present',\n",
       "   'preachy': 'present',\n",
       "   'resolution': 'present',\n",
       "   'featuring': 'present',\n",
       "   'brief': 'present',\n",
       "   'narration': 'present',\n",
       "   'subtle': 'present',\n",
       "   'felt': 'present',\n",
       "   'slap': 'present',\n",
       "   'face': 'present',\n",
       "   'hand': 'present',\n",
       "   'fed': 'present',\n",
       "   'theme': 'present',\n",
       "   'simplistic': 'present',\n",
       "   'exactly': 'present',\n",
       "   'disliked': 'present',\n",
       "   'version': 'present',\n",
       "   'perhaps': 'present',\n",
       "   'problem': 'present',\n",
       "   'imagine': 'present',\n",
       "   'least': 'present',\n",
       "   'pleased': 'present',\n",
       "   'many': 'present',\n",
       "   'timid': 'present',\n",
       "   'weak': 'present',\n",
       "   'manages': 'present',\n",
       "   'compelling': 'present',\n",
       "   'argument': 'present',\n",
       "   'without': 'present',\n",
       "   'advocating': 'present'},\n",
       "  'pos'),\n",
       " ({'one': 'present',\n",
       "   'colleagues': 'present',\n",
       "   'surprised': 'present',\n",
       "   'told': 'present',\n",
       "   'willing': 'present',\n",
       "   'see': 'present',\n",
       "   'betsy': 'present',\n",
       "   'wedding': 'present',\n",
       "   'shocked': 'present',\n",
       "   'hear': 'present',\n",
       "   'actually': 'present',\n",
       "   'liked': 'present',\n",
       "   'reaction': 'present',\n",
       "   'understandable': 'present',\n",
       "   'consider': 'present',\n",
       "   'film': 'present',\n",
       "   'revolves': 'present',\n",
       "   'around': 'present',\n",
       "   'molly': 'present',\n",
       "   'ringwald': 'present',\n",
       "   'made': 'present',\n",
       "   'worthwhile': 'present',\n",
       "   'since': 'present',\n",
       "   '1986': 'present',\n",
       "   'fact': 'present',\n",
       "   'also': 'present',\n",
       "   'alan': 'present',\n",
       "   'alda': 'present',\n",
       "   'making': 'present',\n",
       "   'duds': 'present',\n",
       "   'last': 'present',\n",
       "   'four': 'present',\n",
       "   'years': 'present',\n",
       "   'involved': 'present',\n",
       "   'several': 'present',\n",
       "   'noteworthy': 'present',\n",
       "   'projects': 'present',\n",
       "   'including': 'present',\n",
       "   'crimes': 'present',\n",
       "   'misdemeanors': 'present',\n",
       "   'new': 'present',\n",
       "   'life': 'present',\n",
       "   'written': 'present',\n",
       "   'directed': 'present',\n",
       "   'vibrant': 'present',\n",
       "   'slice': 'present',\n",
       "   'mixing': 'present',\n",
       "   'dramatic': 'present',\n",
       "   'moments': 'present',\n",
       "   'big': 'present',\n",
       "   'bowl': 'present',\n",
       "   'whimsical': 'present',\n",
       "   'humor': 'present',\n",
       "   'comic': 'present',\n",
       "   'elixir': 'present',\n",
       "   'smooth': 'present',\n",
       "   'refreshing': 'present',\n",
       "   '--': 'present',\n",
       "   'welcome': 'present',\n",
       "   'change': 'present',\n",
       "   'pace': 'present',\n",
       "   'usual': 'present',\n",
       "   'summer': 'present',\n",
       "   'fare': 'present',\n",
       "   'bride': 'present',\n",
       "   'groom': 'present',\n",
       "   'dylan': 'present',\n",
       "   'walsh': 'present',\n",
       "   'pivotal': 'present',\n",
       "   'characters': 'present',\n",
       "   'far': 'present',\n",
       "   'least': 'present',\n",
       "   'interesting': 'present',\n",
       "   'nonentity': 'present',\n",
       "   'screen': 'present',\n",
       "   'presence': 'present',\n",
       "   'door': 'present',\n",
       "   'knob': 'present',\n",
       "   'simply': 'present',\n",
       "   'unbearable': 'present',\n",
       "   'easily': 'present',\n",
       "   'weakest': 'present',\n",
       "   'link': 'present',\n",
       "   'chain': 'present',\n",
       "   'looks': 'present',\n",
       "   'hideous': 'present',\n",
       "   'short': 'present',\n",
       "   'cropped': 'present',\n",
       "   'orange': 'present',\n",
       "   'hair': 'present',\n",
       "   'red': 'present',\n",
       "   'lip': 'present',\n",
       "   'stick': 'present',\n",
       "   'grotesque': 'present',\n",
       "   'outfits': 'present',\n",
       "   'supposed': 'present',\n",
       "   'dress': 'present',\n",
       "   'designer': 'present',\n",
       "   'like': 'present',\n",
       "   'clown': 'present',\n",
       "   'make': 'present',\n",
       "   'matters': 'present',\n",
       "   'worse': 'present',\n",
       "   'performance': 'present',\n",
       "   'matches': 'present',\n",
       "   'appearance': 'present',\n",
       "   'thankfully': 'present',\n",
       "   'keeps': 'present',\n",
       "   'time': 'present',\n",
       "   'minimum': 'present',\n",
       "   'interested': 'present',\n",
       "   'colorful': 'present',\n",
       "   'periphery': 'present',\n",
       "   'device': 'present',\n",
       "   'bring': 'present',\n",
       "   'together': 'present',\n",
       "   'working': 'present',\n",
       "   'class': 'present',\n",
       "   'italian': 'present',\n",
       "   'family': 'present',\n",
       "   'rich': 'present',\n",
       "   'gentile': 'present',\n",
       "   'folks': 'present',\n",
       "   'homey': 'present',\n",
       "   'earth': 'present',\n",
       "   'free': 'present',\n",
       "   'spirited': 'present',\n",
       "   'father': 'present',\n",
       "   'madeline': 'present',\n",
       "   'kahn': 'present',\n",
       "   'practical': 'present',\n",
       "   'mother': 'present',\n",
       "   'ally': 'present',\n",
       "   'sheedy': 'present',\n",
       "   'lonely': 'present',\n",
       "   'sister': 'present',\n",
       "   'clan': 'present',\n",
       "   'hand': 'present',\n",
       "   'prim': 'present',\n",
       "   'proper': 'present',\n",
       "   'ostentatious': 'present',\n",
       "   'two': 'present',\n",
       "   'families': 'present',\n",
       "   'meet': 'present',\n",
       "   'mingle': 'present',\n",
       "   'movie': 'present',\n",
       "   'becomes': 'present',\n",
       "   'story': 'present',\n",
       "   'culture': 'present',\n",
       "   'clash': 'present',\n",
       "   'character': 'present',\n",
       "   'puts': 'present',\n",
       "   'money': 'present',\n",
       "   'versus': 'present',\n",
       "   'values': 'present',\n",
       "   'wonderfully': 'present',\n",
       "   'understated': 'present',\n",
       "   'pleasant': 'present',\n",
       "   'surprises': 'present',\n",
       "   'expresses': 'present',\n",
       "   'eyes': 'present',\n",
       "   'entire': 'present',\n",
       "   'body': 'present',\n",
       "   'anthony': 'present',\n",
       "   'lapaglia': 'present',\n",
       "   'however': 'present',\n",
       "   'seizes': 'present',\n",
       "   'spotlight': 'present',\n",
       "   'plays': 'present',\n",
       "   'stevie': 'present',\n",
       "   'dee': 'present',\n",
       "   'suave': 'present',\n",
       "   'overly': 'present',\n",
       "   'polite': 'present',\n",
       "   'mafioso': 'present',\n",
       "   'formally': 'present',\n",
       "   'courting': 'present',\n",
       "   'old': 'present',\n",
       "   'fashioned': 'present',\n",
       "   'chivalry': 'present',\n",
       "   'sincere': 'present',\n",
       "   'dim': 'present',\n",
       "   'witted': 'present',\n",
       "   'riot': 'present',\n",
       "   'uncanny': 'present',\n",
       "   'dead': 'present',\n",
       "   'ringer': 'present',\n",
       "   'robert': 'present',\n",
       "   'de': 'present',\n",
       "   'niro': 'present',\n",
       "   'little': 'present',\n",
       "   'bit': 'present',\n",
       "   'alec': 'present',\n",
       "   'baldwin': 'present',\n",
       "   'thrown': 'present',\n",
       "   'good': 'present',\n",
       "   'measure': 'present',\n",
       "   'seems': 'present',\n",
       "   'attended': 'present',\n",
       "   'school': 'present',\n",
       "   'gangster': 'present',\n",
       "   'acting': 'present',\n",
       "   'inspired': 'present',\n",
       "   'partly': 'present',\n",
       "   'tribute': 'present',\n",
       "   'role': 'present',\n",
       "   'model': 'present',\n",
       "   'rip': 'present',\n",
       "   'know': 'present',\n",
       "   'whether': 'present',\n",
       "   'say': 'present',\n",
       "   'star': 'present',\n",
       "   'born': 'present',\n",
       "   'top': 'present',\n",
       "   'missed': 'present',\n",
       "   'scrumptious': 'present',\n",
       "   'extends': 'present',\n",
       "   'well': 'present',\n",
       "   'beyond': 'present',\n",
       "   'joe': 'present',\n",
       "   'pesci': 'present',\n",
       "   'particular': 'present',\n",
       "   'sinks': 'present',\n",
       "   'teeth': 'present',\n",
       "   'unscrupulous': 'present',\n",
       "   'brother': 'present',\n",
       "   'law': 'present',\n",
       "   'slum': 'present',\n",
       "   'lord': 'present',\n",
       "   'mob': 'present',\n",
       "   'ties': 'present',\n",
       "   'cheating': 'present',\n",
       "   'wife': 'present',\n",
       "   'catherine': 'present',\n",
       "   'hara': 'present',\n",
       "   'faced': 'present',\n",
       "   'challenge': 'present',\n",
       "   'directing': 'present',\n",
       "   'somehow': 'present',\n",
       "   'finds': 'present',\n",
       "   'right': 'present',\n",
       "   'touch': 'present',\n",
       "   'financially': 'present',\n",
       "   'strapped': 'present',\n",
       "   'carpenter': 'present',\n",
       "   'whose': 'present',\n",
       "   'dreams': 'present',\n",
       "   'bigger': 'present',\n",
       "   'wallet': 'present',\n",
       "   'adopts': 'present',\n",
       "   'psychological': 'present',\n",
       "   'point': 'present',\n",
       "   'view': 'present',\n",
       "   'tries': 'present',\n",
       "   'plan': 'present',\n",
       "   'pay': 'present',\n",
       "   'filmmaker': 'present',\n",
       "   'style': 'present',\n",
       "   'remarkably': 'present',\n",
       "   'restrained': 'present',\n",
       "   'tasteful': 'present',\n",
       "   'genius': 'present',\n",
       "   'woody': 'present',\n",
       "   'allen': 'present',\n",
       "   'possess': 'present',\n",
       "   'inspiration': 'present',\n",
       "   'movies': 'present',\n",
       "   'ten': 'present',\n",
       "   'times': 'present',\n",
       "   'entertaining': 'present',\n",
       "   'slop': 'present',\n",
       "   'usually': 'present',\n",
       "   'passes': 'present',\n",
       "   'comedy': 'present'},\n",
       "  'pos'),\n",
       " ({'bloody': 'present',\n",
       "   'clashes': 'present',\n",
       "   'independence': 'present',\n",
       "   'lumumba': 'present',\n",
       "   'refused': 'present',\n",
       "   'pander': 'present',\n",
       "   'belgians': 'present',\n",
       "   'continued': 'present',\n",
       "   'condescending': 'present',\n",
       "   'paternalistic': 'present',\n",
       "   'relationship': 'present',\n",
       "   'congo': 'present',\n",
       "   'officers': 'present',\n",
       "   'particularly': 'present',\n",
       "   'general': 'present',\n",
       "   'janssens': 'present',\n",
       "   'rudi': 'present',\n",
       "   'delhem': 'present',\n",
       "   'force': 'present',\n",
       "   'publique': 'present',\n",
       "   'army': 'present',\n",
       "   'caused': 'present',\n",
       "   'rebellions': 'present',\n",
       "   'undermining': 'present',\n",
       "   'outraged': 'present',\n",
       "   'rape': 'present',\n",
       "   'murder': 'present',\n",
       "   'belgian': 'present',\n",
       "   'nationals': 'present',\n",
       "   'unrest': 'present',\n",
       "   'building': 'present',\n",
       "   'moise': 'present',\n",
       "   'tshombe': 'present',\n",
       "   'pascal': 'present',\n",
       "   'nzonzi': 'present',\n",
       "   'province': 'present',\n",
       "   'katanga': 'present',\n",
       "   'contained': 'present',\n",
       "   '70': 'present',\n",
       "   'percent': 'present',\n",
       "   'country': 'present',\n",
       "   'resources': 'present',\n",
       "   'proclaimed': 'present',\n",
       "   'secession': 'present',\n",
       "   'replaced': 'present',\n",
       "   'making': 'present',\n",
       "   'mobutu': 'present',\n",
       "   'colonel': 'present',\n",
       "   'went': 'present',\n",
       "   'pacification': 'present',\n",
       "   'tour': 'present',\n",
       "   'congolese': 'present',\n",
       "   'president': 'present',\n",
       "   'joseph': 'present',\n",
       "   'kasa': 'present',\n",
       "   'vubu': 'present',\n",
       "   'maka': 'present',\n",
       "   'kotto': 'present',\n",
       "   'late': 'present',\n",
       "   'want': 'present',\n",
       "   'drown': 'present',\n",
       "   'dog': 'present',\n",
       "   'say': 'present',\n",
       "   'rabies': 'present',\n",
       "   'prophesies': 'present',\n",
       "   'fate': 'present',\n",
       "   'peck': 'present',\n",
       "   'bonitzer': 'present',\n",
       "   'exemplary': 'present',\n",
       "   'job': 'present',\n",
       "   'telling': 'present',\n",
       "   'complicated': 'present',\n",
       "   'tale': 'present',\n",
       "   'myriad': 'present',\n",
       "   'players': 'present',\n",
       "   'although': 'present',\n",
       "   'frequently': 'present',\n",
       "   'succumb': 'present',\n",
       "   'cliche': 'present',\n",
       "   'regarding': 'present',\n",
       "   'private': 'present',\n",
       "   'life': 'present',\n",
       "   'script': 'present',\n",
       "   'illuminates': 'present',\n",
       "   'bantu': 'present',\n",
       "   'sayings': 'present',\n",
       "   'like': 'present',\n",
       "   'hand': 'present',\n",
       "   'gives': 'present',\n",
       "   'rules': 'present',\n",
       "   'uses': 'present',\n",
       "   'american': 'present',\n",
       "   'ambassador': 'present',\n",
       "   'direction': 'present',\n",
       "   'less': 'present',\n",
       "   'assured': 'present',\n",
       "   'many': 'present',\n",
       "   'scenes': 'present',\n",
       "   'unfortunately': 'present',\n",
       "   'playing': 'present',\n",
       "   'standard': 'present',\n",
       "   'television': 'present',\n",
       "   'fare': 'present',\n",
       "   'served': 'present',\n",
       "   'well': 'present',\n",
       "   'though': 'present',\n",
       "   'casting': 'present',\n",
       "   'ebouaney': 'present',\n",
       "   'title': 'present',\n",
       "   'role': 'present',\n",
       "   'dynamic': 'present',\n",
       "   'radiating': 'present',\n",
       "   'character': 'present',\n",
       "   'fierce': 'present',\n",
       "   'passion': 'present',\n",
       "   'people': 'present',\n",
       "   'intelligence': 'present',\n",
       "   'ability': 'present',\n",
       "   'strategize': 'present',\n",
       "   'even': 'present',\n",
       "   'cornered': 'present',\n",
       "   'insurmountable': 'present',\n",
       "   'odds': 'present',\n",
       "   'given': 'present',\n",
       "   'subject': 'present',\n",
       "   'lead': 'present',\n",
       "   'actor': 'present',\n",
       "   'elevate': 'present',\n",
       "   'film': 'present',\n",
       "   'mediocre': 'present',\n",
       "   'production': 'present',\n",
       "   'story': 'present',\n",
       "   'deserves': 'present',\n",
       "   'told': 'present',\n",
       "   'performance': 'present',\n",
       "   'makes': 'present',\n",
       "   'tragedy': 'present',\n",
       "   'personally': 'present',\n",
       "   'felt': 'present'},\n",
       "  'pos'),\n",
       " ({'american': 'present',\n",
       "   'action': 'present',\n",
       "   'film': 'present',\n",
       "   'slowly': 'present',\n",
       "   'drowning': 'present',\n",
       "   'death': 'present',\n",
       "   'sea': 'present',\n",
       "   'asian': 'present',\n",
       "   'wire': 'present',\n",
       "   'fu': 'present',\n",
       "   'copycats': 'present',\n",
       "   'pretty': 'present',\n",
       "   'leaving': 'present',\n",
       "   'likes': 'present',\n",
       "   'schwartznager': 'present',\n",
       "   'stallone': 'present',\n",
       "   'van': 'present',\n",
       "   'damme': 'present',\n",
       "   'wearing': 'present',\n",
       "   'cement': 'present',\n",
       "   'galoshes': 'present',\n",
       "   'bottom': 'present',\n",
       "   'kung': 'present',\n",
       "   'sometimes': 'present',\n",
       "   'mix': 'present',\n",
       "   'results': 'present',\n",
       "   'mind': 'present',\n",
       "   'blowing': 'present',\n",
       "   'spectacle': 'present',\n",
       "   'unlike': 'present',\n",
       "   'quality': 'present',\n",
       "   'amazing': 'present',\n",
       "   'exciting': 'present',\n",
       "   'stunt': 'present',\n",
       "   'work': 'present',\n",
       "   '1999': 'present',\n",
       "   'matrix': 'present',\n",
       "   'real': 'present',\n",
       "   'gem': 'present',\n",
       "   'often': 'present',\n",
       "   'hollywood': 'present',\n",
       "   'gets': 'present',\n",
       "   'wrong': 'present',\n",
       "   'even': 'present',\n",
       "   'pay': 'present',\n",
       "   'chinese': 'present',\n",
       "   'directors': 'present',\n",
       "   'flying': 'present',\n",
       "   'ninjas': 'present',\n",
       "   'floating': 'present',\n",
       "   'karate': 'present',\n",
       "   'masters': 'present',\n",
       "   'replaced': 'present',\n",
       "   'soaring': 'present',\n",
       "   'bronx': 'present',\n",
       "   'detectives': 'present',\n",
       "   'slow': 'present',\n",
       "   'motion': 'present',\n",
       "   'kicking': 'present',\n",
       "   'scientists': 'present',\n",
       "   'mostly': 'present',\n",
       "   'laughable': 'present',\n",
       "   'rush': 'present',\n",
       "   'emulate': 'present',\n",
       "   'success': 'present',\n",
       "   'trademark': 'present',\n",
       "   'choreography': 'present',\n",
       "   'become': 'present',\n",
       "   'joke': 'present',\n",
       "   'art': 'present',\n",
       "   'form': 'present',\n",
       "   'iron': 'present',\n",
       "   'monkey': 'present',\n",
       "   'latest': 'present',\n",
       "   'import': 'present',\n",
       "   'shows': 'present',\n",
       "   'us': 'present',\n",
       "   'get': 'present',\n",
       "   'right': 'present',\n",
       "   'actually': 'present',\n",
       "   'reissue': 'present',\n",
       "   '1993': 'present',\n",
       "   'story': 'present',\n",
       "   '19th': 'present',\n",
       "   'vigilante': 'present',\n",
       "   'rongguang': 'present',\n",
       "   'yu': 'present',\n",
       "   'fighting': 'present',\n",
       "   'unique': 'present',\n",
       "   'style': 'present',\n",
       "   'shaolin': 'present',\n",
       "   'rights': 'present',\n",
       "   'oppressed': 'present',\n",
       "   'bellies': 'present',\n",
       "   'hungry': 'present',\n",
       "   'also': 'present',\n",
       "   'piece': 'present',\n",
       "   'narrative': 'present',\n",
       "   'legendary': 'present',\n",
       "   'hero': 'present',\n",
       "   'wong': 'present',\n",
       "   'fei': 'present',\n",
       "   'hong': 'present',\n",
       "   'recently': 'present',\n",
       "   'seen': 'present',\n",
       "   'one': 'present',\n",
       "   'overlooked': 'present',\n",
       "   'possibly': 'present',\n",
       "   'best': 'present',\n",
       "   'films': 'present',\n",
       "   '2000': 'present',\n",
       "   'drunken': 'present',\n",
       "   'master': 'present',\n",
       "   '2': 'present',\n",
       "   'released': 'present',\n",
       "   'u': 'present',\n",
       "   'legend': 'present',\n",
       "   'stars': 'present',\n",
       "   'jackie': 'present',\n",
       "   'chan': 'present',\n",
       "   'adult': 'present',\n",
       "   'finds': 'present',\n",
       "   'much': 'present',\n",
       "   'younger': 'present',\n",
       "   'sze': 'present',\n",
       "   'man': 'present',\n",
       "   'tsang': 'present',\n",
       "   'father': 'present',\n",
       "   'kei': 'present',\n",
       "   'ying': 'present',\n",
       "   'yen': 'present',\n",
       "   'chi': 'present',\n",
       "   'dan': 'present',\n",
       "   'thrust': 'present',\n",
       "   'middle': 'present',\n",
       "   'fight': 'present',\n",
       "   'oppression': 'present',\n",
       "   'succeeds': 'present',\n",
       "   'since': 'present',\n",
       "   'times': 'present',\n",
       "   'styles': 'present',\n",
       "   'especially': 'present',\n",
       "   'devolve': 'present',\n",
       "   'ridiculous': 'present',\n",
       "   'twinkle': 'present',\n",
       "   'toed': 'present',\n",
       "   'like': 'present',\n",
       "   'crouching': 'present',\n",
       "   'tiger': 'present',\n",
       "   'hidden': 'present',\n",
       "   'dragon': 'present',\n",
       "   'director': 'present',\n",
       "   'yuen': 'present',\n",
       "   'wo': 'present',\n",
       "   'ping': 'present',\n",
       "   'eventually': 'present',\n",
       "   'remembers': 'present',\n",
       "   'bring': 'present',\n",
       "   'scenes': 'present',\n",
       "   'back': 'present',\n",
       "   'earth': 'present',\n",
       "   'heart': 'present',\n",
       "   'hardcore': 'present',\n",
       "   'rather': 'present',\n",
       "   'kind': 'present',\n",
       "   'drama': 'present',\n",
       "   'la': 'present',\n",
       "   'however': 'present',\n",
       "   'brief': 'present',\n",
       "   'moments': 'present',\n",
       "   'profoundness': 'present',\n",
       "   'shared': 'present',\n",
       "   'characters': 'present',\n",
       "   'pass': 'present',\n",
       "   'outlaw': 'present',\n",
       "   'good': 'present',\n",
       "   'hearted': 'present',\n",
       "   'misguided': 'present',\n",
       "   'enemy': 'present',\n",
       "   'chief': 'present',\n",
       "   'fox': 'present',\n",
       "   'others': 'present',\n",
       "   'manages': 'present',\n",
       "   'transcend': 'present',\n",
       "   'mindless': 'present',\n",
       "   'nature': 'present',\n",
       "   'touch': 'present',\n",
       "   'hearts': 'present',\n",
       "   'minds': 'present',\n",
       "   'audience': 'present',\n",
       "   'way': 'present',\n",
       "   'equal': 'present',\n",
       "   'masterpiece': 'present',\n",
       "   'dances': 'present',\n",
       "   'quite': 'present',\n",
       "   'nicely': 'present',\n",
       "   'invading': 'present',\n",
       "   'tune': 'present',\n",
       "   'aka': 'present',\n",
       "   'siunin': 'present',\n",
       "   'hung': 'present',\n",
       "   'tsi': 'present',\n",
       "   'titmalau': 'present'},\n",
       "  'pos')]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pos_words[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "776278e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "79.60000000000001"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.classify import NaiveBayesClassifier #load the buildin classifier\n",
    "clf = NaiveBayesClassifier.train(pos_words[:500]+neg_words[:500])  \n",
    "#train it on 50% of records in positive and negative reviews\n",
    "nltk.classify.util.accuracy(clf, pos_words[500:]+neg_words[500:])*100  #test it on remaining 50% records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "25a14f33",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most Informative Features\n",
      "              whatsoever = 'present'         neg : pos    =     14.3 : 1.0\n",
      "                   inept = 'present'         neg : pos    =     13.7 : 1.0\n",
      "               ludicrous = 'present'         neg : pos    =     11.7 : 1.0\n",
      "              nomination = 'present'         pos : neg    =     11.7 : 1.0\n",
      "               stupidity = 'present'         neg : pos    =     11.7 : 1.0\n",
      "                   views = 'present'         pos : neg    =     10.3 : 1.0\n",
      "              vulnerable = 'present'         pos : neg    =     10.3 : 1.0\n",
      "               abilities = 'present'         neg : pos    =      9.7 : 1.0\n",
      "             exceptional = 'present'         pos : neg    =      9.7 : 1.0\n",
      "               fictional = 'present'         pos : neg    =      9.7 : 1.0\n"
     ]
    }
   ],
   "source": [
    "clf.show_most_informative_features()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97eced05",
   "metadata": {},
   "source": [
    "## SpaCy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51dc374b",
   "metadata": {},
   "source": [
    "Spacy is a free, opensource and industrial-strength library for NLP in Python. It provides a huge ecosystem of plugins and components, enables you to build custom components and workflows. Features of SpaCy are :\n",
    "\n",
    "- Support for 72+ languages\n",
    "- 80 trained pipelines for 24 languages\n",
    "- Multi-task learning with pretrained transformers like BERT\n",
    "- Pretrained word vectors\n",
    "- State-of-the-art speed\n",
    "- Production-ready training system\n",
    "- Linguistically-motivated tokenization\n",
    "- Components for named entity recognition, part-of-speech tagging, dependency parsing, sentence segmentation, text classification, lemmatization, morphological analysis, entity linking and more\n",
    "- Easily extensible with custom components and attributes\n",
    "- Support for custom models in PyTorch, TensorFlow and other frameworks\n",
    "- Built in visualizers for syntax and NER\n",
    "- Easy model packaging, deployment and workflow management\n",
    "- Robust, rigorously evaluated accuracy\n",
    "\n",
    "### Download and Install (https://spacy.io/usage)\n",
    "\n",
    "Using pip:\n",
    "<code>\n",
    "pip install -U pip setuptools wheel\n",
    "pip install spacy\n",
    "</code>\n",
    "\n",
    "Using conda:\n",
    "<code>conda install -c conda-forge spacy</code>\n",
    "\n",
    "\n",
    "After installing spacy, we need to download the suitable NLP pipeline of a language we usually work with (English in our case):\n",
    "\n",
    "<code>python -m spacy download en_core_web_sm</code>\n",
    "\n",
    "spaCy’s trained pipelines can be installed as Python packages. Trained pipelines can be installed from a download URL or a local directory, manually or via pip. Their data can be located anywhere on your file system.\n",
    "\n",
    "### Functionality provided by SpaCy:\n",
    "\n",
    "|NAME|\tDESCRIPTION|\n",
    "|:-|:-|\n",
    "|Tokenization|\tSegmenting text into words, punctuations marks etc.|\n",
    "|Part-of-speech (POS) Tagging|\tAssigning word types to tokens, like verb or noun.|\n",
    "|Dependency Parsing\t|Assigning syntactic dependency labels, describing the relations between individual tokens, like subject or object.|\n",
    "|Lemmatization|\tAssigning the base forms of words. For example, the lemma of “was” is “be”, and the lemma of “rats” is “rat”.|\n",
    "|Sentence Boundary Detection (SBD)\t|Finding and segmenting individual sentences.|\n",
    "|Named Entity Recognition (NER)\t|Labelling named “real-world” objects, like persons, companies or locations.|\n",
    "|Entity Linking (EL)|\tDisambiguating textual entities to unique identifiers in a knowledge base.|\n",
    "|Similarity|\tComparing words, text spans and documents and how similar they are to each other.|\n",
    "|Text Classification|\tAssigning categories or labels to a whole document, or parts of a document.|\n",
    "|Rule-based Matching|\tFinding sequences of tokens based on their texts and linguistic annotations, similar to regular expressions.|\n",
    "|Training|\tUpdating and improving a statistical model’s predictions.|\n",
    "|Serialization|\tSaving objects to files or byte strings.|\n",
    "\n",
    "\n",
    "### Pretrained Pipelines / Statistical models\n",
    "\n",
    "Many features of spacy require us to load a trained pipeline. A trained pipeline can consist of multiple components that use a statistical model trained on labeled data. spaCy pipelines can be installed as individual Python modules. The pipeline packages  differ in size, speed, memory usage, accuracy and the data they include. The package you choose depends on your use case and the texts you’re working with. For a general-purpose use case, the small, default packages are always a good start. They typically include the following components:\n",
    "\n",
    "    - Binary weights for the part-of-speech tagger, dependency parser and named entity recognizer to predict those annotations in context.\n",
    "    - Lexical entries in the vocabulary, i.e. words and their context-independent attributes like the shape or spelling.\n",
    "    - Data files like lemmatization rules and lookup tables.\n",
    "    - Word vectors, i.e. multi-dimensional meaning representations of words that let you determine how similar they are to each other.\n",
    "    - Configuration options, like the language and processing pipeline settings and model implementations to use, to put spaCy in the correct state when you load the pipeline.\n",
    "\n",
    "\n",
    "1. Once you’ve downloaded and installed a trained pipeline, you can load it via spacy.load(). This will return a <b>Language </b>object containing all components and data needed to process text. We usually call it <b>nlp</b>. Calling the nlp object on a string of text will return a processed <b>Doc</b> object.\n",
    "\n",
    "2. A Doc consists of individual tokens, and we can iterate over them.\n",
    "\n",
    "3. During further processing, spaCy first tokenizes the text, i.e. segments it into words, punctuation and so on. This is done by applying rules specific to each language. For example, punctuation at the end of a sentence should be split off – whereas “U.K.” should remain one token. Linguistic annotations are available as <b>Token</b> attributes.\n",
    "\n",
    "4. Part-of-speech tagging, identifying word dependencies, Named entity recognition (these tasks need a pretrained pipeline and a model). Named entities are available as the ents property of a Doc\n",
    "\n",
    "### The Processing Pipelines\n",
    "When you call <b>nlp</b> on a text, spaCy first tokenizes the text to produce a <b>Doc</b> object. The Doc is then processed in several different steps – which is referred to as the <b>processing pipeline</b>. The pipeline used by the trained pipelines typically include a <b>tagger, a lemmatizer, a parser and an entity recognizer</b>. Each pipeline component returns the processed Doc, which is then passed on to the next component.\n",
    "\n",
    "<img src = 'pipeline.svg'>\n",
    "<center> Source: https://spacy.io/usage/spacy-101 </center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "84631c1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple Apple PROPN NNP nsubj Xxxxx True False\n",
      "is be AUX VBZ aux xx True True\n",
      "looking look VERB VBG ROOT xxxx True False\n",
      "at at ADP IN prep xx True True\n",
      "buying buy VERB VBG pcomp xxxx True False\n",
      "U.K. U.K. PROPN NNP dobj X.X. False False\n",
      "startup startup NOUN NN dep xxxx True False\n",
      "for for ADP IN prep xxx True True\n",
      "$ $ SYM $ quantmod $ False False\n",
      "1 1 NUM CD compound d False False\n",
      "billion billion NUM CD pobj xxxx True False\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "text = \"Apple is looking at buying U.K. startup for $1 billion\"\n",
    "\n",
    "doc = nlp(text)\n",
    "\n",
    "for token in doc:\n",
    "    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,\n",
    "            token.shape_, token.is_alpha, token.is_stop)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c036063",
   "metadata": {},
   "source": [
    "#### Entity Recognition with Spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "39d46b7e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple 0 5 ORG\n",
      "U.K. 27 31 GPE\n",
      "$1 billion 44 54 MONEY\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"Apple is looking at buying U.K. startup for $1 billion\")\n",
    "\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.start_char, ent.end_char, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82cbe31d",
   "metadata": {},
   "source": [
    "#### Visualization of Dependencies among words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1036479c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"c18f69a4511a4772a03015bdcd782ed2-0\" class=\"displacy\" width=\"750\" height=\"312.0\" direction=\"ltr\" style=\"max-width: none; height: 312.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">This</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PRON</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">is</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">AUX</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">a</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">sentence.</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-c18f69a4511a4772a03015bdcd782ed2-0-0\" stroke-width=\"2px\" d=\"M70,177.0 C70,89.5 220.0,89.5 220.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-c18f69a4511a4772a03015bdcd782ed2-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,179.0 L62,167.0 78,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-c18f69a4511a4772a03015bdcd782ed2-0-1\" stroke-width=\"2px\" d=\"M420,177.0 C420,89.5 570.0,89.5 570.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-c18f69a4511a4772a03015bdcd782ed2-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M420,179.0 L412,167.0 428,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-c18f69a4511a4772a03015bdcd782ed2-0-2\" stroke-width=\"2px\" d=\"M245,177.0 C245,2.0 575.0,2.0 575.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-c18f69a4511a4772a03015bdcd782ed2-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">attr</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M575.0,179.0 L583.0,167.0 567.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import spacy\n",
    "from spacy import displacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(\"This is a sentence.\")\n",
    "displacy.render(doc, style=\"dep\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a5edd7c",
   "metadata": {},
   "source": [
    "## ChatGPT : A Large Language Model and a Generative Pre-trained Transformer (GPT)\n",
    "\n",
    "### What is language modelling?\n",
    "\n",
    "- Language modeling is the task of predicting the next word or character in a document. \n",
    "\n",
    "<img src = 'suggest-min.png' width = 400 height = 400>\n",
    "<center> Source:https://lena-voita.github.io/nlp_course/language_modeling.html </center>\n",
    "\n",
    "- A language model is a machine learning model that has been trained to learn the nuances of a language.  It can be used for a number of NLP tasks like text summarization, question answering, semantic search, dialog systems. \n",
    "\n",
    "- Some language models : GPT3 (Of course!!!), Megatron-LM, BERT\n",
    "\n",
    "To be Contd.."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}