text-nltk/nltk-1-intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab : TXT-1: Exploring NLTK\n",
    "\n",
    "## Overview\n",
    "Understand basic text processing and get to know NLTK library\n",
    "\n",
    "## Run time \n",
    "20 min\n",
    "\n",
    "## NLTK\n",
    "We will be using the excellent python text library called NLTK.\n",
    "Here are some references\n",
    "* [NLTK Home](http://www.nltk.org/)\n",
    "* [NLTK book](http://www.nltk.org/book/)\n",
    "* [NLTK data sets](http://www.nltk.org/nltk_data/)\n",
    "* [Accessing NLTK Corpus](http://www.nltk.org/howto/corpus.html)\n",
    "\n",
    "The following exercises will get you familiar with NLTK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: NLTK\n",
    "Let's make sure NLTK is installed.\n",
    "You can test if NLTK is installed by doing the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "print (\"nltk version : \", nltk.__version__)\n",
    "# is this working?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If not you can install NLTK as follows\n",
    "```bash\n",
    "   $   pip install nltk\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2:  Downloading NLTK Data\n",
    "NLTK library comes with a set of corpus data.  We have downloaded this data already.  It is in `data/nltk_data` directory.\n",
    "If you want to download it you can do so as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "import nltk\n",
    "nltk.download('all')\n",
    "\n",
    "## downloading using command line\n",
    "#! python -m nltk.downloader all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import nltk\n",
    "# nltk.download()\n",
    "# # and choose the directory\n",
    "# # we will add this dierctory to `nltk.data.path`\n",
    "# from os.path import expanduser\n",
    "# nltk.data.path.append( expanduser(\"~\") + \"/data/nltk_data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Exploring NLTK datasets / corpus\n",
    "NLTK ships with lots of data sets.  Here are a few\n",
    "* words\n",
    "* State of the Union\n",
    "* Movie reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Words\n",
    "English words corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "# from os.path import expanduser\n",
    "# nltk.data.path.append( expanduser(\"~\") + \"/data/nltk_data\")\n",
    "from nltk.corpus import words\n",
    "\n",
    "print (words.readme())\n",
    "\n",
    "# TODO-2 : Get 'en' words\n",
    "w = words.words('en')\n",
    "\n",
    "## TODO-2 : how many words are there?  \n",
    "## Hint : len(???)\n",
    "print (\"len(words) : \", len(w))\n",
    "\n",
    "## TODO-2 : Print some words out\n",
    "## Hint : use w[:n] (n is a number)\n",
    "print (w[:20])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 State of The Union text\n",
    "NLTK has lots of SOTU transcripts.  Try the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C-Span State of the Union Address Corpus\n",
      "\n",
      "Annual US presidential addresses 1945-2006\n",
      "\n",
      "http://www.c-span.org/executive/stateoftheunion.asp\n",
      "\n",
      "(Thanks to Kathleen Ahrens for compiling this corpus from\n",
      "the C-Span sources.)\n",
      "\n",
      "\n",
      "----\n",
      "['1945-Truman.txt', '1946-Truman.txt', '1947-Truman.txt', '1948-Truman.txt', '1949-Truman.txt', '1950-Truman.txt', '1951-Truman.txt', '1953-Eisenhower.txt', '1954-Eisenhower.txt', '1955-Eisenhower.txt', '1956-Eisenhower.txt', '1957-Eisenhower.txt', '1958-Eisenhower.txt', '1959-Eisenhower.txt', '1960-Eisenhower.txt', '1961-Kennedy.txt', '1962-Kennedy.txt', '1963-Johnson.txt', '1963-Kennedy.txt', '1964-Johnson.txt', '1965-Johnson-1.txt', '1965-Johnson-2.txt', '1966-Johnson.txt', '1967-Johnson.txt', '1968-Johnson.txt', '1969-Johnson.txt', '1970-Nixon.txt', '1971-Nixon.txt', '1972-Nixon.txt', '1973-Nixon.txt', '1974-Nixon.txt', '1975-Ford.txt', '1976-Ford.txt', '1977-Ford.txt', '1978-Carter.txt', '1979-Carter.txt', '1980-Carter.txt', '1981-Reagan.txt', '1982-Reagan.txt', '1983-Reagan.txt', '1984-Reagan.txt', '1985-Reagan.txt', '1986-Reagan.txt', '1987-Reagan.txt', '1988-Reagan.txt', '1989-Bush.txt', '1990-Bush.txt', '1991-Bush-1.txt', '1991-Bush-2.txt', '1992-Bush.txt', '1993-Clinton.txt', '1994-Clinton.txt', '1995-Clinton.txt', '1996-Clinton.txt', '1997-Clinton.txt', '1998-Clinton.txt', '1999-Clinton.txt', '2000-Clinton.txt', '2001-GWBush-1.txt', '2001-GWBush-2.txt', '2002-GWBush.txt', '2003-GWBush.txt', '2004-GWBush.txt', '2005-GWBush.txt', '2006-GWBush.txt']\n",
      "----\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "# from os.path import expanduser\n",
    "# nltk.data.path.append( expanduser(\"~\") + \"/data/nltk_data\")\n",
    "from nltk.corpus import state_union\n",
    "\n",
    "print(state_union.readme())\n",
    "print(\"----\")\n",
    "\n",
    "## TODO-2 : find all included files\n",
    "print(state_union.fileids())\n",
    "print(\"----\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO-3 : Extract the speech for 2006\n",
    "gw2006 = state_union.raw('2006-GWBush.txt')\n",
    "print (gw2006)\n",
    "print(\"----\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- first 50 words \n",
      "['PRESIDENT', 'GEORGE', 'W', '.', 'BUSH', \"'\", 'S', 'ADDRESS', 'BEFORE', 'A', 'JOINT', 'SESSION', 'OF', 'THE', 'CONGRESS', 'ON', 'THE', 'STATE', 'OF', 'THE', 'UNION', 'January', '31', ',', '2006', 'THE', 'PRESIDENT', ':', 'Thank', 'you', 'all', '.', 'Mr', '.', 'Speaker', ',', 'Vice', 'President', 'Cheney', ',', 'members', 'of', 'Congress', ',', 'members', 'of', 'the', 'Supreme', 'Court', 'and']\n",
      "------\n"
     ]
    }
   ],
   "source": [
    "## TODO-4 : Extract words for the same speech\n",
    "gw2006_words = state_union.words('2006-GWBush.txt')\n",
    "\n",
    "## TODO-5 : Print first 50 words of the speech\n",
    "print (\"--- first 50 words \")\n",
    "print(gw2006_words[:50])\n",
    "print (\"------\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- first 10 sentences ---\n",
      "['PRESIDENT', 'GEORGE', 'W', '.', 'BUSH', \"'\", 'S', 'ADDRESS', 'BEFORE', 'A', 'JOINT', 'SESSION', 'OF', 'THE', 'CONGRESS', 'ON', 'THE', 'STATE', 'OF', 'THE', 'UNION']\n",
      "['January', '31', ',', '2006']\n",
      "['THE', 'PRESIDENT', ':', 'Thank', 'you', 'all', '.']\n",
      "['Mr', '.', 'Speaker', ',', 'Vice', 'President', 'Cheney', ',', 'members', 'of', 'Congress', ',', 'members', 'of', 'the', 'Supreme', 'Court', 'and', 'diplomatic', 'corps', ',', 'distinguished', 'guests', ',', 'and', 'fellow', 'citizens', ':', 'Today', 'our', 'nation', 'lost', 'a', 'beloved', ',', 'graceful', ',', 'courageous', 'woman', 'who', 'called', 'America', 'to', 'its', 'founding', 'ideals', 'and', 'carried', 'on', 'a', 'noble', 'dream', '.']\n",
      "['Tonight', 'we', 'are', 'comforted', 'by', 'the', 'hope', 'of', 'a', 'glad', 'reunion', 'with', 'the', 'husband', 'who', 'was', 'taken', 'so', 'long', 'ago', ',', 'and', 'we', 'are', 'grateful', 'for', 'the', 'good', 'life', 'of', 'Coretta', 'Scott', 'King', '.']\n",
      "['(', 'Applause', '.)']\n",
      "['President', 'George', 'W', '.', 'Bush', 'reacts', 'to', 'applause', 'during', 'his', 'State', 'of', 'the', 'Union', 'Address', 'at', 'the', 'Capitol', ',', 'Tuesday', ',', 'Jan', '.', '31', ',', '2006', '.']\n",
      "['White', 'House', 'photo', 'by', 'Eric', 'DraperEvery', 'time', 'I', \"'\", 'm', 'invited', 'to', 'this', 'rostrum', ',', 'I', \"'\", 'm', 'humbled', 'by', 'the', 'privilege', ',', 'and', 'mindful', 'of', 'the', 'history', 'we', \"'\", 've', 'seen', 'together', '.']\n",
      "['We', 'have', 'gathered', 'under', 'this', 'Capitol', 'dome', 'in', 'moments', 'of', 'national', 'mourning', 'and', 'national', 'achievement', '.']\n",
      "['We', 'have', 'served', 'America', 'through', 'one', 'of', 'the', 'most', 'consequential', 'periods', 'of', 'our', 'history', '--', 'and', 'it', 'has', 'been', 'my', 'honor', 'to', 'serve', 'with', 'you', '.']\n",
      "------\n"
     ]
    }
   ],
   "source": [
    "## TODO-6 : Extract sentences\n",
    "gw2006_sents = state_union.sents('2006-GWBush.txt')\n",
    "## Print first 10 sentences\n",
    "print (\"--- first 10 sentences ---\")\n",
    "for s in gw2006_sents[:10]:\n",
    "    print (s)\n",
    "print (\"------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Novels!\n",
    "NLTK includes a bunch of classic novels from [Gutenberg project](http://www.gutenberg.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['austen-emma.txt', 'austen-persuasion.txt', 'austen-sense.txt', 'bible-kjv.txt', 'blake-poems.txt', 'bryant-stories.txt', 'burgess-busterbrown.txt', 'carroll-alice.txt', 'chesterton-ball.txt', 'chesterton-brown.txt', 'chesterton-thursday.txt', 'edgeworth-parents.txt', 'melville-moby_dick.txt', 'milton-paradise.txt', 'shakespeare-caesar.txt', 'shakespeare-hamlet.txt', 'shakespeare-macbeth.txt', 'whitman-leaves.txt']\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "# from os.path import expanduser\n",
    "# nltk.data.path.append( expanduser(\"~\") + \"/data/nltk_data\")\n",
    "from nltk.corpus import gutenberg\n",
    "\n",
    "# print (gutenberg.readme())\n",
    "print (gutenberg.fileids())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---- moby_dick[1:1000]\n",
      "Moby Dick by Herman Melville 1851]\n",
      "\n",
      "\n",
      "ETYMOLOGY.\n",
      "\n",
      "(Supplied by a Late Consumptive Usher to a Grammar School)\n",
      "\n",
      "The pale Usher--threadbare in coat, heart, body, and brain; I see him\n",
      "now.  He was ever dusting his old lexicons and grammars, with a queer\n",
      "handkerchief, mockingly embellished with all the gay flags of all the\n",
      "known nations of the world.  He loved to dust his old grammars; it\n",
      "somehow mildly reminded him of his mortality.\n",
      "\n",
      "\"While you take in hand to school others, and to teach them by what\n",
      "name a whale-fish is to be called in our tongue leaving out, through\n",
      "ignorance, the letter H, which almost alone maketh the signification\n",
      "of the word, you deliver that which is not true.\" --HACKLUYT\n",
      "\n",
      "\"WHALE. ... Sw. and Dan. HVAL.  This animal is named from roundness\n",
      "or rolling; for in Dan. HVALT is arched or vaulted.\" --WEBSTER'S\n",
      "DICTIONARY\n",
      "\n",
      "\"WHALE. ... It is more immediately from the Dut. and Ger. WALLEN;\n",
      "A.S. WALW-IAN, to roll, to wallow.\" --RICHARDSON'S DICTIONARY\n",
      "\n",
      "----\n",
      "len(moby_dick_words) :  260819\n",
      "---- first 100 words ----\n",
      "['[', 'Moby', 'Dick', 'by', 'Herman', 'Melville', '1851', ']', 'ETYMOLOGY', '.', '(', 'Supplied', 'by', 'a', 'Late', 'Consumptive', 'Usher', 'to', 'a', 'Grammar', 'School', ')', 'The', 'pale', 'Usher', '--', 'threadbare', 'in', 'coat', ',', 'heart', ',', 'body', ',', 'and', 'brain', ';', 'I', 'see', 'him', 'now', '.', 'He', 'was', 'ever', 'dusting', 'his', 'old', 'lexicons', 'and']\n",
      "-----\n"
     ]
    }
   ],
   "source": [
    "## TODO-1 : Get raw text for Moby Dick (use the fileids output above)\n",
    "moby_dick = gutenberg.raw('melville-moby_dick.txt')\n",
    "\n",
    "print (\"---- moby_dick[1:1000]----\")\n",
    "print (moby_dick[1:1000])\n",
    "print ('----')\n",
    "\n",
    "## TODO-2 : Get words for Moby Dick\n",
    "moby_dick_words = gutenberg.words('melville-moby_dick.txt')\n",
    "print (\"len(moby_dick_words) : \", len(moby_dick_words))\n",
    "## TODO-3 print the number of words (Hint : len)\n",
    "## TODO-4 : print first 100 words\n",
    "print ('---- first 100 words ----')\n",
    "print (moby_dick_words[:50])\n",
    "print ('-----')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "len(hamlet_senteces) :   3106\n"
     ]
    }
   ],
   "source": [
    "## TODO-5 : Count the number of sentences for Shakespear's Hamlet\n",
    "hamlet_senteces = gutenberg.sents('shakespeare-hamlet.txt')\n",
    "print (\"len(hamlet_senteces) :  \", len(hamlet_senteces))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "austen-emma.txt :  887071 chars,   192427 words,  7752 sentences\n",
      "\n",
      "austen-persuasion.txt :  466292 chars,   98171 words,  3747 sentences\n",
      "\n",
      "austen-sense.txt :  673022 chars,   141576 words,  4999 sentences\n",
      "\n",
      "bible-kjv.txt :  4332554 chars,   1010654 words,  30103 sentences\n",
      "\n",
      "blake-poems.txt :  38153 chars,   8354 words,  438 sentences\n",
      "\n",
      "bryant-stories.txt :  249439 chars,   55563 words,  2863 sentences\n",
      "\n",
      "burgess-busterbrown.txt :  84663 chars,   18963 words,  1054 sentences\n",
      "\n",
      "carroll-alice.txt :  144395 chars,   34110 words,  1703 sentences\n",
      "\n",
      "chesterton-ball.txt :  457450 chars,   96996 words,  4779 sentences\n",
      "\n",
      "chesterton-brown.txt :  406629 chars,   86063 words,  3806 sentences\n",
      "\n",
      "chesterton-thursday.txt :  320525 chars,   69213 words,  3742 sentences\n",
      "\n",
      "edgeworth-parents.txt :  935158 chars,   210663 words,  10230 sentences\n",
      "\n",
      "melville-moby_dick.txt :  1242990 chars,   260819 words,  10059 sentences\n",
      "\n",
      "milton-paradise.txt :  468220 chars,   96825 words,  1851 sentences\n",
      "\n",
      "shakespeare-caesar.txt :  112310 chars,   25833 words,  2163 sentences\n",
      "\n",
      "shakespeare-hamlet.txt :  162881 chars,   37360 words,  3106 sentences\n",
      "\n",
      "shakespeare-macbeth.txt :  100351 chars,   23140 words,  1907 sentences\n",
      "\n",
      "whitman-leaves.txt :  711215 chars,   154883 words,  4250 sentences\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## TODO-6 print some stats\n",
    "for f in gutenberg.fileids():\n",
    "    name = f\n",
    "    chars = len(gutenberg.raw(f))\n",
    "    words = len(gutenberg.words(f))\n",
    "    sentences = len(gutenberg.sents(f))\n",
    "    print (f\"{f} :  {chars} chars,   {words} words,  {sentences} sentences\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Explore 'Movie Reviews' Corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "# from os.path import expanduser\n",
    "# nltk.data.path.append( expanduser(\"~\") + \"/data/nltk_data\")\n",
    "from nltk.corpus import movie_reviews\n",
    "\n",
    "print(movie_reviews.readme())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 : Tokenizing Text\n",
    "We are going to use `nltk.tokenize` library to tokenize texts.  There are 3 tokenizers we are goign to test\n",
    "* nltk.tokenize.word_tokenize\n",
    "* nltk.tokenize.wordpunct_tokenize\n",
    "* nltk.tokenize.sent_tokenize\n",
    "\n",
    "Try the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text:\n",
      " I went to Starbucks. I bought a latte for $4.50!\n",
      "Yum :-)\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.tokenize import wordpunct_tokenize\n",
    "from nltk.tokenize import sent_tokenize\n",
    "\n",
    "## TODO-1 : check this path\n",
    "# from os.path import expanduser\n",
    "# nltk.data.path.append( expanduser(\"~\") + \"/data/nltk_data\")\n",
    "\n",
    "text=\"\"\"I went to Starbucks. I bought a latte for $4.50!\n",
    "Yum :-)\n",
    "\"\"\"\n",
    "\n",
    "print(\"text:\\n\", text)\n",
    "print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Try `sent_tokenize`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sent_tokenize\n",
      "['I went to Starbucks.', 'I bought a latte for $4.50!', 'Yum :-)']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"sent_tokenize\")\n",
    "print(sent_tokenize(text))  # sent_tokenize(text)\n",
    "print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Try `word_tokenize`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "word_tokenize\n",
      "['I', 'went', 'to', 'Starbucks', '.', 'I', 'bought', 'a', 'latte', 'for', '$', '4.50', '!', 'Yum', ':', '-', ')']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"word_tokenize\")\n",
    "print(word_tokenize(text))\n",
    "print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Try `wordpunct_tokeinze`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "word_tokenize\n",
      "['I', 'went', 'to', 'Starbucks', '.', 'I', 'bought', 'a', 'latte', 'for', '$', '4', '.', '50', '!', 'Yum', ':-)']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"word_tokenize\")\n",
    "print(wordpunct_tokenize(text))\n",
    "print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8: Compare tokenization outputs from above"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}