text-nltk/nltk-1-intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab : TXT-1: Exploring NLTK\n",
    "\n",
    "## Overview\n",
    "Understand basic text processing and get to know NLTK library\n",
    "\n",
    "## Run time \n",
    "20 min\n",
    "\n",
    "## NLTK\n",
    "We will be using the excellent python text library called NLTK.\n",
    "Here are some references\n",
    "* [NLTK Home](http://www.nltk.org/)\n",
    "* [NLTK book](http://www.nltk.org/book/)\n",
    "* [NLTK data sets](http://www.nltk.org/nltk_data/)\n",
    "* [Accessing NLTK Corpus](http://www.nltk.org/howto/corpus.html)\n",
    "\n",
    "The following exercises will get you familiar with NLTK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: NLTK\n",
    "Let's make sure NLTK is installed.\n",
    "You can test if NLTK is installed by doing the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "print (\"nltk version : \", nltk.__version__)\n",
    "# is this working?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If not you can install NLTK as follows\n",
    "```bash\n",
    "   $   pip install nltk\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2:  Downloading NLTK Data\n",
    "NLTK library comes with a set of corpus data.  We have downloaded this data already.  It is in `data/nltk_data` directory.\n",
    "If you want to download it you can do so as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "import nltk\n",
    "nltk.download('all')\n",
    "\n",
    "## downloading using command line\n",
    "#! python -m nltk.downloader all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# import nltk\n",
    "# nltk.download('all')\n",
    "# # and choose the directory\n",
    "# # we will add this dierctory to `nltk.data.path`\n",
    "# from os.path import expanduser\n",
    "# nltk.data.path.append( expanduser(\"~\") + \"/data/nltk_data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Exploring NLTK datasets / corpus\n",
    "NLTK ships with lots of data sets.  Here are a few\n",
    "* words\n",
    "* State of the Union\n",
    "* Movie reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Words\n",
    "English words corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "# from os.path import expanduser\n",
    "# nltk.data.path.append( expanduser(\"~\") + \"/data/nltk_data\")\n",
    "from nltk.corpus import words\n",
    "\n",
    "print (words.readme())\n",
    "\n",
    "# TODO-2 : Get 'en' words\n",
    "w = words.words('en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO-2 : how many words are there?  \n",
    "## Hint : len(???)\n",
    "???\n",
    "\n",
    "## TODO-2 : Print some words out\n",
    "## Hint : use w[:n] (n is a number)\n",
    "???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 State of The Union text\n",
    "NLTK has lots of SOTU transcripts.  Try the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the names of the fileids\n",
    "from nltk.corpus import  state_unionunion\n",
    "print(state_union.fileids())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "# from os.path import expanduser\n",
    "# nltk.data.path.append( expanduser(\"~\") + \"/data/nltk_data\")\n",
    "from nltk.corpus import state_union\n",
    "\n",
    "print(state_union.readme())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO-2 : find all included files\n",
    "print(state_union.fileids())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO-3 : Extract the speech for 2006\n",
    "gw2006 = state_union.raw('???')\n",
    "print (gw2006)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO-4 : Extract words for the same speech\n",
    "gw2006_words = state_union.words('???')\n",
    "\n",
    "## TODO-5 : Print first 50 words of the speech\n",
    "print(gw2006_words[:???])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO-6 : Extract sentences\n",
    "gw2006_sents = state_union.sents('???')\n",
    "## Print first 10 sentences\n",
    "print (gw2006_sents[:???])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Novels!\n",
    "NLTK includes a bunch of classic novels from [Gutenberg project](http://www.gutenberg.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's see what the names of the files are\n",
    "from nltk.corpus import gutenberg\n",
    "\n",
    "print (gutenberg.fileids())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "# from os.path import expanduser\n",
    "# nltk.data.path.append( expanduser(\"~\") + \"/data/nltk_data\")\n",
    "from nltk.corpus import gutenberg\n",
    "\n",
    "print (gutenberg.readme())\n",
    "print (gutenberg.fileids())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO-1 : Get raw text for Moby Dick (use the fileids output above)\n",
    "moby_dick = gutenberg.raw('???')\n",
    "print (moby_dick[1:1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO-2 : Get words for Moby Dick\n",
    "moby_dick_words = gutenberg.???('???')\n",
    "## TODO-3 print the number of words (Hint : len)\n",
    "## TODO-4 : print first 100 words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO-5 : Count the number of sentences for Shakespear's Hamlet\n",
    "hamlet_senteces = gutenberg.???('???')\n",
    "\n",
    "## TODO : print out first 20 sentences\n",
    "print (hamlet_senteces[:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO-6 print some stats\n",
    "for f in gutenberg.fileids():\n",
    "    name = f\n",
    "    chars = len(gutenberg.raw(f))\n",
    "    words = len(???)\n",
    "    sentences = len(???)\n",
    "    print (f\"{f} :  {chars} chars,   {words} words,  {sentences} sentences\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Explore 'Movie Reviews' Corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "# from os.path import expanduser\n",
    "# nltk.data.path.append( expanduser(\"~\") + \"/data/nltk_data\")\n",
    "from nltk.corpus import movie_reviews\n",
    "\n",
    "print(movie_reviews.readme())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 : Tokenizing Text\n",
    "We are going to use `nltk.tokenize` library to tokenize texts.  There are 3 tokenizers we are goign to test\n",
    "* nltk.tokenize.word_tokenize\n",
    "* nltk.tokenize.wordpunct_tokenize\n",
    "* nltk.tokenize.sent_tokenize\n",
    "\n",
    "Try the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.tokenize import wordpunct_tokenize\n",
    "from nltk.tokenize import sent_tokenize\n",
    "\n",
    "# from os.path import expanduser\n",
    "# nltk.data.path.append( expanduser(\"~\") + \"/data/nltk_data\")\n",
    "\n",
    "text=\"\"\"I went to Starbucks. I bought a latte for $4.50!\n",
    "Yum :-)\n",
    "\"\"\"\n",
    "\n",
    "print(\"text:\\n\", text)\n",
    "print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Try `sent_tokenize`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "print(\"sent_tokenize\")\n",
    "print(sent_tokenize(???))  # sent_tokenize(text)\n",
    "print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Try `word_tokenize`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "print(\"word_tokenize\")\n",
    "print(word_tokenize(???))\n",
    "print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Try `wordpunct_tokeinze`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "print(\"word_tokenize\")\n",
    "print(wordpunct_tokenize(???))\n",
    "print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8: Compare tokenization outputs from above"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}