unconditionally.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unconditionally Universal Speeches\n",
    "\n",
    "People speak a lot.\n",
    "\n",
    "For example: politicians when they hold a speech.\n",
    "\n",
    "Playing with the linguistic concept of Language Universals, I wrote some code that weeds through words, and allows the reader a maybe pensive, maybe revealing, but most probably just ten-seconds-fun-amusing, digest of some past US president's mumblings.\n",
    "\n",
    "Hope you'll enjoy : )\n",
    "\n",
    "And choose your words wisely 😜"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unconditional Language Universals\n",
    "\n",
    "Linguistics defines two types of **Language Universals** for natural human languages: _unconditional_ ones and _conditional_ ones.\n",
    "\n",
    "And actually their difference is smartly explained in the derivation and semantics of the two words. (Oh, those linguists... 😉 )\n",
    "\n",
    "While _conditional_ Language Universals rely on some conditions to hold up (e.g. \"if a language has _inflection_, it usually also has _derivation_\"), **unconditional Language Universals** are true without further prerequisites.\n",
    "\n",
    "### Here I will focus on one of the unconditional LUs, namely:\n",
    "\n",
    "> Every language has nouns and verbs.\n",
    "\n",
    "Easy.\n",
    "\n",
    "---\n",
    "\n",
    ">**DISCLAIMER:** THE REST OF THESE MUSINGS FROM HERE DOWNWARDS ARE NOTHING BUT SELF-MADE MIND-GAMES FOR FUN AND PROGRAMMING PRACTICE.\n",
    "Just keep that in mind.\n",
    "\n",
    "So, nouns and verbs are the **essential** Parts-of-speech in every language.\n",
    "\n",
    "I wanted to see what happens to my text understanding when stripping a written text from all but those essential POS.\n",
    "\n",
    "Ready for some code? Here we go!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import state_union\n",
    "from nltk.tokenize import sent_tokenize\n",
    "from nltk.tokenize import word_tokenize\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next I choose one speech for analysis (you can take a different one. there are many other possibilities!)\n",
    "\n",
    "Just choose from the wealth of the corpus.\n",
    "You can check which speeches exists with:\n",
    "\n",
    ">`state_union.fileids()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "kennedy2_full = state_union.raw(\"1962-Kennedy.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time for some functions. I hope they are well explained in docstrings and descriptive naming. Feedback welcome."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def tokenize_text(text):\n",
    "    \"\"\"applies sentence and word tokenization and returns a nested list of sentences and words\"\"\"\n",
    "    s_tknzd = sent_tokenize(text)\n",
    "    all_words = []\n",
    "    for sentence in s_tknzd:\n",
    "        try:\n",
    "            wordset = word_tokenize(sentence)\n",
    "            all_words.append(wordset)\n",
    "        except Exception as e:\n",
    "            print(str(e))\n",
    "    return all_words\n",
    "\n",
    "def tag_POS(nested_word_list):\n",
    "    \"\"\"tags each word in a nested list of words with the POS information\"\"\"\n",
    "    tagged_text = []\n",
    "    for sentence in nested_word_list:\n",
    "        tagged = nltk.pos_tag(sentence)\n",
    "        tagged_text.append(tagged)\n",
    "    return tagged_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "text_POS_tagged = tag_POS(tokenize_text(kennedy2_full))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cloze_text(tagged_text, *args):\n",
    "    \"\"\"applies cloze deletion to all words in a text that have different POS than entered in the *args.\n",
    "    \n",
    "    takes a POS tagged text as input\n",
    "    as well as an arbitrary number of POS abbreviations\n",
    "    replaces each word that is a different POS with a blank\n",
    "    returns a nested list of sentences and words that only keep the words\n",
    "    that associate with the entered POS.\n",
    "    \"\"\"\n",
    "    clozed_text = []\n",
    "    for sen in tagged_text:\n",
    "        clozed_sen = []\n",
    "        for word_inf in sen:\n",
    "            word = word_inf[0]\n",
    "            POS = word_inf[1]\n",
    "            if POS not in args:\n",
    "                # change this to \"_____\" if you prefer a more standard cloze deletion\n",
    "                # I changed this for compactness and better display in the HTML\n",
    "                clozed_sen.append(\".\")\n",
    "            else:\n",
    "                clozed_sen.append(word)\n",
    "        clozed_text.append(clozed_sen)\n",
    "    return clozed_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "essentials = cloze_text(text_POS_tagged, \"NNP\", \"NNPS\", \"NN\", \"NNS\", \"VB\", \"VBD\", \"VBG\", \"VBN\", \"VBP\", \"VBZ\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using all of the different versions of nouns and verbs that have their [POS in NLTK](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html), this amounts to quite some words. Because I didn't want to read so much, I also tried a version with stripping the text down to **only proper nouns**. It's fun 😁"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "noun_speak = cloze_text(text_POS_tagged, \"NNP\", \"NNPS\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def create_speech(clozed_text):\n",
    "    \"\"\"creates a string from a nested list\"\"\"\n",
    "    essentialized_text = \"\"\n",
    "    for s in clozed_text:\n",
    "        sentence = \" \".join(s)\n",
    "        essentialized_text += sentence + \"\\n\"\n",
    "    return essentialized_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So below follows the \"proper-noun version\" of Kennedy's State Union Speech. Each dot represents a deleted word.\n",
    "\n",
    "Of course all those words were utterly unimportant. Or... are they?\n",
    "\n",
    "Hm, let's make a little experiment:\n",
    "> I think you are really an extremely nice person! That is because\n",
    "\n",
    "_(lots and lots of words, let's continue this in \"essentials\")_\n",
    "\n",
    "> . are . . . . . . .\n",
    "\n",
    "Ah, yep! I see why someone would care for adjectives! 😉\n",
    "\n",
    "Anyways, uncomment to read the beginning of the speech after rigorous cloze deletion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PRESIDENT JOHN F. KENNEDY . ANNUAL ADDRESS TO A JOINT SESSION OF CONGRESS ON THE STATE . THE UNION . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . . . .\n",
      ". Mister Sam . Rayburn . . .\n",
      ". . House . . . . . . . . .\n",
      ". . . Congress . . Constitution . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . .\n",
      ". . . . . . . State . . Union . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . North . . South . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . .\n",
      ". . ECONOMY . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . Mr. Khrushchev . . . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . .\n",
      ". . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . Congress . . . First . . Manpower Training . Development Act . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Second . . Youth Employment Opportunities Act . . . . . . . . . . . . Americans . . . . . . . . . . . . . . . . Americans . . . . . . . . . Third . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . First . Presidential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Second . Presidential . . . . . . . . . . . . . . . Federal . . . . . . . Third . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Congress . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . World War II .\n",
      ". . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n",
      ". . . . . . . . . . . . . . . . . . . . . . . . . . . . Government . . . . . . . . . . . . . . . . . . . . Federal Pay Reform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Federal Budget .\n",
      ". . . . . . . . . . . . . . . . . . . . . First . . . . . . . . . . . . . Secondly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Third . . . . . . . . . . . . . . . . . . . . . . . . . . .\n",
      "GETTING AMERICA MOVING . . . . . . . . . . . Budget .\n",
      ". . . . . . . . . . . . . . .\n",
      ". A . America . . . . . America . . . . . America . . .\n",
      ". . . . . . . . . . . . . . . . .\n"
     ]
    }
   ],
   "source": [
    "#print(create_speech(noun_speak)[:3000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, so that's fun for me. Let's get it into a format that is easier to digest!\n",
    "\n",
    "For that aim I'll rewrite the functions to create a simple HTML page for viewing and joyful reading and contemplation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_speech_for_web(clozed_text, filename):\n",
    "    \"\"\"creates a simple HTML website from a nested list\"\"\"\n",
    "    essentialized_text = \"\"\"<!DOCTYPE html>\n",
    "<html>\n",
    "<head>\n",
    "\t<title>Essential Speeches</title>\n",
    "\t<link href=\"https://fonts.googleapis.com/css?family=Amatic+SC\" rel=\"stylesheet\">\n",
    "\t<style type=\"text/css\">\n",
    "\t\tbody {\n",
    "\t\t\tmargin: auto;\n",
    "\t\t\ttext-align: center;\n",
    "\t\t\tmax-width: 800px;\n",
    "\t\t\tfont-family: 'Amatic SC', cursive;\n",
    "\t\t\tfont-size: 2.3em;\n",
    "\t\t\tbackground-image: url(\"https://cdn.pixabay.com/photo/2015/09/22/12/18/paper-951489_960_720.jpg\");\n",
    "\t\t}\n",
    "\t\tp {\n",
    "\t\t\tpadding: 20px;\n",
    "\t\t}\n",
    "\t</style>\n",
    "</head>\n",
    "<body>\"\"\"\n",
    "    for s in clozed_text:\n",
    "        sentence = \" \".join(s)\n",
    "        essentialized_text += \"<p>\" + sentence + \"</p>\"\n",
    "    essentialized_text += \"\"\"</body>\n",
    "</html>\"\"\"\n",
    "    with open(filename, \"w\") as f:\n",
    "        f.write(essentialized_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# calling the function to make some websites\n",
    "#create_speech_for_web(essentials, \"uncondLU-ized_speech.html\")\n",
    "#create_speech_for_web(noun_speak, \"word.html\") # <- I think this one is fun :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_real(president_speech):\n",
    "    \"\"\"creates a cloze-deleted website of a president's speech.\n",
    "    \n",
    "    takes as input a string of a state union speech .txt file\n",
    "    runs it through the pipeline of functions and creates a website.\n",
    "    if the speech is not correctly specified, it returns a list with available options.\n",
    "    \"\"\"\n",
    "    try:\n",
    "        speech = state_union.raw(president_speech)\n",
    "        tokenized_text = tokenize_text(speech)\n",
    "        text_POS_tagged = tag_POS(tokenized_text)\n",
    "        # change the POS *args here for different results\n",
    "        # check this page for all options:\n",
    "        # https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html\n",
    "        essentials = cloze_text(text_POS_tagged, \"NNP\", \"NNPS\", \"NN\", \"NNS\", \"VB\", \"VBD\", \"VBG\", \"VBN\", \"VBP\", \"VBZ\")\n",
    "        create_speech_for_web(essentials, \"index.html\")\n",
    "    except:\n",
    "        speeches = state_union.fileids()\n",
    "        print(\"Please try again. Enter a STRING.\")\n",
    "        print(\"These are your options:\\n\")\n",
    "        print(speeches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "After wrapping and adapting the code some more, it can output a HTML page (with some fun font and background 😜 )\n",
    "\n",
    "# Tldr;: just do the following for VISUAL RESULTS:\n",
    "\n",
    "### Choose your favorite (or least favorite) president, run the code below on his speech, and see what he was really* saying!\n",
    "\n",
    ">*_\"really\"_ = unconditionally-language-universally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "get_real('1961-Kennedy.txt')"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:car]",
   "language": "python",
   "name": "conda-env-car-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}