course-3/module-4-linear-classifier-regularization-assignment-blank.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression with L2 regularization\n",
    "\n",
    "The goal of this second notebook is to implement your own logistic regression classifier with L2 regularization. You will do the following:\n",
    "\n",
    " * Extract features from Amazon product reviews.\n",
    " * Convert an SFrame into a NumPy array.\n",
    " * Write a function to compute the derivative of log likelihood function with an L2 penalty with respect to a single coefficient.\n",
    " * Implement gradient ascent with an L2 penalty.\n",
    " * Empirically explore how the L2 penalty can ameliorate overfitting.\n",
    " \n",
    "# Fire up GraphLab Create\n",
    " \n",
    "Make sure you have the latest version of GraphLab Create. Upgrade by\n",
    "\n",
    "```\n",
    "   pip install graphlab-create --upgrade\n",
    "```\n",
    "See [this page](https://dato.com/download/) for detailed instructions on upgrading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import division\n",
    "import graphlab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and process review dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this assignment, we will use the same subset of the Amazon product review dataset that we used in Module 3 assignment. The subset was chosen to contain similar numbers of positive and negative reviews, as the original dataset consisted of mostly positive reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "products = graphlab.SFrame('amazon_baby_subset.gl/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like we did previously, we will work with a hand-curated list of important words extracted from the review data. We will also perform 2 simple data transformations:\n",
    "\n",
    "1. Remove punctuation using [Python's built-in](https://docs.python.org/2/library/string.html) string functionality.\n",
    "2. Compute word counts (only for the **important_words**)\n",
    "\n",
    "Refer to Module 3 assignment for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The same feature processing (same as the previous assignments)\n",
    "# ---------------------------------------------------------------\n",
    "import json\n",
    "with open('important_words.json', 'r') as f: # Reads the list of most frequent words\n",
    "    important_words = json.load(f)\n",
    "important_words = [str(s) for s in important_words]\n",
    "\n",
    "\n",
    "def remove_punctuation(text):\n",
    "    import string\n",
    "    return text.translate(None, string.punctuation) \n",
    "\n",
    "# Remove punctuation.\n",
    "products['review_clean'] = products['review'].apply(remove_punctuation)\n",
    "\n",
    "# Split out the words into individual columns\n",
    "for word in important_words:\n",
    "    products[word] = products['review_clean'].apply(lambda s : s.split().count(word))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Now, let us take a look at what the dataset looks like (**Note:** This may take a few minutes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "products"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train-Validation split\n",
    "\n",
    "We split the data into a train-validation split with 80% of the data in the training set and 20% of the data in the validation set. We use `seed=2` so that everyone gets the same result.\n",
    "\n",
    "**Note:** In previous assignments, we have called this a **train-test split**. However, the portion of data that we don't train on will be used to help **select model parameters**. Thus, this portion of data should be called a **validation set**. Recall that examining performance of various potential models (i.e. models with different parameters) should be on a validation set, while evaluation of selected model should always be on a test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_data, validation_data = products.random_split(.8, seed=2)\n",
    "\n",
    "print 'Training set   : %d data points' % len(train_data)\n",
    "print 'Validation set : %d data points' % len(validation_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert SFrame to NumPy array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Just like in the second assignment of the previous module, we provide you with a function that extracts columns from an SFrame and converts them into a NumPy array. Two arrays are returned: one representing features and another representing class labels. \n",
    "\n",
    "**Note:** The feature matrix includes an additional column 'intercept' filled with 1's to take account of the intercept term."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def get_numpy_data(data_sframe, features, label):\n",
    "    data_sframe['intercept'] = 1\n",
    "    features = ['intercept'] + features\n",
    "    features_sframe = data_sframe[features]\n",
    "    feature_matrix = features_sframe.to_numpy()\n",
    "    label_sarray = data_sframe[label]\n",
    "    label_array = label_sarray.to_numpy()\n",
    "    return(feature_matrix, label_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We convert both the training and validation sets into NumPy arrays.\n",
    "\n",
    "**Warning**: This may take a few minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "feature_matrix_train, sentiment_train = get_numpy_data(train_data, important_words, 'sentiment')\n",
    "feature_matrix_valid, sentiment_valid = get_numpy_data(validation_data, important_words, 'sentiment') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Are you running this notebook on an Amazon EC2 t2.micro instance?** (If you are using your own machine, please skip this section)\n",
    "\n",
    "It has been reported that t2.micro instances do not provide sufficient power to complete the conversion in acceptable amount of time. For interest of time, please refrain from running `get_numpy_data` function. Instead, download the [binary file](https://s3.amazonaws.com/static.dato.com/files/coursera/course-3/numpy-arrays/module-4-assignment-numpy-arrays.npz) containing the four NumPy arrays you'll need for the assignment. To load the arrays, run the following commands:\n",
    "```\n",
    "arrays = np.load('module-4-assignment-numpy-arrays.npz')\n",
    "feature_matrix_train, sentiment_train = arrays['feature_matrix_train'], arrays['sentiment_train']\n",
    "feature_matrix_valid, sentiment_valid = arrays['feature_matrix_valid'], arrays['sentiment_valid']\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building on logistic regression with no L2 penalty assignment\n",
    "\n",
    "Let us now build on Module 3 assignment. Recall from lecture that the link function for logistic regression can be defined as:\n",
    "\n",
    "$$\n",
    "P(y_i = +1 | \\mathbf{x}_i,\\mathbf{w}) = \\frac{1}{1 + \\exp(-\\mathbf{w}^T h(\\mathbf{x}_i))},\n",
    "$$\n",
    "\n",
    "where the feature vector $h(\\mathbf{x}_i)$ is given by the word counts of **important_words** in the review $\\mathbf{x}_i$. \n",
    "\n",
    "We will use the **same code** as in this past assignment to make probability predictions since this part is not affected by the L2 penalty.  (Only the way in which the coefficients are learned is affected by the addition of a regularization term.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def predict_probability(feature_matrix, coefficients):\n",
    "    '''\n",
    "    produces probablistic estimate for P(y_i = +1 | x_i, w).\n",
    "    estimate ranges between 0 and 1.\n",
    "    '''\n",
    "    # Take dot product of feature_matrix and coefficients  \n",
    "    ## YOUR CODE HERE\n",
    "    ...\n",
    "    \n",
    "    # Compute P(y_i = +1 | x_i, w) using the link function\n",
    "    ## YOUR CODE HERE\n",
    "    predictions = ...\n",
    "    \n",
    "    return predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding  L2 penalty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now work on extending logistic regression with L2 regularization. As discussed in the lectures, the L2 regularization is particularly useful in preventing overfitting. In this assignment, we will explore L2 regularization in detail.\n",
    "\n",
    "Recall from lecture and the previous assignment that for logistic regression without an L2 penalty, the derivative of the log likelihood function is:\n",
    "$$\n",
    "\\frac{\\partial\\ell}{\\partial w_j} = \\sum_{i=1}^N h_j(\\mathbf{x}_i)\\left(\\mathbf{1}[y_i = +1] - P(y_i = +1 | \\mathbf{x}_i, \\mathbf{w})\\right)\n",
    "$$\n",
    "\n",
    "** Adding L2 penalty to the derivative** \n",
    "\n",
    "It takes only a small modification to add a L2 penalty. All terms indicated in **red** refer to terms that were added due to an **L2 penalty**.\n",
    "\n",
    "* Recall from the lecture that the link function is still the sigmoid:\n",
    "$$\n",
    "P(y_i = +1 | \\mathbf{x}_i,\\mathbf{w}) = \\frac{1}{1 + \\exp(-\\mathbf{w}^T h(\\mathbf{x}_i))},\n",
    "$$\n",
    "* We add the L2 penalty term to the per-coefficient derivative of log likelihood:\n",
    "$$\n",
    "\\frac{\\partial\\ell}{\\partial w_j} = \\sum_{i=1}^N h_j(\\mathbf{x}_i)\\left(\\mathbf{1}[y_i = +1] - P(y_i = +1 | \\mathbf{x}_i, \\mathbf{w})\\right) \\color{red}{-2\\lambda w_j }\n",
    "$$\n",
    "\n",
    "The **per-coefficient derivative for logistic regression with an L2 penalty** is as follows:\n",
    "$$\n",
    "\\frac{\\partial\\ell}{\\partial w_j} = \\sum_{i=1}^N h_j(\\mathbf{x}_i)\\left(\\mathbf{1}[y_i = +1] - P(y_i = +1 | \\mathbf{x}_i, \\mathbf{w})\\right) \\color{red}{-2\\lambda w_j }\n",
    "$$\n",
    "and for the intercept term, we have\n",
    "$$\n",
    "\\frac{\\partial\\ell}{\\partial w_0} = \\sum_{i=1}^N h_0(\\mathbf{x}_i)\\left(\\mathbf{1}[y_i = +1] - P(y_i = +1 | \\mathbf{x}_i, \\mathbf{w})\\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: As we did in the Regression course, we do not apply the L2 penalty on the intercept. A large intercept does not necessarily indicate overfitting because the intercept is not associated with any particular feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write a function that computes the derivative of log likelihood with respect to a single coefficient $w_j$. Unlike its counterpart in the last assignment, the function accepts five arguments:\n",
    " * `errors` vector containing $(\\mathbf{1}[y_i = +1] - P(y_i = +1 | \\mathbf{x}_i, \\mathbf{w}))$ for all $i$\n",
    " * `feature` vector containing $h_j(\\mathbf{x}_i)$  for all $i$\n",
    " * `coefficient` containing the current value of coefficient $w_j$.\n",
    " * `l2_penalty` representing the L2 penalty constant $\\lambda$\n",
    " * `feature_is_constant` telling whether the $j$-th feature is constant or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def feature_derivative_with_L2(errors, feature, coefficient, l2_penalty, feature_is_constant): \n",
    "    \n",
    "    # Compute the dot product of errors and feature\n",
    "    ## YOUR CODE HERE\n",
    "    derivative = ...\n",
    "\n",
    "    # add L2 penalty term for any feature that isn't the intercept.\n",
    "    if not feature_is_constant: \n",
    "        ## YOUR CODE HERE\n",
    "        ...\n",
    "        \n",
    "    return derivative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Quiz question:** In the code above, was the intercept term regularized?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To verify the correctness of the gradient ascent algorithm, we provide a function for computing log likelihood (which we recall from the last assignment was a topic detailed in an advanced optional video, and used here for its numerical stability)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\ell\\ell(\\mathbf{w}) = \\sum_{i=1}^N \\Big( (\\mathbf{1}[y_i = +1] - 1)\\mathbf{w}^T h(\\mathbf{x}_i) - \\ln\\left(1 + \\exp(-\\mathbf{w}^T h(\\mathbf{x}_i))\\right) \\Big) \\color{red}{-\\lambda\\|\\mathbf{w}\\|_2^2} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def compute_log_likelihood_with_L2(feature_matrix, sentiment, coefficients, l2_penalty):\n",
    "    indicator = (sentiment==+1)\n",
    "    scores = np.dot(feature_matrix, coefficients)\n",
    "    \n",
    "    lp = np.sum((indicator-1)*scores - np.log(1. + np.exp(-scores))) - l2_penalty*np.sum(coefficients[1:]**2)\n",
    "    \n",
    "    return lp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Quiz question:** Does the term with L2 regularization increase or decrease $\\ell\\ell(\\mathbf{w})$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The logistic regression function looks almost like the one in the last assignment, with a minor modification to account for the L2 penalty.  Fill in the code below to complete this modification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def logistic_regression_with_L2(feature_matrix, sentiment, initial_coefficients, step_size, l2_penalty, max_iter):\n",
    "    coefficients = np.array(initial_coefficients) # make sure it's a numpy array\n",
    "    for itr in xrange(max_iter):\n",
    "        # Predict P(y_i = +1|x_i,w) using your predict_probability() function\n",
    "        ## YOUR CODE HERE\n",
    "        predictions = ...\n",
    "        \n",
    "        # Compute indicator value for (y_i = +1)\n",
    "        indicator = (sentiment==+1)\n",
    "        \n",
    "        # Compute the errors as indicator - predictions\n",
    "        errors = indicator - predictions\n",
    "        for j in xrange(len(coefficients)): # loop over each coefficient\n",
    "            is_intercept = (j == 0)\n",
    "            # Recall that feature_matrix[:,j] is the feature column associated with coefficients[j].\n",
    "            # Compute the derivative for coefficients[j]. Save it in a variable called derivative\n",
    "            ## YOUR CODE HERE\n",
    "            derivative = ...\n",
    "            \n",
    "            # add the step size times the derivative to the current coefficient\n",
    "            ## YOUR CODE HERE\n",
    "            ...\n",
    "        \n",
    "        # Checking whether log likelihood is increasing\n",
    "        if itr <= 15 or (itr <= 100 and itr % 10 == 0) or (itr <= 1000 and itr % 100 == 0) \\\n",
    "        or (itr <= 10000 and itr % 1000 == 0) or itr % 10000 == 0:\n",
    "            lp = compute_log_likelihood_with_L2(feature_matrix, sentiment, coefficients, l2_penalty)\n",
    "            print 'iteration %*d: log likelihood of observed labels = %.8f' % \\\n",
    "                (int(np.ceil(np.log10(max_iter))), itr, lp)\n",
    "    return coefficients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explore effects of L2 regularization\n",
    "\n",
    "Now that we have written up all the pieces needed for regularized logistic regression, let's explore the benefits of using **L2 regularization** in analyzing sentiment for product reviews. **As iterations pass, the log likelihood should increase**.\n",
    "\n",
    "Below, we train models with increasing amounts of regularization, starting with no L2 penalty, which is equivalent to our previous logistic regression implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# run with L2 = 0\n",
    "coefficients_0_penalty = logistic_regression_with_L2(feature_matrix_train, sentiment_train,\n",
    "                                                     initial_coefficients=np.zeros(194),\n",
    "                                                     step_size=5e-6, l2_penalty=0, max_iter=501)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# run with L2 = 4\n",
    "coefficients_4_penalty = logistic_regression_with_L2(feature_matrix_train, sentiment_train,\n",
    "                                                      initial_coefficients=np.zeros(194),\n",
    "                                                      step_size=5e-6, l2_penalty=4, max_iter=501)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# run with L2 = 10\n",
    "coefficients_10_penalty = logistic_regression_with_L2(feature_matrix_train, sentiment_train,\n",
    "                                                      initial_coefficients=np.zeros(194),\n",
    "                                                      step_size=5e-6, l2_penalty=10, max_iter=501)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# run with L2 = 1e2\n",
    "coefficients_1e2_penalty = logistic_regression_with_L2(feature_matrix_train, sentiment_train,\n",
    "                                                       initial_coefficients=np.zeros(194),\n",
    "                                                       step_size=5e-6, l2_penalty=1e2, max_iter=501)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# run with L2 = 1e3\n",
    "coefficients_1e3_penalty = logistic_regression_with_L2(feature_matrix_train, sentiment_train,\n",
    "                                                       initial_coefficients=np.zeros(194),\n",
    "                                                       step_size=5e-6, l2_penalty=1e3, max_iter=501)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# run with L2 = 1e5\n",
    "coefficients_1e5_penalty = logistic_regression_with_L2(feature_matrix_train, sentiment_train,\n",
    "                                                       initial_coefficients=np.zeros(194),\n",
    "                                                       step_size=5e-6, l2_penalty=1e5, max_iter=501)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare coefficients\n",
    "\n",
    "We now compare the **coefficients** for each of the models that were trained above. We will create a table of features and learned coefficients associated with each of the different L2 penalty values.\n",
    "\n",
    "Below is a simple helper function that will help us create this table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "table = graphlab.SFrame({'word': ['(intercept)'] + important_words})\n",
    "def add_coefficients_to_table(coefficients, column_name):\n",
    "    table[column_name] = coefficients\n",
    "    return table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now, let's run the function `add_coefficients_to_table` for each of the L2 penalty strengths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "add_coefficients_to_table(coefficients_0_penalty, 'coefficients [L2=0]')\n",
    "add_coefficients_to_table(coefficients_4_penalty, 'coefficients [L2=4]')\n",
    "add_coefficients_to_table(coefficients_10_penalty, 'coefficients [L2=10]')\n",
    "add_coefficients_to_table(coefficients_1e2_penalty, 'coefficients [L2=1e2]')\n",
    "add_coefficients_to_table(coefficients_1e3_penalty, 'coefficients [L2=1e3]')\n",
    "add_coefficients_to_table(coefficients_1e5_penalty, 'coefficients [L2=1e5]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using **the coefficients trained with L2 penalty 0**, find the 5 most positive words (with largest positive coefficients). Save them to **positive_words**. Similarly, find the 5 most negative words (with largest negative coefficients) and save them to **negative_words**.\n",
    "\n",
    "**Quiz Question**. Which of the following is **not** listed in either **positive_words** or **negative_words**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us observe the effect of increasing L2 penalty on the 10 words just selected. We provide you with a utility function to  plot the coefficient path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = 10, 6\n",
    "\n",
    "def make_coefficient_plot(table, positive_words, negative_words, l2_penalty_list):\n",
    "    cmap_positive = plt.get_cmap('Reds')\n",
    "    cmap_negative = plt.get_cmap('Blues')\n",
    "    \n",
    "    xx = l2_penalty_list\n",
    "    plt.plot(xx, [0.]*len(xx), '--', lw=1, color='k')\n",
    "    \n",
    "    table_positive_words = table.filter_by(column_name='word', values=positive_words)\n",
    "    table_negative_words = table.filter_by(column_name='word', values=negative_words)\n",
    "    del table_positive_words['word']\n",
    "    del table_negative_words['word']\n",
    "    \n",
    "    for i in xrange(len(positive_words)):\n",
    "        color = cmap_positive(0.8*((i+1)/(len(positive_words)*1.2)+0.15))\n",
    "        plt.plot(xx, table_positive_words[i:i+1].to_numpy().flatten(),\n",
    "                 '-', label=positive_words[i], linewidth=4.0, color=color)\n",
    "        \n",
    "    for i in xrange(len(negative_words)):\n",
    "        color = cmap_negative(0.8*((i+1)/(len(negative_words)*1.2)+0.15))\n",
    "        plt.plot(xx, table_negative_words[i:i+1].to_numpy().flatten(),\n",
    "                 '-', label=negative_words[i], linewidth=4.0, color=color)\n",
    "        \n",
    "    plt.legend(loc='best', ncol=3, prop={'size':16}, columnspacing=0.5)\n",
    "    plt.axis([1, 1e5, -1, 2])\n",
    "    plt.title('Coefficient path')\n",
    "    plt.xlabel('L2 penalty ($\\lambda$)')\n",
    "    plt.ylabel('Coefficient value')\n",
    "    plt.xscale('log')\n",
    "    plt.rcParams.update({'font.size': 18})\n",
    "    plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to generate the plot. Use the plot to answer the following quiz question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "make_coefficient_plot(table, positive_words, negative_words, l2_penalty_list=[0, 4, 10, 1e2, 1e3, 1e5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question**: (True/False) All coefficients consistently get smaller in size as the L2 penalty is increased.\n",
    "\n",
    "**Quiz Question**: (True/False) The relative order of coefficients is preserved as the L2 penalty is increased. (For example, if the coefficient for 'cat' was more positive than that for 'dog', this remains true as the L2 penalty increases.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Measuring accuracy\n",
    "\n",
    "Now, let us compute the accuracy of the classifier model. Recall that the accuracy is given by\n",
    "\n",
    "$$\n",
    "\\mbox{accuracy} = \\frac{\\mbox{# correctly classified data points}}{\\mbox{# total data points}}\n",
    "$$\n",
    "\n",
    "\n",
    "Recall from lecture that that the class prediction is calculated using\n",
    "$$\n",
    "\\hat{y}_i = \n",
    "\\left\\{\n",
    "\\begin{array}{ll}\n",
    "      +1 & h(\\mathbf{x}_i)^T\\mathbf{w} > 0 \\\\\n",
    "      -1 & h(\\mathbf{x}_i)^T\\mathbf{w} \\leq 0 \\\\\n",
    "\\end{array} \n",
    "\\right.\n",
    "$$\n",
    "\n",
    "**Note**: It is important to know that the model prediction code doesn't change even with the addition of an L2 penalty. The only thing that changes is the estimated coefficients used in this prediction.\n",
    "\n",
    "Based on the above, we will use the same code that was used in Module 3 assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_classification_accuracy(feature_matrix, sentiment, coefficients):\n",
    "    scores = np.dot(feature_matrix, coefficients)\n",
    "    apply_threshold = np.vectorize(lambda x: 1. if x > 0  else -1.)\n",
    "    predictions = apply_threshold(scores)\n",
    "    \n",
    "    num_correct = (predictions == sentiment).sum()\n",
    "    accuracy = num_correct / len(feature_matrix)    \n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we compare the accuracy on the **training data** and **validation data** for all the models that were trained in this assignment.  We first calculate the accuracy values and then build a simple report summarizing the performance for the various models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_accuracy = {}\n",
    "train_accuracy[0]   = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_0_penalty)\n",
    "train_accuracy[4]   = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_4_penalty)\n",
    "train_accuracy[10]  = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_10_penalty)\n",
    "train_accuracy[1e2] = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_1e2_penalty)\n",
    "train_accuracy[1e3] = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_1e3_penalty)\n",
    "train_accuracy[1e5] = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_1e5_penalty)\n",
    "\n",
    "validation_accuracy = {}\n",
    "validation_accuracy[0]   = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_0_penalty)\n",
    "validation_accuracy[4]   = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_4_penalty)\n",
    "validation_accuracy[10]  = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_10_penalty)\n",
    "validation_accuracy[1e2] = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_1e2_penalty)\n",
    "validation_accuracy[1e3] = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_1e3_penalty)\n",
    "validation_accuracy[1e5] = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_1e5_penalty)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Build a simple report\n",
    "for key in sorted(validation_accuracy.keys()):\n",
    "    print \"L2 penalty = %g\" % key\n",
    "    print \"train accuracy = %s, validation_accuracy = %s\" % (train_accuracy[key], validation_accuracy[key])\n",
    "    print \"--------------------------------------------------------------------------------\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Quiz question**: Which model (L2 = 0, 4, 10, 100, 1e3, 1e5) has the **highest** accuracy on the **training** data?\n",
    "* **Quiz question**: Which model (L2 = 0, 4, 10, 100, 1e3, 1e5) has the **highest** accuracy on the **validation** data?\n",
    "* **Quiz question**: Does the **highest** accuracy on the **training** data imply that the model is the best one?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}