(trainer)_Using_RNN_for_Sentiment_Analysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "(trainer) Using RNN for Sentiment Analysis.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kumar4372/sentiment_analysis_hands_on/blob/master/(trainer)_Using_RNN_for_Sentiment_Analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2DYHYfgAauWJ"
      },
      "source": [
        "# **Sentiment Analysis Using Recurrent Neural Network**\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5PpfRxiBX7HT"
      },
      "source": [
        "## choose Hardware accelarator to \"GPU\" for faster computation. Go to \"Runtime\" -> \"Change runtime type\" to change it.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OXpXUF4cbI0b"
      },
      "source": [
        "In this tutorial, we will use RNN/LSTM for sentiment analysis on movie review dataset.\n",
        "\n",
        "**What is sentiment analysis?**\n",
        "\n",
        "Sentiment Analysis is nothing but finding the sentiments of reviews whether it is positive or negative review.\n",
        "\n",
        "**Example Code to refer**: https://keras.io/examples/nlp/bidirectional_lstm_imdb/\n",
        "\n",
        "**Notes**\n",
        "- RNNs are tricky. Choice of batch size is important,\n",
        "choice of loss and optimizer is critical, etc.\n",
        "Some configurations won't converge.\n",
        "- LSTM loss decrease patterns during training can be quite different\n",
        "from what you see with CNNs/MLPs/etc."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YAZNYKDpbL_4"
      },
      "source": [
        "**Importing Libraries**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jzhfuZIPuajX"
      },
      "source": [
        "We start by importing the required dependencies to preprocess our data and build our model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8jZZT3W6j60O",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7e9663bf-e7d1-4180-d14b-fe972b7410b2"
      },
      "source": [
        "# Import the dependencies\n",
        "from tensorflow.keras.datasets import imdb\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Embedding, Dense, SimpleRNN,LSTM, GRU\n",
        "from tensorflow.keras.preprocessing import sequence\n",
        "print(\"Imported dependencies.\")"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Imported dependencies.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LWcnwxoJjnxd"
      },
      "source": [
        "**Loading Data**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WRzugSCquNN5"
      },
      "source": [
        "We will use IMDB sentiment classification dataset which consists of 50,000 movie reviews from IMDB users that are labeled as either positive (1) or negative (0). \n",
        "\n",
        "Continue downloading the IMDB dataset, which is, fortunately, already built into keras."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jlgiU4PSkLUn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "709ff51f-4197-4773-a98b-45b83149ea85"
      },
      "source": [
        "vocab_size = 10000\n",
        "\n",
        "# Define the training and test dataset\n",
        "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=vocab_size)  # vocab_size is no. of words to consider from the dataset, ordering based on frequency.\n",
        "\n",
        "print(\"Created test and training data.\")\n",
        "print(len(x_train), 'train sequences')\n",
        "print(len(x_test), 'test sequences')"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Created test and training data.\n",
            "25000 train sequences\n",
            "25000 test sequences\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OKwL1Y-aslN7"
      },
      "source": [
        "**Exploring the data**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CmS3ls6YyVLN"
      },
      "source": [
        "You can see in the output above that the dataset is labeled into two categories, — 0 or 1, which represents the sentiment of the review. The whole dataset contains 9,998 unique words and the average review length is 234 words, with a standard deviation of 173 words."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RODKEpHUlaH8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "15b91948-02f9-475f-f9c1-ac22f38b505a"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "#concatenate whole data\n",
        "data = np.concatenate((x_train, x_test), axis=0)\n",
        "targets = np.concatenate((y_train, y_test), axis=0)\n",
        "\n",
        "print(\"Categories:\", np.unique(targets))\n",
        "print(\"Number of unique words:\", len(np.unique(np.hstack(data))))\n",
        "len_sequence_list = [len(i) for i in data]\n",
        "print(\"Average Review length:\", np.mean(len_sequence_list))\n",
        "print(\"Standard Deviation:\", round(np.std(len_sequence_list)))"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Categories: [0 1]\n",
            "Number of unique words: 9998\n",
            "Average Review length: 234.75892\n",
            "Standard Deviation: 173\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r1mgv2N_07F8"
      },
      "source": [
        "We should always check how balanced our training and test data is. This helps in deciding evaluation metrics and observing training progress as well. You will observe that both training and test data is perfectly balanced."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VWIgu1y01I09",
        "outputId": "3971b096-3eda-42d7-c56f-8356497e901b"
      },
      "source": [
        "# since labels are only 0 and 1, np.sum will give you total number of examples with label 1.\n",
        "print(\"percentage of test sequences with label 1 is\", (np.sum(y_test)/len(y_test)*100))\n",
        "print(\"percentage of train sequences with label 1 is\", (np.sum(y_train)/len(y_train)*100))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "percentage of test sequences with label 1 is 50.0\n",
            "percentage of train sequences with label 1 is 50.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G3Ya_GxcuETI"
      },
      "source": [
        "You can see the first review of the dataset, which is labeled as positive (1). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TdOq8BHEmEXg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bfec116f-385b-454b-d36c-ad66857cbfa1"
      },
      "source": [
        "print('---review---')\n",
        "print(x_train[0])\n",
        "print('---label---')\n",
        "print(y_train[0])"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "---review---\n",
            "[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]\n",
            "---label---\n",
            "1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JLWk5R9huA5l"
      },
      "source": [
        "Now we try to map from word index to word so that we can read the reviews.\n",
        "We replace every unknown word with a “#”. It does this by using the get_word_index() function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-TW5kQUWmQDP"
      },
      "source": [
        "index = imdb.get_word_index() # from word to index mapping\n",
        "reverse_index = dict([(value, key) for (key, value) in index.items()]) # from index to word mapping"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-ogflKtFBbwf",
        "outputId": "f31d0555-2bd2-458c-e842-21257b3b3b54",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(index['there']) # we get 47\n",
        "print(reverse_index[47])"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "47\n",
            "there\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kJHHWlzKBcCF",
        "outputId": "06d881e7-8062-4208-ca55-d14a04c8189e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "train_text = []\n",
        "test_text = []\n",
        "for i in range(0,len(x_train)):\n",
        "  train_text.append(\" \".join( [reverse_index.get(j - 3, \"#\") for j in x_train[i]] ))\n",
        "for i in range(0,len(x_test)):\n",
        "  test_text.append(\" \".join( [reverse_index.get(j - 3, \"#\") for j in x_test[i]] ))\n",
        "print(len(train_text),len(test_text))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "25000 25000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "adNJ_dvIV566",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f567ca20-aca6-4404-90ab-48f300506b94"
      },
      "source": [
        "print(train_text[0])"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "# this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert # is an amazing actor and now the same being director # father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for # and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also # to the two little boy's that played the # of norman and paul they were just brilliant children are often left out of the # list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AtEQG_0nqzz3"
      },
      "source": [
        "**Data Preparation**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1B-OZoHct2qi"
      },
      "source": [
        "Now it's time to prepare our data. \n",
        "\n",
        "As we know, each review consists of different number of words. Some reviews could even be of length 1. e.g. \"nice\"\n",
        "\n",
        "We need to fix maximum length of our input sequence. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "842z0XWIuDOo",
        "outputId": "c1321318-6a09-4a8f-8687-f0388f3e283b"
      },
      "source": [
        "from collections import Counter\n",
        "count_length = Counter(len_sequence_list)\n",
        "count_length[200]"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "142"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8XPeGSTsuGuR"
      },
      "source": [
        "Check how many sentences are of length less than x where x is any integer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cr4lzF42uERT",
        "outputId": "11ee3cb5-0a8d-46de-f9ca-45c5413026a9"
      },
      "source": [
        "sum([count_length[i] for i in range(300)]) # x is 300 here"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "38501"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1QvGkeDKuyPq"
      },
      "source": [
        "Here we consider maximum length of our input sequence to be 200.\n",
        "\n",
        "**Please feel free to choose your own maximum length**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DNS1Jr-omVC3"
      },
      "source": [
        "max_review_length = 200\n",
        "x_train_padded = sequence.pad_sequences(x_train, maxlen=max_review_length)\n",
        "x_test_padded = sequence.pad_sequences(x_test, maxlen=max_review_length)"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mDLJtIfowuMZ"
      },
      "source": [
        "To visualize how padding is happening, let's print a few examples of seqeunces for before and after padding\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L_eUysG_fucz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "824fa728-b4c3-42a9-e46a-62257e305831"
      },
      "source": [
        "# for sequence length > maximum sequence length, we remove entries from the beginning. \n",
        "# If you want to remove entries from the end, you should use truncating='post' in pad_sequences in the above cell\n",
        "i = 0\n",
        "print(len(x_train[i]))\n",
        "print(x_train[i])\n",
        "print(len(x_train_padded[i]))\n",
        "print(x_train_padded[i])"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "218\n",
            "[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]\n",
            "200\n",
            "[   5   25  100   43  838  112   50  670    2    9   35  480  284    5\n",
            "  150    4  172  112  167    2  336  385   39    4  172 4536 1111   17\n",
            "  546   38   13  447    4  192   50   16    6  147 2025   19   14   22\n",
            "    4 1920 4613  469    4   22   71   87   12   16   43  530   38   76\n",
            "   15   13 1247    4   22   17  515   17   12   16  626   18    2    5\n",
            "   62  386   12    8  316    8  106    5    4 2223 5244   16  480   66\n",
            " 3785   33    4  130   12   16   38  619    5   25  124   51   36  135\n",
            "   48   25 1415   33    6   22   12  215   28   77   52    5   14  407\n",
            "   16   82    2    8    4  107  117 5952   15  256    4    2    7 3766\n",
            "    5  723   36   71   43  530  476   26  400  317   46    7    4    2\n",
            " 1029   13  104   88    4  381   15  297   98   32 2071   56   26  141\n",
            "    6  194 7486   18    4  226   22   21  134  476   26  480    5  144\n",
            "   30 5535   18   51   36   28  224   92   25  104    4  226   65   16\n",
            "   38 1334   88   12   16  283    5   16 4472  113  103   32   15   16\n",
            " 5345   19  178   32]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7jDIdSwmwFkX",
        "outputId": "8ae07420-8e07-4347-ec4e-9ed2e3be4d42"
      },
      "source": [
        "# for sequence length < maximum sequence length, we put zeros in the beginning. \n",
        "# If you want to put zeros in the end, you should use padding='post' in pad_sequences in the above cell\n",
        "i = 1\n",
        "print(len(x_train[i]))\n",
        "print(x_train[i])\n",
        "print(len(x_train[i]))\n",
        "print(x_train_padded[i])"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "189\n",
            "[1, 194, 1153, 194, 8255, 78, 228, 5, 6, 1463, 4369, 5012, 134, 26, 4, 715, 8, 118, 1634, 14, 394, 20, 13, 119, 954, 189, 102, 5, 207, 110, 3103, 21, 14, 69, 188, 8, 30, 23, 7, 4, 249, 126, 93, 4, 114, 9, 2300, 1523, 5, 647, 4, 116, 9, 35, 8163, 4, 229, 9, 340, 1322, 4, 118, 9, 4, 130, 4901, 19, 4, 1002, 5, 89, 29, 952, 46, 37, 4, 455, 9, 45, 43, 38, 1543, 1905, 398, 4, 1649, 26, 6853, 5, 163, 11, 3215, 2, 4, 1153, 9, 194, 775, 7, 8255, 2, 349, 2637, 148, 605, 2, 8003, 15, 123, 125, 68, 2, 6853, 15, 349, 165, 4362, 98, 5, 4, 228, 9, 43, 2, 1157, 15, 299, 120, 5, 120, 174, 11, 220, 175, 136, 50, 9, 4373, 228, 8255, 5, 2, 656, 245, 2350, 5, 4, 9837, 131, 152, 491, 18, 2, 32, 7464, 1212, 14, 9, 6, 371, 78, 22, 625, 64, 1382, 9, 8, 168, 145, 23, 4, 1690, 15, 16, 4, 1355, 5, 28, 6, 52, 154, 462, 33, 89, 78, 285, 16, 145, 95]\n",
            "189\n",
            "[   0    0    0    0    0    0    0    0    0    0    0    1  194 1153\n",
            "  194 8255   78  228    5    6 1463 4369 5012  134   26    4  715    8\n",
            "  118 1634   14  394   20   13  119  954  189  102    5  207  110 3103\n",
            "   21   14   69  188    8   30   23    7    4  249  126   93    4  114\n",
            "    9 2300 1523    5  647    4  116    9   35 8163    4  229    9  340\n",
            " 1322    4  118    9    4  130 4901   19    4 1002    5   89   29  952\n",
            "   46   37    4  455    9   45   43   38 1543 1905  398    4 1649   26\n",
            " 6853    5  163   11 3215    2    4 1153    9  194  775    7 8255    2\n",
            "  349 2637  148  605    2 8003   15  123  125   68    2 6853   15  349\n",
            "  165 4362   98    5    4  228    9   43    2 1157   15  299  120    5\n",
            "  120  174   11  220  175  136   50    9 4373  228 8255    5    2  656\n",
            "  245 2350    5    4 9837  131  152  491   18    2   32 7464 1212   14\n",
            "    9    6  371   78   22  625   64 1382    9    8  168  145   23    4\n",
            " 1690   15   16    4 1355    5   28    6   52  154  462   33   89   78\n",
            "  285   16  145   95]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hfupbfuaqg0f"
      },
      "source": [
        "**BUILDING AND TRAINING THE MODEL**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oPqDpti_tjHd"
      },
      "source": [
        "Now our data is ready for some modelling!\n",
        "\n",
        "Deep learning models have layers.\n",
        "\n",
        "The top layer takes in the data we've just prepared, the middle layers do some math on this data and the final layer produces an output we can hopefully make use of.\n",
        "\n",
        "In our case, our model has three layers, \n",
        "\n",
        "1. Embedding layer\n",
        "2. LSTM layer\n",
        "3. Dense layer.\n",
        "\n",
        "Our model begins with the line model = Sequential(). Think of this as simply stating \"our model will flow from input to output layer in a sequential manner\" or \"our model goes one step at a time\".\n",
        "\n",
        "**Embedding layer**\n",
        "\n",
        "The Embedding layer creates a database of the relationships between words.\n",
        "\n",
        "model.add(Embedding(max_words, embedding_vector_length, input_length=max_review_length)) is saying: add an Embedding layer to our model and use it to turn each of our words into embedding_vector_length dimensional vector which have some mathematical relationship to each other.\n",
        "\n",
        "So each of our words will become vectors of dimension embedding_vector_length.\n",
        "\n",
        "For example, vector of \"the\" = [0.556433, 0.223122, 0.789654....].\n",
        "\n",
        "Don't worry for now how this is computed, Keras does it for us.\n",
        "\n",
        "**LSTM layer**\n",
        "\n",
        "model.add(LSTM(128)) is saying: add a LSTM layer after our embedding layer in our model and give it 128 units.\n",
        "\n",
        "**Dense layer**\n",
        "\n",
        "model.add(Dense(1, activation='sigmoid')) is saying: add a Dense layer to the end of our model and use a sigmoid activation function to produce a meaningful output.\n",
        "\n",
        "A dense layer is also known as a fully-connected layer. This layer connects the 128 LSTM units in the previous layer to 1 unit. This last unit them takes all this information and runs it through a sigmoid function.\n",
        "\n",
        "**Please feel free to change the model architecture.**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rwu39HI97hcm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "971effc3-77d8-489d-d6da-c945ae21d8e1"
      },
      "source": [
        "# Define how long the embedding vector will be\n",
        "embedding_vector_length = 128\n",
        "\n",
        "# Define the layers in the model\n",
        "model = Sequential()\n",
        "model.add(Embedding(vocab_size, embedding_vector_length, input_length=max_review_length))\n",
        "model.add(LSTM(128))\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "\n",
        "print(\"Model created.\")"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model created.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n6807KVQhXRz"
      },
      "source": [
        "**Compiling the model**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W-FzmE3mqXYc"
      },
      "source": [
        "Now we compile our model, which is nothing but configuring the model for training. We use the “adam” optimizer, an algorithm that changes the weights and biases during training. We also choose binary-crossentropy as loss (because we deal with binary classification) and accuracy as our evaluation metric."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ghfPqcms-S2V",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c9d01529-efb9-4c29-f707-5d48cd38e39f"
      },
      "source": [
        "model.compile(loss = 'binary_crossentropy', optimizer='adam',metrics = ['accuracy'])\n",
        "print(\"Model compiled, ready to be fit to the training data.\")"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model compiled, ready to be fit to the training data.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Hzb0SrkhyBM"
      },
      "source": [
        "**Summarize the model**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CYvRQw0uh4yr"
      },
      "source": [
        "Making a summary of the model will give us an idea of what's happening at each layer.\n",
        "\n",
        "In the embedding layer, each of our words is being turned into a vector of dimension 128. Because there are 10000 words (max_words), there are 1,280,000 parameters (128 x 10000).\n",
        "\n",
        "Parameters are individual pieces of information. The goal of the model is to take a large number of parameters and reduce them down to something we can understand and make use of (less parameters).\n",
        "\n",
        "The LSTM layer reduces the number of parameters to 131584 = 4 × [128(128+128) + 128].\n",
        "\n",
        "The final dense layer connects each of the outputs of the LSTM units into one cell (128 + 1)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5C36PeZgiKjI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "378ecb22-7786-4138-d89f-74eb6b7494bd"
      },
      "source": [
        "# Summarize the different layers in the model\n",
        "print(model.summary())"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding (Embedding)        (None, 200, 128)          1280000   \n",
            "_________________________________________________________________\n",
            "lstm (LSTM)                  (None, 128)               131584    \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 1)                 129       \n",
            "=================================================================\n",
            "Total params: 1,411,713\n",
            "Trainable params: 1,411,713\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8JKKMT6ZiR9Y"
      },
      "source": [
        "**Fitting the model to the training data**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nz3QpxeuiW2o"
      },
      "source": [
        "Now our model is compiled, it's ready to be set loose on our training data.\n",
        "\n",
        "We'll be training for 3 epochs with a batch_size of 64.\n",
        "\n",
        "Because of our loss and optimzation functions, the model accuracy should improve after each cycle.\n",
        "\n",
        "model.fit(X_train, y_train, epochs=3, batch_size=64) is saying: fit the model we've built on the training dataset for 3 cycles and go over 64 reviews at a time.\n",
        "\n",
        "I use test data as validation data. Use validation_split parameter in model.fit if you want to split training data into train and val.\n",
        "\n",
        "**Please Feel free to change the number of epochs or batch_size**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "STmd3VIPpAHk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0eb722ab-1101-4c02-e1ea-bc2ce9a80483"
      },
      "source": [
        "# Fit the model to the training data\n",
        "results = model.fit(x_train_padded, y_train, epochs=3, batch_size=64,validation_data=(x_test_padded, y_test))"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "391/391 [==============================] - 11s 20ms/step - loss: 0.4002 - accuracy: 0.8174 - val_loss: 0.3436 - val_accuracy: 0.8544\n",
            "Epoch 2/3\n",
            "391/391 [==============================] - 7s 18ms/step - loss: 0.2543 - accuracy: 0.9016 - val_loss: 0.3684 - val_accuracy: 0.8519\n",
            "Epoch 3/3\n",
            "391/391 [==============================] - 7s 19ms/step - loss: 0.1869 - accuracy: 0.9306 - val_loss: 0.3755 - val_accuracy: 0.8643\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cn2g3ZRoqGK_"
      },
      "source": [
        "It is time to evaluate our model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2bs_I_gWbIIh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "16c022de-6f72-4d4a-9f07-35f75d4e8d46"
      },
      "source": [
        "loss, acc = model.evaluate(x_test_padded, y_test,\n",
        "                            batch_size=64)\n",
        "print('Test loss:', loss)\n",
        "print('Test accuracy:', acc)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "391/391 [==============================] - 2s 6ms/step - loss: 0.3755 - accuracy: 0.8643\n",
            "Test loss: 0.37549883127212524\n",
            "Test accuracy: 0.8643199801445007\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sXZaFmY5dauk"
      },
      "source": [
        "Let's analyze the results by looking at some examples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aETOH47QY_tE"
      },
      "source": [
        "y_prob = model.predict(x_test_padded)"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JrvqtcqVZXsp",
        "outputId": "b9d74f6f-d611-4254-e212-712a6562cb22"
      },
      "source": [
        "y_prob # probabilty scores"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.03657438],\n",
              "       [0.9958324 ],\n",
              "       [0.3912676 ],\n",
              "       ...,\n",
              "       [0.01443178],\n",
              "       [0.40329775],\n",
              "       [0.98985064]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8r5GkH_umPrX"
      },
      "source": [
        "y_pred = np.round(y_prob)"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "awHoVtqOZ5gj",
        "outputId": "3ec63a18-3aa3-4efd-bbaf-9df8ccd6826d"
      },
      "source": [
        "y_pred.reshape([len(y_pred)]) # predicted labels"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0., 1., 0., ..., 0., 0., 1.], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fBY-Yv-KmXVD"
      },
      "source": [
        "y_pred = y_pred.reshape([len(y_pred)])"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BeiAaANaZjUX"
      },
      "source": [
        "import pandas as pd\n",
        "pd.set_option('display.max_colwidth', None) # for better printing"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iC0uqVH7ZmLx"
      },
      "source": [
        "results = pd.DataFrame({\"review\":test_text, \"ground_truth\":y_test, \"prediction\":y_pred})"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 765
        },
        "id": "Se8vLfYQaBdS",
        "outputId": "0b6d9ff4-8ca9-44dc-f7da-670ad3eb4a7c"
      },
      "source": [
        "results.head(5)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>prediction</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td># please give this one a miss br br # # and the rest of the cast rendered terrible performances the show is flat flat flat br br i don't know how michael madison could have allowed this one on his plate he almost seemed to know this wasn't going to work out and his performance was quite # so all you madison fans give this a miss</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td># this film requires a lot of patience because it focuses on mood and character development the plot is very simple and many of the scenes take place on the same set in frances # the sandy dennis character apartment but the film builds to a disturbing climax br br the characters create an atmosphere # with sexual tension and psychological # it's very interesting that robert altman directed this considering the style and structure of his other films still the trademark altman audio style is evident here and there i think what really makes this film work is the brilliant performance by sandy dennis it's definitely one of her darker characters but she plays it so perfectly and convincingly that it's scary michael burns does a good job as the mute young man regular altman player michael murphy has a small part the # moody set fits the content of the story very well in short this movie is a powerful study of loneliness sexual # and desperation be patient # up the atmosphere and pay attention to the wonderfully written script br br i praise robert altman this is one of his many films that deals with unconventional fascinating subject matter this film is disturbing but it's sincere and it's sure to # a strong emotional response from the viewer if you want to see an unusual film some might even say bizarre this is worth the time br br unfortunately it's very difficult to find in video stores you may have to buy it off the internet</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td># many animation buffs consider # # the great forgotten genius of one special branch of the art puppet animation which he invented almost single # and as it happened almost accidentally as a young man # was more interested in # than the cinema but his # attempt to film two # # fighting led to an unexpected breakthrough in film making when he realized he could # movement by # beetle # and # them one frame at a time this discovery led to the production of amazingly elaborate classic short the # revenge which he made in russia in # at a time when motion picture animation of all sorts was in its # br br the political # of the russian revolution caused # to move to paris where one of his first productions # was a dark political satire # known as # or the # who wanted a king a strain of black comedy can be found in almost all of films but here it is very dark indeed aimed more at grown ups who can appreciate the satirical aspects than children who would most likely find the climax # i'm middle aged and found it pretty # myself and indeed # of the film intended for english speaking viewers of the 1920s were given title cards filled with # and # in order to help # the sharp # of the finale br br our tale is set in a swamp the # # where the citizens are unhappy with their government and have called a special session to see what they can do to improve matters they decide to # # for a king the crowds are # animated in this opening sequence it couldn't have been easy to make so many frog puppets look alive simultaneously while # for his part is depicted as a # white # guy in the clouds who looks like he'd rather be taking a # when # sends them a tree like god who regards them the # decide that this is no improvement and demand a different king irritated # sends them a # br br delighted with this # looking new king who towers above them the # welcome him with a # of # dressed # the mayor steps forward to hand him the key to the # as # cameras record the event to everyone's horror the # promptly eats the mayor and then goes on a merry rampage # citizens at random a title card # reads news of the king's # throughout the kingdom when the now terrified # once more # # for help he loses his temper and # their community with lightning # the moral of our story delivered by a hapless frog just before he is eaten is let well enough alone br br considering the time period when this startling little film was made and considering the fact that it was made by a russian # at the height of that # country's civil war it would be easy to see this as a # about those events # may or may not have had # turmoil in mind when he made # but whatever # his choice of material the film stands as a # tale of universal # # could be the soviet union italy germany or japan in the 1930s or any country of any era that lets its guard down and is overwhelmed by # it's a fascinating film even a charming one in its macabre way but its message is no joke</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td># i generally love this type of movie however this time i found myself wanting to kick the screen since i can't do that i will just complain about it this was absolutely idiotic the things that happen with the dead kids are very cool but the alive people are absolute idiots i am a grown man pretty big and i can defend myself well however i would not do half the stuff the little girl does in this movie also the mother in this movie is reckless with her children to the point of neglect i wish i wasn't so angry about her and her actions because i would have otherwise enjoyed the flick what a number she was take my advise and fast forward through everything you see her do until the end also is anyone else getting sick of watching movies that are filmed so dark anymore one can hardly see what is being filmed as an audience we are # involved with the actions on the screen so then why the hell can't we have night vision</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td># like some other people wrote i'm a die hard mario fan and i loved this game br br this game starts slightly boring but trust me it's worth it as soon as you start your hooked the levels are fun and # they will hook you # your mind turns to # i'm not kidding this game is also # and is beautifully done br br to keep this spoiler free i have to keep my mouth shut about details but please try this game it'll be worth it br br story 9 9 action 10 1 it's that good # 10 attention # 10 average 10</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          review  ...  prediction\n",
              "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     # please give this one a miss br br # # and the rest of the cast rendered terrible performances the show is flat flat flat br br i don't know how michael madison could have allowed this one on his plate he almost seemed to know this wasn't going to work out and his performance was quite # so all you madison fans give this a miss  ...         0.0\n",
              "1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         # this film requires a lot of patience because it focuses on mood and character development the plot is very simple and many of the scenes take place on the same set in frances # the sandy dennis character apartment but the film builds to a disturbing climax br br the characters create an atmosphere # with sexual tension and psychological # it's very interesting that robert altman directed this considering the style and structure of his other films still the trademark altman audio style is evident here and there i think what really makes this film work is the brilliant performance by sandy dennis it's definitely one of her darker characters but she plays it so perfectly and convincingly that it's scary michael burns does a good job as the mute young man regular altman player michael murphy has a small part the # moody set fits the content of the story very well in short this movie is a powerful study of loneliness sexual # and desperation be patient # up the atmosphere and pay attention to the wonderfully written script br br i praise robert altman this is one of his many films that deals with unconventional fascinating subject matter this film is disturbing but it's sincere and it's sure to # a strong emotional response from the viewer if you want to see an unusual film some might even say bizarre this is worth the time br br unfortunately it's very difficult to find in video stores you may have to buy it off the internet  ...         1.0\n",
              "2  # many animation buffs consider # # the great forgotten genius of one special branch of the art puppet animation which he invented almost single # and as it happened almost accidentally as a young man # was more interested in # than the cinema but his # attempt to film two # # fighting led to an unexpected breakthrough in film making when he realized he could # movement by # beetle # and # them one frame at a time this discovery led to the production of amazingly elaborate classic short the # revenge which he made in russia in # at a time when motion picture animation of all sorts was in its # br br the political # of the russian revolution caused # to move to paris where one of his first productions # was a dark political satire # known as # or the # who wanted a king a strain of black comedy can be found in almost all of films but here it is very dark indeed aimed more at grown ups who can appreciate the satirical aspects than children who would most likely find the climax # i'm middle aged and found it pretty # myself and indeed # of the film intended for english speaking viewers of the 1920s were given title cards filled with # and # in order to help # the sharp # of the finale br br our tale is set in a swamp the # # where the citizens are unhappy with their government and have called a special session to see what they can do to improve matters they decide to # # for a king the crowds are # animated in this opening sequence it couldn't have been easy to make so many frog puppets look alive simultaneously while # for his part is depicted as a # white # guy in the clouds who looks like he'd rather be taking a # when # sends them a tree like god who regards them the # decide that this is no improvement and demand a different king irritated # sends them a # br br delighted with this # looking new king who towers above them the # welcome him with a # of # dressed # the mayor steps forward to hand him the key to the # as # cameras record the event to everyone's horror the # promptly eats the mayor and then goes on a merry rampage # citizens at random a title card # reads news of the king's # throughout the kingdom when the now terrified # once more # # for help he loses his temper and # their community with lightning # the moral of our story delivered by a hapless frog just before he is eaten is let well enough alone br br considering the time period when this startling little film was made and considering the fact that it was made by a russian # at the height of that # country's civil war it would be easy to see this as a # about those events # may or may not have had # turmoil in mind when he made # but whatever # his choice of material the film stands as a # tale of universal # # could be the soviet union italy germany or japan in the 1930s or any country of any era that lets its guard down and is overwhelmed by # it's a fascinating film even a charming one in its macabre way but its message is no joke  ...         0.0\n",
              "3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 # i generally love this type of movie however this time i found myself wanting to kick the screen since i can't do that i will just complain about it this was absolutely idiotic the things that happen with the dead kids are very cool but the alive people are absolute idiots i am a grown man pretty big and i can defend myself well however i would not do half the stuff the little girl does in this movie also the mother in this movie is reckless with her children to the point of neglect i wish i wasn't so angry about her and her actions because i would have otherwise enjoyed the flick what a number she was take my advise and fast forward through everything you see her do until the end also is anyone else getting sick of watching movies that are filmed so dark anymore one can hardly see what is being filmed as an audience we are # involved with the actions on the screen so then why the hell can't we have night vision  ...         0.0\n",
              "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                # like some other people wrote i'm a die hard mario fan and i loved this game br br this game starts slightly boring but trust me it's worth it as soon as you start your hooked the levels are fun and # they will hook you # your mind turns to # i'm not kidding this game is also # and is beautifully done br br to keep this spoiler free i have to keep my mouth shut about details but please try this game it'll be worth it br br story 9 9 action 10 1 it's that good # 10 attention # 10 average 10  ...         1.0\n",
              "\n",
              "[5 rows x 3 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DQsFZZDCad2T"
      },
      "source": [
        "## some samples of wrong predictions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 731
        },
        "id": "G1tBg9mkaFp7",
        "outputId": "e4fbddaf-801a-4d2f-bd0f-32d3589e9fc2"
      },
      "source": [
        "results[results['ground_truth'] != results['prediction']].sample(5)"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>prediction</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>11788</th>\n",
              "      <td># once i heard that the greatest and oldest # # heroic poem was transformed into a film it almost became my obsession to see it the first # of its appearance i caught never disappointed me a futuristic interpretation with # our favourite # and tomb # to be in leading roles # appealing though some doubts came to life an important female character in beowulf two hours ago i saw the film after i had read the director's name my world fell apart as i said from that point on there was not many surprises first and foremost the film has nothing to do with the original beowulf if we disregard a couple of violently and # stolen names if they had not stolen the names and # it to be a new story it might have passed as an f class action stupidity with nice costumes and # this way it is simply a crime an attack on a legend and its ideology as well as on common sense ok let me be positive for a second apart from the general # # atmosphere which is nice it also has good music that was it for both the positive part and this comment</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18008</th>\n",
              "      <td># i saw this at the screening at # in # i had some time to kill and decided to check it out it played to about 1000 people in a packed standing room only ballroom br br wow what a ride the script was tight the action tense the pacing perfect the character exposition excellent one thing i really appreciated was that you knew going in that this wasn't a big budget film yet it soon became obvious that the creators pushed their sets and effects as far as they could despite their limitations and it was more than enough br br it's true that this film was targeted at a certain audience # # players the creators make no effort to hide that but other filmmakers could learn a lot from them for in going for the # in scene after scene and not worrying about if mom who happens to be watching will get it they got the biggest laughs time and time again but there's enough # there that mom will be laughing too even if she's not in on every joke i think too many times i see films that try so hard to lower the bar to the lowest common # so that they will appeal to the most people but the movie just ends up suffering for it br br but not this flick indeed this film was so solid that it had the audience wrapped around it's finger from the opening credits and while the viewers around me really wanted to like the film they weren't # # can be among the most critical # out there br br i'm so glad i got to see this in a big crowd at least 10 times the audience was having such a good time that they # into applause at a joke or scene during the film how often does that happen at # it should be no surprise that there was a huge standing # when the closing credits rolled br br for my own part i can't wait for this to be released after it ended one of the producers said they were shooting for a # tv dvd release that date cannot come soon enough</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3101</th>\n",
              "      <td># yeah i guess this movie is kinda dull compared to some of pam # other films the plot is overly familiar the dialog stilted and some of the acting isn't too good but it's worth seeing for the lengthy stretch near the end of the film where we see ms # in a sexy blue with the # half yeah it seems like a # point when discussing an actress of pam # talent but she also happens to be an extremely gorgeous woman and back in the day she had a body that wouldn't quit it's nice to see it being # in a tight rent the dvd and then tell me i'm wrong can't can you that's because you know i'm right and yes i really did give a 10 just for the scenes</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22356</th>\n",
              "      <td># this film was shot in randolph county in central north # in 1968 when a film crew in the state was a rare thing the locations were the of liberty and and the surrounding rural countryside it is not a particularly good movie it did have # # and it brought life to the # for a few minutes br br the plot is standard the cinematography is that fuzzy stuff that came out of the late sixties and early seventies the local folks were thrilled to be a part of the enterprise br br if viewers have difficulty finding a copy of this film a record copy is available in # br br actors not credited include ben jones # # tommy # bill #</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19745</th>\n",
              "      <td># this soap is worse than bad it's # of the many television shows that have had a # influence on british society over the past twenty years # is the prime example for two decades this show has celebrated the # the thug the wide boy the # the the violent the sexually the criminal the ignorant the br br how many times has someone or other # that # mirrors life life on which planet exactly br br it's written about working class characters as imagined by middle class people who have taken a course in creative writing eager to show to their middle class peers how familiar they are with the working class they dream up the # # that is the # of # br br this has a toxic effect on some minds less well # than others to handle fiction and so we find members of the real population assuming the attitudes and # of the inhabitants of br br thus it came to pass that # mirrors life but only after life had been # into # # br br other # have followed in footsteps filled to their # with ugly # faced # headed pot # characters # at each other and # # constantly this is the # as perceived by the writers who produce this trash the writers will grow rich on the proceeds of such # and will go on to enjoy the # things of life in their # meanwhile the # number of new tv induced # will proceed # toward cultural # br br and there you have the new priests and the new creatures of the early 21st century much of this is due to the # power of that # of dancing in the corner of your living room it's your fault gentle reader that's what you chose as the only window through which to look out from your prison</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     review  ...  prediction\n",
              "11788                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                # once i heard that the greatest and oldest # # heroic poem was transformed into a film it almost became my obsession to see it the first # of its appearance i caught never disappointed me a futuristic interpretation with # our favourite # and tomb # to be in leading roles # appealing though some doubts came to life an important female character in beowulf two hours ago i saw the film after i had read the director's name my world fell apart as i said from that point on there was not many surprises first and foremost the film has nothing to do with the original beowulf if we disregard a couple of violently and # stolen names if they had not stolen the names and # it to be a new story it might have passed as an f class action stupidity with nice costumes and # this way it is simply a crime an attack on a legend and its ideology as well as on common sense ok let me be positive for a second apart from the general # # atmosphere which is nice it also has good music that was it for both the positive part and this comment  ...         1.0\n",
              "18008  # i saw this at the screening at # in # i had some time to kill and decided to check it out it played to about 1000 people in a packed standing room only ballroom br br wow what a ride the script was tight the action tense the pacing perfect the character exposition excellent one thing i really appreciated was that you knew going in that this wasn't a big budget film yet it soon became obvious that the creators pushed their sets and effects as far as they could despite their limitations and it was more than enough br br it's true that this film was targeted at a certain audience # # players the creators make no effort to hide that but other filmmakers could learn a lot from them for in going for the # in scene after scene and not worrying about if mom who happens to be watching will get it they got the biggest laughs time and time again but there's enough # there that mom will be laughing too even if she's not in on every joke i think too many times i see films that try so hard to lower the bar to the lowest common # so that they will appeal to the most people but the movie just ends up suffering for it br br but not this flick indeed this film was so solid that it had the audience wrapped around it's finger from the opening credits and while the viewers around me really wanted to like the film they weren't # # can be among the most critical # out there br br i'm so glad i got to see this in a big crowd at least 10 times the audience was having such a good time that they # into applause at a joke or scene during the film how often does that happen at # it should be no surprise that there was a huge standing # when the closing credits rolled br br for my own part i can't wait for this to be released after it ended one of the producers said they were shooting for a # tv dvd release that date cannot come soon enough  ...         0.0\n",
              "3101                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      # yeah i guess this movie is kinda dull compared to some of pam # other films the plot is overly familiar the dialog stilted and some of the acting isn't too good but it's worth seeing for the lengthy stretch near the end of the film where we see ms # in a sexy blue with the # half yeah it seems like a # point when discussing an actress of pam # talent but she also happens to be an extremely gorgeous woman and back in the day she had a body that wouldn't quit it's nice to see it being # in a tight rent the dvd and then tell me i'm wrong can't can you that's because you know i'm right and yes i really did give a 10 just for the scenes  ...         0.0\n",
              "22356                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     # this film was shot in randolph county in central north # in 1968 when a film crew in the state was a rare thing the locations were the of liberty and and the surrounding rural countryside it is not a particularly good movie it did have # # and it brought life to the # for a few minutes br br the plot is standard the cinematography is that fuzzy stuff that came out of the late sixties and early seventies the local folks were thrilled to be a part of the enterprise br br if viewers have difficulty finding a copy of this film a record copy is available in # br br actors not credited include ben jones # # tommy # bill #  ...         1.0\n",
              "19745                                                                                                                                                                                                                                                         # this soap is worse than bad it's # of the many television shows that have had a # influence on british society over the past twenty years # is the prime example for two decades this show has celebrated the # the thug the wide boy the # the the violent the sexually the criminal the ignorant the br br how many times has someone or other # that # mirrors life life on which planet exactly br br it's written about working class characters as imagined by middle class people who have taken a course in creative writing eager to show to their middle class peers how familiar they are with the working class they dream up the # # that is the # of # br br this has a toxic effect on some minds less well # than others to handle fiction and so we find members of the real population assuming the attitudes and # of the inhabitants of br br thus it came to pass that # mirrors life but only after life had been # into # # br br other # have followed in footsteps filled to their # with ugly # faced # headed pot # characters # at each other and # # constantly this is the # as perceived by the writers who produce this trash the writers will grow rich on the proceeds of such # and will go on to enjoy the # things of life in their # meanwhile the # number of new tv induced # will proceed # toward cultural # br br and there you have the new priests and the new creatures of the early 21st century much of this is due to the # power of that # of dancing in the corner of your living room it's your fault gentle reader that's what you chose as the only window through which to look out from your prison  ...         1.0\n",
              "\n",
              "[5 rows x 3 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gGoFsTbhcdTI"
      },
      "source": [
        "## some samples of correct predictions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 527
        },
        "id": "ihLVvshlaFs2",
        "outputId": "3947ec41-138d-4c9d-a8b3-89cf334cb889"
      },
      "source": [
        "results[results['ground_truth'] == results['prediction']].sample(5)"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>prediction</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>4006</th>\n",
              "      <td># i have never read sarah # book although i have not read the book the 3 hour movie is very interesting it begins with an interesting storyline with a twisted ending i have to say these 2 actresses are amazing sally # is stunning successfully portrayed the character in love with her mistress and betrayed by her love their romance slowly # as they spend more and more time together the love making scene is very tender and emotional well acted the end is quite intriguing and these 2 ended up together after all they have been thru which is a bless overall it is a great movie to see a very interesting plot with excellent performances</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6226</th>\n",
              "      <td># crazy six is torture it must be albert worst film even blast and # are better i # believe how boring this film is how this even got # i saw this movie about 3 years ago and the only thing i remember is how bad it was this # good bad movie it is simply bad bad bad bad bad movie br br 1 out of 10 # out of</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12831</th>\n",
              "      <td># this film for what it was set out to be succeeded it's a short tragic film although my choice of film are ones that really develop characters and their relationships this film is meant to just give a taste leaving you with the what happens next factor after watching it i really was wanting more more of the characters back story what influences they had to make them into the people they were i think thats what the makers intended the viewing audience to think the acting is amazing there aren't many lines in the film so their body language facial expressions and overall presence needed to be powerful enough to # a scene both franco and # have that element and it shows for them especially franco to take the time to make this obviously says they believed in this film and wanted to be apart of it and for that i appreciated the film for what it was also i'm happy i own it so i can share it with other people that would've never known it existed</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9483</th>\n",
              "      <td># for real though this game is where it's at i'm 20 years old and that's basically where it started for me 4 bit graphics was fabulous i hope you all remember this game with as much # as i do that # is a real</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21712</th>\n",
              "      <td># a rich old lady calls on a # # to woo a # away from her silly soon to be married # br br let us be gay is an interesting little domestic comedy which features some # dialogue courtesy of celebrated screenwriter frances marion good performances while perhaps a bit # at times this can probably be blamed on the difficulties with early sound technology which tended to limit action movement br br norma # can be credited with appearing in this minor film rather than using her # # as irving # # to insist upon only a grade pictures she is especially effective in her first few scenes where # flat makeup makes her almost # her extreme from # to # could only happen in hollywood but it's # # to spend much time worrying about that br br rod # doesn't come off too well as # # husband quite popular during silent days the # were not especially kind to him and his career would suffer here his role is not in the least sympathetic and one has to wonder what # # moves women to desire the # so much br br magnificent marie # is on hand as an eccentric long island # as a great friend of frances marion one can easily imagine that the part was written # for her full of # she is very humorous however the tremendous warmth essential goodness which would very shortly make her hollywood's biggest star are largely missing br br among the supporting cast hopper scores as a # society # as does # playing a comic butler movie # will spot little # moore as # young son elderly mary gordon as her # both uncredited</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 review  ...  prediction\n",
              "4006                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       # i have never read sarah # book although i have not read the book the 3 hour movie is very interesting it begins with an interesting storyline with a twisted ending i have to say these 2 actresses are amazing sally # is stunning successfully portrayed the character in love with her mistress and betrayed by her love their romance slowly # as they spend more and more time together the love making scene is very tender and emotional well acted the end is quite intriguing and these 2 ended up together after all they have been thru which is a bless overall it is a great movie to see a very interesting plot with excellent performances  ...         1.0\n",
              "6226                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 # crazy six is torture it must be albert worst film even blast and # are better i # believe how boring this film is how this even got # i saw this movie about 3 years ago and the only thing i remember is how bad it was this # good bad movie it is simply bad bad bad bad bad movie br br 1 out of 10 # out of  ...         0.0\n",
              "12831                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         # this film for what it was set out to be succeeded it's a short tragic film although my choice of film are ones that really develop characters and their relationships this film is meant to just give a taste leaving you with the what happens next factor after watching it i really was wanting more more of the characters back story what influences they had to make them into the people they were i think thats what the makers intended the viewing audience to think the acting is amazing there aren't many lines in the film so their body language facial expressions and overall presence needed to be powerful enough to # a scene both franco and # have that element and it shows for them especially franco to take the time to make this obviously says they believed in this film and wanted to be apart of it and for that i appreciated the film for what it was also i'm happy i own it so i can share it with other people that would've never known it existed  ...         1.0\n",
              "9483                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   # for real though this game is where it's at i'm 20 years old and that's basically where it started for me 4 bit graphics was fabulous i hope you all remember this game with as much # as i do that # is a real  ...         1.0\n",
              "21712  # a rich old lady calls on a # # to woo a # away from her silly soon to be married # br br let us be gay is an interesting little domestic comedy which features some # dialogue courtesy of celebrated screenwriter frances marion good performances while perhaps a bit # at times this can probably be blamed on the difficulties with early sound technology which tended to limit action movement br br norma # can be credited with appearing in this minor film rather than using her # # as irving # # to insist upon only a grade pictures she is especially effective in her first few scenes where # flat makeup makes her almost # her extreme from # to # could only happen in hollywood but it's # # to spend much time worrying about that br br rod # doesn't come off too well as # # husband quite popular during silent days the # were not especially kind to him and his career would suffer here his role is not in the least sympathetic and one has to wonder what # # moves women to desire the # so much br br magnificent marie # is on hand as an eccentric long island # as a great friend of frances marion one can easily imagine that the part was written # for her full of # she is very humorous however the tremendous warmth essential goodness which would very shortly make her hollywood's biggest star are largely missing br br among the supporting cast hopper scores as a # society # as does # playing a comic butler movie # will spot little # moore as # young son elderly mary gordon as her # both uncredited  ...         1.0\n",
              "\n",
              "[5 rows x 3 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XRpQSZ51f3gp"
      },
      "source": [
        "## Below we have some more sample architectures you can try !!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DgPdDWvfVWW_"
      },
      "source": [
        "**Extensions**\n",
        "\n",
        "Let us use LSTM variants. We use check the accuracy by replacing LSTM cell with GRU cell."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jNe2GAad7_rr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a2315f4f-481d-466d-8748-80d5e4ec5e69"
      },
      "source": [
        "# Define the layers in the model\n",
        "model = Sequential()\n",
        "model.add(Embedding(vocab_size, embedding_vector_length, input_length=max_review_length))\n",
        "#model.add(LSTM(128))\n",
        "model.add(GRU(32,dropout=0.2, recurrent_dropout=0.2))\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "print(model.summary())\n",
        "\n",
        "#num_params_layer 3 × [h(h+i) + h]  = 3 × [32(32+64) + 32] = 9312"
      ],
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Layer gru_1 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
            "Model: \"sequential_6\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_6 (Embedding)      (None, 200, 128)          1280000   \n",
            "_________________________________________________________________\n",
            "gru_1 (GRU)                  (None, 32)                15552     \n",
            "_________________________________________________________________\n",
            "dense_6 (Dense)              (None, 1)                 33        \n",
            "=================================================================\n",
            "Total params: 1,295,585\n",
            "Trainable params: 1,295,585\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "316Ha7ug8EEu"
      },
      "source": [
        "model.compile(loss='binary_crossentropy', \n",
        "             optimizer='adam', \n",
        "             metrics=['accuracy'])"
      ],
      "execution_count": 57,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WMbarkxdWFHM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1afc1e6e-e479-4d8f-c8df-fb7c387e8913"
      },
      "source": [
        "# Fit the model to the training data\n",
        "results = model.fit(x_train_padded, y_train, epochs=3, batch_size=64,validation_data=(x_test_padded, y_test))"
      ],
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "391/391 [==============================] - 264s 670ms/step - loss: 0.4317 - accuracy: 0.7960 - val_loss: 0.3289 - val_accuracy: 0.8606\n",
            "Epoch 2/3\n",
            "391/391 [==============================] - 260s 664ms/step - loss: 0.2581 - accuracy: 0.8981 - val_loss: 0.3092 - val_accuracy: 0.8707\n",
            "Epoch 3/3\n",
            "391/391 [==============================] - 259s 661ms/step - loss: 0.1967 - accuracy: 0.9257 - val_loss: 0.3427 - val_accuracy: 0.8548\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ja11LMYJ8cPx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "597bdbf3-6f9f-41e1-ae0a-043cc5a0fb09"
      },
      "source": [
        "loss, acc = model.evaluate(x_test_padded, y_test,\n",
        "                            batch_size=64)\n",
        "print('Test loss:', loss)\n",
        "print('Test accuracy:', acc)"
      ],
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "391/391 [==============================] - 18s 45ms/step - loss: 0.3427 - accuracy: 0.8548\n",
            "Test loss: 0.34270304441452026\n",
            "Test accuracy: 0.8547999858856201\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vzzYeeiAWPXh"
      },
      "source": [
        "**Using LSTM stack layers**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RIOwL8VH8o_B",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f591b24d-3314-493c-dfe4-fd64f0016a7a"
      },
      "source": [
        "model= Sequential()\n",
        "model.add(Embedding(vocab_size, embedding_vector_length, input_length=max_review_length))\n",
        "model.add(LSTM(units=64, return_sequences=True))\n",
        "model.add(LSTM(units=64, return_sequences=True))\n",
        "model.add(LSTM(units=4))\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "print(model.summary())"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_4\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_4 (Embedding)      (None, 200, 128)          1280000   \n",
            "_________________________________________________________________\n",
            "lstm_5 (LSTM)                (None, 200, 64)           49408     \n",
            "_________________________________________________________________\n",
            "lstm_6 (LSTM)                (None, 200, 64)           33024     \n",
            "_________________________________________________________________\n",
            "lstm_7 (LSTM)                (None, 4)                 1104      \n",
            "_________________________________________________________________\n",
            "dense_4 (Dense)              (None, 1)                 5         \n",
            "=================================================================\n",
            "Total params: 1,363,541\n",
            "Trainable params: 1,363,541\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VtMPv6pR8sBW"
      },
      "source": [
        "model.compile(loss='binary_crossentropy', \n",
        "             optimizer='adam', \n",
        "             metrics=['accuracy'])"
      ],
      "execution_count": 47,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h0_eb8nT8s0b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e63aabee-92e8-4fdf-cf95-2197a027a34f"
      },
      "source": [
        "# Fit the model to the training data\n",
        "results = model.fit(x_train_padded, y_train, epochs=3, batch_size=64,validation_data=(x_test_padded, y_test))"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "391/391 [==============================] - 77s 35ms/step - loss: 0.6605 - accuracy: 0.6037 - val_loss: 0.6885 - val_accuracy: 0.5296\n",
            "Epoch 2/3\n",
            "391/391 [==============================] - 13s 33ms/step - loss: 0.6602 - accuracy: 0.5954 - val_loss: 0.5763 - val_accuracy: 0.7264\n",
            "Epoch 3/3\n",
            "391/391 [==============================] - 13s 33ms/step - loss: 0.4755 - accuracy: 0.8030 - val_loss: 0.4701 - val_accuracy: 0.7820\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7CrZgxje8w2m",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fb590ccd-224b-42d5-920e-7bc9040c3e71"
      },
      "source": [
        "loss, acc = model.evaluate(x_test_padded, y_test, batch_size=64)\n",
        "print('Test loss:', loss)\n",
        "print('Test accuracy:', acc)"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "391/391 [==============================] - 4s 11ms/step - loss: 0.4701 - accuracy: 0.7820\n",
            "Test loss: 0.4700876474380493\n",
            "Test accuracy: 0.7819600105285645\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PCE3OEZNzQzA"
      },
      "source": [
        "**Using Simple RNN**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qklhcSWay9f-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aae22981-c82d-4006-a037-0fe63397ca73"
      },
      "source": [
        "# Define the layers in the model\n",
        "model = Sequential()\n",
        "model.add(Embedding(vocab_size, embedding_vector_length, input_length=max_review_length))\n",
        "#model.add(LSTM(128))\n",
        "model.add(SimpleRNN(128,dropout=0.2, recurrent_dropout=0.2))\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "print(model.summary())"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_5\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_5 (Embedding)      (None, 200, 128)          1280000   \n",
            "_________________________________________________________________\n",
            "simple_rnn (SimpleRNN)       (None, 128)               32896     \n",
            "_________________________________________________________________\n",
            "dense_5 (Dense)              (None, 1)                 129       \n",
            "=================================================================\n",
            "Total params: 1,313,025\n",
            "Trainable params: 1,313,025\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kB-Msn79zZgi"
      },
      "source": [
        "model.compile(loss='binary_crossentropy', \n",
        "             optimizer='adam', \n",
        "             metrics=['accuracy'])"
      ],
      "execution_count": 53,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rfYP4xEuzeK7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9a8af872-3d38-4656-acdc-46a844d63a85"
      },
      "source": [
        "# Fit the model to the training data\n",
        "results = model.fit(x_train_padded, y_train, epochs=3, batch_size=64, validation_data=(x_test_padded, y_test))"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "391/391 [==============================] - 102s 259ms/step - loss: 0.6929 - accuracy: 0.5381 - val_loss: 0.6663 - val_accuracy: 0.5965\n",
            "Epoch 2/3\n",
            "391/391 [==============================] - 102s 260ms/step - loss: 0.6662 - accuracy: 0.5806 - val_loss: 0.6510 - val_accuracy: 0.6064\n",
            "Epoch 3/3\n",
            "391/391 [==============================] - 102s 260ms/step - loss: 0.6130 - accuracy: 0.6492 - val_loss: 0.4927 - val_accuracy: 0.7705\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RVKPgiTfzfgr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a9a828b4-d850-45d5-98c1-29fe9090f6bb"
      },
      "source": [
        "loss, acc = model.evaluate(x_test_padded, y_test,\n",
        "                            batch_size=64)\n",
        "print('Test loss:', loss)\n",
        "print('Test accuracy:', acc)"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "391/391 [==============================] - 6s 15ms/step - loss: 0.4927 - accuracy: 0.7705\n",
            "Test loss: 0.492692232131958\n",
            "Test accuracy: 0.7704799771308899\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}