03_Recommender_Introduction/01_content_based.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To open this notebook in Google Colab and start coding, click on the Colab icon below.\n",
    "\n",
    "<table style=\"border:2px solid orange\" align=\"left\">\n",
    "    <td style=\"border:2px solid orange\">\n",
    "        <a target=\"_blank\" href=\"https://colab.research.google.com/github/neuefische/ds-meetups/blob/main/03_Recommender_Introduction/01_content_based.ipynb\">\n",
    "        <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "    </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# content-based recommender system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recommender systems are algorithms which suggest information of interest to the user. If you ever used the internet, you will surely have made contact with a recommender system. Netflix, Amazon, Spotify, all make use of such functions to improve user satisfaction as well as buying behavior. Based on your previous activity the recommender system will suggest you the next video, product, song or whatever. The most common recommender types are content-based, collaborative filtering and hybrid models of those two types.\n",
    "\n",
    "Having enough user data do build a proper collaborative model is usually an obstacle at the beginning, as there is little known about the users behavior. This is also referred to as the cold start problem in recommender systems. In this notebook you will learn how to build a content-based recommender-system. A content-based model has the advantage that no user data is required and the model can be scaled to a large number of users independently of their preferences. Solving the cold start problem. For a content-based recommender system the filtering is based on the item features only.\n",
    "<br/><br/>\n",
    "<p align=\"center\">\n",
    "    <img src=\"images/book_text.png\" alt=\"drawing\" width=\"400\"/>\n",
    "</p>\n",
    "<br/><br/>\n",
    "An example: If you have a book that has the features \"aquarium\" and \"fish\" you would get recommendations with the same features, so books about aquarium fish without any usage of ratings or user data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with building your own content-based recommender system!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#requirements\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics.pairwise import linear_kernel\n",
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cosine similarity \n",
    "\n",
    "To get meaningful recommendations, it is important to measure the similarity of two items. To do so, cosine similarity is the most popular metric for content-based models. Mathematically, the cosine similarity is the dot product of normalized vectors and measures the cosine of the angle between two vectors projected in a n-dimensional space (n = number of features). Output values are limited between 0 and 1, with a value of 0 indicating no similarity, whereas 1 means that both the items are identical. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you got a two-dimensional space and the angle of two data points is 45° the cosine would be 0.7 while it would be zero for an angle of 90°. In a data frame each feature represents a dimension. This means, every observation (row) could be plotted in a n-dimensional space based on it's feature-properties. Let's have a look at two examples:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: 2-dimensional space "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's assume, we got two fish with different colors. One fish is solid blue, while the other is solid orange. Plotted in a 2-dimensional space, where one axis represents the color blue and the other the color orange, one fish would have the coordinates (1,0) and the other (0,1). A fish with both colors would be in between (1,1). To calculate the cosine similarity score between two fish, we simply measure the cosine of the two coordinate vectors. \n",
    "\n",
    "<br/><br/>\n",
    "<p align=\"center\">\n",
    "    <img src=\"images/fish_text.png\" alt=\"drawing\" width=\"300\"/>\n",
    "</p>\n",
    "<br/><br/>\n",
    "The vectors of the blue and orange fish and the solid orange fish would have a 45° angle, therefore a cosine similarity of 0.7. Just the same if we compare the solid blue and the blue-orange fish. The solid blue and the solid orange fish would have an angle of 90° and hence a cosine of 0, so no similarity at all with respect to the feature color. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: 4-dimensional space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create an example data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create example data frame\n",
    "d = {'green': [1, 1, 0, 1, 1], 'blue': [1, 0, 0, 1, 1], 'yellow': [0, 0, 1, 1, 0], 'black': [1, 0, 0, 1, 1]}\n",
    "df_example2 = pd.DataFrame(data = d, index= [\"fish1\", \"fish2\", \"fish3\", \"fish4\", \"fish5\"])\n",
    "df_example2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this example let's think of 5 different fish. Each fish has a different combination of the colors green, blue, yellow and black. Those colors are the features of the fish and likewise the dimensionality of the data. Each feature is represented by a value of 0 or 1, while 0 means the fish doesn't have this color and 1 it has. In our example fish1 has the colors green, blue and yellow. If we assign colors to dimensions we get the coordinates (1,1,0,1). The following equation allows us to calculate the cosine similarity for multi-dimensional data: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "  CosSim(x,y) = \\frac{\\sum_{i}x_{i}y_{i}}{\\sqrt{\\sum_{i}x_{i}^2}\\sqrt{\\sum_{i}y_{i}^2}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we calculate the cosine similarity on our data frame, we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#calculate cosine similarity\n",
    "cosine_sim_sk = pd.DataFrame(cosine_similarity(df_example2).round(2), columns=[\"fish1\", \"fish2\", \"fish3\", \"fish4\", \"fish5\"], index= [\"fish1\", \"fish2\", \"fish3\", \"fish4\", \"fish5\"])\n",
    "cosine_sim_sk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How to interpret this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see, that fish1 is of course totally similar to itself, indicated by a value of 1. Then fish1 is less similar to fish 2 (only one overlap) as to fish 4 (three overlaps) and has no similarity to fish3 while it is totally similar to fish5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now, let's start to build our first recommender system :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading in the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#read data\n",
    "df = pd.read_csv(\"https://raw.githubusercontent.com/neuefische/ds-meetups/main/03_Recommender_Introduction/fish_1.csv\")\n",
    "df.reset_index(drop= True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To calculate the cosine similarity of the fish in our data frame, we need to transform the features into binary vectors. To do this we use the get_dummies function implemented in scikit-learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create dummy vectors\n",
    "df_vectorized = pd.get_dummies(df.iloc[:, 1:])\n",
    "df_vectorized.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that this increases the numbers of features from 24 to 114. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's calculate the cosine similarity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#calculate cosine similarity\n",
    "cosine_sim_sk = pd.DataFrame(cosine_similarity(df_vectorized))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally the function to get recommendations!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the cosine similarities, we can simply sort the similarities with respect to a specific item. This function takes in the name of a fish, checks the similarity scores compared to other fish, sorts the values and gives out the 10 most similar fish based on similarity score. Just type in a name of a fish in the data frame and you will get recommended the 10 most similar fish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build index with fish names\n",
    "names = df['name']\n",
    "indices = pd.Series(df.index, index=df['name'])\n",
    "\n",
    "# Function that get fish recommendations based on the cosine similarity \n",
    "def fish_recommendations(name):\n",
    "    #get the index of the fish we put into the function\n",
    "    idx = indices[name]\n",
    "    #calculate all cosine similarities to that fish and store it in a list\n",
    "    sim_scores = list(enumerate(cosine_sim_sk[idx]))\n",
    "    #sort the list staring with the highest similarity\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    #get the similarities from 1:11 (not starting with 0 because it is the same fish)\n",
    "    sim_scores = sim_scores[1:11]\n",
    "    #get the indeces of that 10 fish\n",
    "    fish_indices = [i[0] for i in sim_scores]\n",
    "    #return the fish names of that 10 fish\n",
    "    return names.iloc[fish_indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, use this function to get recommendations :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get recommendations\n",
    "fish_recommendations('Blue Neon - Paracheirodon simulans')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text related content-based recommender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you think of online warehouses or movie databases, many items got descriptions in text form. Content-based recommender systems are often build on these descriptions as they contain useful information about the article. In the following you will find an example of how to build a text based recommender system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we have to create a text block, we can work with. Let's join our features to text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#set column types to string for whole data frame\n",
    "df = df.astype(str)\n",
    "columns = df.columns\n",
    "#merge features to text block\n",
    "df['text'] = df[columns].agg(' '.join, axis=1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#check first element\n",
    "df.text[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text vectorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Different tools are available to vectorize texts. Dependent on your problem, you can use NLP methods like CountVectorizer, TfidfVectorizor or words2vec. In this notebook we will use the TfidfVectorizer. Documentation can be found [here](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html). TF-IDF or Term Frequency Inverse Document Frequency is a statistical measure to determine the significance of different words in a document. Subsequently every word gets a TF-IDF value. This value is the product of the Term frequency (TF) and the inverse document frequency (IDF).</br></br>\n",
    "\n",
    "Let's have a look at an example with 3 sentences where each sentence represents a document:\n",
    "\n",
    "1. Recommender are cool\n",
    "2. Recommender are helpful\n",
    "3. Recommender use your data\n",
    "\n",
    "In these 3 sentences the term \"recommender\" occurs 3 times, the therm \"are\" 2 times and \"use\" 1 time.</br></br>\n",
    "\n",
    "\n",
    "Let's calculate the TF (term-frequency) for those sentences.\n",
    "\n",
    "1. [0.33, 0.33, 0.33] \n",
    "2. [0.33, 0.33, 0.33] \n",
    "3. [0.25, 0.25, 0.25, 0.25]\n",
    "\n",
    "In sentence 1. are 3 unique words, the frequency of each word is therefore 1/3, same in sentence 2.Sentence 4 consists of 4 unique words with a corresponding frequency of 1/4.</br></br>\n",
    "\n",
    "Now, let's have a look at the IDF (inverse document frequency)</br></br>\n",
    "\n",
    "The IDF is the logarithm of the ratio of the total number of documents to the number of documents having this term. \n",
    "\n",
    "* The word \"recommender\" appears in all 3 documents and has therefore a Log(3/3) = 0\n",
    "* The word \"are\" appears in two documents with a corresponding Log(3/2) = 0.176\n",
    "* The words \"cool\", \"helpful\", etc appear in only one document leading to a Log(3/1) = 0.477</br></br>\n",
    "\n",
    "The IDF is a measure of how much information the word provides and as you can see, the IDF is higher the rarer the word is. This means rare words are more meaningful to get the context of a document.</br></br>\n",
    "\n",
    "\n",
    "The TF-IDF of a word then is the product of the TF and the IDF value. The higher the score, the more relevant is this word for that particular document and with regards to recommender system the more important they are to distinguish categories. \n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've learned how TF-IDF works, let's vectorize our data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#vectorize text column\n",
    "tf = TfidfVectorizer(analyzer='word',ngram_range=(1, 2),min_df=0)\n",
    "tfidf_matrix = tf.fit_transform(df['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we now have a vecorized matrix of our data, we can likewise above calculate the cosine similarity on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate cosine similarity\n",
    "cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix) # linear_kernel is a useful function to calculate cosine sim on large data sets "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And use the cosine similarity matrix to get our top 10 recommendations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build index with fish names\n",
    "names = df['name']\n",
    "indices = pd.Series(df.index, index=df['name'])\n",
    "\n",
    "# Function that get fish recommendations based on the cosine similarity \n",
    "def fish_recommendations(name):\n",
    "    #get the index of the fish we put into the function\n",
    "    idx = indices[name]\n",
    "    #calculate all cosine similarities to that fish and store it in a list\n",
    "    sim_scores = list(enumerate(cosine_sim[idx]))\n",
    "    #sort the list staring with the highest similarity\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    #get the similarities from 1:11 (not starting with 0 because it is the same fish)\n",
    "    sim_scores = sim_scores[1:11]\n",
    "    #get the indeces of that 10 fish\n",
    "    fish_indices = [i[0] for i in sim_scores]\n",
    "    #return the fish names of that 10 fish\n",
    "    return names.iloc[fish_indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get recommendations\n",
    "fish_recommendations('Blue Neon - Paracheirodon simulans')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Input:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Blue neon             |  \n",
    ":-------------------------:|\n",
    "<img src=\"https://blog.tetra.net/de-de/wp-content/uploads/2019/05/Platinum-Green-Neon-Fish-Paracheirodon-simulans_shutterstock_630636488_bearb-1200x800.jpg\" alt=\"drawing\" width=\"100\"/> |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pictures of Top 5 recommendations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Red Neon            |  Red Neon XL           |  Neon Tetra          |  Red-gold Neon         |  Dwarf tetra          \n",
    ":-------------------------:|:-------------------------: |:-------------------------: |:-------------------------: |:-------------------------:\n",
    "<img src=\"https://www.garnelio.de/media/image/a3/36/17/Paracheirodon-axelrodi-roter-neon2VaNjbtTw3umpb.jpg\" alt=\"drawing\" width=\"100\"/>  |  <img src=\"https://www.garnelio.de/media/image/a3/36/17/Paracheirodon-axelrodi-roter-neon2VaNjbtTw3umpb.jpg\" alt=\"drawing\" width=\"100\"/>   |  <img src=\"https://www.aquarium-ratgeber.com/wp-content/uploads/2021/08/neonsalmler-paracheirodon-innesi.jpg\" alt=\"drawing\" width=\"100\"/>  |  <img src=\"https://cdn02.plentymarkets.com/idwditcg5ajj/item/images/85046/full/Paracheirodon-axelrodi-GOLD.jpg\" alt=\"drawing\" width=\"100\"/>  |  <img src=\"https://www.zierfische-kotterba.de/wp-content/uploads/nanostomus-marginatus.jpg\" alt=\"drawing\" width=\"100\"/> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspecting the outcome, you can see that the most similar fish to the blue neon is the red neon, a close relative. If you want you can now play around with different fish names or use this notebook as a template for your own recommender :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.4 64-bit ('3.9.4')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "12546a0569d4971a52b421a01225a3969b17a7b5652796c625ce62be2ee0cfb1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}