02_Web_Scraping_With_Beautiful_Soup/02_webscraping_bs4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To open this notebook in Google Colab and start coding, click on the Colab icon below.\n",
    "\n",
    "<table style=\"border:2px solid orange\" align=\"left\">\n",
    "    <td style=\"border:2px solid orange\">\n",
    "        <a target=\"_blank\" href=\"https://colab.research.google.com/github/neuefische/ds-meetups/blob/main/02_Web_Scraping_With_Beautiful_Soup/02_webscraping_bs4.ipynb\">\n",
    "        <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "    </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping\n",
    "\n",
    "Web scraping is the process of extracting and storing data from websites for analytical or other purposes. Therefore, it is useful to know the basics of html and css, because you have to identifiy the elements of a webpage you want to scrape. If you want to refresh your knowledge about these elements, check out the [HTML basics notebook](./01_HTML_Basics.ipynb).\n",
    "\n",
    "We will go through all the important steps performed during web scraping with python and BeautifulSoup in this Notebook.\n",
    "\n",
    "### Learning objectives for this Notebook\n",
    "\n",
    "At the end of this notebook you should:\n",
    "- be able to look at the structure of a real website\n",
    "- be able to figure out what information is relevant to you and how to find it (Locating Elements)\n",
    "- know how to download the HTML content with BeautifulSoup\n",
    "- know how to loop over an entire website structure and extract information\n",
    "- know how to save the data afterwards\n",
    "\n",
    "\n",
    "For web scraping it is useful to know the basics of html and css, because you have to identifiy the elements of a webpage you want to scrape. The easiest way to locate an element is to open your Chrome dev tools and inspect the element that you need. A cool shortcut for this is to highlight the element you want with your mouse and then press Ctrl + Shift + C or on macOS Cmd + Shift + C instead of having to right click + inspect each time (same in mozilla).\n",
    "\n",
    "## Locating Elements\n",
    "\n",
    "For locating an element on a website you can use:\n",
    "\n",
    "- Tag name\n",
    "- Class name\n",
    "- IDs\n",
    "- XPath\n",
    "- CSS selectors\n",
    "\n",
    "![alt text](./images/html_elements.png)\n",
    "\n",
    "XPath is a technology that uses path expressions to select nodes or node- sets in an XML document (or in our case an HTML document). [Read here for more information](https://www.scrapingbee.com/blog/practical-xpath-for-web-scraping/)\n",
    "\n",
    "## Is Web Scraping Legal?\n",
    "\n",
    "Unfortunately, there’s not a cut-and-dry answer here. Some websites explicitly allow web scraping. Others explicitly forbid it. Many websites don’t offer any clear guidance one way or the other.\n",
    "\n",
    "Before scraping any website, we should look for a terms and conditions page to see if there are explicit rules about scraping. If there are, we should follow them. If there are not, then it becomes more of a judgement call.\n",
    "\n",
    "Remember, though, that web scraping consumes server resources for the host website. If we’re just scraping one page once, that isn’t going to cause a problem. But if our code is scraping 1,000 pages once every ten minutes, that could quickly get expensive for the website owner.\n",
    "\n",
    "Thus, in addition to following any and all explicit rules about web scraping posted on the site, it’s also a good idea to follow these best practices:\n",
    "\n",
    "### Web Scraping Best Practices:\n",
    "\n",
    "- Never scrape more frequently than you need to.\n",
    "- Consider caching the content you scrape so that it’s only downloaded once.\n",
    "- Build pauses into your code using functions like time.sleep() to keep from overwhelming servers with too many requests too quickly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Problem we want to solve\n",
    "\n",
    "![](images/sad_larissa.png)\n",
    "\n",
    "Larissa's sister broke her aquarium. And we decided to get her a new one because christmas is near and we want to cheer Larissa up! And because we know how to code and can't decide what fish we want to get, we will solve this problem with web scraping!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BeautifulSoup\n",
    "\n",
    "The library we will use today to find fishes we can gift Larissa for christmas is [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/). It is a library to extract data out of HTML and XML files.\n",
    "\n",
    "The first thing we’ll need to do to scrape a web page is to download the page. We can download pages using the Python requests.\n",
    "\n",
    "The requests library will make a GET request to a web server, which will download the HTML contents of a given web page for us. There are several different types of requests we can make using requests, of which GET is just one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the content of the website\n",
    "page = requests.get(\"https://www.interaquaristik.de/tiere/zierfische\")\n",
    "html = page.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the BeautifulSoup library to parse this document, and extract the information from it.\n",
    "\n",
    "We first have to import the library, and create an instance of the BeautifulSoup class to parse our document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# parse the html and save it into a BeautifulSoup instance\n",
    "bs = BeautifulSoup(html, 'html.parser')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now print out the HTML content of the page, formatted nicely, using the prettify method on the BeautifulSoup object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(bs.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This step isn't strictly necessary, and we won't always bother with it, but it can be helpful to look at prettified HTML to make the structure of the page clearer and nested tags are easier to read.\n",
    "\n",
    "As all the tags are nested, we can move through the structure one level at a time. We can first select all the elements at the top level of the page using the children property of ``bs``.\n",
    "\n",
    "Note that children returns a list generator, so we need to call the list function on it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(bs.findChildren())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we can have a closer look on the children. For example the ```head```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.find('head')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you can try out different tags like ```body```, headers like ```h1``` or ```title```:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.find('insert your tag here')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But what if we have more than one element with the same tag? Then we can just use the ```.find_all()``` method of BeautifulSoup:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.find_all('article')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also you can search for more than one tag at once for example if you want to look for all headers on the page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titles = bs.find_all(['h1', 'h2','h3','h4','h5','h6'])\n",
    "print([title for title in titles])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often we are not interested in the tags themselves, but in the content they contain. With the ```.get_text()``` method we can easily extract the text from between the tags. So let's find out if we really scrape the right page to buy the fishes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.find('title').get_text()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Searching for tags by class and id\n",
    "We introduced ```classes``` and ```ids``` earlier, but it probably wasn’t clear why they were useful.\n",
    "\n",
    "Classes and ```ids``` are used by ```CSS``` to determine which ```HTML``` elements to apply certain styles to. For web scraping they are also pretty useful as we can use them to specify the elements we want to scrape. In our case the ```ìds``` are not that useful there are only a few of them but one example would be:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.find_all('div', id='page-body')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But it seems like that the ```classes``` could be useful for finding the fishes and their prices, can you spot the necessary tags in the DevTool of your browser?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tag of the description of the fishes\n",
    "bs.find_all(class_=\"insert your tag here for the name\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tag of the price of the fishes\n",
    "bs.find_all(class_=\"insert your tag here for the price\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting all the important information from the page\n",
    "Now that we know how to extract each individual piece of information, we can save these informations to a list. Let's start with the price:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will search for the price\n",
    "prices = bs.find_all(class_= \"price\")\n",
    "\n",
    "prices_lst = [price.get_text() for price in prices]\n",
    "prices_lst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We seem to be on the right track but like you can see it doesn't handle the special characters, spaces and paragraphs. So web scraping is coming hand in hand with cleaning your data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prices_lst = [price.strip() for price in prices_lst]\n",
    "prices_lst[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks a little bit better but we want only the number to work with the prices later. We have to remove the letters and convert the string to a float:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We are removing the letters from the end of the string and keeping only the first part\n",
    "prices_lst = [price.replace('\\xa0€ *', '') for price in prices_lst]\n",
    "prices_lst[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we have to replace the comma with a dot to convert the string to a float\n",
    "prices_lst = [price.replace(',', '.') for price in prices_lst]\n",
    "prices_lst[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# So lets convert the string into a float\n",
    "prices_lst = [float(price) for price in prices_lst]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But if we want to convert the string to a flaot we get an error message there seems to be prices which start with ```ab```.\n",
    "So let me intodruce you to a very handy thing called ```Regular expressions``` or short ```regex```. It is a sequence of characters that specifies a search pattern. In python you can use regex with the ```re``` library. So lets have a look how many of the prices still contain any kind of letters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with the regex sequence we are looking for strings that contain any\n",
    "# kind of letters\n",
    "for price in prices_lst:\n",
    "    if re.match(\"^[A-Za-z]\", price):\n",
    "        print(price)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So there are some prices with an \"ab\" in front of them, so lets remove the letters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we have to replace the comma with a dot to convert the string to a float\n",
    "prices_lst = [float(price.replace('ab ', '')) for price in prices_lst]\n",
    "prices_lst[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it worked! so let's do the same with the description of the fishes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find all the descriptions of the fish and save them in a variable\n",
    "descriptions = bs.find_all(class_='thumb-title small')\n",
    "\n",
    "# Get only the text of the descriptions\n",
    "descriptions_lst = [description.get_text() for description in descriptions]\n",
    "descriptions_lst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean the text by removing spaces and paragraphs\n",
    "descriptions_lst = [description.strip() for description in descriptions_lst]\n",
    "descriptions_lst[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look if we can get the links to the images of the fish, so that we later can look up how the fish are looking, we can use the ```img``` tag for that in most cases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find all images of the fish\n",
    "image_lst = bs.find('ul', {'class': 'product-list row grid'})\n",
    "images = image_lst.find_all('img')\n",
    "images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are only two results for the image tag so let's have a look what the tag of the other images are.\n",
    "\n",
    "So they have the tag: ```picture``` so lets extract those:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract all the pictures for the fish by using first the tag ul and than the tag picture\n",
    "picture_lst = bs.find('ul', {'class': 'product-list row grid'})\n",
    "pictures = picture_lst.find_all('picture')\n",
    "pictures[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks more like all pictures! \n",
    "Although, it seems some of the fish have specials like 'Sonderangebot' or 'Neuheit'. Wouldn't it be nice if we would have this information as well?  Here it gets a little bit tricky because the 'Sonderangebot' and 'Neuheit' do not have the same ```classes``` in the ```span``` but if we go one tag higher we can get all of them: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extracting all the special offers by using the div tag and the class 'special-tags p-2'\n",
    "specials = bs.find_all('div', {'class' : 'special-tags p-2'})\n",
    "specials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want only the text from the ```span``` we now can iterate over the specials list and extract the text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to get only the text from the specials we are iterating over all specials\n",
    "for special in specials:\n",
    "    # and than get the text of all spans from the special objects\n",
    "    special_text = special.find(\"span\").get_text().strip()\n",
    "    print(special_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice that will help us for making a decision what fish to buy!\n",
    "\n",
    "But so far we only scraped the first page there are more fish on the next pages. There are 29 pages of fish. So how can we automate this? <br>\n",
    "So this is the link of the first page: https://www.interaquaristik.de/tiere/zierfische <br>\n",
    "The second link of the second page looks like this: https://www.interaquaristik.de/tiere/zierfische?page=2 <br>\n",
    "The third: https://www.interaquaristik.de/tiere/zierfische?page=3 <br>\n",
    "\n",
    "So the only thing that changes is the ending... Let's use this! But don't forget each request is causing traffic for the server, so we will set a sleep timer between requests!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "link = 'https://www.interaquaristik.de/tiere/zierfische'\n",
    "for _ in range(30):\n",
    "    time.sleep(3)\n",
    "    if _ == 0:\n",
    "        page = requests.get(link)\n",
    "        html = page.content\n",
    "    else:\n",
    "        print(link + f'?page={_}')\n",
    "        page = requests.get(link + f'?page={_}')\n",
    "        html = page.content\n",
    "```\n",
    "\n",
    "This will be our starting point!\n",
    "We will save our results in a pandas data frame so that we can work with the data later. Therefore we will create a empty data frame and append our data to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating an empty Dataframe for later use\n",
    "df = pd.DataFrame()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But first lets create some functions for the scraping part:\n",
    "1. for the description\n",
    "2. for the price\n",
    "3. for the images\n",
    "4. for specials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a function to get all the description\n",
    "def get_description(lst_name):\n",
    "    ''' \n",
    "    Get all the description from the fish by class_ = 'thumb-title small'\n",
    "    and saving it to an input list.\n",
    "    Input: list\n",
    "    Output: list\n",
    "    '''\n",
    "    # find all the descriptions and save them to a list\n",
    "    fish = bs.find_all(class_='thumb-title small')\n",
    "    # iterate over the list fish to get the text and strip the strings\n",
    "    for names in fish:\n",
    "        lst_name.append(\n",
    "            names.get_text()\\\n",
    "                .strip()\n",
    "        )\n",
    "    return lst_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a function to get all the prices\n",
    "def get_price(lst_name):\n",
    "    ''' \n",
    "    Get all the prices from the fish by class_ = 'prices'\n",
    "    and saving it to an input list.\n",
    "    Input: list\n",
    "    Output: list\n",
    "    '''\n",
    "    # find all the prices and save them to a list\n",
    "    prices = bs.find_all(class_='prices')\n",
    "    # iterate over the prices\n",
    "    for price in prices:\n",
    "        # try to clean the strings from spaces, letters and paragraphs and convert it into a float\n",
    "        try:\n",
    "            price = float(price.get_text()\\\n",
    "                .strip()\\\n",
    "                .replace('\\xa0€ *','')\\\n",
    "                .replace(',','.')\\\n",
    "                .replace('ab ', '')\n",
    "            )\n",
    "        except:\n",
    "            # in some cases there is no * in the string like here: '\\xa0€ *' with the except we try to intercept this\n",
    "            price = price.get_text()\\\n",
    "                .split('\\n')[0]\\\n",
    "                .replace('\\xa0€','')\n",
    "            if price != '':\n",
    "                price = 0.0                  \n",
    "            else:\n",
    "                price = float(price)  \n",
    "        # append the prices to the lst_name list\n",
    "        lst_name.append(\n",
    "           price\n",
    "        )\n",
    "    return lst_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a function to get all the images\n",
    "def get_image(lst_name_1, lst_name_2):\n",
    "    ''' \n",
    "    Get all the images from the fish by tag = 'ul' and class_ = 'product-list row grid'\n",
    "    and saving the name to one lst_name_1 and the link of the image to another lst_name_2.\n",
    "    Input: list_1, list_2\n",
    "    Output: list_1, list_2\n",
    "    '''\n",
    "    # find all images\n",
    "    images_listings = bs.find('ul', {'class': 'product-list row grid'})\n",
    "    images = images_listings.find_all('img')\n",
    "    # find all pictures\n",
    "    pictures_listings = bs.find('ul', {'class': 'product-list row grid'})\n",
    "    pictures = pictures_listings.find_all('picture')\n",
    "    # iterate over the images and save the names of the fish in one list and the link to the image in another one\n",
    "    for image in images:\n",
    "        lst_name_1.append(image['src'])\n",
    "        lst_name_2.append(image['alt'].strip())\n",
    "    # iterate over the pictures and save the names of the fish in one list and the link to the image in another one\n",
    "    for picture in pictures:\n",
    "        lst_name_1.append(picture['data-iesrc'])\n",
    "        lst_name_2.append(picture['data-alt'].strip())\n",
    "    return lst_name_1, lst_name_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_special(lst_name_1, lst_name_2):\n",
    "    ''' \n",
    "    Get all the images from the fish by tag = 'div' and class_ = 'thumb-inner'\n",
    "    and saving the name to one lst_name_1 and the index to another lst_name_2.\n",
    "    Input: list_1, list_2\n",
    "    Output: list_1, list_2\n",
    "    '''\n",
    "    # use the article as tag to get the index of all articles\n",
    "    article_lst = bs.find_all('div', {'class' : 'thumb-inner'})\n",
    "    # iterate over all articles with enumerate to get the single articles and the index\n",
    "    for idx,article in enumerate(article_lst):\n",
    "        # get all specials\n",
    "        spans = article.find('div', {'class' : 'special-tags p-2'})\n",
    "        # and if there is a special save the special and the index each to a list\n",
    "        if spans != None:\n",
    "            special = spans.find(\"span\").get_text().strip()\n",
    "            lst_name_1.append(special)\n",
    "            lst_name_2.append(idx)\n",
    "    return lst_name_1, lst_name_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will combine it all and could scrape all pages:\n",
    "\n",
    "**NOTE:** We have commented out the code, because we don't want to overwhelm the server with the requests of participants in the meetup. Feel free to run the code after the meetup. We ran the code once and uploaded the result in a csv file to github so the following code will still work!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#link = 'https://www.interaquaristik.de/tiere/zierfische'\n",
    "#\n",
    "## for loop to get the page numbers\n",
    "#for _ in range(30):\n",
    "#    # sleep timer to reduce the traffic for the server\n",
    "#    time.sleep(3)\n",
    "#    # create the lists for the functions\n",
    "#    fish_names = []\n",
    "#    fish_prices = []\n",
    "#    picture_lst = []\n",
    "#    picture_name = []\n",
    "#    index_lst =[]\n",
    "#    special_lst = []\n",
    "#    # first iteration is the main page\n",
    "#    if _ == 0:\n",
    "#        # get the content\n",
    "#        page = requests.get(link)\n",
    "#        html = page.content\n",
    "#        bs = BeautifulSoup(html, 'html.parser')\n",
    "#        # call the functions to get the information\n",
    "#        get_description(fish_names)\n",
    "#        get_price(fish_prices)\n",
    "#        get_image(picture_lst, picture_name)\n",
    "#        get_special(special_lst, index_lst)\n",
    "#        # create a pandas dataframe for the names and prices\n",
    "#        fish_dict = {\n",
    "#            'fish_names': fish_names,\n",
    "#            'fish_prices in EUR': fish_prices\n",
    "#        }\n",
    "#        df_fish_info = pd.DataFrame(data=fish_dict)\n",
    "#        # create a pandas dataframe for the pictures\n",
    "#        picture_dict = {\n",
    "#            'fish_names': picture_name,\n",
    "#            'pictures': picture_lst\n",
    "#        }\n",
    "#        df_picture = pd.DataFrame(data=picture_dict)\n",
    "#        \n",
    "#        # merge those two dataframes on the fishnames\n",
    "#        df_ = pd.merge(df_fish_info, df_picture, on='fish_names', how='outer')\n",
    "#        \n",
    "#        # create a pandas dataframe for the specials\n",
    "#        specials_dict = {\n",
    "#            'special': special_lst\n",
    "#        }\n",
    "#        real_index = pd.Series(index_lst)\n",
    "#        df_specials = pd.DataFrame(data=specials_dict)\n",
    "#        df_specials.set_index(real_index, inplace=True)\n",
    "#        \n",
    "#        # merge the dataframes on the index\n",
    "#        df_ = pd.merge(df_, df_specials, left_index=True,right_index=True, how='outer')\n",
    "#        # append the temporary dataframe to the dataframe we created earlier outside the for loop\n",
    "#        df = df.append(df_)\n",
    "#    # else-statment for the next pages\n",
    "#    else:\n",
    "#        # get the content from the links we create with a f-string an the number we get from the for-loop\n",
    "#        page = requests.get(link+f'?page={_}')\n",
    "#        html = page.content\n",
    "#        bs = BeautifulSoup(html, 'html.parser')\n",
    "#        # call the functions to get the information\n",
    "#        get_description(fish_names)\n",
    "#        get_price(fish_prices)\n",
    "#        get_image(picture_lst, picture_name)\n",
    "#        get_special(special_lst, index_lst)\n",
    "#        # create a pandas dataframe for the names and prices\n",
    "#        fish_dict = {\n",
    "#            'fish_names': fish_names,\n",
    "#            'fish_prices in EUR': fish_prices\n",
    "#        }\n",
    "#        df_fish_info = pd.DataFrame(data=fish_dict)\n",
    "#        # create a pandas dataframe for the pictures\n",
    "#        picture_dict = {\n",
    "#            'fish_names': picture_name,\n",
    "#            'pictures': picture_lst\n",
    "#        }\n",
    "#        df_picture = pd.DataFrame(data=picture_dict)\n",
    "#        \n",
    "#        # merge those two dataframes on the fishnames\n",
    "#        df_ = pd.merge(df_fish_info, df_picture, on='fish_names', how='outer')\n",
    "#        \n",
    "#        # create a pandas dataframe for the specials\n",
    "#        specials_dict = {\n",
    "#            'special': special_lst\n",
    "#        }\n",
    "#        real_index = pd.Series(index_lst)\n",
    "#        df_specials = pd.DataFrame(data=specials_dict)\n",
    "#        df_specials.set_index(real_index, inplace=True)\n",
    "#        \n",
    "#        # merge the dataframes on the index\n",
    "#        df_ = pd.merge(df_, df_specials, left_index=True,right_index=True, how='outer')\n",
    "#        # append the temporary dataframe to the dataframe we created earlier outside the for loop\n",
    "#        df = df.append(df_)\n",
    "#        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#checking if everything worked\n",
    "#df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The web scraping part is over and the following part is only looking at the data.\n",
    "We will save the dataframe to a csv file so that we don't have to scrape the info again!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking for duplicates something that can happen quickly while scraping\n",
    "df.pivot_table(columns=['fish_names'], aggfunc='size')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems like we have some duplicates. Let's drop them!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df.drop_duplicates(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the dataframe to a csv file without index\n",
    "#df.to_csv('fish_data.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we haven't run the code for scraping all pages, we uploaded the data we scraped before to github and we now can load it into pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reading the csv file from github\n",
    "df = pd.read_csv('https://raw.githubusercontent.com/neuefische/ds-meetups/main/02_Web_Scraping_With_Beautiful_Soup/fish_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#checking if everything worked\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want fish for Larissa that she has never had before, that is why we are looking for new items (Neuheiten)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Query over the dataframe and keeping only the fish with the special Neuheit\n",
    "df_special_offer = df.query('special == \"Neuheit\"')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_special_offer.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a budget of around 250 € and we want to buy at least 10 fish so we will filter out fishes that are more expensive than 25 €!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filtering only for the fish that are cheaper than 25 EUR\n",
    "df_final = df_special_offer[df_special_offer['fish_prices in EUR'] <= 25]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let's write some code that chooses the fish for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# our budget\n",
    "BUDGET = 250\n",
    "# a list for the fish we will buy\n",
    "shopping_bag = []\n",
    "# a variable here we save the updating price in\n",
    "price = 0\n",
    "# we are looking for fish until our budget is reached\n",
    "while price <= BUDGET:\n",
    "    # samples the dataframe randomly\n",
    "    df_temp = df_final.sample(1)\n",
    "    # getting the name from the sample\n",
    "    name = df_temp['fish_names'].values\n",
    "    # getting the price from the sample\n",
    "    fish_price = df_temp['fish_prices in EUR'].values\n",
    "    # updating our price\n",
    "    price += fish_price\n",
    "    # adding the fish name to the shopping bag\n",
    "    shopping_bag.append((name[0],fish_price[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('display.max_colwidth', None)\n",
    "\n",
    "print(f\"We are at a price point of {price[0].round(2)} Euro and this are the fish we chose:\")\n",
    "res=pd.DataFrame(shopping_bag,columns=[\"Name\",\"Price [€]\"])\n",
    "display(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Christmas can come!\n",
    "\n",
    "![](images/happy_larissa.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "c52cf4adb870298e6b71ade4770958cc553b28308cc1a016d002d57ee09d01e1"
  },
  "kernelspec": {
   "display_name": "Python 3.9.4 64-bit ('.venv': venv)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}