notebooks/03_reading_writing_files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 3 - Reading and writing files  \n",
    "--------------------------------------------------------\n",
    "\n",
    "## Table of Content <a id='toc'></a>\n",
    "\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;[**Introduction**](#1)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Code design aspects: where is my file](#1.1)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Practical aspects: the `with open()` context manager](#1.2)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[filename argument](#1.21)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[mode argument](#1.22)  \n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;[**Reading from files**](#3)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Reading lines manually](#3.1)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[End-of-line characters](#3.2)  \n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;[**Writing to files**](#4)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;[**Exercises 3.1 and 3.2**](#5)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;[**Additional Material**](#6)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Reading a file's entire content at once](#6.1)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Easier reading of .csv formatted file](#6.2)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Opening files without context managers](#6.3)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Reading files using a while loop](#6.4)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[Some new cool syntax for Python >= 3.8](#6.5)  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[What does the `flush` argument of `print()` do?](#6.6)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "\n",
    "## Introduction <a id='1'></a>\n",
    "-------------------\n",
    "\n",
    "In many use cases, you will want your python code to **read/write from/to files** stored on your local hard drive.\n",
    "\n",
    "### Code design aspects: where is my file <a id='1.1'></a>\n",
    "\n",
    "Here are a few important points to consider when working with files:\n",
    "\n",
    "* **Where** is my file?\n",
    "* Do I need to read the **entire dataset/file into memory**?\n",
    "    * Accessing the hard drive is among the slower operations a computer performs. \n",
    "      Reading an entire file when you only need the first few lines is costly.\n",
    "    * If you are reading a very large file, having the entire file in memory at \n",
    "      once may overburden your computer's memory (RAM).\n",
    "* Are there **concurrency issues**?\n",
    "    * If another software (or even your own code if you have messed up) writes to a file you \n",
    "      are currently reading, you could run into trouble.\n",
    "\n",
    "<br>\n",
    "\n",
    "### Practical aspects: the `with open()` context manager <a id='1.2'></a>\n",
    "\n",
    "Whether it is for reading or for writing, operations on files require a **file object** (sometimes also referred-to as **file handle**).\n",
    "\n",
    "In modern python, files are opened with the syntax: **`with open(...) as ...`**.\n",
    "* Such statements staring with the **`with`** keyword are called **_context managers_**.\n",
    "* Context managers are code blocks that **do something** automatically when you enter them, and\n",
    "  also when you exit them.\n",
    "  * In the case of `with open(...) as ...`, the context manager opens the file when enter\n",
    "    the context manager, and automatically closes the file once you exit it (in addition, the\n",
    "    context manager also closes the file if an error occurs while the file is being accessed).  \n",
    "  * This is a nice feature, because it means that you will **never forget to close the file**.\n",
    "* Since context managers are code blocks, their content must be indented:\n",
    "\n",
    "    ```python\n",
    "    # Open a file in a given \"mode\" (e.g. read, write, or append).\n",
    "    with open(filename, mode) as file_handle:\n",
    "\n",
    "        # Do something with the file... (note the indentation)\n",
    "        #\n",
    "        # Inside the context manager block, we interact with the file\n",
    "        # using the \"file_handle\" variable (or however we named it).\n",
    "\n",
    "\n",
    "    # When you exit the context manager block, the file is automatically closed.\n",
    "\n",
    "    ```\n",
    "\n",
    "<br>\n",
    "\n",
    "#### `filename` argument: where is my file located ?  <a id='1.21'></a>\n",
    "\n",
    "The first argument that `open()` takes is the **file name**, which **may include the file path** (relative or absolute) if the file is not located in the current working directory.\n",
    "\n",
    "To pass the correct value to `filename`, you will need to know:\n",
    " 1. Where your file is,\n",
    " 2. Where you *code is executing from* (i.e. what is the working directory when you code runs)\n",
    " \n",
    "We leave question 1. to you, but with Jupyter Notebook, the second question has an easy answer: your code is executed where the jupyter notebook is saved (with a more classical python script, the code executes from the directory where you called `python`).\n",
    "\n",
    "Now, we need to make sure that our code can find the file:\n",
    "* **If the file is the same folder as the code**, then you can just use the name of the file,\n",
    "  no need for further modification.\n",
    "* **If the file is elsewhere**, you will have to specify a path to the file.\n",
    "  This can be either a:\n",
    " \n",
    "    * **Absolute path:** from the root of the computer to the file. e.g.,\n",
    "      - `\"C:\\Users\\JohnDoe\\Desktop\\ProjectP\\data\\myFile.txt\"` (Windows)\n",
    "      - `\"/home/JaneDoe/Documents/ProjectP/data/myFile.txt\"` (Linux, Mac)\n",
    "    \n",
    "    * **Relative path:** from your code to the file. e.g.,\n",
    "      - `\"data/myFile.txt\"` - here the file is in a subdirectory named `data`.\n",
    "      - `\"../otherProject/myFile.txt\"` - more complex, the file is in a subdirectory of the parent directory.\n",
    "      - *Note:* `..` indicates the parent directory of the current directory.\n",
    "\n",
    "<br>\n",
    "\n",
    "The last case - `\"../otherProject/myFile.txt\"` - depicts a situation like this:\n",
    "```\n",
    "parentFolder:\n",
    " | \n",
    " ├─ ProjectA\n",
    " |     └─ myCode.ipynb\n",
    " |\n",
    " └─ otherProject\n",
    "       └─ myFile.txt\n",
    "```\n",
    "\n",
    "<br>\n",
    "\n",
    "#### `mode` argument: the file opening mode  <a id='1.22'></a>\n",
    "\n",
    "When using the `open()` function, the optional **`mode`** argument can be passed to the function. This specifies the type of access you need on the file. For instance, the `\"r\"` mode will only allow to read the content of a file, and not to write to it (this is useful to avoid accidental writing to the file).\n",
    "\n",
    "There are several possible modes when opening files:\n",
    "* **`\"r\"`**: open file in read-only mode. This is the **default value** of the `mode` argument.\n",
    "* **`\"w\"`**: open file in write-only mode. This **overwrites** existing files with the same name.\n",
    "* **`\"a\"`**: open file in write-only mode, **appending** to existing files with the same name\n",
    "  (if the file does not exist yet, it is created).\n",
    "* **`\"rb\"`**, **`\"wb\"`**, **`\"ab\"`**: same as `\"r\"`, `\"w\"` and `\"a\"`, but reading/writing to/from binary files (such as `.zip` or `.bmp` image files). The content is read/written as bytes objects without any decoding.\n",
    "\n",
    "See `help(open)` or the [python online documentation](https://docs.python.org/3/library/functions.html#open) for a full list of modes and details about them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "\n",
    "[Back to ToC](#toc)\n",
    "\n",
    "## Reading from files  <a id='3'></a>\n",
    "--------------------------\n",
    "\n",
    "To initiate reading a file, we creates a **file object** using the `with open()` context manager with **`mode=\"r\"`**.\n",
    "\n",
    "```python\n",
    "with open(\"path/to/file\", mode=\"r\") as file_handle:\n",
    "    ...\n",
    "```\n",
    "\n",
    "### Reading lines manually <a id='3.1'></a>\n",
    "\n",
    "Now that we have a **file object** (file handle), we can use it to read from the file.  \n",
    "When reading a file with python, you can consider your file object a bit like a **cursor which starts at the beginning of your file**, and progresses toward the end of the file (it can go backward, but it is often a bit hacky to do so).\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "<img src=\"img/file_pointer_1.svg\" alr=\"a file pointer at opening a file\" width=\"500\">\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "\n",
    "You can read elements (ie. make the cursor advance) using the following methods:\n",
    " * **`.readline()`**: reads a **single line**.\n",
    " * **`.read()`**: reads the **entire remainder of the file** (from the current cursor position) in one go\n",
    "   and returns it as a **single string**.\n",
    " * **`.readlines()`**: reads the **entire remainder of the file** (from the current cursor position)\n",
    "   in one go, and returns it as a **`list` of `str` (string)**, where each element of the list is a\n",
    "   line of the file that was read: `[\"line 1...\", \"line 2...\", \"line 3...\", ...]`.\n",
    "\n",
    "> **Note:** all these methods return the text they read as `str` (string objects). This means that if you read\n",
    "  a number and want to use it as a `float` (e.g. to do math with it), you will need to convert it\n",
    "  from `str` to `float` yourself (with `float(x)`).\n",
    "\n",
    "<br>\n",
    "\n",
    "Let's focus on **`.readline()`**, the most used method, which reads a single line at a time. When we call it, the file pointer progresses to the next line and returns the line that was just read as a `str` object.\n",
    "Note how the returned strings contain the **end of line character `\\n`**.\n",
    "\n",
    "<br>\n",
    "\n",
    "We'll now examine `.readline()`, a frequently used method for reading files. Upon execution, `.readline()` moves the file pointer to the subsequent line and returns the line's content as a string (`str`). It's important to observe that the returned string contains the newline character `\\n`.\n",
    "\n",
    "<img src=\"img/file_pointer_2.svg\" alr=\"a file pointer after reading the first line\" width=\"500\">\n",
    "\n",
    "With each invocation of `.readline()`, a new line is read and the file pointer moves forward to the subsequent line.\n",
    "\n",
    "<img src=\"img/file_pointer_3.svg\" alr=\"a file pointer after reading the second line\" width=\"500\">\n",
    "\n",
    "After reading the last line, `.readline()` will **return an empty string** (`\"\"`). This empty string serves as a signal that the end of the file has been reached.\n",
    "\n",
    "<img src=\"img/file_pointer_4.svg\" alr=\"a file pointer at the end of a file\" width=\"500\">\n",
    "\n",
    "Note that we employed the `.read()` method to read the rest of the file's content from the file pointer's last position all at once.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "\n",
    "**Let's see an example with actual code:**\n",
    "\n",
    "Here we are going to read the file `data/fresh_fruits.txt`, which has the following content:\n",
    "\n",
    "```\n",
    "passionfruit\n",
    "oranges\n",
    "apples\n",
    "grapefruit (whole and segments)\n",
    "pointed sticks\n",
    "```\n",
    "\n",
    "<br>\n",
    "\n",
    "* Open the file, and **read a single line** from it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read and print the first line of the file \"data/fresh_fruits.txt\".\n",
    "\n",
    "with open(\"data/fresh_fruits.txt\", mode=\"r\") as reading_handle:\n",
    "    \n",
    "    line = reading_handle.readline()  # Reads a single line from the file.\n",
    "    print(\"line 1:\", line)            # Print the line to the screen.\n",
    "    \n",
    "\n",
    "# At this point the file is closed...\n",
    "# reading_handle.readline()  # This would fail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "* Open the file, and **read all its content**, line by line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "with open(\"data/fresh_fruits.txt\" , mode=\"r\") as reading_handle:\n",
    "    \n",
    "    # Read and print the first line in the file.\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 1:\", line)\n",
    "    \n",
    "    # Keep reading and printing lines.\n",
    "    # Problem: how many time should I do this ?\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 2:\", line)\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 3:\", line)\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 4:\", line)\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 5:\", line)\n",
    "    \n",
    "    # There are only 5 lines in this file.\n",
    "    # Once there are no more lines to read, .readline() returns an empty string.\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 6:\", line)\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 7:\", line)\n",
    "    line = reading_handle.readline()\n",
    "    print(\"line 8:\", line)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "Luckily for us, file objects are **iterable**, and we can thus use a **`for` loop** to read through files:\n",
    "* Each iteration reads 1 line.\n",
    "* The `for` loop ends when the entire file was read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/fresh_fruits.txt\", mode=\"r\") as reading_handle:\n",
    "    \n",
    "    for line in reading_handle:\n",
    "        print(line)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### End-of-line characters  <a id='3.2'></a>\n",
    "As you can see in the example above, there are additional empty lines in between our printed lines. This is because the lines are read from the file with their **end-of-line** characters, which generally is `\\n` .  \n",
    "\n",
    "To avoid this issue, one typically uses the **`.strip()`** method of string, which removes any whitespace or end-of-line character at the start and end of the string.\n",
    "\n",
    "* **Example:** use `.strip()` on a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = \"   hello    world !     \"\n",
    "print(\"Original: ---->\", s, \"<----\", sep=\"\")\n",
    "print(\"Stripped: ---->\", s.strip(), \"<----\", sep=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Example:** using `.strip()` when reading content from a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/fresh_fruits.txt\", 'r') as reading_handle:\n",
    "    \n",
    "    # Reminder: \"enumerate()\" is our friend that automatically enumerates items\n",
    "    # and creates tuples of the form \"(index, element)\".\n",
    "    for i, line in enumerate(reading_handle):\n",
    "        print(\"line\", i, \":\", line.strip())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "    \n",
    "### Micro Exercise 1 - reading a file with a for loop\n",
    "\n",
    "* Read the content of `data/titanic_head.csv` and print it. Make sure that no white space is printed between lines.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "\n",
    "\n",
    "[Back to ToC](#toc)\n",
    "\n",
    "## Writing to files  <a id='4'></a>\n",
    "----------------------\n",
    "\n",
    "Writing to a file is achieved in pretty much the same way as reading from it, but the opening mode is now **`\"w\"`**.  \n",
    "And instead of reading lines, we now `print()` them to the file.\n",
    "\n",
    "* **To print to a file** (instead of printing to standard-output as we do by default), we need to use the optional **`file` argument**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "help(print)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> *Note*: for people wondering what the `flush` argument does, please see the additional material  \n",
    "> Spoiler alert - it's a fairly minor argument, only useful in some edge cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"shopping_list.txt\", mode=\"w\") as f:\n",
    "    print(\"onion\", file=f)\n",
    "    print(34, \"potato\", file=f)\n",
    "    print(\"shrubbery\", file=f)\n",
    "    print(\"tomato sauce\", file=f)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By passing the file object (file handle) to the **`file` argument** of the `print()` function, we now print to the file rather than to our terminal (standard output).\n",
    "\n",
    "> **Reminder**: the **`\"w\"`** mode **overwrites** the opened file - if you use it on an existing file,\n",
    "> its original content is lost.\n",
    ">\n",
    "> **Pro tip:** you can open more than one file using a single `with` statement by using multiple\n",
    "> context managers in the same code block:\n",
    "> * Either on the same line:\n",
    ">      \n",
    ">  ```python\n",
    ">     with open(\"input.txt\", 'r') as in_file, open(\"output.txt\", 'w') as out_file:\n",
    ">         do_something()\n",
    ">  ```\n",
    ">  \n",
    "> * Or by having one context manager per line, and enclosing them all in parentheses `()`\n",
    ">   as shown below (better for readability) - *note: this only works with python >= 3.10* :\n",
    ">  ```python\n",
    ">    with (\n",
    ">        open(\"input.txt\", 'r') as in_file,\n",
    ">        open(\"output.txt\", 'w') as out_file,\n",
    ">    ):\n",
    ">        do_something()\n",
    ">  ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "#### Additional material\n",
    "\n",
    "You might sometimes see some Python code - especially older one - that uses the **`.write()`** method of the **file object** to write/print content to a file.\n",
    "\n",
    "* There are some differences between the `print()` method and `.write()`; the most important being\n",
    "  that `.write()` does not do any formatting and even the end-of-line (carriage return) characters\n",
    "  need to be manually written.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"shopping_list2.txt\", mode=\"w\") as f:\n",
    "    f.write(\"onion\\n\")\n",
    "    f.write(\"{} potato\\n\".format(34))\n",
    "    f.write(\"shrubbery\\n\")\n",
    "    f.write(\"tomato sauce\\n\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "### Micro Exercise 2 - copy a file's content\n",
    "\n",
    "Write some code to read, print and copy the content of the `shopping_list.txt` file we just created.  \n",
    "\n",
    "Specifically, your code should:\n",
    "  * Print the content of `shopping_list.txt` to screen.\n",
    "  * Make a copy of the content in a new file `shopping_list_copy.txt`.\n",
    "  * Make sure that no white space is printed between lines.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "\n",
    "## Exercises 3.1 and 3.2   <a id='5'></a>\n",
    "------------------------------\n",
    "\n",
    "* Exercises are found in a separate Jupyter Notebook.\n",
    "* If you have time, feel free to try the **additional exercises**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "[Back to ToC](#toc)\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "# Additional Material  <a id='6'></a>\n",
    "-------------------------------------\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### `readlines()` - reading a file's entire content at once <a id=\"6.1\"></a>\n",
    "\n",
    "Sometimes it is necessary to load the entire content of a file into memory at once. One way to achieve this is using the **`.readlines()`** method (note the plural marker \"s\" in the method's name).\n",
    "\n",
    "As its name suggests, `readlines()` reads more than one line at a time (by default, all lines in the file)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/fresh_fruits.txt\", mode=\"r\") as f:\n",
    "\n",
    "    entire_file = f.readlines()                       # The whole content of the file is now in memory.\n",
    "    print(f.name, \"has\", len(entire_file), \"lines:\")  # We can check how many lines there are\n",
    "                                                      # before we start looping over them.\n",
    "    for i, line in enumerate(entire_file):\n",
    "        print(\"line\", i, \":\", line.strip())\n",
    "\n",
    "print(entire_file)\n",
    "print(\"The file has\", len(entire_file), \"lines.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** while our examples both `readline()` and `readlines()` work equally well, there can be important implications in using one or the other, especially when dealing with large files. Can you think of a drawback of using `readlines()`?\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "**Answer** (enter edit mode or highlight to reveal): <font color='white'>using readlines() will (by default) load the entire file in memory, and this can be problematic when working with large files as is often the case in bioinformatics. Always consider the file sizes you are dealing with when using readlines().</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### Easier reading of .csv formatted file using modules  <a id='6.2'></a>\n",
    "\n",
    "`csv` (**C**omma **S**eparated **V**alue) is one of the most common file format when it comes to storing tabular data. In this format, each line contain a fixed number of values (columns), separated by a specific character (typically `\",\"`).\n",
    "\n",
    "\n",
    "Classically, when reading these files, we want to create some form of structure which reflects their tabular structure.  \n",
    "For instance, we can create a `list` where each row is a dictionary whose keys are the column name (found in the file's first line):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List where we will store the content of the \".csv\" file.\n",
    "# Each line of the file will be stored as a separate element (dict)\n",
    "# in the list.\n",
    "data = []\n",
    "\n",
    "with open(\"data/titanic_head.csv\") as f:\n",
    "    \n",
    "    line = f.readline()\n",
    "    \n",
    "    # The column names are in the first line\n",
    "    columnNames = line.strip().split(',')  # .split(',') is our best ally here : it cuts a str into a list. \n",
    "    \n",
    "    for line in f:\n",
    "        # Split the line in its different fields.\n",
    "        sl = line.strip().split(',')\n",
    "        \n",
    "        # Now we map the fields onto their constituent columns.\n",
    "        row = {}\n",
    "        for i in range(len(sl)):\n",
    "            row[columnNames[i]] = sl[i]\n",
    "\n",
    "        # Add the dictionary for the current row (line) to the list.\n",
    "        data.append(row)\n",
    "        \n",
    "\n",
    "print(\"Full data:\")\n",
    "for row in data:\n",
    "    print(row)\n",
    "\n",
    "print(\"***\")\n",
    "print(\"Name of passenger 4 : \", data[4][\"Name\"])\n",
    "print(\"Age of passenger 4 : \", data[4][\"Age\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sure, this works, but it is also fairly tedious to write.\n",
    "\n",
    "Because csv is such a widespread format, python comes with functions that that can help us out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the external \"csv\" module. Ignore this for now, we'll talk about it in the next notebook\n",
    "import csv\n",
    "\n",
    "data = []\n",
    "\n",
    "with open(\"data/titanic_head.csv\") as f:\n",
    "    \n",
    "    reader = csv.DictReader(f)  # Returns a DictReader object.\n",
    "    for row in reader:\n",
    "        data.append(row)        # Row is a dictionary whose keys correspond to the columns!\n",
    "\n",
    "for row in data:\n",
    "    print(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"full data:\")\n",
    "for row in data:\n",
    "    print(row)\n",
    "\n",
    "print(\"***\")\n",
    "print(\"Name of passenger 4 : \" , data[4][\"Name\"])\n",
    "print(\"Age of passenger 4 : \" , data[4][\"Age\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is much simpler.\n",
    "\n",
    "If the file uses **another field delimiter** (eg, `';'`), you can specify it when creating the `DictReader`:\n",
    "\n",
    "   ```python\n",
    "   reader = csv.DictReader( readingHandle , delimiter=';')\n",
    "   ```\n",
    "\n",
    "Additionally, libraries dedicated to data analysis often have functions that read directly from a csv file and create their specific data structure.\n",
    "\n",
    "For instance, for `pandas` (that's a sneak-peak into day3 modules ;-) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd  # ignore this, we'll talk about it in the next notebook.\n",
    "\n",
    "# Reading the csv file as a pandas.DataFrame, their custom type for tabular data.\n",
    "df = pd.read_csv(\"data/titanic_head.csv\") \n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### Opening files without context managers  <a id='6.3'></a>\n",
    "\n",
    "So far we have always used the build-in `open()` function as a *context manager* (i.e. `with open() as ...`).  \n",
    "However:\n",
    "\n",
    "* **`open()`** can also be used as a regular function.\n",
    "* But **don't forget the close it!** (usually the `with open()` context manager takes care of that for you)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_handle = open(\"data/fresh_fruits.txt\", \"r\")\n",
    "\n",
    "for i, line in enumerate(file_handle):\n",
    "    print(\"line\", i, \":\", line.strip())\n",
    "        \n",
    "# Don't forget to close the file!\n",
    "file_handle.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### Reading files using a while loop  <a id='6.4'></a>\n",
    "\n",
    "Here is an example of file reading where, instead of a `for` loop, we use a `while` loop and `.readline()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reading_handle = open(\"data/fresh_fruits.txt\", \"r\")\n",
    "line = reading_handle.readline()\n",
    "i = 0\n",
    "\n",
    "# When the file has been entirely read, readline() returns an empty string\n",
    "# and the while loop ends.\n",
    "# In python a non-empty string evaluates to \"True\", and therefore we can use\n",
    "# \"while line\" as a shortcut for \"while line != '' \".\n",
    "\n",
    "while line:\n",
    "    print(\"line\", i, \":\", line.strip())    # Note: we use the \"strip()\" method of \"str\" to remove the \n",
    "                                           # trailing \"\\n\" (carriage return) of each line.\n",
    "    line = reading_handle.readline()       # Don't forget this or you will have an infinite loop.\n",
    "    i += 1\n",
    "\n",
    "reading_handle.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### The walrus operator: a new syntax for Python >= 3.8  <a id='6.5'></a>\n",
    "\n",
    "Starting with Python 3.8, a new operator **`:=`** (a.k.a, the **walrus operator**) allows to do a variable assignment (`line` in the example below), while at the same time evaluating an expression.\n",
    "\n",
    "This can be used when reading a file to reduce the number of lines of code, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "with open(\"data/fresh_fruits.txt\", \"r\") as f:\n",
    "    i = 0\n",
    "    while (line := f.readline()):             # := assigns values to variables as part of a larger expression. \n",
    "        print(\"line\", i, \":\", line.strip())   # It is known as the \"walrus operator” and it works really well\n",
    "        i += 1                                # together with the while-loop\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "[Back to ToC](#toc)\n",
    "\n",
    "\n",
    "### What does the `flush` argument of `print()` do? <a id='6.6'></a>\n",
    "\n",
    "By default (i.e. when `flush=False`), the output of `print()` is buffered until either:\n",
    "* The buffer is full, or\n",
    "* a `\\n` is printed (or some other triggering event occurs).\n",
    "\n",
    "By passing `flush=True` to `print()`, the value passed to `print()` is printed immediately (along with everything that is still in the buffer at that time).\n",
    "\n",
    "<br>\n",
    "\n",
    "**Example:**\n",
    "* To observe the difference between `flush=False` and `flush=True` in practice, you can copy/paste\n",
    "  the following code into a python interpreter.\n",
    "* **Important:** this **will not work as expected in Jupyter Notebook**, as Jupyter Notebook always flushes the buffer.\n",
    "\n",
    "<br>\n",
    "\n",
    "```python\n",
    "import time\n",
    "\n",
    "# Default behavior: the buffer is flushed when a `\\n` gets printed at the end of the loop.\n",
    "# Note that there are no `\\n` printed inside the loop as we manually specify `end=\" \"`.\n",
    "for i in range(5):\n",
    "    print(i, end=\" \")\n",
    "    time.sleep(0.5)\n",
    "print(\"end\")\n",
    "\n",
    "# With `flush=True` the values passed to `print()` are immediately printed.\n",
    "for i in range(5):\n",
    "    print(i, end=\" \", flush=True)\n",
    "    time.sleep(0.5)\n",
    "print(\"end\")\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}