lab_exercises/lab3_1_NumpyAlgebra.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "\n",
    "\n",
    "\n",
    "## Linear Algebra\n",
    "\n",
    "Those exercises will involve vector and matrix math, the  <a href=\"http://wiki.scipy.org/Tentative_NumPy_Tutorial\">NumPy</a> Python package.\n",
    "\n",
    "This exercise will be divided into two parts:\n",
    "\n",
    "#### 1. Math checkup\n",
    "Where you will do some of the math by hand.\n",
    "\n",
    "#### 2. NumPy and Spark linear algebra\n",
    "You will do some exercise using the NumPy package.\n",
    "\n",
    "<br>\n",
    "In the following exercises you will need to replace the code parts in the cell that starts with following comment: \"#Replace the `<INSERT>`\"\n",
    "\n",
    "To go through the notebook fill in the `<INSERT>`:s with appropriate code in the cells. \n",
    "To run a cell press Shift-Enter to run it and advance to the following cell or Ctrl-Enter to only run the code in the cell. You should do the exercises from the top to the bottom in this notebook, because following cells may depend on code in previous cells.\n",
    "\n",
    "If you want to execute these lines in a python script, you will need to create first a spark context:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#from pyspark import SparkContext, StorageLevel \\\n",
    "#from pyspark.sql import SQLContext \\\n",
    "#sc = SparkContext(master=\"local[*]\") \\\n",
    "#sqlContext = SQLContext(sc) \\"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "But since we are using the notebooks, those lines are not needed here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Math checkup\n",
    "\n",
    " 1.1 Euclidian norm\n",
    "\n",
    "$$\n",
    "\\mathbf{v} = \\begin{bmatrix}\n",
    "  666 \\\\\n",
    "  1337 \\\\\n",
    "  1789 \\\\\n",
    "  1066 \\\\\n",
    "  1945 \\\\\n",
    " \\end{bmatrix}\n",
    " \\qquad\n",
    " \\|\\mathbf{v}\\| = ?\n",
    " $$\n",
    "\n",
    "Calculate the euclidian norm for the $\\mathbf{v}$ using the following definition:\n",
    "\n",
    "$$\n",
    "\\|\\mathbf{v}\\|_2 = \\sqrt{\\sum\\limits_{i=1}^n {x_i}^2} = \\sqrt{{x_1}^2+\\cdots+{x_n}^2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "import math\n",
    "import numpy as np\n",
    "v = [666, 1337, 1789, 1066, 1945]\n",
    "rdd = sc.parallelize(v)\n",
    "sumOfSquares = rdd.map(<INSERT>).reduce(<INSERT>) \n",
    "# <INSERT round to 8 decimals > \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Helper function to check results\n",
    "import hashlib\n",
    "def hashCheck(x, hashCompare): #Defining a help function\n",
    "    hash = hashlib.md5(str(x).encode('utf-8')).hexdigest()\n",
    "\n",
    "    if hash == hashCompare:\n",
    "        print('Yay, you succeeded!')\n",
    "    else:\n",
    "        print('Try again!')\n",
    "        \n",
    "def check(x,y,label):\n",
    "    if(x == y):\n",
    "        print(\"Yay, \"+label+\" is correct!\")\n",
    "    else:\n",
    "        print(\"Nay, \"+label+\" is incorrect, please try again!\")\n",
    "\n",
    "def checkArray(x,y,label):\n",
    "    if np.allclose(x,y):\n",
    "        print(\"Yay, \"+label+\" is correct!\")\n",
    "    else:\n",
    "        print(\"Nay, \"+label+\" is incorrect, please try again!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Check if the norm is correct\n",
    "hashCheck(norm_numpy, '6de149ccbc081f9da04a0bbd8fe05d8c')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2 Transpose\n",
    "\n",
    "$$\n",
    "\\mathbf{A} = \\begin{bmatrix}\n",
    "  1 & 2 & 3\\\\\n",
    "  4 & 5 & 6\\\\\n",
    "  7 & 8 & 9\\\\\n",
    " \\end{bmatrix}\n",
    " \\qquad\n",
    " \\mathbf{A}^T = ?\n",
    "$$\n",
    "\n",
    "Tranpose is an operation on matrices that swaps the row for the columns.\n",
    "\n",
    "$$\n",
    "\\begin{bmatrix}\n",
    "  2 & 7 \\\\\n",
    "  3 & 11\\\\\n",
    "  5 & 13\\\\\n",
    " \\end{bmatrix}^T\n",
    " \\Rightarrow\n",
    " \\begin{bmatrix}\n",
    "  2 & 3 & 5 \\\\\n",
    "  7 & 11 & 13\\\\\n",
    " \\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Do the transpose of A by hand and write it in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "#Input aT like this: AT = [[1, 2, 3],[4, 5, 6],[7, 8, 9]]\n",
    "#At = <INSERT>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Check if the transpose is correct\n",
    "hashCheck(At, '1c8dc4c2349277cbe5b7c7118989d8a5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3 Scalar matrix multiplication\n",
    "\n",
    "$$\n",
    "\\mathbf{A} = 3\\times\\begin{bmatrix}\n",
    "  1 & 2 & 3\\\\\n",
    "  4 & 5 & 6\\\\\n",
    "  7 & 8 & 9\\\\\n",
    " \\end{bmatrix}\n",
    "=?\n",
    "\\qquad\n",
    "\\mathbf{B} = 5\\times\\begin{bmatrix}\n",
    "  1\\\\\n",
    "  -4\\\\\n",
    "  7\\\\\n",
    " \\end{bmatrix}\n",
    "=?\n",
    "$$\n",
    "\n",
    "The operation is done element-wise, e.g. $k\\times\\mathbf{A}=\\mathbf{C}$ then $k\\times a_{i,j}={k}c_{i,j}$.\n",
    "\n",
    "$$\n",
    " 2\n",
    " \\times\n",
    " \\begin{bmatrix}\n",
    "  1 & 6 \\\\\n",
    "  4 & 8 \\\\\n",
    " \\end{bmatrix} \n",
    " = \n",
    " \\begin{bmatrix}\n",
    "  2\\times1& 2\\times6 \\\\\n",
    "  2\\times4 & 2\\times8\\\\\n",
    " \\end{bmatrix}\n",
    " =\n",
    " \\begin{bmatrix}\n",
    "  2& 12 \\\\\n",
    "  8 & 16\\\\\n",
    " \\end{bmatrix}\n",
    " $$\n",
    " \n",
    " $$\n",
    " 11\n",
    " \\times\n",
    " \\begin{bmatrix}\n",
    "  2  \\\\\n",
    "  3  \\\\\n",
    "  5  \\\\\n",
    " \\end{bmatrix} \n",
    " = \n",
    " \\begin{bmatrix}\n",
    "  11\\times2  \\\\\n",
    "  11\\times3  \\\\\n",
    "  11\\times5  \\\\\n",
    " \\end{bmatrix}\n",
    " =\n",
    " \\begin{bmatrix}\n",
    " 22\\\\\n",
    " 33\\\\\n",
    " 55\\\\\n",
    " \\end{bmatrix}\n",
    " $$\n",
    "\n",
    "Do the scalar multiplications of $\\mathbf{A}$ and $\\mathbf{B}$ by hand and write them in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "#Input A like this: A = [[1, 2, 3],[4, 5, 6],[7, 8, 9]]\n",
    "#And B like this: B = [1, -4, 7]\n",
    "\n",
    "#A = <INSERT>\n",
    "#B = <INSERT>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Check if the scalar matrix multiplication is correct\n",
    "hashCheck(A, '91b9508ec9099ee4d2c0a6309b0d69de')\n",
    "hashCheck(B, '88bddc0ee0eab409cee011770363d007')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.4 Dot product\n",
    "$$\n",
    "c_1=\\begin{bmatrix}\n",
    "  11  \\\\\n",
    "  2  \\\\\n",
    " \\end{bmatrix} \n",
    " \\cdot\n",
    " \\begin{bmatrix}\n",
    "  3 \\\\\n",
    "  5 \\\\\n",
    " \\end{bmatrix}\n",
    " =?\n",
    "\\qquad\n",
    "c_2=\\begin{bmatrix}\n",
    "  1  \\\\\n",
    "  2  \\\\\n",
    "  3  \\\\\n",
    " \\end{bmatrix} \n",
    " \\cdot\n",
    " \\begin{bmatrix}\n",
    "  4 \\\\\n",
    "  5 \\\\\n",
    "  6 \\\\\n",
    " \\end{bmatrix}\n",
    " =?\n",
    "$$\n",
    "The operations are done element-wise, e.g. $\\mathbf{v}\\cdot\\mathbf{w}=k$ then $\\sum v_i \\times w_i =k$\n",
    "\n",
    "$$\n",
    " \\begin{bmatrix}\n",
    "  2  \\\\\n",
    "  3  \\\\\n",
    "  5  \\\\\n",
    " \\end{bmatrix} \n",
    " \\cdot\n",
    " \\begin{bmatrix}\n",
    "  1 \\\\\n",
    "  4 \\\\\n",
    "  6 \\\\\n",
    " \\end{bmatrix}\n",
    " = 2\\times1+3\\times4+5\\times6=44\n",
    " $$\n",
    " \n",
    " Calculate the values of $c_1$ and $c_2$ by hand and write them in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "#Input c1 and c2 like this: c = 1337\n",
    "#c1 = <INSERT>\n",
    "#c2 = <INSERT>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Check if the dot product is correct\n",
    "hashCheck(c1, '17e62166fc8586dfa4d1bc0e1742c08b')\n",
    "hashCheck(c2, '6364d3f0f495b6ab9dcf8d3b5c6e0b01')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.5 Matrix multiplication\n",
    " $$\n",
    " \\mathbf{A}=\n",
    " \\begin{bmatrix}\n",
    " 682 &  848 & 794 & 954 \\\\\n",
    " 700 & 1223 & 1185 &  816 \\\\\n",
    " 942 & 428 &  324 &  526 \\\\\n",
    " 321 &  543 &  532 &  614 \\\\\n",
    " \\end{bmatrix}\n",
    " \\qquad\n",
    "  \\mathbf{B}=\n",
    " \\begin{bmatrix}\n",
    "  869 & 1269 & 1306 & 358 \\\\\n",
    "  1008 & 836 & 690 & 366 \\\\\n",
    "  973 & 619 &  407 & 1149 \\\\\n",
    "  323 & 42 & 405 & 117 \\\\\n",
    " \\end{bmatrix}\n",
    " \\qquad\n",
    " \\mathbf{A}\\times\\mathbf{B}=\\mathbf{C}=?\n",
    " $$\n",
    "\n",
    "The $c_{i,j}$ entry is the dot product of the i-th row in $\\mathbf{A}$ and the j-th column in $\\mathbf{B}$\n",
    "\n",
    "Calculate $\\mathbf{C}$ by implementing the naive matrix multiplication algotrithm with $\\mathcal{O}(n^3)$ run time, by using the tree nested for-loops below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The convention is to import NumPy as the alias np\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "A = [[ 682,  848,  794,  954],\n",
    "     [ 700, 1223, 1185,  816],\n",
    "     [ 942,  428,  324,  526],\n",
    "     [ 321,  543,  532,  614]]\n",
    "\n",
    "B = [[ 869, 1269, 1306,  358],\n",
    "     [1008,  836,  690,  366],\n",
    "     [ 973,  619,  407, 1149],\n",
    "     [ 323,   42,  405,  117]]\n",
    "\n",
    "C = [[0]*4 for i in range(4)]\n",
    "\n",
    "#Iterate through rows of A\n",
    "for i in range(len(A)):\n",
    "   #Iterate through columns of B\n",
    "   for j in range(len(B[0])):\n",
    "       #Iterate through rows of B\n",
    "       for k in range(len(B)):\n",
    "           C[i][j] += A[i][k] * B[k][j]\n",
    "        \n",
    "print(np.matrix(C))\n",
    "print(np.matrix(A)*np.matrix(B))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Check if the matrix multiplication is correct\n",
    "hashCheck(C, 'f6b7b0500a6355e8e283f732ec28fa76')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. NumPy and Spark linear algebra\n",
    "\n",
    "A python library to utilize arrays is  <a href=\"http://wiki.scipy.org/Tentative_NumPy_Tutorial\">NumPy</a>. The library is optimized to be fast and memory efficient, and provide abstractions corresponding to vectors, matrices and the operations done on these objects.\n",
    "\n",
    "Numpy's array class is called <a href=\"http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.html\">ndarray</a>, it is also known by the alias array. This is a multidimensional array of fixed-size that contains numerical elements of one type, e.g. floats or integers.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1 Scalar matrix multiplication using NumPy\n",
    "\n",
    "$$\n",
    "\\mathbf{A} = \\begin{bmatrix}\n",
    "  1 & 2 & 3\\\\\n",
    "  4 & 5 & 6\\\\\n",
    "  7 & 8 & 9\\\\\n",
    " \\end{bmatrix}\n",
    "\\quad\n",
    "5\\times\\mathbf{A}=\\mathbf{C}=?\n",
    "\\qquad\n",
    "\\mathbf{B} = \\begin{bmatrix}\n",
    "  1&-4& 7\\\\\n",
    " \\end{bmatrix}\n",
    " \\quad\n",
    "3\\times\\mathbf{B}=\\mathbf{D}=?\n",
    "$$\n",
    "\n",
    "Utilizing the <a href=\"http://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html\">np.array()</a> function create the above matrix $\\mathbf{A}$ and vector $\\mathbf{B}$ and multiply it by 5 and 3 correspondingly.\n",
    "\n",
    "Note that if you use a Python list of integers to create an array you will get a one-dimensional array, which is, for our purposes, equivalent to a vector.\n",
    "\n",
    "Calculate C and D by inputting the following statements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Replace the <INSERT>. You will use np.array()\n",
    "A = <INSERT>\n",
    "B = <INSERT>\n",
    "C = <INSERT>\n",
    "D = <INSERT>\n",
    "print(C)\n",
    "print(D)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the scalar matrix multiplication is correct\n",
    "checkArray(C,[[5, 10, 15],[20, 25, 30],[35, 40, 45]], \"the scalar multiplication\")\n",
    "checkArray(D,[3, -12,  21], \"the scalar multiplication\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2 Dot product and element-wise multiplication\n",
    "\n",
    "Both dot product and element-wise multiplication is supported by ndarrays.\n",
    "\n",
    "Element-wise multiplication is the standard between two arrays, of the same dimension, using the operator *. \n",
    "\n",
    "The dot product you can use either <a href=\"http://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html#numpy.dot\">np.dot()</a> or <a href=\"http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.dot.html\">np.array.dot()</a>. The dot product is a commutative operation, i.e. the order of the arrays doe not matter, e.g. if you have the ndarrays x and y, you can write the dot product as any of the following four ways: np.dot(x, y), np.dot(y, x), x.dot(y), or y.dot(x).\n",
    "\n",
    "Calculate the element wise product and the dot product by filling in the following statements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "u = np.arange(0, 5)\n",
    "v = np.arange(5, 10)\n",
    "elementWise = <INSERT>\n",
    "dotProduct = <INSERT>\n",
    "print(elementWise)\n",
    "print(dotProduct)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the dot product and element wise is correct\n",
    "checkArray(elementWise,[0,6,14,24,36], \"the element wise multiplication\")\n",
    "check(dotProduct, 80, \"the dot product\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Cosine similarity\n",
    "The cosine similarity between two vectors is defined as the following equation:\n",
    "\n",
    "$$\n",
    "cosine\\_similarity(u,v)=\\cos\\theta=\\frac{\\mathbf{u}\\cdot\\mathbf{v}}{\\|u\\|\\|v\\|}\n",
    "$$\n",
    "\n",
    "The norm of a vector $\\|v\\|$ can be calculated by using <a href=\"http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html#numpy.linalg.norm\">np.linalg.norm()</a>.\n",
    "\n",
    "Implement the following function that calculates the cosine similarity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cosine_similarity(u,v):\n",
    "    dotProduct = <INSERT>\n",
    "    normProduct = <INSERT>\n",
    "    return dotProduct/normProduct\n",
    "\n",
    "u = np.array([2503,2992,1042])\n",
    "v = np.array([2217,2761,990])\n",
    "\n",
    "w = np.array([0,1,1])\n",
    "x = np.array([1,0,1])\n",
    "\n",
    "uv = cosine_similarity(u,v)\n",
    "wx = cosine_similarity(w,x)\n",
    "\n",
    "print(uv)\n",
    "print(wx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the cosine similarity is correct\n",
    "check(round(uv,5),0.99974,\"cosine similarity between u and v\")\n",
    "check(round(wx,5),0.5,\"cosine similarity between w and x\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Matrix math\n",
    "To represent matrices, you can use the following class: <a href=\"http://docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.html\">np.matrix()</a>. To create a matrix object either pass it a two-dimensional ndarray, or a list of lists to the function, or a string e.g. '1 2; 3 4'. Instead of element-wise multiplication, the operator *, does matrix multiplication.\n",
    "\n",
    "To transpose a matrix, you can use either <a href=\"http://docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.transpose.html\">np.matrix.transpose()</a> or <a href=\"http://docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.T.html\">.T</a> on the matrix object.\n",
    "\n",
    "To calculate the inverse of a matrix, you can use <a href=\"http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.inv.html\">np.linalg.inv()</a> or <a href=\"docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.I.htmll\">.I</a> on the matrix object, remember that the inverse of a matrix is only defined on square matrices, and is does not always exist (for sufficient requirements of invertibility look up the: <a href=\"https://en.wikipedia.org/wiki/Invertible_matrix#The_invertible_matrix_theorem\">The invertible matrix theorem</a>) and it will then raise a LinAlgError. If you multiply the original matrix with its inverse, you get the identity matrix, which is a square matrix with ones on the main diagonal and zeros elsewhere., e.g. $\\mathbf{A} \\mathbf{A}^{-1} = \\mathbf{I_n}$\n",
    "\n",
    "In the following exercise, you should calculate $\\mathbf{A}^T$ multiply it by $\\mathbf{A}$ and then inverting the product $\\mathbf{AA}^T$ and finally multiply $\\mathbf{AA}^T[\\mathbf{AA}^T]^{-1}=\\mathbf{I}_n$ to get the identity matrix:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "\n",
    "#We generate a Vandermonde matrix\n",
    "A = np.mat(np.vander([2,3], 5))\n",
    "print(A)\n",
    "\n",
    "#Calculate the transpose of A\n",
    "At = <INSERT>\n",
    "print(At)\n",
    "\n",
    "#Calculate the multiplication of A and A^T\n",
    "AAt = <INSERT>\n",
    "print(AAt)\n",
    "\n",
    "#Calculate the inverse of AA^T\n",
    "AAtInv = <INSERT>\n",
    "print(AAtInv)\n",
    "\n",
    "#Calculate the multiplication of AA^T and (AA^T)^-1\n",
    "I = <INSERT>\n",
    "print(I)\n",
    "\n",
    "#To get the identity matrix we round it because of numerical precision\n",
    "I = I.round(13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Check if the matrix math is correct\n",
    "checkArray(I,[[1.,0.], [0.,1.]], \"the matrix math\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Slices\n",
    "\n",
    "It is possible to select subsets of one-dimensional arrays using <a href=\"http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html\">slices</a>. The basic syntax for slices is $\\mathbf{v}$[i:j:k] where i is the starting index, j is the stopping index, and k is the step ($k\\neq0$), the default value for k, if it is not specified, is 1. If no i is specified, the default value is 0, and if no j is specified, the default value is the end of the array.\n",
    "\n",
    "For example [0,1,2,3,4][:3] = [0,1,2] i.e. the three first elements of the array. You can use negative indices also, for example [0,1,2,3,4][-3:] = [2,3,4] i.e. the three last elements.\n",
    "\n",
    "The following function can be used to concenate 2 or more arrays: <a href=\"http://docs.scipy.org/doc/numpy/reference/generated/numpy.concatenate.html\">np.concatenate</a>, the syntax is np.concatenate((a1, a2, ...)).\n",
    "\n",
    "Slice the following array in 3 pieces and concenate them together to form the original array:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "v = np.arange(1, 9)\n",
    "print(v)\n",
    "#The first two elements of v\n",
    "v1 = <INSERT>\n",
    "\n",
    "#The last two elements of v\n",
    "v3 = <INSERT>\n",
    "\n",
    "#The middle four elements of v\n",
    "v2 = <INSERT>\n",
    "print(v1,v2,v3)\n",
    "#Concatenating the three vectors to get the original array\n",
    "u = np.concatenate((v1, v2, v3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Check if the slices are correct\n",
    "checkArray(u,[1,2,3,4,5,6,7,8], \"the slicing\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6 Stacking\n",
    "There exist many functions provided by the NumPy library to <a href=\"http://docs.scipy.org/doc/numpy/reference/routines.array-manipulation.html\">manipulate</a> existing arrays. We will try out two of these methods <a href=\"docs.scipy.org/doc/numpy/reference/generated/numpy.hstack.html\">np.hstack()</a> which takes two or more arrays and stack them horizontally to make a single array (column wise, equvivalent to np.concatenate), and <a href=\"docs.scipy.org/doc/numpy/reference/generated/numpy.vstack.html\">np.vstack()</a> which takes two or more arrays and stack them vertically (row wise). The syntax is the following np.vstack((a1, a2, ...)).\n",
    "\n",
    "Stack the two following array $\\mathbf{u}$ and $\\mathbf{v}$ to create a 1x20 and a 2x10 array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "u = np.arange(1, 11)\n",
    "v = np.arange(11, 21)\n",
    "\n",
    "#A 1x20 array\n",
    "oneRow = <INSERT>\n",
    "print(oneRow)\n",
    "\n",
    "#A 2x10 array\n",
    "twoRows = <INSERT>\n",
    "print(twoRows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the stacks are correct\n",
    "checkArray(oneRow,[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20], \"the hstack\")\n",
    "checkArray(twoRows,[[1,2,3,4,5,6,7,8,9,10],[11,12,13,14,15,16,17,18,19,20]], \"the vstack\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.7 PySpark's DenseVector\n",
    "In PySpark there exists a <a href=\"https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.linalg.DenseVector\">DenseVector</a> class within the module <a href=\"https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#module-pyspark.mllib.linalg\">pyspark.mllib.linalg</a>. The DenseVector stores the values as a NumPy array and delegates the calculations to this object. You can create a new DenseVector by using DenseVector() and passing it an NumPy array or a Python list.\n",
    "\n",
    "The DenseVector class implements several functions, one important is the dot product, DenseVector.dot(), which operates just like np.ndarray.dot().\n",
    "\n",
    "The DenseVector save all values as np.float64, so even if you pass it an integer vector, the resulting vector will contain floats. Using the DenseVector in a distributed setting, can be done by either passing functions that contain them to resilient distributed dataset (RDD) transformations or by distributing them directly as RDDs.\n",
    "\n",
    "Create the DenseVector $\\mathbf{u}$ containing the 10 elements [0.1,0.2,...,1.0] and the DenseVector $\\mathbf{v}$ containing the 10 elements [1.0,2.0,...,10.0] and calculate the dot product of $\\mathbf{u}$ and $\\mathbf{v}$: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#To use the DenseVector first import it\n",
    "from pyspark.mllib.linalg import DenseVector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "#[0.1,0.2,...,1.0]\n",
    "u = <INSERT>\n",
    "print(u)\n",
    "\n",
    "#[1.0,2.0,...,10.0]\n",
    "v = <INSERT>\n",
    "print(v)\n",
    "\n",
    "#The dot product between u and v\n",
    "dotProduct = <INSERT>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the dense vectors are correct\n",
    "check(dotProduct, 38.5, \"the dense vectors\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}