chapter_attention-mechanisms/attention-cues.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0
   },
   "source": [
    "# Attention Cues\n",
    ":label:`sec_attention-cues`\n",
    "\n",
    "Thank you for your attention\n",
    "to this book.\n",
    "Attention is a scarce resource:\n",
    "at the moment\n",
    "you are reading this book\n",
    "and ignoring the rest.\n",
    "Thus, similar to money,\n",
    "your attention is being paid with an opportunity cost.\n",
    "To ensure that your investment of attention\n",
    "right now is worthwhile,\n",
    "we have been highly motivated to pay our attention carefully\n",
    "to produce a nice book.\n",
    "Attention\n",
    "is the keystone in the arch of life and\n",
    "holds the key to any work's exceptionalism.\n",
    "\n",
    "\n",
    "Since economics studies the allocation of scarce resources,\n",
    "we are\n",
    "in the era of the attention economy,\n",
    "where human attention is treated as a limited, valuable, and scarce commodity\n",
    "that can be exchanged.\n",
    "Numerous business models have been\n",
    "developed to capitalize on it.\n",
    "On music or video streaming services,\n",
    "we either pay attention to their ads\n",
    "or pay money to hide them.\n",
    "For growth in the world of online games,\n",
    "we either pay attention to\n",
    "participate in battles, which attract new gamers,\n",
    "or pay money to instantly become powerful.\n",
    "Nothing comes for free.\n",
    "\n",
    "All in all,\n",
    "information in our environment is not scarce,\n",
    "attention is.\n",
    "When inspecting a visual scene,\n",
    "our optic nerve receives information\n",
    "at the order of $10^8$ bits per second,\n",
    "far exceeding what our brain can fully process.\n",
    "Fortunately,\n",
    "our ancestors had learned from experience (also known as data)\n",
    "that *not all sensory inputs are created equal*.\n",
    "Throughout human history,\n",
    "the capability of directing attention\n",
    "to only a fraction of information of interest\n",
    "has enabled our brain\n",
    "to allocate resources more smartly\n",
    "to survive, to grow, and to socialize,\n",
    "such as detecting predators, preys, and mates.\n",
    "\n",
    "\n",
    "\n",
    "## Attention Cues in Biology\n",
    "\n",
    "To explain how our attention is deployed in the visual world,\n",
    "a two-component framework has emerged\n",
    "and been pervasive.\n",
    "This idea dates back to William James in the 1890s,\n",
    "who is considered the \"father of American psychology\" :cite:`James.2007`.\n",
    "In this framework,\n",
    "subjects selectively direct the spotlight of attention\n",
    "using both the *nonvolitional cue* and *volitional cue*.\n",
    "\n",
    "The nonvolitional cue is based on\n",
    "the saliency and conspicuity of objects in the environment.\n",
    "Imagine there are five objects in front of you:\n",
    "a newspaper, a research paper, a cup of coffee, a notebook, and a book such as in :numref:`fig_eye-coffee`.\n",
    "While all the paper products are printed in black and white,\n",
    "the coffee cup is red.\n",
    "In other words,\n",
    "this coffee is intrinsically salient and conspicuous in\n",
    "this visual environment,\n",
    "automatically and involuntarily drawing attention.\n",
    "So you bring the fovea (the center of the macula where visual acuity is highest) onto the coffee as shown in :numref:`fig_eye-coffee`.\n",
    "\n",
    "![Using the nonvolitional cue based on saliency (red cup, non-paper), attention is involuntarily directed to the coffee.](https://raw.githubusercontent.com/d2l-ai/d2l-en/master/img/eye-coffee.svg)\n",
    ":width:`400px`\n",
    ":label:`fig_eye-coffee`\n",
    "\n",
    "After drinking coffee,\n",
    "you become caffeinated and\n",
    "want to read a book.\n",
    "So you turn your head, refocus your eyes,\n",
    "and look at the book as depicted in :numref:`fig_eye-book`.\n",
    "Different from\n",
    "the case in :numref:`fig_eye-coffee`\n",
    "where the coffee biases you towards\n",
    "selecting based on saliency,\n",
    "in this task-dependent case you select the book under\n",
    "cognitive and volitional control.\n",
    "Using the volitional cue based on variable selection criteria,\n",
    "this form of attention is more deliberate.\n",
    "It is also more powerful with the subject's voluntary effort.\n",
    "\n",
    "![Using the volitional cue (want to read a book) that is task-dependent, attention is directed to the book under volitional control.](https://raw.githubusercontent.com/d2l-ai/d2l-en/master/img/eye-book.svg)\n",
    ":width:`400px`\n",
    ":label:`fig_eye-book`\n",
    "\n",
    "\n",
    "## Queries, Keys, and Values\n",
    "\n",
    "Inspired by the nonvolitional and volitional attention cues that explain the attentional deployment,\n",
    "in the following we will\n",
    "describe a framework for\n",
    "designing attention mechanisms\n",
    "by incorporating these two attention cues.\n",
    "\n",
    "To begin with, consider the simpler case where only\n",
    "nonvolitional cues are available.\n",
    "To bias selection over sensory inputs,\n",
    "we can simply use\n",
    "a parameterized fully-connected layer\n",
    "or even non-parameterized\n",
    "max or average pooling.\n",
    "\n",
    "Therefore,\n",
    "what sets attention mechanisms\n",
    "apart from those fully-connected layers\n",
    "or pooling layers\n",
    "is the inclusion of the volitional cues.\n",
    "In the context of attention mechanisms,\n",
    "we refer to volitional cues as *queries*.\n",
    "Given any query,\n",
    "attention mechanisms\n",
    "bias selection over sensory inputs (e.g., intermediate feature representations)\n",
    "via *attention pooling*.\n",
    "These sensory inputs are called *values* in the context of attention mechanisms.\n",
    "More generally,\n",
    "every value is paired with a *key*,\n",
    "which can be thought of the nonvolitional cue of that sensory input.\n",
    "As shown in :numref:`fig_qkv`,\n",
    "we can design attention pooling\n",
    "so that the given query (volitional cue) can interact with keys (nonvolitional cues),\n",
    "which guides bias selection over values (sensory inputs).\n",
    "\n",
    "![Attention mechanisms bias selection over values (sensory inputs) via attention pooling, which incorporates queries (volitional cues) and keys (nonvolitional cues).](https://raw.githubusercontent.com/d2l-ai/d2l-en/master/img/qkv.svg)\n",
    ":label:`fig_qkv`\n",
    "\n",
    "Note that there are many alternatives for the design of attention mechanisms.\n",
    "For instance,\n",
    "we can design a non-differentiable attention model\n",
    "that can be trained using reinforcement learning methods :cite:`Mnih.Heess.Graves.ea.2014`.\n",
    "Given the dominance of the framework in :numref:`fig_qkv`,\n",
    "models under this framework\n",
    "will be the center of our attention in this chapter.\n",
    "\n",
    "\n",
    "## Visualization of Attention\n",
    "\n",
    "Average pooling\n",
    "can be treated as a weighted average of inputs,\n",
    "where weights are uniform.\n",
    "In practice,\n",
    "attention pooling aggregates values using weighted average, where weights are computed between the given query and different keys.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load ../utils/djl-imports\n",
    "%load ../utils/plot-utils\n",
    "%load ../utils/Functions.java"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NDManager manager = NDManager.newBaseManager();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 4
   },
   "source": [
    "To visualize attention weights,\n",
    "we define the `showHeatmaps` function.\n",
    "Its input `matrices` has the shape (number of rows for display, number of columns for display, number of queries, number of keys).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "public static Figure showHeatmaps(\n",
    "            NDArray matrices,\n",
    "            String xLabel,\n",
    "            String yLabel,\n",
    "            String[] titles,\n",
    "            int width,\n",
    "            int height) {\n",
    "    int numRows = (int) matrices.getShape().get(0);\n",
    "    int numCols = (int) matrices.getShape().get(1);\n",
    "\n",
    "    Trace[] traces = new Trace[numRows * numCols];\n",
    "    int count = 0;\n",
    "    for (int i = 0; i < numRows; i++) {\n",
    "        for (int j = 0; j < numCols; j++) {\n",
    "            NDArray NDMatrix = matrices.get(i).get(j);\n",
    "            double[][] matrix =\n",
    "                    new double[(int) NDMatrix.getShape().get(0)]\n",
    "                            [(int) NDMatrix.getShape().get(1)];\n",
    "            Object[] x = new Object[matrix.length];\n",
    "            Object[] y = new Object[matrix.length];\n",
    "            for (int k = 0; k < NDMatrix.getShape().get(0); k++) {\n",
    "                matrix[k] = Functions.floatToDoubleArray(NDMatrix.get(k).toFloatArray());\n",
    "                x[k] = k;\n",
    "                y[k] = k;\n",
    "            }\n",
    "            HeatmapTrace.HeatmapBuilder builder = HeatmapTrace.builder(x, y, matrix);\n",
    "            if (titles != null) {\n",
    "                builder = (HeatmapTrace.HeatmapBuilder) builder.name(titles[j]);\n",
    "            }\n",
    "            traces[count++] = builder.build();\n",
    "        }\n",
    "    }\n",
    "    Grid grid =\n",
    "            Grid.builder()\n",
    "                    .columns(numCols)\n",
    "                    .rows(numRows)\n",
    "                    .pattern(Grid.Pattern.INDEPENDENT)\n",
    "                    .build();\n",
    "    Layout layout =\n",
    "            Layout.builder()\n",
    "                    .title(\"\")\n",
    "                    .xAxis(Axis.builder().title(xLabel).build())\n",
    "                    .yAxis(Axis.builder().title(yLabel).build())\n",
    "                    .width(width)\n",
    "                    .height(height)\n",
    "                    .grid(grid)\n",
    "                    .build();\n",
    "    return new Figure(layout, traces);\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 6
   },
   "source": [
    "For demonstration,\n",
    "we consider a simple case where\n",
    "the attention weight is one only when the query and the key are the same; otherwise it is zero.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NDArray attentionWeights = manager.eye(10).reshape(new Shape(1, 1, 10, 10));\n",
    "showHeatmaps(attentionWeights, \"Keys\", \"Queries\", null, 700, 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 8
   },
   "source": [
    "In the subsequent sections,\n",
    "we will often invoke this function to visualize attention weights.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Human attention is a limited, valuable, and scarce resource.\n",
    "* Subjects selectively direct attention using both the nonvolitional and volitional cues. The former is based on saliency and the latter is task-dependent.\n",
    "* Attention mechanisms are different from fully-connected layers or pooling layers due to inclusion of the volitional cues.\n",
    "* Attention mechanisms bias selection over values (sensory inputs) via attention pooling, which incorporates queries (volitional cues) and keys (nonvolitional cues). Keys and values are paired.\n",
    "* We can visualize attention weights between queries and keys.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. What can be the volitional cue when decoding a sequence token by token in machine translation? What are the nonvolitional cues and the sensory inputs?\n",
    "1. Randomly generate a $10 \\times 10$ matrix and use the softmax operation to ensure each row is a valid probability distribution. Visualize the output attention weights.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Java",
   "language": "java",
   "name": "java"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "Java",
   "pygments_lexer": "java",
   "version": "14.0.2+12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}