chapter_attention-mechanisms/nadaraya-watson.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0
   },
   "source": [
    "# Attention Pooling: Nadaraya-Watson Kernel Regression\n",
    ":label:`sec_nadaraya-watson`\n",
    "\n",
    "Now you know the major components of attention mechanisms under the framework in :numref:`fig_qkv`.\n",
    "To recapitulate,\n",
    "the interactions between\n",
    "queries (volitional cues) and keys (nonvolitional cues)\n",
    "result in *attention pooling*.\n",
    "The attention pooling selectively aggregates values (sensory inputs) to produce the output.\n",
    "In this section,\n",
    "we will describe attention pooling in greater detail\n",
    "to give you a high-level view of\n",
    "how attention mechanisms work in practice.\n",
    "Specifically,\n",
    "the Nadaraya-Watson kernel regression model\n",
    "proposed in 1964\n",
    "is a simple yet complete example\n",
    "for demonstrating machine learning with attention mechanisms.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load ../utils/djl-imports\n",
    "%load ../utils/plot-utils\n",
    "%load ../utils/Functions.java\n",
    "%load ../utils/Animator.java\n",
    "%load ../utils/PlotUtils.java"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NDManager manager = NDManager.newBaseManager();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 3
   },
   "source": [
    "## Generating the Dataset\n",
    "\n",
    "To keep things simple,\n",
    "let us consider the following regression problem:\n",
    "given a dataset of input-output pairs $\\{(x_1, y_1), \\ldots, (x_n, y_n)\\}$,\n",
    "how to learn $f$ to predict the output $\\hat{y} = f(x)$ for any new input $x$?\n",
    "\n",
    "Here we generate an artificial dataset according to the following nonlinear function with the noise term $\\epsilon$:\n",
    "\n",
    "$$y_i = 2\\sin(x_i) + x_i^{0.8} + \\epsilon,$$\n",
    "\n",
    "where $\\epsilon$ obeys a normal distribution with zero mean and standard deviation 0.5.\n",
    "Both 50 training examples and 50 testing examples\n",
    "are generated.\n",
    "To better visualize the pattern of attention later, the training inputs are sorted.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "int nTrain = 50; // No. of training examples\n",
    "NDArray xTrain = manager.randomUniform(0, 1, new Shape(nTrain)).mul(5).sort(); // Training inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Function<NDArray, NDArray> f = x -> x.sin().mul(2).add(x.pow(0.8));\n",
    "NDArray yTrain =\n",
    "        f.apply(xTrain)\n",
    "                .add(\n",
    "                        manager.randomNormal(\n",
    "                                0f,\n",
    "                                0.5f,\n",
    "                                new Shape(nTrain),\n",
    "                                DataType.FLOAT32)); // Training outputs\n",
    "NDArray xTest = manager.arange(0f, 5f, 0.1f); // Testing examples\n",
    "NDArray yTruth = f.apply(xTest); // Ground-truth outputs for the testing examples\n",
    "int nTest = (int) xTest.getShape().get(0); // No. of testing examples\n",
    "System.out.println(nTest);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 7
   },
   "source": [
    "The following function plots all the training examples (represented by circles),\n",
    "the ground-truth data generation function `f` without the noise term (labeled by \"Truth\"), and the learned prediction function (labeled by \"Pred\").\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "public static Figure plot(\n",
    "        NDArray yHat,\n",
    "        String trace1Name,\n",
    "        String trace2Name,\n",
    "        String xLabel,\n",
    "        String yLabel,\n",
    "        int width,\n",
    "        int height) {\n",
    "    ScatterTrace trace =\n",
    "            ScatterTrace.builder(\n",
    "                            Functions.floatToDoubleArray(xTest.toFloatArray()),\n",
    "                            Functions.floatToDoubleArray(yTruth.toFloatArray()))\n",
    "                    .mode(ScatterTrace.Mode.LINE)\n",
    "                    .name(trace1Name)\n",
    "                    .build();\n",
    "\n",
    "    ScatterTrace trace2 =\n",
    "            ScatterTrace.builder(\n",
    "                            Functions.floatToDoubleArray(xTest.toFloatArray()),\n",
    "                            Functions.floatToDoubleArray(yHat.toFloatArray()))\n",
    "                    .mode(ScatterTrace.Mode.LINE)\n",
    "                    .name(trace2Name)\n",
    "                    .build();\n",
    "\n",
    "    ScatterTrace trace3 =\n",
    "            ScatterTrace.builder(\n",
    "                            Functions.floatToDoubleArray(xTrain.toFloatArray()),\n",
    "                            Functions.floatToDoubleArray(yTrain.toFloatArray()))\n",
    "                    .mode(ScatterTrace.Mode.MARKERS)\n",
    "                    .marker(Marker.builder().symbol(Symbol.CIRCLE).size(15).opacity(.5).build())\n",
    "                    .build();\n",
    "\n",
    "    Layout layout =\n",
    "            Layout.builder()\n",
    "                    .height(height)\n",
    "                    .width(width)\n",
    "                    .showLegend(true)\n",
    "                    .xAxis(Axis.builder().title(xLabel).domain(0, 5).build())\n",
    "                    .yAxis(Axis.builder().title(yLabel).domain(-1, 5).build())\n",
    "                    .build();\n",
    "\n",
    "    return new Figure(layout, trace, trace2, trace3);\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 9
   },
   "source": [
    "## Average Pooling\n",
    "\n",
    "We begin with perhaps the world's \"dumbest\" estimator for this regression problem:\n",
    "using average pooling to average over all the training outputs:\n",
    "\n",
    "$$f(x) = \\frac{1}{n}\\sum_{i=1}^n y_i,$$\n",
    ":eqlabel:`eq_avg-pooling`\n",
    "\n",
    "which is plotted below. As we can see, this estimator is indeed not so smart.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NDArray yHat = yTrain.mean().tile(nTest);\n",
    "plot(yHat, \"Truth\", \"Pred\", \"x\", \"y\", 700, 500);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 12
   },
   "source": [
    "## Nonparametric Attention Pooling\n",
    "\n",
    "Obviously,\n",
    "average pooling omits the inputs $x_i$.\n",
    "A better idea was proposed\n",
    "by Nadaraya :cite:`Nadaraya.1964`\n",
    "and Watson :cite:`Watson.1964`\n",
    "to weigh the outputs $y_i$ according to their input locations:\n",
    "\n",
    "$$f(x) = \\sum_{i=1}^n \\frac{K(x - x_i)}{\\sum_{j=1}^n K(x - x_j)} y_i,$$\n",
    ":eqlabel:`eq_nadaraya-watson`\n",
    "\n",
    "where $K$ is a *kernel*.\n",
    "The estimator in :eqref:`eq_nadaraya-watson`\n",
    "is called *Nadaraya-Watson kernel regression*.\n",
    "Here we will not dive into details of kernels.\n",
    "Recall the framework of attention mechanisms in :numref:`fig_qkv`.\n",
    "From the perspective of attention,\n",
    "we can rewrite :eqref:`eq_nadaraya-watson`\n",
    "in a more generalized form of *attention pooling*:\n",
    "\n",
    "$$f(x) = \\sum_{i=1}^n \\alpha(x, x_i) y_i,$$\n",
    ":eqlabel:`eq_attn-pooling`\n",
    "\n",
    "\n",
    "where $x$ is the query and $(x_i, y_i)$ is the key-value pair.\n",
    "Comparing :eqref:`eq_attn-pooling` and :eqref:`eq_avg-pooling`,\n",
    "the attention pooling here\n",
    "is a weighted average of values $y_i$.\n",
    "The *attention weight* $\\alpha(x, x_i)$\n",
    "in :eqref:`eq_attn-pooling`\n",
    "is assigned to the corresponding value $y_i$\n",
    "based on the interaction\n",
    "between the query $x$ and the key $x_i$\n",
    "modeled by $\\alpha$.\n",
    "For any query, its attention weights over all the key-value pairs are a valid probability distribution: they are non-negative and sum up to one.\n",
    "\n",
    "To gain intuitions of attention pooling,\n",
    "just consider a *Gaussian kernel* defined as\n",
    "\n",
    "$$\n",
    "K(u) = \\frac{1}{\\sqrt{2\\pi}} \\exp(-\\frac{u^2}{2}).\n",
    "$$\n",
    "\n",
    "\n",
    "Plugging the Gaussian kernel into\n",
    ":eqref:`eq_attn-pooling` and\n",
    ":eqref:`eq_nadaraya-watson` gives\n",
    "\n",
    "$$\\begin{aligned} f(x) &=\\sum_{i=1}^n \\alpha(x, x_i) y_i\\\\ &= \\sum_{i=1}^n \\frac{\\exp\\left(-\\frac{1}{2}(x - x_i)^2\\right)}{\\sum_{j=1}^n \\exp\\left(-\\frac{1}{2}(x - x_j)^2\\right)} y_i \\\\&= \\sum_{i=1}^n \\mathrm{softmax}\\left(-\\frac{1}{2}(x - x_i)^2\\right) y_i. \\end{aligned}$$\n",
    ":eqlabel:`eq_nadaraya-watson-gaussian`\n",
    "\n",
    "In :eqref:`eq_nadaraya-watson-gaussian`,\n",
    "a key $x_i$ that is closer to the given query $x$ will get\n",
    "*more attention* via a *larger attention weight* assigned to the key's corresponding value $y_i$.\n",
    "\n",
    "Notably, Nadaraya-Watson kernel regression is a nonparametric model;\n",
    "thus :eqref:`eq_nadaraya-watson-gaussian`\n",
    "is an example of *nonparametric attention pooling*.\n",
    "In the following, we plot the prediction based on this\n",
    "nonparametric attention model.\n",
    "The predicted line is smooth and closer to the ground-truth than that produced by average pooling.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "// Shape of `xRepeat`: (`nTest`, `nTrain`), where each row contains the\n",
    "// same testing inputs (i.e., same queries)\n",
    "NDArray xRepeat = xTest.repeat(nTrain).reshape(new Shape(-1, nTrain));\n",
    "// Note that `xTrain` contains the keys. Shape of `attention_weights`:\n",
    "// (`nTest`, `nTrain`), where each row contains attention weights to be\n",
    "// assigned among the values (`yTrain`) given each query\n",
    "NDArray attentionWeights = xRepeat.sub(xTrain).pow(2).div(2).mul(-1).softmax(-1);\n",
    "// Each element of `yHat` is weighted average of values, where weights are\n",
    "// attention weights\n",
    "yHat = attentionWeights.dot(yTrain);\n",
    "plot(yHat, \"Truth\", \"Pred\", \"x\", \"y\", 700, 500);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 15
   },
   "source": [
    "Now let us take a look at the attention weights.\n",
    "Here testing inputs are queries while training inputs are keys.\n",
    "Since both inputs are sorted,\n",
    "we can see that the closer the query-key pair is,\n",
    "the higher attention weight is in the attention pooling.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PlotUtils.showHeatmaps(\n",
    "        attentionWeights.expandDims(0).expandDims(0),\n",
    "        \"Sorted training inputs\",\n",
    "        \"Sorted testing inputs\",\n",
    "        new String[] {\"\"},\n",
    "        500,\n",
    "        700);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 18
   },
   "source": [
    "## Parametric Attention Pooling\n",
    "\n",
    "Nonparametric Nadaraya-Watson kernel regression\n",
    "enjoys the *consistency* benefit:\n",
    "given enough data this model converges to the optimal solution.\n",
    "Nonetheless,\n",
    "we can easily integrate learnable parameters into attention pooling.\n",
    "\n",
    "As an example, slightly different from :eqref:`eq_nadaraya-watson-gaussian`,\n",
    "in the following\n",
    "the distance between the query $x$ and the key $x_i$\n",
    "is multiplied a learnable parameter $w$:\n",
    "\n",
    "\n",
    "$$\\begin{aligned}f(x) &= \\sum_{i=1}^n \\alpha(x, x_i) y_i \\\\&= \\sum_{i=1}^n \\frac{\\exp\\left(-\\frac{1}{2}((x - x_i)w)^2\\right)}{\\sum_{j=1}^n \\exp\\left(-\\frac{1}{2}((x - x_i)w)^2\\right)} y_i \\\\&= \\sum_{i=1}^n \\mathrm{softmax}\\left(-\\frac{1}{2}((x - x_i)w)^2\\right) y_i.\\end{aligned}$$\n",
    ":eqlabel:`eq_nadaraya-watson-gaussian-para`\n",
    "\n",
    "In the rest of the section,\n",
    "we will train this model by learning the parameter of\n",
    "the attention pooling in :eqref:`eq_nadaraya-watson-gaussian-para`.\n",
    "\n",
    "\n",
    "### Batch Matrix Multiplication\n",
    ":label:`subsec_batch_dot`\n",
    "\n",
    "To more efficiently compute attention\n",
    "for minibatches,\n",
    "we can leverage batch matrix multiplication utilities\n",
    "provided by deep learning frameworks.\n",
    "\n",
    "\n",
    "Suppose that the first minibatch contains $n$ matrices $\\mathbf{X}_1, \\ldots, \\mathbf{X}_n$ of shape $a\\times b$, and the second minibatch contains $n$ matrices $\\mathbf{Y}_1, \\ldots, \\mathbf{Y}_n$ of shape $b\\times c$. Their batch matrix multiplication\n",
    "results in\n",
    "$n$ matrices $\\mathbf{X}_1\\mathbf{Y}_1, \\ldots, \\mathbf{X}_n\\mathbf{Y}_n$ of shape $a\\times c$. Therefore, given two tensors of shape ($n$, $a$, $b$) and ($n$, $b$, $c$), the shape of their batch matrix multiplication output is ($n$, $a$, $c$).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NDArray X = manager.ones(new Shape(2, 1, 4));\n",
    "NDArray Y = manager.ones(new Shape(2, 4, 6));\n",
    "\n",
    "X.batchDot(Y).getShape()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 21
   },
   "source": [
    "In the context of attention mechanisms, we can use minibatch matrix multiplication to compute weighted averages of values in a minibatch.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NDArray weights = manager.ones(new Shape(2, 10)).mul(0.1);\n",
    "NDArray values = manager.arange(20f).reshape(new Shape(2, 10));\n",
    "weights.expandDims(1).batchDot(values.expandDims(-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 24
   },
   "source": [
    "### Defining the Model\n",
    "\n",
    "Using minibatch matrix multiplication,\n",
    "below we define the parametric version\n",
    "of Nadaraya-Watson kernel regression\n",
    "based on the parametric attention pooling in\n",
    ":eqref:`eq_nadaraya-watson-gaussian-para`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "public class NWKernelRegression extends AbstractBlock {\n",
    "\n",
    "    private Parameter w;\n",
    "    public NDArray attentionWeights;\n",
    "\n",
    "    public NWKernelRegression() {\n",
    "        w = Parameter.builder().optShape(new Shape(1)).setName(\"w\").optInitializer(new UniformInitializer()).build();\n",
    "        addParameter(w);\n",
    "    }\n",
    "\n",
    "    @Override\n",
    "    protected NDList forwardInternal(\n",
    "            ParameterStore parameterStore,\n",
    "            NDList inputs,\n",
    "            boolean training,\n",
    "            PairList<String, Object> params) {\n",
    "        // Shape of the output `queries` and `attentionWeights`:\n",
    "        // (no. of queries, no. of key-value pairs)\n",
    "        NDArray queries = inputs.get(0);\n",
    "        NDArray keys = inputs.get(1);\n",
    "        NDArray values = inputs.get(2);\n",
    "        queries =\n",
    "                queries.repeat(keys.getShape().get(1))\n",
    "                        .reshape(new Shape(-1, keys.getShape().get(1)));\n",
    "\n",
    "        this.attentionWeights =\n",
    "                queries.sub(keys).mul(this.w.getArray()).pow(2).div(2).mul(-1).softmax(-1);\n",
    "        // Shape of `values`: (no. of queries, no. of key-value pairs)\n",
    "        return new NDList(\n",
    "                attentionWeights.expandDims(1).batchDot(values.expandDims(-1)).reshape(-1));\n",
    "    }\n",
    "\n",
    "    @Override\n",
    "    public Shape[] getOutputShapes(Shape[] inputShapes) {\n",
    "        throw new UnsupportedOperationException(\"Not implemented\");\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 27
   },
   "source": [
    "### Training\n",
    "\n",
    "In the following, we transform the training dataset\n",
    "to keys and values to train the attention model.\n",
    "In the parametric attention pooling,\n",
    "any training input takes key-value pairs from all the training examples except for itself to predict its output.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "// Shape of `xTile`: (`nTrain`, `nTrain`), where each column contains the\n",
    "// same training inputs\n",
    "NDArray xTile = xTrain.tile(new long[] {nTrain, 1});\n",
    "// Shape of `Y_tile`: (`nTrain`, `nTrain`), where each column contains the\n",
    "// same training outputs\n",
    "NDArray yTile = yTrain.tile(new long[] {nTrain, 1});\n",
    "// Shape of `keys`: ('nTrain', 'nTrain' - 1)\n",
    "NDArray keys =\n",
    "        xTile.get(new NDIndex().addBooleanIndex(manager.eye(nTrain).mul(-1).add(1))).reshape(new Shape(nTrain, -1));\n",
    "// Shape of `values`: ('nTrain', 'nTrain' - 1)\n",
    "values = yTile.get(new NDIndex().addBooleanIndex(manager.eye(nTrain).mul(-1).add(1))).reshape(new Shape(nTrain, -1));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 30
   },
   "source": [
    "Using the squared loss and stochastic gradient descent,\n",
    "we train the parametric attention model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NWKernelRegression net = new NWKernelRegression();\n",
    "Loss loss = Loss.l2Loss();\n",
    "Tracker lrt =\n",
    "        Tracker.fixed(0.5f * nTrain); // Since we are using sgd, to be able to put the right\n",
    "                                      // scale, we need to multiply by batchSize\n",
    "Optimizer sgd = Optimizer.sgd().setLearningRateTracker(lrt).build();\n",
    "DefaultTrainingConfig config =\n",
    "        new DefaultTrainingConfig(loss)\n",
    "                .optOptimizer(sgd) // Optimizer (loss function)\n",
    "                .addTrainingListeners(TrainingListener.Defaults.logging()); // Logging\n",
    "Model model = Model.newInstance(\"\");\n",
    "model.setBlock(net);\n",
    "\n",
    "Trainer trainer = model.newTrainer(config);\n",
    "Animator animator = new Animator();\n",
    "ParameterStore ps = new ParameterStore(manager, false);\n",
    "\n",
    "\n",
    "// Create trainer and animator\n",
    "for (int epoch = 0; epoch < 5; epoch++) {\n",
    "    try (GradientCollector gc = trainer.newGradientCollector()) {\n",
    "        NDArray result = net.forward(ps, new NDList(xTrain, keys, values), true).get(0);\n",
    "        NDArray l = trainer.getLoss().evaluate(new NDList(yTrain), new NDList(result));\n",
    "\n",
    "        gc.backward(l);\n",
    "        animator.add(epoch + 1, (float) l.getFloat(), \"Loss\");\n",
    "        animator.show();\n",
    "    }\n",
    "    trainer.step();\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 33
   },
   "source": [
    "After training the parametric attention model,\n",
    "we can plot its prediction.\n",
    "Trying to fit the training dataset with noise,\n",
    "the predicted line is less smooth\n",
    "than its nonparametric counterpart that was plotted earlier.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "// Shape of `keys`: (`nTest`, `nTrain`), where each column contains the same\n",
    "// training inputs (i.e., same keys)\n",
    "keys = xTrain.tile(new long[] {nTest, 1});\n",
    "\n",
    "// Shape of `value`: (`nTest`, `nTrain`)\n",
    "values = yTrain.tile(new long[] {nTest, 1});\n",
    "yHat = net.forward(ps, new NDList(xTest, keys, values), true).get(0);\n",
    "plot(yHat, \"Truth\", \"Pred\", \"x\", \"y\", 700, 500);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 36
   },
   "source": [
    "Comparing with nonparametric attention pooling,\n",
    "the region with large attention weights becomes sharper\n",
    "in the learnable and parametric setting.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PlotUtils.showHeatmaps(\n",
    "        net.attentionWeights.expandDims(0).expandDims(0),\n",
    "        \"Sorted training inputs\",\n",
    "        \"Sorted testing inputs\",\n",
    "        new String[] {\"\"},\n",
    "        500,\n",
    "        700);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 39
   },
   "source": [
    "## Summary\n",
    "\n",
    "* Nadaraya-Watson kernel regression is an example of machine learning with attention mechanisms.\n",
    "* The attention pooling of Nadaraya-Watson kernel regression is a weighted average of the training outputs. From the attention perspective, the attention weight is assigned to a value based on a function of a query and the key that is paired with the value.\n",
    "* Attention pooling can be either nonparametric or parametric.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Increase the number of training examples. Can you learn  nonparametric Nadaraya-Watson kernel regression better?\n",
    "1. What is the value of our learned $w$ in the parametric attention pooling experiment? Why does it make the weighted region sharper when visualizing the attention weights?\n",
    "1. How can we add hyperparameters to nonparametric Nadaraya-Watson kernel regression to predict better?\n",
    "1. Design another parametric attention pooling for the kernel regression of this section. Train this new model and visualize its attention weights.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Java",
   "language": "java",
   "name": "java"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "Java",
   "pygments_lexer": "java",
   "version": "14.0.2+12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}