python-sdk/tutorials/automl-with-azureml/automl-nlp-ner/automl-nlp-ner.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/experimental/automl-nlp-ner/automl-nlp-ner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Machine Learning\n",
    "_**Named Entity Recognition Using AutoML NLP**_\n",
    "\n",
    "## Contents\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Setup](#Setup)\n",
    "1. [Data](#Data)\n",
    "1. [Train](#Train)\n",
    "1. [Inference](#Inference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "This notebook demonstrates Named Entity Recognition (NER) with text data using AutoML NLP.\n",
    "\n",
    "AutoML highlights here include using end to end deep learning for NLP tasks like NER.\n",
    "\n",
    "Make sure you have executed the [configuration](../../../configuration.ipynb) before running this notebook.\n",
    "\n",
    "Notebook synopsis:\n",
    "\n",
    "1. Creating an Experiment in an existing Workspace\n",
    "2. Configuration and remote run of AutoML for CoNLL 2003 dataset for NER task\n",
    "3. Evaluating the trained model on a test set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import tempfile\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "import azureml.core\n",
    "from azureml.core import Dataset\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.core.dataset import Dataset\n",
    "from azureml.core.compute import AmlCompute\n",
    "from azureml.core.compute import ComputeTarget\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "from azureml.data.datapath import DataPath\n",
    "from azureml.core.run import Run\n",
    "from azureml.core.script_run_config import ScriptRunConfig\n",
    "from azureml.train.automl import AutoMLConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"This notebook was created using version 1.39.0 of the Azure ML SDK\")\n",
    "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As part of the setup you have already created a <b>Workspace</b>. To run AutoML, you also need to create an <b>Experiment</b>. An Experiment corresponds to a prediction problem you are trying to solve, while a Run corresponds to a specific approach to the problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()\n",
    "\n",
    "# Choose an experiment name.\n",
    "experiment_name = \"automl-nlp-text-ner\"\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "\n",
    "output = {}\n",
    "output[\"Subscription ID\"] = ws.subscription_id\n",
    "output[\"Workspace Name\"] = ws.name\n",
    "output[\"Resource Group\"] = ws.resource_group\n",
    "output[\"Location\"] = ws.location\n",
    "output[\"Experiment Name\"] = experiment.name\n",
    "pd.set_option(\"display.max_colwidth\", None)\n",
    "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
    "outputDf.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up a compute cluster\n",
    "This section uses a user-provided compute cluster (named \"gpu-compute\" in this example). If a cluster with this name does not exist in the user's workspace, the below code will create a new cluster. You can choose the parameters of the cluster as mentioned in the comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_nodes = 1\n",
    "\n",
    "# Choose a name for your cluster.\n",
    "amlcompute_cluster_name = \"gpu-compute\"\n",
    "\n",
    "# Verify that cluster does not exist already\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
    "    print(\"Found existing cluster, use it.\")\n",
    "except ComputeTargetException:\n",
    "    compute_config = AmlCompute.provisioning_configuration(\n",
    "        vm_size=\"STANDARD_NC6\", max_nodes=num_nodes  # Use GPU only\n",
    "    )\n",
    "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
    "\n",
    "compute_target.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload dataset to datastore\n",
    "data_dir = \"data\"  # Local directory to store data\n",
    "blobstore_datadir = data_dir  # Blob store directory to store data in\n",
    "\n",
    "datastore = ws.get_default_datastore()\n",
    "target = DataPath(datastore=datastore, path_on_datastore=blobstore_datadir)\n",
    "Dataset.File.upload_directory(\n",
    "    src_dir=data_dir, target=target, overwrite=True, show_progress=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datastore_path = [(datastore, blobstore_datadir + \"/train.txt\")]\n",
    "train_data = Dataset.File.from_files(path=datastore_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datastore_path = [(datastore, blobstore_datadir + \"/dev.txt\")]\n",
    "val_data = Dataset.File.from_files(path=datastore_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = train_data.register(\n",
    "    workspace=ws,\n",
    "    name=\"CoNLL_2003_train\",\n",
    "    description=\"NER train data\",\n",
    "    create_new_version=True,\n",
    ")\n",
    "\n",
    "val_data = val_data.register(\n",
    "    workspace=ws,\n",
    "    name=\"CoNLL_2003_val\",\n",
    "    description=\"NER val data\",\n",
    "    create_new_version=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train\n",
    "\n",
    "## Submit AutoML run\n",
    "\n",
    "Here we do not set `primary_metric` parameter as we only train one model and we do not need to rank trained models. The run will use default primary metrics, `accuracy`. But it is only for reporting purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"verbosity\": logging.INFO,\n",
    "}\n",
    "\n",
    "automl_config = AutoMLConfig(\n",
    "    task=\"text-ner\",\n",
    "    debug_log=\"automl_errors.log\",\n",
    "    compute_target=compute_target,\n",
    "    training_data=train_data,\n",
    "    validation_data=val_data,\n",
    "    **automl_settings\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Submit AutoML Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_run = experiment.submit(automl_config, show_output=False)\n",
    "_ = automl_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download Metrics\n",
    "\n",
    "These metrics logged with the training run are computed with the trained model on validation dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_metrics = automl_run.get_metrics()\n",
    "pd.DataFrame(\n",
    "    {\"metric_name\": validation_metrics.keys(), \"value\": validation_metrics.values()}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also get the best run id and the best model with `get_output` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run, best_model = automl_run.get_output()\n",
    "best_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference\n",
    "\n",
    "Now you can use the trained model to do inference on unseen data. We use a `ScriptRun` to do this, with script that we provide. The following blocks will register the test dataset, download the inference script and trigger the inference run. Unlink multiclass or multilabel scenario, the inference runs for NER saves the evaluation metrics. So we do not have to download the predictions, but directly get the metrics.\n",
    "\n",
    "## Submit Inference Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datastore_path = [(datastore, blobstore_datadir + \"/test.txt\")]\n",
    "test_data = Dataset.File.from_files(path=datastore_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data = test_data.register(\n",
    "    workspace=ws, name=\"CoNLL_2003_test\", description=\"NER test data\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load training script run corresponding to AutoML run above.\n",
    "training_run_id = best_run.id\n",
    "training_run = Run(experiment, training_run_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inference script run arguments\n",
    "arguments = [\n",
    "    \"--run_id\",\n",
    "    training_run_id,\n",
    "    \"--experiment_name\",\n",
    "    experiment.name,\n",
    "    \"--input_dataset_id\",\n",
    "    test_data.as_named_input(\"test_data\"),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_args = arguments\n",
    "with tempfile.TemporaryDirectory() as tmpdir:\n",
    "    # Download required files from training run into temp folder.\n",
    "    entry_script_name = \"score_script.py\"\n",
    "    output_path = os.path.join(tmpdir, entry_script_name)\n",
    "    training_run.download_file(\n",
    "        \"outputs/\" + entry_script_name, os.path.join(tmpdir, entry_script_name)\n",
    "    )\n",
    "\n",
    "    script_run_config = ScriptRunConfig(\n",
    "        source_directory=tmpdir,\n",
    "        script=entry_script_name,\n",
    "        compute_target=compute_target,\n",
    "        environment=training_run.get_environment(),\n",
    "        arguments=scoring_args,\n",
    "    )\n",
    "    scoring_run = experiment.submit(script_run_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = scoring_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Evaluation Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_metrics = scoring_run.get_metrics()\n",
    "test_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(\n",
    "    {\"metric name\": list(test_metrics.keys()), \"value\": list(test_metrics.values())}\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "anshirga"
   }
  ],
  "compute": [
   "AML Compute"
  ],
  "datasets": [
   "None"
  ],
  "deployment": [
   "None"
  ],
  "exclude_from_index": false,
  "framework": [
   "None"
  ],
  "friendly_name": "DNN Text Featurization",
  "index_order": 2,
  "interpreter": {
   "hash": "cc0892e042a269bcf4aec58f0c86eb5e2be478ff7be4e5f6b2680e2af1718f2e"
  },
  "kernelspec": {
   "display_name": "Python 3.8 - AzureML",
   "language": "python",
   "name": "python38-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "tags": [
   "None"
  ],
  "task": "Text featurization using DNNs for classification"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}