sdk/jobs/pipelines/1h_automl_in_pipeline/automl-image-instance-segmentation-in-pipeline/automl-image-instance-segmentation-in-pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AutoML Image Instance Segmentation in pipeline\n",
    "\n",
    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
    "- A basic understanding of Machine Learning\n",
    "- An Azure account with an active subscription - [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
    "- An Azure ML workspace with computer cluster - [Configure workspace](../../configuration.ipynb)\n",
    "- A python environment\n",
    "- Installed Azure Machine Learning Python SDK v2 - [install instructions](../../../README.md) - check the getting started section\n",
    "\n",
    "**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
    "- Create a pipeline with Image Instance Segmentation AutoML task.\n",
    "\n",
    "**Motivations** - This notebook explains how to use Image Instance Segmentation AutoML task inside pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Connect to Azure Machine Learning Workspace\n",
    "\n",
    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n",
    "\n",
    "## 1.1 Import the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import required libraries\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "from azure.ai.ml import MLClient, Input, command, Output\n",
    "from azure.ai.ml.automl import (\n",
    "    image_instance_segmentation,\n",
    "    ImageObjectDetectionSearchSpace,\n",
    ")\n",
    "from azure.ai.ml.dsl import pipeline\n",
    "from azure.ai.ml.sweep import BanditPolicy, Choice, Uniform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2. Configure workspace details and get a handle to the workspace\n",
    "\n",
    "To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](../../configuration.ipynb) for more details on how to configure credentials and connect to a workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "credential = DefaultAzureCredential()\n",
    "ml_client = None\n",
    "try:\n",
    "    ml_client = MLClient.from_config(credential)\n",
    "except Exception as ex:\n",
    "    print(ex)\n",
    "    # Enter details of your AML workspace\n",
    "    subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "    resource_group = \"<RESOURCE_GROUP>\"\n",
    "    workspace = \"<AML_WORKSPACE_NAME>\"\n",
    "    ml_client = MLClient(credential, subscription_id, resource_group, workspace)\n",
    "print(ml_client)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Download Data\n",
    "\n",
    "Load the 'fridge items' dataset from a JSON file and MLTable definition.\n",
    "\n",
    "In order to generate models for computer vision, you will need to bring in labeled image data as input for model training in the form of an Azure Machine Learning MLTable. \n",
    "\n",
    "In this notebook, we use a toy dataset called Fridge Objects, which consists of 134 images of 4 classes of beverage container {can, carton, milk bottle, water bottle} photos taken on different backgrounds.\n",
    "\n",
    "All images in this notebook are hosted in [this repository](https://github.com/microsoft/computervision-recipes) and are made available under the [MIT license](https://github.com/microsoft/computervision-recipes/blob/master/LICENSE).\n",
    "\n",
    "**NOTE:** In this PRIVATE PREVIEW we're defining the MLTable in a separate folder and .YAML file.\n",
    "In later versions, you'll be able to do it all in Python APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib\n",
    "from zipfile import ZipFile\n",
    "\n",
    "# download data\n",
    "download_url = \"https://cvbp-secondary.z19.web.core.windows.net/datasets/object_detection/odFridgeObjectsMask.zip\"\n",
    "data_file = \"./data/odFridgeObjectsMask.zip\"\n",
    "urllib.request.urlretrieve(download_url, filename=data_file)\n",
    "\n",
    "# extract files\n",
    "with ZipFile(data_file, \"r\") as zip:\n",
    "    print(\"extracting files...\")\n",
    "    zip.extractall(path=\"./data\")\n",
    "    print(\"done\")\n",
    "# delete zip file\n",
    "os.remove(data_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a sample image from this dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "\n",
    "sample_image = \"./data/odFridgeObjectsMask/images/31.jpg\"\n",
    "Image(filename=sample_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload the images to Datastore through an AML Data asset (URI Folder)\n",
    "\n",
    "In order to use the data for training in Azure ML, we upload it to our default Azure Blob Storage of our  Azure ML Workspace.\n",
    "\n",
    "Reference to URI FOLDER data asset example for further details: https://github.com/Azure/azureml-examples/blob/samuel100/data-samples/sdk/assets/data/data.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uploading image files by creating a 'data asset URI FOLDER':\n",
    "\n",
    "from azure.ai.ml.entities import Data\n",
    "from azure.ai.ml.constants import AssetTypes\n",
    "\n",
    "my_data = Data(\n",
    "    path=\"./data/odFridgeObjectsMask\",\n",
    "    type=AssetTypes.URI_FOLDER,\n",
    "    description=\"Fridge-items images instance segmentation\",\n",
    "    name=\"fridge-items-images-instance-segmentation\",\n",
    ")\n",
    "\n",
    "uri_folder_data_asset = ml_client.data.create_or_update(my_data)\n",
    "\n",
    "print(uri_folder_data_asset)\n",
    "print(\"\")\n",
    "print(\"Path to folder in Blob Storage:\")\n",
    "print(uri_folder_data_asset.path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The jsonl_converter below relies on scikit-image and simplification.\n",
    "# If you don't have them installed, install them before converting data by runing this cell.\n",
    "%pip install \"scikit-image==0.17.2\" \"simplification\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from jsonl_converter import convert_mask_in_VOC_to_jsonl\n",
    "\n",
    "data_path = \"./data/odFridgeObjectsMask/\"\n",
    "convert_mask_in_VOC_to_jsonl(data_path, uri_folder_data_asset.path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Basic pipeline job with Image Instance Segmentation task\n",
    "\n",
    "## 2.1 Build pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note that the used docker image doesn't suit for all size of gpu compute. Please use the following command to create gpu compute if experiment failed\n",
    "# !az ml compute create -n gpu-cluster --type amlcompute --min-instances 0 --max-instances 4 --size Standard_NC12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define pipeline\n",
    "@pipeline(\n",
    "    description=\"AutoML Image Instance Segmentation Pipeline\",\n",
    ")\n",
    "def automl_image_instance_segmentation(\n",
    "    image_instance_segmentation_train_data, image_instance_segmentation_validation_data\n",
    "):\n",
    "    # define the automl image_instance_segmentation task with automl function\n",
    "    image_instance_segmentation_node = image_instance_segmentation(\n",
    "        training_data=image_instance_segmentation_train_data,\n",
    "        validation_data=image_instance_segmentation_validation_data,\n",
    "        target_column_name=\"label\",\n",
    "        primary_metric=\"MeanAveragePrecision\",\n",
    "        # currently need to specify outputs \"mlflow_model\" explictly to reference it in following nodes\n",
    "        outputs={\"best_model\": Output(type=\"mlflow_model\")},\n",
    "    )\n",
    "    image_instance_segmentation_node.set_limits(timeout_minutes=180)\n",
    "    image_instance_segmentation_node.set_image_model(nms_iou_threshold=0.7)\n",
    "    image_instance_segmentation_node.extend_search_space(\n",
    "        [\n",
    "            ImageObjectDetectionSearchSpace(\n",
    "                model_name=Choice([\"maskrcnn_resnet50_fpn\"]),\n",
    "                learning_rate=Uniform(0.0001, 0.001),\n",
    "                optimizer=Choice([\"sgd\", \"adam\", \"adamw\"]),\n",
    "                min_size=Choice([600, 800]),\n",
    "            ),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    image_instance_segmentation_node.set_sweep(\n",
    "        max_trials=10,\n",
    "        max_concurrent_trials=2,\n",
    "        sampling_algorithm=\"Random\",\n",
    "        early_termination=BanditPolicy(\n",
    "            evaluation_interval=2, slack_factor=0.2, delay_evaluation=6\n",
    "        ),\n",
    "    )\n",
    "\n",
    "    # define command function for registering the model\n",
    "    command_func = command(\n",
    "        inputs=dict(\n",
    "            model_input_path=Input(type=\"mlflow_model\"),\n",
    "            model_base_name=\"image_instance_segmentation_example_model\",\n",
    "        ),\n",
    "        code=\"./register.py\",\n",
    "        command=\"python register.py \"\n",
    "        + \"--model_input_path ${{inputs.model_input_path}} \"\n",
    "        + \"--model_base_name ${{inputs.model_base_name}}\",\n",
    "        environment=\"AzureML-sklearn-0.24-ubuntu18.04-py37-cpu:1\",\n",
    "    )\n",
    "    register_model = command_func(\n",
    "        model_input_path=image_instance_segmentation_node.outputs.best_model\n",
    "    )\n",
    "\n",
    "\n",
    "data_folder = \"./data\"\n",
    "pipeline = automl_image_instance_segmentation(\n",
    "    image_instance_segmentation_train_data=Input(\n",
    "        path=f\"{data_folder}/training-mltable-folder/\", type=\"mltable\"\n",
    "    ),\n",
    "    image_instance_segmentation_validation_data=Input(\n",
    "        path=f\"{data_folder}/validation-mltable-folder/\", type=\"mltable\"\n",
    "    ),\n",
    ")\n",
    "\n",
    "# set pipeline level compute\n",
    "pipeline.settings.default_compute = \"gpu-cluster\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.2 Submit pipeline job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# submit the pipeline job\n",
    "pipeline_job = ml_client.jobs.create_or_update(\n",
    "    pipeline, experiment_name=\"pipeline_samples\"\n",
    ")\n",
    "pipeline_job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wait until the job completes\n",
    "ml_client.jobs.stream(pipeline_job.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next Steps\n",
    "You can see further examples of running a pipeline job [here](../)"
   ]
  }
 ],
 "metadata": {
  "description": {
   "description": "Create pipeline with automl node"
  },
  "kernelspec": {
   "display_name": "Python 3.10.0 ('sdk-cli-v2')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  },
  "vscode": {
   "interpreter": {
    "hash": "c8b460ceac8e800778f31ce93ea472d62385502baa28f64e7ac3023280a461ed"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}