sdk/endpoints/online/triton/single-model/online-endpoints-triton.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy a model to online endpoints using Triton\n",
    "Learn how to deploy a model using Triton as an online endpoint in Azure Machine Learning.\n",
    "\n",
    "Triton is multi-framework, open-source software that is optimized for inference. It supports popular machine learning frameworks like TensorFlow, ONNX Runtime, PyTorch, NVIDIA TensorRT, and more. It can be used for your CPU or GPU workloads.\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "* To use Azure Machine Learning, you must have an Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the [free or paid version of Azure Machine Learning](https://azure.microsoft.com/free/).\n",
    "\n",
    "* Install and configure the [Python SDK v2](sdk/setup.sh).\n",
    "\n",
    "* You must have an Azure resource group, and you (or the service principal you use) must have Contributor access to it.\n",
    "\n",
    "* You must have an Azure Machine Learning workspace.\n",
    "\n",
    "* You must have additional Python packages installed for scoring and may install them with the code below. They include:\n",
    "    * Numpy - An array and numerical computing library \n",
    "    * [Triton Inference Server Client](https://github.com/triton-inference-server/client) - Facilitates requests to the Triton Inference Server\n",
    "    * Pillow - A library for image operations\n",
    "    * Gevent - A networking library used when connecting to the Triton Server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install numpy\n",
    "%pip install tritonclient[http]\n",
    "%pip install pillow\n",
    "%pip install gevent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Please note, for Triton no-code-deployment, testing via local endpoints is currently not supported, so this tutorial will only show how to set up on online endpoint.\n",
    "\n",
    "## 1. Connect to Azure Machine Learning Workspace\n",
    "\n",
    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Configure workspace details\n",
    "To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "1-configure-workspace-details"
   },
   "outputs": [],
   "source": [
    "subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "resource_group = \"<RESOURCE_GROUP>\"\n",
    "workspace_name = \"<AML_WORKSPACE_NAME>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Generate an endpoint name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "1-generate-endpoint-name"
   },
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "endpoint_name = f\"endpoint-{random.randint(0, 10000)}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Get a handle to the workspace\n",
    "\n",
    "We use these details above in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](../../jobs/configuration.ipynb) for more details on how to configure credentials and connect to a workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "1-get-handle"
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "ml_client = MLClient(\n",
    "    DefaultAzureCredential(),\n",
    "    subscription_id,\n",
    "    resource_group,\n",
    "    workspace_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Configure deployment and associated resources\n",
    "\n",
    "A deployment is a set of resources required for hosting the model that does the actual inferencing. We will create a deployment for our endpoint using the `ManagedOnlineDeployment` class.\n",
    "\n",
    "### Key aspects of deployment \n",
    "- `name` - Name of the deployment.\n",
    "- `endpoint_name` - Name of the endpoint to create the deployment under.\n",
    "- `model` - The model to use for the deployment. This value can be either a reference to an existing versioned model in the workspace or an inline model specification.\n",
    "- `environment` - The environment to use for the deployment. This value can be either a reference to an existing versioned environment in the workspace or an inline environment specification.\n",
    "- `code_configuration` - the configuration for the source code and scoring script\n",
    "    - `path`- Path to the source code directory for scoring the model\n",
    "    - `scoring_script` - Relative path to the scoring file in the source code directory\n",
    "- `instance_type` - The VM size to use for the deployment. For the list of supported sizes, see [Managed online endpoints SKU list](https://docs.microsoft.com/en-us/azure/machine-learning/reference-managed-online-endpoints-vm-sku-list).\n",
    "- `instance_count` - The number of instances to use for the deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Configure online endpoint\n",
    "`endpoint_name`: The name of the endpoint. It must be unique in the Azure region. Naming rules are defined under [managed online endpoint limits](https://docs.microsoft.com/azure/machine-learning/how-to-manage-quotas#azure-machine-learning-managed-online-endpoints-preview).\n",
    "\n",
    "`auth_mode` : Use `key` for key-based authentication. Use `aml_token` for Azure Machine Learning token-based authentication. A `key` does not expire, but `aml_token` does expire. \n",
    "\n",
    "Optionally, you can add description, tags to your endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "2-configure-online-endpoint"
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import ManagedOnlineEndpoint\n",
    "\n",
    "endpoint = ManagedOnlineEndpoint(name=endpoint_name, auth_mode=\"key\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Configure online deployment\n",
    "A deployment is a set of resources required for hosting the model that does the actual inferencing. We will create a deployment for our endpoint using the `ManagedOnlineDeployment` class and define a Model inline.\n",
    "\n",
    "#### Key aspects of deployment \n",
    "- `name` - Name of the deployment.\n",
    "- `endpoint_name` - Name of the endpoint to create the deployment under.\n",
    "- `model` - The model to use for the deployment. This value can be either a reference to an existing versioned model in the workspace or an inline model specification.\n",
    "- `instance_type` - The VM size to use for the deployment. For the list of supported sizes, see [Managed online endpoints SKU list](https://docs.microsoft.com/en-us/azure/machine-learning/reference-managed-online-endpoints-vm-sku-list).\n",
    "- `instance_count` - The number of instances to use for the deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "2-configure-online-deployment"
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import ManagedOnlineDeployment, Model\n",
    "\n",
    "deployment = ManagedOnlineDeployment(\n",
    "    name=\"blue\",\n",
    "    endpoint_name=endpoint_name,\n",
    "    model=Model(path=\"./models\", type=\"triton_model\"),\n",
    "    instance_type=\"Standard_NC6s_v3\",\n",
    "    instance_count=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Deploy to Azure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Create the endpoint\n",
    "Using the `MLClient` created earlier, we will now create the Endpoint in the workspace. This command will start the endpoint creation and return a confirmation response while the endpoint creation continues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "3-create-endpoint"
   },
   "outputs": [],
   "source": [
    "endpoint = ml_client.online_endpoints.begin_create_or_update(endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Create the deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `MLClient` created earlier, we will now create the deployment in the workspace. This command will start the deployment creation and return a confirmation response while the deployment creation continues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "3-create-deployment"
   },
   "outputs": [],
   "source": [
    "ml_client.online_deployments.begin_create_or_update(deployment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Set traffic to 100% for deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "3-set-traffic"
   },
   "outputs": [],
   "source": [
    "endpoint.traffic = {\"blue\": 100}\n",
    "ml_client.online_endpoints.begin_create_or_update(endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Test the endpoint with sample data\n",
    "This version of the triton server requires pre- and post-image processing. Below we show how to invoke the endpoint with this processing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Retrieve the scoring URI\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "4-scoring-uri"
   },
   "outputs": [],
   "source": [
    "endpoint = ml_client.online_endpoints.get(endpoint_name)\n",
    "scoring_uri = endpoint.scoring_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Retrieve the endpoint auth key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "4-auth-key"
   },
   "outputs": [],
   "source": [
    "keys = ml_client.online_endpoints.list_keys(endpoint_name)\n",
    "auth_key = keys.primary_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Test the endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below script imports pre- and post-processing functions from `scoring_utils/prepost.py`. We first test the model/server readiness and then use those functions to convert the image into a triton readable format and issue the scoring request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "4-test-endpoint"
   },
   "outputs": [],
   "source": [
    "# Test the blue deployment with some sample data\n",
    "import requests\n",
    "import gevent.ssl\n",
    "import numpy as np\n",
    "import tritonclient.http as tritonhttpclient\n",
    "from pathlib import Path\n",
    "import prepost\n",
    "\n",
    "img_uri = \"http://aka.ms/peacock-pic\"\n",
    "\n",
    "# We remove the scheme from the url\n",
    "url = scoring_uri[8:]\n",
    "\n",
    "# Initialize client handler\n",
    "triton_client = tritonhttpclient.InferenceServerClient(\n",
    "    url=url,\n",
    "    ssl=True,\n",
    "    ssl_context_factory=gevent.ssl._create_default_https_context,\n",
    ")\n",
    "\n",
    "# Create headers\n",
    "headers = {}\n",
    "headers[\"Authorization\"] = f\"Bearer {auth_key}\"\n",
    "\n",
    "# Check status of triton server\n",
    "health_ctx = triton_client.is_server_ready(headers=headers)\n",
    "print(\"Is server ready - {}\".format(health_ctx))\n",
    "\n",
    "# Check status of model\n",
    "model_name = \"model_1\"\n",
    "status_ctx = triton_client.is_model_ready(model_name, \"1\", headers)\n",
    "print(\"Is model ready - {}\".format(status_ctx))\n",
    "\n",
    "if Path(img_uri).exists():\n",
    "    img_content = open(img_uri, \"rb\").read()\n",
    "else:\n",
    "    agent = f\"Python Requests/{requests.__version__} (https://github.com/Azure/azureml-examples)\"\n",
    "    img_content = requests.get(img_uri, headers={\"User-Agent\": agent}).content\n",
    "\n",
    "img_data = prepost.preprocess(img_content)\n",
    "\n",
    "# Populate inputs and outputs\n",
    "input = tritonhttpclient.InferInput(\"data_0\", img_data.shape, \"FP32\")\n",
    "input.set_data_from_numpy(img_data)\n",
    "inputs = [input]\n",
    "output = tritonhttpclient.InferRequestedOutput(\"fc6_1\")\n",
    "outputs = [output]\n",
    "\n",
    "result = triton_client.infer(model_name, inputs, outputs=outputs, headers=headers)\n",
    "max_label = np.argmax(result.as_numpy(\"fc6_1\"))\n",
    "label_name = prepost.postprocess(max_label)\n",
    "print(label_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Managing endpoints and deployments\n",
    "\n",
    "### 5.1 Get the logs for the new deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "5-get-logs"
   },
   "outputs": [],
   "source": [
    "ml_client.online_deployments.get_logs(\n",
    "    name=\"blue\", endpoint_name=endpoint_name, lines=50\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Delete resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 Get model name and version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment = ml_client.online_deployments.get(name=\"blue\", endpoint_name=endpoint_name)\n",
    "\n",
    "model_uri = deployment.model.split(\"/\")\n",
    "model_name = model_uri[-3]\n",
    "model_version = model_uri[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Delete the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "6-delete-endpoint"
   },
   "outputs": [],
   "source": [
    "ml_client.online_endpoints.begin_delete(name=endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3 Archive the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.models.archive(name=model_name, version=model_version)"
   ]
  }
 ],
 "metadata": {
  "description": {
   "description": "Deploy a custom container as an online endpoint. Use web servers other than the default Python Flask server used by Azure ML without losing the benefits of Azure ML's built-in monitoring, scaling, alerting, and authentication."
  },
  "kernelspec": {
   "display_name": "Python 3.8.13 ('triton1')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "42a2b98eb521da9fbb037bb21477fc99ac2282fc3b56c201a68e3aa412fb4376"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}