2_fine_tuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine-Tuning the Text to Image Model\n",
    "\n",
    "Let's teach our text to image generator what my dog looks like.\n",
    "\n",
    "![redhat dog](https://rhods-public.s3.amazonaws.com/sample-data/images/redhat-dog-small.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check GPU\n",
    "\n",
    "**It's recommended that you shut down any other notebook kernels.**\n",
    "\n",
    "This fine tuning process uses a lot of video memory.   Here, we'll check on how much we have available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XU7NuMAA2drw",
    "outputId": "2e5ad228-fdf6-4828-e558-8c8c323ffdad",
    "tags": []
   },
   "outputs": [],
   "source": [
    "!nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv,noheader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wnTMyW41cC1E",
    "tags": []
   },
   "source": [
    "## Install Requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip list | grep -E \"boto|grpcio|pandas|torch|torchvision|diffusers|transformers|accelerate|flash-attn|ftfy|xformers|protobuf\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G0NV324ZcL9L"
   },
   "source": [
    "## Settings\n",
    "\n",
    "Here we set up all the options for training.  Most are environment variables which will allow us to override values from pipelines and run this notebook with different setting such as the base model or number of training steps and learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Rxg0y5MBudmd",
    "outputId": "1c67fe1f-5652-439c-abae-920824533765",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from datetime import datetime\n",
    "\n",
    "date = datetime.now()\n",
    "date_string = date.strftime(\"%Y%m%d%H%M%S\")\n",
    "\n",
    "VERSION = os.environ.get(\"VERSION\", f\"notebook-output\")\n",
    "MODEL_NAME = os.environ.get(\"MODEL_NAME\", \"stabilityai/stable-diffusion-2-1\")\n",
    "OUTPUT_DIR = os.path.join(os.getcwd(), f\"{VERSION}/stable_diffusion_weights/redhat-dog\")\n",
    "DATA_DIR = os.path.join(os.getcwd(), f\"{VERSION}/data\")\n",
    "INSTANCE_DATA_URL = os.environ.get(\"INSTANCE_DATA_URL\", \"https://rhods-public.s3.amazonaws.com/sample-data/images/redhat-dog.tar.gz\")\n",
    "INSTANCE_DIR = os.path.join(DATA_DIR, \"instance_dir\")\n",
    "CLASS_DIR = os.path.join(DATA_DIR, \"class_dir\")\n",
    "INSTANCE_PROMPT = os.environ.get(\"INSTANCE_PROMPT\", \"photo of a rhteddy dog\")\n",
    "CLASS_PROMPT = os.environ.get(\"CLASS_PROMPT\", \"a photo of dog\")\n",
    "\n",
    "NUM_CLASS_IMAGES = int(os.environ.get(\"NUM_CLASS_IMAGES\", \"200\"))\n",
    "MAX_TRAIN_STEPS = int(os.environ.get(\"MAX_TRAIN_STEPS\", \"800\"))\n",
    "\n",
    "S3_PREFIX = f\"models/{VERSION}/redhat-dog\"\n",
    "\n",
    "os.makedirs(OUTPUT_DIR, exist_ok=True)\n",
    "os.makedirs(INSTANCE_DIR, exist_ok=True)\n",
    "\n",
    "print(f\"Weights will be saved at {OUTPUT_DIR}\")\n",
    "print(f\"It will be based on the model {MODEL_NAME}\")\n",
    "print(f\"Training data located in downloaded from {INSTANCE_DATA_URL}\")\n",
    "print(f\"We're going to train the difference between \\\"{INSTANCE_PROMPT}\\\" and \\\"{CLASS_PROMPT}\\\"\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qn5ILIyDJIcX"
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Training Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "import tarfile\n",
    "import urllib\n",
    "\n",
    "url = INSTANCE_DATA_URL\n",
    "output = f\"instance-images.tar.gz\"\n",
    "urllib.request.urlretrieve(url, output)\n",
    "\n",
    "!tar -xzf instance-images.tar.gz -C $INSTANCE_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up the Training Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!accelerate config default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!wget -O train_dreambooth.py https://raw.githubusercontent.com/huggingface/diffusers/main/examples/dreambooth/train_dreambooth.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Training\n",
    "\n",
    "Here we kick off the training job with our chosen settings.  This will take about 15 minutes depending on settings and hardware."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!echo \"MODEL_NAME=$MODEL_NAME\"\n",
    "!echo \"OUTPUT_DIR=$OUTPUT_DIR\"\n",
    "!echo \"DATA_DIR=$DATA_DIR\"\n",
    "!echo \"INSTANCE_DIR=$INSTANCE_DIR\"\n",
    "!echo \"CLASS_DATA_URL=$CLASS_DATA_URL\"\n",
    "!echo \"CLASS_DIR=$CLASS_DIR\"\n",
    "!echo \"INSTANCE_PROMPT=$INSTANCE_PROMPT\"\n",
    "!echo \"CLASS_PROMPT=$CLASS_PROMPT\"\n",
    "!echo \"NUM_CLASS_IMAGES=$NUM_CLASS_IMAGES\"\n",
    "!echo \"MAX_TRAIN_STEPS=$MAX_TRAIN_STEPS\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!accelerate launch train_dreambooth.py \\\n",
    "  --pretrained_model_name_or_path=$MODEL_NAME  \\\n",
    "  --instance_data_dir=$INSTANCE_DIR \\\n",
    "  --class_data_dir=$CLASS_DIR \\\n",
    "  --output_dir=$OUTPUT_DIR \\\n",
    "  --with_prior_preservation --prior_loss_weight=1.0 \\\n",
    "  --instance_prompt=\"$INSTANCE_PROMPT\" \\\n",
    "  --class_prompt=\"$CLASS_PROMPT\" \\\n",
    "  --resolution=512 \\\n",
    "  --train_batch_size=1 \\\n",
    "  --gradient_accumulation_steps=2 --gradient_checkpointing \\\n",
    "  --use_8bit_adam \\\n",
    "  --learning_rate=5e-6 \\\n",
    "  --lr_scheduler=\"constant\" \\\n",
    "  --lr_warmup_steps=0 \\\n",
    "  --num_class_images=$NUM_CLASS_IMAGES \\\n",
    "  --max_train_steps=$MAX_TRAIN_STEPS \\\n",
    "  --enable_xformers_memory_efficient_attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examine the output\n",
    "\n",
    "If everything went well, we should have generated model weights in the `$OUTPUT_DIR` including those for the tokenizer, vae, unet, and text encoder.  You could now load these weights and generate images with our finetuned model.  You could simply replace any references to our Huggingface model (`runwayml/stable-diffusion-v1-5`) with the path to our new weights (e.g. `./notebook-output/stable_diffusion_weights/redhat-dog`)\n",
    "\n",
    "It's also ready for upload to Huggingface, such as [cfchase/stable-diffusion-rhteddy](https://huggingface.co/cfchase/stable-diffusion-rhteddy), where you can use those weights in another notebook or application.  To try out the custom weight from huggingface (or any other weights) with multiple images generated, try the notebook [misc-notebooks/generate_multiple.ipynb](misc-notebooks/generate_multiple.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!ls $OUTPUT_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save to S3\n",
    "\n",
    "Now we have our model in a portable ONNX format, however, it's not doing much good in this notebook. We need to push the model to our connected storage location. Then we can use it in another notebook or serve the models for use within an application.\n",
    "Note: This requires a data connection to an S3 compatible bucket. As part of the setup for this project, you added the setup-s3.yaml which created a local s3 bucket and data connections.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import boto3\n",
    "import botocore\n",
    "\n",
    "aws_access_key_id = os.environ.get('AWS_ACCESS_KEY_ID')\n",
    "aws_secret_access_key = os.environ.get('AWS_SECRET_ACCESS_KEY')\n",
    "endpoint_url = os.environ.get('AWS_S3_ENDPOINT')\n",
    "region_name = os.environ.get('AWS_DEFAULT_REGION')\n",
    "bucket_name = os.environ.get('AWS_S3_BUCKET')\n",
    "\n",
    "session = boto3.session.Session(aws_access_key_id=aws_access_key_id,\n",
    "                                aws_secret_access_key=aws_secret_access_key)\n",
    "\n",
    "s3_resource = session.resource(\n",
    "    's3',\n",
    "    config=botocore.client.Config(signature_version='s3v4'),\n",
    "    endpoint_url=endpoint_url,\n",
    "    region_name=region_name)\n",
    "\n",
    "bucket = s3_resource.Bucket(bucket_name)\n",
    "\n",
    "\n",
    "def upload_directory_to_s3(local_directory, s3_prefix):\n",
    "    for root, dirs, files in os.walk(local_directory):\n",
    "        for filename in files:\n",
    "            file_path = os.path.join(root, filename)\n",
    "            relative_path = os.path.relpath(file_path, local_directory)\n",
    "            s3_key = os.path.join(s3_prefix, relative_path)\n",
    "            print(f\"{file_path} -> {s3_key}\")\n",
    "            bucket.upload_file(file_path, s3_key)\n",
    "\n",
    "\n",
    "def list_objects(prefix):\n",
    "    filter = bucket.objects.filter(Prefix=prefix)\n",
    "    for obj in filter.all():\n",
    "        print(obj.key)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "S3_PREFIX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "upload_directory_to_s3(OUTPUT_DIR, S3_PREFIX)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "list_objects(S3_PREFIX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remember the S3 Prefixes \n",
    "\n",
    "Take note of the s3 prefixes. \n",
    "\n",
    "For example, `models/notebook-output/redhat-dog/`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Next\n",
    "\n",
    "The next notebook [Notebook 3 - Remote Inference](3_remote_inference.ipynb), tests out Red Hat OpenShift Data Science Serving capabilities.  Before trying it out, you must have deployed a custom runtime for Diffusers, and created a Single Model Server.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "vscode": {
   "interpreter": {
    "hash": "e7370f93d1d0cde622a1f8e1c04877d8463912d04d973331ad4851f04de6915a"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0cee055de82f4a4d8019e90d5ed1dfbb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "180888618e5a4d5d9c5bace5ef15c21a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_da70b684ad3f44789cf0f213708f5da5",
      "placeholder": "​",
      "style": "IPY_MODEL_83954ef920464529acf6015cb9d31693",
      "value": "100%"
     }
    },
    "195e8f3e983342d9b7227ee9e4461153": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "1d7ea19cc56a43c68df69bcc9655b8a7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a58f6b987c9f4f5e9a1b0a73c4ae549e",
       "IPY_MODEL_d2740a085b95416b88fa742e058402be",
       "IPY_MODEL_8252ddb68924432b87a1083b7e6aaa6e"
      ],
      "layout": "IPY_MODEL_e7113dfd596d4ecc8b1ebd76ee979d92"
     }
    },
    "28491b57195048b99c3714b60c8a25c2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "582d3b3efd32478da7754bf2cb5530c3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_28491b57195048b99c3714b60c8a25c2",
      "max": 200,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_b662d5f27cd5413db237b78bd137cae7",
      "value": 200
     }
    },
    "6eacdf7c06d041218c0b96745106e712": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "746ae93e85b94b0c9371e1dff99fc620": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8252ddb68924432b87a1083b7e6aaa6e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fe3f23ff7b8b46938a53c5e1bfdbaaca",
      "placeholder": "​",
      "style": "IPY_MODEL_916a795f943e475cbdc41ed25dab3deb",
      "value": " 200/200 [00:31&lt;00:00,  6.29it/s]"
     }
    },
    "83954ef920464529acf6015cb9d31693": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "916a795f943e475cbdc41ed25dab3deb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9b8392f2d04e476ea94e7c5889d9fdd7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a58f6b987c9f4f5e9a1b0a73c4ae549e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_be1ef96102b545a782d54b3dfd698696",
      "placeholder": "​",
      "style": "IPY_MODEL_746ae93e85b94b0c9371e1dff99fc620",
      "value": "100%"
     }
    },
    "b662d5f27cd5413db237b78bd137cae7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "b9f236f54bf640f791c17beb11c95c66": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9b8392f2d04e476ea94e7c5889d9fdd7",
      "placeholder": "​",
      "style": "IPY_MODEL_e3116fa65f6b4abcbc9ce7e61bdfb45f",
      "value": " 200/200 [00:31&lt;00:00,  6.34it/s]"
     }
    },
    "be1ef96102b545a782d54b3dfd698696": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d2740a085b95416b88fa742e058402be": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6eacdf7c06d041218c0b96745106e712",
      "max": 200,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_195e8f3e983342d9b7227ee9e4461153",
      "value": 200
     }
    },
    "da70b684ad3f44789cf0f213708f5da5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e3116fa65f6b4abcbc9ce7e61bdfb45f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e7113dfd596d4ecc8b1ebd76ee979d92": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "eca04bb8e40f49c7869f00512e0767a6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_180888618e5a4d5d9c5bace5ef15c21a",
       "IPY_MODEL_582d3b3efd32478da7754bf2cb5530c3",
       "IPY_MODEL_b9f236f54bf640f791c17beb11c95c66"
      ],
      "layout": "IPY_MODEL_0cee055de82f4a4d8019e90d5ed1dfbb"
     }
    },
    "fe3f23ff7b8b46938a53c5e1bfdbaaca": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}