Added Inference Spec & CI

[AlejandroEsquivel]

Added:

• Added Inference Spec class to support Ray, Sagemaker, FastAPI in one go
• Added CI to support to deploy/update validator's inference endpoint on the cluster

Example Running FastAPI with Inference Spec:

pip install git+https://github.com/guardrails-ai/models-host@feat/adding-ray-setup
uvicorn models_host.fastapi:app

Example Running Ray Serve with Inference Spec:

# Refer to models-host repo for instructions if you don't have a GPU on the machine running ray serve
pip install git+https://github.com/guardrails-ai/models-host@feat/adding-ray-setup
serve run models_host.ray_serve:app

[JosephCatrambone]

Double checking that serve should not have the .py extension and that app.py is still required? Can we remove app.py now that inference_spec is in place or is that required for the heartbeat path?

[AlejandroEsquivel]

Double checking that serve should not have the .py extension and that app.py is still required? Can we remove app.py now that inference_spec is in place or is that required for the heartbeat path?

yep good call outs, serve is a standarized filename for Sagemaker to load models for inference and has no extension but has the top part #!/usr/bin/env python

as for app.py not technically needed but I believe it's purpose was to showcase to other "validator developers" how they can serve the validator themselves but would be good to double check with Z cause once their impl starts diverging we might not want to maintain those unless there's a better reason to keep them around