SkyPilot for Model Serving?

[sandkoan]

Has anyone tried to serve ML models through a web server (e.g., Flask, FastAPI) through SkyPilot (for autoscaling, job restart, etc)? Are there any examples detailing how one might go about doing so?

[concretevitamin]

Hey @sandkoan, we're aware of a few basic inference use cases on SkyPilot, as well as #994 by @iojw which prototyped using Flask on top of SkyPilot.

If model serving is of interest to you, I'd love to connect to learn more about your use case. Feel free to send me an email or a message through the community Slack!

[sandkoan]

Hey Zongheng, Hope you're doing well. SkyPilot is brilliant, and quite useful for running jobs in the cloud—it seems to have all the functionality necessary to run something like an Nvidia Triton server that's launched on-demand, and yet, remains long running, in a sense (capable of scaling as necessary). Could you perhaps explicate some of those other ways people have adapted SkyPilot for inference? Best, Govind

…

________________________________ From: Zongheng Yang ***@***.***> Sent: Saturday, January 14, 2023 10:16:52 PM To: skypilot-org/skypilot ***@***.***> Cc: Govind Gnanakumar ***@***.***>; Mention ***@***.***> Subject: Re: [skypilot-org/skypilot] SkyPilot for Model Serving? (Discussion #1544) Hey @sandkoan<https://github.com/sandkoan>, we're aware of a few basic inference use cases on SkyPilot, as well as #994<#994> by @iojw<https://github.com/iojw> which prototyped using Flask on top of SkyPilot. If model serving is of interest to you, I'd love to connect to learn more about your use case. Feel free to send me an email or a message through the community Slack<https://github.com/skypilot-org/skypilot#issues-feature-requests-and-questions>! — Reply to this email directly, view it on GitHub<#1544 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALNP7QKTUQKQ65H5TRZQ4HDWSNT2JANCNFSM6AAAAAATDGHJEQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[concretevitamin]

@pounde and co have built a model serving server/application called xView2. Hoping he can chime in more, but off the top of my head these inference requests are heavier computations (10s of minutes) running computer vision models. @dongreenberg and co have also built Runhouse using SkyPilot for all cluster needs, including serving I believe which is slightly less heavier weight(?).