Proposal: Adoption of Feast to Kubeflow

[franciscojavierarceo]

History with Kubeflow

Feast has a long history with Kubeflow, as an add-on and previously included in the manifest dating back to March of 2021.

After discussing with the @feast-dev maintainers and getting their agreement, I am proposing donating Feast to Kubeflow to officially serve as Kubeflow's recommended open source feature store of choice.

Benefits

1. Incorporating Feast into Kubeflow (and the manifest) will help formally fill a needed gap for Kubeflow in the AI/ML Lifecycle (image for reference).
   

2. It will also allow the Data WG to have an answer for the online serving of features. Additionally, this will nicely complement the Spark Operator as Feast supports batch and stream processing using Spark as an offline store.

3. The Feast community is healthy and the users will further grow the Kubeflow community.

4. Feast is expanding its scope to support Generative AI and RAG as a first-class citizen (retrieval/vector search in particular), which will help ensure Kubeflow has a solution for RAG.

5. With the inclusion of Feast, we can provide end-to-end demos of development and production AI/ML and we can also provide suggested patterns for stitching the Kubeflow products together so that MLOps engineers, ML Engineers, and AI engineers can be impactful immediately after deploying Kubeflow.

6. I am just as committed to Feast as I have ever been and I believe this will meaningfully enhance Kubeflow and result in Kubeflow getting the benefit of my contributions and the contributions of the Feast community.

[franciscojavierarceo]

CC @kubeflow/kubeflow-steering-committee

[danbaron63]

This sounds really exciting! We're currently running KubeFlow in GCP Vertex AI pipelines, how easy would be integration be from here?

[blaketastic2]

Will Feast still be a standalone component we can continue to use w/ other options like Flyte?

[franciscojavierarceo]

Yes, @blaketastic2!

[franciscojavierarceo]

This sounds really exciting! We're currently running KubeFlow in GCP Vertex AI pipelines, how easy would be integration be from here?

That's a good question @danbaron63. @hbelmiro is working on a demo with Feast and KFP and the idea would be that a KFP would run either batch processing tasks or the materialization to the online store. The former ends up being quite a lot of jobs so our plan is to invest there, as that's a very common user need. If you have feedback or an integration you'd like to see, do feel free to share it! Would love to hear about it!

[hbelmiro]

This sounds really exciting! We're currently running KubeFlow in GCP Vertex AI pipelines, how easy would be integration be from here?

That's a good question @danbaron63. @hbelmiro is working on a demo with Feast and KFP and the idea would be that a KFP would run either batch processing tasks or the materialization to the online store. The former ends up being quite a lot of jobs so our plan is to invest there, as that's a very common user need. If you have feedback or an integration you'd like to see, do feel free to share it! Would love to hear about it!

@danbaron63
I have never used GCP Vertex AI, but Feast provides a Python SDK and a REST API, which makes me think it should not be hard to integrate.