Proposal: Donate Feast to Kubeflow

[franciscojavierarceo]

I am proposing donating Feast to Kubeflow (kubeflow/community#804).

I have copied the benefits I outlined in the Kubeflow issue:

Benefits

1. Incorporating Feast into Kubeflow (and the manifest) will help formally fill a needed gap for Kubeflow in the AI/ML Lifecycle (image for reference).
   

2. It will also allow the Data WG to have an answer for the online serving of features. Additionally, this will nicely complement the Spark Operator as Feast supports batch and stream processing using Spark as an offline store.

3. The Feast community is healthy and the users will further grow the Kubeflow community.

4. Feast is expanding its scope to support Generative AI and RAG as a first-class citizen (retrieval/vector search in particular), which will help ensure Kubeflow has a solution for RAG.

5. With the inclusion of Feast, we can provide end-to-end demos of development and production AI/ML and we can also provide suggested patterns for stitching the Kubeflow products together so that MLOps engineers, ML Engineers, and AI engineers can be impactful immediately after deploying Kubeflow.

6. I am just as committed to Feast as I have ever been and I believe this will meaningfully enhance Kubeflow and result in Kubeflow getting the benefit of my contributions and the contributions of the Feast community.

Beyond the benefits to Kubeflow, I believe it will provide significant benefits to Feast in growing our community and embedding ourselves in a complete platform. This will also allow both communities (particularly Data Scientists/MLEs) to benefit from the GenAI/LLM/NLP work that I have been doing (see here: #4964 and #4364).

I have discussed this with the Feast maintainers and have gotten approval. We also wanted to have this discussion open in the public to welcome any feedback from users.

--
For reference, the google doc is available here.

[woop]

Thanks @franciscojavierarceo!

Given that this is such a large decision for a project like Feast, it's worth spelling out the case for moving (vs not moving), and the specific implications of the donation. I can see a donation going in many different directions, some of which are very disruptive to existing users, while other approaches are nearly invisible and a net win.

Is there a document that lays out the plan and implications?

[franciscojavierarceo]

Is there a document that lays out the plan and implications?

@woop yes, see this document and this PR with that document added (will update it to reflect latest changes soon).

Given that this is such a large decision for a project like Feast, it's worth spelling out the case for moving (vs not moving), and the specific implications of the donation. I can see a donation going in many different directions, some of which are very disruptive to existing users, while other approaches are nearly invisible and a net win.

I can add a section to the doc on that 👍

[woop]

Thanks @franciscojavierarceo! The document above is mostly focused on the value to Kubeflow, so as long as we can document the value to the Feast side as well, then we're in a good spot to make a decision.

[franciscojavierarceo]

Sounds good, will update to include that as well.

[ngoduykhanh]

Out of curiosity, If Feast joins Kubeflow, will Feast still be available as a standalone open-source product that we can use as of today? I don't have access to the documentation so I can't see the detailed plan.

[franciscojavierarceo]

I'll update the access to make it public. Each Kubeflow component is meant to be standalone and integrated!

[franciscojavierarceo]

@ngoduykhanh updated!

[franciscojavierarceo]

@woop updated to include a "Benefits for Feast section", as follows:

Benefits for Feast

Having Feast as an official part of the Kubeflow community has the following benefits:

1. Being integrated into an end-to-end AI/ML ecosystem .
2. Increased adoption from the Kubeflow community.
3. Aligned product roadmap and vision that best serves the community.
4. Coupled with open source products that are maintained by a neutral foundation that allows for great product experiences for end-users.
5. Benefit from Kubeflow’s marketing and social media support rather than duplicating effort.
6. Brand benefits from Kubeflow and its enterprise appeal.

[woop]

Generally I'm supportive of closer collaboration with Kubeflow - I just want to make sure we're solving the right problems and choosing the best path forward for Feast and its users.

What problem are we trying to solve?

You mentioned that maintainers are spending too much time on community management rather than technical development.

Can you share more about what specific challenges you're facing day-to-day? Is it about:

• Limited bandwidth for code reviews and feature development?
• Challenges with marketing and community growth?
• Concerns about long term sustainability?

Understanding what's actually painful would help us figure out if donation is the right solution.

Is donation the only option?

Before we go all in on donation, I'm curious if we've thought about:

• Keeping Feast independent but deeply integrated with Kubeflow - similar to how KServe works. This would give us the integration benefits without completely merging the projects.

• Sharing specific resources - Maybe we could get help with the parts that are most burdensome (docs, community management) while keeping the core project separate?

What would this mean for current Feast users?

I'm also thinking about our existing users:

• Will they need to change how they install or use Feast?
• How disruptive would API changes be? Are there any guarantees being made in terms of breaking changes?
• What's our plan for a smooth transition of docs and support channels?
• Are there any other downsides to the move (I'm only seeing benefits)?

Next steps

Let's figure out:

1. What specific problems we're trying to solve with this move
2. Whether donation is the only/best option
3. A clear transition plan if we do move forward with donation

Happy to get into the details over a call, but I do want to make sure that this discussion is public for posterity sake.

I'm all for making Feast better and more sustainable - just want to make sure we're taking the right approach.

What do you think?

[franciscojavierarceo]

Can you share more about what specific challenges you're facing day-to-day? Is it about:

Limited bandwidth for code reviews and feature development?
Challenges with marketing and community growth?
Concerns about long term sustainability?

It is mostly about "Limited bandwidth for code reviews and feature development" and "Concerns about long term sustainability".

My work over the past year and a half has helped continue to grow the community (I don't intend on stopping that) but my bandwidth is limited and I don't want maintainers to have to focus on marketing and events. Kubeflow has a large community with folks explicitly focusing on community and attending events—Feast doesn't (excluding my best attempts).

Before we go all in on donation, I'm curious if we've thought about:

Keeping Feast independent but deeply integrated with Kubeflow - similar to how KServe works. This would give us the integration benefits without completely merging the projects.

Yes, I discussed this extensively with the Kubeflow Steering Committee and their feedback was that they are getting rid of Add-Ons as a concept entirely, so that took this option off the table and is what ultimately led to my proposal. You can see the full discussion openly in this PR: kubeflow/community#741.

I'm also thinking about our existing users:

Will they need to change how they install or use Feast?

Nope, they will not have to. It'll still be the same pip install and we'll even incorporate a proper Feast Client in the Kubeflow SDK to be more deeply integrated with the rest of the Kubeflow components.

How disruptive would API changes be? Are there any guarantees being made in terms of breaking changes?

Absolutely no breaking changes due to Kubeflow.

What's our plan for a smooth transition of docs and support channels?

I intend on keeping the documentation. Our docs are a great example of what docs should look like (though of course we can still make some improvements).

Are there any other downsides to the move (I'm only seeing benefits)?

Candidly, not really. The Feast brand, code, deployment, and structure would still remain, we do forego some control to the Data WG that is being proposed in the sense that they can influence our decisions but I haven't actually seen that exercised. Part of giving influence to the WG is to foster a data community inside Kubeflow that also focuses on GenAI (it would be included with Spark, Model Registry, and others). FWIW, I would be a lead of that WG as well and hope to invest in making the Spark KF experience even better (especially for embedding lots of documents 🤗).

Lastly, I want to be transparent to the community, I proposed this because I think it's what's best for Feast and Kubeflow, as well as each individually. Candidly, Kubeflow has a massive gap in having 0 answer on how to serve data if it drops Feast when dropping Add-Ons. It's a very clear win-win and, I believe, it is part of the reason I was elected to the KSC (all of the feedback on social channels that I have seen about the proposal have been extremely positive).

[franciscojavierarceo]

And to answer these explicitly:

1. What specific problems we're trying to solve with this move

I believe my note above addressed this explicit as well as the Google Document

2. Whether donation is the only/best option

I believe I also explicitly addressed this above with this comment:

Before we go all in on donation, I'm curious if we've thought about:

Keeping Feast independent but deeply integrated with Kubeflow - similar to how KServe works. This would give us the integration benefits without completely merging the projects.

Yes, I discussed this extensively with the Kubeflow Steering Committee and their feedback was that they are getting rid of Add-Ons as a concept entirely, so that took this option off the table and is what ultimately led to my proposal. You can see the full discussion openly in this PR: kubeflow/community#741.

4. A clear transition plan if we do move forward with donation

This is also outlined in the Google Document under the Migration Plan section.