Muxing Support for Model Routing

[lukehinds]

Introduce support for "Muxing" in CodeGate, enabling users to route specific types of logic to different large language models (LLMs) based on file type, individual files and workspace. This would provide cost savings , a common complaint from users is the expense of tokens and how to optimise those costs along with a way of isolation use to specific models (based on privacy, security , protection of IP)

It would also enable context switching (context window refers to the amount of text data a language model can consider at one time when generating responses) , whereby we could switch models during a long protracted session of prompt / response.

For example:

• Per Workspace: A user can set a LLM choice for the
• Per File Type: Documentation files (e.g., .md, .rst) could be processed by a free, local model, while source code files (e.g., .py, .js) are handled by an advanced model like Claude Sonnet 3.5.
• Per File: Users could specify models for particular files within a project based on complexity or other criteria.

Why is this feature important?

1. Granular Control: Developers can tune model usage based on file type or project needs.
2. Resource Efficiency: Allocate lightweight or local models to simpler tasks, while reserving advanced models for more complex challenges, providing more cost efficiency.
3. Workspace Integration: Aligns with the workflow of managing multiple workspaces

NOTE: As always, start small, simple and validate, the following acts as a guideline of where this could lead.

Possible Solution

Expand CodeGate’s functionality to include:

1. Repository-Level Configuration:

   • Enable users to set model preferences for an entire workspace.
   • Example: Repository A -> local_model, Repository B -> advanced_model.

2. File-Type-Based Routing:

   • Route tasks based on file extensions or MIME types.
   • Example: .md -> local_model, .py -> claude_sonnet_3.5.

3. Per-File Routing:

   • Allow specific files in a workspace to be linked to a designated model.
   • Example: config.py -> local_model_v2.

Challenges and Considerations

• Extensible: It's likely a lot of features will sit on top of muxing, and some synergies will be mapped to pipelines.
• Latency and Performance: Optimize routing logic to prevent delays in large or multi-repository setups.
• User Experience: Provide clear documentation and possibly a UI component for managing these configurations.

[lukehinds]

Moving this to ready, as I think we can at least start prototyping and thinking about this, albeit we should land #454 first to really make it useful.

[jtenniswood]

Filter by file: can do support wildcards (eg doc*.txt), as well as specific file names?
Would we also want to know the directory structure too?

[jtenniswood]

Would we want to specify a version of a model, or something broader (eg claude_sonnet vs claude_sonnet_3.5).
What would happen if 3.5 wasn't available?

[jtenniswood]

Would users select their repo, or just type it in (eg can we see a list or is it just manual?)

[jtenniswood]

Would this be on a global level or per workspace?

[jhrozek]

Filter by file: can do support wildcards (eg doc*.txt), as well as specific file names?

I think it would be nice, I can envision being OK to send tests/database.sql but not prod/database.sql (or anything under prod)

Would we also want to know the directory structure too?

We would want to but I don't think it was easy. Tagging @aponcedeleonch and @JAORMX who were looking into this. I suspect it's not fully possible until codegate is hooked into the IDE, but I don't know 100%

[jhrozek]

(Similar to the other reply, this is my 2 öre, not a canonical reply)

Would we want to specify a version of a model, or something broader (eg claude_sonnet vs claude_sonnet_3.5).

This is a question about model aliases right_

What would happen if 3.5 wasn't available?

I think we could start with just being specific - in other words, just require the full model name to by typed. Many providers have aliases anyway, e.g. anthropic supports claude-3-5-sonnet-latest. OpenAI has the same. The "major versions" are usually different enough I would let the user type the version.

But I can also see that codegate having some logic to pick the best model might be good, e.g. "for simple requests, use anthropic haiku, for more complex changes use anthropic sonnet, for very complex tasks use anthropic opus".

[JAORMX]

We would want to but I don't think it was easy. Tagging @aponcedeleonch and @JAORMX who were looking into this. I suspect it's not fully possible until codegate is hooked into the IDE, but I don't know 100%

That's right, codegate can't know unless it's actually hooked into the IDE, which it isn't

[jtenniswood]

First pass at model routing UI, love to hear your feedback, or if I have misunderstood anything

Demo
https://www.figma.com/proto/TtIS3Uk8qBdoNijeezfJYl/Dashboard?page-id=138%3A14936&node-id=138-15105&p=f&viewport=-3424%2C558%2C0.31&t=twZKYhVBCkhHlqpp-1&scaling=min-zoom&content-scaling=fixed&starting-point-node-id=138%3A15105

Designs
https://www.figma.com/design/TtIS3Uk8qBdoNijeezfJYl/Dashboard?node-id=156-8633&t=TJnz3nwd0u64aokC-11)

[jtenniswood]

[aponcedeleonch]

The work on this issue has finished. The next enhancement is tracked in #1059