Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codegate Workspaces (repos) #454

Open
lukehinds opened this issue Dec 26, 2024 · 3 comments
Open

Codegate Workspaces (repos) #454

lukehinds opened this issue Dec 26, 2024 · 3 comments

Comments

@lukehinds
Copy link
Contributor

lukehinds commented Dec 26, 2024

Workspaces is something that myself and @jhrozek discussed, and the more our discussion unfolded, the more it was realised how useful this could be in unlocking many features that would be highly beneficial to users. It is also an ‘inevitable’ that we will need such a feature, so sooner the better and less painful in the long run.

This document attempts to describe the potential scope of Workspaces, what might they look like, and what they could unlock feature-wise.

Non-Goals

Let’s start off with non-goals, as this is important. Workspaces would not mean a multi-tenant system with access control, hierarchy etc. If we need that (which will be much later) then a system such as Minder would be a more logical choice.

So what is a Workspace?

A workspace (in respect of this document) is essentially a means of grouping elements under a single codebase / repository.

At present, we track the following elements, which are collated together as singular object:

  • Prompt (chat or FIM based, either from the user or the IDE / Agent)
  • LLM Output
  • Secrets
  • Malicious / Deprecated Packages

These can be further broken down into pipelines, where Secrets and Malicious Packages are inserted into a flow of checks via the pipeline system.

Over time, we plan to introduce more elements, such as token count, suspicious shell patterns, and folder obfuscation etc. It’s at this point that you can start to see a large amount of data all sitting in the same pile, making for an organized mess in the dashboard.

A workspace is a representation of a git repository, perhaps mounted in a volume or by other means (TBD)

The first obvious benefit of Workspace in CodeGate is the grouping together of elements, neatly under a parent (a repository). This will make the UX far more user-friendly than a big long ever increasing list of prompts, secrets and malicious packages.

Root Element (Repository)

The Nexus of a Workspace is the Repository. This is a logical component to use, for the following reasons:

  1. Prompts are specific to a codebase / repository (outside of generic chatbot scenarios)
  2. Outputs are specific to a codebase / repository
  3. Files and their paths are specific to a codebase / repository
  4. Secrets and Malicious packages are are specific to a codebase / repository

So we see we have logical grouping with a repository.

With a repository as the root element / grouping and local file access of the repository being made available to codegate, this then allows some highly useful features

Better UI Organization

Right now we lump everything in together, with separate buckets for each repo, it will be far more useful to helping users find what they need.

Workspace Based System Prompt

Having a code specific workspace, allows users to set a system prompt to a specific project (e.g., "You are an expert python developer with expertise in asynchronous networking and software security"). Of the few coding assistants that provide a customizable prompt, its global and often they are forced into adopting the IDE’s root system prompt

Image *Custom instructions in Cline*

Snapshot Generation

Snapshots are becoming increasingly popular, as a means of giving a model a holistic overview of the entire codebase, thereby making it produce more ‘project wide’ specific recommendations

Many of them exist, but they are all singular with no way of associating them with workspaces or checking their freshness:

acme
├── src
│   ├── backend
│   │   └── hello_world.py
│   └── frontend
│       └── hello_world.js


### Workspace Files
- `acme/src/python/hello_world.py`
- `acme/src/javascript/hello_world.js`

#### acme/src/backend/hello_world.py
def main():
    print("Hello, World!")

if __name__ == "__main__":
    main()

#### acme/src/frontend/hello_world.js
console.log("Hello, World!");

Token Metering and MUXing

Token usage per prompt / conversation, with something like provider filtering can offer good insight into what is costing the user more money and what optimizations could be made.

This is just a few of the ideas that workspace could bring about.

@JAORMX
Copy link
Contributor

JAORMX commented Jan 14, 2025

Let's rename "projects" to "workspaces" instead, this helps us avoid tenancy implications and thus not confuse folks.

@JAORMX JAORMX changed the title Codegate Projects (repos) Codegate Workspaces (repos) Jan 14, 2025
@JAORMX
Copy link
Contributor

JAORMX commented Jan 14, 2025

@lukehinds from a user perspective, is the idea that a user would spawn a Codegate container per each git repo they're planning to work on?

@JAORMX JAORMX self-assigned this Jan 14, 2025
@aponcedeleonch
Copy link
Contributor

@JAORMX no, the idea is that a single Codegate container can handle multiple git repos

aponcedeleonch added a commit that referenced this issue Jan 14, 2025
…path

Related: #454

This is the initial work to create workspaces when the server is initialized.
The idea is that the user mounts a volume at the specific location: `/app/codegate_workspaces`
and read from there the git repositories.
aponcedeleonch added a commit that referenced this issue Jan 15, 2025
Related: #454

We noticed most of the incoming-requests which contain a code-snippet
only list a relative path with respect to where the code editor is
opened. This would make difficult to accurately distinguish between
repositories in Codegate. For example, a user could open 2 different
code Python repositorites in different session and both repositories
contain a `pyproject.toml`. It would be impossible for Codegate to
determine the real repository of the file only using the relative path.

Hence, the initial implementation of Workspaces will rely on a pipeline
step that is able to take commands a process them. Some commands could be:
- List workspaces
- Add workspace
- Switch active workspace
- Delete workspace

It would be up to the user to select the desired active workspace.

This PR introduces an initial migration for Workspaces and the pipeline
step with the `list` command.
aponcedeleonch added a commit that referenced this issue Jan 16, 2025
Related: #454

We noticed most of the incoming-requests which contain a code-snippet
only list a relative path with respect to where the code editor is
opened. This would make difficult to accurately distinguish between
repositories in Codegate. For example, a user could open 2 different
code Python repositorites in different session and both repositories
contain a `pyproject.toml`. It would be impossible for Codegate to
determine the real repository of the file only using the relative path.

Hence, the initial implementation of Workspaces will rely on a pipeline
step that is able to take commands a process them. Some commands could be:
- List workspaces
- Add workspace
- Switch active workspace
- Delete workspace

It would be up to the user to select the desired active workspace.

This PR introduces an initial migration for Workspaces and the pipeline
step with the `list` command.
aponcedeleonch added a commit that referenced this issue Jan 16, 2025
* feat: Initial migration for Workspaces and pipeline step

Related: #454

We noticed most of the incoming-requests which contain a code-snippet
only list a relative path with respect to where the code editor is
opened. This would make difficult to accurately distinguish between
repositories in Codegate. For example, a user could open 2 different
code Python repositorites in different session and both repositories
contain a `pyproject.toml`. It would be impossible for Codegate to
determine the real repository of the file only using the relative path.

Hence, the initial implementation of Workspaces will rely on a pipeline
step that is able to take commands a process them. Some commands could be:
- List workspaces
- Add workspace
- Switch active workspace
- Delete workspace

It would be up to the user to select the desired active workspace.

This PR introduces an initial migration for Workspaces and the pipeline
step with the `list` command.

* Reformatting changes

* Make unique workspaces name

* Introduced Sessions table and added add and activate commands

* Formatting changes and unit tests

* Classes separation into a different file
aponcedeleonch added a commit that referenced this issue Jan 17, 2025
Related: #454

This PR is not ready yet. For the moment it adds the system prompts
to DB and associates it to a workspace.

It's missing to use the system prompt and actually send it to the
LLM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants