Allow creation of master API keys to be shared across users

[krschacht]

Some people want to setup an instance of HostedGPT and let a couple special people use it. In this scenario, they don’t want everyone to have to enter their own API keys they want to let them just piggy back on some master API keys and foot the bill. In this scenario, my assumption was: we could read the API keys out of some central config.

If we implemented it this way then we would want to allow trusted people to register but prevent random people from registering. Maybe there needs to be an option to set a registration code? And, when I tell a trusted friend my registration code and they sign up, if that friend goes to their account settings how should the API key fields behave? Maybe the fields should still be there so they can input their own keys instead of using the master ones. Probably that’s the case.

Note: We’ll need some way to lock down user registration once this is implemented.

[krschacht]

@olimart I mentioned this in a PR, but I’m pulling it out into a real issue. FYI in case you want to tackle it.

[olimart]

Food for thoughts:

• Implement a new model ApiKey with type/provider Claude, ChatGPT,... & shared attribute.
• Update logic for current user key to become user's key || shared ApiKey

This implies:

• To keep fields on user settings to override shared key
• New admin/api_keys controller & views to manage keys
  • Add admin attribute to the very first user created to restrict keys management
• Migrate user keys to the new ApiKey model
• Eventually, new onboarding screen to input API key(s)

[krschacht]

I like how you’re thinking. What you laid out makes sense as the place to end up. My inclination would be to get there in a least a couple PRs. One PR could lay down the core logic and just use an ENV as the fallback “shared key” and then a second PR could implement the new admin views & routes.

Want to take this one?

[stephan-buckmaster]

Has this got anywhere? More api-key configurations coming with my next changes, for #341

[krschacht]

Nope, no progress on this one so you're safe to changes in this arena

[matthewbennink]

This is interesting for our use case where we'll have a handful of students who are learning about AI chat interfaces. We'll be providing the API keys and yet we want to be sure we can keep usage within reasonable limits on a per-user basis.

As for cost tracking, is that something like keeping track of token usage per conversation? It seems like it'd be hard to do per message, especially if there's a moment where HostedGPT is unable to send all of the preceding messages, but instead sends summarized messages or perhaps only the last N messages of a conversation to keep the token count underneath the max_tokens limit. Or were you imagining tracking in terms of USD based on a pricing table?

Also, fwiw, we have a very small change that uses two env variables: DEFAULT_OPENAI_KEY and DEFAULT_ANTHROPIC_KEY. I suppose those names could be changed. We've also got REGISTRATION_FEATURE set to false to ensure we're in control of user account creation. I'd be happy to open a PR that adds those couple of ENV variables.

[krschacht]

@matthewbennink My initial thought on cost tracking was:

• Add a column to the messages table such as estimated_cost
• At the time of new message creation we count the number of tokens and update this on the message
• Then in various places such as account settings we can estimate total $ incurred this month with a simple Message.created_after(Date.beginning_of_month).sum(estimated_cost)
• I think we'd need some cost-per-token added to our LanguageModel model
• Then within backend/anthropic & backend/open_ai when we send the API call to the server, in that moment we count how many tokens we are sending (I have the tiktoken ruby gem already in the app for doing this). At this point, the new message currently is blank but we go ahead and save a preliminary estimated_cost based on how many tokens we're sending.
• Then in get_next_message_job near the bottom where we do the final message.save we count the total number of tokens received back from the API (I think it's just Tiktoken.count(message.content_text) and += to add that onto the preliminary estimated_cost

[krschacht]

Closing this and opening #402 as a placeholder for token cost tracking