AutotitleConversationJob not working for Ollama

[mattlindsey]

Conversations with Ollama don't get their title updated from 'New chat' because this job returns before executing. This is because Ollama does not require an API key.
If this line is removed, AutotitleConversationJob works fine. Is the line definitely required? Or how should we fix it?

hostedgpt/app/jobs/autotitle_conversation_job.rb

Line 10 in b8c9c52

return false if @conversation.assistant.api_service.effective_token.blank? # should we use anthropic key if that's all the user has?

[mattlindsey]

There's a workaround for now by putting something in Token in the Settings:

[krschacht]

@mattlindsey Hmm, I see what you're saying. The reason I added this check is before we had this job failing and it would keep retrying so I was attempting to gracefully short circuit.

It's notable that the app does not cleanly handle the case where a token is not required. In fact, I just searched the code base and found this line:

hostedgpt/app/services/ai_backend/open_ai.rb

Line 20 in da4b146

raise ::OpenAI::ConfigurationError if assistant.api_service.requires_token? && assistant.api_service.effective_token.blank?

This makes it seem like the open_ai backend should not work at all if the effective token is blank. But you're saying this LLM works with a blank token?

[mattlindsey]

Yeah. If you just download the Ollama app and run it you don't need a key I think. If you comment out that line at least, it works fine with no key entered in Settings.

You can just do this with no key:

curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "llama3.1",
        "messages": [
            {
                "role": "user",
                "content": "Hello!"
            }
        ]
    }'

[krschacht]

Okay so it sounds like what we need to do is add official support for “no key”. Maybe for the services setting screen we need a checkbox beneath token which says “No token needed” and if the user checks that, it greys out the token input and we save this boolean state. Or if we wanted to get clever about it we could try to differentiate between a token value of nil and a token value of ””. Maybe we don’t add a boolean column but we instead add a visual checkbox which sets the value of that textbox to “” whereas if the textbox is left empty it gets set to nil? That would take some testing because I don’t know how rails controllers & models currently save empty inputs. I don’t know which would be better. It may just be easier to add a new column possibly named token_unnecessary or something like that.

[krschacht]

Fixed by #583

[krschacht]

Thanks @mattlindsey !