Add meta prompt to ensure model is not trying to use tools he can't use

[tdraier]

fixes: https://github.com/dust-tt/tasks/issues/1162

Description

This is to prevent an assistant from using tools even if it sees in the conversation other agents having access to.

Risk

should be safe, but can be rollbacked.

Deploy Plan

deploy front

[fontanierh]

Kind of bearish to add that for all models. I think only some models have this issue no ? Also, did you confirm that this actually reduce the likelihood of this happening ?

[fontanierh]

Also bearish to add "important:" in the prompt. All meta-prompts we add are important

[fontanierh]

we try to reduce the amount of meta-prompting we do, as these can break / change the behavior of assistants

[spolu]

Aligned with @fontanierh comments.

How wide-spread is this behavior? Is it specific to a model? This would be indeed useful to know before we add a potentially very long meta prompt.

Technically this is a model bug in a sense as we already clearly specific which tool a model has access to. We should first and foremost report examples to the associated model providers.

[spolu]

Looking at the initial eng runner issue the problem is mostly when the model has no more tools to use because maxToolUserPerRun is exhausted.

Anthropic has update a few things on the parameter we can pass in case of tool use we should definitely dig there if there isn't a parameter that can guarantee us no tool use (which works well on OpenAI I believe).

[spolu]

(commented on the parent eng runner issue)

[tdraier]

It seems the issue is only with claude indeed. @claude has 0 tools, but it's trying to take tools from the conversation messages for itself and maxToolUserPerRun is immediately exhausted - If it's only claude we could add a prompt only there.
In the end yes the prompting fixes the issue, it does not try to use a tool.

[spolu]

Can we before we go the meta prompt route share an example to Anthropic for them to comment on what is the best path forward? Also look at the options we have in the API in that case as we wrote the integration before some changes were made on that front AFAICT.

[fontanierh]

It seems the issue is only with claude indeed. @claude has 0 tools, but it's trying to take tools from the conversation messages for itself and maxToolUserPerRun is immediately exhausted

Then your fix doesn't address this right, because the new prompt is only added when the assistant has tools, which is the case where we don't actually want the prompt

NVM me there is the else -- maybe we should only keep the else part then ? and maybe only for claude ?
Could be a new noToolsMetaPrompt on the ModelConfiguration (if we decide to actually go the meta-prompt route)

[tdraier]

Here's the request done to anthropic :

req.json

There's no tool defined apart from the dummy_do_not_use

Then in the stream I receive a :

{
  "type": "content_block_start",
  "index": 1,
  "content_block": {
    "type": "tool_use",
    "name": "query_tables",
    "input": {},
    "id": "toolu_012FDrTWues3eo2xjMgzqaZA"
  }
}

and :

{
  "type": "tool_use",
  "name": "query_tables",
  "input": {},
  "id": "toolu_012FDrTWues3eo2xjMgzqaZA"
}

Looks very weird to me - do you confirm it's not expected ?

[fontanierh]

Yeah it's a know issue, Claude sometimes try to steal tools from previous messages.
Also, once you have a convo with tools, Anthropic enforces (or at least used to enforce) that at least one tool is passed, hence the dummy tool don't use.

But I think some things have changed in their API, so would be interesting to check if:

• dummy too is still required
• there's maybe a param that we can pass to prevent tools use altogether

Otherwise, adding a metaprompt in case there are tools in the convo but Claude isn't allowed to use them might be the way to go

[spolu]

The core of the problem might be the dummy tool (which puts the model in tool use mode and make it possible for a tool to be emitted even if stolen). It is possible Anthropic fixed this bug.

Can you try not passing the dummy tool when conversations previously used tools but won't use tool for the current turn. Anthropic used to require the presence of at least one tool hence the dummy tool being injected. If we can get rid of it I presume Claude will not output any tool as it will be prevented from it at a lower level

[tdraier]

We can't remove the tools section, as anthropic sees some "tool_use" in the messages list :
[invalid_request_error] Requests which include tool_useortool_result blocks must define tools..

So another option for anthropic could be to remove all tool_use / tool_result from the conversation .. ? It won't comlpain and won't try to use any tool.

[fontanierh]

I think the new version of Claude might fix the issue.

another option for anthropic could be to remove all tool_use / tool_result from the conversation .. ? It won't comlpain and won't try to use any tool.

Not really an option as the assistant needs to see the tools results to answert the question effectively

[fontanierh]

We can't remove the tools section, as anthropic sees some "tool_use" in the messages list :
[invalid_request_error] Requests which include tool_useortool_result blocks must define tools..

@tdraier just to be extra sure, that's also true if we keep it but pass an empty array ?

[spolu]

If/once we confirm it's still a bug, let's raise it again to the team 👍

[spolu]

If there is no change, I would add the meta-prompt in the dummy-tool versus the assistant => in the dummy tool dscription add a setence saying no other tool should be used either

[tdraier]

If there is no change, I would add the meta-prompt in the dummy-tool versus the assistant => in the dummy tool dscription add a setence saying no other tool should be used either

yes, same error :

"AnthropicError: [invalid_request_error] Requests which include tool_use or tool_result blocks must define tools."

[tdraier]

I've checked with the latest model, we still have the same behaviour.
Also tried to move the instruction in the dummy-tool description, it seems to be ignored.
I'll check on how to put a metaprompt only for this specific case