Single agent tool not being recognised by agents

[Achronus]

Hey guys,

I'm toying around with the RAG example on your website and stumbled upon a pretty nasty issue. I'm not sure if Gemini is extremely stupid or if there's a tool bug going on so wanted to drop a ticket just in case.

Full files are in a toy-script repo. Main one to look at is the rag.py file.

I've transformed the example in the docs to one that uses PydanticAI and setup everything with docker using the following commands:

mkdir postgres-data
docker run --rm -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d -v /postgres-data:/var/lib/postgresql/data pgvector/pgvector:pg17

Adding the logfire_docs.json file to the database is the easy part:

python -m toy_script.rag build

Works like a dream (note: embedding size has changed for the SentenceTransformer I'm using). Issue is that when I run:

python -m toy_script.rag search

Gemini responds with stupid things like:

• I am sorry, I cannot find information on how to configure Logfire to work with FastAPI using the available tools. My knowledge about Logfire is limited.
• I am sorry, I cannot find information on how to configure Logfire to work with FastAPI using the available tools. The provided retrieve function does not seem to have access to any documentation that would contain this information.

These are with the system prompt:

m = GeminiModel("gemini-1.5-flash", api_key=settings.GEMINI_API_KEY)
agent = Agent(
    m,
    deps_type=Deps,
    retries=3,
    system_prompt="Use the 'retrieve' tool to answer user queries.",
)

Even in logfire, tools aren't being detected or run.

I've done a few experiments and had some luck with adding an extra agent tool and asking it to use that first, like so:

# Init code + imports

@dataclass
class Deps:
    pool: asyncpg.Pool
    name: str = "Test"

m = GeminiModel("gemini-1.5-flash", api_key=settings.GEMINI_API_KEY)
agent = Agent(
    m,
    deps_type=Deps,
    retries=3,
    system_prompt="Return the users name and use the 'retrieve' tool to answer user queries.",
)

@agent.tool
def get_users_name(ctx: RunContext[Deps]) -> str:
    return ctx.deps.name

# Other code

That doesn't address the problem though. Let me know if you need anymore details!

Ryan

[samuelcolvin]

I can look soon, bit works be good to know if this happens with other models, e.g. openai?

[Achronus]

I've done a few more experiments this morning, looks like other models work fine. Sorry about that! Learned something today - always try different LLM models first 😅.

Experiments

Main questions to answer:

1. Are the tools being detected in the Agent() instance? ✔
2. Are other models having the same problem? E.g., is gemini-1.5-flash garbage? ❌

1. Are the tools being detected in the Agent() instance?

From what I can tell, yes - assuming that agent._function_tools is the storage container.

# retrieve() tool only
agent._function_tools
# => {
#  'retrieve': Tool(function=<function retrieve at 0x00000256E864FEC0>, ...)
# }

# retrieve() + get_users_name() tool
# {
#  'get_users_name': Tool(function=<function get_users_name at 0x00000256E95514E0>, ...), 
#  'retrieve': Tool(function=<function retrieve at 0x00000256E9551260>, ...)
# }

2. Are other models having the same problem? E.g., is gemini-1.5-flash garbage?

Nope. gemini-1.5-flash is garbage. Test results focusing on the first prompt:

Model	Class	Works?
gemini-1.5-flash	GeminiModel	❌
gemini-1.5-pro	GeminiModel	✔
claude-3-5-haiku-latest	AnthropicModel	✔
gpt-3.5-turbo	OpenAiModel	✔

[sydney-runkle]

I think this isn't an issue with the newer gemini models, so going to close this as resolved / as a model limitation that can be avoided. Feel free to follow up if you think this should remain open.