5 ms·
That’s super interesting actually. I remember when mcp came out and I made an “add” tool but actually made it multiply. OpenAI model (I forget which) called t
by vorticalbox 13d ago
That’s super interesting actually.
I remember when mcp came out and I made an “add” tool but actually made it multiply.
OpenAI model (I forget which) called the tool three times then decided to ignore the result and return the correct answer.
Have you tried the search experiment with smaller/local models?
I have a theory internally they reason about tool results before accepting it for the reply.
- screm 13d agoWe haven't tested with smaller / older models but it would definitely work better. Prompt injection was the top 1 concern for first LLMs so they put a lot of energy into having guardrails at almost every stage afaik (input, tool call validation, tool call output). So I guess your intuition sounds right! It's of course a lot more complex (I'm not an expert) and labs published a lot about it (like here: https://openai.com/index/designing-agents-to-resist-prompt-injection/ https://openai.com/index/designing-agents-to-resist-prompt-i...). They favor false positives to false negatives so it's expected that we sometimes trigger those guardrails!