6 ms·
I had a hunch that opus 4.7 hedged more than other models - and it turns out it's true model total_claims hedged_count hedged_pct cla
by faxmeyourcode 4mo ago
I had a hunch that opus 4.7 hedged more than other models - and it turns out it's true
model total_claims hedged_count hedged_pct
claude-opus-4-7 1000 451 45.1
sonar-pro 1000 391 39.1
gpt-5.4 1000 277 27.7
gemini-3-retrieval 1000 129 12.9
gemini-3-pro 1000 60 6.0
datasette query here
https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwillison.net%2Fstatic%2Fcors-allow%2F2026%2Flenz-llm-disagreement.csv#/data?sql=WITH+verdicts+AS+%28%0A++SELECT+claim_id%2C+%27gpt-5.4%27+AS+model%2C+%22gpt-5.4_verdict%22+AS+verdict%0A++FROM+%22lenz-llm-disagreement%22%0A%0A++UNION+ALL%0A++SELECT+claim_id%2C+%27claude-opus-4-7%27%2C+%22claude-opus-4-7_verdict%22%0A++FROM+%22lenz-llm-disagreement%22%0A%0A++UNION+ALL%0A++SELECT+claim_id%2C+%27gemini-3-pro%27%2C+%22gemini-3-pro_verdict%22%0A++FROM+%22lenz-llm-disagreement%22%0A%0A++UNION+ALL%0A++SELECT+claim_id%2C+%27gemini-3-retrieval%27%2C+%22gemini-3-retrieval_verdict%22%0A++FROM+%22lenz-llm-disagreement%22%0A%0A++UNION+ALL%0A++SELECT+claim_id%2C+%27sonar-pro%27%2C+%22sonar-pro_verdict%22%0A++FROM+%22lenz-llm-disagreement%22%0A%29%0ASELECT%0A++model%2C%0A++COUNT%28*%29+AS+total_claims%2C%0A++SUM%28CASE+WHEN+verdict+NOT+IN+%28%27True%27%2C+%27False%27%29+AND+verdict+IS+NOT+NULL+THEN+1+ELSE+0+END%29+AS+hedged_count%2C%0A++ROUND%28%0A++++100.0+*+SUM%28CASE+WHEN+verdict+NOT+IN+%28%27True%27%2C+%27False%27%29+AND+verdict+IS+NOT+NULL+THEN+1+ELSE+0+END%29+%2F+COUNT%28*%29%2C%0A++++1%0A++%29+AS+hedged_pct%0AFROM+verdicts%0AGROUP+BY+model%0AORDER+BY+hedged_count+DESC%3B https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil...
- kostaj 4mo agoThis is in line with my observations and tests as well. Also supported by the distribution of the verdicts across the 4-buckets -- Gemini uses the middle buckets (Mostly True and Misleading) much less often - 6% combined for Gemini w/o search. And Opus uses them the most - 45% combined. Looks like Gemini is calibrated to be confident and Opus to be careful.