6 ms·
Two of the five models used (Gemini+Search and Sonar Pro) have retrieval capabilities and used search when classifying the claims. The disagreement between them
by kostaj 4mo ago
Two of the five models used (Gemini+Search and Sonar Pro) have retrieval capabilities and used search when classifying the claims. The disagreement between them is still quite significant - 42%.
- simonw 4mo agoHere are those disagreements: https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwillison.net%2Fstatic%2Fcors-allow%2F2026%2Flenz-llm-disagreement.csv#/data?sql=select+atomic_claim%2C+%22gemini-3-retrieval_verdict%22%2C+%22sonar-pro_verdict%22+from+%5Blenz-llm-disagreement%5D+%0Awhere+%22gemini-3-retrieval_verdict%22+%21%3D++%22sonar-pro_verdict%22 https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil... One example: Researchers estimate that the average person ingests about 5 grams of plastic per week, which is approximately the weight of a credit card. Gemini retrieval: Misleading Sonar pro: Mostly True
- jeffbee 4mo agoInternally the statement is perfectly true: some researchers did estimate this, and the credit card is a fair proxy for a 5g mass. Was the research flagrantly incorrect? Yes. But that does not affect the truth of the statement.