kostaj

7 ms·

1.
by kostaj 1mo ago
I'm Kosta, co-author of the research and founder of Lenz. Our goal is to assess to what extent the frontier LLMs are interchangeable as verifiers of factual claims. In this revision v1.1 of the research: improved methodology, latest fr
2.
Show HN: Research on LLM Disagreement on Factual Claims(zenodo.org)
2 points by kostaj 1mo ago|1 comments
3.
by kostaj 2mo ago
Yes, you can check out Lenz without the API here: https://lenz.io/verify Also examples of claims other people have verified with Lenz: https://lenz.io/library
4.
Lenz – A fact-checking API for AI-generated content(lenz.io)
5 points by kostaj 2mo ago|2 comments
5.
by kostaj 4mo ago
Some models struggle combining JSON schema and web search capabilities.
6.
by kostaj 4mo ago
Good point. Will publish in the next version also the results with a prompt that allows the models to "think out loud" before providing the final verdict.
7.
by kostaj 4mo ago
Awesome. We do plan to human-label the 1,000 claims and then compare Lenz' performance vs the 5 models. We've done some limited internal research with 150 claims, but more are needed for statistical significance.
8.
by kostaj 4mo ago
Agree that some of the claims are forward-looking. The messiness of the real-world and real-user fact checks. No ground-truth verdicts are provided or used in the study though. It only measures the level of agreement between the selected mo
9.
by kostaj 4mo ago
Good idea about publishing intra-model variance data! Will include in the next version. Even if we put aside the two middle buckets (Mostly True and Misleading), that are somewhat subject to interpretation and hedging: On 21% of the claims
10.
by kostaj 4mo ago
Good point. Processing the substance of the answer might be too labor-consuming (1,000 claims x 5 models), but "thinking out loud" might improve the quality of the answers indeed. And we can still force/ask them to respond wi
11.
by kostaj 4mo ago
This is in line with my observations and tests as well. Also supported by the distribution of the verdicts across the 4-buckets -- Gemini uses the middle buckets (Mostly True and Misleading) much less often - 6% combined for Gemini w/o
12.
by kostaj 4mo ago
Indeed. For algorithms and coding, my personal routine nowadays is to review every detailed plan with Opus 4.7 and GPT-5.5. They tend to find very different type of gaps.
13.
by kostaj 4mo ago
Agree that True and Mostly True might be very close and could be a calibration difference. Misleading and False, as well. A better headline number might be the 34% claims with substantial or polar-opposite verdicts.
14.
by kostaj 4mo ago
Agree. Human experts also struggle agreeing on this type of claims. The inter-annotator agreement on the verdicts on the AVeriTeC corpus across 50 organizations is κ=0.619 - substantial but well short of perfect.
15.
by kostaj 4mo ago
Agree with @pjdesno, that the 34% substantive or polar disagreement might be a better headline number. Or even the 21% polar disagreement (at least one model True, and at least one model False), which is still high for many real-world appli
16.
by kostaj 4mo ago
That's a valid point. During the preliminary research, we did try also more explicit prompts (with explanation for each of the 4 buckets), as well as a five-bucket rubric (with Abstain option). Will show in a follow-up paper how the co
17.
by kostaj 4mo ago
Quick note on the second effect - how LLMs reduce that to a four-category judgment: On 21% of the claims at least two models provide polar-opposite verdicts (at least one model False, and at least one model True). This might be a better mea
18.
by kostaj 4mo ago
Agree about comparing models with and without search capabilities. Even the two models with search capabilities (Sonar Pro and Gemini) agree only on 58% of the claims.
19.
by kostaj 4mo ago
Will add a human-labelled expected response and measure against it in a follow up research. This one only captures the disagreement between the models, but not which model is write/wrong.
20.
by kostaj 4mo ago
The reason for the "No explanations, no qualifiers" in the prompt was to force the models to put the claim in one of the four buckets and answer with the bucket name only. It's a pure quantitive analysis (first in a series) a
21.
by kostaj 4mo ago
@john_strinlai @gcr, depends on the application. In many cases an "I don't know" answer is indeed better than a forced answer. But in many production systems, LLMs generate content/response anyway. Although inheriting th
22.
by kostaj 4mo ago
Search was enabled for 2 of the 5 models -- Gemini and Sonar Pro. The disagreement between them is still high - different verdict on 42% of the claims. Fully agree, that some of those claims are hard to classify for a human as well -- the r
23.
by kostaj 4mo ago
Two of the five models used (Gemini+Search and Sonar Pro) have retrieval capabilities and used search when classifying the claims. The disagreement between them is still quite significant - 42%.
24.
by kostaj 4mo ago
Indeed. I prompted each model ones, plus one retry on errors. Very good point to measure the inter-model disagreement! Will add in the next version. Section "4.2 Agreement w/ peer majority" shows the level of agreement of eac
25.
by kostaj 4mo ago
GPT-5.4 and Opus 4.7, specifically, agree between themselves on 65% of the claims - 95% CI 62–68%. I.e., in at least 35% of the claims, one of the two models is wrong under this 4-bucket rubric.
26.
by kostaj 4mo ago
Tried initially with a fifth bucket, Abstain. It was actually heavily used by some of the models. But it felt as if they are using this to "avoid" some of the hard questions, and we dropped this bucket to force them to provide a v
27.
by kostaj 4mo ago
Yes, inter-human-annotator disagreement is also high on similar type of questions (AVeriTeC) - inter-panel agreement: κ=0.619. Tried giving the models a fifth option, Abstain, but some models seem to use it to "avoid answering hard que
28.
by kostaj 4mo ago
This paper covers only the disagreement between models and established only the floor of the error, based on the disagreement, but not which model is better. Planning to follow up with another study to benchmark against human-labelled verdi
29.
by kostaj 4mo ago
Two of the models used have retrieval capabilities and can access newer information via search. Valid point for the other 3 models. All of the claims were submitted after February 15, 2026, but many of them were not time-sensitive (e.g. did
30.
by kostaj 4mo ago
Btw, sometimes that do that too -- all agree on the wrong answer.

More ›