4 ms·

The blog post appears to get confused and devotes its entire second half to pitching Keenable itself. If the idea is to build credibility for the new benchmark,

by mpalmer 20d ago

The blog post appears to get confused and devotes its entire second half to pitching Keenable itself. If the idea is to build credibility for the new benchmark, this maybe was not the best choice. Search engines with their own indexes can retrieve documents no one else has. That means your agents will be able to answer questions previously unanswerable. Besides the clear AI smell, this nonsensical claim also plainly contradicts the methodology's key evaluation claim that the quality of an engine's results should be measured against how much it overlaps with the reranked aggregate of the other engines. The benchmark thus seemingly values an engine's ability to "answer unanswerable questions" at zero. Search engines should improve from being used in a way where the quality improvement is not an engineering push but a property of the architecture. That's a learning machine, and it's the only kind of system that can adapt to agents. Yeah? Care to cite anything for that?

matt4711 20d ago
> Yeah? Care to cite anything for that? This automatically optimizing for clicks using ML is the main way google and other "human" focused search engines have been improving for 20 years. Not sure what citation is needed here.
matt4711 20d ago
> Besides the clear AI smell, this nonsensical claim also plainly contradicts the methodology's key evaluation claim that the quality of an engine's results should be measured against how much it overlaps with the reranked aggregate of the other engines. The benchmark thus seemingly values an engine's ability to "answer unanswerable questions" at zero. The engine itself is part of the reranked aggregate so if it finds something useful and everybody else does not it gets full credit.
phoenix120 18d ago
> The blog post appears to get confused and devotes its entire second half to pitching Keenable itself. If the idea is to build credibility for the new benchmark, this maybe was not the best choice. The exact sentence you cite is an example that I've tried everything I could to avoid that. In one of the original revisions it was "We can retrieve very unique documents no one else has", not "Search engines with their own indexes can retrieve documents no one else has". > Besides the clear AI smell, this nonsensical claim also plainly contradicts the methodology's key evaluation claim that the quality of an engine's results should be measured against how much it overlaps with the reranked aggregate of the other engines. The benchmark thus seemingly values an engine's ability to "answer unanswerable questions" at zero. It is fun to hear "AI smell" accusations about these specific sentences. I do use Claude to fix my grammar mistakes and give editorial opinions, mostly because I'm not a native English speaker. And as for these specific sentences, I even had a session where Claude fixed an obvious mistake in my writing: https://snipboard.io/rkm4LD.jpg https://snipboard.io/rkm4LD.jpg There is no contradiction to the methodology. We use ultimate engine to calculate IDCG, but that doesn't mean that the methodology favors the overlap. 5 relevant unique results would produce exactly the same nDCG as 5 relevant non-unique results. The main part of the benchmark (the one that measures quality) doesn't care about result uniqueness (if it is not about full URL duplicates). > Yeah? Care to cite anything for that? The closest source is probably LRAT (https://arxiv.org/abs/2604.04949 https://arxiv.org/abs/2604.04949), it is cited in the benchmark, we used their trajectories in AgenticRare.

by mpalmer 20d ago

matt4711 20d ago

> Yeah? Care to cite anything for that? This automatically optimizing for clicks using ML is the main way google and other "human" focused search engines have been improving for 20 years. Not sure what citation is needed here.

matt4711 20d ago

> Besides the clear AI smell, this nonsensical claim also plainly contradicts the methodology's key evaluation claim that the quality of an engine's results should be measured against how much it overlaps with the reranked aggregate of the other engines. The benchmark thus seemingly values an engine's ability to "answer unanswerable questions" at zero. The engine itself is part of the reranked aggregate so if it finds something useful and everybody else does not it gets full credit.

phoenix120 18d ago

> The blog post appears to get confused and devotes its entire second half to pitching Keenable itself. If the idea is to build credibility for the new benchmark, this maybe was not the best choice. The exact sentence you cite is an example that I've tried everything I could to avoid that. In one of the original revisions it was "We can retrieve very unique documents no one else has", not "Search engines with their own indexes can retrieve documents no one else has". > Besides the clear AI smell, this nonsensical claim also plainly contradicts the methodology's key evaluation claim that the quality of an engine's results should be measured against how much it overlaps with the reranked aggregate of the other engines. The benchmark thus seemingly values an engine's ability to "answer unanswerable questions" at zero. It is fun to hear "AI smell" accusations about these specific sentences. I do use Claude to fix my grammar mistakes and give editorial opinions, mostly because I'm not a native English speaker. And as for these specific sentences, I even had a session where Claude fixed an obvious mistake in my writing: https://snipboard.io/rkm4LD.jpg https://snipboard.io/rkm4LD.jpg There is no contradiction to the methodology. We use ultimate engine to calculate IDCG, but that doesn't mean that the methodology favors the overlap. 5 relevant unique results would produce exactly the same nDCG as 5 relevant non-unique results. The main part of the benchmark (the one that measures quality) doesn't care about result uniqueness (if it is not about full URL duplicates). > Yeah? Care to cite anything for that? The closest source is probably LRAT (https://arxiv.org/abs/2604.04949 https://arxiv.org/abs/2604.04949), it is cited in the benchmark, we used their trajectories in AgenticRare.