6 ms·

That has been proven not to work and a variation of the "make no mistakes" meme. What about the lawyer doing the actual work, he/she is being paid for?

Iolaum 23d ago
?! It does work but you need separate (sub)agents (and potentially loop it until the reviewer agent finds nothing). You also need to set it up a bit so all citations mentioned are easy to identify. Ideally this is also done programmatically, ie parse text find citations and ensure they say what you think they say (llm part).
- johnbarron 23d ago
  It does not work because the errors compound and also are most times correlated. "Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels" - https://arxiv.org/abs/2605.29800 https://arxiv.org/abs/2605.29800 "From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaboration" - https://arxiv.org/abs/2605.29800 https://arxiv.org/abs/2605.29800 "Large Language Models Cannot Self-Correct Reasoning Yet" - https://arxiv.org/abs/2310.01798 https://arxiv.org/abs/2310.01798 "Correlated Errors in Large Language Models" - https://proceedings.mlr.press/v267/kim25e.html https://proceedings.mlr.press/v267/kim25e.html "Partially Correlated Verifier Cascades in LLM Harnesses: Concave Log-Odds, Polynomial Reliability, and Blind-Spot Ceilings" - https://arxiv.org/abs/2607.13918 https://arxiv.org/abs/2607.13918 "Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysis" - https://arxiv.org/abs/2603.21454 https://arxiv.org/abs/2603.21454
- roryirvine 22d ago
  Sure, and there are existing legal research tools which would help here (LexisNexis, amongst others, will happily sell you a suitable service). Constraining an agent by forcing it to pass some form of external validation is one of the best ways we have of minimising hallucinations. But is a lawyer who is this lazy and this dishonest ever likely to bother doing that?

Iolaum 23d ago

?! It does work but you need separate (sub)agents (and potentially loop it until the reviewer agent finds nothing). You also need to set it up a bit so all citations mentioned are easy to identify. Ideally this is also done programmatically, ie parse text find citations and ensure they say what you think they say (llm part).

johnbarron 23d ago
It does not work because the errors compound and also are most times correlated. "Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels" - https://arxiv.org/abs/2605.29800 https://arxiv.org/abs/2605.29800 "From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaboration" - https://arxiv.org/abs/2605.29800 https://arxiv.org/abs/2605.29800 "Large Language Models Cannot Self-Correct Reasoning Yet" - https://arxiv.org/abs/2310.01798 https://arxiv.org/abs/2310.01798 "Correlated Errors in Large Language Models" - https://proceedings.mlr.press/v267/kim25e.html https://proceedings.mlr.press/v267/kim25e.html "Partially Correlated Verifier Cascades in LLM Harnesses: Concave Log-Odds, Polynomial Reliability, and Blind-Spot Ceilings" - https://arxiv.org/abs/2607.13918 https://arxiv.org/abs/2607.13918 "Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysis" - https://arxiv.org/abs/2603.21454 https://arxiv.org/abs/2603.21454
roryirvine 22d ago
Sure, and there are existing legal research tools which would help here (LexisNexis, amongst others, will happily sell you a suitable service). Constraining an agent by forcing it to pass some form of external validation is one of the best ways we have of minimising hallucinations. But is a lawyer who is this lazy and this dishonest ever likely to bother doing that?