5 ms·
Long-form factuality in large language models
- olesya1979 2y ago[dead]
- rst 2y agoHmmm... checking against external sources is an interesting idea -- but using Google as a source of ground truth is a little bit tricky, given how often these days Google itself is spitting up confabulated AI-generated crud (or other low-quality stuff).
- barfbagginus 2y agoUse books and papers from the Library of Genesis - that gives you good context, even while the search engines collapse End Google Long live The Library!
- cl42 2y agoFor those interested in using search-augmented "reasoning", I implemented something similar in Emerging Trajectories[1], an open source package that forecasts geopolitical and economic events. We extract facts[2] from various websites (Google searches, news articles, RSS feeds) and have the LLM generate a hypothesis on a metric. We're tracking the info forecasts to see how well this does for future events. For example, we're pitting the LLMs against each other to predict March 2024 CPI[3]. [1] https://emergingtrajectories.com/ https://emergingtrajectories.com/ [2] Sample code: https://github.com/wgryc/emerging-trajectories/blob/main/emergingtrajectories/factsforecaster.py https://github.com/wgryc/emerging-trajectories/blob/main/eme... [3] https://emergingtrajectories.com/a/statement/28 https://emergingtrajectories.com/a/statement/28