5 ms·
Do database dumps of it still exist? Maybe it’s time to just download it and Wikipedia, then point an LLM / vectordb at it.
by hedora 22d ago
Do database dumps of it still exist?
Maybe it’s time to just download it and Wikipedia, then point an LLM / vectordb at it.
- novok 22d agoYes they do, there are archives that are up to date for the month.
- toomuchtodo 22d agoSee if https://github.com/Tryhard-cs/reddit-download-tool https://github.com/Tryhard-cs/reddit-download-tool works, the readme enumerates the data sources it pulls from. Perhaps the way to solve for X is having a scraper serve the data via torrent and/or IPFS vs centralized systems like nitter and xcancel.