5 ms·
Where does ArchiveTeam find all the reddit posts and comments to archive? Do they have a script automatically going through the "New" section or are they findin
by ajsfoux234 5y ago
Where does ArchiveTeam find all the reddit posts and comments to archive? Do they have a script automatically going through the "New" section or are they finding posts through Google or link crawling?
- giantrobot 5y agoBesides their Archive Warrior distributed crawler I imagine PushShift[0] is probably a starting point for them. [0] https://files.pushshift.io/ https://files.pushshift.io/
- Gigachad 5y agoIn general, ArchiveTeam has scripts which hit random links to see if there is any content. They have coordination servers which share info on which slugs have been checked before to avoid duplicate effort.