6 ms·
Last time I downloaded Wikipedia, it was 4.5GB. If I were to knock up this hack, I would definitely scrape pages instead.
by tim_iles 15y ago
Last time I downloaded Wikipedia, it was 4.5GB. If I were to knock up this hack, I would definitely scrape pages instead.
- blahedo 15y agoOn-demand page scrape + memoisation is almost certainly a win here. Even if thousands of people are hitting this, a lot will choose some of the same queries (I'm sure Kevin Bacon and xkcd and philosophy are in there a bunch), especially in the tails of the paths (Latin, Mathematics, ...)