9 ms·

You mean like this? https://archive.org/download/dnmarchives https://archive.org/download/dnmarchives

• gwern 11y ago
  I don't think that works. It's not remotely browsable or searchable. It would be quite challenging to put these scrapes up, anyway. They're regular wget crawls with a regular directory/file structure, the problem is that there's so much material and so many files that it can be almost impossible to find what you are looking for... (Plus you need to rewrite links into relative links to make everything render properly.)
  • joshmn 11y ago
    Hmm. Now I'm thinking that I might end up using your idea (scraping the dark web) and using something like httrack[0] to do exactly that: structure. [0] https://en.wikipedia.org/wiki/HTTrack https://en.wikipedia.org/wiki/HTTrack
    • gwern 11y ago
      I once tried using HTTrack, but I found it was doing too much magic under the hood and was hard to work with. As dumb as wget is (that blacklist bug is over 12 years old now!), it at least is understandable.
      • joshmn 11y ago
        Thanks for saving me the headache :)