5 ms·

Wayback Machine was down

112 points by k-ian 6y ago

tgsovlerkhgsel 6y ago
That would explain the random issues I saw recently (within the past ~12 hours) where I asked for a page version from 2019 and got one from 2018.
deleted 6y ago
[deleted]
nikisweeting 6y ago
In the meantime, distributed archiving ftw, run your own archives with Webrecorder.io, ArchiveBox.io, SingleFile, kiwix.org, etc!
- trevyn 6y ago
  The kiwix Wikipedia-en full scrape with images has been broken for over 18 months, I think they could use some technical help. I tried running their scraper myself on a nice AWS instance and it just stalls after many days of downloading articles. Could probably use a rewrite. ;) https://sourceforge.net/p/kiwix/discussion/604121/thread/1f27659cdb/ https://sourceforge.net/p/kiwix/discussion/604121/thread/1f2... https://github.com/openzim/mwoffliner/issues/1020 https://github.com/openzim/mwoffliner/issues/1020 https://github.com/openzim/mwoffliner https://github.com/openzim/mwoffliner
  - Rebelgecko 6y ago
    I'm not sure what your use case is so maybe this isn't helpful, but Wikipedia has weekly or so database dumps that you can download, as well as static HTML (although that might be more out of date) https://en.wikipedia.org/wiki/Wikipedia:Database_download https://en.wikipedia.org/wiki/Wikipedia:Database_download
    - trevyn 6y ago
      Kiwix has a mobile app that runs a local copy of full Wikipedia (~80GB) from the device, with formatting and pictures and everything. I'm not aware of any other fully-baked solution that does this. It looks like the pictures are not available as a dump from Wikimedia; the page you link to implies that they are, but the latest dump is from 2012.
  - traverseda 6y ago
    The whole zim file infrastructure is pretty broken. I've been trying to put together a system for generating a WARC file by rendering all the wikitext content in a database dump, which is a lot more reasonable of an approach. Rendering wikitext is challenging though, since wikitext can include chunks of other wikitext, and wikitext can use some pretty complicated templating functionality. Oddly enough where I've run into the biggest issues is in weird slowdowns of the python WARCIO library that making dealing with large archives just about impossible. I haven't had time to really track that down, but if anyone want to it's pretty easy to reproduce, just try adding a few million lorum-ipsum articles and look at how far from linear time it's running. There are a lot of advantages to starting from a dump, you can provide much better tools for filtering articles, probably even provide rudimentary document classification. You can also do things like re-compress and minify images, a dump intended for a cellphone probably doesn't need 4k images. WARC is also probably a better tool for distributing web-archive type content, like wikipedia dumps. You can distribute a package of text content and image content as separate files, for example. Generally I have not been very impressed with the quality of ZIM file tooling. One disadvantage is you need to provide separate search indexing, but that's doable. I'd love to be able to get a wikimedia grant to work on this, and take on less contract work, but so far their grant process is pretty hard to follow.
    - 20after4 6y ago
      This sounds like a really cool project, maybe someone from Wikimedia will be willing to help you figure out the process. I'll try to dig up the right contacts for you. You could start by asking here: https://meta.wikimedia.org/wiki/Grants_talk:Project https://meta.wikimedia.org/wiki/Grants_talk:Project
    - nikisweeting 6y ago
      I'm actually currently working on the ZIM toolchain for Kiwix on a contract basis, so I'd be interested to hear more about your pain points, they might be something I can help out with. In general, I'd say that ZIM and WARC are not really direct competitors or solutions to the same problems, they're really for distinct use-cases. ZIM is a highly-compressed format that's designed solely for static articles and flat content, it doesn't really store headers or anything else that WARC does in order to support full request/response replaying. ZIM is optimized for storing thousands to millions of pages of homogenous content, WARC is optimized for high-fidelity collections of smaller amounts of content. If you want to help out with our efforts, feel free to DM me on Twitter @theSquashSH or reply here and I can introduce you to the ZIM people (who get grants to improve this process on the regular, and are open to hiring contract workers).
  - nikisweeting 6y ago
    I'm actually helping work on that right now, we're improving the node-libzim bindings that mwoffliner uses to write the ZIM files, and providing some additional server power to do larger archives and hopefully catch up on the backlog of wikipedia-en dumps.
- Kaiyou 6y ago
  Is there a tool that downloads every website I visit locally and then, upon revisit, shows me my local copy for instant load, but does a diff in the background with the online version and asks me to show newer version only if there are differences?
  - nikisweeting 6y ago
    Try https://github.com/WorldBrain/Memex https://github.com/WorldBrain/Memex, it has annotation and lets you review previously seen versions of a site as you're browsing. Or check out some of the other options here: - https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Projects https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Comm... - https://github.com/iipc/awesome-web-archiving https://github.com/iipc/awesome-web-archiving
username2020 6y ago
Massive layoffs today I heard.
- wideasleep1 6y ago
  Any kinks would be appreciated.
  - saagarjha 6y ago
    I think you may have had a Freudian slip ;)
    - wideasleep1 6y ago
      Indeed! Kinks still welcome. (was going to make a joke about fat-fingering, butt decided to let it lay).
- progman32 6y ago
  Source?
  - nvahalik 6y ago
    I don't believe GP means at Archive.org. But there are a number of companies letting people go.
    - username2020 6y ago
      I definitely mean at Archive.org. No links yet.
jarfil 6y ago
The Internet Archive just announced a no-waitlist book lending due to COVID-19, I'd guess their servers might not be too happy about the inrush of users. http://blog.archive.org/2020/03/24/announcing-a-national-emergency-library-to-provide-digitized-books-to-students-and-the-public/ http://blog.archive.org/2020/03/24/announcing-a-national-eme...
- mirimir 6y ago
  Right, and https://archive.org/details/nationalemergencylibrary https://archive.org/details/nationalemergencylibrary is up and responsive. So I'm guessing that they've shifted resources.
dublinben 6y ago
Now would be a great time to donate to the Internet Archive if you're able to. They can surely use the help.
- dheera 6y ago
  If only shareholders would think the same way about PG&E and other companies that could use infrastructure upgrades ...
  - dheera 6y ago
    Downvotes! Yep, this is exactly why we have crappy infrastructure.
    - robocat 6y ago
      FYI: downvotes because (0) off-topic, (a) not constructive, (b) the world is bigger than California and there’s a lot of great infrastructure elsewhere, (c) PG&E is probably about average, (d) complaining about downvotes without thinking about why people had reason to downvote you.
      - labster 6y ago
        Downvoted you only because PG&E is way worse than average. Just yesterday they agreed to plead guilty to 84 counts of manslaughter, and I could go on and on about their wrongdoing. Much related to not doing maintenance and then falsifying maintenance reports. I moved out of their service area a decade ago, still hate them. All other points are fair, though.
        brian_herman__ 6y ago
        I'm glad that you are out of their service area we need more people like you
    - deleted 6y ago
      [deleted]
thegeekpirate 6y ago
I sent them a bug yesterday where I was being blocked for "Too Many Requests" regarding an endpoint I wasn't actually using (they thought I was attempting to submit URLs using the "Save Page Now" feature), so they've been having issues across the board. This is good though, as they're now hopefully aware of some previously unknown deficiencies. Best of luck to the Archive team to get things up and running again with minimal stress!
pcdoodle 6y ago
It's been down for a few weeks for cnn.com (Can't load feb 1st to current day). I wonder if they're getting pressure from somewhere. Check it out for yourself.

112 points by k-ian 6y ago

That would explain the random issues I saw recently (within the past ~12 hours) where I asked for a page version from 2019 and got one from 2018.

deleted 6y ago

[deleted]

nikisweeting 6y ago

In the meantime, distributed archiving ftw, run your own archives with Webrecorder.io, ArchiveBox.io, SingleFile, kiwix.org, etc!

trevyn 6y ago
The kiwix Wikipedia-en full scrape with images has been broken for over 18 months, I think they could use some technical help. I tried running their scraper myself on a nice AWS instance and it just stalls after many days of downloading articles. Could probably use a rewrite. ;) https://sourceforge.net/p/kiwix/discussion/604121/thread/1f27659cdb/ https://sourceforge.net/p/kiwix/discussion/604121/thread/1f2... https://github.com/openzim/mwoffliner/issues/1020 https://github.com/openzim/mwoffliner/issues/1020 https://github.com/openzim/mwoffliner https://github.com/openzim/mwoffliner
- Rebelgecko 6y ago
  I'm not sure what your use case is so maybe this isn't helpful, but Wikipedia has weekly or so database dumps that you can download, as well as static HTML (although that might be more out of date) https://en.wikipedia.org/wiki/Wikipedia:Database_download https://en.wikipedia.org/wiki/Wikipedia:Database_download
  - trevyn 6y ago
    Kiwix has a mobile app that runs a local copy of full Wikipedia (~80GB) from the device, with formatting and pictures and everything. I'm not aware of any other fully-baked solution that does this. It looks like the pictures are not available as a dump from Wikimedia; the page you link to implies that they are, but the latest dump is from 2012.
- traverseda 6y ago
  The whole zim file infrastructure is pretty broken. I've been trying to put together a system for generating a WARC file by rendering all the wikitext content in a database dump, which is a lot more reasonable of an approach. Rendering wikitext is challenging though, since wikitext can include chunks of other wikitext, and wikitext can use some pretty complicated templating functionality. Oddly enough where I've run into the biggest issues is in weird slowdowns of the python WARCIO library that making dealing with large archives just about impossible. I haven't had time to really track that down, but if anyone want to it's pretty easy to reproduce, just try adding a few million lorum-ipsum articles and look at how far from linear time it's running. There are a lot of advantages to starting from a dump, you can provide much better tools for filtering articles, probably even provide rudimentary document classification. You can also do things like re-compress and minify images, a dump intended for a cellphone probably doesn't need 4k images. WARC is also probably a better tool for distributing web-archive type content, like wikipedia dumps. You can distribute a package of text content and image content as separate files, for example. Generally I have not been very impressed with the quality of ZIM file tooling. One disadvantage is you need to provide separate search indexing, but that's doable. I'd love to be able to get a wikimedia grant to work on this, and take on less contract work, but so far their grant process is pretty hard to follow.
  - 20after4 6y ago
    This sounds like a really cool project, maybe someone from Wikimedia will be willing to help you figure out the process. I'll try to dig up the right contacts for you. You could start by asking here: https://meta.wikimedia.org/wiki/Grants_talk:Project https://meta.wikimedia.org/wiki/Grants_talk:Project
  - nikisweeting 6y ago
    I'm actually currently working on the ZIM toolchain for Kiwix on a contract basis, so I'd be interested to hear more about your pain points, they might be something I can help out with. In general, I'd say that ZIM and WARC are not really direct competitors or solutions to the same problems, they're really for distinct use-cases. ZIM is a highly-compressed format that's designed solely for static articles and flat content, it doesn't really store headers or anything else that WARC does in order to support full request/response replaying. ZIM is optimized for storing thousands to millions of pages of homogenous content, WARC is optimized for high-fidelity collections of smaller amounts of content. If you want to help out with our efforts, feel free to DM me on Twitter @theSquashSH or reply here and I can introduce you to the ZIM people (who get grants to improve this process on the regular, and are open to hiring contract workers).
- nikisweeting 6y ago
  I'm actually helping work on that right now, we're improving the node-libzim bindings that mwoffliner uses to write the ZIM files, and providing some additional server power to do larger archives and hopefully catch up on the backlog of wikipedia-en dumps.
Kaiyou 6y ago
Is there a tool that downloads every website I visit locally and then, upon revisit, shows me my local copy for instant load, but does a diff in the background with the online version and asks me to show newer version only if there are differences?
- nikisweeting 6y ago
  Try https://github.com/WorldBrain/Memex https://github.com/WorldBrain/Memex, it has annotation and lets you review previously seen versions of a site as you're browsing. Or check out some of the other options here: - https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Projects https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Comm... - https://github.com/iipc/awesome-web-archiving https://github.com/iipc/awesome-web-archiving

username2020 6y ago

Massive layoffs today I heard.

wideasleep1 6y ago
Any kinks would be appreciated.
- saagarjha 6y ago
  I think you may have had a Freudian slip ;)
  - wideasleep1 6y ago
    Indeed! Kinks still welcome. (was going to make a joke about fat-fingering, butt decided to let it lay).
progman32 6y ago
Source?
- nvahalik 6y ago
  I don't believe GP means at Archive.org. But there are a number of companies letting people go.
  - username2020 6y ago
    I definitely mean at Archive.org. No links yet.

jarfil 6y ago

The Internet Archive just announced a no-waitlist book lending due to COVID-19, I'd guess their servers might not be too happy about the inrush of users. http://blog.archive.org/2020/03/24/announcing-a-national-emergency-library-to-provide-digitized-books-to-students-and-the-public/ http://blog.archive.org/2020/03/24/announcing-a-national-eme...

mirimir 6y ago
Right, and https://archive.org/details/nationalemergencylibrary https://archive.org/details/nationalemergencylibrary is up and responsive. So I'm guessing that they've shifted resources.

dublinben 6y ago

Now would be a great time to donate to the Internet Archive if you're able to. They can surely use the help.

dheera 6y ago
If only shareholders would think the same way about PG&E and other companies that could use infrastructure upgrades ...
- dheera 6y ago
  Downvotes! Yep, this is exactly why we have crappy infrastructure.
  - robocat 6y ago
    FYI: downvotes because (0) off-topic, (a) not constructive, (b) the world is bigger than California and there’s a lot of great infrastructure elsewhere, (c) PG&E is probably about average, (d) complaining about downvotes without thinking about why people had reason to downvote you.
    - labster 6y ago
      Downvoted you only because PG&E is way worse than average. Just yesterday they agreed to plead guilty to 84 counts of manslaughter, and I could go on and on about their wrongdoing. Much related to not doing maintenance and then falsifying maintenance reports. I moved out of their service area a decade ago, still hate them. All other points are fair, though.
      - brian_herman__ 6y ago
        I'm glad that you are out of their service area we need more people like you
  - deleted 6y ago
    [deleted]

thegeekpirate 6y ago

I sent them a bug yesterday where I was being blocked for "Too Many Requests" regarding an endpoint I wasn't actually using (they thought I was attempting to submit URLs using the "Save Page Now" feature), so they've been having issues across the board. This is good though, as they're now hopefully aware of some previously unknown deficiencies. Best of luck to the Archive team to get things up and running again with minimal stress!

pcdoodle 6y ago

It's been down for a few weeks for cnn.com (Can't load feb 1st to current day). I wonder if they're getting pressure from somewhere. Check it out for yourself.