7 ms·

Show HN: Tesoro – Personal internet archive

174 points by agamble 9y ago

pbhjpbhj 9y ago
How are you handling copyright infringement? Outside USAs Fair Use terms this looks like pretty blatant infringement.
- iso-8859-1 9y ago
  What is there to handle? You take down stuff when you get an email? Most users of this will be so small that they'll never get noticed. Maybe they won't even be online, how are you going to know you were infringed? Maybe the crawler allows for spoofing the user-agent.
  - pbhjpbhj 9y ago
    So, ignoring it basically. If a person in the UK uses your service you're committing contributory infringement for commercial purposes, AFAICT. Moreover, the ECD has different protections than DMCA. Particularly getting a takedown notice isn't required. >Maybe the crawler allows for spoofing the user-agent. // As a tort you only need to get a preponderance of evidence. IP of the crawler that made the copy puts the owner of that IP in court for contributory infringement, no? If you make copies of parts of BBC sites and serve those copies from your server how is that not copyright infringement by you?? FWIW I like the service and do not like the copyright regime as it stands, particularly how UK law lacks the breadth of liberties of Fair Use.
- brianobush 9y ago
  Probably in the same manner as archive.org
Piskvorrr 9y ago
That's just as much "my own" as The Internet Archive: a website Out There somewhere. Worse, it's much more likely to rot and disappear than archive.org. Now, if I could run this locally... (Yes, yes, `wget --convert-links`, I know. Not quite as convenient, though.)
- agamble 9y ago
  OP here. The internet archive is great, but it's not so awesome if there's some ephemeral content you need to save right away, like Tweets or social media posts. Being able to trigger an archive immediately let's you save temporary content such as that which is more prone to deletion. I'm going to build a Chrome extension to click and make cloud copy of the page you're on, hopefully that will make it seem more personally controllable. Do you think being able to download the archive locally would be useful?
  - detaro 9y ago
    You can trigger the Internet Archive manually as well.
    - agamble 9y ago
      Oh neat didn't realise that, my mistake :)
      - Sophira 9y ago
        To be fair, the easiest way to do it (to my knowledge) is by direct URL entry - for example (replace 'hxxps' with 'https' for this - I didn't want crawlers to pick this up): hxxps://web.archive.org/save/https://tesoro.io/ https://tesoro.io/
  - dschep 9y ago
    So like another toplevel commenter asked. Why build this or use this instead of archive.is? And there are already multiple extensions available for chrome for it ;) I agree with GP here, that anything billed as "My own internet archive" should be run on my computer. Not some one elses.
  - jtrip 9y ago
    A local download only increases the redundancy. Tesoro keeps a copy, and the user keeps a local copy that they can also use however they want. A bit like keeping newspaper clippings, that are found decades later by some relative to be then posted on social websites as something interesting.
zichy 9y ago
So this is like archive.is, but I can't search through archived sites?
lozzo 9y ago
it would be nice to have a bit of explanation on how it works and why we can be confident that we can rely upon it
- agamble 9y ago
  OP here. Definitely, great idea :) Briefly: Sites are archived using a system written in Golang and uploaded to a Google Cloud bucket. More: The system downloads the remote HTML, parses it to extract the relevant dependencies (<script>, <link>, <img> etc) and then downloads these as well. Tesoro is even parsing CSS files to extract the url('...') file dependencies from here as well, meaning most background images and fonts should continue to work. All dependencies (even those hosted at remote domains) are downloaded and hosted with the archive, meaning the src attributes on the original page tags are wrangled to support the new location. The whole thing is hosted on GCP Container Engine and I deploy with Kubernetes. I'll write up a more comprehensive blog post in some time, which portion of this would you like to hear more about?
  - 19eightyfour 9y ago
    The issue is cost. Your costs are disk space for people's archives, instances for people's use, and bandwidth for the fetches and crawls and access. How can you pay for this if it's free? It's unreliable unless its financially viable.
    - agamble 9y ago
      Totally right, great observation :) For now it's a free service with a single rate-limited form. Now it's time to work on adding specialty features that are worth paying for.
  - Faaak 9y ago
    How about avoiding redundancies ? Are same CSS files cached twice or referenced by their hash ? The page URI is a bit obscure though. I think a tresoro.io/example.tld/page/foobar/timestamp would look good. What about big media content and/or small differences between them ?
akerro 9y ago
Nice, post it on https://www.reddit.com/r/DataHoarder/ https://www.reddit.com/r/DataHoarder/ They will love it!
smoyer 9y ago
It's not mine unless it's running on my own servers or computer - I created a really rough version of this several years ago that is saved to my computer (and from there into box).
- pbhjpbhj 9y ago
  I adapted a bash script someone posted here, it uses Firefox bookmarks (pages.sqlite). Cron runs the script and downloads every page I've bookmarked that month (after some filtering). I don't use it often but sometimes I'll awk-grep it; I'm a hoarder in real life too!
  - gkya 9y ago
    Would you mind posting it?
    - pbhjpbhj 9y ago
      See sibling comment.
  - kchr 9y ago
    Please share!
    - pbhjpbhj 9y ago
      See sibling comment.
  - dddw 9y ago
    indeed interested in this too
    - pbhjpbhj 9y ago
      https://www.gwern.net/Archiving%20URLs https://www.gwern.net/Archiving%20URLs was the initial idea and base for my code. I see now he's updated his page considerably - IIRC - from what was there before. My normal medium is clay so I'd rely on his stuff rather than mine (linked below). * https://gist.github.com/pbhj/6636d0908d0d11885809a2545b138694 https://gist.github.com/pbhj/6636d0908d0d11885809a2545b13869... main script * https://gist.github.com/pbhj/4dedca1e980d6a102433403c0f435524 https://gist.github.com/pbhj/4dedca1e980d6a102433403c0f43552... filter script
dbz 9y ago
This is pretty cool. I have a chrome extension that let's you view the cached version of a web page [1]. Would I be able to use this through an API? I currently support Google Cache, WayBack Machine, and CoralCDN, but Coral doesn't work well and I'd like to replace it with something else. [1] https://chrome.google.com/webstore/detail/cmmlgikpahieigpcclckfmhnchdlfnjd https://chrome.google.com/webstore/detail/cmmlgikpahieigpccl...
- agamble 9y ago
  OP here. Yup, API and chrome extension are next on the feature list. :)
Retr0spectrum 9y ago
Is this any different to archive.is? If I want my own archive, Ctrl+S in Firefox usually works fine for me.
venning 9y ago
Thoughts: I like the look. Very clean. I like how fast it's responding; better than archive.org (though, obviously, they have different scaling problems). "Your own internet archive" might be overselling it, as other commenters have pointed out; the "Your" feels a bit misleading. I think "Save a copy of any webpage." gives a better impression, which you use on the site itself. The "Archive!" link probably shouldn't work if there's nothing in the URL box. It just gives me an archive link that errors. Example: [1] Using it on news.YC as a test gave me errors with the CSS & JS [2]. This might be due to the fact that HN uses query parameters in their CSS and JS, which repeat in the tesoro URL, which you may not be parsing correctly. Maybe have something in addition to an email link for submitting error reports like the above, just cause I'd be more likely to file a GitHub issue (even if the repo is empty) than send a stranger an email. As other commenters have pointed out, archive.is also does this, and their longevity helps me feel confident that they'll still be around. Perhaps, if you wish to differentiate, offer some way for me to "own" the copy of the page, like downloading it or emailing it to myself or sharing it with another site (like Google Docs or Imgur) to leverage redundancy, or something like that. Just a thought. All in all, nice Show HN. EDIT: You also may want to adjust the header to work properly on mobile devices. Still though, nice job. Sorry if I'm sounding critical. [1] https://archive.tesoro.io/320b55cc9b78e271c94716ee23554da8 https://archive.tesoro.io/320b55cc9b78e271c94716ee23554da8 [2] https://archive.tesoro.io/a7bf03e247224bc3b4e5a7c1f2ad42b1 https://archive.tesoro.io/a7bf03e247224bc3b4e5a7c1f2ad42b1
- agamble 9y ago
  Thanks! These are great comments - I'll look into the issue with saving Hacker News CSS + JS.
unicornporn 9y ago
In what way could this considered to be “your own internet archive”? I see no way to register a user and save pages to a collection. If you really want to create your own archive, set up a Live Archiving HTTP Proxy[1], run SquidMan [2] or check out WWWOFFLE[3]. If you want something simpler, have a look at Webrecorder[4] or a paid Pinboard account with the “Bookmark Archive”[5]. [1] http://netpreserve.org/projects/live-archiving-http-proxy/ http://netpreserve.org/projects/live-archiving-http-proxy/ [2] http://squidman.net/squidman/index.html http://squidman.net/squidman/index.html [3] http://www.gedanken.org.uk/software/wwwoffle/ http://www.gedanken.org.uk/software/wwwoffle/ [4] https://webrecorder.io/ https://webrecorder.io/ [5] https://pinboard.in/upgrade/ https://pinboard.in/upgrade/
- agamble 9y ago
  Great points. You're right, for now it's a single rate-limited HTML form and you'll have to manually collate the links to the archives you create. I'll be adding specialty features (with accounts) next. :)
- falcolas 9y ago
  Another pair of even simpler solutions: Print and store pages as PDFs. Download and save entire pages as webarchives (Safari, wget)
jpalomaki 9y ago
This might be a good use case for distributed storage (IPFS?). Instead of hosting this directly on my computer, it would be interesting to have a setup where the archiving is done via the service and I would just provide somewhere a storage space where the content would end up being mirrored (just to guarantee that my valuable things are saved at least somewhere, should the the other nodes decide to remove the content). I would prefer this setup, because it would be easily accessible for me from any device and I would not need to worry about running some always available system. With some suitable P2P setup my storage node would have less strict uptime requirements.
- johnaberlin 9y ago
  Hi jpalomaki, Have you heard of InterPlanetary Wayback (ipwb)? https://github.com/oduwsdl/ipwb https://github.com/oduwsdl/ipwb InterPlanetary Wayback (ipwb) facilitates permanence and collaboration in web archives by disseminating the contents of WARC files into the IPFS network.
prirun 9y ago
I think you should explain why you're paying Google to archive web pages for others, ie, how do plan on benefiting from this? If you have some business model in mind, let people know now. It's the first question that comes to my mind when someone offers a service that is free yet costs the provider real money. You obviously can't pay Google to archive everyone's web pages just for the fun of it.
- agamble 9y ago
  OP here. Great point. Right now this is just a single rate-limited HTML form to gauge interest. Next is to build specialty features that are worth paying for and make this sustainable. :)
- crispytx 9y ago
  Haters gonna hate.
rahiel 9y ago
An internet archive can only provide value if it's there for the long-term. What's your plan to keep this service running if it gets popular? For example, archive.is costed about $2000/month at the start of 2014 [1]. I expect it to cost even more now. [1]: http://blog.archive.is/post/72136308644/how-much-does-it-cost-you-to-host-a-website-of http://blog.archive.is/post/72136308644/how-much-does-it-cos...
JackC 9y ago
For personal web archiving, I highly recommend http://webrecorder.io http://webrecorder.io. The site lets you download archives in standard WARC format and play them back in an offline (Electron) player. It's also open source and has a quick local setup via Docker - https://github.com/webrecorder/webrecorder https://github.com/webrecorder/webrecorder . Webrecorder is by a former Internet Archive engineer, Ilya Kreymer, who now captures online performance art for an art museum. What he's doing with capture and playback of Javascript, web video, streaming content, etc. is state of the art as far as I know. (Disclaimer - I use bits of Webrecorder for my own archive, perma.cc.) For OP, I would say consider building on and contributing back to Webrecorder -- or alternatively figure out what Webrecorder is good at and make sure you're good at something different. It's a crazy hard problem to do well and it's great to have more ideas in the mix.
- agamble 9y ago
  Thanks Jack, I hadn't heard of webrecorder before, but I'll check it out. :)
- shasheene 9y ago
  Last I played with it, the latency on webrecorder was uncomfortably high for always-on recording of personal web usage (the pages only display once fully rendered). I wish webpages would render as normal and get asynchronously archived once loading is complete. That would allow constant archival of every webpage a user ever visits -- an immutable record over the user's years of crawling the web.
  - unicornporn 9y ago
    > That would allow constant archival of every webpage a user ever visits -- an immutable record over the user's years of crawling the web. This is usually solved by using a proxy: http://netpreserve.org/projects/live-archiving-http-proxy/ http://netpreserve.org/projects/live-archiving-http-proxy/
    - owenversteeg 9y ago
crispytx 9y ago
You know your site actually does a better job reproducing webpages than archive.org. I've noticed that if you use a CDN to serve up CSS & JS for a webpage that you're trying to archive on archive.org, it won't render correctly. On your site, there doesn't seem to be a problem including CSS & JS from an external domain. Thumbs up :)
- agamble 9y ago
  OP here. Thanks! Could you point me to the pages where it worked well for you vs archive.org?
gorbachev 9y ago
You should try and rewrite relative links in websites that get archived. I tested your app with a news site, and all the links go to archive.tesoro.io/sites/internal/url/structure/article.html I also second the need for user accounts. If I am to use your site as my personal archive, then I would need to log in and create a collection of my own archived sites.
NicoJuicy 9y ago
When a company went down, i downloaded every one of their clients with httrack and wget. Just to be sure their clients wouldn't lose their site. ( and some other things) I wonder what this site uses
CM30 9y ago
When you said 'own internet archive' I thought you meant some sort of program you could download that'd save your browsing history (or whatever full website you wanted) to your hard drive. I think that would have been significantly more useful here. As is it, while it's a nice service, it's still got all the issues of other archive ones: 1. It's online only, so one failed domain renewal or hosting payment takes everything offline. 2. It being online also means I can't access any saved pages if my connection goes down or has issues. 3. The whole thing is wide open to having content taken down by websites wanting to cover their tracks. I mean, what do you do if someone tells you to remove a page? What about with a DMCA notice? It's a nice alternative to archive.is, but still doesn't really do what the title suggests if you ask me.
idlewords 9y ago
You're going to get this service shut down if you let anonymous people republish arbitrary content while running everything on Google. I (obviously) think personal archives are a great idea, but republishing is a hornets' nest.
jdc0589 9y ago
> Tesoro saves linked assets, such as images, Javascript and CSS files. I'm confused. It looks like image sources in "archived" pages on Tesoro still point back to the origin domain. Edit: it works as expected. I just didn't notice the relative paths.
- agamble 9y ago
  OP here. The site will rewrite absolute image URLs as relative ones pointing to Tesoro. For example, in the Chicken Teryaki example on the homepage, the main image is sourced from the relative location "static01.nyt.com/.../28COOKING-CHICKEN-TERIYAKI1-articleLarge.jpg", which looks like it's coming from nytimes.com, but you can check in the Chrome dev console that it isn't. Have you found an example where it isn't working correctly? If so would you mind posting it here and I'll fix it :).
  - jdc0589 9y ago
    Nope, you are right. I just missed that there wasn't a protocol on the src I was looking at.
  - ikreymer 9y ago
    Unfortunately, this approach alone will only work for sites that are mostly static, eg. do not use JS to load dynamic content. That is a small (and shrinking) percent of the web. Once JS is involved, all bets are off -- JS will attempt to load content via ajax, or generate new html, load iframes, etc and you will have 'live leaks' where the content seems to be coming form the archive but is actually coming form the live web. Here is an example from archiving nytimes home page: https://archive.tesoro.io/665dbeab57a4d57d8140f89cfedc69b5 https://archive.tesoro.io/665dbeab57a4d57d8140f89cfedc69b5 If you look at network traffic (domain in devtools), you'll see that only a small % is coming from archive.tesoro.io -- the rest of the content is loaded from the live web. This can be misleading and possibly a security risk as well. Not to discourage you, but this is a hard problem and I've been working on for years now. This area is a moving target, but we think live leaks are mostly eliminated in Webrecorder and pywb, although there are lots of areas to work on to maintain high-fidelity preservation. If you want chat about possible solutions or want to collaborate (we're always looking for contributors!), feel free to reach out to us at support [at] webrecorder.io or find my contact on GH.
bfirsh 9y ago
What's the best way to automatically archive all of the data I produce on websites? Facebook, Twitter, Instagram, blogs, and so on. At some point these services will disappear, and I want to preserve them. I know a lot of these sites have archiving features, but want something centralised and automatic.
- donpdonp 9y ago
  The IndieWeb(.org) group recommends 'Publish (on your) Own Site, Syndicate Elsewhere' (https://indieweb.org/POSSE https://indieweb.org/POSSE) that you might find interesting.
- aethertron 9y ago
  The hypothetical system that makes most sense to me for this: a process that runs 24/7 on a server, watching your feeds on those services. Grabbing and saving everything via APIs or screen-scraping.
  - fiatjaf 9y ago
    Is that creepy resource-eater bug-prone service what makes most sense to you?
    - aethertron 9y ago
      Yes. Why do you call it 'creepy'? It's supposed to be a personal service, owned and controlled by the user who wants to archive stuff. And all computations consume resources, and may have bugs. So what? They can be optimised. Bugs can get fixed. The process would, ideally, auto-update.
salmonfamine 9y ago
Worth noting that Tesoro is the name of a major oil/fuel company in Texas.
zippoxer 9y ago
Cool tool, but by using it, you depend on it staying alive for longer than any page you archive on it. This got me thinking about how a decentralized p2p internet archive could solve the trust problem that exists in centralized internet archives. Such solution could also increase the capacity of archived pages and the frequency at which archived pages are updated. It is true that keeping the entire history of the internet on your local drive is likely impossible, but a solution similar to what Sia is doing could solve this problem: split each page to 20 pieces and distribute each piece to 10 peers such that every y pieces can recover the original page. So, you only have to trust that 10 peers out of 20 that store a page are still alive to get the complete page. The main problem I can see right now would be lack of motivation to contribute to the system -- why would people run nodes? Just because it would feature a yet another cryptocurrency? Sure, this could hold now, but when the cryptocurrency craze quiets down and people stop buying random cryptocurrencies just for the sake of trading them, what then? Who would run the nodes and why?
- burkemw3 9y ago
  IPFS [0] and it's sibling Filecoin [1] are dealing in a very similar space as your wonderings [0]: https://ipfs.io/ https://ipfs.io/ [1]: https://filecoin.io/ https://filecoin.io/
j_s 9y ago
The discussion 3 months ago on bookmarks mentioned several options for archiving pages (some locally): Ask HN: Do you still use browser bookmarks? | https://news.ycombinator.com/item?id=14064096 https://news.ycombinator.com/item?id=14064096 extensions: Firefox "Print Edit" Addon / Firefox Scrapbook X / Chrome Falcon / Firefox Recoll open source: Zotero / WorldBrain / Wallabag commercial: Pinboard / InstaPaper / Pocket / Evernote / Mochimarks / Diigo / PageDash / URL Manager Pro / Save to Google / OneNote / Stash / Fetching public: http://web.archive.org http://web.archive.org / https://archive.is/ https://archive.is/
j_s 9y ago
I would be interested in an attestation service that can provide court-admissable evidence that a particular piece of content was publically accessible on the web at a particular point in time via a particular url. I believe the only way to incentivise participation in such a system is by paying for timestamp'ed signatures, eg. "some subset of downloaded [content] from [url] at [time] hashed to [hash]" all tucked into a Bitcoin transaction or something. There are services that will do this with user-provided content[1]; I am looking for something that will pull a url and timestamp the content. This would also be a way to detect when different users are being served different content at the same url, thus the need for a global network of validators. [1] https://proofofexistence.com/ https://proofofexistence.com/
- rjeli 9y ago
  Interesting - it is trivial to prove something was done today rather than yesterday, by hashing with the most recent bitcoin block or some new info. Is it possible to prove something was done in the past? All I can think of is some sort of scheme involving destroyed information.
  - deleted 9y ago
    [deleted]
  - an27 9y ago
    Isn't it the other way around? I can prove I had today's papers today, but once I've seen it I can prove it any day. So you can say "this information existed at day X or earlier". There's also the issue of proving the content you hashed actually came from the place you say it came from. To me it seems that would require proofs of authenticity, by the source itself; not something that's easy to come by.
  - j_s 9y ago
    trivial to prove something was done today My focus is on the something much more so than the when. I can do my own doctoring of any data, or use some service to make something that looks real[1]. Getting some proof that this fake data existed is not what I'm after. Instead, I want multiple, completely separate (and ideally as independent and diverse as possible) attestations that something was out there online, as proof that some person or organization intended for it to be seen by everyone as their content. Being able to prove that irrefutably seems nearly impossible today even for the present time, particularly against insider threats. Your question regarding proving something in the past is going far beyond what I'm hoping for; it will take me quite a while to come up with anything that might be helpful for such a situation. I assume most would hit up the various archive sites, but my gut feeling is that it winds up being a probability based on how well forensics holds up / are not falsifiable. [1] simitator.com - not linking because ads felt a bit extra-sketch!
skdotdan 9y ago
Nice. How are you planning to pay the servers? Your service seems quite storage-intensive.
- WhiteOwlLion 9y ago
  I got a dedicated server in France that cost me less than $20 USD/month. 16GB RAM, 1TB storage: https://www.online.net/en/dedicated-server/dedibox-xc https://www.online.net/en/dedicated-server/dedibox-xc
  - mkroman 9y ago
    With no redundancy, no backup and no way to extend storage. I'm not sure how you'd archive the internet with low-range dedicated server deals.
arkenflame 9y ago
I made a simple Chrome extension to automatically save local copies of pages you bookmark, if you prefer that instead: https://chrome.google.com/webstore/detail/backmark-back-up-the-page/cmbflafdbcidlkkdhbmechbcpmnbcfjf https://chrome.google.com/webstore/detail/backmark-back-up-t...
itsjamesfoster 9y ago
that's a great tool.

174 points by agamble 9y ago

pbhjpbhj 9y ago

How are you handling copyright infringement? Outside USAs Fair Use terms this looks like pretty blatant infringement.

iso-8859-1 9y ago
What is there to handle? You take down stuff when you get an email? Most users of this will be so small that they'll never get noticed. Maybe they won't even be online, how are you going to know you were infringed? Maybe the crawler allows for spoofing the user-agent.
- pbhjpbhj 9y ago
  So, ignoring it basically. If a person in the UK uses your service you're committing contributory infringement for commercial purposes, AFAICT. Moreover, the ECD has different protections than DMCA. Particularly getting a takedown notice isn't required. >Maybe the crawler allows for spoofing the user-agent. // As a tort you only need to get a preponderance of evidence. IP of the crawler that made the copy puts the owner of that IP in court for contributory infringement, no? If you make copies of parts of BBC sites and serve those copies from your server how is that not copyright infringement by you?? FWIW I like the service and do not like the copyright regime as it stands, particularly how UK law lacks the breadth of liberties of Fair Use.
brianobush 9y ago
Probably in the same manner as archive.org

Piskvorrr 9y ago

That's just as much "my own" as The Internet Archive: a website Out There somewhere. Worse, it's much more likely to rot and disappear than archive.org. Now, if I could run this locally... (Yes, yes, `wget --convert-links`, I know. Not quite as convenient, though.)

agamble 9y ago
OP here. The internet archive is great, but it's not so awesome if there's some ephemeral content you need to save right away, like Tweets or social media posts. Being able to trigger an archive immediately let's you save temporary content such as that which is more prone to deletion. I'm going to build a Chrome extension to click and make cloud copy of the page you're on, hopefully that will make it seem more personally controllable. Do you think being able to download the archive locally would be useful?
- detaro 9y ago
  You can trigger the Internet Archive manually as well.
  - agamble 9y ago
    Oh neat didn't realise that, my mistake :)
    - Sophira 9y ago
      To be fair, the easiest way to do it (to my knowledge) is by direct URL entry - for example (replace 'hxxps' with 'https' for this - I didn't want crawlers to pick this up): hxxps://web.archive.org/save/https://tesoro.io/ https://tesoro.io/
- dschep 9y ago
  So like another toplevel commenter asked. Why build this or use this instead of archive.is? And there are already multiple extensions available for chrome for it ;) I agree with GP here, that anything billed as "My own internet archive" should be run on my computer. Not some one elses.
- jtrip 9y ago
  A local download only increases the redundancy. Tesoro keeps a copy, and the user keeps a local copy that they can also use however they want. A bit like keeping newspaper clippings, that are found decades later by some relative to be then posted on social websites as something interesting.

zichy 9y ago

So this is like archive.is, but I can't search through archived sites?

lozzo 9y ago

it would be nice to have a bit of explanation on how it works and why we can be confident that we can rely upon it

agamble 9y ago
OP here. Definitely, great idea :) Briefly: Sites are archived using a system written in Golang and uploaded to a Google Cloud bucket. More: The system downloads the remote HTML, parses it to extract the relevant dependencies (<script>, <link>, <img> etc) and then downloads these as well. Tesoro is even parsing CSS files to extract the url('...') file dependencies from here as well, meaning most background images and fonts should continue to work. All dependencies (even those hosted at remote domains) are downloaded and hosted with the archive, meaning the src attributes on the original page tags are wrangled to support the new location. The whole thing is hosted on GCP Container Engine and I deploy with Kubernetes. I'll write up a more comprehensive blog post in some time, which portion of this would you like to hear more about?
- 19eightyfour 9y ago
  The issue is cost. Your costs are disk space for people's archives, instances for people's use, and bandwidth for the fetches and crawls and access. How can you pay for this if it's free? It's unreliable unless its financially viable.
  - agamble 9y ago
    Totally right, great observation :) For now it's a free service with a single rate-limited form. Now it's time to work on adding specialty features that are worth paying for.
- Faaak 9y ago
  How about avoiding redundancies ? Are same CSS files cached twice or referenced by their hash ? The page URI is a bit obscure though. I think a tresoro.io/example.tld/page/foobar/timestamp would look good. What about big media content and/or small differences between them ?

akerro 9y ago

Nice, post it on https://www.reddit.com/r/DataHoarder/ https://www.reddit.com/r/DataHoarder/ They will love it!

smoyer 9y ago

It's not mine unless it's running on my own servers or computer - I created a really rough version of this several years ago that is saved to my computer (and from there into box).

pbhjpbhj 9y ago
I adapted a bash script someone posted here, it uses Firefox bookmarks (pages.sqlite). Cron runs the script and downloads every page I've bookmarked that month (after some filtering). I don't use it often but sometimes I'll awk-grep it; I'm a hoarder in real life too!
- gkya 9y ago
  Would you mind posting it?
  - pbhjpbhj 9y ago
    See sibling comment.
- kchr 9y ago
  Please share!
  - pbhjpbhj 9y ago
    See sibling comment.
- dddw 9y ago
  indeed interested in this too
  - pbhjpbhj 9y ago
    https://www.gwern.net/Archiving%20URLs https://www.gwern.net/Archiving%20URLs was the initial idea and base for my code. I see now he's updated his page considerably - IIRC - from what was there before. My normal medium is clay so I'd rely on his stuff rather than mine (linked below). * https://gist.github.com/pbhj/6636d0908d0d11885809a2545b138694 https://gist.github.com/pbhj/6636d0908d0d11885809a2545b13869... main script * https://gist.github.com/pbhj/4dedca1e980d6a102433403c0f435524 https://gist.github.com/pbhj/4dedca1e980d6a102433403c0f43552... filter script

dbz 9y ago

This is pretty cool. I have a chrome extension that let's you view the cached version of a web page [1]. Would I be able to use this through an API? I currently support Google Cache, WayBack Machine, and CoralCDN, but Coral doesn't work well and I'd like to replace it with something else. [1] https://chrome.google.com/webstore/detail/cmmlgikpahieigpcclckfmhnchdlfnjd https://chrome.google.com/webstore/detail/cmmlgikpahieigpccl...

agamble 9y ago
OP here. Yup, API and chrome extension are next on the feature list. :)

Retr0spectrum 9y ago

Is this any different to archive.is? If I want my own archive, Ctrl+S in Firefox usually works fine for me.

venning 9y ago

Thoughts: I like the look. Very clean. I like how fast it's responding; better than archive.org (though, obviously, they have different scaling problems). "Your own internet archive" might be overselling it, as other commenters have pointed out; the "Your" feels a bit misleading. I think "Save a copy of any webpage." gives a better impression, which you use on the site itself. The "Archive!" link probably shouldn't work if there's nothing in the URL box. It just gives me an archive link that errors. Example: [1] Using it on news.YC as a test gave me errors with the CSS & JS [2]. This might be due to the fact that HN uses query parameters in their CSS and JS, which repeat in the tesoro URL, which you may not be parsing correctly. Maybe have something in addition to an email link for submitting error reports like the above, just cause I'd be more likely to file a GitHub issue (even if the repo is empty) than send a stranger an email. As other commenters have pointed out, archive.is also does this, and their longevity helps me feel confident that they'll still be around. Perhaps, if you wish to differentiate, offer some way for me to "own" the copy of the page, like downloading it or emailing it to myself or sharing it with another site (like Google Docs or Imgur) to leverage redundancy, or something like that. Just a thought. All in all, nice Show HN. EDIT: You also may want to adjust the header to work properly on mobile devices. Still though, nice job. Sorry if I'm sounding critical. [1] https://archive.tesoro.io/320b55cc9b78e271c94716ee23554da8 https://archive.tesoro.io/320b55cc9b78e271c94716ee23554da8 [2] https://archive.tesoro.io/a7bf03e247224bc3b4e5a7c1f2ad42b1 https://archive.tesoro.io/a7bf03e247224bc3b4e5a7c1f2ad42b1

agamble 9y ago
Thanks! These are great comments - I'll look into the issue with saving Hacker News CSS + JS.

unicornporn 9y ago

In what way could this considered to be “your own internet archive”? I see no way to register a user and save pages to a collection. If you really want to create your own archive, set up a Live Archiving HTTP Proxy[1], run SquidMan [2] or check out WWWOFFLE[3]. If you want something simpler, have a look at Webrecorder[4] or a paid Pinboard account with the “Bookmark Archive”[5]. [1] http://netpreserve.org/projects/live-archiving-http-proxy/ http://netpreserve.org/projects/live-archiving-http-proxy/ [2] http://squidman.net/squidman/index.html http://squidman.net/squidman/index.html [3] http://www.gedanken.org.uk/software/wwwoffle/ http://www.gedanken.org.uk/software/wwwoffle/ [4] https://webrecorder.io/ https://webrecorder.io/ [5] https://pinboard.in/upgrade/ https://pinboard.in/upgrade/

agamble 9y ago
Great points. You're right, for now it's a single rate-limited HTML form and you'll have to manually collate the links to the archives you create. I'll be adding specialty features (with accounts) next. :)
falcolas 9y ago
Another pair of even simpler solutions: Print and store pages as PDFs. Download and save entire pages as webarchives (Safari, wget)

jpalomaki 9y ago

This might be a good use case for distributed storage (IPFS?). Instead of hosting this directly on my computer, it would be interesting to have a setup where the archiving is done via the service and I would just provide somewhere a storage space where the content would end up being mirrored (just to guarantee that my valuable things are saved at least somewhere, should the the other nodes decide to remove the content). I would prefer this setup, because it would be easily accessible for me from any device and I would not need to worry about running some always available system. With some suitable P2P setup my storage node would have less strict uptime requirements.

johnaberlin 9y ago
Hi jpalomaki, Have you heard of InterPlanetary Wayback (ipwb)? https://github.com/oduwsdl/ipwb https://github.com/oduwsdl/ipwb InterPlanetary Wayback (ipwb) facilitates permanence and collaboration in web archives by disseminating the contents of WARC files into the IPFS network.

prirun 9y ago

I think you should explain why you're paying Google to archive web pages for others, ie, how do plan on benefiting from this? If you have some business model in mind, let people know now. It's the first question that comes to my mind when someone offers a service that is free yet costs the provider real money. You obviously can't pay Google to archive everyone's web pages just for the fun of it.

agamble 9y ago
OP here. Great point. Right now this is just a single rate-limited HTML form to gauge interest. Next is to build specialty features that are worth paying for and make this sustainable. :)
crispytx 9y ago
Haters gonna hate.

rahiel 9y ago

An internet archive can only provide value if it's there for the long-term. What's your plan to keep this service running if it gets popular? For example, archive.is costed about $2000/month at the start of 2014 [1]. I expect it to cost even more now. [1]: http://blog.archive.is/post/72136308644/how-much-does-it-cost-you-to-host-a-website-of http://blog.archive.is/post/72136308644/how-much-does-it-cos...

JackC 9y ago

For personal web archiving, I highly recommend http://webrecorder.io http://webrecorder.io. The site lets you download archives in standard WARC format and play them back in an offline (Electron) player. It's also open source and has a quick local setup via Docker - https://github.com/webrecorder/webrecorder https://github.com/webrecorder/webrecorder . Webrecorder is by a former Internet Archive engineer, Ilya Kreymer, who now captures online performance art for an art museum. What he's doing with capture and playback of Javascript, web video, streaming content, etc. is state of the art as far as I know. (Disclaimer - I use bits of Webrecorder for my own archive, perma.cc.) For OP, I would say consider building on and contributing back to Webrecorder -- or alternatively figure out what Webrecorder is good at and make sure you're good at something different. It's a crazy hard problem to do well and it's great to have more ideas in the mix.

agamble 9y ago
Thanks Jack, I hadn't heard of webrecorder before, but I'll check it out. :)
shasheene 9y ago
Last I played with it, the latency on webrecorder was uncomfortably high for always-on recording of personal web usage (the pages only display once fully rendered). I wish webpages would render as normal and get asynchronously archived once loading is complete. That would allow constant archival of every webpage a user ever visits -- an immutable record over the user's years of crawling the web.
- unicornporn 9y ago
  > That would allow constant archival of every webpage a user ever visits -- an immutable record over the user's years of crawling the web. This is usually solved by using a proxy: http://netpreserve.org/projects/live-archiving-http-proxy/ http://netpreserve.org/projects/live-archiving-http-proxy/
  - owenversteeg 9y ago

crispytx 9y ago

You know your site actually does a better job reproducing webpages than archive.org. I've noticed that if you use a CDN to serve up CSS & JS for a webpage that you're trying to archive on archive.org, it won't render correctly. On your site, there doesn't seem to be a problem including CSS & JS from an external domain. Thumbs up :)

agamble 9y ago
OP here. Thanks! Could you point me to the pages where it worked well for you vs archive.org?

gorbachev 9y ago

You should try and rewrite relative links in websites that get archived. I tested your app with a news site, and all the links go to archive.tesoro.io/sites/internal/url/structure/article.html I also second the need for user accounts. If I am to use your site as my personal archive, then I would need to log in and create a collection of my own archived sites.

NicoJuicy 9y ago

When a company went down, i downloaded every one of their clients with httrack and wget. Just to be sure their clients wouldn't lose their site. ( and some other things) I wonder what this site uses

CM30 9y ago

When you said 'own internet archive' I thought you meant some sort of program you could download that'd save your browsing history (or whatever full website you wanted) to your hard drive. I think that would have been significantly more useful here. As is it, while it's a nice service, it's still got all the issues of other archive ones: 1. It's online only, so one failed domain renewal or hosting payment takes everything offline. 2. It being online also means I can't access any saved pages if my connection goes down or has issues. 3. The whole thing is wide open to having content taken down by websites wanting to cover their tracks. I mean, what do you do if someone tells you to remove a page? What about with a DMCA notice? It's a nice alternative to archive.is, but still doesn't really do what the title suggests if you ask me.

idlewords 9y ago

You're going to get this service shut down if you let anonymous people republish arbitrary content while running everything on Google. I (obviously) think personal archives are a great idea, but republishing is a hornets' nest.

jdc0589 9y ago

> Tesoro saves linked assets, such as images, Javascript and CSS files. I'm confused. It looks like image sources in "archived" pages on Tesoro still point back to the origin domain. Edit: it works as expected. I just didn't notice the relative paths.

agamble 9y ago
OP here. The site will rewrite absolute image URLs as relative ones pointing to Tesoro. For example, in the Chicken Teryaki example on the homepage, the main image is sourced from the relative location "static01.nyt.com/.../28COOKING-CHICKEN-TERIYAKI1-articleLarge.jpg", which looks like it's coming from nytimes.com, but you can check in the Chrome dev console that it isn't. Have you found an example where it isn't working correctly? If so would you mind posting it here and I'll fix it :).
- jdc0589 9y ago
  Nope, you are right. I just missed that there wasn't a protocol on the src I was looking at.
- ikreymer 9y ago
  Unfortunately, this approach alone will only work for sites that are mostly static, eg. do not use JS to load dynamic content. That is a small (and shrinking) percent of the web. Once JS is involved, all bets are off -- JS will attempt to load content via ajax, or generate new html, load iframes, etc and you will have 'live leaks' where the content seems to be coming form the archive but is actually coming form the live web. Here is an example from archiving nytimes home page: https://archive.tesoro.io/665dbeab57a4d57d8140f89cfedc69b5 https://archive.tesoro.io/665dbeab57a4d57d8140f89cfedc69b5 If you look at network traffic (domain in devtools), you'll see that only a small % is coming from archive.tesoro.io -- the rest of the content is loaded from the live web. This can be misleading and possibly a security risk as well. Not to discourage you, but this is a hard problem and I've been working on for years now. This area is a moving target, but we think live leaks are mostly eliminated in Webrecorder and pywb, although there are lots of areas to work on to maintain high-fidelity preservation. If you want chat about possible solutions or want to collaborate (we're always looking for contributors!), feel free to reach out to us at support [at] webrecorder.io or find my contact on GH.

bfirsh 9y ago

What's the best way to automatically archive all of the data I produce on websites? Facebook, Twitter, Instagram, blogs, and so on. At some point these services will disappear, and I want to preserve them. I know a lot of these sites have archiving features, but want something centralised and automatic.

donpdonp 9y ago
The IndieWeb(.org) group recommends 'Publish (on your) Own Site, Syndicate Elsewhere' (https://indieweb.org/POSSE https://indieweb.org/POSSE) that you might find interesting.
aethertron 9y ago
The hypothetical system that makes most sense to me for this: a process that runs 24/7 on a server, watching your feeds on those services. Grabbing and saving everything via APIs or screen-scraping.
- fiatjaf 9y ago
  Is that creepy resource-eater bug-prone service what makes most sense to you?
  - aethertron 9y ago
    Yes. Why do you call it 'creepy'? It's supposed to be a personal service, owned and controlled by the user who wants to archive stuff. And all computations consume resources, and may have bugs. So what? They can be optimised. Bugs can get fixed. The process would, ideally, auto-update.

salmonfamine 9y ago

Worth noting that Tesoro is the name of a major oil/fuel company in Texas.

zippoxer 9y ago

Cool tool, but by using it, you depend on it staying alive for longer than any page you archive on it. This got me thinking about how a decentralized p2p internet archive could solve the trust problem that exists in centralized internet archives. Such solution could also increase the capacity of archived pages and the frequency at which archived pages are updated. It is true that keeping the entire history of the internet on your local drive is likely impossible, but a solution similar to what Sia is doing could solve this problem: split each page to 20 pieces and distribute each piece to 10 peers such that every y pieces can recover the original page. So, you only have to trust that 10 peers out of 20 that store a page are still alive to get the complete page. The main problem I can see right now would be lack of motivation to contribute to the system -- why would people run nodes? Just because it would feature a yet another cryptocurrency? Sure, this could hold now, but when the cryptocurrency craze quiets down and people stop buying random cryptocurrencies just for the sake of trading them, what then? Who would run the nodes and why?

burkemw3 9y ago
IPFS [0] and it's sibling Filecoin [1] are dealing in a very similar space as your wonderings [0]: https://ipfs.io/ https://ipfs.io/ [1]: https://filecoin.io/ https://filecoin.io/

j_s 9y ago

The discussion 3 months ago on bookmarks mentioned several options for archiving pages (some locally): Ask HN: Do you still use browser bookmarks? | https://news.ycombinator.com/item?id=14064096 https://news.ycombinator.com/item?id=14064096 extensions: Firefox "Print Edit" Addon / Firefox Scrapbook X / Chrome Falcon / Firefox Recoll open source: Zotero / WorldBrain / Wallabag commercial: Pinboard / InstaPaper / Pocket / Evernote / Mochimarks / Diigo / PageDash / URL Manager Pro / Save to Google / OneNote / Stash / Fetching public: http://web.archive.org http://web.archive.org / https://archive.is/ https://archive.is/

j_s 9y ago

I would be interested in an attestation service that can provide court-admissable evidence that a particular piece of content was publically accessible on the web at a particular point in time via a particular url. I believe the only way to incentivise participation in such a system is by paying for timestamp'ed signatures, eg. "some subset of downloaded [content] from [url] at [time] hashed to [hash]" all tucked into a Bitcoin transaction or something. There are services that will do this with user-provided content[1]; I am looking for something that will pull a url and timestamp the content. This would also be a way to detect when different users are being served different content at the same url, thus the need for a global network of validators. [1] https://proofofexistence.com/ https://proofofexistence.com/

rjeli 9y ago
Interesting - it is trivial to prove something was done today rather than yesterday, by hashing with the most recent bitcoin block or some new info. Is it possible to prove something was done in the past? All I can think of is some sort of scheme involving destroyed information.
- deleted 9y ago
  [deleted]
- an27 9y ago
  Isn't it the other way around? I can prove I had today's papers today, but once I've seen it I can prove it any day. So you can say "this information existed at day X or earlier". There's also the issue of proving the content you hashed actually came from the place you say it came from. To me it seems that would require proofs of authenticity, by the source itself; not something that's easy to come by.
- j_s 9y ago
  trivial to prove something was done today My focus is on the something much more so than the when. I can do my own doctoring of any data, or use some service to make something that looks real[1]. Getting some proof that this fake data existed is not what I'm after. Instead, I want multiple, completely separate (and ideally as independent and diverse as possible) attestations that something was out there online, as proof that some person or organization intended for it to be seen by everyone as their content. Being able to prove that irrefutably seems nearly impossible today even for the present time, particularly against insider threats. Your question regarding proving something in the past is going far beyond what I'm hoping for; it will take me quite a while to come up with anything that might be helpful for such a situation. I assume most would hit up the various archive sites, but my gut feeling is that it winds up being a probability based on how well forensics holds up / are not falsifiable. [1] simitator.com - not linking because ads felt a bit extra-sketch!

skdotdan 9y ago

Nice. How are you planning to pay the servers? Your service seems quite storage-intensive.

WhiteOwlLion 9y ago
I got a dedicated server in France that cost me less than $20 USD/month. 16GB RAM, 1TB storage: https://www.online.net/en/dedicated-server/dedibox-xc https://www.online.net/en/dedicated-server/dedibox-xc
- mkroman 9y ago
  With no redundancy, no backup and no way to extend storage. I'm not sure how you'd archive the internet with low-range dedicated server deals.

arkenflame 9y ago

I made a simple Chrome extension to automatically save local copies of pages you bookmark, if you prefer that instead: https://chrome.google.com/webstore/detail/backmark-back-up-the-page/cmbflafdbcidlkkdhbmechbcpmnbcfjf https://chrome.google.com/webstore/detail/backmark-back-up-t...

itsjamesfoster 9y ago

that's a great tool.