5 ms·

Hi, I run the datacenter/infrastructure team at the Internet Archive! We would love to see you at our various events this fall but if paying for the ticket is

Hi, I run the datacenter/infrastructure team at the Internet Archive! We would love to see you at our various events this fall but if paying for the ticket is difficult for you, please email me (in bio) and we'll get you in (if possible).

awesomeMilou 1y ago
What events are we talking about here?
- jackling 1y ago
  Probably these: https://blog.archive.org/events/ https://blog.archive.org/events/
NetOpWibby 1y ago
I would love to work for IA but openings are rare
- pabs3 1y ago
  If you are in Europe, consider Software Heritage (similar to IA but for source code) too: https://www.softwareheritage.org/jobs/ https://www.softwareheritage.org/jobs/
  - msephton 1y ago
    Internet Archive now have a presence in Amsterdam
moralestapia 1y ago
Hey, Q., so what's the size of the internet archive?
- deleted 1y ago
  [deleted]
- metalman 1y ago
  it is large enough that I am wondering if the data captured by the actual physical magnetic charges has a heft, that a person could feel. obviously the hardware would fill a house or something, but at what point does the worlds data become a discernable physical reality, at least in theory
- the_real_cher 1y ago
  I'm betting exabyte or close maybe
- textfiles 11mo ago
  For the purposes of ballpark, between 150-200 petabytes of unique data, probably on the lowish end of that last I checked.
psychoslave 1y ago
Are they distributed events all around the world of just in wherever the team is gathered (San Francisco I guess?) By the way, thank you all the teams in IA, what you provide is such an important thing for humanity.
vettyvignesh 1y ago
would love technical details around this feat. ex: how you even crawl to begin with, storage, etc
WhereIsTheTruth 1y ago
We all know the NSA has access to servers hosted in the U.S. How are you protecting the archive from malicious tampering? Are you using any form of immutable storage? Is it post-quantum secure?
- gosub100 1y ago
  Why would they do that? Have you previously seen a case where they "maliciously tampered" with anyone's website?
  - WhereIsTheTruth 1y ago
    I just question the integrity and immutability of the data IA is archiving, that's all You want to know why they'd tamper data? https://seclab.cs.washington.edu/2017/10/30/rewriting-history-manipulating-the-archived-web-from-the-present/ https://seclab.cs.washington.edu/2017/10/30/rewriting-histor... https://blog.archive.org/2018/04/24/addressing-recent-claims-of-manipulated-blog-posts-in-the-wayback-machine/ https://blog.archive.org/2018/04/24/addressing-recent-claims... NSA already paid to back-door RSA, got caught shiping pre-hacked routers, can rewrite pages mid-flight with QUANTUM, penetrate and siphon data from remote infected machines.. what else could they do? https://www.amnesty.org/en/latest/news/2022/09/myanmar-facebooks-systems-promoted-violence-against-rohingya-meta-owes-reparations-new-report/ https://www.amnesty.org/en/latest/news/2022/09/myanmar-faceb...
    - gosub100 1y ago
      IA themselves could tamper with the data, no? It was never meant to be an official historical snapshot to be pulled up for any serious or official purposes. Although it has been used that way for high profile internet drama. It's just a matter of time (maybe during an election) before it's surreptitiously altered and referenced for nefarious purposes.
southernplaces7 1y ago
Most of all, i'm curious about how you reliably and securely store or host so many archived pages. Would you mind briefly explaining such a huge undertaking? Also, total congratulations on the fantastic achievement of this. You guys are my go-to for so much information. Edit: And how many terabytes it all amounts to.
zhynn 1y ago
Thanks for helping to run my favorite library on earth.

awesomeMilou 1y ago

What events are we talking about here?

jackling 1y ago
Probably these: https://blog.archive.org/events/ https://blog.archive.org/events/

NetOpWibby 1y ago

I would love to work for IA but openings are rare

pabs3 1y ago
If you are in Europe, consider Software Heritage (similar to IA but for source code) too: https://www.softwareheritage.org/jobs/ https://www.softwareheritage.org/jobs/
- msephton 1y ago
  Internet Archive now have a presence in Amsterdam

moralestapia 1y ago

Hey, Q., so what's the size of the internet archive?

deleted 1y ago
[deleted]
metalman 1y ago
it is large enough that I am wondering if the data captured by the actual physical magnetic charges has a heft, that a person could feel. obviously the hardware would fill a house or something, but at what point does the worlds data become a discernable physical reality, at least in theory
the_real_cher 1y ago
I'm betting exabyte or close maybe
textfiles 11mo ago
For the purposes of ballpark, between 150-200 petabytes of unique data, probably on the lowish end of that last I checked.

psychoslave 1y ago

Are they distributed events all around the world of just in wherever the team is gathered (San Francisco I guess?) By the way, thank you all the teams in IA, what you provide is such an important thing for humanity.

vettyvignesh 1y ago

would love technical details around this feat. ex: how you even crawl to begin with, storage, etc

WhereIsTheTruth 1y ago

We all know the NSA has access to servers hosted in the U.S. How are you protecting the archive from malicious tampering? Are you using any form of immutable storage? Is it post-quantum secure?

gosub100 1y ago
Why would they do that? Have you previously seen a case where they "maliciously tampered" with anyone's website?
- WhereIsTheTruth 1y ago
  I just question the integrity and immutability of the data IA is archiving, that's all You want to know why they'd tamper data? https://seclab.cs.washington.edu/2017/10/30/rewriting-history-manipulating-the-archived-web-from-the-present/ https://seclab.cs.washington.edu/2017/10/30/rewriting-histor... https://blog.archive.org/2018/04/24/addressing-recent-claims-of-manipulated-blog-posts-in-the-wayback-machine/ https://blog.archive.org/2018/04/24/addressing-recent-claims... NSA already paid to back-door RSA, got caught shiping pre-hacked routers, can rewrite pages mid-flight with QUANTUM, penetrate and siphon data from remote infected machines.. what else could they do? https://www.amnesty.org/en/latest/news/2022/09/myanmar-facebooks-systems-promoted-violence-against-rohingya-meta-owes-reparations-new-report/ https://www.amnesty.org/en/latest/news/2022/09/myanmar-faceb...
  - gosub100 1y ago
    IA themselves could tamper with the data, no? It was never meant to be an official historical snapshot to be pulled up for any serious or official purposes. Although it has been used that way for high profile internet drama. It's just a matter of time (maybe during an election) before it's surreptitiously altered and referenced for nefarious purposes.

southernplaces7 1y ago

Most of all, i'm curious about how you reliably and securely store or host so many archived pages. Would you mind briefly explaining such a huge undertaking? Also, total congratulations on the fantastic achievement of this. You guys are my go-to for so much information. Edit: And how many terabytes it all amounts to.

zhynn 1y ago

Thanks for helping to run my favorite library on earth.