6 ms·
Wayback Machine full archive is like less than 50PB. Let's say you could strip multimedia and remove every patterned data to compress that into about a petabyte
by numpad0 4d ago
Wayback Machine full archive is like less than 50PB. Let's say you could strip multimedia and remove every patterned data to compress that into about a petabyte. The per-bit cheapest disk right now is consumer Seagate 24TB(SI; 21.8TiB usable) at ~$500, or around $1200k for just the disks.
Doable if you had couple million dollars to burn. Cheaper than private jets new.
- nijave 4d agoSeems like the most practical path is some kind of distributed peer to peer contraption where you can allocate some storage and optionally participate in crawling. I imagine if set some constraints you could get index size down quite a bit but I still suspect it'd be hard keeping up with content churn. Edit: Looks like enwiki bz2 is coming in around 46Gi which isn't too bad considering the amount of content it contains.