5 ms·

We need something like SETI@home/Folding@home but for crawling and archiving the web or maybe something as simple as a browser extension that can (with permissi

We need something like SETI@home/Folding@home but for crawling and archiving the web or maybe something as simple as a browser extension that can (with permission) archive pages you view.

ryoshu 7mo ago
This is a good idea. Not sure what ToS it would violate. But a good idea.
pclmulqdq 7mo ago
Your TV probably does that, and you definitely gave it permission when you clicked "accept" on the terms.
- macinjosh 7mo ago
  good thing I don't have a TV!
ninjagoo 7mo ago
In the US at least, there is no expectation of privacy in public. Why should these websites that are public-facing get an exemption from that? Serving up content to the public should imply archivability. Sometimes it feels like ai-use concerns are a guise to diminish the public record. While on the other hand services like Ring or Flock are archiving the public forever.
- sejje 7mo ago
  Ring and Flock are not a standard we should be striving towards. Their massive databases tracking citizens need to go.
dunder_cat 7mo ago
This exists although not in the traditional BOINC space, it's Archiveteam^1. I run two of their warrior^2 instances in my home k3s instance via the docker images. One of them is set to the "Team's choice" where it spends most of its time downloading Telegram chats. However, when they need the firepower for sites with imminent risk of closure, it will switch itself to those. The other one is set to their URL shortener project, "Terror of Tiny Town"^3. Their big requirement is you need to not be doing any DNS filtering or blocking of access to what it wants, so I've got the pod DNS pointed to the unfiltered quad9 endpoint and rules in my router to allow the machine it's running on to bypass my PiHole enforcement+outside DNS blocks. ^1 https://wiki.archiveteam.org/ https://wiki.archiveteam.org/ ^2 https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior ^3 https://wiki.archiveteam.org/index.php/URLTeam https://wiki.archiveteam.org/index.php/URLTeam
deleted 7mo ago
[deleted]
cagrimmett 7mo ago
I run an ArchiveBox instance locally. Recommended! https://archivebox.io/ https://archivebox.io/

We need something like SETI@home/Folding@home but for crawling and archiving the web or maybe something as simple as a browser extension that can (with permission) archive pages you view.

ryoshu 7mo ago

This is a good idea. Not sure what ToS it would violate. But a good idea.

pclmulqdq 7mo ago

Your TV probably does that, and you definitely gave it permission when you clicked "accept" on the terms.

macinjosh 7mo ago
good thing I don't have a TV!

ninjagoo 7mo ago

In the US at least, there is no expectation of privacy in public. Why should these websites that are public-facing get an exemption from that? Serving up content to the public should imply archivability. Sometimes it feels like ai-use concerns are a guise to diminish the public record. While on the other hand services like Ring or Flock are archiving the public forever.

sejje 7mo ago
Ring and Flock are not a standard we should be striving towards. Their massive databases tracking citizens need to go.

dunder_cat 7mo ago

This exists although not in the traditional BOINC space, it's Archiveteam^1. I run two of their warrior^2 instances in my home k3s instance via the docker images. One of them is set to the "Team's choice" where it spends most of its time downloading Telegram chats. However, when they need the firepower for sites with imminent risk of closure, it will switch itself to those. The other one is set to their URL shortener project, "Terror of Tiny Town"^3. Their big requirement is you need to not be doing any DNS filtering or blocking of access to what it wants, so I've got the pod DNS pointed to the unfiltered quad9 endpoint and rules in my router to allow the machine it's running on to bypass my PiHole enforcement+outside DNS blocks. ^1 https://wiki.archiveteam.org/ https://wiki.archiveteam.org/ ^2 https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior ^3 https://wiki.archiveteam.org/index.php/URLTeam https://wiki.archiveteam.org/index.php/URLTeam

deleted 7mo ago

[deleted]

cagrimmett 7mo ago

I run an ArchiveBox instance locally. Recommended! https://archivebox.io/ https://archivebox.io/