TheTechRobo

5 ms·

1.
by TheTechRobo 2y ago
Browsertrix (and Webrecorder tools in general) also violate the standard by modifying response data. It's supposed to be the raw bytes as they are sent over the network (minus TLS). The entire WARC ecosystem is kind of a mess.
2.
by TheTechRobo 2y ago
Unfortunately the Raspberry Pi version isn't supported anymore because their version of Wget isn't well-tested on ARM. I'm surprised it's still there in the repo. You can get Docker to emulate x86, though.
3.
by TheTechRobo 2y ago
That doesn't really confirm that, IMHO. The line before claims the opposite, in fact: "The Internet Archive has been around since 1996, and while that does not guarantee anything, it shows continuity of support and strong commitme
4.
by TheTechRobo 3y ago
Everything transmitted over HTTP. That includes cookies, passwords, etc. You need to be careful when writing WARCs.
5.
by TheTechRobo 3y ago
Yeah, the format really needs an update. For starters, WARC only officially supports HTTP/1.1. Webrecorder has started faking HTTP/1.1 data in WARC files in order to save other versions, but I don't think faking data is great
6.
by TheTechRobo 3y ago
ArchiveTeam's general idea is to archive it first, and make it usable later. Wayback Machine is generally how stuff is "made usable", but in cases like this, a searchable index might need to made. (That's been done befor
7.
by TheTechRobo 3y ago
Yes, what is your point?
8.
by TheTechRobo 3y ago
Archive Team hasn't really done much yet - all they've done is an ArchiveBot job, which isn't getting anywhere because of DPReview's ratelimits, and some discovery scrapes. This is probably a different issue.
9.
by TheTechRobo 3y ago
You select the project you run at any given moment. You can also select Archiveteam's Choice, which is usually the most urgent or the one with the most IPs needed, but if you do want to select it yourself, you pick the project.
10.
by TheTechRobo 5y ago
Similar: github.com/InternetArchive/warcprox
11.
by TheTechRobo 5y ago
There's github.com/ArchiveTeam/grab-site, but unfortunately it's not maintained very well.
12.
Forked a small Python text calculator and added a LOT of features(github.com)
1 points by TheTechRobo 6y ago|0 comments