5 ms·
> Why is git.kernel.org “interesting” to crawlers I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit
by mzajc 19d ago
> Why is git.kernel.org “interesting” to crawlers
I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests.
The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this means billions of links for all combinations of parameters and hashes. That, or it's a deliberate DDoS attack.
- vintermann 19d agoA lot of work is apparently put into bypassing any kind of anti-scraping, no work is apparently put into figuring out if the site freely gives a way to get all that information in a less wasteful way.
- sigbottle 19d agoCode (not AI) in general is the worst at semantics. Not an excuse for their practices, but it makes sense they wouldn't try to figure out something more advanced. If they did, their solution would probably be something like, "try all current approaches we have"; it wouldn't be fine grained or truly reasoning at all unless they stuck an actual AI in front of it.
- TonyTrapp 19d agoExactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP. They probably don't even recognize what's inside those pages and that there's an easier way to obtain the same result.
- diegocg 19d agoWhich, as the post notes, it's incredibly stupid. So much for artificial "intelligence"
- emsign 19d agoMakes me wonder how much garbage they actually collect across the web. That can't be good for the quality of the LLM.
- jeremyjh 19d agoThe crawlers are not AI. The crawlers are deterministic. They are collecting data to train AIs.
- jonhohle 19d agoBut gitweb is probably the second most used method of hosting a git repo and easily recognizable through heuristics. If it’s gitweb, fallback to git access and save everyone, including the crawler, time and resources.
- jeremyjh 19d agoUnderstood. The comment I replied to suggests the crawlers should figure this out on their own.
- TonyTrapp 19d agoNo, when I said this behaviour is stupid I did not imply that it needs to be solved with "intelligence". This class if problem is already solved by traditional crawlers, AI companies just actively chose to disregard any of that in their race to the top / bottom.
- TonyTrapp 19d agoSo? You can design a deterministic crawler for AI that - Respects robots.txt - Detects if something is likely a git forge and as a result - Clone the git repository instead of trying to crawl the forge None of this needs to be happening, and yet it does at an unprecedented scale.
- deleted 19d ago[deleted]
- asddubs 19d agoI run a non source code service and I also get these, also getting hit by literally millions of residential IPs being cycled after 1 - 2 hits. Ingesting thousands of copies of near identical pages. The bots seem to be programmed in an extremely simple manner, using non-obfuscated javascript to write the href attribute seems to stop them from seeing links, where previously they would endlessly crawl generated links. Unfortunately it also means noscript users get shafted.
- xiconfjs 19d agoSame here. Have a small gitea instance with cloned projects from github I‘m keeping in case the github version gets removed. Every few days there is an army of bots hammering my small vm with 40k req/min for 20 min straight. I had to install anubis to keep the instance online (while still slowed down).
- panki27 19d agoI don't even have anything worthwile on my gitea instance, and the same thing happened to me. The only thing that ended up helping was disabling access for anonymous users.