5 ms·
I also run a public cgit instance and get over 1M hits every day, although my pet projects are nowhere near the size or impact of kernel. I had to block (via ng
by virgoerns 17d ago
I also run a public cgit instance and get over 1M hits every day, although my pet projects are nowhere near the size or impact of kernel. I had to block (via nginx conf) cgit endpoints for diffs, blame, snapshots and historical commits, because nothing else works. Now they return 402 (payment required). I consider this my total defeat and it's killing me inside, but it is what it is.
- inigyou 17d agoYou could also publish a list of IP addresses.
- mzajc 17d agoAs the article describes, it doesn't help, because the traffic originates from millions of unique residential IPs across hundreds of ASNs and countries.
- inigyou 17d agoSo?
- VladVladikoff 17d agoHave you tried blocking a million IPs before? Fail2ban gets pretty shaky at even 200,000 The AI crawler traffic I’ve seen sends one request per ip and seemingly has an infinite pool of residential IPs. You can’t block the ASNs becuase you also block honest clients. IP blocks are the wrong solution. And because I’m being negative I’ll also be constructive, IMHO the correct solution for fighting residential proxy crawlers is using RTT diffs this is one example https://github.com/Sakura-sx/Aroma https://github.com/Sakura-sx/Aroma
- pmlnr 16d agoFail2ban becomes a serious bottleneck at significant traffic. I've replaced it with a shell script and direct pf commands that run every few minutes.
- virgoerns 16d agoDo you mind sharing your script? I also noticed that fail2ban currently eats ~25% of CPU and started thinking if it's even worth running at this point
- pmlnr 16d agohttps://petermolnar.net/article/asn-blocking-freebsd-pf-nginx/index.html https://petermolnar.net/article/asn-blocking-freebsd-pf-ngin... It's not the most sophisticated thing.
- left-struck 17d agoRead the article, it will tell you why it won’t work.