Y
HN Search
Hacker News Search
new
|
comments
|
top
|
jobs
rli
searching Neon…
1.
▲
2.
▲
3.
▲
4.
▲
5.
▲
6.
▲
4 ms
·
1.
▲
by
rli
14y ago
Nothing to be seen in Chrome v23.0.1271.95
2.
▲
by
rli
14y ago
robots.txt can be ignored, it's just a reference for honest spiders. I think the way described above, of listing top requestors, doing statistics and then automating blocking is indeed the best way. Could also be there's a blocklist or two