Y
HN Search
Hacker News Search
new
|
comments
|
top
|
jobs
freddielarge
searching Neon…
1.
▲
2.
▲
3.
▲
4.
▲
5.
▲
6.
▲
5 ms
·
1.
▲
by
freddielarge
12y ago
fun fact: robots.txt can also be used by attackers to find admin interfaces or other sensitive tidbits that you don't want search engines to crawl lots of target-detection crawlers will look at robots.txt as the first thing they do to