Y
HN Search
Hacker News Search
new
|
comments
|
top
|
jobs
node56
searching Neon…
1.
▲
2.
▲
3.
▲
4.
▲
5.
▲
6.
▲
5 ms
·
1.
▲
by
node56
15y ago
It seems like the web server should be able to report what has changed since a timestamp. The first step when crawling a site would be first to do something like an HTTP STATUS request, if that succeeds the bot could merely crawl the pages