5 ms·
> if you don't crawl as frequently as google you are reliant on your crawl frequency being quicker than the 'content stealing' speed. No problem there. If a pa
by addingnumbers 5y ago
> if you don't crawl as frequently as google you are reliant on your crawl frequency being quicker than the 'content stealing' speed.
No problem there. If a page hasn't been crawled, it won't be a search result.
- Closi 5y agoI mean you need to crawl often enough that you can determine which site stole content. If a full crawl of the web takes 3 months and A steals content from B after 2 weeks, it may not be possible to easily infer that A has stolen from B rather than visa-versa. If I crawl every week and see that A posted it on the first crawl, and B suddenly also had that content on the second crawl, I can infer that B stole from A.