6 ms·
There is no legal restriction for crawling any web site. But you should respect robot.txt document for every single web site. Such as: https://www.facebook.com/
by yamural 14y ago
There is no legal restriction for crawling any web site. But you should respect robot.txt document for every single web site. Such as: https://www.facebook.com/robots.txt https://www.facebook.com/robots.txt
http://web.archive.org/robots.txt http://web.archive.org/robots.txt