7 ms·
They do respect robots.txt matter of fact, it will remove the website outright and most of its archives will be hidden until that website's robots.txt is offlin
by jrwr 5y ago
They do respect robots.txt matter of fact, it will remove the website outright and most of its archives will be hidden until that website's robots.txt is offline.
Their documentation about it is rather crap right now, but its in several of their FAQs about it
- pseudalopex 5y agoRespecting robots.txt would mean not saving pages blocked by robots.txt. Not just hiding.