5 ms·Is it only considered malicious if it disobeys robots.txt?by baltcode 15y agoIs it only considered malicious if it disobeys robots.txt?