4 ms·I didn't see it mentioned, but why not just use robots.txt? Does Bytespider ignore it?by jd20 2y agoI didn't see it mentioned, but why not just use robots.txt? Does Bytespider ignore it?chptung 2y agoNot super sure so this felt like a faster way to plug it ASAP.hajimuz 2y agoYes. It ignores robots.txt.