5 ms·
That is a very good point. Security crawlers should probably use a masked user-agent.
by idoco 7y ago
That is a very good point. Security crawlers should probably use a masked user-agent.
- marksomnian 7y agoI'm fairly sure Google's search crawler already uses a masked UA, to detect when pages serve it different content than they do to users.