6 ms·
> - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is a very effective wa
by datatrashfire 1y ago
> - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string.
This is a very effective way to make sure you won't get any scraping done!
- lucb1e 1y agoTell that to Googlebot, Bingbot, Petalbot, SemrushBot, MJ12bot, MojeekBot, DotBot, YandexBot, SeznamBot, Barkrowler, AhrefsBot, DuckDuckBot, AcademicBotRTU, Bytespider, Applebot, ZoominfoBot, TelegramBot, TwitterBot, SemanticScholarBot, redditbot, Pinterestbot... From a quick peek at my access log, all include either a link (most) or an email address (zoom, tiktok/bytedance, dotbot, and that academic bot) Very few individual bots don't follow this good practice. Most of the IP ranges of violating bots are owned by Huawei (a few is Huawei Cloud so it could be anyone, but the majority seems to be Huawei themselves) and the remainder is all small beans as far as I remember (few thousand accesses in a day and then disappear forever, for example)
- datatrashfire 1y agoNone of the institutional market intelligence products I’ve ever worked on in nearly a decade of doing this do. Why? Cause they wouldn’t otherwise work. Many APIs require specific user agents. Tools like curl impersonate require specific user agents.