7 ms·
They also write this: > Since a user requested the fetch, this fetcher generally ignores robots.txt rules.
by hadrien01 1y ago
They also write this:
> Since a user requested the fetch, this fetcher generally ignores robots.txt rules.
- dabeeeenster 1y agoI mean, that's just not true.
- esskay 1y agoWhich part? It's widely established and known that many AI crawlers are ignoring the robots.txt file, perplexity being one of them [1] [1]https://www.tomshardware.com/tech-industry/artificial-intelligence/several-ai-companies-said-to-be-ignoring-robots-dot-txt-exclusion-scraping-content-without-permission-report https://www.tomshardware.com/tech-industry/artificial-intell...
- simonw 1y agoOh wow, I missed that! That's from the docs for that Perplexity‑User user-agent, at which point presumably there's no point in listing that in robots.txt at all?
- bitpush 1y ago> Since a user requested the fetch, this fetcher generally ignores robots.txt rules. Normally the expecation is that the user-agent faithfully presents the content it fetched. If I make a browser that fetches bbc.com, and strips away ads and presented it to users - I would expect BBC to not like it and block the user-agent from accessing it. It isnt a robots.txt thing. It is a user-agent thing.