6 ms·

"bro": https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt?op=1 https://www.businessinsider.com/openai-anthropic-ai-i

"bro": https://www.businessinsider.com/openai-anthropic-ai-ignore-rule-scraping-web-contect-robotstxt?op=1 https://www.businessinsider.com/openai-anthropic-ai-ignore-r...

wilg 6mo ago
should be pretty easy to test and not rely on an anonymous source from a weird analytics company via business insider. are these bots actually from openai or are they just using their user agent? are they coming from openai ip ranges? etc. https://openai.com/gptbot.json https://openai.com/gptbot.json
- stefanka 6mo ago
  Are all of OpenAI’s ip ranges known?
- ImPostingOnHN 6mo ago
  > should be pretty easy to test I look forward to your results, whether or not they disprove the article.
stefanka 6mo ago
The article is from 2024. Is this still happening?
- ImPostingOnHN 6mo ago
  Do we have any evidence they started complying? If not, we can conclude they did not, until such evidence shows up.
  - stefanka 6mo ago
    I’m genuinely curious to know whether there was a change in behavior especially after OpenAI informed about how to prevent scraping (robot.txt, etc.).
    - ImPostingOnHN 6mo ago
      I am as well. Like, is there any evidence of a change, or can we assume nothing changed?