5 ms·
The number of disallow we found in the robots.txt files actually surprises us. Companies like OpenAI also have to do a lot of things to ensure the compliance t
by flowinghorse 3y ago
The number of disallow we found in the robots.txt files actually surprises us.
Companies like OpenAI also have to do a lot of things to ensure the compliance to the regulation.
- LinuxBender 3y agoto ensure the compliance to the regulation Did legislation pass requiring people and their bots to obey robots.txt? If so I totally missed it. That would be big news if so.