6 ms·
Worse, the constant AI scraping is actually costing content providers additional money for no return. At least Google/Bing/Yahoo scraping would then be used
by Ensorceled 4mo ago
Worse, the constant AI scraping is actually costing content providers additional money for no return. At least Google/Bing/Yahoo scraping would then be used to provide links back to your content.
- fiedzia 4mo ago> At least Google/Bing/Yahoo scraping would then be used to provide links back That doesn't work anymore. Google provides AI generated summary, nobody looks at the original site.
- bolangi 4mo agoNot only costing money. Constant AI scraping constitutes a denial-of-service attack that has brought down websites.
- devsda 4mo agoHow do you distinguish Google/MS scraping for Gemini/Copilot vs Google Search/Bing? In the case of Google, the UA is the same and you are entirely at their mercy to honor the Google-Extended instructions in robots.txt Google has further complicated it with new search announcement blurring lines between regular search and AI search. And AI likes to not honor any licenses or instructions when it is hungry for training material. It is once again an example of Google using its dominant position to abuse and promote cross functional products.
- cute_boi 4mo agoIf company like Meta are downloading pirated books etc.. to train their AI, they will surely honor robots.txt.
- johneth 4mo agoI wouldn't be surprised if there isn't some sort of legal action against Google, the monopoly, to make the distinction in how their crawlers use scraped content.