5 ms·
It was meant to be an open-source compilation of the crawled internet so that research could be done on web search given how opaque Google's process is. It was
by jaybna 2y ago
It was meant to be an open-source compilation of the crawled internet so that research could be done on web search given how opaque Google's process is. It was NOT meant to be a cheap source of data for for-profit LLM's to train on.
*edit: added "for-profit"
- CamperBob2 2y ago(Shrug) Multiple not-for-profit LLMs have trained on it as well. If something I worked on turned out to play a significant part in something that turned out to be that big a deal, I'd be OK with it. And nobody's stopping people from doing web-search studies with it, to this day.