6 ms·
This sounds very wrong to me. Take the C4 training dataset for example. The uncompressed, uncleaned, size of the dataset is ~6TB, and contains an exhaustive E
by jesse__ 7mo ago
This sounds very wrong to me.
Take the C4 training dataset for example. The uncompressed, uncleaned, size of the dataset is ~6TB, and contains an exhaustive English language scrape of the public internet from 2019. The cleaned (still uncompressed) dataset is significantly less than 1TB.
I could go on, but, I think it's already pretty obvious that 1TB is more than enough storage to represent a significant portion of the internet.
- FeepingCreature 7mo agoThis would imply that the English internet is not much bigger than 20x the English Wikipedia. That seems implausible.
- jesse__ 7mo ago> That seems implausible. Why, exactly? Refuting facts with "I doubt it, bro" isn't exactly a productive contribution to the conversation..
- onraglanroad 7mo agoBecause we can count? How could you possibly think that Wikipedia was 5% of the whole Internet? It's just such a bizarrely foolish idea.