8 ms·

I think it depends on what you mean. I don't think that just the number of bytes of data should be used to measure the size of your dataset. For example, if I h

I think it depends on what you mean. I don't think that just the number of bytes of data should be used to measure the size of your dataset. For example, if I have 1TB of all 1's, this is a lot of data, but not very interesting. I think a more nuanced notion of size is the /information content/ of the datasets. I haven't thought about it carefully, but I'm sure you can quantify this more explicitly in terms of information theory or other non-bit-based complexity measures. From this point of view, literature may not be large (in terms of the number of bytes), but in terms of information content, it is incredibly dense. A large portion of human knowledge is written somewhere. From this point of view, it is in fact big data.

baddox 10y ago
Sounds like you may be interested in https://en.m.wikipedia.org/wiki/Kolmogorov_complexity https://en.m.wikipedia.org/wiki/Kolmogorov_complexity
- hyporthogon 10y ago
  Everything I've read by Chaitin is interesting and thoughtful, but this piece is particularly accessible and entertaining: https://arxiv.org/abs/math/0303352 https://arxiv.org/abs/math/0303352
Isamu 10y ago
Maybe you would find the theory of information interesting: http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf http://math.harvard.edu/~ctm/home/text/others/shannon/entrop... Here in his famous tour de force Claude Shannon lays out the way we can estimate the amount of actual information in an act of communication (e.g. a literary work) and relates it to system entropy. To your point, !TB of all 1's compresses to a just few bits of actual information. But I suspect you are not actually speaking about the information contained in a literary work. You are probably thinking: how much expansive commentary and explanation could a literary work spawn? That is another question. The answer is always: unbounded.
- killjoywashere 10y ago
  > The answer is always: unbounded. But after initial perturbations die out, the growth rate will likely look something like O(log n) or less.
killjoywashere 10y ago
> 1TB ... is a lot of data "You have a terabyte of data? For $30,000 you can put that in main memory. If your dataset fits in main memory, it's not big data." Source: Michael Stonebraker, January 2015.

baddox 10y ago

Sounds like you may be interested in https://en.m.wikipedia.org/wiki/Kolmogorov_complexity https://en.m.wikipedia.org/wiki/Kolmogorov_complexity

hyporthogon 10y ago
Everything I've read by Chaitin is interesting and thoughtful, but this piece is particularly accessible and entertaining: https://arxiv.org/abs/math/0303352 https://arxiv.org/abs/math/0303352

Isamu 10y ago

Maybe you would find the theory of information interesting: http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf http://math.harvard.edu/~ctm/home/text/others/shannon/entrop... Here in his famous tour de force Claude Shannon lays out the way we can estimate the amount of actual information in an act of communication (e.g. a literary work) and relates it to system entropy. To your point, !TB of all 1's compresses to a just few bits of actual information. But I suspect you are not actually speaking about the information contained in a literary work. You are probably thinking: how much expansive commentary and explanation could a literary work spawn? That is another question. The answer is always: unbounded.

killjoywashere 10y ago
> The answer is always: unbounded. But after initial perturbations die out, the growth rate will likely look something like O(log n) or less.

killjoywashere 10y ago

> 1TB ... is a lot of data "You have a terabyte of data? For $30,000 you can put that in main memory. If your dataset fits in main memory, it's not big data." Source: Michael Stonebraker, January 2015.