6 ms·
What does it compress the full 1GB file to? http://prize.hutter1.net/ http://prize.hutter1.net/
by 7373737373 3mo ago
What does it compress the full 1GB file to? http://prize.hutter1.net/ http://prize.hutter1.net/
- spidy__ 3mo agoI tried it on a enwik9 100 mb slice and was able to compress it to 20 mb + 900kb transformer so 21mb. I know the top submission was able to get it to 13 mb. Still trying some ideas to get better compression.
- gravypod 3mo agoSince you know the size of the file beforehand you may be able to overfit some kind of text diffusion model instead of a transformer? May allow you to partially correct the model output using some other method and then fill in the blanks that were wrong from previous generations.
- spidy__ 3mo agoOh, sounds interesting. I hadn't considered using a diffusion model for this. My current approach generates the document byte by byte with an autoregressive transformer, so I'm curious how a diffusion model would improve memorization or reconstruction quality. Can you point me to something that i can read? I really wanna try this approach , diffusion model does sounds interesting for compression.
- atiedebee 3mo agoWhich slice? The large text compression benchmark uses enwik8 for a "smaller" input that is easily reproducible. The predictability of enwik9 can vary significantly depending on where in the file you are, as shown by Matt Mahoney https://www.mattmahoney.net/dc/textdata.html https://www.mattmahoney.net/dc/textdata.html
- purple-leafy 3mo agoThanks for the link!
- cellular 3mo agoMaybe everyone should compress the 1st 100MB worth of digits of pi, for an apples-to-apples comparison? Edit: oh wait that's too easy. Need to generate /publish random digits so everyone can use it.
- saulpw 3mo agorandom digits aren't compressible though?
- SV_BubbleTime 3mo agoRandom digits are compressible though. Random data does not mean it does not match a pattern in your dictionary for example.
- deleted 3mo ago[deleted]
- gnabgib 3mo agoNo.. they're not. Do you understand random (the apparent or actual lack of definite patterns or predictability[0]) or compression (reduces bits by identifying and eliminating statistical redundancy[1])? [0]: https://en.wikipedia.org/wiki/Randomness https://en.wikipedia.org/wiki/Randomness [1]: https://en.wikipedia.org/wiki/Data_compression https://en.wikipedia.org/wiki/Data_compression
- thin_carapace 3mo agoby this definition, a random dataset could apparently present no patterns, while presenting non apparent patterns.
- ufocia 3mo agoSounds like presenting no patterns, apparently or otherwise, would be a pattern in itself.