spidy__

8 ms·

1.
by spidy__ 3mo ago
yep!!
2.
by spidy__ 3mo ago
Ummm I don't have a lot of idea about crypto but in compression the idea was that you just share the model along with the metadata file to the other user and they de compress it, not sure how things will differ in crypto
3.
by spidy__ 3mo ago
If I were only using a transformer that would have been true, but we use arithmetic coding alongside our transformer to fix those mistakes (layman terms). You can read about arithmetic coding, its a pretty cool topic.
4.
by spidy__ 3mo ago
Ohh that "what I tried that didn't work" was a section that I specifically wrote myself (and then polished with AI) because I wanted to document what are the different approaches I tried to compress more but failed. Also than
5.
by spidy__ 3mo ago
I mean neural compressors provide great compression, BUTT the issue is they are really slow like in my project it takes around 45 minutes for de compression of 100 mb so I doubt if it would be useful, also using a transformer in user's
6.
by spidy__ 3mo ago
I tried with zstd 19
7.
by spidy__ 3mo ago
Ofcourse, am doing this just because I enjoy it. While we are on the discussion I have mentioned a question at the end of the discussion around an assumption am trying, can you please check it out and see if you have any suggestions? Would
8.
by spidy__ 3mo ago
Ohh yeah , I took it from Project Nayuki as mentioned in the file as well, i tried to pip install it but there were some issues so just took the file and kept the copy right as it is. Its not an issue is it? I am not sure.
9.
by spidy__ 3mo ago
Oh, sounds interesting. I hadn't considered using a diffusion model for this. My current approach generates the document byte by byte with an autoregressive transformer, so I'm curious how a diffusion model would improve memorizat
10.
by spidy__ 3mo ago
1. I wrote the content as what i want to mention in the documentation and just used AI to polish it so that its easy to understand, is it hard to understand the documentation right now? 2. Have added the link for downloading both the enwik9
11.
by spidy__ 3mo ago
I might be confused by the question, but I overfit the model on a single file and then transport the model along with the arithmetic coding file. There have been ideas where you generalize a model (constant weights) and then pass the arithm
12.
by spidy__ 3mo ago
Yeah yeah, I just found the idea kinda interesting so wanted to implement it
13.
by spidy__ 3mo ago
Sounds interesting man, soo am a bit confused maybe but can you run this on enwik9?
14.
by spidy__ 3mo ago
For the model overhead to become significant enough to eat into the gains, the file size would need to be fairly small, right? I assumed nobody would use this for compressing anything below 100 MB. I tested with 100 MB files because anythin
15.
by spidy__ 3mo ago
Reallly?? So have you published something so far? Can i read something? Sounds like you got some interesting ideas.
16.
by spidy__ 3mo ago
I tried it on a enwik9 100 mb slice and was able to compress it to 20 mb + 900kb transformer so 21mb. I know the top submission was able to get it to 13 mb. Still trying some ideas to get better compression.
17.
Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB
112 points by spidy__ 3mo ago|71 comments