6 ms·
Yes, it's absolutely possible—just like how diffusion LLMs work, we can do the same with DDN LLMs. I made an initial attempt to combine [DDN with GPT](https://
by diyer22 11mo ago
Yes, it's absolutely possible—just like how diffusion LLMs work, we can do the same with DDN LLMs.
I made an initial attempt to combine [DDN with GPT](https://github.com/Discrete-Distribution-Networks/Discrete-Distribution-Networks.github.io/issues/1 https://github.com/Discrete-Distribution-Networks/Discrete-D...), aiming to remove tokenizers and let LLMs directly model binary strings. In each forward pass, the model adaptively adjusts the byte length of generated content based on generation difficulty (naturally supporting speculative sampling).
- vintermann 11mo agoThis is what I find most impressive, that it's a natural hierarchial method which seems so general, yet is actually quite competitive. I feel like the machine learning community has been looking for that for a long time. Non-generative uses (like hierarchial embeddings, maybe? Making Dewey's decimal like embeddings for anything!) are even more exciting.
- diyer22 11mo agoExactly! The paragraph on Efficient Data Compression Capability in the original paper also highlights: > To our knowledge, Taiji-DDN is the first generative model capable of directly transforming data into a semantically meaningful binary string which represents a leaf node on a balanced binary tree. This property excites me just as much.
- cubefox 11mo agoThis sounds a bit like H-Net [1] or Byte Latent Transformer [2]. 1: https://arxiv.org/abs/2507.07955 https://arxiv.org/abs/2507.07955 2: https://arxiv.org/abs/2412.09871 https://arxiv.org/abs/2412.09871
- lukan 11mo agoI vouched for this comment. Your account seems to be shadow banned, but your last comments look fine to me, so you maybe want to email dang to revoke that status ..
- cubefox 11mo agoThanks. I sent an email.
- diyer22 11mo agoIt does seem that way — we’re both trying to overcome the limitations imposed by LLM tokenization to achieve a truly end-to-end model. And, their work is far more polished; I’ve only put together a quick GPT+DDN proof-of-concept. Thank you for sharing.