5 ms·
Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.
by itkovian_ 2mo ago
Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.
- ModelForge 2mo agoAnd adding to that, there is also the recurrent state in the Kimi Delta Attention. I wouldn't call it position information but more sth like "position sensitivity"
- cma 2mo agoThe paper says: "Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA."