9 ms·

I believe OP posted it because the new Kimi K3 has 69 KDA layers (the rest are 24 Gated MLA), I think previous large Kimi models had only MLA layers.

• yorwba 2mo ago
  It's not the same KDA as used in Kimi Linear, though.
  • GaggiX 2mo ago
    What's the difference? They are both called Kimi Delta Attention.
    • yorwba 2mo ago
      The differences are explained in section 2.1.1 of the Kimi K3 technical report: https://arxiv.org/pdf/2607.24653#page=4 https://arxiv.org/pdf/2607.24653#page=4