6 ms·
Kimi K3 Architecture Overview and Notes
- gokohl 2mo agoInteresting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.
- wongarsu 2mo agoIs frontier scale larger than this? Kimi K3 seems to benchmark in the same range as Opus and Fable. I would have expected they are all in the 2-4T range, with quality of the training and architecture differences as the major differentiators
- porridgeraisin 2mo agoThe number of active parameters is vastly different. Deepseek CEO hinted that he estimates it as an order of magnitude difference in one of his recent interviews. > Seems to benchmark yes, but in human usage the differences show up
- kamranjon 2mo agoWould you happen to have a link to that interview? Sounds like an interesting read.
- porridgeraisin 2mo agoIt was posted to HN a few days back 1. https://news.ycombinator.com/item?id=49019012 https://news.ycombinator.com/item?id=49019012 (original chinese) 2. https://news.ycombinator.com/item?id=49052912 https://news.ycombinator.com/item?id=49052912 (translated english [pdf])
- onlyrealcuzzo 2mo agoIt's related to this: https://news.ycombinator.com/item?id=49052912 https://news.ycombinator.com/item?id=49052912
- thunderbird120 2mo agoKimi Delta Attention (KDA), despite having "Attention" in the name, isn't really attention at all in any conventional sense. It's more like an RNN which can be efficiently parallelized during training. It's a very small modification to Gated DeltaNet, which can be described as an RNN whose hidden state acts like a small, editable attention memory. Because it's RNN-like, it has an inherent idea that X comes before Y which comes before Z in the sequence XYZ. Transformers, by default, don't have that. They operate on sets, unordered collections of unique items. They have no idea where those items are in relation to each-other so you have to clue them in. Because There are 3 KDA layers per attention layer, and 3 KDA layers before the first attention layer, every single token position is going to be able to learn information about where it is in the sequence before the first actual attention layer. RoPE is actually a bit destructive, so being able to omit it like this is very convenient. Models like Gemma-4 have a similar structure with 5:1 Sliding Window Attention (SWA) layers for every global attention layer. These are cheap, shitty attention layers which handle local information and which go in-between the big powerful ones, KDA serves the same role in this model. In Gemma only the SWA layers have RoPE while the global attention layers omit it. SWA is actual attention, even if it only operates on a small sliding window, so it needs the positional embedding. KDA isn't, so it doesn't. >Curious to see if it holds up at frontier scale. I don't know how much more frontier scale you can get than this, but yes, there's no reason why that wouldn't work at larger scales. Honestly, more parameters just makes it easier for the KDA layers to communicate that positional information better.
- samuelknight 2mo agoIt has some weird side effects though. for example KV-caches are implemented in fixed incremental token blocks (1024 from the providers I used) instead of simply caching up to the most recent input prompt input. It results in up to 1023 additional input (cache miss) tokens per inference.
- wren6991 2mo agoSounds a lot like running the Qwen3.5/3.6-series models at home: you need checkpoints for the recurrent state (GDN in the case of Qwen). You avoid the miss for the common case of 100% prefix match (e.g. during interleaved tool calls and thinking) by keeping an additional checkpoint for the actual last generated token. If you're a cloud provider serving many concurrent clients then you might prefer to skip that complexity and always take the 1k worst-case prefill hit. SWA has a similar issue. Unless you keep the entire KV prefix lying around (which is not unreasonable: you retain flop + bandwidth benefits but lose capacity benefits), you need to start 1 window back from the rollback point, in order to refill the sliding window before going into normal prefill.
- cubefox 2mo agoThis comment smells LLM generated.
- abroadwin 2mo ago"Quietly"
- cubefox 2mo ago"at frontier scale"
- nextaccountic 2mo ago"—"
- Bolwin 2mo agoScreams it in fact
- losvedir 2mo agoHuh, which model? I didn't pick up any of the tells. It seemed normal enough to me. (I vouched it, oops.)
- cubefox 2mo agoTelling which model wrote it is much harder than telling that some model wrote it.
- alealvarezarg 2mo agoGreat breakdown. After using Kimi extensively, it's fascinating to see how architectural choices like KDA and NoPE translate into such strong real-world performance. Really impressive engineering.
- thatsgcasey 2mo agoSabastian Raschka is one of the great LLM researchers/authors. I highly recommend his substack
- 3abiton 2mo agoI second that. I've always enjoyed his work!
- rsolva 2mo agoThis seems like a rare gem in a space swamped with lazily LLM generated summaries. Added to my RSS feed!
- tomrod 2mo agoHis was the only Packt book (on ML, circa 2019 IIRC) that I have never regretted purchasing. Clear thinker, attention to details, and clean coder. Highly recommend.
- hugopuybareau 2mo agoNever heard of him before but I was amazed how straight to the point yet very easy to grasp it was
- Upvoter33 2mo agoI would describe him as a leading educator / explainer / teacher in this space.
- souravsspace 2mo agoi like your detailed breakdown. Thanks. <3
- Ilaurens 2mo ago"Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead." It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?
- yhocd 2mo agoThe primary author of RoPE, Jianlin Su, is currently working at Moonshot (KIMI)
- itkovian_ 2mo agoCausal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.
- ModelForge 2mo agoAnd adding to that, there is also the recurrent state in the Kimi Delta Attention. I wouldn't call it position information but more sth like "position sensitivity"
- cma 2mo agoThe paper says: "Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA."
- dwohnitmok 2mo agoAs a sibling comment points out you don't strictly need positional embeddings for decoder-only causal transformers. You definitely need it for non-causal ones (e.g. the encoder of the original transformer paper!). And yes accumulation is a good intuition for what's going on. You could imagine a part of the attention head that just kept writing to the same part of the residual stream causing that to keep accumulating (simply via attention summation) as more input tokens come in thereby functioning as a kind of index without the need for any positional encoding.
- pullrun 2mo ago[dead]
- myronkeir1968 2mo ago[flagged]
- myronkeir1968 2mo ago[dead]
- dumpstertechops 2mo ago[flagged]
- constantlm 2mo agoSo, unlike what leaders of western labs labs would like you to believe (that Kimi is just the result of distillation attacks), they are introducing new and novel approaches.
- boomskats 2mo agoNo! It's just not fair!! What about our financial bubble!
- hlynurd 2mo ago"attack"
- nostromo 2mo agoEven if they are distilling, I don't particularly care. Anthropic and others have been distilling copyrighted material by to build these models, largely without permission.
- wnsdy95 2mo ago[flagged]
- mensetmanusman 2mo agoIf you redefine the technical world distilling to mean training on text, you are correct.
- deleted 2mo ago[deleted]
- brookst 2mo agoWe’re all distillers now.
- xdennis 2mo agoPeople don't remember now, but in the old HN comments like "this is so true honestly" would have been downvoted. If a comment just agrees with the parent, people would have said that's what the upvote button is for. Adding a "I agree" comment is just noise.
- RobLach 2mo agoConcise. Nice.
- rglover 2mo agoJust tried K3 out for the first time today and it's a legitimate threat. Temporarily (maybe permanently) using it as my daily driver but it's wild how comparable it is to Opus 4.7/4.8 (what's been my go to for a bit now—wrote a quick post on what I found today [1]). [1] https://graybearding.bearblog.dev/kimi-k3-is-insane/ https://graybearding.bearblog.dev/kimi-k3-is-insane/
- kroaton 2mo agoBetter than Opus 4.8 on complex tasks but tends to overthink. It found a bunch of bugs and architecture issues that only 5.6 Sol Max and Fable on my C++ projects.
- ATMLOTTOBEER 2mo agoAgreed. Dropped my $200/mo Claude I was saturating for the $100/mo moonshot sub and have no complaints Plus I can actually see the thinking traces and I’m not getting downgraded/denied by anthropic constsntly…
- nmfisher 2mo agoHow much usage do you get out of the $100 Moonshot plan? I haven't heard great things.
- KronisLV 2mo agoFor me it got exhausted faster than I'd like, subjectively less value than the Anthropic 100 USD subscription, BUT their 5 hour limits are very generous, whereas I keep bumping into Anthropic's all the time: https://blog.kronis.dev/blog/kimi-k3-is-out-is-anthropic-done/ https://blog.kronis.dev/blog/kimi-k3-is-out-is-anthropic-don... Their 200 USD tier is pretty good and I might move to them because their annual discounts also make it even cheaper. I dislike the way how Opus/Fable write and Kimi has less of that slop writing problem, but also ends up feeling kinda slower (especially on Max reasoning), technical abilities are not 100% the same but close (better in some tasks, worse on others).
- 2mo ago
- jartan2002 2mo ago[flagged]
- mickael-kerjean 2mo agoGenuine question: how reproducible / usable / verifiable are these architectures from the published documentation? Are they similar to PDF/DWG/PSD specifications, where the format look like an open spec at first sight until you attempt to implement it and realize the crucial implementation details are undocumented?
- GaggiX 2mo agoThe code is open source, and there probably several different implementations.
- calebkaiser 2mo agoImplementing models directly from papers is typically pretty doable (and is of course more straightforward when the full implementation is open sourced). Often there is some amount of specific knowledge, like particular hyperparameters, that is missing and has to be trial and errored by the community, but generally speaking, getting the core model architecture implemented is a reasonable task for most well documented models. Reproducing the exact training run, however, is basically impossible without the original dataset and training pipeline (here meaning all of the code + infra involved in actually executing the pre and post training loops). Also, it would be exorbitantly expensive to do if you weren't also a lab trying to train a similar model. But you can still scale the architecture down and experiment as a solo researcher using the published research. There are probably some open source implementations already on GitHub for any given big open model release.
- marcyb5st 2mo agoThe exact training run is basically impossible anyway. Randomness plays a role. Even if you fix your RNG seed, in a distributed training scenario like this one some weight updates might come at different times and be included in different update steps. Should have minimal impact on the final outcome, but would still be a different model as some of the weights will differ in the end.
- eru 2mo ago
- augment_me 2mo agoI feel like the Kimi team is amongst the best in the industry to pick and choose what is meaningful from the other models. For example, avoiding the expensive and empirically uncertain mHC in favor of simpler residuals. Latent MoE. My only doubts are around Linear Attention instead of DSA as this is inherently lossy. You are kind of banking on that your query is inherently in the embedding space of the model already and can be lossy.
- mensetmanusman 2mo agoThey need to work on token cost for unrestrained problem solving.
- gboss 2mo agoAnybody getting the result that Kimi 3 is more expensive than Opus 5 or Sol on Cursor? Pretty sure Kimi 3 sucked up a good chunk of my ultimate plan in a few prompts. Anyone have any tools or ways to understand per model usage towards cursor subscriptions? I know there are alternatives to cursor just haven’t made the move yet. (Edit spelling)
- cadamsdotcom 2mo agoOh no! First time dealing with an American corporation?
- igravious 2mo agoThe assertions doing the rounds that Kimi K3 and GLM 5.2 are way cheaper than Claude/GPT are not true -- DeepSeek V4 Pro is a lot cheaper but K3 and 5.2 ain't. Turns out that you actually have to fork out some cash for frontier-esque models, be they Chinese or American. Hope that helps. Source: my bank balance
- negativezone 2mo agois it cheaper for medium scale folks? not everyone needs to go big on ai subscriptions and a solid, reliable midrange option wouldn't go amiss
- TokenLat 2mo ago[flagged]
- madhu_ghalame 2mo ago[flagged]
- anentropic 2mo agoAsk Kimi
- rekpero 2mo agokimi K3 felt on par with my claude opus 4.8 in my first tests and is exceptionally good at frontend and website motion design, but I still find it hard to switch back and forth between claude and kimi, and i’m not sure if others are experiencing the same thing.
- dools 2mo agoThey may have all this amazing architecture but Kimi has been super dumb recently. I reckon they’re under compute pressure and quantising to stay afloat. I was a heavy k2.5/2.6 user earlier in the year and built serious features with it, but even k3 now does stupid shit like fail tool calls and get stuck in endless thought trains. K2.6 was spinning its wheels on a problem for over 10 minutes today and then deepseek v4 fixed it in under a minute. Something is wrong at moonshot.
- rurban 2mo agoDeepseek is also struggling with simple tasks for the last two days. Claude is perfect, with the additional credits.
- dools 2mo agoI’ve had no issues with deepseek at all. Maybe it’s just luck of the draw
- rurban 2mo agoGood to hear. Most likely one of those monthly hickups, destroying main. Needed lots of reverts, and manual fixing. If in a branch I wouldn't care that much.
- Aldipower 2mo agoI have a lot of problems with Opus 5 in the last days. So, maybe the problem is to expect reliability from something probabilistic?
- KronisLV 2mo agoMight be a bit more than just noise: https://marginlab.ai/trackers/claude-code-historical-performance/ https://marginlab.ai/trackers/claude-code-historical-perform... Edit: well, crap. Model overloaded retrying (6/10) 1m 40s Thanks, Anthropic.
- 2mo ago
- Giho 2mo agoI wonder if Kimis KDA architecture is a way closer to online models(models that learn during interference).As KDA updates its state matrix: St =St−1 +βt (vt −St−1 kt )ktT The expression (vt −St−1 kt )ktT is literally the gradient step of a linear regression model updating its weights (S) on a single sample using Mean Squared Error loss. Interesting what other labs comes up with from this.
- deleted 2mo ago[deleted]
- deleted 2mo ago[deleted]
- ngcazz 2mo agoI know some of these words, but I'd like to understand all of it better. Would anyone be so kind as to recommend a good learning resource for LLM design and the math involved?