8 ms·

Kimi-K3 Technical Report [pdf]

• kiaansaraiya 2mo ago
  [dead]
• m00dy 2mo ago
  I would want to see three things before drawing strong conclusions: End-to-end tokens/sec and cost on realistic coding agent trajectories, including tool outputs and retries, not isolated decode benchmarks. Cache hit rates and prefill cost for branching, multi-turn sessions. Router-load distributions after post-training, where expert collapse or specialization problems often show up.
• storus 2mo ago
  What would be the current best method to fine-tune it for my own specific agentic tasks? LoRA + DPO? GRPO? Something else?
  • whimsicalism 2mo ago
    LoRA + SFT, but it'll be big - better to wait for a finetuning API from one of the providers, I wouldn't jump straight to RL or off-policy pseudo-RL like DPO.
• m_ke 2mo ago
  Also open sourced a bunch of infra to go with it. Anyone who claims open source and open weights models are "decel" needs to get their head checked https://github.com/MoonshotAI/MoonEP https://github.com/MoonshotAI/MoonEP https://github.com/kvcache-ai/AgentEnv https://github.com/kvcache-ai/AgentEnv https://github.com/MoonshotAI/FlashKDA https://github.com/MoonshotAI/FlashKDA
  • jvanderbot 2mo ago
    This comment would be much better without the second line
    • ike_a 2mo ago
      I'm not sure I understand the case for open-source models being decelerationist, is this it? Decel: - Potentially reduces investor appetite for funding big labs. - More risk of powerful AI getting in bad hands -> more regulation. Accel: - More competition so big labs can't rest on laurels. - More research in open, so all labs can accrete advancements faster. I feel like open-source = acceleration has a much more clear argument. (and how bad would deceleration be in any case?)
      • Iolaum 2mo ago
        Open Source models decelerate growth of closed AI. For people who think (or want) AI = closed_AI then that argument has weight. Good luck getting them to update their priors.
      • StevenWaterman 2mo ago
        I think it's basically open weights => more inference competition => less profit from inference => less training competition
        • f311a 2mo ago
          > less training competition I think you meant less research and experiments in big labs because they don't get all the AI money. Training is expensive, but they also have more than 10 000 of employees combined and they cost a lot of money.
      • zozbot234 2mo ago
        Open source AI is actually a lot less "powerful" than genuine frontier models, i.e. it has a much tighter inherent capability ceiling. This is "decelerationist" from a purely AGI-pilled point of view but it's actually great if you're worried about a capabilities arms race putting AI Safety at severe risk. Kimi K3 is plausibly a lot less dangerous than a totally jailbroken ChatGPT/Gemini/Claude Sonnet (let alone Opus or Fable!) and it's quite deeply weird how no one seems to be calling for those models to be banned or restrained by further regulation. Why the double standard against the less concerning (but more efficient!) open weight models?
• samxli 2mo ago
  [flagged]
• fahrradflucht 2mo ago
  License: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE > If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 20 million US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must enter into a separate agreement with Moonshot AI before using the Software or its derivative works for any commercial purpose. + the existing 100 million monthly active users, or more than 20 million US dollars for commercial products have to name Kimi clause
  • throwaway27448 2mo ago
    I wonder how they'll figure out who to target for litigation when this license is violated.
    • ffsm8 2mo ago
      They haven't litigated the last public non-compliance... Despite that one being extremely public. So probably not at all for now.
      • Iolaum 2mo ago
        Maybe because a non public agreement was in place?
      • JumpCrisscross 2mo ago
        > They haven't litigated the last public non-compliance What was it?
        • deleted 2mo ago
          [deleted]
        • drawnwren 2mo ago
          Cursor was thought to be but they were later found to be using an authorized provider [1] 1 - https://x.com/Kimi_Moonshot/status/2035074972943831491?lang=en https://x.com/Kimi_Moonshot/status/2035074972943831491?lang=...
• whimsicalism 2mo ago
  It's funny that we've finally returned to tanh activation functions, time is a circle.
  • pinkmuffinere 2mo ago
    Wow this is fascinating enough that I’m actually going to read tfa lol
  • rhdunn 2mo ago
    From the paper (page 6 with a comparison to GLU and SwiGLU) they are not using tanh directly (i.e. f(x) = tanh(x)) but: f_gate(b,x) = b * tanh(x / b) * sigmoid(x) f_up(b,x) = b * tanh(x / b) Looking at the graph I wonder if this is to try and get the best of both GLU (better representation at higher values of x >~ 5) and SwiGLU (the value bump just before 0).
    • whimsicalism 2mo ago
      Still more of a tanh than I've seen in years
    • lostmsu 2mo ago
      Which part of that is doing gating? I thought gate generally looks like f_gate(x, y) = f(x) * y for some f. In your case b is a constant hyperparameter though, e.g. f_gate(x) = 4 * tanh(x / 4) * sigmoid(x) So where's the gate?
      • rhdunn 2mo ago
        In the linked "Kimi-K3 Technical Report [pdf]" paper, section 2.3 (Stable LatentMoE, p6) has the table with those equations on (top of p7, using β_1 for the gate branch and β_2 for the up branch). They talk specifically about the function in section 2.3.2 (Sigmoid Tanh Unit GLU, bottom of p7). Figure 2 (p3) has the architecture diagram for K3 (which is the same one on the Kimi K3 blog post https://www.kimi.com/blog/kimi-k3 https://www.kimi.com/blog/kimi-k3). AFAICT, that diagram along with section 2.3 should answer your question. Note: I'm not familiar with LLM/NN architecture to answer it more precisely than that; I only have a surface level understanding from watching various YouTube channels like 3B1B and Welch Labs on the subject.
• a-dub 2mo ago
  knowledge graph guided task synthesis. very cool! i have long wondered about the "how do you get good coverage of all the tasks" problem. maybe some interesting theoretical work there around the rate of production of new knowledge itself and various mechanisms (human approaches, mechanistic approaches, etc).
• histiq 2mo ago
  [flagged]
• eamag 2mo ago
  Can someone explain what are teachers in Multi-Teacher On-Policy Distillation? I can imagine math, coding and other verifiable domains, but they also have biology? Is it where distillation from bigger models come in?
  • porridgeraisin 2mo ago
    They're other models yes.
  • sosodev 2mo ago
    They reference https://thinkingmachines.ai/blog/on-policy-distillation/ https://thinkingmachines.ai/blog/on-policy-distillation/ If I understand correctly, it's distillation via having a teacher model score each of the student's tokens for a problem based on their own probabilities of generating that token at each step in the sequence. The reward/loss is then applied as RL. The multi-teacher bit seems to imply they're distilling from multiple models. It's light on the details, but it seems like it could be part of distilling from frontier/closed models. Provided they calculate the logprobs, which OpenAI seems to allow via API but not Anthropic. Maybe they have a way of estimating the logprobs externally? This method can be used to learn any domain from the teacher. Biology included.
    • mike_hearn 2mo ago
      It's not that light on the details. I read the paper and they train several different models in parallel over a few different domains and then they distill from their own models to get the final model.
    • htrp 2mo ago
      > which OpenAI seems to allow via API but not Anthropic Does openai still allow logprobs in their current gen models?
• colesantiago 2mo ago
  This is amazing to witness. Moonshot open sourcing Kimi K3, a frontier AI and other components really means we are getting abundant AI for all of humanity. Kudos to Moonshot for truly being what OpenAI should have been. Fable-level and frontier AI should be open source and available to everyone for free.
  • embedding-shape 2mo ago
    > Kudos to Moonshot for truly being what OpenAI should have been. Kudos to Moonshot for making these weights available for download. Lets not fool ourselves and claim these are "open source" by any understanding of the concept though, there are usage restrictions (even if you download them) and also training data isn't clearly broken down either, nor it it actually using a FOSS license.
    • lossolo 2mo ago
      > Kudos to Moonshot for making these weights available for download. Lets not fool ourselves and claim these are "open source" by any understanding of the concept though, there are usage restrictions (even if you download them) and also training data isn't clearly broken down either, nor it it actually using a FOSS license. They will probably never release the training data because that represents a large part of their competitive moat. The same is true of US companies (Google, OpenAI, Meta etc) none of which has released the full training data for its open models. They use private datasets that cost a lot to acquire, synthetic datasets and a lot copyrighted material for which they don't have licensing. What matters most is that, with the necessary hardware, I can download a near frontier model, run it and modify it however I want. The other concerns you mentioned are just noise. And if my company is generating $20 million in revenue or serving 100 million users, it can probably afford a relatively inexpensive commercial license.
      • bilbo0s 2mo ago
        >They will probably never release the training data because that represents a large part of their competitive moat Not that what you've written isn't the case. However, in addition to what you've written, (or probably even before what you've written), there's the fact that everything they're training on is stolen IP. Same with US LLM labs. Let's not kid ourselve's about where the training data is coming from. They are not asking artists, writers, coders, content creators, etc etc etc for permission to use their creations. Anthropic, Moonshot et al are doing incredible things, but we shouldn't gloss over the costs. Both present and future costs are kind of enormous.
• eamag 2mo ago
  > we build a self-evolving, hierarchically organized knowledge graph that agents continuously expand through web-scale exploration across knowledge-intensive and coding domains That's interesting!
• tudou527 2mo ago
  [flagged]
• lenerdenator 2mo ago
  What would it take to get an American open model to compete with this?
  • boomskats 2mo ago
    An act of G̶o̶d̶ Congress?
  • chrsw 2mo ago
    Something beyond my imagination
  • embedding-shape 2mo ago
    Latest "big" release from any of the bigger American lab must have been GPT-OSS-120b I think? Released ~summer 2025, so pretty much one years ago. Doesn't seem like it'll happen by itself, so something either forcing their hand figuratively, or something forcing their hand literally. Personally I was wishing/hoping for one of the recent Gemma releases to be in the ~100B class at least, but sadly Google is keeping that all for themselves.
    • layer8 2mo ago
      August 2025.
      • embedding-shape 2mo ago
        Bah, of course, thanks :)
    • lossolo 2mo ago
      Yeah, besides that, the only big other open source US model that was worth looking at was Inkling (975B params), Jul 15, 2026. https://thinkingmachines.ai/news/introducing-inkling/ https://thinkingmachines.ai/news/introducing-inkling/
    • Stagnant 2mo ago
      NVIDIA-Nemotron-3-Ultra-550B-A55B was released in June 4th 2026 and I think it was the largest open US model until thinking machines' Inkling (975B) was released a couple of weeks ago.
• brcmthrowaway 2mo ago
  Is this p-hacking?
• deleted 2mo ago
  [deleted]
• abratabia 2mo ago
  [flagged]
• weberer 2mo ago
  Does anyone know if a torrent is available? I think it would take quite a while to download 1.5tb from their servers.
  • se4u 2mo ago
    Your best bet will be to start a network optimized ec2 (or gcp compute instance) and save to some bucket. Cloud network will be a lot more stable than home.
• GodelNumbering 2mo ago
  Back of the envelope calculation (could be off, correct me if I am) If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 parallel agentic workflows (each with ~100k context on average) at ~30 tok/s. Assuming the annual amortization+electricity at $1.5M/year and about 50% average annual utilization, you get less than 60 cents (USD) per million output token, for a frontier model with plenty of capacity to share, all your data never leaving premises and well over an order of magnitude cheaper! As long as a company believes that the openweight models will continue to get more capable and 'AI is here to stay', this model provides the first solid footing for a decision to just buy a rack.
  • reckless 2mo ago
    I think the licensing that would likely apply to a company that's able to afford ~$6M rack and the associated infrastructure muddies this somewhat
    • vidarh 2mo ago
      As far as I can tell, license fees are only applicable if you have more than 20m USD/year revenue from services provided using the model, or serve more than 100m users.
    • petu 2mo ago
      I think internal use is allowed at any scale in the license? > 4. The requirements set forth in Sections 2 and 3 do not apply to: (a) internal use of the Software, defined as any use that does not make the Software, its outputs, or its underlying capabilities available to third parties; [...]
      • jsnell 2mo ago
        It's very hard to make sure none of the outputs are ever made available to third parties. Source code can end up widely distributed (e.g. client-side js, open source). Prose will frequently get shared across organization boundaries (e.g. emails, websites, documents).
• Tejush 2mo ago
  Any guess on the pre-training tokens/flops they consumed?
  • lostmsu 2mo ago
    Technical report has a graph vs Kimi K2 with 1e21 FLOPs (but they don't claim that's the entirety of pretraining)
    • JacobAsmuth 2mo ago
      1e21 flops is hilariously wrong. for reference the llama 3 8B model (https://arxiv.org/pdf/2407.21783 https://arxiv.org/pdf/2407.21783) used 10 times that many flops. This model is 350x bigger in total params and 12x bigger in active params and was trained on 3x the data.
• NguyenDat377 2mo ago
  That's pretty impressive, I don't want to be the one undermine but I personally think the industry is shifting to rather designs that enable Models to work instead of chugging it all to data and compute ?
• Mohiuddin7 2mo ago
  That's the great achievement, but ahh, wanted to try it, but my RAM had other plans
• Mkadri 2mo ago
  [dead]
• htrp 2mo ago
  Why is everyone's pricing the same? licensing agreements with moonshot as an inference provider?
• logicallee 2mo ago
  What are the coding/agent harnesses (like Claude Code or ChatGPT Codex) that can be used with this?
• tesnorindian 2mo ago
  > Two findings in the Linux kernel illustrate the depth of these results. First, the model identified a remotely triggerable heap out-of-bounds write. The bug was introduced by an incomplete upstream fix and affects all subsequent releases, up to and including the latest upstream code Hope these Linux Kernel bugs due to some incomplete upstream fix identified by K3 patched? Moonshot did not provide much details on this. https://www.opensourceforu.com/2026/07/linux-maintainers-battle-record-ai-fuelled-cve-surge/?utm_source=krypteiasec&utm_medium=email&utm_campaign=daily-threat-brief-july-26-2026 https://www.opensourceforu.com/2026/07/linux-maintainers-bat... Looks like K3 is good with identifying zero days https://www.heise.de/en/news/Kimi-K3-Chinese-AI-finds-several-zero-day-vulnerabilities-in-redis-database-11377430.html https://www.heise.de/en/news/Kimi-K3-Chinese-AI-finds-severa...