Moonshot AI fully revealed Kimi K3’s technical guts on July 27, releasing not just the model weights but comprehensive architecture documentation that exposes how they engineered a frontier-scale competitor to Western models at lower computational cost.
The 1.56 TB repository on Hugging Face contains what engineers will spend weeks dissecting: a 2.8 trillion-parameter multimodal mixture-of-experts machine with a staggering one-million-token context window.
The numbers paint a picture of aggressive architectural choices. Kimi K3 contains 896 distinct experts, but uses only 16 per token. That sparse routing is deliberate. It reduces computational load while preserving the benefit of having massive parameter pools available.
Only 104 billion parameters activate on any given inference call, meaning the model runs far lighter than its raw parameter count suggests. The context window of 1,048,576 tokens puts it in elite company alongside similarly extended-context models, enabling long-horizon reasoning across enormous documents or code repositories.
The model uses 93 layers, a 160K-token vocabulary, and native vision support powered by MoonViT-V2, a 401-million-parameter vision encoder Moonshot AI built for the task. But the real innovation sits in the attention mechanisms. Kimi K3 combines 69 Kimi Delta Attention layers with 24 Gated Mixture-of-Latents layers.
This hybrid approach balances long-range dependency tracking with computational efficiency. Moonshot AI also applied quantization-aware training, using low-precision MXFP4 weights and MXFP8 activations to compress without destroying capability.
Moonshot AI claims the architecture delivers 2.5 times better scaling efficiency than its predecessor Kimi K2. That efficiency gain matters enormously because it means less compute produces more capability per dollar or watt. The company attributes the leap to Kimi Delta Attention, Attention Residuals, a sparser MoE configuration and revised training recipes, but has not yet opened those recipes to independent scrutiny.
What researchers now face is a practical constraint beyond the architecture itself. The full repository runs 1.56 terabytes, making local experimentation an infrastructure problem. Running Kimi K3 requires serious GPU memory, storage bandwidth, and accelerator interconnect capacity.
Teams that want to finetune, quantize further, or add custom tools need to budget for those requirements. The weights are accessible. Making them practical requires solving deployment challenges that most researchers face for the first time.
The release also included infrastructure components for attention kernels, Mixture-of-Experts communication, and large-scale agent deployment tools. These are less glamorous than the base model but matter deeply for teams building production systems around Kimi K3.
