Kimi K3 2.8T Open-Source Model Deep Dive: Mixed Attention KDA/Gated MLA, 896-Expert MoE, and AgentENV Sandbox Training

Kimi K3 2.8T Open-Source Model Deep Dive: Mixed Attention KDA/Gated MLA, 896-Expert MoE, and AgentENV Sandbox Training

1. Introduction: A Historic Milestone for Open-Source AI

On July 27, 2026, Moonshot AI officially open-sourced Kimi K3—the world’s first trillion-parameter-class open-source model at the 3-trillion scale. This is no ordinary model release; it represents a paradigm shift for the open-source AI community. Kimi K3 boasts 2.88 trillion total parameters, 104 billion active parameters per token, native support for 1-million-token context windows, and native multimodal capabilities. Across multiple benchmarks, K3 comprehensively surpasses Claude Opus 4.8, GLM-5.2, and GPT-5.5, approaching the current strongest closed-source models Claude Fable 5 and GPT-5.6 Sol.

Kimi K3 Core Specifications:

ParameterValue
Total Parameters≈2.88T
Active Parameters per Token≈104B
ArchitectureMixed Attention MoE (KDA + Gated MLA)
Decoder Layers93 (69 KDA + 24 MLA)
Routed Experts896, Top-16 activated per token
Shared Experts2
Hidden Dimension7168
Attention Heads96
Max Context1,000,000 Tokens
Vision EncoderMoonViT-V2 (trained from scratch)
LicenseOpen Weights

K3’s significance lies not only in its parameter scale but in three core architectural innovations: the KDA mixed linear attention mechanism, Attention Residuals, and the Stable LatentMoE ultra-sparse MoE framework. Together, these innovations improve K3’s overall scaling efficiency by approximately 2.5× compared to the previous generation K2. This article provides a deep technical analysis of each key technology in K3, complete with full code implementations.

2. Mixed Attention: KDA + Gated MLA

2.1 Why Mixed Attention?

Standard Transformer softmax attention has O(n²) time and space complexity, where n is the sequence length. At 1 million tokens, a standard attention mechanism’s KV Cache would consume terabytes of memory—completely infeasible. For a 96-head attention with hidden dimension 7168 and 1M tokens:

  • Standard attention KV Cache: 2 × 96 × 7168 × 1,000,000 × 2 bytes (FP16) ≈ 2.75 TB
  • Even with MLA (Multi-head Latent Attention) compressing to 3584: 2 × 96 × 3584 × 1,000,000 × 2 ≈ 1.37 TB

Kimi K3’s solution is a 3:1 mixed attention architecture: every 3 KDA (linear attention) layers are followed by 1 Gated MLA (global attention) layer, totaling 93 layers (69 KDA + 24 MLA).

2.2 KDA: Kimi Delta Attention Mathematics

KDA is a linear attention variant whose core idea is to replace the sequence-length-growing KV Cache with a fixed-size recurrent state. KDA simplifies attention computation into a recurrence:

Given input sequence x₁, x₂, ..., xₙ
KDA maintains a state matrix Sₜ ∈ ℝ^(d_k × d_v) at each time step
Sₜ = λ ⊙ S_{t-1} + (kₜ ⊗ vₜ)
Output oₜ = Sₜ · qₜ

Where:

  • kₜ, qₜ ∈ ℝ^(d_k) are the key and query at position t
  • vₜ ∈ ℝ^(d_v) is the value at position t
  • λ ∈ (0,1) is a learnable decay factor
  • ⊙ is element-wise multiplication, ⊗ is outer product

Python Implementation of KDA:

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Optional, Tuple

class KimiDeltaAttention(nn.Module):
    """
    Kimi Delta Attention (KDA) Implementation
    Uses fixed-size recurrent state instead of growing KV Cache
    """
    def __init__(self, hidden_dim: int, d_k: int, d_v: int, dropout: float = 0.0):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.d_k = d_k
        self.d_v = d_v
        
        self.q_proj = nn.Linear(hidden_dim, d_k, bias=False)
        self.k_proj = nn.Linear(hidden_dim, d_k, bias=False)
        self.v_proj = nn.Linear(hidden_dim, d_v, bias=False)
        self.out_proj = nn.Linear(d_v, hidden_dim, bias=False)
        
        self.log_lambda = nn.Parameter(torch.zeros(d_k))
        self.state_init = nn.Parameter(torch.zeros(d_k, d_v))
        self.dropout = nn.Dropout(dropout)
    
    def forward(
        self, 
        x: torch.Tensor, 
        state: Optional[torch.Tensor] = None,
        return_state: bool = False
    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
        batch_size, seq_len, _ = x.shape
        
        q = self.q_proj(x)
        k = self.k_proj(x)
        v = self.v_proj(x)
        
        decay = torch.sigmoid(self.log_lambda)
        
        if state is None:
            state = self.state_init.unsqueeze(0).expand(batch_size, -1, -1)
        
        outputs = []
        current_state = state
        
        for t in range(seq_len):
            k_t = k[:, t, :]
            v_t = v[:, t, :]
            q_t = q[:, t, :]
            
            outer_product = torch.bmm(
                k_t.unsqueeze(2),
                v_t.unsqueeze(1)
            )
            
            current_state = decay.unsqueeze(0).unsqueeze(2) * current_state + outer_product
            
            o_t = torch.bmm(
                current_state.transpose(1, 2),
                q_t.unsqueeze(2)
            ).squeeze(2)
            
            outputs.append(o_t)
        
        output = torch.stack(outputs, dim=1)
        output = self.out_proj(output)
        output = self.dropout(output)
        
        if return_state:
            return output, current_state
        return output, None

2.3 Attention Residuals

The second key innovation in Kimi K3 is Attention Residuals. In traditional Transformers, each residual block only connects adjacent layers. Attention Residuals allow the current layer to directly read feature representations from all previous layers:

class AttentionResidualBlock(nn.Module):
    """
    Attention Residual Block: enables cross-layer access to historical representations
    """
    def __init__(self, hidden_dim: int, num_layers: int):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.num_layers = num_layers
        
        self.cross_layer_weights = nn.ParameterList([
            nn.Parameter(torch.randn(i) * 0.01)
            for i in range(num_layers)
        ])
        
        self.fusion_proj = nn.Linear(hidden_dim, hidden_dim, bias=False)
        
    def forward(self, x: torch.Tensor, historical_states: list, layer_idx: int) -> torch.Tensor:
        if layer_idx == 0 or not historical_states:
            return x
        
        weights = self.cross_layer_weights[layer_idx]
        weights = F.softmax(weights, dim=0)
        
        fused = x * weights[-1]
        for i, h_state in enumerate(historical_states):
            fused = fused + h_state * weights[i]
        
        return self.fusion_proj(fused)

3. Stable LatentMoE: 896-Expert Ultra-Sparse Routing

3.1 MoE Architecture Overview

Kimi K3’s MoE layer is the largest in any open-source model—896 routed experts with 16 activated per token, plus 2 shared experts. Total parameters reach 2.88T, but only approximately 104B parameters are activated per token.

3.2 Quantile Balancing

Traditional MoE training uses an auxiliary load-balancing loss. Kimi K3’s Quantile Balancing takes a fundamentally different approach—dynamic bias adjustment without auxiliary losses:

class QuantileBalancingRouter(nn.Module):
    """
    Quantile Balancing Router
    Dynamically adjusts expert biases instead of using auxiliary loss
    """
    def __init__(self, hidden_dim: int, num_experts: int, top_k: int, 
                 balancing_factor: float = 0.01):
        super().__init__()
        self.router = nn.Linear(hidden_dim, num_experts, bias=False)
        self.expert_bias = nn.Parameter(torch.zeros(num_experts))
        self.register_buffer('expert_load', torch.zeros(num_experts))
        self.register_buffer('step_count', torch.zeros(1))
        self.top_k = top_k
        self.balancing_factor = balancing_factor
        self.window_size = 1000
    
    def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
        logits = self.router(x)
        adjusted_logits = logits + self.expert_bias.unsqueeze(0).unsqueeze(0)
        expert_weights, expert_indices = torch.topk(
            F.softmax(adjusted_logits, dim=-1), self.top_k, dim=-1
        )
        
        if self.training:
            with torch.no_grad():
                load = torch.zeros(self.num_experts, device=x.device)
                indices_flat = expert_indices.view(-1)
                load.scatter_add_(0, indices_flat, torch.ones_like(indices_flat, dtype=torch.float))
                load = load / (x.shape[0] * x.shape[1] * self.top_k)
                
                decay = 0.99
                self.expert_load = decay * self.expert_load + (1 - decay) * load
                self.step_count += 1
                
                if self.step_count % self.window_size == 0:
                    self._update_quantile_bias()
        
        return expert_weights, expert_indices
    
    def _update_quantile_bias(self):
        with torch.no_grad():
            target_load = 1.0 / self.num_experts
            load_diff = target_load - self.expert_load
            self.expert_bias += self.balancing_factor * load_diff

3.3 SiTU-GLU Activation Function

Kimi K3 uses SiTU-GLU instead of the traditional SwiGLU in expert FFN layers. SiTU-GLU solves the activation explosion problem in ultra-sparse scenarios:

class SiTU_GLU(nn.Module):
    """
    SiTU-GLU: sigmoid(x) * x * tanh(gate) * up
    More stable than SwiGLU for ultra-sparse MoE
    """
    def __init__(self, hidden_dim: int):
        super().__init__()
        self.W_gate = nn.Linear(hidden_dim, hidden_dim, bias=False)
        self.W_up = nn.Linear(hidden_dim, hidden_dim, bias=False)
    
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        gate = self.W_gate(x)
        up = self.W_up(x)
        gate_activated = torch.sigmoid(gate) * up
        tanh_gate = torch.tanh(gate)
        return gate_activated * tanh_gate

4. AgentENV: MicroVM-Based Sandbox Training System

4.1 Why AgentENV?

Kimi K3’s reinforcement learning training is fully customized for “1-million-token long-horizon agents.” Traditional training data consists of static Q&A pairs and documents, but K3’s training environment is a real simulator—a microVM-based sandbox system called AgentENV.

class MicroVMSandbox:
    """
    MicroVM Sandbox Environment
    Simulates Kimi K3's AgentENV training environment
    """
    def __init__(self, sandbox_dir: str = "./sandbox"):
        self.sandbox_dir = sandbox_dir
        os.makedirs(sandbox_dir, exist_ok=True)
        
        self.env_state = {
            "filesystem": {},
            "processes": [],
            "tools": {
                "editor": self._simulate_editor,
                "terminal": self._simulate_terminal,
                "browser": self._simulate_browser,
                "git": self._simulate_git,
                "slack": self._simulate_slack,
                "notion": self._simulate_notion,
                "gmail": self._simulate_gmail,
            }
        }
    
    def execute_step(self, agent_action: Dict[str, Any]) -> Dict[str, Any]:
        tool = agent_action.get("tool", "")
        params = agent_action.get("params", {})
        
        if tool in self.env_state["tools"]:
            return self.env_state["tools"][tool](**params)
        return {"status": "error", "message": f"Unknown tool: {tool}"}

5. Infrastructure: MoonEP and FlashKDA

5.1 MoonEP: Zero-Waste Expert Parallelism

Training a 2.88T parameter MoE model requires expert parallelism. MoonEP uses online integer linear programming and zero-copy communication to achieve fully dynamic and perfectly balanced compute allocation:

from scipy.optimize import linear_sum_assignment
import numpy as np

class MoonEPAllocator:
    """Zero-waste expert parallelism allocator"""
    def __init__(self, num_gpus: int, num_experts: int):
        self.num_gpus = num_gpus
        self.num_experts = num_experts
        self.expert_load_history = np.zeros((num_experts,))
    
    def optimize_allocation(self, expert_loads: np.ndarray) -> np.ndarray:
        alpha = 0.9
        self.expert_load_history = (
            alpha * self.expert_load_history + 
            (1 - alpha) * expert_loads
        )
        
        cost_matrix = np.zeros((self.num_experts, self.num_gpus))
        for e in range(self.num_experts):
            for g in range(self.num_gpus):
                cost_matrix[e, g] = gpu_loads[g] + self.expert_load_history[e]
        
        row_ind, col_ind = linear_sum_assignment(cost_matrix)
        return col_ind[:self.num_experts] % self.num_gpus

6. Benchmark Results and Industry Impact

6.1 Comprehensive Benchmark Results

BenchmarkKimi K3Claude Opus 4.8GPT-5.5Claude Fable 5GPT-5.6 Sol
MMLU-Pro92.4%89.1%90.2%93.8%94.1%
HumanEval93.7%87.3%89.5%94.2%94.5%
SWE-Bench68.2%52.1%55.6%71.3%72.8%
MATH-50097.1%94.8%95.3%97.8%98.1%
Agent-Bench81.3%70.2%73.1%83.5%84.2%

6.2 Cost Advantage

On Artificial Analysis, Kimi K3 achieves first-tier performance at half the per-task cost of GPT-5.6 Sol, and nearly an order of magnitude cheaper than Claude Fable 5. This is enabled by quantization-aware training (MXFP4 weights/MXFP8 activations), speculative decoding, and aggressive system-level optimization.

7. Conclusion

Kimi K3’s open-source release is one of the most important milestones in AI during 2026. It proves that:

  1. Open-source models can rival the strongest closed-source models
  2. Mixed attention architecture (KDA + MLA) is a viable path for long-context inference
  3. Ultra-sparse MoE (896 experts, 1.8% activation rate) is engineering-feasible at scale
  4. AgentENV sandbox training is effective for developing long-horizon agent capabilities

With K3’s weights fully open, we can expect a wave of community-optimized versions, local deployment solutions, and vertical applications. K3 represents a new era for open-source AI.

References

  1. Moonshot AI, “Kimi K3 Tech Blog: Open Frontier Intelligence”, 2026
  2. AMD, “Day 0 Kimi-K3 Inference Deployment with ATOM on AMD Instinct MI355X GPUs”, 2026
  3. LMSys, “Day-0 Support for Kimi K3 in SGLang and Miles”, 2026
  4. Moonshot AI, “Kimi K3 Technical Report”, 2026