Moonshot AI CEO Challenges Deep Learning Orthodoxy in GTC Debut, Unveiling ‘Efficiency-First’ Architecture
Yang Zhilin, the founder of Moonshot AI, used his solitary platform as the only Chinese large model entrepreneur at NVIDIA’s GTC 2026 to propose a radical overhaul of deep learning’s foundational blocks. His thesis is stark: the decade-old standards governing AI scaling—from optimizers to residual connections—have become bottlenecks that must be discarded.
In a technical deep-dive titled "How We Scaled Kimi K2.5," Yang outlined a roadmap that prioritizes architectural efficiency over brute-force resource accumulation. The presentation marked the first systematic disclosure of the technology stack behind Kimi, Moonshot’s flagship large language model (LLM), which now integrates a triad of innovations: doubled token efficiency, linear attention for massive context windows, and autonomous agent swarms.
The unveiling comes as the global AI sector shifts its focus from parameter size to inference economics. Yang’s strategy suggests that for Chinese developers—often navigating constrained compute environments—the path to AGI lies not just in more GPUs, but in smarter mathematics.
Shattering the Optimization Ceiling
For over a decade, the Adam optimizer has been the unquestioned default for training neural networks. Yang argues this consensus is now a liability. Moonshot AI introduced MuonClip, a proprietary optimization method that effectively doubles the efficiency of token training compared to the industry-standard AdamW. Under equivalent compute budgets, MuonClip converts training data into model capability at twice the rate.
Scaling this new optimizer to trillion-parameter models initially caused severe instability, with logits exploding beyond manageable thresholds. The team countered this by combining Newton-Schulz iterations with a QK-Clip mechanism, strictly constraining max logits under 100 to ensure stability without degrading loss performance. To handle the infrastructure demands of 2026, the company also deployed "Distributed Muon," a system that fragments optimizer states across data-parallel groups to maximize memory efficiency in large-scale GPU clusters.
Decoupling Context from Latency
While 1 million-token context windows are now a market standard, the inference cost remains prohibitive for enterprise application. Moonshot’s solution, Kimi Linear, dismantles the traditional "Full Attention" mechanism that slows down processing as data grows.
By adopting a hybrid architecture that mixes Kimi Delta Attention (KDA) with global attention in a 3:1 ratio, the model achieves a 5x to 6x increase in decoding speed within the 128k to 1M context range. This architecture transforms long-context capabilities from a theoretical feature into a commercially viable tool, maintaining high fidelity in recall and reasoning tasks while drastically cutting memory overhead.
Rewriting the Neural Backbone
Perhaps the most theoretically aggressive move is Moonshot’s redesign of the residual connection—a structural staple of deep learning since the ResNet era of the mid-2010s. Yang contends that traditional additive connections dilute information in deep networks. The proposed replacement, Attention Residuals (AttnRes), utilizes Softmax-based dynamic aggregation, allowing layers to actively "select" information from preceding layers rather than passively accumulating it.
This approach, recently validated by industry heavyweights including Andrej Karpathy and Elon Musk, treats attention as an expansion of the information channel. By open-sourcing this architecture, Moonshot is positioning itself as a contributor to fundamental AI theory, moving beyond the role of a mere application developer.
Orchestrating Autonomous Swarms
Moving up the stack, Yang detailed the transition from single-agent chatbots to Agent Swarms. Kimi K2.5 utilizes a central "Orchestrator" to spawn and manage specialized sub-agents—such as "Physics Researchers" or "Fact Checkers"—to execute complex workflows in parallel.
To prevent "serial collapse"—a common failure mode where multi-agent systems devolve into sequential processing—Moonshot implemented a tiered reinforcement learning reward system. By incentivizing instantiation, task completion, and outcome quality separately, the system enforces genuine parallel execution. This structure allows Kimi to handle tasks requiring "Actions at Scale," significantly reducing execution time for complex queries compared to linear processing.
The presentation also highlighted a 1.7% to 2.2% gain in pure text benchmarks (MMLU-Pro, GPQA-Diamond) derived solely from visual reinforcement learning, confirming that multi-modal training is now yielding cognitive gains across the board.
Yang concluded by noting that the "Scaling Ladder" has fundamentally changed. The tools of the past decade—Adam (11 years old), Attention (8 years old), and Residuals (10 years old)—are being systematically replaced by Moonshot’s new stack, signaling a new phase of competition defined by architectural ingenuity.
Related Coverage:
Kimi's Overseas Revenue Surpasses Domestic Sales as AI Startup Targets Global Productivity Market