Moonshot AI Unveils “Attention Residuals,” a Bid to Make Kimi’s Next Models Train Deeper and Reason Better

Moonshot AI has published a paper detailing “Attention Residuals,” a redesign of the residual connections inside Transformer-style networks that aims to improve deep-model training stability and boost reasoning performance without increasing compute budgets—an architectural bet that could reshape how Kimi’s next-generation models scale.

The work, released on March 16, 2026 with a public PDF on GitHub, quickly drew attention from prominent industry figures including former OpenAI researcher Jerry Tworek and OpenAI co-founder Andrej Karpathy. Moonshot said an X post about the paper approached 600,000 reads, signaling strong developer and investor interest in efficiency-oriented upgrades as AI capex tightens globally.

Replacing fixed sums reshapes how models use depth

Transformers typically use residual connections that add each layer’s output to the running hidden state with equal weight. That “uniform accumulation” stabilizes optimization but can dilute early-layer information as depth increases, limiting expressiveness and making very deep networks harder to exploit.

Moonshot’s “Attention Residuals” (AttnRes) replaces that fixed add with an attention mechanism over prior layer representations. Each layer acts as a query that retrieves and reweights useful signals from historical layers via softmax weights, turning depth into a learnable routing problem rather than a fixed pipeline. For investors tracking model roadmaps, the key claim is that better layer-wise information access can raise the ceiling for reasoning tasks—without requiring proportional increases in parameter count or training tokens.

Engineering block attention targets compute and communication bottlenecks

Full attention over all historical layers can raise overhead as models deepen. Moonshot proposed “Block AttnRes,” which groups layers into blocks: layers compute normally within a block, while cross-block interactions use compressed representations to control cost.

The paper also focuses on systems constraints—pipeline-parallel training communication and inference memory pressure—highlighting cross-stage caching to reduce redundant transfers. Moonshot reported end-to-end training time increased by less than 4% versus a standard Transformer setup, while inference added under 2% latency overhead, using a two-stage computation strategy plus sequence-dimension sharding of cached block representations across tensor-parallel devices.

Benchmarks show a compute-efficiency trade that matters for model economics

Moonshot integrated AttnRes into its Kimi Linear architecture—48B total parameters with 3B active parameters—and pretrained on 1.4 trillion tokens. The company said AttnRes mitigated “pre-norm dilution,” producing more uniform activation and gradient magnitudes across depth, a stability lever that can reduce the need for conservative hyperparameters or costly retraining runs.

On evaluation, Moonshot reported consistent gains across benchmarks, with larger improvements on tasks requiring multi-step reasoning such as math, science QA, and code generation. In GPQA-Diamond, a multi-step reasoning benchmark, the AttnRes model’s score improved by more than 20% in the company’s experiments.

On efficiency, Block AttnRes achieved a validation loss of 1.692 versus a 1.714 baseline, which the authors framed as roughly a 1.25x compute-efficiency advantage—i.e., matching the loss of a baseline trained with about 1.25 times the compute. That metric speaks directly to model economics: if the result generalizes, it could shift the cost curve for training reasoning-focused models in China’s competitive frontier-model market.

The architecture signals a new battleground beyond parameters and tokens

The paper positions attention not only as a sequence mechanism but as a depth mechanism—effectively turning layer aggregation into an adaptive retrieval step. For the supply chain, that emphasis on “more capability per unit of compute” aligns with a 2026 industry pattern: developers and cloud vendors are optimizing for throughput, memory bandwidth, and training stability as much as for raw parameter scale.

Moonshot credited a larger team effort, listing researchers Guangyu Chen, Yu Zhang (also first author on Kimi-Linear), and Jianlin Su among the top contributors, and said co-founders Yang Zhilin, Wu Yuxin and Zhou Xinyu led the work.

The paper is available at: 

https://github.com/MoonshotAI/Attention-Residuals/blob/master/Attention_Residuals.pdf

Related Coverage:

Kimi Surges Ahead in Global AI Race as Overseas Revenue Overtakes Domestic Market

Kimi's Overseas Revenue Surpasses Domestic Sales as AI Startup Targets Global Productivity Market