Moonshot introduces Attention Residuals with 1.25x compute gains on Kimi Linear

• Moonshot says its new launch thread swaps fixed residual accumulation for learned, input-dependent lookbacks over earlier layers, turning residual paths into a selective retrieval mechanism instead of a uniform sum.
• In Moonshot's scaling-law post, Attention Residuals show a consistent 1.25x compute advantage across model sizes, while the main announcement says inference overhead stays below 2%.
• The company says the method was validated on Kimi Linear, a 48B-parameter sparse architecture with 3B activated parameters, where the announcement reports consistent downstream gains.
• Moonshot's training-dynamics post claims the approach also curbs hidden-state magnitude growth and produces a more uniform gradient distribution across depth, pointing to training-stability benefits beyond raw efficiency.

Moonshot is pitching Attention Residuals as a drop-in replacement for standard residual stacks. Instead of every layer receiving the same accumulated history, the launch thread says each layer can "selectively retrieve past representations" through learned attention over prior layers.

That is the main architectural shift: residual connections stop being fixed depth-wise recurrence and become an attention problem over model depth. Moonshot says Block AttnRes makes that usable at scale by grouping layers into compressed blocks, and the

from a supporting explainer shows the intended tradeoff clearly: full lookback behavior with a blockwise memory reduction. The company links the full method description in its paper.

A practitioner summary from Cedric Chee's note captures the practical pitch in plain language: old stacks "keep piling layers on top," while Attention Residuals let a layer "look back" and pull the most useful earlier state.

Moonshot's benchmark claim is narrow but concrete. Its scaling-law post says experiments show a "consistent 1.25× compute advantage" across model sizes, and the main announcement adds that the latency cost is negligible at "<2%" during inference.

The training-side argument is that learned lookbacks improve optimization, not just efficiency. According to the training-dynamics post, AttnRes mitigates hidden-state growth and yields a "more uniform gradient distribution across depth." The same thread says results were validated on Kimi Linear, with 48B total parameters and 3B activated, and Moonshot reports "consistent downstream performance gains" there launch thread.

What Moonshot has not shown in the thread is a broad reproduction set outside Kimi Linear or a detailed public accounting of memory costs under different stack depths and block settings. The strongest current evidence remains Moonshot's own paper and thread-level summaries, including a supporting recap from AlphaSignalAI's summary that repeats the headline numbers but does not add independent results.