Kimi Delta Attention (recurrent state)

Attention Residuals (LSTM residual rotated into attention space)

Both are recurrence-flavored ideas injected into the transformer: one replaces full attention with a gated recurrent memory, the other generalizes residual connections along the same recurrence/attention duality.

1 source videos