VideoDeltaNet on Minimax H3

Video DeltaNet: Near-Lossless Hybrid Attention for Video Models

Attention is the dominant cost of video generation^[1],[2]. On a frontier omni-model like MiniMax H3^[3], softmax attention over long token sequences accounts for more than 85% of total runtime. To overcome this, frontier LLMs replace softmax, whose cost is quadratic in sequence length, with linear attention^[4],[5]. While significantly faster, this comes at the cost of generation quality. For instance, linear attention struggles to maintain global features like subject identity, scene layout, and temporal dependencies over long sequences^{[8],[9],[10],[11]}.

We propose a hybrid attention architecture^[6],[7], Video DeltaNet (VDN), that approaches the speed of linear attention while maintaining near-lossless quality. We have applied this architecture improvement to the MiniMax H3 model to train VDN-H3. In only 11.23 seconds our model can produce a 14.4 second video clip that is visually nearly indistinguishable from the original H3's output. In effect, it can stream H3-quality generated content continuously on 8 NVIDIA B200.

Here is what VDN-H3 produces. Each clip is generated in 11.23 seconds, less time than it takes to watch. This speedup comes with near-lossless visual quality compared to the original MiniMax H3 and demonstrates higher quality and better instruction-following ability than MiniMax FastH3.

Local Softmax and Long-Range Linear Memory

Video DeltaNet splits the video–video attention into two complementary branches: a sliding-window softmax branch for local frame pairs and a linear branch for long-range context.

The bidirectional sliding window computes exact attention between nearby frames, preserving fine-grained local detail and promoting short-term temporal stability. Specifically, sets of five consecutive latent frames are grouped into chunks, following the original H3 architecture. Each chunk attends to itself, the previous chunk, and the following chunk.

We further augment the softmax branch with 4-way boundary anchors to maintain global consistency.^[13] Every frame attends to all tokens in the first and last frames—the boundary frames—and both boundary frames attend to every other frame. This design adds only 3.57% attention density, yet substantially improves long-range stability.

We process distant context using bidirectional linear attention. For Frame \(t\), the forward state \(S_t^{\rightarrow}\) summarizes frames before the local Softmax window, while the reverse state \(S_t^{\leftarrow}\) summarizes those after it. Their temporal regions are disjoint, and do not include the frames covered by sliding window softmax and boundary frames. This allows the query \(q_t\) to read states from both sides and add their outputs without double-counting any video frames.

Here is how the mechanism works:

Both attention branches are conditioned on the text prompt. In the softmax branch, text tokens remain globally visible to every frame. In the linear branch, the text prompt is incorporated once at initialization: we write the text tokens into both the forward and reverse states via the delta rule, each scaled by one half, so that summing the two readouts counts the text contribution exactly once.

To make the attention pattern concrete, we visualize the attention map for a representative workload with 1 text token, 16 video frames, and 1 audio token. For legibility, the VAE groups every 2 consecutive frames into a chunk here, rather than 5 as in the full model. Softmax attention handles the text and audio tokens, the local frame pairs within the sliding window, and the boundary anchors. Linear attention handles the remaining long-range context along with the text prompt written into its states at initialization as described above.

Gating the Softmax and Linear Branch Outputs

The outputs of the two branches cannot simply be summed as their scales differ. Instead, we apply a content-dependent sigmoid gate to each branch, project each through its own output projection, and add both results into the residual stream.

The two gates serve different purposes. In the softmax branch, the normalizer sums over only the local window, so the branch redistributes full attention mass across nearby frames even though most key positions have been removed. This potentially leads to an over-weighting of the local context. The softmax gate learns to calibrate this, controlling how much of the local output to preserve.

For the linear branch, the output scale may not naturally match the softmax branch. Following Gated DeltaNet and Kimi Delta Attention,^[4][5] we apply RMSNorm followed by an element-wise sigmoid gate to its output. The linear branch also receives its own output projection since the two branches may require different output spaces. We illustrate the full mechanism below.

Video Delta Attention: A Frame-Wise Solution

With the overall hybrid architecture in place, we can now look more closely at how to design our linear attention method: Video Delta Attention. The major bottleneck is how to turn the token-wise delta rule into a frame-wise delta-rule, making the linear attention video-native. Before explaining our method, we first review the standard linear attention update that is widely used in LLMs.

Let \(S_{t-1}\) be the linear state entering the current step. KDA^[5] first applies the channel-wise decay gate \(\alpha\) to form the decayed state \(\bar S=S_{t-1}\operatorname{Diag}(\alpha)\). The classical delta rule then processes one token at a time: it reads the value associated with key \(k\), compares it with target \(v\), and writes the residual along the same direction. The write strength \(\beta\), acting as the update's adaptive learning rate, controls how strongly the old association is erased and the new target written.

For video, however, a strictly token-by-token update is not the most natural unit of computation. Video generation is organized around latent frames, so the spatial tokens within one frame should update the linear state together. SANA-WM therefore parallelizes the delta rule across all \(U\) spatial tokens in one frame.^[11] Let \(K\) collect the unit-normalized keys before frame-size scaling. SANA-WM applies that additional scaling to the keys themselves to address stability issues, using \(\widehat K=K/\sqrt U\), whose \(u\)-th row is \(\hat k_u^\top=k_u^\top/\sqrt U\). To keep the unscaled frame statistics available for comparison, define

From independent token updates to one frame-wise solve

In the SANA-WM update above, every token reads from the same frozen state \(\bar S\), and each write is computed without seeing the other proposed writes. Consequently, one token cannot adjust its correction when another has a similar key but a different target. Although the batched update is applied once per frame, its tokens do not interact or negotiate; correlated and repeated evidence simply accumulates.

Our motivation is to make both reading and writing genuinely frame-wise. All tokens in the frame should determine one new state together, so correlated patches can resolve their shared directions inside the update itself. At the same time, the new state should remain close to the memory already accumulated from earlier frames.

The Delta Rule already suggests how to do this. Its one-token correction is exactly a gradient-descent step with learning rate \(\beta\) on the prediction error:

We turn the desired reads \(Sk_u\approx v_u\) into one joint objective and add a proximity term that prevents the new state from moving too far from \(\bar S\):

Using the same \(A=K^\top\operatorname{Diag}(\beta)K\) and \(B=V^\top\operatorname{Diag}(\beta)K\) as above, differentiating with respect to \(S\) gives

This formulation has two useful consequences, which we explain below.

1. 01
   Stability of the Inherited State

   For \(A\succeq0\), the transition \((I+A)^{-1}\) has operator norm at most one, so a single frame update cannot amplify the inherited-state component. See the stability derivation below.

2. 02
   Correlation-Aware Frame Updates

   The joint solve preconditions the batch residual with \((I+A)^{-1}\), formed from the weighted key Gram matrix \(A\), damping redundant directions without globally shrinking independent ones. See how repeated and independent key directions produce different frame updates below.

Stability of the Inherited State

During generation, frame-wise transitions compose over time. If one update amplifies a stored direction, later frames with similar keys can amplify it repeatedly and corrupt the accumulated context. We therefore require the transition acting on the inherited state to be non-expansive, with operator norm at most one.

The three rules differ only in which transition they apply, so we check this single requirement three times. Each check explains one scaling decision encountered above: why GDN and KDA unit-scale their keys,^[4][5] why SANA-WM must add a frame-size factor on top of that, and why VDA needs no size-dependent scaling at all.

Why GDN and KDA unit-scale their keys

Start with one token. The value write \(\beta_u v_u k_u^\top\) is purely additive, so the inherited state is touched only by the rank-one erase \(M_u\), and the question is simply whether \(\lVert M_u\rVert_2\le1\). Applying the erase to the key itself reveals its only nontrivial eigendirection:

This reveals that \(k_u\) is an eigenvector of \(M_u\), with eigenvalue \(\mu_u\triangleq1-\beta_u\lVert k_u\rVert_2^2\). For any vector \(z\) orthogonal to \(k_u\), we have \(M_u z=z\), so every orthogonal direction has eigenvalue one. Consequently, \(\lVert M_u\rVert_2=\max\{1,|\mu_u|\}\), and the update is non-expansive exactly when \(|\mu_u|\le1\). Since \(0\le\beta_u\le1\), unit scaling \(\lVert k_u\rVert_2=1\) ensures \(\mu_u=1-\beta_u\in[0,1]\). Therefore \(\lVert M_u\rVert_2=1\), and the inherited state cannot expand. Without normalization, the key norm can instead make \(|\mu_u|>1\), violating this guarantee.

Why SANA-WM must also scale by frame size

In one video frame, SANA-WM processes \(U\) spatial tokens in parallel. Each token contributes a rank-one erase, and their sum is the weighted key Gram matrix \(A\). Setting aside SANA-WM’s frame-size scaling for a moment, the inherited state therefore passes through \(I-A\). To determine whether this transition is non-expansive, we first examine the spectrum of \(A\):

Since \(\beta_u\ge0\), the middle line shows that \(A\succeq0\). By the spectral theorem, \(A=Q\Lambda Q^\top\) has orthonormal eigenvectors and nonnegative eigenvalues \(\lambda_i(A)\); therefore \(I-A\) scales each eigenvector by \(1-\lambda_i(A)\). Unit scaling bounds each token’s contribution, but not these frame-wide eigenvalues. When many keys align, their contributions accumulate in one eigendirection and make the corresponding \(\lambda_i(A)\) large:

For unit keys, \(\lambda_{\max}(A)\le\operatorname{tr}(A)=\sum_u\beta_u\), so the largest eigenvalue can grow with the number of spatial tokens. If aligned keys make \(\lambda_i(A)>2\), then \(|1-\lambda_i(A)|>1\), and \(I-A\) becomes expansive. Unit-scaling each key is therefore insufficient for a frame-wide batch update.

SANA-WM prevents this by scaling each unit key by \(1/\sqrt U\): \(\hat k_u=k_u/\sqrt U\). This changes \(A\) to \(A/U\), and since \(\lambda_{\max}(A)\le\sum_u\beta_u\le U\), we have \(\lambda_{\max}(A/U)\le1\). Therefore \(I-A/U\) has eigenvalues in \([0,1]\) and is non-expansive.

Why VDA needs no frame-size scaling

VDA never forms the additive transition, so it never needs the worst-case rescale. Its joint frame-wise solve yields \((I+A)^{-1}\) directly from the unscaled weighted key Gram matrix, and because \(I+A\) has the same eigenvectors as \(A\), the new transition can be read in exactly the same eigenbasis:

For every \(\lambda_i\ge0\), the multiplier \(1/(1+\lambda_i)\) lies in \((0,1]\): a direction with little frame evidence is nearly preserved, and a heavily crowded one is strongly suppressed. Where the additive multiplier \(1-\lambda_i\) keeps falling and crosses \(-1\) as crowding grows, \(1/(1+\lambda_i)\) only flattens toward zero and never crosses it. VDA is therefore natively non-expansive for any number or alignment of tokens: the required contraction comes from the observed Gram spectrum rather than a global frame-size scale.

The decay gate preserves the same property. Since \(\bar S=S_{t-1}\operatorname{Diag}(\alpha_t)\) with \(0\le\alpha_t\le1\), the inherited component is multiplied by \(\operatorname{Diag}(\alpha_t)(I+A_t)^{-1}\), whose operator norm is at most one by submultiplicativity.

Correlation-Aware Frame Updates

VDA’s correlation-aware solve has another benefit: it uses unit-normalized keys directly, without attenuating each value–key write by \(1/\sqrt U\). SANA-WM chooses its frame scale from the token count \(U\) alone, so it cannot distinguish a frame that repeats one memory direction from one that spans \(U\) independent directions. VDA’s \((I+A)^{-1}\) adapts to the observed correlations instead:

The first-order term \(I-A\) treats the proposed corrections additively. The higher-order powers \(A^2,A^3,\ldots\) account for their interactions through shared key directions. VDA evaluates the inverse directly, so the solve remains well-defined for every \(A\succeq0\), including outside the series’ convergence regime. Two extreme key geometries make this difference concrete.

Assume unit keys and a shared write strength \(\beta\). In the first geometry every token addresses the same direction, \(k_u=k\), and the targets collapse to their mean \(\bar v=U^{-1}\sum_u v_u\); in the second the keys are orthonormal, with no overlap at all. Because SANA-WM stores associations for its scaled keys \(\hat k_u=k_u/\sqrt U\) while VDA uses \(k_u\), we compare the prediction each state produces on its own key. Substituting each geometry’s \(A\) and \(B\) into the two update rules gives

These two cases expose the limitation of a single \(U\)-based scale. With repeated keys, SANA-WM ignores how many tokens support the same association; with orthogonal keys, it weakens each independent write by \(1/U\). VDA instead adapts each eigendirection through \((I+A)^{-1}\), preserving repeated evidence without penalizing unrelated directions.

Adapting the Pretrained H3 backbone

In this section, we explain how to adapt the existing H3 model to our hybrid architecture. Introducing a randomly initialized linear branch into the hybrid architecture can easily perturb the already well-calibrated softmax path, since all linear state, gate, and output projection begin with random initialization. Therefore, naively end-to-end training can easily degrade generation quality.

We therefore propose a three-stage training pipeline.

In Stage A1, we calibrate one layer at a time. Following the per-layer calibration strategy introduced by Taylor-Calibrate,^[14] each randomly initialized linear branch needs to be trained independently before full model training. This offers a better initialization for the next stage. In Stage A2, we perform end-to-end branch adaptation. We insert all hybrid blocks in the backbone and train them jointly. In both Stage A1 and A2, we freeze the softmax branch, QKVO projection, and FFN to isolate the effect of the linear branch. In Stage B, we perform end-to-end co-adaptation. We add LoRA adapters to the QKV projection and O projection, then jointly train these adapters and the linear branch. All remaining parameters remain frozen.

Crucially, the softmax gate remains frozen throughout Stage A1 and A2. Before the linear branch is correctly trained, a trainable softmax gate gives the optimizer an easier but undesirable way to reduce the loss. It might over-suppress the softmax branch instead of learning the new linear branch. Therefore, we only train it in Stage B, as we believe the linear branch can provide meaningful signal at that time.

During training, we use separate learning rate groups for different parameter families. Following Chimera’s principle,^[15] we use a higher learning rate for those linear attention parameters that have a smaller fan-in size, and use a lower learning rate for others. LoRA adapters also has a higher learning rate compared to standard fine-tuning recipes.

Having built the training pipeline to convert the base H3 model into a hybrid architecture, we next measure how this translates into practical inference speedup.

On a single B200, the 50-step Dense Attention DiT takes about 13.95 minutes to generate a 15 second, 768p video. VDN-H3 reduces this to about 5.35 minutes with optimized kernels and FP8 linear layers.

To see where the gain comes from, we first isolate a single H3 block. Dense H3 takes 332.5ms on a single B200. By applying Video Delta Attention and Sliding Window Softmax, this number is reduced to 192.1ms. With optimized inference kernels and FP8 linear attention, we achieve 125.3ms inference speed per layer, corresponding to 2.65x speedup.

Parallelization & Few-Step Generation

Our next goal is to use parallelization and few step distillation techniques to further accelerate VDN-H3. Standard Ulysses parallelism shards the sequence across 8 B200s, so each GPU processes only a fraction of the sequence or attention heads. This reduces latency from 6.46s / NFE to 1.62s / NFE^†, a 3.99x per-step speedup.

However, this is far from optimal. We identify a further speedup opportunity by placing the softmax branch and VDA branch on separate GPUs, rather than letting every GPU execute both branches simultaneously.

The tensor remains sharded along the sequence dimension during QKV projection. After that, an uneven all-to-all gathers the heads in a way that 5 GPUs receive the Softmax branch, and 3 GPUs receive the VDA branch. This split is chosen based on profiling results.

After both branches finish, a reverse all-to-all sends their outputs back to the original sequence-sharded layout. This design reduces inference latency to 1.405 s / NFE, reducing the latency by 13.3% beyond the standard Ulysses algorithm.

Few-step distillation provides further gains. Starting from LarryVRH’s community MiniMax-H3 turbo LoRA,^[16] we continue training it with DMD2^[17] to obtain our own VDN-H3 LoRA for 8-step distillation. We deliberately avoid an aggressive four-step setting, prioritizing generation quality over latency. An 8-step generation offers better quality-speed trade-off.

Overall, the DiT denoising process only takes 11.23 seconds to generate a 14.3s video, leading to a 74.5× speedup over the dense MiniMax H3 single-GPU baseline. Compared against the dense MiniMax H3 8-GPU baseline, VDN-H3 still demonstrates a 10.7× speedup.

Results

Below, we show additional qualitative comparisons. VDN-H3 achieves generation quality nearly identical to the dense H3 baseline while running significantly faster.

References

1. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. Xi et al., 2025.
2. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation. Yang et al., 2025.
3. MiniMax H3: Breaking the Boundaries of Tasks and Modalities. MiniMax, 2026.
4. Gated Delta Networks: Improving Mamba2 with Delta Rule. Yang et al., 2024.
5. Kimi Linear: An Expressive, Efficient Attention Architecture. Kimi Team, 2025.
6. SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention. Zhang et al., 2025.
7. SLA2: Sparse-Linear Attention with Learnable Routing and QAT. Zhang et al., 2026.
8. SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers. Xie et al., 2024.
9. SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer. Xie et al., 2025.
10. SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer. Chen et al., 2025.
11. SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer. Zhu et al., 2026.
12. FastVideo FastH3 V1: Open-Weight 4-Step Sparse Distilled Minimax H3 for 14x Speedup on NVIDIA Blackwell GPU. FastVideo Team, 2026.
13. Efficient Streaming Language Models with Attention Sinks. Xiao et al., 2023.
14. Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation. Zhou et al., 2026.
15. Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers. Ge et al., 2026.
16. MiniMax-H3 Turbo LoRA. LarryVRH, 2026.
17. Improved Distribution Matching Distillation for Fast Image Synthesis. Yin et al., 2024.

@misc{xi2026videodeltanet,
  title  = {VideoDeltaNet on MiniMax H3},
  author = {Haocheng Xi and Yiming Xie and Hexu Zhao and Yiwen Zhang and Michael Liu and Thomas Creavin and Kurt Keutzer and Xiuyu Li and Zhaoyang Lv and Chenfeng Xu and Haiwen Feng},
  year   = {2026},
  url    = {https://openvdn.github.io/}
}