Nunchux AI releases VC-Attention to speed up video diffusion models

Nunchux AI has released VC-Attention, a training-free low-bit attention kernel built for video Diffusion Transformers (DiTs). It addresses two problems at once: value quantization error and a slow softmax stage.

Video DiTs flatten a clip into one sequence of spatiotemporal tokens and run full self-attention at every layer. A 5-second 720p Wan2.2-14B clip spans roughly 70K tokens. On the RTX 5090, attention consumes more than 64% of generation time. The research team says attention accounts for about two thirds of every MiniMax-H3 denoising step on a single B200.

Two bottlenecks

Earlier methods such as SageAttention2 smooth queries and keys. After QK smoothing and rotation, the value term accounts for 82% of output error on Wan2.2. Meanwhile the softmax between the two matrix products still runs in FP32, and on B200 and H200 the exponential plus its FP8 cast becomes the longest pipeline stage.

Value outliers sit in a few tokens and their channels shift across heads, layers and steps. A Hadamard rotation preserves token norms, so rotating V changes value error by only 0.2%. Nunchux's block mean, averaged over 100 Wan2.2 heads, removes 8% of block energy in sequence order and 36% after sorting, at a cost of 0.125 bit per value element.

ExpCast-FP8 exploits the fact that an E4M3 byte already approximates the logarithm of the value it stores, writing the byte directly from the log-domain score with one fused multiply-add. Across 204.8K Wan2.2 attention rows the measured average deviation is 1.6%.

Measurements

Tests cover four open-weight video DiTs: Wan2.2-T2V-A14B, LongCat-Video, HunyuanVideo-1.5 and MiniMax-H3. On B200, VC-Attention is 6.02x faster than SageAttention2, which ships no Blackwell kernel; on H200 the gap narrows to 1.16x. On MiniMax-H3 at 1344x768, attention runs 1.60x faster than BF16 FlashAttention-4 on B200, with PSNR of 20.2 dB against 19.9 dB for SageAttention2.

Because the method changes only per-interaction cost, the company says it can compose with sparse attention approaches, distillation and multi-GPU execution. Full details are in the MarkTechPost report.