DeepSeek-V4.1-Flash arrives with a 1M context and 890 bytes of KV cache per token

DeepSeek AI has released DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts model with 552B backbone parameters, 196B additional Engram parameters and a 1M-token context window. It activates 8B parameters per token during prefill and 16B during decode.

The headline figure is the global KV cache footprint: 890 bytes per token, roughly a quarter of DeepSeek-V4-Flash and about 437x smaller than DeepSeek-V1.

How the architecture changes

The 40-layer backbone splits into a 20-layer causal encoder and a 20-layer decoder. Inspired by YOCO, the decoder computes no global KV of its own; per-layer projection weights derive it from the final encoder hidden state. Prompt tokens stop at the encoder, which nearly halves prefill compute. Sliding-window attention with a 128-token window still runs in every layer, and decoder SWA states are rebuilt by replaying only the last 128 prompt tokens, a step the team calls Decoder SWA Bounded Replay.

V4.1-Flash uses pure CSA2 and cuts cache size along the layer axis. The 18 encoder layers use a compression ratio of 2 in three groups of six, while the 20 decoder layers use ratio 1 in five groups of four. A Hierarchical Sparse Indexer lets the Full decoder layer build a candidate pool of up to 16,384 positions, so later Reindex layers score a bounded set.

The main KV cache is quantized to E2M1 with one E4M3 scale per 16 channels, following NVFP4 without its global scale, which nearly halves storage against V4's FP8 cache. SWA KV is no longer persisted to SSD: it sits in a distributed pool carved from 10% of host DRAM with a TTL of minutes, while global KV keeps a guaranteed 72-hour lifetime.

Training and availability

Pre-training covers 45T multimodal tokens at a 7:1 text-to-multimodal ratio. Sparse attention is trained from scratch at 64K sequence length with no dense warmup, and context is extended to 1M at 34T tokens. The team says the base model matches DeepSeek-V4-Pro-Base on world knowledge and coding using a third of the total and a quarter of the activated parameters.

Open weights ship under an MIT license on Hugging Face with vLLM, SGLang and Transformers paths. Full details are in the MarkTechPost report.