llama.cpp 0.4.0: initial Qwen3.8-Flash-Next and Nemotron-3-Puzzle support, plus two changes that cut memory
The project most people use to run models on their own hardware has a new release. Two new model families gain initial support, but the useful part is memory: tensors are now read on demand and the quantizer is capped. On the server side, context limits are enforced per slot.
llama.cpp, the de facto standard for running models locally, has published version 0.4.0. The announcement came from the project's own account.
The release notes come down to three things: initial support for two new model families, on-demand tensor reading, and per-slot context limits on the server.
New models
- Qwen3.8-Flash-Next (the
qwen4exparchitecture). The notes state this is initial support, with optimization work still pending. - NVIDIA Nemotron-3-Puzzle-75B-A9B
- Nemotron 3.5 DSpark variant
- nanbeige4.2-3B
Sparse flash attention support also landed for DeepSeek-V4, GLM and qwen4exp.
The real story is memory
The version number is not the news; these two changes are. Both address the same problem: the memory spike you hit when running a large model on a modest machine.
On-demand tensor reading. Tensors are now loaded lazily, as needed, rather than all at once, which lowers the memory overhead during model initialization.
Quantizer RAM capping and row-slab streaming. The quantizer now has a ceiling and streams data in row slabs, to prevent memory spikes during quantization.
Together these mean room for a bigger model on the same hardware, or the same model on less memory.
Server side: context per slot
The llama.cpp server can now enforce the context size limit separately for each slot. That makes resource management across concurrent requests easier: a single long-context request no longer consumes the whole server's memory.
Two more additions sit alongside it: KV-cell token tracking with n-gram history lookup, and Apple RDMA transport support for RPC. The underlying ggml dependency moves from 0.22.0 to 0.23.0.
For the corporate reader
This release is aimed at the organization that wants to run a model on its own machine rather than send data to a cloud. The obstacle to local inference is usually not the model but the memory needed to open it, and memory is measured at the peak, not the average. On-demand tensor reading and the quantizer cap touch exactly that peak.
Practically: a model that did not fit on your server before may fit now. Measure it on your own hardware before committing, because the gain varies by model and quantization format.
Sources
- Primary source: llama.cpp v0.4.0 release notes, ggml-org: github.com/ggml-org/llama.cpp
- Announcement: the release post from the project's own account, 4 September.
