llama.cpp brings decision models and faster generation to local servers

llama.cpp released version 0.6.0 on 5 October 2026. llama.cpp is an open-source program that lets you run AI models on your own computer or server, with no need to send data to the cloud; it is developed by the ggml-org community. According to the release notes, the changes that matter most for people running models locally fall into three areas: a new server API for decision models, wider options for speeding up response generation, and support for new models.

The most notable addition is an endpoint called /v1/systemone. It lets decision models run inside llama-server, the server program that ships with llama.cpp. A decision model reads a situation and assigns a probability to each of a set of predefined options. That means classification work, such as routing a support ticket to the right team or identifying the type of a document, can run locally without data leaving the organization.

Decision models return probabilities in llama-server

According to the release notes, the new API arrived with five models, laya, julia-1, lev, openjev and kev, and nimble was added in the same release. Clef, which Cloudflare released a few days ago, is now fully supported with both text and image input. According to the server documentation, the API follows the TypeSafe API format. A request carries the state to evaluate and a set of questions, which come in three types: multiple choice (choice), graded levels (score) and yes/no (noul). The response gives the most likely option, the probability of each option and a confidence value. Since the model does not generate text, the answer comes back in one piece, and streaming is not supported.

The documentation adds a few details. Other models answer each question independently, while Clef reads all the questions in a request together and decides them jointly. A server running Clef serves only this endpoint, so text generation is unavailable there. Image input needs a model that supports it and a multimodal projector file loaded with --mmproj; openjev and Clef are among those models. In addition, GET /v1/models now reports what each model can read and produce. Native decision models are marked with a "decisions" output, so client software can spot a decision model without loading it.

Speculative decoding speeds up generation

Language models produce a response one token, or word piece, at a time, and the whole large model runs at every step. With speculative decoding, a fast method first drafts the next few tokens, then the large model checks the draft in a single pass and accepts the part that is correct. When the draft is often right, responses get noticeably faster, and the output still consists of tokens the large model has approved.

llama-server offers several methods for this, and two stand out in this release. MTP (multi-token prediction) uses a layer inside the model itself that was trained to predict more than one token ahead; no separate draft model is needed, and it is switched on with --spec-type draft-mtp. According to the release notes, MTP support was added for the Qwen3.8-Flash-Next family, with a decode speedup of about 1.5 times measured on NVIDIA DGX Spark. A probabilistic sampling option also arrived for draft models and MTP; it aims to get drafts accepted more often when the temperature setting is above zero.

DFlash works with a separate, small draft model. According to the documentation, this draft model produces a whole block of tokens in a single step (a method called block diffusion) and reads the main model's internal states while predicting. A draft is always trained for one specific target model. An example: llama-server -m model.gguf -md dflash-model.gguf --spec-type draft-dflash --spec-draft-n-max 7; the --spec-draft-n-max value is capped at the block size the draft was trained with. In its Qwen3.8-27B repository on Hugging Face, ggml-org publishes ready-made files prefixed dflash- and mtp- next to the main model files, so both methods can be tried without any conversion. This release also adds DFlash support for MiMo models.

GLM-5.3-Flash and other models gain support

The largest new model is GLM-5.3-Flash (GLM5-Next). According to the release notes, it is a 320-billion-parameter hybrid model with a mixture-of-experts (MoE) design that reads both text and images. Ling 3.0 VL and the LFM2.5-Encoder models also joined the supported list.

The web UI can download models directly

The web interface that comes with llama-server has been reworked. It gains a data layer connected to the Hugging Face Hub and a model download pipeline, plus an estimate of whether a model will fit in the computer's memory. This makes it easier to pick and download a model from the interface without hunting for the file on the command line.

On Apple devices, the Metal backend gets two updates: a new flash attention kernel for an F16 KV cache, and new matrix multiplication kernels used in speculative and batched decoding. According to the release notes, the new kernels make matrix multiplication up to about 3 times faster on Apple GPUs. Vulkan gains sparse flash attention for quantized K/V, and the underlying ggml library moves to version 0.26.0.

MTP sped up generation in GlobalFeed's own test

In GlobalFeed's own test, a fine-tuned model based on Qwen3.8-27B (Q4 quantization) ran on a single NVIDIA RTX 3090 graphics card. With MTP switched on, generation on Turkish business text rose from about 39 to about 58 tokens per second, and the decision outputs stayed the same. The DFlash draft, which is trained for the base Qwen3.8-27B, had only about 5 percent of its drafts accepted with this fine-tuned model on free-form Turkish text and roughly halved the speed there, while JSON extraction got faster. The lesson: a draft trained for a base model may not suit fine-tuned models derived from it, so measure any speed-up method on your own model and in your own language.

The speed figures in the release notes were given for specific models and hardware; the gain varies with the model, the quantization format, the language and the type of task. The server documentation notes that the probabilities returned by decision models are scaled with settings stored in the model file and are not guaranteed to match the real rates in your own data, so it is worth testing them on your own examples before using them on non-English text. This release also raises the format version of saved session files, so anyone relying on sessions saved with an older version should check this before updating. For a fuller picture, follow evaluations from independent researchers as well.

Sources: llama.cpp v0.6.0 release notes, llama-server documentation: /v1/systemone, llama.cpp: speculative decoding documentation, Hugging Face: ggml-org Qwen3.8-27B-GGUF.