Kimi K3 ran on a single CPU in 8 GB of memory, by streaming from disk instead of fitting

An inference engine has been released that runs Kimi K3 on a single CPU, with no GPU, in 8.24 GB of measured peak memory. The whole engine is written in portable C99 and compiles to a single 176 KB binary. No BLAS, no framework, no CUDA. It is licensed under Apache 2.0.

The number looks impossible at first. The weights of a 2.78-trillion-parameter model do not fit in 8 gigabytes, and indeed they do not. What the project changes is not the model's size but the way it is read.

The memory ceiling was a property of loading, not of the model

Kimi K3 is a mixture-of-experts model: only a small share of the weights is used at each step. The engine leans directly on that. The dense trunk stays resident in memory, while 1.45 TB of routed experts remain on disk and are read from NVMe through O_DIRECT calls that bypass the operating system's cache. An LRU cache sits in front of the ones used most often.

The interesting consequence is that memory stops being a threshold and becomes a dial. Whatever RAM you have, the same model returns byte-identical answers. Only speed moves:

• 26.5 seconds per token in 8 GB
• 5.6 seconds per token at 128 GB and above

That is minutes for a sentence and half an hour for a paragraph. This is a demonstration, not a product.

Where the cost lands

What is saved in memory is paid in storage: the setup needs a 1.56 TB checkpoint plus a 109 GB packed trunk, roughly 1.7 TB of local disk and a download measured in hours. The CPU has to support AVX2 and FMA. Linux x86-64 is the primary target, with macOS/ARM64 and Windows also supported.

There is one more limit that tends to get lost under the headline: what runs here is the base model, with no chat templating. It continues the text you give it rather than answering questions, so on its own it is not usable as an assistant.

Why it matters

The value of the project is not its speed but what it makes visible. Anyone looking at running models locally arrives at the same sentence: "this model will not fit on my card." This work shows that the sentence is not always true. Not fitting is often a property of our habit of loading everything into memory at once, rather than of the model itself. If most experts are not used on a given step anyway, keeping all of them resident turns out to be a choice rather than a requirement.

In practice nobody will do real work at one word every 26.5 seconds. But the same idea, applied to smaller mixture-of-experts models on faster storage, lands in far more reasonable territory. Fareed Khan, who wrote the code, sums the approach up in one line in the repository: same model, same answers, different speeds depending on the memory you have.

We covered Kimi K3 before, when it went rentable on CoreWeave. The question there was who could afford to run it; this project answers the same question from an entirely different direction.