Cohere open-sources its Megakernel serving system
Cohere has released under Apache-2.0 a serving system that fuses the entire decode step into one CUDA kernel. It is 1.58x faster than vLLM on a single H100, but it runs one model only, on H100 only, at batch sizes 1 to 8.
Cohere has open-sourced its serving system, Cohere Megakernel, under Apache-2.0. The claim: the first full-fledged serving system built around a decode megakernel, one that fuses an entire decode step into a single CUDA kernel launch while still supporting everything a real server needs.
What a megakernel is
In conventional inference, producing one token means launching dozens of separate GPU kernels in sequence, each launch and each synchronisation costing time. The megakernel inverts that: one persistent CUDA kernel executes the complete decode forward pass. The work is decomposed into a task graph where each task is a GEMM tile or an attention split, which removes the full-grid synchronisation barriers between operations.
The numbers
Measurements are on a single NVIDIA H100 with North Mini Code, against vLLM v0.24 running FlashAttention-3 and the Triton MoE backend.
| Measurement | Versus vLLM |
|---|---|
| Decode only, batch size 1, 8K context | 1.58x |
| AIME 2025, end to end, batch size 8 | 1.41x |
| SciCode | 1.37x |
| MMLU-Pro (CS) | 1.33x |
| LiveCodeBench v6 | 1.28x |
| GPQA | 1.25x |
The repository also reports reaching 62% of H100 speed-of-light at batch size 1. The 1.58 figure in the announcement comes from the best case, the decode-only measurement; with real prompts the end-to-end gain is 1.25x to 1.41x.
The limits are written down
The repository does not hide the constraints, and the list is not short. The system does decode only; prefill runs on separate PyTorch kernels, and prefill pauses decode. Hardware is H100 only (SM90a) and BF16 only. Batch sizes are 1 to 8. Sampling is restricted: greedy and temperature, no top_p, limited top_k. Because the task schedule is model-specific, the only supported model is North Mini Code.
So this is not a general-purpose vLLM replacement. It is an inference engine compressed around one model, one card and a narrow batch range. In exchange it does support the real server work: continuous batching, ragged sequence lengths, a paged KV cache, sliding-window attention, prefix caching, preemption, streaming responses and tool calling, behind an OpenAI-compatible API.
