> ## Content Index
> Fetch the complete content index at: https://globalfeed.ai/llms.txt
> Use this file to discover other available public pages before exploring further.

# Cohere open-sources its Megakernel serving system
- URL: https://globalfeed.ai/en/cohere-open-sources-cohere-megakernel-up-to-1-58x-faster-than-vllm-on-a-single-h100/
- Published: 2026-09-08T20:41:38.000Z
- Updated: 2026-09-08T20:46:37.000Z
- Description: Cohere has released under Apache-2.0 a serving system that fuses the entire decode step into one CUDA kernel. It is 1.58x faster than vLLM on a single H100, but it runs one model only, on H100 only, at batch sizes 1 to 8.
- Author: GlobalFeed Editor
- Tags: Cohere, Cohere Megakernel, North Mini Code, inference, open source, x-cohere, dil-en, elle, video, x-gitti

**Cohere** has open-sourced its serving system, **Cohere Megakernel**, under **Apache-2.0**. The claim: the first full-fledged serving system built around a decode megakernel, one that fuses an entire decode step into **a single CUDA kernel launch** while still supporting everything a real server needs.

## What a megakernel is

In conventional inference, producing one token means launching dozens of separate GPU kernels in sequence, each launch and each synchronisation costing time. The megakernel inverts that: **one persistent CUDA kernel executes the complete decode forward pass**. The work is decomposed into a task graph where each task is a GEMM tile or an attention split, which removes the full-grid synchronisation barriers between operations.

## The numbers

Measurements are on a single **NVIDIA H100** with **North Mini Code**, against vLLM v0.24 running FlashAttention-3 and the Triton MoE backend.

| Measurement                           | Versus vLLM |
| ------------------------------------- | ----------- |
| Decode only, batch size 1, 8K context | 1.58x       |
| AIME 2025, end to end, batch size 8   | 1.41x       |
| SciCode                               | 1.37x       |
| MMLU-Pro (CS)                         | 1.33x       |
| LiveCodeBench v6                      | 1.28x       |
| GPQA                                  | 1.25x       |

The repository also reports reaching **62% of H100 speed-of-light at batch size 1**. The 1.58 figure in the announcement comes from the best case, the decode-only measurement; with real prompts the end-to-end gain is **1.25x to 1.41x**.

## The limits are written down

The repository does not hide the constraints, and the list is not short. The system does **decode only**; prefill runs on separate PyTorch kernels, and prefill **pauses** decode. Hardware is **H100 only** (SM90a) and **BF16 only**. Batch sizes are **1 to 8**. Sampling is restricted: greedy and temperature, no top\_p, limited top\_k. Because the task schedule is model-specific, the only supported model is **North Mini Code**.

So this is not a general-purpose vLLM replacement. It is an inference engine compressed around one model, one card and a narrow batch range. In exchange it does support the real server work: continuous batching, ragged sequence lengths, a paged KV cache, sliding-window attention, prefix caching, preemption, streaming responses and tool calling, behind an OpenAI-compatible API.

Source: [github.com/cohere-ai/cohere-megakernel](https://github.com/cohere-ai/cohere-megakernel?ref=globalfeed.ai).