Cohere
Cohere open-sources its Megakernel serving system
Cohere has released under Apache-2.0 a serving system that fuses the entire decode step into one CUDA kernel. It is 1.58x faster than vLLM on a single H100, but it runs one model only, on H100 only, at batch sizes 1 to 8.