Perplexity details the GPU stack serving pplx-embed: Ivy, Tulip and ROSE
Perplexity's engineering team has described how it serves the embedding and ranking models behind its search engine. The stack has three parts: Ivy, Tulip and ROSE. The main finding is that on small embedding models latency tracks token count, not the number of sequences.
Perplexity's engineering team has described how it serves the embedding and ranking models behind its search engine, in a post titled "Fast Embeddings on GPUs" published on 4 September. The stack: Ivy, Tulip and ROSE. Its main finding: latency on small embedding models tracks token count, not sequence count, and under a billion parameters 512 tokens saturate the GPU.
The stack
- Ivy: an HTTP gateway in Rust. It handles JSON parsing, tokenization, input templating and batch splitting, converts the request to a custom gRPC protocol and balances load across replicas. The company says its own unigram tokenizer cut latency significantly against off-the-shelf ones.
- Tulip: a gRPC server in Rust with tokio and tonic; it schedules first come, first served, batches requests and passes them to ROSE.
- ROSE (Runtime-Optimized Serving Engine): the inference engine, mostly Python. It builds no KV cache for embedding and uses ragged attention kernels that need no padding; in the company's words, pplx-embed and Qwen3.5 parsing run through the same kernels.
Where the gain comes from
The CPU-side kernel launch cost can outweigh the GPU work on small batches, so the whole forward pass is captured as a single CUDA graph launched with one call. Capture yields thousands of graphs and can take minutes, so it happens lazily during serving: warm-up on the first hit, capture on the second, replay after that. LazyTensor tracks an asynchronous copy into a page-locked host buffer instead of waiting for the result, so the next batch queues before the previous finishes.
The measurements
The comparison uses vLLM v0.22.0, BF16 and real model weights across four scenarios. Perplexity reports lower latency and higher throughput, but the numbers appear only in charts, and there is no independent testing.
For the corporate reader
For a team building its own search or RAG stack, the gain comes from the shell around the model more than the model: tokenization, batching, CPU and GPU overlap. Larger batches do not buy throughput on a small embedding model, the threshold being around 512 tokens. The measurement setup is spelled out and can be repeated on another workload; the post carries no statement that the stack is open source.
Sources
- Perplexity Engineering, "Fast Embeddings on GPUs", 4 September: perplexity.ai/hub/blog/fast-embeddings-on-gpus
- MarkTechPost write-up, 5 September.
