Perplexity open sources Lily, a Rust and Metal inference engine for Apple Silicon
Perplexity released Lily, the local engine running Qwen3.6-35B-A3B on Apple Silicon, reporting 1.23x faster prefill and 1.35x faster decode than MLX-LM.
Perplexity has open sourced Lily, the local inference engine behind Hybrid Compute in Perplexity Computer. It is a single-process runtime: a Rust layer loads the checkpoint and drives the generation loop, an OpenAI-compatible chat-completions API streams tokens, and hand-written Metal kernels execute the model. Neither PyTorch nor MLX sits in the execution path.
Lily is deliberately narrow, one model (Qwen3.6-35B-A3B) on one hardware family, and Perplexity presents that narrowness as the performance argument: model structure, execution plans and kernel selection all live in the same runtime. A standalone demo is available in the pplx-garden repository, serving greedy text generation over a minimal OpenAI-compatible HTTP API.
Requirements and model layout
The 4-bit checkpoint is 19.4 GB, making an Apple silicon Mac with 32 GB or more of unified memory the realistic floor. The shipping Hybrid Compute product lists macOS 15+, 24 GB minimum and 32 GB for best results.
Qwen3.6-35B-A3B stores 35B parameters and activates roughly 3B per token. A router scores 256 experts and picks eight, plus one shared expert that sees every token. The model mixes 10 full-attention layers using grouped-query attention (16 query heads, two KV heads) with 30 Gated DeltaNet layers. Weights use groupwise affine 4-bit quantization with every group of 64 sharing a bfloat16 scale and bias, compressing about 70 GB of bfloat16 weights to 19.4 GB.
Measurements
In Perplexity's ablation, dequantizing one tile at a time inside the grouped GEMM raised end-to-end prefill 77.4% at a 512-token prompt, and keeping routing work inside a single GPU command buffer added 89% at the same length. Expert GEMMs account for roughly 90% of prefill time. On decode, GQA packing improved throughput 23.8% at 32K, while a fixed-block attention layout gained 40.2% at 128K.
On one 40-core, 128 GB M5 Max at batch 1, loading identical 4-bit checkpoint bytes against MLX-LM's fastest direct-generation path across ten lengths from 256 to 128K tokens, Lily averaged 4,156 prefill tokens/s versus 3,388 (1.23x) and 170.0 decode tokens/s versus 126.4 (1.35x). A teacher-forced check across 192 positions put Lily's perplexity 0.04% higher, with the same top-ranked token 96.35% of the time.
Technical details and the repository are covered in MarkTechPost's report.
