Redis launches LangCache, a managed semantic cache for LLM apps

Redis has released a managed service that stops production LLM apps from paying for the same question twice. LangCache sits between the application and the model, compares each incoming prompt against previously answered ones by meaning rather than exact wording, and returns the stored response when a close enough match exists.

The service is available now as a public preview on Redis Cloud, accessed through a REST API with Python and JavaScript SDKs. Redis notes that features and behavior may change during the preview.

Not the same as prefix caching

Prefix caching reuses the KV states computed for a shared system prompt, but the request still reaches the model and the answer is still decoded token by token. As Redis puts it, a prefix-cache hit is a cheaper generation call, not an avoided one. LangCache moves the cache outside the model and stores the generated response itself.

Embedding generation is handled by the service, with default models or bring-your-own. Behavior is controlled through similarity thresholds, TTLs and eviction policies, along with adaptive controls for precision and recall. Because it runs on Redis's vector database and is exposed over REST, it works with any LLM provider and any language. Hit rates and savings are tracked from the Redis Cloud console.

The reported numbers

Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster than re-querying the model; the preview announcement cited up to 15x faster responses and up to 70% lower token usage. In a demo run on a paraphrased question, direct inference took 2.232 seconds and used 514 input plus 250 output tokens, while LangCache returned the earlier answer in 0.37 seconds with zero LLM tokens.

The documentation is more measured about how savings accrue: a cached response costs no output tokens, while input token costs are typically offset by embedding and storage costs. The suggested estimate multiplies monthly output token costs by the cache hit rate. On $200 of monthly spend with 60% going to output tokens and a 50% hit rate, that is $60 saved per month.

Customer Mangoes.ai reports a 70% hit rate on its patient-care voice app, 70% lower LLM spend and 4x faster responses. Redis says data stays on the customer's Redis servers and is not used to train models. Full details at MarkTechPost.