Semantic Caching for LLMs: Cut Costs Without Serving a Confidently Wrong Answer
Say your support bot gets asked “how do I reset my password” four hundred times a day. Same question, four hundred phrasings. And every time, you're paying full price to generate the exact same answer from scratch. Wasteful, like reheating a fresh meal every time someone asks what's for dinner.
Semantic caching is the fix for that specific kind of waste. The short version: it stores past questions and answers as vector embeddings, and when a new question means roughly the same thing as one it's already answered, it hands back the stored response instead of calling the LLM again. No new generation, no new token bill.
Here's the part most articles gloss over, though. Semantic caching is the cheapest big win you'll find in LLM cost engineering, and it's also the one most likely to confidently serve the wrong answer if you get one number wrong. That number is the similarity threshold, and we'll spend a good chunk of this piece on it, because it's where things quietly go sideways in production.
We'll cover how it works end to end, how to pick and test a threshold instead of guessing, a real cost model with actual arithmetic, when this is a genuinely bad idea, and what the tooling landscape looks like.
What Is Semantic Caching, Actually?
Semantic caching stores previous LLM queries and their responses as vector embeddings, then returns a stored response when a new query is semantically similar enough to one already sitting in the cache. Unlike a normal cache, it's matching on meaning, not on an exact string.
Compare that to the cache you already know from web development. A regular cache keys off an exact URL or a request hash. Change one character and it's a different key entirely. Natural language questions are basically never byte-identical, so exact-match caching on user queries gets you a hit rate close to zero in practice. That's the gap semantic caching closes.
One quick disambiguation before we go further, because people mix these up constantly: this article is about caching responses at the application layer. It's not about the transformer's internal KV cache, which is a different thing entirely that stores attention key-value tensors while the model is generating tokens. Different layer, different problem, different article.
Here's the property that explains basically everything difficult about this technique: a semantic cache is lossy. A regular cache either has your exact key or it doesn't, never wrong, just absent. A semantic cache makes a judgement call on every lookup, and judgement calls have an error rate. Keep that in mind, it comes up again.
This also isn't some scrappy hack anymore. AWS documents it as a first-class feature of ElastiCache, and Microsoft ships it for Azure Cosmos DB. If you're still treating this as a weekend side project, the cloud providers have already moved past you.
Semantic Caching vs Exact-Match Caching vs Prompt Caching
This is probably the most useful section here, and it's also where most articles on this topic get muddled. Redis, cloud vendors, gateway companies, all of them blur semantic caching with provider-side prompt caching. They are not the same thing.
| Dimension | Exact-match caching | Provider prompt caching | Semantic caching |
|---|---|---|---|
| What's stored | Full response, keyed on prompt hash | Provider-side state for a prompt prefix | Full response, keyed on query embedding |
| Where it lives | Your app (Redis, Memcached...) | The model provider's infra | Your app + a vector store |
| Match condition | Byte-identical prompt | Identical prefix, token-exact | Similarity score above your threshold |
| Input tokens saved | All | Most (at a discount) | All |
| Output tokens saved | All | None, regenerated every time | All |
| Risk of a wrong answer | None | None | Real, and threshold-dependent |
| Best for | Identical repeated calls, batch jobs | Long stable system prompts, big documents | Paraphrased, repetitive user questions |
Here's the economic point almost nobody spells out: provider prompt caching only discounts input tokens. In our cost model below, output tokens make up the majority of the per-request bill. So no matter how well you configure prompt caching, it can't touch the biggest chunk of your spend on a short-prompt, long-answer workload. Semantic caching, on the other hand, skips the call entirely. Rule of thumb: the longer your prompt relative to your answer, the more prompt caching helps. The shorter your prompt relative to your answer, the more semantic caching helps.
And here's the thing people miss: these aren't rivals fighting for the same job, they stack. Put a semantic cache in front so repeat questions never reach the model, then let prompt caching handle the misses so they pay less for their stable prefix.
All the major providers now ship some version of prefix or context caching, worth reading the official docs directly rather than trusting a summary that'll be outdated in three months: Anthropic, OpenAI, and Google Gemini.
How It Actually Works, Step by Step
Strip away the vendor branding and every semantic cache is doing roughly the same seven things on every request:
1. Normalise the query. Trim whitespace, handle casing, strip anything sensitive. Small step, disproportionate impact on hit rate.
2. Embed the query. One call to an embedding model, cheap and fast, usually tens of milliseconds.
3. Search the cache. ANN (approximate nearest-neighbour) search finds the closest stored query and its score, “close enough, fast” rather than checking every entry.
4. Check the threshold. Clears the bar, candidate hit. Doesn't, it's a miss.
5. Validate the candidate. Expired? Right namespace? For high-stakes cases, does a verifier agree the stored answer fits?
6. Hit: return the stored response. No model call, no output tokens, no wait.
7. Miss: call the LLM normally, then write the new vector and response back for the next paraphrase.
Here's a bare-bones, framework-agnostic version of the logic, stripped of any vendor SDK so you can actually see what's happening instead of copy-pasting a black box:
Two lines matter more than the rest, and naive implementations skip both: the namespace (invalidation, below) and source_doc_ids (staleness, also below). Everything else is standard plumbing.
One practical gotcha: cache the user's raw question, not the entire assembled prompt. Embed the whole thing, retrieved chunks included, and those chunks end up dominating the vector. Two genuinely different questions that happen to pull the same documents will look identical to the cache. Real bug, more common than you'd think.
Build an AI-First Career, Master the Complete Skillset
Choose from our industry-leading programs designed for career success
Modern Software and AI Engineering Program
Master full-stack development with AI integration
+1000 moreModern Data Science and ML with specialisation in AI
Advanced data science techniques with AI specialization
+1000 moreAdvanced AIML with Specialisation in Agentic AI
Deep dive into AIML with focus on Agentic systems
+1000 moreDevOps, Cloud & AI Platform Engineering
Build and manage AI-powered cloud infrastructure
+1000 moreAI Engineering Advanced Certification by IIT-Roorkee
Premier AI engineering certification from IIT-Roorkee
AI Forward Deployed Engineer Program
Full-stack engineering, production AI and client-facing consulting
+1000 moreWhat Semantic Caching Actually Saves (With Real Math)
Every vendor page loves the phrase save up to 90%. None show a token price, a request volume, or an infrastructure cost behind it. So let's just do the arithmetic. Prices below are illustrative, check live provider pricing before basing a real decision on them.
Scenario: a support assistant handling 100,000 LLM queries a month. Average request is 1,500 input tokens (system prompt, retrieved context, the actual question) and 400 output tokens. Model priced at 15.00 per million output tokens. Embeddings at 50/month.
● Input cost per request: 1,500 ÷ 1,000,000 × $3.00 = $0.0045
● Output cost per request: 400 ÷ 1,000,000 × $15.00 = $0.0060
● Total per request: $0.0105, output tokens alone are 57% of that
● Baseline monthly spend: 100,000 × $0.0105 = $1,050.00
● Embedding every query costs about $0.06/month, basically free, despite the common fear that embedding everything gets expensive
● Cache overhead: roughly $50.06/month, mostly the vector store
Now, how that plays out at different hit rates:
| Hit rate | LLM spend | Cache overhead | Total | Net saving |
|---|---|---|---|---|
| 0% | $1,050.00 | $50.06 | $1,100.06 | 4.8% |
| 10% | $945.00 | $50.06 | $995.06 | 5.2% |
| 20% | $840.00 | $50.06 | $890.06 | 15.2% |
| 30% | $735.00 | $50.06 | $785.06 | 25.2% |
| 40% | $630.00 | $50.06 | $680.06 | 35.2% |
| 50% | $525.00 | $50.06 | $575.06 | 45.2% |
| 70% | $315.00 | $50.06 | $365.06 | 65.2% |
So at a realistic 30% hit rate, this saves KaTeX parse error: Expected 'EOF', got '%' at position 23: …month, about 25%̲. Roughly 3,180 a year. That's genuinely good, and completely different from the “up to 90%” headline you keep seeing everywhere.
Here's the part nobody else publishes, and honestly the most useful thing in this whole article: the break-even point. The cache costs money whether it works or not, so it has to clear that bar before it saves you anything.
● At 100,000 queries/month, break-even sits at 4.8% hit rate. Easy.
● At 1,000,000 queries/month, break-even drops to 0.48%. Basically unavoidable to clear.
● At 10,000 queries/month, break-even jumps to 47.6%. Most workloads never get there. At small scale, the cache often costs more than it saves.
So, rule of thumb: semantic caching is a volume game. Below roughly ten thousand queries a month, you're probably better off with plain exact-match caching and a well-set-up prompt cache instead.
There's also the latency angle, which honestly matters more to users than the cost line. A cache hit is tens of milliseconds. A generation is one to several seconds, an order of magnitude difference, and it hits your most common questions. Before building anything, spend an afternoon on a one-week offline estimate: export real production queries, cluster them by similarity at your candidate threshold, count the hits. Cheap experiment, honest answer.
The Threshold Problem: Where This Actually Breaks
Every competing article names a threshold number, usually 0.8 or 0.85, and moves on. None explain why, or how to check it's right for your data. This is where semantic caching quietly breaks in production.
Think of the threshold as a dial between two failure modes, and no setting dodges both. Too loose, and the cache confidently answers a question the user didn't ask, zero indication anything's wrong. Worse than a miss. Too tight, and hit rate collapses, you pay for infra and save nothing, and someone declares the whole idea doesn't work.
Why one number can't cover everything
Embedding similarity measures how related two questions are topically. It does not reliably measure whether they have the same answer. That gap is where every false hit comes from. Here's a few pairs that make the problem obvious:
| Query A | Query B | Similarity | Same answer? |
|---|---|---|---|
| How do I enable 2FA? | How do I disable 2FA? | Very high | No, opposite |
| Capital of Austria? | Capital of Australia? | Very high | No |
| Is Python faster than Java? | Is Java faster than Python? | Near-identical | No, inverted |
| Cancel my subscription | Refund after cancelling | High | No, related but not the same |
The reason: embedding models are systematically weak on negation, antonyms, and argument order, exactly the features most likely to flip an answer. Two questions can score 0.95 similar and have opposite correct answers. Not a tuning failure, just how embeddings represent meaning. A domain full of enable/disable, add/remove pairs needs a much tighter threshold, or a verifier, than a domain of open informational questions.
Small but annoying trap: check whether your vector store returns similarity or distance. Some return distance, where lower means more similar, so a “0.2 threshold” there is actually roughly 0.8 similarity. Get this backwards and you've inverted your entire cache. Fails loudly enough to eventually notice, quietly enough to reach production first. Ask me how I know. (Kidding. Mostly.)
How to actually evaluate a threshold
Hit rate alone is not a success metric. A threshold of zero gives you a 100% hit rate and a completely useless product. You need hit rate and precision together, always.
● Hit rate = cache hits ÷ total queries. How much you save.
● Cache precision = correct hits ÷ total hits. How much you can trust it.
A quick, runnable evaluation method:
1. Sample 200 to 500 real queries from production logs.
2. For each, write a paraphrase that must hit, and an adversarial near-miss that must not.
3. Label each pair: does the stored answer actually answer the new question?
4. Sweep the threshold from 0.75 to 0.99, computing hit rate and precision at each step.
5. Plot both curves, find the knee, move one notch tighter than feels comfortable.
6. Re-run whenever you switch embedding models. Scores aren't comparable across models.
Tooling defaults commonly sit around 0.8 to 0.9 cosine similarity, precision-sensitive domains often run tighter. Treat any published default as a starting point, never a setting you ship blind.
The best escape hatch: a two-stage cache. Use vector search for cheap recall, then confirm with a lightweight verifier, a small cross-encoder or a fast LLM asked one yes/no question, before returning anything. Costs a fraction of a full generation and lets you run a looser threshold safely. If you want the academic version of this argument, there's decent work on domain-specific semantic caching thresholds.
Cache Invalidation and Staleness
A semantic cache decouples the answer from the knowledge that produced it. Update your refund policy today, and it'll happily keep quoting the old one to every paraphrase of “what's your refund policy” until the entry expires. Nothing errors, nothing alerts. Looks perfectly healthy while quietly lying to your users.
| Strategy | How it works | Watch out for |
|---|---|---|
| TTL | Every entry expires after a fixed period | It's a guess about how fast your knowledge changes. Set per content type, not globally. |
| Version-keyed namespaces | Hash of model ID, prompt version, index version in the cache key | Old namespaces get orphaned, not deleted. Pair with a size cap. |
| Event-driven eviction | Source document changes, evict every cached answer that used it | Needs provenance stored at write time, the step most people skip. |
| Feedback-driven eviction | A thumbs-down evicts the entry, not just logs it | Needs a feedback signal actually wired to the cache. |
Suggested starting points: stable documentation, hours to a day. Semi-dynamic stuff like pricing, minutes to an hour. Genuinely real-time data, don't cache at all. Rule of thumb: TTL should always be shorter than the interval at which the answer can change.
Version-keyed namespaces deserve a mention because they're elegant and underused. Change your system prompt and you've changed the correct answer to a bunch of questions. A version-keyed namespace invalidates the whole affected slice atomically at deploy time, no purge job, no race condition. Most teams never think to put the prompt version in the cache key, then burn a day debugging why the assistant's new personality keeps reverting.
Also worth doing: bound the cache size, evict on LRU, and guard against a stampede (a popular entry expires, a hundred requests miss it at once) with a lock or stale-while-revalidate. And log the age of every entry served, median staleness catches invalidation bugs faster than anything else here.
Semantic Caching in RAG and Agent Systems
A RAG pipeline gives you three different places to cache, and they carry very different risk profiles:
| Layer | What's cached | Saving | Risk |
|---|---|---|---|
| Embedding cache | Vector for text you've embedded before | Tiny | None, deterministic for a fixed model |
| Retrieval cache | Chunk IDs returned for a query | Small | Low, model still reasons over context |
| Response cache | The final generated answer | Large | Highest, bypasses fresh retrieval entirely |
Sensible adoption order: embedding cache first, it's free and safe. Retrieval cache next, usually fine. Response cache last, only once you've got a tuned threshold, real namespacing, and invalidation tied to index version. Most teams reach for the response cache first because it saves the most, then learn the hard way why it's listed last.
One detail that keeps the retrieval cache safe: cache chunk IDs, not chunk text. Re-fetching by ID pulls current text from the store, so a document edit shows up immediately even though retrieval was cached. Cache the text instead and you've quietly built a second staleness problem on top of the first.
Agents multiply both upside and risk. One request can trigger a dozen model calls, so caching sub-steps saves a lot, but an early wrong step propagates through everything after it. Safer to cache: query rewriting, classification, routing, summarising static docs. Riskier: anything depending on a fresh tool result or conversation state.
The conversational trap deserves its own line: “what about the second one” is a stable embedding with a completely different correct answer depending on which thread it's in. Cache key just the query text, and you'll serve one conversation's answer into another. Fold conversation state into the key, or don't cache mid-conversation turns at all.
When Semantic Caching Is a Bad Idea
Nobody selling you a caching product is going to write this section, which is exactly why it needs writing. A semantic cache is safe only when the meaning of the question fully decides the answer. Every case below has something else deciding it too.
● Personalised answers. “What's my balance” embeds identically for every user but answers differently for each. Partition per user and it only hits when someone repeats themselves, hit rate collapses back to near zero.
● Time-sensitive data. Prices, inventory, scores, weather, anything “latest.” Decays faster than any TTL you'd risk setting.
● Conversational context. Already covered above, worth saying twice.
● High-stakes domains. Medical, legal, financial. A near-miss here isn't slightly wrong, it's harmful. Run a verifier and a tight threshold if you cache here at all.
● Creative tasks. Brainstorming, copy, code generation. Users want variety, a cache returns the same string every time, feels broken.
● Low-repetition workloads. If questions rarely repeat, hit rate stays near zero and you're paying infra for nothing.
● Low volume, full stop. Below roughly ten thousand queries a month, break-even needs about a 48% hit rate. Most workloads never get there.
Quick checklist: cache when queries repeat, the answer depends only on the question, some staleness is fine, volume is high, and a near-miss is embarrassing rather than dangerous. Don't cache when answers are personalised, time-sensitive, or conversation-dependent, when queries rarely repeat, when volume is low, or when a near-miss could actually hurt someone.
One war story: a team watched their hit rate jump from 30% to 80% overnight and celebrated, only to find the embedding service had started returning zero vectors, which are maximally similar to literally everything. Every query hit the same cached answer. Treat a sudden hit-rate jump as an incident until proven otherwise, not a win to screenshot for the team Slack.
How Scaler Transformed Careers in Different Fields
Scaler learners achieved 2.5x salary growth with average post-Scaler CTC reaching ₹23L.
Google
Amazon
Microsoft
Flipkart
Adobe1200+ moreSemantic Caching Tools and Libraries
Kept strictly neutral here, open source listed first, no winner declared:
| Tool | Type | Best for |
|---|---|---|
| GPTCache | Open-source library | Learning the pattern, self-hosted Python stacks |
| LangChain cache integrations | Framework feature | Teams already on LangChain |
| Redis (RedisVL / LangCache) | Library + managed service | Teams already running Redis |
| LiteLLM / Portkey | Gateway proxy | Multi-model orgs, fastest pilot, zero code change |
| Momento | Serverless cache | Avoiding infra ownership entirely |
| Amazon ElastiCache / Azure Cosmos DB | Managed cloud feature | AWS-native or Azure-native stacks |
| Roll your own on pgvector / Qdrant / Milvus | DIY | Full control over threshold, verifier, invalidation |
Worth actually reading before picking a tool: the original GPTCache paper if you want the source of the pattern, the GPTCache repo if you want to poke at code, and the LangChain caching docs if you're already in that ecosystem.
When evaluating any of these, check whether it lets you change the threshold, supports a verifier stage, lets you namespace by tenant and version, stores provenance for invalidation, and exposes hit-rate and precision metrics. Most tools are weak on that last group.
Honest advice: start with a gateway for a two-day pilot, measure real hit rate with zero code changes. If the numbers justify it, move to a library or your own build where you control threshold, verifier, and invalidation, the three things that decide whether this is safe.
How to Measure a Semantic Cache in Production
Once it's live, track: hit rate (the saving), cache precision (the safety, sample hits weekly and check them), a similarity histogram split by hits and misses, p50/p95 latency by hit and miss, cost per 1,000 queries, median staleness, and cache size plus eviction rate.
Counterintuitive part: alert on a sudden hit-rate increase, that's usually a bug, not a win. Alert on precision dropping after a knowledge-base update, invalidation isn't working. Alert if hit-path latency starts approaching miss-path latency, the cache has stopped earning its keep.
Keep a cache bypass flag you can flip per request or deployment. When an answer looks wrong, the first question is always “was that cached,” and you want an answer in seconds, not by staring at a similarity score guessing.
FAQs
What is a semantic cache?
A store of past questions and answers indexed by vector embedding instead of exact text. When a new question is close enough in meaning to a stored one, it returns the stored answer instead of calling the LLM.
Turn Learning into Career Growth
How is semantic caching different from prompt caching?
Prompt caching discounts input tokens for a repeated prefix, the answer is still generated fresh. Semantic caching skips generation entirely, saving both input and output tokens. They stack well together.
What similarity threshold should I use?
There's no universal number. Defaults commonly sit around 0.8 to 0.9 cosine similarity, but sweep it against a labelled set of paraphrases and near-misses and pick where precision actually holds up for your data.
How much does semantic caching actually save?
It scales with hit rate. At 100,000 queries a month and roughly KaTeX parse error: Expected 'EOF', got '%' at position 35: … a realistic 30%̲ hit rate saves…265 a month, around 25%. Below roughly 10,000 queries a month, infrastructure often costs more than it saves.
Can Redis be used for semantic caching?
Yes, Redis supports vector search and ships semantic-cache tooling, which makes it a common pick for teams already running it. Any vector store, pgvector, Qdrant, Milvus, works just as well.
What are the three types of cache in LLM systems?
Exact-match caching, provider prompt caching, and semantic caching. Only the third one handles rephrased questions, and only the third one can actually be wrong.
When should you not use semantic caching?
When answers are personalised, time-sensitive, or depend on conversation context, when a near-miss could cause real harm, or when query volume is too low to cover the cache's own cost.
Wrapping Up
Semantic caching is one of the rare LLM optimisations that improves cost and latency at the same time, and the only one here that can quietly hand someone a wrong answer with total confidence. The hard part was never the vector search, that's a library call. It's the threshold, the namespacing, and knowing which questions should never touch the cache.
Realistic expectations matter. A 30% hit rate on 100,000 monthly queries saves around 25%, not 90%, and at low volume it might save nothing. Teams tracking precision alongside hit rate ship this safely. Teams watching only hit rate eventually ship a confident, wrong answer to a real customer.
Before writing a line of code, run the one-week offline hit-rate estimate. Costs an afternoon, tells you honestly whether it's worth building. If you'd rather go deeper into building LLM systems that are cheap, fast, and correct in production, retrieval, caching, evaluation, deployment, the whole stack, that's basically the premise of Scaler's AI & Machine Learning program.
