Should you cache LLM prompts or embeddings? A cost-vs-flexibility tradeoff guide.

Every API call to a modern large language model (LLM) costs money, and many applications send the same or similar prompts hundreds of times per day. Engineers building LLM-powered systems face a choice: cache prompts to slash costs, or cache embeddings to handle semantic variation. This guide walks through the tradeoffs, implementation patterns, and tuning decisions that determine which approach fits your workload.

How prompt caching trades flexibility for 50-90% cost reductions

Prompt caching stores the internal representation of a prompt—the key-value (KV) cache—so the model reuses it across requests instead of reprocessing tokens from scratch. When a new request shares a prefix with a cached prompt, the provider charges only for the new tokens. Anthropic's Claude and OpenAI's GPT-4 Turbo offer this as a built-in feature.

The savings are dramatic. A 10,000-token system prompt that normally costs $0.03 per request drops to $0.003 when cached, assuming 90% reuse. If an app sends that prompt 1,000 times per day, spending falls from $30 to $3. The catch: the cache only activates when the prompt prefix matches exactly, token-for-token. Change one word in system instructions and the cache misses, forcing a full recomputation.

This makes prompt caching ideal for workloads with stable prefixes. Chatbots with long system prompts, retrieval-augmented generation (RAG) pipelines that inject the same context documents repeatedly, and code-generation tools that prepend fixed instructions all benefit. The rigidity is the price: tweaking phrasing or reordering context invalidates the cache.

Prompt caching delivers the best ROI when your system prompt is large, static, and reused across many user requests.

Vector embedding-based caching: when semantic matching outperforms exact token reuse

Semantic caching stores responses keyed by embedding vectors instead of exact tokens. When a new query arrives, the system computes its embedding and searches a vector database (Pinecone, Weaviate, pgvector) for similar cached queries. If the cosine similarity exceeds a threshold—say, 0.95—the cached response returns without hitting the LLM.

Think of it like a spellcheck dictionary. Exact caching only matches "colour" if the input is "colour." Semantic caching matches "colour," "color," and "what color is the sky" if they map to nearby points in embedding space. The tradeoff: generating embeddings costs pennies per thousand queries (with models like OpenAI's text-embedding-3-small), and occasional false positives occur when dissimilar queries land near each other.

Semantic caching shines when users rephrase the same question. Customer support bots fielding "How do I reset my password?" and "I forgot my password, help" can serve one cached answer. E-commerce search handling "red sneakers" and "crimson running shoes" can reuse product descriptions. The flexibility comes at the cost of complexity: tuning similarity thresholds, managing a vector store, and handling cache invalidation when underlying data changes.

Architectural tradeoffs: latency, cost, and complexity of each caching layer

Prompt caching lives at the provider layer. Anthropic and OpenAI handle the cache internally. Send a prompt, and if it matches a recent prefix, billing drops automatically. Zero infrastructure needed. Latency impact is negligible—cached tokens are nearly free to replay. The downside: no control over eviction policy (caches typically expire after five minutes of inactivity), and caches cannot be shared across different API keys or accounts.

Semantic caching lives at the application layer. Stand up a vector database, generate embeddings for incoming queries, and perform a nearest-neighbor search before deciding whether to call the LLM. This adds 10-50 milliseconds of latency depending on vector store performance and network proximity. Embedding generation costs roughly $0.0001 per query with small embedding models—tiny compared to a full LLM call ($0.01-$0.10+ per request).

The complexity cost is real. Requirements include:

• Deploy and maintain a vector database (Postgres with pgvector, Qdrant, Pinecone).
• Handle cache invalidation when source data updates. If "What's our return policy?" is cached and the policy changes, stale answers create support headaches.
• Tune similarity thresholds to balance cache hits against false positives.

For many teams, the operational burden outweighs cost savings unless query volume is high. A rule of thumb: below 10,000 LLM requests per day, prompt caching's simplicity often wins. Above that threshold, or when queries vary semantically but not topically, semantic caching pays off.

Implementation patterns: pairing caching strategies with Claude, GPT, or open-source LLMs

Prompt caching with Claude or GPT-4 Turbo requires minimal code changes. In Anthropic's API, add a cache_control parameter to designate which parts of the prompt should be cached. A RAG pipeline might mark the system prompt and retrieved documents as cacheable, leaving only the user query uncached:

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    system=[
        {"type": "text", "text": "You are a helpful assistant.", "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": long_context_document, "cache_control": {"type": "ephemeral"}}
    ],
    messages=[{"role": "user", "content": user_query}]
)

OpenAI's prompt caching (available in gpt-4-turbo and later) works automatically when the prefix of messages matches a recent request. No code changes needed; just send requests with identical prefixes and billing reflects the savings.

Semantic caching with any LLM follows a standard pattern:

1. Hash the user query or compute its embedding using a small, fast model (e.g., text-embedding-3-small or an open-source alternative like gte-large).
2. Query the vector database for embeddings with cosine similarity above the threshold (typically 0.90-0.98).
3. If a match is found, return the cached response. If not, call the LLM, store the new embedding and response, and return the result.

A minimal example with Postgres and pgvector:

import openai
import psycopg2

# Generate embedding for incoming query
embedding = openai.Embedding.create(input=user_query, model="text-embedding-3-small")["data"][0]["embedding"]

# Search cache
conn = psycopg2.connect("dbname=cache")
cur = conn.cursor()
cur.execute("""
    SELECT response FROM semantic_cache
    WHERE embedding <=> %s < 0.05
    ORDER BY embedding <=> %s LIMIT 1
""", (embedding, embedding))
result = cur.fetchone()

if result:
    return result[0]
else:
    llm_response = openai.ChatCompletion.create(model="gpt-4", messages=[{"role": "user", "content": user_query}])
    cur.execute("INSERT INTO semantic_cache (query, embedding, response) VALUES (%s, %s, %s)",
                (user_query, embedding, llm_response["choices"][0]["message"]["content"]))
    conn.commit()
    return llm_response["choices"][0]["message"]["content"]

Open-source LLMs (Llama, Mistral) do not offer provider-level prompt caching, but KV cache sharing can be implemented when self-hosting with frameworks like vLLM or TGI (Text Generation Inference). These engines deduplicate prefixes across requests in memory, giving prompt-caching-like behavior without per-token billing. The tradeoff: managing inference infrastructure and scaling it to handle load.

Tuning similarity thresholds and vector databases for production semantic cache systems

The similarity threshold is the single most important knob in a semantic cache. Set it too low (0.80) and false positives occur—"How do I cancel my subscription?" returns a cached answer for "How do I upgrade my plan?" Set it too high (0.99) and valid paraphrases miss, wasting the effort of maintaining the vector store.

Start at 0.95 cosine similarity (or 0.05 cosine distance) and adjust based on precision-recall tradeoffs. Log every cache hit and false positive in production. If users report wrong answers, raise the threshold. If cache hit rate is below 20%, lower it or revisit the embedding model—some domains (legal, medical) need fine-tuned embeddings to cluster similar queries accurately.

Embedding model choice matters. OpenAI's text-embedding-3-small (1536 dimensions, $0.00002 per 1K tokens) is fast and cheap. Cohere's embed-english-v3.0 offers better semantic clustering for English text. Open-source models like gte-large eliminate API costs but require hosting. Benchmark on real queries: generate embeddings for 100 user questions, compute pairwise similarities, and measure how often semantically identical queries land above the threshold.

Vector database selection depends on scale and stack:

• pgvector (Postgres extension): Best if Postgres is already in the stack. Handles millions of vectors, supports HNSW indexing for fast approximate search.
• Qdrant or Weaviate: Purpose-built vector stores with filtering, multi-tenancy, and horizontal scaling. Overkill for small deployments but necessary above 10M vectors.
• Pinecone or Chroma: Managed options that eliminate ops burden. Pinecone charges per vector stored and queried; Chroma is open-source but requires hosting.

Index the vector database with HNSW (Hierarchical Navigable Small World) or IVF (Inverted File) for sub-50ms search latency. HNSW trades memory for speed; IVF clusters vectors into buckets, trading recall for lower memory usage. For most caches, HNSW on 1536-dimensional embeddings delivers 10-20ms p99 latency on commodity hardware.

The right similarity threshold is not a number—it is a function of your users' tolerance for stale or slightly wrong answers.

Monitor cache staleness. In a customer support bot, if the knowledge base updates weekly, expire cached responses older than seven days. In a code assistant, if the codebase changes daily, lower the time-to-live (TTL) or tag cache entries with a version hash. Stale caches cost user trust faster than they save money.

What this means for builders

If an LLM application sends the same system prompt thousands of times per day, enable provider-level prompt caching first. It is a one-line change with zero infrastructure cost and delivers immediate 50-90% savings on cached portions.

If users ask semantically similar questions but phrase them differently, invest in semantic caching. Budget for a vector database (pgvector is a pragmatic starting point with Postgres), allocate one to two weeks to tune similarity thresholds on real traffic, and plan for cache invalidation logic tied to data update cadence.

Combine both strategies when appropriate: use prompt caching for static system prompts and semantic caching for user queries. The two layers stack—prompt caching handles exact prefix reuse, semantic caching handles variation. Be mindful of ops cost: running a vector store and tuning thresholds is only worth it if query volume justifies the effort.

Do not cache blindly. Log cache hit rates, false positives, and latency overhead. If cache hit rate drops below 15%, embedding and search costs may exceed LLM savings. If users complain about stale answers, tighten similarity thresholds or shorten cache TTLs.

Conclusion

Caching LLM prompts and embeddings is not an either-or decision—it is a layered optimization problem. Prompt caching delivers dramatic cost cuts with zero complexity when prefixes are stable. Semantic caching unlocks savings on varied user queries at the price of infrastructure and tuning. The best systems layer both: provider-level caching for static prompts, application-level semantic caching for dynamic queries. The key is matching the caching strategy to workload variance and team operational capacity. Done right, caching transforms LLM economics from a per-call tax into a manageable, predictable cost structure.