AI semantic caching

Semantic caching caches AI responses by prompt embedding similarity: a paraphrased prompt within the similarity threshold returns the cached response with no provider call and no token spend. This is a significant cost saver for workloads with repeated or paraphrased prompts.

Semantic caching is compiled into the default build (the the hnsw_rs HNSW ANN index adds binary size). Build with cargo build to enable it. Without the feature, the config is accepted but the cache is inert (all requests hit the provider).

Configuration

ai:
  semantic_cache:
    enabled: true
    embedding_url: http://localhost:11434/v1/embeddings
    embedding_model: all-MiniLM-L6-v2
    embedding_dim: 384
    threshold: 0.85
    ttl_secs: 3600
    max_entries: 10000
    embedding_timeout_ms: 5000
    embedding_api_key: ${EMBEDDING_API_KEY}

Field	Default	Notes
enabled	false	Must be true to activate the cache
embedding_url	(required)	URL of an OpenAI-compatible /v1/embeddings API
embedding_model	(required)	Model name passed to the embedding service
embedding_dim	(required)	Vector dimension; must match the embedding service's output
threshold	0.85	Cosine similarity threshold (0.0 to 1.0); higher = stricter
ttl_secs	3600	Entry TTL in seconds; stale entries are not returned
max_entries	10000	Max cached entries; when full, the least-recently-used entry is evicted (LRU)
embedding_timeout_ms	5000	Timeout for the embedding service HTTP call
embedding_api_key	(optional)	Sent as Authorization: Bearer <key>; supports ${...} refs

How it works

A lookup walks a two-tier path before any provider call happens:

Exact-match fast tier

Before calling the embedding service, the cache checks an exact-match tier (prompt text hash). If the exact prompt is cached and within TTL, the response is returned immediately without an embedding call. This is the common case for repeated identical prompts and adds zero latency overhead beyond a hash lookup.

Semantic similarity lookup

1. On a non-streaming AI request, AFTER guardrails and BEFORE the provider call, the gateway sends the prompt text to the configured embedding service and receives a vector embedding.
2. The embedding is searched against the HNSW ANN index for the nearest cached entry (k=1).
3. If the nearest entry's cosine similarity is >= the threshold, the entry is within TTL, and the model alias matches, the cached response is returned immediately (no provider call, no token spend).
4. On a cache miss, the request proceeds to the provider. After a successful response, the prompt embedding and response are stored in the cache (fire-and-forget -- the store never blocks the response path).

Streaming cache

Streaming responses (stream: true) are also cached. The gateway collects streaming SSE frames up to 1 MiB in a bounded tee buffer as they are forwarded to the client. If the stream completes within the cap, the collected frames are stored in the cache. On a subsequent cache hit, the stored frames are replayed as text/event-stream -- the client sees the same streaming experience as the original call.

If the stream exceeds the 1 MiB cap, it continues to the client normally but is not cached. This is a graceful degradation: large streams are served correctly, just not cached for replay.

Eviction

When the cache reaches max_entries, the least-recently-accessed entry is evicted (LRU policy). Each cache entry tracks its last-access timestamp; eviction scans for the oldest entry. This is more predictable than a wholesale reset and preserves recently-used entries when the cache is full.

Fail-open behavior

If the embedding service is unavailable or times out, the cache fails open: the request proceeds to the provider as a cache miss. The embedding error is logged but never blocks the request path.

Limitations

• Streaming cache size limit: streaming responses exceeding 1 MiB are served to the client but not cached.
• External dependency: the embedding service must be reachable and responsive for semantic similarity lookups. The exact-match tier works without the embedding service. If the embedding call fails or times out, the cache fails open (the request proceeds to the provider as a miss).
• Per-model: the cache is keyed by model alias; the same prompt with different models does not cross-hit.
• No persistence: the HNSW index and cached entries are in-memory only; they are lost on restart. The cache persists across config reloads (the index and entries survive; config updates in place).
• Chat only: the semantic cache applies to chat-completions endpoints. Non-chat endpoints (embeddings, images, audio, moderation) are served via passthrough and are not cached.

Cost savings

The primary value of semantic caching is provider-call avoidance. For a workload with N requests and M cache hits, the provider is called N - M times. The cost savings are:

• Token cost: zero tokens spent on cache hits (the response is served from cache, not from the provider).
• Latency: cache hits skip the provider round-trip (the embedding call is typically faster than a full LLM completion).
• Embedding cost: each cache lookup and store makes one embedding API call. The embedding cost should be significantly cheaper than the LLM completion cost for the savings to be positive.

The dwara_ai_semantic_cache_hits_total{model} and dwara_ai_semantic_cache_misses_total{model} metrics quantify the hit rate; combine with dwara_ai_cost_micros_total to measure the dollar savings.

Runnable demo

Run this feature against a live gateway: demos/07-ai-gateway/ (test script: test-05-semantic-caching.sh) in the repository. The category README covers prerequisites and teardown.

See also

• AI gateway