Module semantic_cache 

AI semantic caching (DW-083): embedding-similarity cache for AI prompts. A paraphrased prompt within the similarity threshold returns the cached response (no provider call, no token spend).

Uses an external embedding service (OpenAI-compatible /v1/embeddings API) to vectorize prompts and hnsw_rs (pure Rust HNSW) for approximate nearest neighbor search.

Feature-gated behind the semantic_cache cargo feature. Without it, the module compiles to an inert placeholder that always returns None (the config is accepted but the cache is a no-op).

PERF-02 enhancements

• Exact-match fast tier: before calling the embedding service, a HashMap<prompt_hash, entry_id> checks for an exact text match. A hit skips the embedding call entirely (the common case for repeated identical prompts — RAG traffic, retry storms, etc.).
• LRU eviction: when the cache reaches max_entries, the least recently accessed entry is evicted (not a wholesale reset). Each entry carries a last_accessed_ms timestamp; the eviction scan is O(n) but runs once per insert, so the amortized cost is O(1).
• Streaming support: streaming responses (SSE frame sequences) are cached and replayed. On a cache miss, the stream is TEE’d: frames are forwarded to the client AND collected in a bounded buffer; on stream completion, the collected frames are stored. On a cache hit, the cached frames are replayed as an SSE response. The tee buffer is bounded (stream_cache_max_bytes, default 1 MiB): a stream exceeding the cap is not cached (the response is still forwarded to the client unmodified).

Lifecycle

The engine is constructed once at startup and stored on the dataplane behind an ArcSwapOption. It PERSISTS across reloads: the HNSW index and cached entries survive config refreshes, and a reload updates the config in place via update_config (so a threshold or TTL change applies to the next lookup with no cache reset).

Request path

The LOOKUP runs in serve_ai AFTER guardrails (the prompt may have been redacted) and BEFORE model routing + the provider call. A hit returns the cached response (JSON for non-streaming, SSE frames for streaming) with no provider call. The lookup is async (it may make an HTTP call to the embedding service) — acceptable because it runs in the already-async serve_ai. The STORE is fire-and-forget (a spawned task): the embedding call and the HNSW insert happen AFTER the response is sent, so they never block the response path.

Structs

SemanticCacheEngine

The semantic cache engine (DW-083). Constructed once at startup and stored on the dataplane (persists across reloads — the HNSW index and cached entries survive config refreshes). Config is updated in place via update_config.

Enums

CachedResponse

What a cache lookup returns (PERF-02): the cached response, either a JSON body (non-streaming) or SSE frames (streaming).