Module ai_proxy 

The AI proxy action (DW-075): the dataplane half of the provider-adapter pack — extended by DW-076 with routing and failover.

Runs when a route’s action is ai: parse the client’s OpenAI chat-completions body, resolve the model alias through the generation’s AiRuntime, translate via the provider’s ProviderAdapter, place the call through the provider’s UPSTREAM (pooling, TLS, timeouts, breaker, health — the standard machinery), and translate the response back. All failures answer in the OpenAI error shape so standard client SDKs parse them; the gateway’s request id rides along inside error.request_id.

Routing and failover (DW-076)

AiRuntime::route turns the alias into an ordered candidate list: [primary, failover...] for a chained alias, or exactly one deterministically-picked canary version for a split alias (see ai::routing). The action walks the candidates and advances on RETRYABLE outcomes — a 429, a 5xx, a transport error, or a dialect-specific translation rejection — and treats everything else as final. Failover is invisible to the client by construction: a provider response is read and translated WHOLE before any byte reaches the client, so a failing candidate emits nothing client-visible. Same-provider retries are deliberately not the chain’s job — the provider’s upstream breaker owns those; the chain only ever moves to a DIFFERENT provider/model pair. Usage and request metrics attribute to the provider and canary version that actually served; the access record follows the serving provider.

Body bounds: an AI request must be read whole to translate it (the zero-buffering rule yields here by necessity, opt-in by choosing the action), and both directions are capped INCREMENTALLY — the read aborts as soon as the cap is crossed, never after buffering the whole body — at MAX_AI_REQUEST_BYTES inbound and MAX_AI_PROVIDER_RESPONSE_BYTES from the provider, so a hostile or runaway peer cannot turn the translation into an unbounded buffer.

Streaming (DW-077): stream: true requests stream back to the client as text/event-stream, translated frame-by-frame with ZERO added buffering (AiStreamBody + StreamTranslator): each complete provider frame becomes client frames in the same poll. Token counts are provider-reported only (locked decision) and accumulate mid-stream into one terminal usage chunk; the gateway owns the data: [DONE] terminator. The failover chain applies until the streaming response is returned (the commit point); after the first forwarded frame a provider abort ends the stream cleanly with an error chunk — already-forwarded content stands.

Structs

AiStreamBody

The zero-buffer AI streaming body (DW-077): wraps the provider’s upstream body, translates each COMPLETE provider SSE frame into OpenAI-shaped client frames as the bytes arrive, and forwards them in the same poll — nothing waits for the stream to finish. Usage events accumulate (provider-reported only) and are emitted as one terminal chunk; the gateway owns the data: [DONE] terminator. A provider body error after forwarding becomes a terminal error chunk (already-forwarded content stands), never a reset.

Constants

MAX_AI_ERROR_BYTES

Provider error body cap: 1 MiB.

MAX_AI_PROVIDER_RESPONSE_BYTES

Provider response body cap: 32 MiB (non-streaming completions of extreme length). Provider ERROR bodies are capped tighter at MAX_AI_ERROR_BYTES — they are never legitimately large.

MAX_AI_REQUEST_BYTES

Inbound AI request body cap: 16 MiB, generous against long-context requests while bounding the translation buffer.