Module observability

Expand description

Observability substrate (DW-021, feature analysis sections 4.17 and 4.19): tracing spans per request phase, structured JSON access logs, Prometheus metrics, request IDs, and the unified error envelope.

§Tracing and spans

tracing (MIT) is the substrate. Every request opens a root span named request carrying request_id, method, path (WITHOUT the query string — redaction: query strings carry tokens), consumer, route, and listener fields, plus child spans at each phase: authn, authz, ratelimit, admission, upstream_pick (emitted inside the upstream handle where the pick actually happens), and one upstream_attempt span per send. The binary installs a tracing-subscriber JSON formatter over STDOUT filtered by DWARA_LOG (RUST_LOG syntax, default dwara=info).

§OTLP decision

The OTLP exporter (opentelemetry + opentelemetry-otlp) was evaluated and NOT added in v1: the crates are heavy (their own HTTP transport, tonic/protonic codegen) against a musl <25MB binary-size budget (DW-026) and a compute-conscious CI. The span STRUCTURE ships now and is proven by an in-process span-capture test (one trace shows all phases). Since #126 the exporter exists behind the default-off otlp cargo FEATURE on dwara-bin (the env var DWARA_OTLP_ENDPOINT is a binary-level knob and the tracing subscriber the exporter must hook lives in the bin; this module stays dependency-light by design). Default build: DWARA_OTLP_ENDPOINT remains RESERVED but inert; feature build: the spans defined here export over http/protobuf.

§Access logs

One event at target dwara::access per completed request (rendered as one JSON line by the binary’s subscriber): ts (subscriber timestamp), request_id, method, path (sanitized: no query), status, duration_ms, route, consumer, upstream, endpoint, attempts, and the rate_limited / broken / shed flags. bytes_in/bytes_out are deliberately omitted: the proxy path is zero-buffering by design (DW-009) and body sizes are not cheaply available for streamed bodies — recorded here as an honest omission rather than a wrong number. Sampling: DWARA_ACCESS_LOG_ SAMPLE (0.0-1.0, default 1.0); responses with status >= 500 are ALWAYS logged regardless of sampling (documented rule).

§Metrics

Prometheus text format on /metrics, reserved on every HTTP(S) listener exactly like /healthz (a configured route matching it is shadowed — same accepted v1 behavior). The prometheus crate (MIT, default features off: text format only, no protobuf exporter) backs the registry. Metric families:

requests_total{route,listener,status_class} counter
request_duration_seconds{route} histogram
upstream_attempts_total{upstream,endpoint,status_class} counter
retries_total{upstream} counter
dwara_hedge_sent_total{upstream} counter (DW-063) — speculative hedge copies sent after the tail-latency timer fired
dwara_mirror_sent_total{upstream} counter (DW-062) — shadow traffic mirror requests sent to a mirror upstream
rate_limited_total{route} counter
dwara_rate_limiter_adaptive_factor{policy} gauge (DW-089) — the current adaptive rate-limit factor per policy (1.0 = configured rate; < 1.0 tightens, > 1.0 relaxes; bounded [min, max])
dwara_rate_limiter_origin_signal_total{policy,signal} counter (DW-089) — upstream origin signals received (retry_after), by policy and signal (both config-bounded)
dwara_rate_limiter_adaptive_tightened_total{policy} counter (DW-089) — adaptive factor decreases, by policy
dwara_rate_limiter_adaptive_relaxed_total{policy} counter (DW-089) — adaptive factor increases, by policy
dwara_canary_promotions_total{group} counter (DW-091) — canary promotions, by group (service name or AI alias)
dwara_canary_rollbacks_total{group} counter (DW-091) — canary rollbacks (including severe-regression immediate rollbacks to 0)
dwara_canary_weight{group} gauge (DW-091) — current canary weight, by group (the total weight stays constant)
shed_total{priority} counter
dwara_policy_dry_run_total{phase,route} counter (DW-041) — requests a dry-run (monitor) policy would have rejected, by phase (route_limits, authz, rate_limit, load_shed) and route (unrouted for pre-404 listener/global policies); the request was ALLOWED. Together with the dwara::policy warn events (phase, would-be status, reason, route, consumer, request id) this family IS the dry-run report — no separate endpoint.
breaker_state{upstream} gauge (0 closed, 1 open, 2 half-open)
endpoint_health{upstream,endpoint} gauge (1 available, 0 ejected)
upstream_fail_open_picks{upstream} gauge (scrape-time snapshot of the balancer’s monotonic fail-open counter; a gauge rather than a counter so the hot pick path stays free of registry coupling)
active_requests gauge
config_generation gauge
jwks_refresh_total{provider} counter
dwara_rate_limiter_evictions_total gauge (scrape-time snapshot of the rate-limit engine’s monotonic eviction counter, aggregated over every compiled rule; resets when a reload rebuilds the engine; a gauge rather than a counter so the hot check path stays free of registry coupling)
dwara_rate_limiter_live_keys gauge (live per-key cells across every compiled rule, approximate, bounded by the sharded store cap; aggregate and unlabeled — cardinality is never per key)
dwara_webhook_events_total{kind,outcome} counter (DW-044): webhook deliveries by event kind and outcome — delivered (2xx on some attempt), failed (retries exhausted / non-transient answer / budget spent), dropped (never tried: envelope over the byte cap, or delivery concurrency saturated). Both labels are closed sets, so cardinality is bounded by construction.
dwara_events_dropped_total / dwara_events_emitted_total gauges (DW-044, scrape-time snapshots of the event bus’s monotonic counters — gauges rather than counters so the emit path stays a plain atomic with zero registry coupling, the same model as the rate-limiter eviction gauge): events dropped at EMIT time (full queue / no deliverer) and events handed to the queue.
dwara_access_records_streamed_total{outcome} counter (DW-121): access records counted per flushed-batch delivery outcome — delivered / failed (tried and not taken) / dropped (per-record byte cap) — a closed three-value label set, counted by the stream flusher, never the request path.
dwara_access_records_offered_total / dwara_access_records_dropped_total gauges (DW-121, scrape-time snapshots of the record stream’s monotonic counters, the same gauge-not-counter model as the event bus): records offered at request completion and records dropped at OFFER time (channel full / stream disabled).
dwara_quota_denied_total{consumer,budget} counter (DW-033) — requests refused by a consumer request budget (429), by consumer (the quota-configured set, config-bounded) and binding budget (daily/monthly, a closed set). Distinct from rate_limited_total: budgets are not rates.
dwara_quota_used{consumer,budget} / dwara_quota_limit{consumer, budget} gauges (DW-033, scrape-time snapshots of the state store’s quota counters and the config caps — the same snapshot gauge model as the rate-limiter live-keys gauge).
dwara_admission_queued_total{outcome} counter (DW-053) — admission queue outcomes by closed set admitted (a permit was acquired after queueing), timeout (the queue timeout expired; shed with 503), queue_full (the queue was at capacity; shed immediately with 503).
dwara_admission_queue_depth gauge (DW-053) — current number of requests waiting in the admission queue for a concurrency permit.
dwara_license_status gauge (DW-032) — 0 = no license (OSS mode), 1 = valid, 2 = expired within grace period, 3 = expired past grace period (degraded to OSS). Set at startup and on reload; no labels.
dwara_dns_discovery_endpoints{upstream} gauge (DW-042) — current resolved endpoint count for DNS-discovered upstreams. The upstream label is config-bounded; cardinality is never per resolved address.
dwara_dns_discovery_refresh_total{upstream} counter (DW-042) — DNS discovery refresh attempts, by upstream (config-bounded label).
dwara_dns_discovery_refresh_failures_total{upstream} counter (DW-042) — DNS discovery refresh failures, by upstream.
dwara_mcp_sessions_total{state} counter (DW-087) — MCP session lifecycle transitions, by state (initialized, closed, expired). A closed three-value label set, so cardinality is three series total.
dwara_mcp_tool_calls_total{tool,status} counter (DW-087) — MCP tool calls, by tool (config-bounded) and status (success, error, denied). No consumer label (cardinality rule).
dwara_mcp_tool_duration_seconds{tool} histogram (DW-087) — MCP tool call duration (authz check through upstream response), by tool (config-bounded label).
dwara_a2a_requests_total{agent,outcome} counter (DW-114) — A2A (agent-to-agent) requests, by agent (config-bounded) and outcome (success, error, stubbed). No consumer label (cardinality rule).
dwara_a2a_sessions_total{state} counter (DW-114) — A2A session lifecycle transitions, by state (initialized, closed, expired). A closed three-value label set (mirrors MCP), so cardinality is three series total.
dwara_tls_pq_handshakes_total{result} counter (DW-105) — post-quantum TLS handshake outcomes, by result (success, fallback, disabled). A closed three-value label set, so cardinality is three series total.
dwara_graphql_requests_total{route,outcome} counter (DW-099) — GraphQL check outcomes, by route (config-bounded) and outcome (allowed, depth_exceeded, complexity_exceeded, persisted_query_required — a closed four-value set).
dwara_graphql_depth{route} gauge (DW-099) — last query depth for a GraphQL-configured route (config-bounded label).
dwara_nano_service_requests_total{route,outcome} counter (DW-106) — nano-service (WASM route handler) request outcomes, by route (config-bounded) and outcome (success, error, timeout — a closed three-value set). No consumer label (cardinality rule).
dwara_nano_service_duration_seconds{route} histogram (DW-106) — nano-service handle call duration, by route (config-bounded label).
dwara_plugin_failures_total{name,reason} counter (DW-157) — plugin dispatch failures, by plugin name (config-bounded) and reason (crashed, disabled, not_loaded, not_registered, instantiate_failed, trap, body_too_large, response_stream_ended — a closed set). One increment per request the plugin chain answered with an error (a plugin short-circuit via send_http_response is a plugin DECISION, not a failure, and is not counted; the access-log plugin_short_circuit flag marks those).
dwara_plugin_total{state} gauge (DW-158) — plugins loaded per lifecycle state (healthy, crashed, disabled, not_loaded, not_registered — a closed five-value set over the generation’s declared plugins, the same enumeration GET /plugins serves). Refreshed at publish time (every config generation); state changes that happen between publishes (a crash counted at request time) land on the next publish or are visible per-request in dwara_plugin_failures_total.
dwara_graphql_complexity{route} gauge (DW-099) — last query complexity for a GraphQL-configured route (config-bounded label).
dwara_l4_connections_total{listener,protocol} counter (DW-103) — L4 connections proxied, by listener (config-bounded) and protocol (tcp, udp — a closed two-value set). Counted once per accepted L4 connection (before the splice is established).
dwara_l4_connection_duration_seconds{listener} histogram (DW-103) — L4 connection duration (splice establishment through close), by listener (config-bounded label).
dwara_ai_request_duration_seconds{provider,route} histogram (AI-07) — AI request latency (request parse through response translation), by provider and route (both config-bounded). Observed once per completed AI request (success and provider/translation errors alike; client-side rejections never reach a provider).
dwara_ai_tokens_per_request{provider,model,kind} histogram (AI-07) — distribution of provider-reported tokens per request, by provider (config-bounded), provider_model (config-bounded), and kind (the closed set prompt | completion). Provider-reported only; the gateway never estimates. No consumer label (cardinality).

§OTel GenAI semantic conventions (AI-07)

AI request spans carry the OpenTelemetry GenAI semantic-convention gen_ai.* attributes (https://opentelemetry.io/docs/specs/semconv/gen-ai/): gen_ai.system (the provider kind), gen_ai.request.model (the client alias), gen_ai.response.model (the provider model that served), gen_ai.request.{max_tokens,temperature,top_p}, and the provider-reported gen_ai.usage.{prompt_tokens,completion_tokens, total_tokens}, gen_ai.response.finish_reasons, and gen_ai.response.id. The helpers record_gen_ai_request_attrs and record_gen_ai_response_attrs record them onto a span created with every gen_ai.* field declared Empty (the span lives in dataplane::ai_proxy::serve_ai).

§Error envelope (section 4.19)

Every gateway-generated non-success body — including the reserved /healthz//readyz responses, which are aligned to the same shape — is {"error":{"code","message","request_id"}}. Messages never leak upstream internals: classification strings only (“upstream unavailable”, not hyper error text).

§Request IDs

An inbound X-Request-Id is respected when it is printable ASCII of at most 128 bytes (anything else is replaced — a hostile ID must not smuggle control bytes into logs); otherwise a process-unique ID is generated (wall-clock nanoseconds + per-process counter — cheaper than a UUID, no extra dependency). The ID is echoed on the response as X-Request-Id and available to every span and log line.

§Redaction rules (hard requirements)

Paths are logged WITHOUT query strings.
No Authorization / Proxy-Authorization / Cookie / Set-Cookie / X-API-Key VALUES are ever emitted by this module or the tracing calls it fronts; field lists below are exhaustive.
JWKS bodies, credentials, and keys are never logged.

Structs§

AccessRecord: The accumulated access-log record for one request. The proxy fills fields as the request progresses; finish stamps status/duration.
ListenerLabel: Extension inserted by the listener frontend (dwara-bin) naming the listener that accepted the request; absent when proxy::handle is driven directly (tests), in which case the label is “unknown”.
Observability: Per-dataplane observability state: the metric families plus the access-log sampling knob. One instance per DataPlane, so parallel tests never share a registry (duplicate-metric registration errors).
SloTargets: Plain per-route SLO targets. Primitives on purpose: observability depends on nothing, so the config types are converted at the call site (dataplane refresh).

Constants§

X_CORRELATION_ID: Request header carrying the business correlation ID (DW-093): inbound respected (falls back to X-Request-Id), outbound echoed on the response so a client can tie a reply to its journey across multiple gateway requests.
X_REQUEST_ID: Request header carrying the correlation ID (inbound respected, outbound always set by the gateway).

Functions§

emit_access: Emit one access-log event (one JSON line under the binary’s subscriber). The field list is exhaustive and redacted by construction: no headers, no query string, no credentials. bytes_in/bytes_out (PERF-12, #209): bytes_in is the declared Content-Length (0 for chunked); bytes_out is the frame-counted response body size (read from the atomic the CountingBody wrapper updated as frames streamed to the client).
envelope_body: The unified error body (section 4.19): a JSON object an operator can grep and a client can correlate. code is a stable machine token, message a human string with no upstream internals, request_id ties the response to the trace/access log.
generate_request_id: Public for testing the request-id generation contract.
record_gen_ai_request_attrs: Record the request-side OTel GenAI semantic-convention attributes (AI-07) onto an AI span. The span MUST have been created with every gen_ai.* field declared as tracing::field::Empty (see gen_ai_span_fields); Span::record only fills fields the span’s metadata already declared. Request-side fields are always available once the canonical crate::ai::types::ChatRequest is parsed; the system is the serving provider’s kind (crate::config::ai::AiProviderKind::gen_ai_system).
record_gen_ai_response_attrs: Record the response-side OTel GenAI semantic-convention attributes (AI-07) onto an AI span. Each argument is recorded only when present (provider-reported only; the gateway never estimates). finish_reasons is the per-choice finish-reason token list (one entry per choice).
resolve_correlation_id: Resolve the request’s business correlation ID (DW-093): a valid inbound X-Correlation-Id verbatim, else the already-resolved request_id (so every request carries a non-empty correlation handle for the journey/funnel query even when the client did not send one). The validity gate mirrors valid_inbound_request_id (printable ASCII, at most 128 bytes) so hostile IDs cannot smuggle control characters into the analytics store.
resolve_request_id: Resolve the request’s correlation ID: a valid inbound X-Request-Id verbatim, else a generated one (req- + hex wall-clock nanoseconds + hex process counter; unique per process, no dependency pulled in for randomness — request IDs are correlation handles, not secrets).
stamp_correlation_id: Echo the correlation ID onto a response (insert, never append: exactly one X-Correlation-Id, the gateway’s resolved one).
stamp_request_id: Echo a request ID onto a response (insert, never append: exactly one X-Request-Id, the gateway’s resolved one).
status_class: Prometheus status-class label (“2xx”, “5xx”, …).
valid_inbound_request_id: Validate an inbound X-Request-Id value: printable ASCII (0x20-0x7E), at most 128 bytes. Anything else (control bytes, UTF-8 multibyte, overlong) is rejected so hostile IDs cannot smuggle control characters into logs.