Observability

Logs, request IDs, metrics, and error bodies share one goal: an operator can correlate any client complaint to exactly one gateway request.

Logs

The binary emits structured JSON on stdout (one JSON object per log line, easy to parse by log aggregators), filtered by DWARA_LOG (RUST_LOG syntax — a filter syntax like dwara=info,debug, default dwara=info). One access-log line per completed request carries: timestamp, request_id, method, path, status, duration_ms, route, consumer, upstream, endpoint, attempts, and rate_limited/broken/shed flags. route is unrouted for 404s and reserved paths; consumer is anonymous without authentication.

DWARA_ACCESS_LOG_SAMPLE (0.0–1.0, default 1.0) sets the fraction of non-error lines emitted — responses with status ≥ 500 are always logged regardless of sampling, and an invalid value falls back to 1.0 so a broken knob can never silence error visibility.

Redaction is exhaustive: logged paths never include the query string, and no credential material (Authorization, Proxy-Authorization, Cookie, Set-Cookie, X-API-Key values, keys, JWKS bodies (a JSON document of signing keys)) is ever logged.

Request IDs

An inbound X-Request-Id is respected when it's printable ASCII of at most 128 bytes; otherwise Dwara generates one (req-<hex nanoseconds>-<counter>). The resolved ID is echoed on every response as X-Request-Id and appears in every log line, span, and error body — use it as the single correlation key across client reports, gateway logs, and (if enabled) traces.

Metrics

/metrics serves Prometheus text format (a plain-text metrics format Prometheus scrapes — Prometheus is a monitoring system that scrapes metrics endpoints), reserved on every terminate and cleartext listener just like /healthz (see Operations).

Metric	Type	Labels
requests_total	counter (a value that only goes up)	route, listener, status_class
request_duration_seconds	histogram (counts observations into buckets for percentiles)	route
upstream_attempts_total	counter	upstream, endpoint, status_class
retries_total	counter	upstream
rate_limited_total	counter	route
shed_total	counter	priority
dwara_policy_dry_run_total	counter	phase, route
breaker_state	gauge (0/1/2 = closed/open/half-open; a gauge is a value that can go up or down, like a queue depth)	upstream
endpoint_health	gauge (1/0 = available/ejected)	upstream, endpoint
upstream_fail_open_picks	gauge	upstream
active_requests	gauge	—
config_generation	gauge	—
jwks_refresh_total	counter	provider
dwara_rate_limiter_evictions_total	gauge	—
dwara_rate_limiter_live_keys	gauge	—
dwara_webhook_events_total	counter	kind, outcome
dwara_events_dropped_total	gauge	—
dwara_events_emitted_total	gauge	—
dwara_slo_burn_rate	gauge	route, objective, window
dwara_slo_target	gauge	route, objective
dwara_adaptive_factor	gauge	policy
dwara_adaptive_min_factor	gauge	policy
dwara_adaptive_max_factor	gauge	policy
dwara_adaptive_origin_signal_total	counter	policy, signal
dwara_adaptive_tightened_total	counter	policy
dwara_adaptive_relaxed_total	counter	policy
dwara_ai_requests_total	counter	provider, route, outcome, version
dwara_ai_tokens_total	counter	provider, kind, version
dwara_ai_request_duration_seconds	histogram	provider, route
dwara_ai_tokens_per_request	histogram	provider, model, kind
dwara_ai_first_token_seconds	histogram	provider
dwara_ai_stream_chunks_total	counter	provider
dwara_ai_stream_duration_seconds	histogram	provider
dwara_ai_budget_denied_total	counter	kind
dwara_ai_semantic_cache_hits_total	counter	model
dwara_ai_semantic_cache_misses_total	counter	model
dwara_plugin_total	gauge	state
dwara_plugin_failures_total	counter	name, reason

Label cardinality is deliberately config-bounded — there is no consumer-name label anywhere, and the rate-limiter series are aggregate/unlabeled even though the engine tracks many per-key cells internally. The SLO series (dwara_slo_*) exist only for routes carrying an slo block: dwara_slo_burn_rate is the bad-request fraction over a 5m or 1h process-local sliding window divided by the allowed fraction — 1.0 consumes the error budget at exactly the allowed rate, and the dashboard's SLO panel draws the 6x (slow burn) and 14.4x (fast burn) alerting lines. dwara_policy_dry_run_total counts requests a dry-run policy would have rejected, by phase (route_limits, authz, rate_limit, load_shed) and route — its log counterpart is the dwara::policy warn event. A starter dashboard ships at grafana/dwara-overview.json; import it in Grafana via Dashboards → New → Import and point it at a Prometheus instance scraping the gateway's /metrics.

Plugin metrics

The two plugin families describe the same runtime from two angles — stock (how many plugins are in each state) and flow (how often plugin dispatch fails):

• dwara_plugin_total{state} — a gauge counting plugins per lifecycle state: healthy, crashed (the .wasm could not be read or compiled; its routes answer 500 plugin_unavailable), disabled, not_loaded (declared but absent from the running runtime), and not_registered (a native: filter name with no registered implementation). Refreshed on every config publish; the per-plugin detail (digest, error text, affected routes) is in GET /plugins and the plugins section of dwara-cli status.
• dwara_plugin_failures_total{name,reason} — a counter incremented once per request the plugin chain answered with an error, by plugin name (config-bounded) and reason (crashed, disabled, not_loaded, not_registered, instantiate_failed, trap, body_too_large, response_stream_ended). A plugin that answers a request itself (send_http_response) is making a decision, not failing — those requests are marked plugin_short_circuit in the access log instead. See Plugin lifecycle.

Error envelope

Every gateway-generated non-success response body (including the reserved /healthz//readyz paths) uses one shape:

{"error":{"code":"no_route","message":"no route","request_id":"req-..."}}

code is a stable machine token, message is a human string that never leaks upstream internals, and request_id ties the response to logs and traces.

Distributed tracing (OTLP)

Trace export over OTLP (OpenTelemetry Protocol — the standard for exporting traces) requires a build with the otlp cargo feature (off by default to keep the release binary small):

cargo build -p dwara-bin

Point it at a collector with DWARA_OTLP_ENDPOINT (e.g. http://collector:4318 — /v1/traces is appended, or a full .../v1/traces URL is accepted as-is). In a default build, this environment variable is reserved but inert — setting it has no effect unless the binary was built with the feature.

AI traces: GenAI semantic conventions

AI route requests carry OpenTelemetry GenAI semantic convention attributes on their span, so AI traffic is observable in standard OTel tooling (Jaeger, Tempo, Grafana, Datadog) without custom configuration. The attributes include:

• gen_ai.system — the provider system (openai, anthropic, gemini, azure_openai, bedrock).
• gen_ai.request.model — the client-facing model alias.
• gen_ai.request.max_tokens, gen_ai.request.temperature, gen_ai.request.top_p — request parameters, when present.
• gen_ai.response.model — the provider model that served.
• gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens, gen_ai.usage.total_tokens — provider-reported token counts.
• gen_ai.response.finish_reasons — the finish reasons.
• gen_ai.response.id — the provider response ID.

These attributes appear on the AI request span (a child of the request root span) in any OTel-compatible trace viewer.

SLOs and error budgets

Routes can declare service-level objectives (SLOs — a Service Level Objective is a target like 99.9% availability); the gateway exports them as burn-rate metrics (how fast you are consuming the error budget your SLO allows) for multiwindow alerting:

routes:
  - name: checkout
    # ...
    slo:
      availability: 99.9        # percent of requests that must not be a 5xx
      latency_ms: 250           # optional latency objective threshold
      latency_target: 99        # percent of requests within latency_ms (default 99)

dwara_slo_burn_rate{route,objective,window} is the error-budget consumption rate — the bad-request fraction over a 5m or 1h sliding window divided by the allowed fraction. availability counts a request bad only when the GATEWAY answers 5xx (client errors are the caller's policy, not availability); latency counts a request bad when its end-to-end duration exceeds latency_ms. Alert on the standard pair: 14.4x over 1h pages (the 28-day budget would burn in ~2 days), 6x over 1h is the slow-burn signal. The windows are process-local and start empty at boot; the shipped dashboard's "SLO burn rate" panel draws both the 6x and 14.4x thresholds. Routes without an slo block export nothing.

tokio-console

For development and debugging, Dwara can expose tokio-console diagnostics for its async task runtime. tokio-console is an optional async task inspector -- connect with the tokio-console TUI to see live task polls, waker stats, and task lifetimes, which is invaluable for diagnosing stuck tasks, waker churn, or scheduler starvation.

This is controlled by the DWARA_TOKIO_CONSOLE_ADDR environment variable (default OFF) and is off at runtime unless explicitly enabled:

cargo build -p dwara-bin

Enable it at runtime with the DWARA_TOKIO_CONSOLE_ADDR environment variable, which sets the address the console server binds to:

DWARA_TOKIO_CONSOLE_ADDR=127.0.0.1:6666 ./dwara run

Then connect from another terminal:

tokio-console http://127.0.0.1:6666

WARNING

tokio-console adds per-poll instrumentation overhead. It is intended for development and debugging only -- never enable it in a production build serving live traffic.

When DWARA_TOKIO_CONSOLE_ADDR is not set, DWARA_TOKIO_CONSOLE_ADDR is reserved but inert -- setting it has no effect. When the feature is on but the variable is unset, the console server does not bind.

Runnable demo

Run the signals against a live gateway: demos/06-observability/ in the repository. Test scripts assert the /metrics request counter (test-01-metrics.sh), X-Request-Id echoed on every response (test-02-request-id.sh), access logs landing in the gateway's container output (test-03-access-logs.sh), and dwara_slo_* gauges for a route carrying an SLO (test-11-slo.sh). The category README covers prerequisites and teardown.