Struct Observability 

pub struct Observability { /* private fields */ }

Per-dataplane observability state: the metric families plus the access-log sampling knob. One instance per DataPlane, so parallel tests never share a registry (duplicate-metric registration errors).

Implementations

impl Observability

pub fn from_env() -> Self

Build with DWARA_ACCESS_LOG_SAMPLE applied (default 1.0; values outside [0.0, 1.0] fall back to 1.0 — a malformed knob must never silence logs).

pub fn new() -> Self

pub fn set_access_sample(&self, rate: f64)

Access-log sample rate in [0.0, 1.0]. Env-driven at construction (DWARA_ACCESS_LOG_SAMPLE); settable directly for tests.

pub fn access_sample(&self) -> f64

pub fn should_log_access(&self, status: u16) -> bool

Whether this request’s access log line is emitted. Errors (status >= 500) are ALWAYS logged; everything else follows the configured sample rate.

pub fn record_request( &self, route: &str, listener: &str, status: u16, elapsed: Duration, )

Count and observe one completed request.

pub fn record_upstream_attempt( &self, upstream: &str, endpoint: &str, status: u16, )

Count one upstream send attempt (endpoint = “unpicked” when the dispatch never resolved an endpoint).

pub fn record_retry(&self, upstream: &str)

Count one retry (a send attempt after the first).

pub fn record_hedge_sent(&self, upstream: &str)

Count one hedge copy sent (DW-063): a speculative duplicate the hedge timer fired. Called once per copy, not once per request.

pub fn record_mirror_sent(&self, upstream: &str)

Count one mirror (shadow traffic) request sent (DW-062).

pub fn record_rate_limited(&self, route: &str)

Count one rate-limit denial (429).

pub fn set_adaptive_factor(&self, policy: &str, factor: f64)

Set the current adaptive rate-limit factor for a policy (DW-089). factor is the bounded [min_factor, max_factor] value the controller computed on its last record_outcome.

pub fn record_adaptive_origin_signal(&self, policy: &str, signal: &str)

Count one upstream origin signal received by the adaptive controller (DW-089). signal is the closed set retry_after.

pub fn record_adaptive_tightened(&self, policy: &str)

Count one adaptive factor decrease / tighten (DW-089).

pub fn record_adaptive_relaxed(&self, policy: &str)

Count one adaptive factor increase / relax (DW-089).

pub fn record_canary_promotion(&self, group: &str)

Count one canary promotion (DW-091), by group (config-bounded: service name or AI alias).

pub fn record_canary_rollback(&self, group: &str)

Count one canary rollback (DW-091), by group (config-bounded: service name or AI alias). Includes severe-regression immediate rollbacks to 0.

pub fn set_canary_weight(&self, group: &str, weight: f64)

Set the current canary weight for a group (DW-091). Set on every applied adjustment; the total weight stays constant.

pub fn record_mesh_connection(&self, mode: &str, direction: &str)

Count one sidecar connection proxied (DW-107), by mode and direction. mode is the closed set sidecar; direction is the closed set inbound, outbound. Inbound: terminate mTLS, apply policies, forward to the local app. Outbound: apply policies, wrap in mTLS, dial the remote sidecar.

pub fn record_spiffe_svid_refresh(&self, result: &str)

Count one SPIFFE SVID refresh attempt (DW-107), by result. result is the closed set success, error. An error keeps serving the previous SVID until it expires.

pub fn set_spiffe_svid_expiry_seconds(&self, seconds: i64)

Set the seconds-until-expiry gauge for the current SPIFFE SVID (DW-107). Set on every successful refresh; counts down to 0 as the SVID nears expiry.

pub fn record_quota_denied(&self, consumer: &str, budget: &str)

Count one quota denial (429, DW-033) by consumer and binding budget. Deliberately NOT the rate_limited_total family: budgets and rates are separate mechanisms (the issue’s headline distinction); a quota wall is not a rate spike.

pub fn set_quota_used(&self, consumer: &str, budget: &str, used: i64)

Set a budget’s current-window usage gauge (DW-033, scrape-time snapshot; budget is daily or monthly).

pub fn set_quota_limit(&self, consumer: &str, budget: &str, limit: i64)

Set a budget’s configured cap gauge (DW-033, scrape-time).

pub fn record_shed(&self, priority: u8)

Count one gateway-cap shed (503), by priority class.

pub fn record_admission_queued(&self, outcome: &str)

Count one admission queue outcome (DW-053). outcome is one of the closed set admitted (a permit was acquired after queueing), timeout (the queue timeout expired; the request was shed with 503), queue_full (the queue was at capacity; the request was shed immediately with 503).

pub fn set_admission_queue_depth(&self, depth: i64)

Set the current admission queue depth gauge (DW-053). Called from the dataplane’s atomic-backed depth counter (the hot path stays free of registry coupling — the same gauge model as the coalescing-waiters gauge).

pub fn record_waf(&self, route: &str, filter: &str, outcome: &str)

Count one WAF-lite inspection outcome (DW-051). route is the matched route name; filter is the filter category that matched (sqli, xss, path_traversal) or all for the passed outcome (no match); outcome is blocked (403 returned), logged (dry-run match, request allowed), or passed (no match). All three labels are config-bounded (route names and the closed filter/outcome sets).

pub fn record_anomaly(&self, route: &str, outcome: &str)

Count one anomaly-scoring inspection outcome (DW-090). route is the matched route name; outcome is blocked (403 returned), logged (dry-run match, request allowed), or passed (score below the threshold). Both labels are config-bounded (route names and the closed outcome set).

pub fn record_graphql(&self, route: &str, outcome: &str)

Count one GraphQL check outcome (DW-099). route is the matched route name; outcome is one of the closed set allowed, depth_exceeded, complexity_exceeded, persisted_query_required. Both labels are config-bounded.

pub fn set_graphql_depth(&self, route: &str, depth: i64)

Set the last GraphQL query depth gauge (DW-099), by route.

pub fn set_graphql_complexity(&self, route: &str, complexity: i64)

Set the last GraphQL query complexity gauge (DW-099), by route.

pub fn record_policy_dry_run(&self, phase: &str, route: &str)

Count one dry-run (monitor) would-have-rejected observation (DW-041): a policy phase evaluated a DENY (or shed, or limit violation) and the request was allowed anyway. phase is one of route_limits, authz, rate_limit, load_shed; route is the matched route name (unrouted for the pre-404 listener/ global policy pass) — both config-bounded label spaces.

pub fn emit_policy_dry_run( &self, phase: &str, would_be_status: u16, route: &str, consumer: Option<&str>, request_id: &str, detail: &str, )

The dry-run report’s log side (DW-041): one structured warn event at target dwara::policy per would-have-rejected request, carrying the phase, the status the phase WOULD have answered, the route and consumer (when known), and the phase’s own reason — the request-id correlation key ties it to the trace and access line of the request that was allowed. Field list is exhaustive and redacted by construction (no headers, no query string).

pub fn jwks_refresh_counter(&self, provider: &str) -> IntCounter

The JWKS refresh counter child for one provider (handed to the authenticator; incremented on every JWKS fetch attempt).

pub fn record_cache_lookup(&self, outcome: &str)

Count one response-cache lookup outcome (DW-037). outcome is one of the closed set hit / stale / miss / bypass (see dataplane::response_cache for the decision table).

pub fn record_cache_store(&self, outcome: &str)

Count one response-cache store outcome (DW-037). outcome is one of the closed set stored / vetoed / over_cap / error.

pub fn record_cache_revalidated(&self)

Count one 304-confirmed revalidation served from the stored body (DW-037).

pub fn record_cache_purge(&self, scope: &str)

Count one cache purge (DW-037). scope is route, all, tag, url, or url_prefix (the DP-04 tag/url purge axes).

pub fn set_cache_entries(&self, entries: i64)

Set the approximate live-entries gauge (DW-037) — a scrape-time snapshot walk of the backing store (the same gauge model as the rate-limiter live-keys gauge: state lives outside observability, the dataplane walks it at scrape time).

pub fn record_coalescing_leader(&self)

DW-038: this cache miss became the coalescing leader.

pub fn record_coalescing_follower(&self, outcome: &str)

DW-038: a coalescing follower resolved — outcome is one of the closed set served / fell_back_timeout / fell_back_unshared / fell_back_epoch (see the field docs; callers pass literals).

pub fn record_coalescing_saved(&self)

DW-038: one upstream call avoided (a follower was served).

pub fn coalescing_waiter(&self, parked: bool)

DW-038: follower waiters gauge, +/- on park/resolve.

pub fn active_requests(&self) -> &IntGauge

pub fn set_config_generation(&self, generation: u64)

Publish the current config generation number.

pub fn set_license_status(&self, status: i64)

Set the dwara_license_status gauge (DW-032): 0 = no license (OSS), 1 = valid, 2 = expired within grace, 3 = expired past grace (degraded to OSS). Set once at startup and on every reload.

pub fn set_dns_discovery_endpoints(&self, upstream: &str, count: i64)

Set the dwara_dns_discovery_endpoints gauge for one upstream (DW-042): the current resolved endpoint count. The upstream label is config-bounded; cardinality is never per resolved address. Set by the discovery task after each refresh.

pub fn record_dns_discovery_refresh(&self, upstream: &str)

Count one DNS discovery refresh attempt (DW-042) in dwara_dns_discovery_refresh_total{upstream}. The upstream label is config-bounded.

pub fn record_dns_discovery_refresh_failure(&self, upstream: &str)

Count one DNS discovery refresh failure (DW-042) in dwara_dns_discovery_refresh_failures_total{upstream}. The upstream label is config-bounded.

pub fn set_config_convergence_generation(&self, instance: &str, generation: i64)

Set this instance’s current convergence generation (DW-054) in dwara_config_convergence_generation{instance}. The instance label is config-bounded (one series per process). Set by the convergence coordinator after every local or remote publish.

pub fn set_config_convergence_instances(&self, count: i64)

Set the total instances the convergence backend reports (DW-054) in dwara_config_convergence_instances. Set by the coordinator’s drift check from the cluster view.

pub fn set_config_convergence_drift(&self, drift: bool)

Set the convergence drift flag (DW-054) in dwara_config_convergence_drift: 1 when one or more instances serve a different config hash than the majority, 0 when converged. Set by the coordinator’s drift check.

pub fn record_config_convergence_refresh(&self)

Count one convergence refresh attempt (DW-054) in dwara_config_convergence_refresh_total.

pub fn record_ai_budget_denied(&self, kind: &str)

Count one AI budget denial (DW-078) in dwara_ai_budget_denied_total{kind} (kind: tokens | cost).

pub fn record_ai_governance_denied(&self, reason: &str)

Count one AI governance denial (DW-084) in dwara_ai_governance_denied_total{reason} (reason: the denial cause string, config-bounded — no consumer/model labels).

pub fn record_ai_guardrail_blocked(&self, rule: &str, kind: &str)

Count one AI guardrail block (DW-082) in dwara_ai_guardrail_blocked_total{rule,kind} (rule: the rule name, config-bounded; kind: the phase/kind label — prompt, response, schema, pii, injection, banned). No consumer label (cardinality rule).

pub fn record_ai_semantic_cache_hit(&self, model: &str)

Count one AI semantic cache hit (DW-083) in dwara_ai_semantic_cache_hits_total{model} (model: the client-facing alias, config-bounded). No consumer label (cardinality rule).

pub fn record_ai_semantic_cache_miss(&self, model: &str)

Count one AI semantic cache miss (DW-083) in dwara_ai_semantic_cache_misses_total{model} (model: the client-facing alias, config-bounded). No consumer label (cardinality rule).

pub fn record_ai_routing_policy_escalation(&self, policy: &str)

Count one AI routing-policy FallbackChain escalation (DW-085) in dwara_ai_routing_policy_escalations_total{policy} (policy: the routing-policy name, config-bounded). No consumer label (cardinality rule).

pub fn record_ai_routing_policy_cheap(&self, policy: &str)

Count one AI routing-policy FallbackChain cheap-model selection (DW-085) in dwara_ai_routing_policy_cheap_total{policy} (policy: the routing-policy name, config-bounded). No consumer label (cardinality rule).

pub fn record_ai_routing_policy_latency_cost_selection(&self, policy: &str)

Count one AI routing-policy LatencyCost selection (DW-085) in dwara_ai_routing_policy_latency_cost_selections_total{policy} (policy: the routing-policy name, config-bounded). No consumer label (cardinality rule).

pub fn record_ai_experiment_variant_selection( &self, experiment: &str, variant: &str, )

Count one AI experiment (A/B test) variant selection (DW-086) in dwara_ai_experiment_variant_selections_total{experiment, variant} (experiment: the A/B test name, variant: the selected variant name, both config-bounded labels). No consumer label (cardinality rule).

pub fn record_ai_stream_chunk(&self, provider: &str)

Count forwarded streaming chunks (DW-077) in dwara_ai_stream_chunks_total{provider}.

pub fn record_ai_first_token(&self, provider: &str, seconds: f64)

Record the time to a stream’s first forwarded chunk (DW-077) in dwara_ai_first_token_seconds{provider}. Call once per stream.

pub fn record_ai_stream_end(&self, provider: &str, seconds: f64)

Record a completed stream’s total duration (DW-077) in dwara_ai_stream_duration_seconds{provider}.

pub fn record_ai_request_duration( &self, provider: &str, route: &str, seconds: f64, )

Record one AI request’s end-to-end latency (AI-07) in dwara_ai_request_duration_seconds{provider,route}. provider and route are config-bounded labels. Call once per completed AI request (success and provider/translation errors alike); client-side rejections never reach a provider and are not observed here.

pub fn record_ai_tokens_per_request( &self, provider: &str, model: &str, prompt: u64, completion: u64, )

Observe the per-request token distribution (AI-07) in dwara_ai_tokens_per_request{provider,model,kind}. kind is the closed set prompt | completion. Call once per request that reported usage, for each direction that has a value (provider-reported only; the gateway never estimates). model is the provider’s own model identifier (config-bounded).

pub fn record_mcp_session(&self, state: &str)

Count one MCP session lifecycle transition (DW-087) in dwara_mcp_sessions_total{state}. state is one of the closed set initialized (a new session created by initialize), closed (a session deleted by shutdown), or expired (a session reaped by the TTL cleanup).

pub fn record_mcp_tool_call(&self, tool: &str, status: &str)

Count one MCP tool call (DW-087) in dwara_mcp_tool_calls_total{tool,status}. tool is the config-declared tool name (config-bounded); status is one of the closed set success, error, denied. No consumer label (cardinality rule); the consumer is in the access log and the analytics mcp_tool_calls table.

pub fn record_mcp_tool_duration(&self, tool: &str, seconds: f64)

Record an MCP tool call duration (DW-087) in dwara_mcp_tool_duration_seconds{tool}. tool is the config-declared tool name (config-bounded label). Call once per tool call (success, error, and denied alike).

pub fn record_a2a_request(&self, agent: &str, outcome: &str)

Count one A2A (agent-to-agent) request (DW-114) in dwara_a2a_requests_total{agent,outcome}. agent is the config-declared agent name (config-bounded); outcome is one of the closed set success, error, stubbed (stubbed = the task lifecycle is not yet implemented pending spec freeze). No consumer label (cardinality rule); the consumer is in the access log.

pub fn record_a2a_session(&self, state: &str)

Count one A2A session lifecycle transition (DW-114) in dwara_a2a_sessions_total{state}. state is one of the closed set initialized (a new task session created), closed (a session deleted), or expired (a session reaped by the TTL cleanup). Mirrors MCP’s session counter.

pub fn record_nano_service_request(&self, route: &str, outcome: &str)

Count one nano-service (WASM route handler) request outcome (DW-106) in dwara_nano_service_requests_total{route,outcome}. route is the matched route name (config-bounded); outcome is one of the closed set success, error, timeout. No consumer label (cardinality rule); the consumer is in the access log.

pub fn record_nano_service_duration(&self, route: &str, seconds: f64)

Record a nano-service handle call duration (DW-106) in dwara_nano_service_duration_seconds{route}. route is the matched route name (config-bounded label). Call once per request (success, error, and timeout alike).

pub fn record_plugin_failure(&self, name: &str, reason: &str)

Count one plugin dispatch failure (DW-157) in dwara_plugin_failures_total{name,reason}. name is the config-declared plugin name (config-bounded); reason is one of the closed set crashed, disabled, not_loaded, not_registered, instantiate_failed, trap, body_too_large, response_stream_ended. Counted once per request the plugin chain answered with an error — a plugin’s send_http_response short-circuit is a decision, not a failure, and is not counted (the access-log plugin_short_circuit flag marks those).

pub fn record_plugin_callout(&self, name: &str, outcome: &str)

Count one plugin HTTP callout outcome (DW-167) in dwara_plugin_callouts_total{name,outcome}. name is the config-declared plugin name (config-bounded); outcome is one of the closed set ok (a completed response was delivered to the plugin, any status), timeout, error, loop_guard. Counted once per performed callout (loop_guard once per guard trip); a callout ERROR additionally increments dwara_plugin_failures_total{name,reason=callout_failed} because the route fails closed.

pub fn set_plugin_state_total(&self, state: &str, total: i64)

Set one state’s plugin count (DW-158) in dwara_plugin_total{state}. state is one of the closed set healthy, crashed, disabled, not_loaded, not_registered. The dataplane sets every state’s series on each publish (reload_plugins), so a reload that removes the last plugin of a state drives its series back to zero instead of stranding the previous count.

pub fn record_tls_pq_handshake(&self, result: &str)

Count one post-quantum TLS handshake outcome (DW-105) in dwara_tls_pq_handshakes_total{result}. result is one of the closed set success (the X25519+ML-KEM hybrid kx group was negotiated), fallback (the client did not support the hybrid group and a classical group was negotiated), or disabled (PQ was off or the experimental API was inert). Use security::pq::pq_handshake_metric to derive the label from a security::pq::PqHandshakeResult.

pub fn record_config_convergence_refresh_failure(&self)

Count one convergence refresh failure (DW-054) in dwara_config_convergence_refresh_failures_total.

pub fn record_ai_request( &self, provider: &str, route: &str, outcome: &str, version: &str, )

Count one AI route request outcome (DW-075) in dwara_ai_requests_total{provider,route,outcome,version}. The provider, route, and version labels are config-bounded (version is the canary version that served, DW-076, or “default” for non-canary aliases).

pub fn record_ai_tokens( &self, provider: &str, prompt: u64, completion: u64, version: &str, )

Add provider-reported token usage (DW-075) to dwara_ai_tokens_total{provider,kind,version} (kind: prompt | completion; version: the canary version that served, DW-076, or “default”). Provider-reported values only.

pub fn record_ai_cost(&self, provider: &str, model: &str, cost_micros: u64)

Add priced AI spend (DW-079) to dwara_ai_cost_micros_total{provider,model} in micro-USD. The provider and model labels are config-bounded (no consumer label — cardinality rule).

pub fn record_ai_unknown_pricing(&self, model: &str)

Count one request served by a model with no pricing entry (DW-AI-03) under the alert unknown-model policy. The model label is the provider model identifier (config-bounded).

pub fn config_convergence_drift(&self) -> i64

The current dwara_config_convergence_drift gauge value (DW-054): 1 when drift was last reported, 0 when converged. Test seam + operator introspection (the gauge is also scraped via /metrics).

pub fn config_convergence_instances(&self) -> i64

The current dwara_config_convergence_instances gauge value (DW-054): the last cluster-view instance count. Test seam + operator introspection.

pub fn set_breaker_state(&self, upstream: &str, state: i64)

Set the breaker_state gauge for one upstream (0 closed, 1 open, 2 half-open). Scrape-time snapshot setter: the walk that computes the values from live upstream state lives on the dataplane (see refresh_observation_gauges in dataplane::upstream), keeping this module free of runtime-state dependencies — it only records what it is handed.

pub fn set_fail_open_picks(&self, upstream: &str, picks: i64)

Set the upstream_fail_open_picks gauge for one upstream (scrape-time snapshot of the balancer’s monotonic fail-open counter; a gauge rather than a counter so the hot pick path stays free of metrics coupling).

pub fn set_rate_limiter_evictions(&self, evictions: i64)

Set the dwara_rate_limiter_evictions_total gauge: aggregate cells dropped by eviction sweeps across every compiled rule (#132). Scrape-time snapshot of the engine’s monotonic counter — a gauge rather than a counter so the hot check path stays free of metrics coupling; the value resets when a reload rebuilds the engine. See Self::set_breaker_state for the refresh model.

pub fn set_rate_limiter_live_keys(&self, keys: i64)

Set the dwara_rate_limiter_live_keys gauge: live per-key cells across every compiled rule, approximate under concurrent checks and bounded by the sharded store cap (#132). Aggregate and unlabeled by design — cardinality is never per key. Scrape-time snapshot setter; see Self::set_breaker_state.

pub fn record_webhook_event(&self, kind: &str, outcome: &str)

Count one webhook delivery outcome (DW-044) in dwara_webhook_events_total{kind,outcome}. kind is the event-kind spelling, outcome one of delivered, failed, dropped (the deliverer is the only caller, so both label spaces stay the closed sets documented on the family).

pub fn set_events_dropped(&self, dropped: i64)

Set the dwara_events_dropped_total gauge (DW-044): events dropped at EMIT time. Scrape-time snapshot of the event bus’s monotonic counter; see Self::set_rate_limiter_evictions for the gauge-not-counter model.

pub fn set_events_emitted(&self, emitted: i64)

Set the dwara_events_emitted_total gauge (DW-044): events handed to the event queue since process start. Scrape-time snapshot setter; see Self::set_events_dropped.

pub fn record_access_stream(&self, outcome: &str, records: u64)

Count records access records into dwara_access_records_streamed_total{outcome} (DW-121). The stream flusher is the only caller; outcome is one of delivered, failed, dropped (the closed set documented on the family), counted at BATCH granularity by the records each flushed batch carried.

pub fn set_access_records_offered(&self, offered: i64)

Set the dwara_access_records_offered_total gauge (DW-121): records offered to the stream channel since process start. Scrape-time snapshot of the stream’s monotonic counter; see Self::set_events_dropped for the gauge-not-counter model.

pub fn set_access_records_dropped(&self, dropped: i64)

Set the dwara_access_records_dropped_total gauge (DW-121): records dropped at OFFER time. Scrape-time snapshot setter; see Self::set_access_records_offered.

pub fn set_access_records_sampled_dropped(&self, dropped: i64)

Set the dwara_access_records_sampled_dropped_total gauge (REL-10, #221): records dropped by sampled degradation. Scrape-time snapshot setter; see Self::set_access_records_dropped.

pub fn record_split_pick(&self, service: &str, upstream: &str)

Count one split dispatch decision (DW-040) in dwara_split_picks_total{service,upstream}. Both labels are config-declared names; the caller is the dispatch path.

pub fn record_sticky_session(&self)

Count one minted sticky session (DW-040).

pub fn record_websocket_policy(&self, route: &str, outcome: &str)

Count one WebSocket policy decision (DW-039) in dwara_websocket_policy_total{route,outcome}. outcome is one of origin_denied, rate_closed (the closed set documented on the family); route is the config-declared route name.

pub fn set_endpoint_health(&self, upstream: &str, endpoint: &str, up: bool)

Set the endpoint_health gauge for one endpoint (1 available, 0 ejected). Scrape-time snapshot setter; see Self::set_breaker_state for the refresh model.

pub fn render(&self) -> String

Render every family in Prometheus text format.

pub fn gather(&self) -> Vec<MetricFamily>

Gather the raw metric families (DW-073: OTLP metrics export). Used by the OTLP metrics exporter to convert prometheus metric families into OTLP protobuf for export to a collector.

impl Observability

pub fn set_route_slos(&self, slos: Vec<(String, SloTargets)>)

DW-052: swap the configured per-route SLO set. Called by the dataplane on every generation refresh; routes without an slo block record nothing (the completion-path call is a no-op for them).

pub fn record_slo(&self, route: &str, status: u16, duration_ms: f64)

DW-052: record one completed request against its route’s SLO windows (request-completion path; no-op when unconfigured).

pub fn record_l4_connection(&self, listener: &str, protocol: &str)

Count one L4 connection (DW-103) in dwara_l4_connections_total{listener,protocol}. listener is the config-declared listener name (config-bounded); protocol is one of the closed set tcp, udp. Counted once per accepted L4 connection (before the splice is established).

pub fn record_l4_connection_duration(&self, listener: &str, seconds: f64)

Record an L4 connection duration (DW-103) in dwara_l4_connection_duration_seconds{listener}. listener is the config-declared listener name (config-bounded label). Call once per connection (forwarded and closed alike).

Trait Implementations

impl Default for Observability

fn default() -> Self

Returns the “default value” for a type.