Expand description
In-process event bus (DW-044, feature analysis section 5 “Platform”): the notice plumbing behind alert & event webhooks.
§What it is
A bounded tokio::sync::mpsc channel plus cheap clone-able
Emitter handles. Subsystems that own a state machine worth
alerting on (the circuit breaker, passive/active health, the config
publish pipeline) emit a small Event at every transition; the
webhook deliverer (webhook) drains the channel in a background
task and POSTs each event to the configured targets. std + tokio
only — no dependency beyond config (the schema types and their
bounds) and observability (delivery-outcome metrics).
§Emission contract: never backpressure the dataplane
Emitter::emit is a try_send onto the bounded channel. When the
queue is full (or nobody is draining it) the event is DROPPED and the
drop is counted (EventBus::dropped_total, surfaced as the
dwara_events_dropped_total gauge at scrape time). There is no
blocking wait, no unbounded buffer growth, and no failure mode that
can stall the caller — emission sites sit on request paths (the
breaker check/report wire) and on the reload path, and a
misbehaving webhook target must never show up as gateway latency.
The drop policy is drop-NEWEST at emit time (the oldest queued
events keep their place in line): a slow drain sheds the burst, not
the history.
§Event kinds and payloads
Kinds are a closed set (EventKind::ALL) — a webhook’s events
list is validated against it, so a config cannot subscribe to an
event the gateway will never emit. The payload is a flat struct of
bounded config labels and numbers (EventPayload); there is no
free-form field, so the serialized envelope is small by construction
(the deliverer still enforces a hard byte cap). No request-derived
data (paths, headers) enters a payload — identifiers are
CONFIG-DECLARED labels only (upstream, and consumer for quota
budgets, whose consumers are config records by construction in this
edition) — an envelope is safe to POST to a third party by
construction.
Emission sites:
- breaker state transitions (DW-015): opened (with the rule that
tripped), half-open, closed (
resilience::breaker); - outlier ejection and recovery (DW-012 passive + DW-013 active —
both report through the same tracker, so both emit):
endpoint_ejected,endpoint_recovered(resilience::health); - config published (generation + content hash + route count) and
config rejected (validation issue count) (
snapshot); - a consumer’s request budget crossing the near-limit threshold
(DW-033):
quota_near_limit, edge-triggered once per (consumer, budget, window) from the dataplane’s quota phase (dataplane::proxy— the emit lives there because the state domain must not import events).
Deliberately NOT emitted (documented hook points, not gaps):
- rate-limiter eviction (#132): already observable as
dwara_rate_limiter_evictions_total; an event per eviction would be high-frequency noise, not an alert.
§Failure isolation
The deliverer (see webhook) runs as its own task, dispatches each
delivery as a further task, bounds concurrent deliveries with a
semaphore (over-cap deliveries are dropped and counted, never
queued), and gives every delivery ONE total timeout shared by all
its retry attempts. A slow, hung, or dead target can therefore
consume at most timeout_ms of one semaphore slot — never the
queue, never the emit path, never the dataplane.
§Bus identity and lifecycle
One bus per gateway process, owned by the DataPlane and attached
to the ConfigState it was built from (so config publish events and
dataplane events share one queue and one deliverer). The
gateway field of every envelope identifies the instance:
dwara-<pid>-<boot unix ms> — unique per process start, no
hostname dependency, stable for the process lifetime.
Modules§
- stream
- Real-time access-record stream (DW-121, feature analysis section 5 “Platform”): the record plumbing that mirrors the notice plumbing of DW-044. Where the event bus carries a handful of discrete operational events per minute (breaker transitions, ejections, config publishes), this pipeline carries ONE record per completed request — the raw firehose an operator points at their own warehouse or SIEM when the embedded analytics store (DW-043) is not the system of record. It complements, never replaces: the two are configured independently (a deployment can run either, both, or neither) and share nothing at runtime — different channels, different tasks, different failure domains.
- webhook
- Webhook delivery (DW-044, feature analysis section 5 “Platform”): drains the event bus and POSTs each event to the configured targets.
Structs§
- Emitter
- A clone-able emission handle (an
Arc<EventBus>).Option<Emitter>at every emission site:None(tests, unwired constructions) is a documented no-op, so the subsystems never need a bus to exist. - Endpoint
Events UpstreamEmitterfurther bound to one endpoint label (DW-044): held by the per-endpoint passive health tracker (resilience::health::EndpointHealth), which knows its state machine but not its ownaddress:port(the balancer does).- Event
- One emitted event.
idandtimestamp_msare assigned by the bus at emit time (single place),gatewayidentifies the process. - Event
Bus - The bounded event queue plus its identity and counters. Share via
Arc; emitters hold anEmitter(a cheap clone of that Arc). - Event
Payload - The payload of one event: a flat set of bounded labels and numbers. Every field is optional (each event kind sets the fields it owns); unset fields are omitted from the serialized envelope, and there is deliberately NO free-form/detail-string-from-input field — an envelope must stay small and safe to hand to a third party.
- Upstream
Emitter Emitterpre-bound to an upstream label (DW-044): held by the per-upstream circuit breaker and balancer.
Enums§
- Event
Kind - Every event kind the gateway emits (DW-044). Closed set: webhook
eventslists are validated againstEventKind::from_config.
Constants§
- EVENT_
CHANNEL_ CAPACITY - Capacity of the event channel (DW-044). Bound chosen for alert-rate
shapes: transition events are rare per upstream (a breaker flapping
once per
open_ms, ejections once pereject_ms), so 256 queued events is already a gateway in serious trouble — beyond it, drops (counted, gauge-exported) are the right answer, not buffering.
Traits§
- Event
Durability - REL-14 (#249): the durability seam for the event bus. The events
domain owns the trait; the dataplane (which already holds both the
StateStoreand theEventBus) provides a concrete implementation that wrapsStateStoremethods. This keeps the dependency direction correct:eventsdoes not importstate.
Functions§
- generate_
instance_ id - Process-unique instance label:
dwara-<pid>-<boot unix ms hex>. Two gateway processes on one host never share it; it carries no hostname (a third-party envelope reader needs process identity, not topology). - refresh_
event_ gauges - Refresh the event-bus observation gauge at scrape time (the
dwara_rate_limiter_evictions_totalmodel: the emit path bumps a plain atomic, and only the scrape couples it to the registry — a gauge rather than a counter so the hot path stays registry-free).