Module breaker 

Per-upstream circuit breaker (DW-015, feature analysis 4.11).

What it is

The breaker gates an ENTIRE upstream (all endpoints) when that upstream is failing as a whole. It is a different layer from DW-012 per-ENDPOINT ejection: ejection removes individual endpoints from load-balancer rotation, the breaker stops sending ANY traffic to the upstream for a cooling-off period. When every endpoint is ejected the balancer fail-opens and picks anyway (DW-012); that pick still flows THROUGH the breaker — the breaker gates the upstream, ejection gates endpoints within it. The two never consume each other’s state: an open breaker short-circuits before the balancer is consulted, so a breaker-open period ejects nothing (passive health sees no traffic, hence no failures, hence no ejections), and ejections never open the breaker.

States and transitions

• Closed (healthy): requests flow. Failures are counted two ways — a consecutive-failure streak and a rolling-window error ratio. The breaker OPENS when either trips: the streak reaches consecutive_failures (default 5), or the window holds at least error_volume (default 20) observations AND failures / observations >= error_ratio (default 0.5). The window is 60 seconds, the same rolling-event pattern as the retry budget.
• Open (tripped): every request fails fast with 503 (body “upstream circuit open”) and a Retry-After header carrying the whole seconds until half-open (rounded up, minimum 1). No attempts are made — no endpoint pick, no retries. Requests already in flight when the breaker opened complete normally (documented).
• HalfOpen (probing): after open_ms (default 30000) the next request is admitted as a trial probe (up to half_open_probes concurrent probes, default 1). A successful probe CLOSES the breaker (all counters reset, window cleared); a failed probe re-OPENS it for another open_ms. While all probes are in flight, further requests fail fast like Open (Retry-After: 1 second, the documented probing hint — exact half-open time is unknowable while a probe runs). A retried request consumes one half-open probe slot per attempt: each attempt is a trial.

Observation points

The breaker is evaluated at the SAME observation points as passive health: when an attempt’s response HEADERS resolve — transport errors and statuses >= 500 are failures, everything else (1xx-4xx) is a success and counts toward the ratio denominator. Every retry attempt reports too (each attempt is a real exchange with the upstream). A mid-BODY abort after headers resolved does not open the breaker: the exchange was classified at header time (it is still reported to endpoint health per DW-014).

Failure classification is intentionally identical to passive health so operators reason about one notion of “failure” across both layers.

Timing is wall-clock (SystemTime): an NTP step can lengthen or shorten a wall-clock Open period.

Reload semantics

Breaker STATE (current state, streak, window) is carried across config reloads keyed by upstream name — exactly like balancer state and the retry budget. Breaker PARAMETERS apply from the new config (BreakerParams is resolved per generation, Breaker holds state only, and every method takes the params explicitly — the same split as RetryParams/RetryBudget).

Events (DW-044)

A breaker constructed with Breaker::with_clock_and_events emits one event per STATE TRANSITION (never per report): breaker_opened (with the rule that tripped, or “half_open_probe_failed” for a failed probe re-opening it), breaker_half_open (the first admitted probe after the cooling-off), and breaker_closed (a successful probe). Emission is the event bus’s bounded non-blocking hand-off — a full queue drops and counts, never blocks the report path. A None emitter (every direct/test construction) is a documented no-op.

Structs

Breaker

Per-upstream circuit breaker state (see the module docs). Share via Arc from the upstream handle; the lock is taken once per attempt (report) and once per request admission (check).

BreakerParams

Resolved (validated) breaker parameters for one upstream. None on the upstream handle disables the breaker entirely — the no-config behavior is bit-identical to pre-DW-015.

Enums

BreakerDecision

Admission decision for one request about to be dispatched.

BreakerState

Breaker state snapshot (observability/tests).

Constants

BREAKER_WINDOW_MS

Rolling window the error ratio accounts over (milliseconds).

HALF_OPEN_RETRY_AFTER_MS

Retry-After (milliseconds) advertised while half-open probes are in flight (exact half-open time is unknowable until a probe resolves).