Module health 

Passive health checking / outlier detection (DW-012, feature analysis 4.5).

Passive health watches REAL traffic outcomes — no synthetic probes are sent while an endpoint is serving. Each endpoint of an upstream carries one EndpointHealth tracker; the proxy’s send path classifies every dispatched request’s outcome and reports it to the picked endpoint’s tracker (EndpointHealth::report). The load balancer consults EndpointHealth::acquire on every pick (lock-free: atomics only) and skips endpoints that are currently ejected.

Ejection model

An endpoint is ejected when EITHER:

• consecutive failures: consecutive_failures (default 5) transport or 5xx outcomes in a row, regardless of volume; or
• failure ratio with volume: within the rolling window_ms (default 60 s) observation window, the 5xx/transport share of observations is >= failure_ratio (default 0.5) AND the window saw at least failure_min_volume (default 20) observations. The volume gate keeps a 2-of-3 blip from ejecting on a trickle of traffic.

Outcome classification (reported at response-header resolution, the same point the in-flight guard releases): transport errors (connect timeout, refused, reset, client framing errors) are failures; HTTP status >= 500 is a failure; 1xx-4xx are successes. 429 and 408 are deliberately successes in v1: they describe the CALLER or queueing pressure, not the endpoint’s health, and ejecting on them would remove healthy capacity exactly when backpressure is needed (documented choice).

Active health probes report through EndpointHealth::report_probe, which drives the SAME consecutive streak and ejection/recovery transitions but never inserts an observation into the rolling window: synthetic probes never mix into real-traffic ratios.

Recovery (half-open)

After eject_ms (default 30 s) the tracker moves to half-open on the next pick: the first half_open_probes (default 1) requests are allowed through (each pick consumes one probe); everything else keeps skipping the endpoint. A successful probe restores the endpoint to healthy and clears its failure history (so the old failures cannot immediately re-eject it); a failed probe re-ejects for another full eject_ms. A success observed on an EJECTED endpoint (only possible via the all-ejected fail-open path) also restores health — real traffic to it just worked, which is the strongest possible health signal.

Fail-open

If EVERY endpoint of an upstream is ejected, picks fall back to the full set rather than blackholing traffic (documented choice; a gateway that returns 503 for a fully-ejected pool has converted an upstream brownout into a guaranteed outage). The balancer counts these fallbacks (UpstreamLb::fail_open_picks) so operators can see the degraded state.

Time

All timing runs on a millisecond clock resolved per call from the owning balancer (see UpstreamLb::now_ms), which defaults to the system clock and can be swapped for a deterministic clock in tests (UpstreamLb::set_health_clock) — the same injection pattern the rate limiter (DW-004) uses.

Concurrency

Status transitions are CAS loops on a status atomic; the observation window is a Mutex<VecDeque> mutated only on the report path (never on picks). Races between concurrent reports and picks resolve best-effort: a probe slot may occasionally be granted to a request that observes slightly stale state, which is harmless — the next report re-synchronizes the tracker. Tracker state (Arc<EndpointHealth>) is keyed by endpoint address:port and carried across config rebuilds like the in-flight counters; config changes to the health parameters apply to NEW observations (the window and consecutive counter live on).

Events (DW-044)

A tracker constructed with EndpointHealth::with_events emits endpoint_ejected and endpoint_recovered on the two state transitions (bound to its upstream and address:port labels by the balancer at construction). Both passive reports and ACTIVE probe reports funnel through the same transition code, so either signal ejecting or recovering an endpoint emits. Emission is the event bus’s bounded non-blocking hand-off (full queue = drop and count); None (direct/test constructions) is a documented no-op.

Structs

EndpointHealth

Per-endpoint passive health tracker. All pick-path reads are atomic; only the observation window takes a lock (report path). Share via Arc from the balancer state; carried across rebuilds for unchanged address:port.

HealthDispatch

The per-dispatch health report handle carried out of a pick: the endpoint’s tracker plus the parameter generation the pick ran against. The send path reports the outcome exactly once.

HealthParams

Resolved (validated) passive-health parameters, shared per upstream and snapshotted inside each balancer state generation. A rebuild with new config swaps the whole Arc; in-flight reports pin the generation they were dispatched with, so parameter changes apply only to new observations.

Constants

DEFAULT_CONSECUTIVE_FAILURES

Default consecutive-failure ejection threshold.

DEFAULT_EJECT_MS

Default ejection duration in milliseconds.

DEFAULT_FAILURE_MIN_VOLUME

Default minimum observations in the window before the ratio applies.

DEFAULT_FAILURE_RATIO

Default failure ratio (0 < r <= 1).

DEFAULT_HALF_OPEN_PROBES

Default trial requests allowed through per half-open recovery attempt.

DEFAULT_WINDOW_MS

Default rolling observation window in milliseconds.