Module active 

Active health checks (DW-013, feature analysis 4.5).

Passive health (DW-012) watches real traffic; active health PROBES. For every upstream that configures active_health (and therefore the passive health block), one probe loop task per endpoint runs:

loop { sleep(interval + uniform(0..jitter)); probe; report }

Probe mechanics

Probes go DIRECTLY to the endpoint (address:port), bypassing load balancing and the pooled upstream client — a probe must examine one specific endpoint, not whoever the balancer would pick. Each http probe speaks HTTP/1.1 over its own connection (plaintext toward http1 upstreams, rustls over the SAME trust roots as the pooled connector — webpki by default, the upstream’s trusted_ca_file bundle when configured, #121 — with the endpoint address as server name toward https/http2 upstreams), issues GET {path} with Connection: close, and classifies the FIRST status line it reads: 2xx = success; anything else (3xx included — redirects are not followed — 4xx, 5xx, truncation, timeout, transport error) = failure. tcp probes succeed when the TCP connect completes within timeout_ms. Documented limitation: http2-protocol upstreams that refuse HTTP/1.1 on a separate connection will read as failing; use kind: tcp for those.

Timing

Each cycle sleeps interval + uniform(0..jitter) BEFORE probing, so the realized period is interval + jitter + probe duration and the tick drifts late by that sum per cycle (no fixed-wallclock scheduling); the jitter exists to keep probe storms from synchronizing across endpoints.

Reporting model (precedence over passive)

Probe results report into the SAME per-endpoint crate::resilience::health::EndpointHealth tracker the passive checker and the load balancer use, so both systems feed one ejection state and the existing pick filter removes failing endpoints from rotation:

• Both run concurrently. Either signal can eject: a passive report ejects at the passive consecutive_failures threshold; an active report ejects at failure_threshold. They share the tracker’s consecutive-failure streak, so the last report’s parameters decide whether that streak ejects — the coherent reading is “either can eject, on its own threshold, from one shared streak”.
• Active reports use the consecutive path only. Probe reports go through EndpointHealth::report_probe, which never inserts an observation into the rolling window — synthetic probes must not be mixed into real-traffic ratios — and the probe’s report parameters additionally disable the failure-ratio/volume rule (failure_min_volume = u32::MAX) as a second line of defense. The passive window/eject/half-open parameters stay in force.
• Ejection follows the passive model: after failure_threshold consecutive failed probes the endpoint leaves rotation for health.eject_ms. Probe failures while already ejected leave the window alone (they cannot extend it).
• Recovery is probe-driven: success_threshold consecutive successful probes report a success, and a success observed on an ejected (or half-open) tracker restores it outright — with active health on, a healthy probe streak re-admits the endpoint even before eject_ms expires, which is the point of probing. A probe success below the threshold is reported only when the endpoint is currently available (it just resets the failure streak; it never re-admits). Best-effort under races, exactly like the rest of the tracker.

Task lifecycle

ActiveProbes owns one [tokio::task::JoinSet] for every probe loop. The operator (dwara-bin) calls ActiveProbes::respawn after every snapshot swap (startup and each reload): all previous loops are aborted and new ones spawned for the new generation’s upstreams. Endpoints whose address:port persists across the swap keep their tracker (the balancer carries it, DW-011/DW-012 semantics), so an ejection streak survives a reload even though the probe task restarts. Dropping ActiveProbes (or calling ActiveProbes::abort_all) aborts every loop — the graceful-shutdown path.

Structs

ActiveParams

Resolved (validated) active-probe parameters, copied into each probe loop so a config rebuild cannot change a running loop mid-flight.

ActiveProbes

Owns every active-probe loop task for the running generation. Call ActiveProbes::respawn on startup and after every snapshot swap; dropping it (or shutdown) aborts all loops.

Functions

catch_unwind_future

catch_unwind for futures (std has no FutureExt::catch_unwind): each poll of fut runs under catch_unwind, so a panic raised at any await point inside fut surfaces as Err instead of unwinding the loop task.

next_below

Public for testing the probe-jitter bound contract.

probe_once

One direct probe of address:port, bounded by timeout (connect plus response for http). tls is the upstream’s outbound TLS client config: Some for https/http2 upstreams (built from the SAME root store as the pooled connector — webpki by default, the upstream’s trusted_ca_file bundle when configured, #121 — with HTTP/1.1 ALPN), None for plaintext http1 upstreams. happy is the upstream’s resolved happy-eyeballs delay (DW-030): probes dial through the same RFC 8305 racing as the pooled connector, so a dual-stack endpoint’s probe does not silently behave differently from its traffic. Returns whether the endpoint answered healthy.

report_params

Report parameters for probe outcomes: the passive window/eject/half-open parameters with the ACTIVE thresholds substituted and the ratio rule disabled (see the module docs).