Module experiments 

Prompt experimentation (DW-086): A/B model comparison, prompt versioning, regression evals, and feedback ingestion.

A/B tests

A model alias declares ab_test: <name> to be served by an experiment. The experiment’s variants each name a plain model alias (chain/canary), an optional prompt version, and a weight. The CompiledAbTest resolves each variant’s model alias to its primary RouteTarget at compile time (the same composition pattern as DW-085 routing policies). At request time, a deterministic weighted pick by request id selects a variant — the same slot semantics as canary splits (ratios hold per request, and re-sending a request with the same id lands on the same variant).

Prompt versioning

Each prompt declares one or more versions (each with a system message) and an active version. The active version can be overridden at runtime via the admin API (stored in the state store). When a variant references a prompt version, or when an eval references a prompt version, the system message is prepended to the request’s messages (BEFORE any existing system message).

Eval runner

The eval runner makes DIRECT provider calls via hyper_util (the same pattern as the DW-083 semantic cache and DW-085 routing policy classifier). Each golden-set case is sent as a user message to the provider; the response is scored by the case’s scorer (exact_match, contains, regex). Results are stored in analytics. The verdict is computed from stored eval results (pass rate, cost tiebreaker).

Dependency direction

ai depends on config only. The eval runner’s HTTP call reuses the same hyper_util client pattern as the DW-083 semantic cache and DW-085 routing policy (no new dependencies).

Structs

AutoPromotionConfig

AI-10 (#199): Auto-promotion configuration. When attached to an A/B test, the eval runner can automatically promote the winning variant when the pass-rate improvement exceeds the threshold.

AutoPromotionDecision

AI-10 (#199): The result of an auto-promotion decision.

CompiledAbTest

A compiled A/B test (DW-086): two or more variants with resolved targets. At request time, a deterministic weighted pick by request id selects a variant. Built at AiRuntime compile time from the config block; immutable once built.

CompiledAbVariant

One compiled A/B test variant (DW-086): the resolved model alias target, the optional prompt-version system message, and the variant’s weight and name (for analytics attribution).

EvalCaseResult

One eval case result (DW-086): the output of running one golden- set case against a provider.

EvalRunResult

The result of running one eval against one model (DW-086).

ExperimentDecision

The decision an A/B test made for one request (DW-086). The dataplane records this as a metric and an analytics assignment row. Carries the variant name and the model alias for attribution.

ExperimentVerdict

The verdict of an A/B test comparison (DW-086): computed from stored eval results. The variant with the higher pass rate wins; on a tie, the lower average latency wins (the cost tiebreaker — latency is a proxy for cost when the same provider serves both variants, and the actual cost is in the spend table).

Enums

EvalScorer

The scorer for an eval case (DW-086, AI-10/#199).

Functions

active_prompt_system

Resolve the active version for a prompt, considering runtime overrides. Returns the system message of the active (or overridden) version, or None when the prompt does not exist.

compute_verdict

Compute the verdict from a set of eval run results (DW-086). Each result is one variant’s eval run. The variant with the highest pass rate wins; on a tie, the lowest average latency wins; on a full tie (same pass rate AND same latency), the verdict is a tie (winner = None).

evaluate_auto_promotion

AI-10 (#199): Evaluate whether the winning variant should be auto-promoted based on the eval results and the promotion config. Returns a decision describing whether to promote and why.

run_eval

Run an eval against a model alias (DW-086). The provider_url is the full URL, auth_header is the optional Authorization header, provider_model is the provider’s own model identifier, and system_message is the optional prompt-version system message (from the eval’s prompt reference, resolved by the caller). Returns the per-case results.