Analytics

Observability answers "is it healthy right now" (see Operations); analytics answers "what happened over time, to whom, and why". The embedded analytics store gives a single-instance gateway a durable, queryable traffic history with zero external dependencies and bounded disk usage.

When to use this

The embedded analytics store gives a single-instance gateway a durable, queryable traffic history with zero external dependencies — useful for usage reporting, billing, and post-incident review. For multi-instance fleets or warehouses that want the raw firehose, use the analytics stream instead.

Enabling

Add an analytics block to the gateway config and restart (the database opens at startup, like the listener bind set):

analytics:
  path: /var/lib/dwara/analytics.db
  flush_ms: 1000              # optional, default 1s, 100..=60000
  retention:                  # optional; defaults shown
    raw_ms: 86400000          # raw records: 24h
    m1_ms: 172800000          # 1-minute rollups: 48h
    m5_ms: 604800000          # 5-minute rollups: 7d
    h1_ms: 2592000000         # 1-hour rollups: 30d
    d1_ms: 31536000000        # 1-day rollups: 365d
  dimensions:                 # optional custom dimensions
    - name: plan
      header: x-plan

The database is a separate SQLite file (an embedded SQL database stored as a single file) from the state store (DWARA_STATE_DB) on purpose: retention deletes and vacuum churn never touch the identity store. Without the block, the gateway runs exactly as before — the admin analytics endpoints answer 404, and request recording is a no-op.

Every completed request is recorded fire-and-forget: the gateway never waits on analytics (a saturated analytics pipeline drops and counts records rather than adding so much as a lock to the request path).

Custom dimensions

dimensions[] tags requests with values from request headers — x-plan: pro becomes the analytics dimension plan=pro — which then group and aggregate like the built-in dimensions (listener, route, upstream, consumer, method, status class). The first value of a repeated header wins; non-UTF-8 and over-128-byte values are skipped. Dimensions never appear in the access log.

The rollup model

Raw records live briefly (default 24 h) and feed a fixed cascade of pre-aggregated rollups (pre-aggregated summaries at fixed time granularities) — 1 minute, 5 minutes, 1 hour, 1 day — each holding request/error/rate-limit/shed counts, duration sum/max, and a 13-bucket latency histogram (counts observations into buckets for percentile estimation) per dimension tuple. Because every stored number is additive, any time range at any granularity merges exactly, and percentiles (e.g. p95 = the latency 95% of requests are under) are estimated from the merged histogram. Retention is enforced per granularity with incremental vacuum (SQLite's space-reclaim operation), so disk usage is bounded by configuration, not by traffic.

Querying

All endpoints live on the admin API (mTLS; the dev fallback works too) and answer 404 when analytics is not configured.

Dashboard series — per-window metrics with drill-down and filters:

GET /analytics/dashboard?from_ms=1690000000000&to_ms=1690086400000&gran=2
    &group_by=consumer&route=checkout

gran is 0..=3 (1m/5m/1h/1d); group_by and the equality filters (listener, route, upstream, consumer, method, status_class) are optional. Each point carries requests, errors, error_rate, rate_limited, shed, avg_ms, p50_ms, p95_ms, p99_ms.

Top-N reports — the five built-in rankings:

GET /analytics/top?kind=slowest&from_ms=...&to_ms=...&n=10

kind is one of consumers, routes, slowest, error_prone, rate_limited.

Structured query — a closed JSON grammar translated to parameterized SQL (SQL with placeholders, never string-interpolated — injection-safe) (SQL text is never accepted):

POST /analytics/query
{
  "from_ms": 1690000000000,
  "to_ms": 1690086400000,
  "gran": 1,
  "group_by": ["consumer", "status_class"],
  "filters": { "route": "checkout" },
  "limit": 100
}

group_by accepts exactly the six dimension columns. A week of traffic answers in well under 100 ms — the query reads rollup tables, never raw records.

Usage reports and exports

An exports block inside analytics turns the store into a scheduled reporter: once per configured window the gateway writes a per-consumer usage statement — requests, errors, error rate, rate-limited and shed counts, average latency, and quota budget figures — as one deterministic file per format. Durable input for a billing pipeline, distinct from the ad hoc query API. The quota figures come from consumer budgets (see Consumer quotas); live per-window counters are also readable via GET /quotas/usage on the admin API.

analytics:
  path: /var/lib/dwara/analytics.db
  exports:
    directory: /var/lib/dwara/usage-reports
    window: daily            # hourly | daily | monthly; default daily (UTC)
    formats: [csv, json]     # default both

The directory is created on demand. Files are named dwara-usage-{window}-{utc-stamp}.{ext} — for example dwara-usage-daily-2026-08-29.json — and a re-export of the same window simply overwrites the file, so output is idempotent and safe to re-run. A background worker checks every 30 seconds and exports each closed window about 5 minutes after it closes (so rollups settle); after a restart it backfills missed windows oldest-first. Reloads apply live. Validation rejects an empty directory and duplicated formats — an omitted or empty format list simply means both; Parquet (a columnar storage format for analytics) is not offered yet.

The statement's numbers are the query API's numbers: the export runs the same aggregation as POST /analytics/query (grouped by consumer, plus a totals row), so a statement always reconciles with an ad hoc query for the same period.

Quota columns follow a strict alignment rule: a budget's used/limit figures appear only when its quota window fully contains the export window — a daily report carries the same-day daily counter plus the month-to-date monthly counter, a monthly report carries only the monthly counter. Monthly figures are month-to-date as of generation time (the store's live counter, so a re-exported window can differ from the original run); each budget's window_start_epoch_s and reset_epoch_s bound what used covers. Empty cells mean "no applicable budget", never zero.

Two admin API endpoints (they answer 404 when analytics is not configured):

GET /analytics/exports?limit=25
POST /analytics/exports/run   {"window": "daily", "window_start_ms": 1770000000000}

GET /analytics/exports returns the run ledger, newest first (limit 1..=100, default 25): one record per exported window with status (ok/failed), the partial flag, formats, and consumer/request counts. POST /analytics/exports/run triggers one export by hand; both body fields are optional (defaults: the configured window kind and its most recent closed window), and the endpoint rejects a misspelled window or a window_start_ms that is unaligned or not yet closed.

A window older than the rollup retention may undercount without the store being able to know; such statements carry "partial": true so a billing consumer can reject them, and the scheduler never auto-exports past that horizon — only a manual trigger can, still flagged. Windows with no traffic are not loss: counts are exact whenever partial is false.

Live sketches

The durable store answers "what happened"; live sketches answer "what is happening right now" with sub-second freshness. An opt-in live_sketches block adds in-process, hand-rolled aggregation with no new dependencies:

analytics:
  path: /var/lib/dwara/analytics.db
  live_sketches:
    enabled: true               # default true when the block is present
    freshness_target_ms: 500    # rolling window size; 100..=5000, default 500

A per-route rolling window (size = freshness_target_ms) is maintained synchronously on the request-completion hot path — request/error counts and capped latency samples. Percentiles (p50/p95/p99) are computed at snapshot time, never on the hot path. The update never blocks the dataplane.

GET /analytics/live

Returns a JSON snapshot of all routes' current windows: request count, error count, error rate, p50/p95/p99 latency, and the window's time bounds. When the block is absent, the endpoint answers 404 (analytics_not_configured).

ML insights

An opt-in insights block adds hand-rolled ML traffic insights over the live sketch window rotations — no ML crates, no external services:

analytics:
  path: /var/lib/dwara/analytics.db
  live_sketches:
    freshness_target_ms: 500
  insights:
    forecast: true              # capacity forecasting; default false
    anomaly_baseline: true      # anomaly detection; default false
    baseline_windows: 1440      # 24h of 1-minute windows; default 1440

The engine maintains two models: an EWMA (exponentially-weighted moving average) over recent window counts for the short-term trend, and a minute-of-day ring buffer (1440 entries) for the seasonal baseline.

GET /analytics/forecast

Returns a prediction for the next window — the seasonal average for the upcoming minute adjusted by the EWMA trend — with a confidence score. Requires forecast: true.

GET /analytics/anomalies

Returns the current window's anomaly status: an is_anomalous boolean and a score (0..1) from comparing the current window's shape to the seasonal baseline. Requires anomaly_baseline: true.

Both endpoints answer 404 (analytics_not_configured) when the insights block is absent or the relevant capability is disabled.

Business metrics dimensions

In addition to the header-derived custom dimensions described in Custom dimensions, routes can attach arbitrary key-value business dimensions to every access record they produce. These are configured per-route via a dimensions block in the route config:

routes:
  - name: checkout
    match: { path: "/checkout" }
    dimensions:
      product_line: electronics
      channel: web

Each key-value pair is stored on the access record alongside the built-in and header-derived dimensions. Business dimensions appear in query results and are accepted in group_by by the structured query endpoint, so you can slice traffic by product_line or channel exactly as you would by route or consumer. The same additive-rollup model applies: any time range at any granularity merges exactly.

Federated analytics (Enterprise)

In a CP/DP split (see CP/DP split), each edge records analytics locally just as a single-instance gateway does. The controller aggregates every edge's data into one store so that admin API queries return fleet-wide numbers.

• Edge-side: a FederatedAnalyticsSink (an implementation of the AnalyticsSink extension trait) batches completed-request events and streams them to the controller over gRPC via the PublishAnalytics RPC. Batching is bounded; a saturated stream drops and counts rather than blocking the request path.
• Controller-side: an AnalyticsCollector aggregates all edges' streamed data into a single analytics store. The standard admin API analytics endpoints (see Querying) then answer fleet-wide data -- a query with no edge filter returns the aggregate across every edge.
• Per-edge filter: an optional ?edge=<edge-id> query parameter narrows a query to a single edge's records.

This is an enterprise feature, gated behind the ent cargo feature and a valid license. In an OSS build the FederatedAnalyticsSink is inert.

Runnable demo

Query a live analytics store: demos/06-observability/ in the repository. Test scripts exercise the mTLS admin dashboard query (test-04-analytics-query.sh) and a custom-dimension rollup keyed on the X-API-Version request header (test-05-analytics-dimensions.sh). The category README covers prerequisites and teardown.

Federated analytics specifically: demos/11-enterprise/ in the repository (test script: test-07-federated-analytics.sh) documents the enterprise edge-to-controller streaming and verifies the embedded local analytics store.