Module rollup 

Rollup cascade and retention (DW-043).

Four granularities roll in a fixed cascade — raw → 1m → 5m → 1h → 1d — each stage aggregating the PREVIOUS stage’s completed windows, never raw twice (except by deterministic recompute, see below):

• A window is COMPLETE when its end plus a safety grace (writer lag headroom) is in the past; the grace means a straggler record flushed late still lands in the window it belongs to. Rows arriving later than the grace are documented as rollup-lost (they remain queryable in raw until raw retention expires).
• Each granularity keeps a CURSOR (exclusive ms frontier) in meta, advanced in the SAME transaction as the rows it covers: a crash mid-cascade never double-counts.
• Every aggregation is a wholesale window RECOMPUTE (INSERT OR REPLACE over the window’s full source range), so re-running any window — after a crash, a restored backup, or a cursor reset — reproduces byte-identical rows. Idempotence comes from determinism, not from merge arithmetic (upsert-merge would double-count on re-run).
• Custom-dimension rollups aggregate the same raw range in RUST (the source column is a JSON object; keeping JSON out of SQL avoids depending on the bundled SQLite’s JSON1 build flags).

Retention deletes expired rows per granularity (raw by ts_ms, rollups by window_start) and incrementally vacuums, which is the bounded-disk half of the done-when: pages return to the file system without a full VACUUM ever blocking the writer.

Constants

GRANULARITIES_MS

The four granularities, in cascade order (index = the gran column value stored in the rollup tables).

Functions

cascade_range

Aggregate the finer table’s [lo, hi) into the next coarser granularity, wholesale. Every stored column merges by SUM (or MAX for the max), which is exactly why the schema stores NON-cumulative bucket counts and additive sums.

cursor

Read a cursor (0 when never set).

roll_cascade

One cascade stage: aggregate granularity from’s completed windows into granularity to (from + 1). Wholesale recompute per coarse window, cursor-guarded exactly like the raw pass.

roll_raw_range

Aggregate raw [lo, hi) into rollup_fixed/rollup_dim at granularity 0, wholesale over the range (idempotent recompute). Split from roll_raw_to_1m so tests can drive exact ranges.

roll_raw_to_1m

One raw→1m pass: aggregate raw rows in [cursor, frontier) into completed 1-minute windows, recompute-style. frontier must be the end of the last COMPLETE window (the caller computes it with the grace). Returns the number of 1m windows written.

sweep_retention

Retention sweep: delete expired rows per table and incrementally vacuum up to vacuum_pages pages. For the raw table, only intra-day rows older than the retention are deleted (SCALE-09 #188: daily partition tables are dropped by partition::drop_expired_partitions, which is O(1) DROP TABLE vs O(n) DELETE). Returns rows deleted.

window_of

Window start (floor to granularity) of a timestamp.