Historical statistics charts rollout¶
Delivery reconciliation, 28 September 2026. The four observed-history consumers #3568/#3573/#3574/#3578 and the daily producer #3563 have merged. Their separate history gates remain default off in code. Merge does not prove a deployed operating schedule, retained observations, activated flags, authorized browser rendering or rollout acceptance; see STATUS.md for the programme record.
This is the concrete chart placement within the existing FEAT-024 phases, verified against main
9ddb8ec659bd63fa3b7b9fb65a05bad205471790 on 22 September 2026. It supplements the
technical plan and current status.
It plans the requested area charts; it does not enable new readers or approve new historical semantics.
Actual consumer inventory¶
| Surface | Exact current consumer | Verified current behavior / planned scope |
|---|---|---|
| Screening Overview, screening area | screening/screening-overview/screening-overview.component.ts → TimeProgressComponent |
Guarded, bounded observed project-screening history merged in #3568; unavailable points remain explicit |
| Stage Overview, screening area | stage/stage-overview/stage-overview.component.ts uses the project-screening history consumer |
It remains project-wide screening, not independent stage votes; #3573 reused #3568 |
| Stage Overview, annotation area | Same component → TimeProgressComponent |
Guarded selected-stage annotation observations with retained definitions merged in #3573 |
| Reviewer screening history dialog | MemberTimeProgressComponent |
Scoped, authorized reviewer screening observations merged in #3574 |
| Reviewer annotation history dialog | MemberTimeProgressComponent |
Paired stage/member observations and authorization merged in #3578 |
| Stage Review header and open progress dialog | stage-progress and StageOverviewDialogComponent current-progress views |
Current progress, not an existing historical area chart. Adding history here is a new placement decision; reuse the approved history reader if adopted |
The original null-selector and empty-mapper inventory described main on 22 September. The merged guarded readers now supply real retained checkpoint observations in those existing chart locations. Flag-off and unavailable history still require honest empty/gap handling; a FullStats refresh cannot reconstruct missing past checkpoints. The source layout and exact selectors have changed since the original inventory, so use the linked PRs for implementation detail.
Historical meaning and acceptance contract¶
Use real observed state at a checkpoint, as already required by FEAT-024. Each point answers “what was the authorized state when this checkpoint was captured?” It is neither a count of actions performed during that day nor a cumulative sum of daily event counts. Values may legitimately decrease after corrections, deletion, reopening, exclusion or changed configuration.
- Time: ordinary observation days are UTC, with the actual checkpoint capture time retained. Display the timezone explicitly. Local display formatting must not move points between UTC buckets. Midnight, DST and client-timezone fixtures are required. Bootstrap and repair points carry their own trigger/capture time; do not relabel them as midnight daily observations.
- Frequency versus accumulation: daily checkpoint state is the initial chart contract. Do not cumulatively sum already accumulated state totals. A future daily-throughput/cumulative-action chart needs an approved event definition and reliable event history; it is not a label toggle on this data.
- Screening: preserve project/study/reviewer decision identity and the retained agreement settings at each root. Include/exclude/disagreement/insufficient and overscreened cells must retain catalogue formulas and permitted overlaps. A Stage Overview screen chart uses the project scope unless a separate stage-attribution metric is approved. Never count the same project decision again per stage.
- Stage attribution: later correction can change a screening record's current StageId. That must not rewrite earlier retained checkpoints or fabricate historical stage ownership. Legacy creation timestamps cannot reconstruct overwritten decisions or original attribution.
- Annotation: retain the stage definition and target/grouping settings with each checkpoint. Separate no-session, incomplete/in-progress, completed and reconciliation states using catalogue distributions. Reserved/allocated capacity is not completed annotation. Reviewer views require authorized membership/stage scope; current permission governs access to retained data.
- Corrections/deletions: new checkpoints reflect changed state; immutable old observations remain unchanged. Removed stages/questions use retained definitions only when authorized. Do not substitute current configuration or current data for an unavailable historical root.
- Gaps, unchanged days and observations (#3636): every day is one of three states. Observed: a checkpoint root was published. Unchanged: the daily runner saw no clock-visible change since the latest checkpoint (see the limits below) and recorded an explicit "unchanged since checkpoint X" day; the chart carries that checkpoint's values forward and marks the point as carried forward (diamond marker, "unchanged; value carried forward" tooltip, table note). Gap: no observation and no unchanged record (the job did not run, failed, was disabled, or retention expired); render a gap, never zero and never a value carried from a neighbouring checkpoint. An unchanged day whose referenced checkpoint was reclaimed or is unavailable is a gap too: the API lists unchanged days only on an available root, matched by identity and build token.
- Verification state (#3636 part 2): every available point also carries
verificationState. Confirmed: every value was calculated from source (every checkpoint before part 2, and every checkpoint while its copy option is off), or a later drift check confirmed it. Unconfirmed: some values were copied from the current materialised rows and no drift check has confirmed them yet. Suspect: a drift check found the materialised rows disagreeing with source after this snapshot, so its copied values may be wrong; history is never recomputed. The chart marks an unconfirmed point with a triangle and a suspect one with a downward triangle, names the state in the tooltip, and the checkpoint table states it in words, so the state never depends on colour. Days carried forward from a checkpoint keep their diamond and share its state. Weekly/monthly retained observations retain their true dates; no smoothing or interpolation that implies additional observations. Stacked percentages need same-root denominators and explicit zero handling; reject incompatible series rather than mixing roots.
Current chart formatters call new Date(...); this alone does not establish the UTC-day contract.
Current percentage/decision toggles require formula and denominator tests before reuse.
Backfill, scheduling and provenance¶
A first backfill establishes the current authoritative baseline and an honest bootstrap checkpoint. It cannot reconstruct pre-enable daily states from surviving screening/session creation times. The production read-only audit confirmed those timestamps do not preserve every correction/save time. Historical data earlier than the first provable checkpoint is unavailable, not zero.
Every returned series must expose checkpoint identity, observed time, trigger, schema/catalogue version, retained configuration, scope and availability. Bound time range, page size, point count and response bytes. Use retained checkpoint/history infrastructure; never run one broad authoritative aggregation per plotted day or silently fall back to present data for missing past points.
Unchanged days (#3636 part 1). The default-off Quartz daily observation runner first compares the
project control's clocks with the latest checkpoint: when the current identity (source high-water
mark, committed projection revision and mode epoch) equals a retained root's identity, and that root's
configuration digest and source/catalogue versions still match the control, it writes one small record
to pmProjectStatisticsUnchangedDay (keyed by project and UTC day, naming the root's identity and build
token) instead of enumerating, calculating and building. That consumes no admission slot, build marker,
reference page or root, so it cannot reach the 512-marker or 256-root ceilings. It also refuses while a staged bulk
operation (import, bulk screening) holds a publication-guard fence, because those are admitted without
advancing the source clock, and when the families requested today differ from the families the root
references. The clocks, root and guards are re-read in the transaction that writes the record, and the
record's time is taken before that transaction starts; activity committed after the first check refuses
the record and the day falls through to an ordinary checkpoint.
Limits. An unchanged day means "no clock-visible change", not a recalculation. A write path that
changes data without advancing the clocks, or a period with statistics writes switched off, is invisible
to this check. That residual is what the #3636 part 3 drift check
catches, by recalculating from source regardless of the clocks; unchanged days carry the same trust as
the checkpoint they cite.
A requested family with no scopes (for example no searches) is absent from the root, so such projects
fall back to the ordinary path, which is conservative. Records are retained for the daily-retention window, at
most 128 per project, and are deleted when their checkpoint is no longer retained, by the bounded
history maintenance pass (at most 32 per pass). History responses add unchangedAtUtc to each available
point; nothing existing changed meaning. No new flag: this is part of the already default-off daily
observation path (ProjectStatisticsDailyObservations:Enabled) and history consumers.
Copied snapshots (#3636 part 2). When the default-off deployment option
ProjectStatisticsDailyObservations:CopyFreshMaterializedRows (environment variable
ProjectStatisticsDailyObservations__CopyFreshMaterializedRows) is true, an active day's ordinary
daily observation copies each scope's current materialised row instead of recalculating it from
source, but only when that row passes the serving equality gate right now. That is the same
ProjectStatisticsServingGate the bundle reader applies before serving a row: fleet and project gates
including the serving and allowlist flags, no source-visibility fence, compatible versions, family
requested and enabled, a Fresh publication guard with no active fence, the selected published row
Fresh with equal versions, configuration digest and write epoch, LastChangedRevision <=
CommittedProjectionRevision, and no active operation fence on the scope. The row must also carry the
versions and digest the family's own calculation would stamp, and the scope must still exist in the
Project the way the calculator decides it. Any scope that fails is calculated from source as before;
a Stale, Rebuilding, Missing, Incompatible, fenced or tombstoned row is never copied. Search
population (one document read) and derived summaries are never copied. Copied counters are normalised
and carry the same definition metadata a calculation would, so a copied observation equals a
calculated one whenever the row is correct.
The copy reads the fleet control, the project control, the guards, every row, every fence and the Project in one pinned snapshot. If that snapshot's identity (source high-water mark, committed projection revision, mode epoch) is not the admitted checkpoint identity, the build is abandoned as a lost race and retried at the newer boundary. Copied values carry the snapshot's boundary, so the builder's existing captured-boundary check refuses any calculated scope that saw a later write, and publication re-verifies every captured input. A snapshot therefore never mixes revisions. Admission, markers, staging, the configuration-digest refusal and publication are unchanged.
Before publication the builder writes one record per root to pmProjectStatisticsCheckpointVerification
(keyed by project and build token): the checkpoint identity, the copied scopes (per-scope provenance;
every other scope was calculated), the counts, and the state. A root with no copied scope is
Confirmed at birth; any copied scope makes it Unconfirmed. The record lives outside the immutable
root, pages and observations, so the part 3 drift check moves only its State field
(TryTransitionAsync, a compare-and-set) and never rewrites history. A root without a record reads as
Confirmed. History maintenance deletes at most 32 records per pass whose root no longer exists.
Observations are shared between roots, so provenance is recorded per root, never on an observation.
The option is configuration, not a runtime feature flag, for the same reasons as Enabled: it changes
how immutable history is produced, it is read by a background job rather than a request, and it must
stay fixed for a whole soak. Turning it off stops new copies only; existing Unconfirmed snapshots keep
their state until part 3 decides it.
IsAuthoritativeBuild is protocol, not provenance (#3727). A copied root is published through
the authoritative build protocol, so its build marker and root still carry IsAuthoritativeBuild =
true. That flag says which publication protocol admitted the root (captured source, projection and
mode-epoch identity, compare-and-set at publication); it does not say the values were calculated
from source. Value provenance lives only in the root's pmProjectStatisticsCheckpointVerification
record: CopiedScopes lists every copied scope, and the state is Unconfirmed until the drift check
decides it. The drift check, history consumers and any future reader must take provenance from that
record (a root without one was fully calculated) and must never read IsAuthoritativeBuild as proof
that values were calculated.
Definition changes that move counters (#3727). A copied row is only as good as the invalidation
of the writes that feed it. The one stage setting that changes stored counters without any Study
write is the stage's effective SessionCountTarget: Reviewer annotation evaluates
SufficientlyAllocated against it for every Study in the stage. Both stage-settings owners (the
guarded PUT .../review-settings and the legacy PATCH api/projects/{id}/stages/{stageId}) now
invalidate that family inside the Project save's own transaction when the effective target changes:
the source and client clocks advance, the family's write epoch advances and its guard becomes Stale,
so neither the reader nor the copy can pair counters from the old target with the new stage
definition, and an admitted daily build or rebuild that captured the old identity is refused. The
other stage settings do not change stored counters (Stage, Membership-stage and Domain reconciliation
count against the fixed session minimum of 2; MaxInProgress, HideExcludedStudiesFromReviewers and
ExcludedSessionStatsGrouping are applied at read time from the definition in the same bundle or
root), and an agreement-threshold edit is owned by the inclusion-recalculation fence and changes the
configuration digest the copy compares. See ProjectStageConfigurationChange for the field-by-field
rule.
A copier fault releases the build at once (#3727). An exception inside the copy (a transient
Mongo error, SnapshotTooOld, cancellation) abandons and cleans the admitted build before it
propagates, exactly as the typed refusals do, so the next attempt is admitted immediately instead of
waiting for the build lease to expire.
Enablement order for the copy (#3727). Enable CopyFreshMaterializedRows only after every API
replica and every Project Management replica runs a build containing #3725 and #3727. Older API
replicas ignore verification records, so they would show copied points as ordinary (unmarked)
history; older API replicas also change a stage's session-count target without invalidating Reviewer
annotation, so a copy could pair old counters with the new definition. Confirm the rollout has
completed (no older replica set still serving) before setting the option in any environment, and
keep it off during a rollback to an older binary. The fleet operations runbook carries the checklist:
enabling copied daily snapshots.
Not yet done: copy batching. The copy evaluates each scope's gate with its own row and fence reads inside one snapshot transaction. For a very large membership × stage project that is many round trips inside one snapshot; if the snapshot's lifetime is exceeded the copy fails and the build is released (above), but a retry of the same project can fail the same way. Batching the reads by family without splitting the one consistent snapshot is a tracked follow-up; until it lands, pilot the copy on projects whose scope count has been measured.
The periodic drift check (#3636 part 3). A default-off Project Management job
(ProjectStatisticsDriftCheck:Enabled, weekly by default) recalculates every allowlisted project from
source on its own cadence, regardless of the clocks and of unchanged-day records, because drift is
exactly what the clocks cannot see. For each enabled scope it compares the authoritative calculation
(ProjectStatisticsRoutingScopeCalculator) with the row the reader would serve now
(ProjectStatisticsServingGate.EvaluateScopeAsync), both normalised by
ProjectStatisticsCounterComparison.Normalize. The two halves are compared only when the calculation's
captured source and projection revisions and the fleet mode epoch equal those of the pinned snapshot
the served row was read in, and only when the calculation pinned its own snapshot (a non-null
snapshot boundary), so a write landing during the check is retried rather than reported as drift. The
failure is recorded only in the transaction that re-verifies the identity and the row and makes it
Stale. A scope the gate does not serve, that the calculator reports absent, or that is calculated under
other versions or another configuration is not compared and is remembered as unverified.
- Pass. The check's start identity becomes the project's last passing boundary (persisted with its
time in
pmProjectStatisticsDriftCheck). When an earlier passing check exists, every Unconfirmed snapshot after that boundary and at or before this one, bracketed by two passing checks, moves to Confirmed, provided every scope its record copied was compared and found in parity during the final visit. Evidence from earlier budgeted visits is discarded because source drift can bypass activity clocks between runs. The pass first claims the versioned state as confirmation-pending; an interrupted confirmation finishes before the next check starts. The first ever passing check confirms nothing, so snapshots from before it stay Unconfirmed. A pass needs at least one compared scope; scopes it could not compare do not stop it but are not verified. A check that compared nothing isUnverified: it confirms nothing and does not move the last passing boundary. - Caveat. Bracketing is strong evidence, not per-snapshot proof: if a copied row drifted and was then republished from source before the next passing check, that check passes and confirms the snapshot that copied the drifted value.
- Fail. The drifted served row is made Stale through the ordinary non-servable path, so reads fall back to the authoritative path at once. Nothing republishes it automatically: an administrator's family backfill or rebuild (or a fleet run) does. The same transaction advances the project's source clock, which ends any unchanged-day streak: the next daily observation builds a new root (calculating the Stale scope) instead of carrying the drifted root forward. Every Unconfirmed snapshot since the last passing check moves to Suspect; the failure does not move the last passing boundary.
- Confirmed snapshots are never touched and history is never recomputed; only the
Statefield of verification records moves, throughTryTransitionAsync. Afalseanswer means the record already moved and is ignored. Unchanged days are judged only through the root they cite.
Operator enablement, cadence, cost, telemetry and rollback are in the fleet runbook.
Measured locally (real Mongo, MeasuresSourceAggregationsWithTheOptionOffAndOn): for an active
project with 41 scopes across all nine physical families, the option off runs 41 per-scope
calculations and 23 legacy full-stats aggregations; the option on runs 1 calculation (the search
population scope) and 0 full-stats aggregations. Scope enumeration is unchanged in both modes.
Staging numbers remain to be measured.
The four observed-history charts request 20 checkpoints initially. Each “Show more observations”
action requests one further 20-checkpoint page using the API's nextCursor; the button disappears
only when the cursor is exhausted. Every request stays under the API's 50-checkpoint cap. New first-page
observations from polling or SignalR are combined with already loaded older pages by checkpoint identity.
When a refreshed first page no longer overlaps the loaded range, the chart resets that range and
uses the new first-page cursor so “Show more” cannot skip an unseen interval.
An overlapping head distinguishes a retention omission from a checkpoint that slipped into the
continuation by the API's source/projection/mode checkpoint order; observation timestamps may be
skewed and do not define cursor order.
Every loaded older page is rechecked in rotation, one page per successful head poll, for both
verification changes and retention compaction. A point first seen in the head joins that rotation
when overlapping newer heads push it into the continuation. New cursors join the back of the
rotation, so successive shifted heads cannot starve previously loaded pages. This eventually refreshes provenance
and removes compacted checkpoints, including Confirmed or Suspect points, without refetching every
loaded page in one burst. A maintenance read for a slipped head point updates only checkpoints
already visible; it does not reveal the rest of that page before “Show more” is clicked. Unaffected
loaded pages remain visible. When the deepest explicitly loaded page refreshes after retention
backfills it, its returned cursor becomes the next “Show more” boundary, so the next click advances
past the new visible rows. An error after a previously empty page shows the retry status without
simultaneously claiming there are no observations.
Changing the selected project, stage, reviewer or viewer discards previous pages and cancels pending
requests; authorization refusal clears them. Unavailable observations remain gaps, and nothing before
the first retained checkpoint is reconstructed. This UI correction uses the existing history consumer
flags; it does not introduce another activation gate or change retained observations.
On Screening and Stage Overview, each route owns its project-screening history subscription and its "Show more" cursor. Demand from mounted consumers shares one poll and SignalR stream; leaving the route, changing the project or viewer, losing Project.View or screening-graph permission, or turning off either existing history gate clears retained observations and cancels pending reads. The chart remains a presentation component and uses the same pager.
The daily producer and default-off Quartz daily/startup schedule with bounded same-day retry merged in #3563. Actual deployed execution, missed-day handling and published-result evidence remain gaps. A catch-up job must not stamp today's state as yesterday's observation; retry still-valid captured work or record a missed-day gap. The approved retention policy remains changed daily points for 90 days, real weekly observations through one year, real month-end observations through five years, and at most 256 checkpoints per project, subject to bounded storage and reachability rules.
Placement in the existing programme phases¶
| Programme phase | Required contribution / dependency | Reviewable delivery |
|---|---|---|
| 0 — catalogue and baseline | Record exact consumers above, formulas, observed-state semantics and honest backfill limits | This inventory repairs the current chart coverage gap; no runtime change |
| 1 — history foundation | Snapshot checkpoint roots/observations, read authorization, bounded response/retention, daily producer and recovery contract | Audit existing primitives; complete producer scheduling/retry/gap correctness before claiming durable daily user history |
| 2 — project screening | Transactional current screening and retained screening observations across submit/correct/import/delete/config changes | Existing family implementation is reused; prove snapshot parity and historical immutability |
| 3 — annotation families | Stage, member/reviewer, reconciliation/question lifecycle and retained definitions | Complete missing operational family baselines; prove mixed-state/config-change histories |
| 4 — remaining summaries | Coherent derived ratios/member views where requested | Existing search population/reviewer maintenance remain delivered; no restart of those implementations |
| 5.1 / 5.2 / 5.6 — consumer cutovers | Bounded authorized history API and real selectors; screening overview/project-scope stage screening first, stage annotation next, reviewer dialogs afterward | Each family/page is a separately reviewable flag-gated slice with empty/gap/error states and parity proof |
| 6 — fleet, soak, retirement | Scheduled observation/retention operations at fleet scale, load/soak, consumer replacement before legacy removal | No production enablement or collection cleanup without existing gates |
The API and selector migration belongs to phase 5, not an invented new “phase 1”. Supporting family/history correctness remains phases 1–3. The user's request now authorizes implementing phases 0–3 followed by 4–6. The minimal phase-5 history slice uses the existing area-chart locations and truthful checkpoint-state points. The epic/basic-history versus optional-UI distinction does not block those existing surfaces; richer analytics and new placements remain outside this slice.
Live refresh, flags and validation¶
A history-enabled page first reads authorized bounded history. A supported source change refreshes current state; it does not imply a new historical checkpoint exists. Refresh the historical tail when a root is published/retained history changes, using monotonic invalidation and authorized refetch. Deduplicate and ignore older revisions/responses, retain a trailing read, and recover on reconnect/resubscription plus bounded missed-event polling. Test two viewers on different API pods. Unsaved annotation drafts survive; chart refresh is independent of action eligibility and reservation revocation.
The existing materializedProjectStatisticsPages, ProjectOverview, StageOverview and SignalR
flags cover current-statistics slices, not historical consumers. Merged PRs #3568, #3573, #3574 and
3578 added separate default-off materializedProjectStatisticsHistory,¶
materializedProjectStatisticsStageHistory, materializedProjectStatisticsReviewerHistory and
materializedProjectStatisticsReviewerAnnotationHistory flags respectively. Prove flag dependencies,
per-project allowlisting, old-client behavior, rollback, authorized history and operating observation
capture before activation. Rollback restores the existing honest
no-history state, not synthetic points. Chart flags never change screening eligibility.
Required acceptance before each consumer activation:
- Exact count/bucket parity at captured roots for all screening modes and annotation states, authorized reviewer scopes and all existing decision/percentage toggles; one coherent retained denominator.
- Corrections, rescreen attribution, deletion, reopening, definition/target changes and source-only periods preserve old points and produce truthful new observations. Empty, disabled, missing, pruned, incompatible and never-enabled history must remain distinguishable.
- UTC midnight/DST/client timezone boundaries, sparse points, equal-time roots, revision ordering, pagination/cursors, bounded points/bytes, selected-stage changes and permission revocation.
- Reconnect, missed/duplicate/out-of-order events, simultaneous viewers across API pods, in-flight refetches and draft preservation. A progress poll passing is not actual SignalR transport proof.
- Fixed benchmark datasets and explicit history-endpoint/consumer p95 and read-budget measurements; preserve programme read/write gates and prove no per-day authoritative aggregation. Record actual storage/retention growth and seven-day/10,000-read/1,000-mutation rollout evidence as applicable.
The existing basic-history chart placements are implemented behind default-off gates. They still need the observation, parity, authorization, browser, retention and rollback acceptance above. Any additional Stage Review placement is a separate product decision.