Skip to content

ADR-019: Single-document point saves with an asynchronous statistics fold

Context

The recovered-design reconciliation chose to update materialized statistics in the same MongoDB transaction as each Study point save (its "Primary update path" row). It left the contention threshold as an open evidence gate (item 5 of "Decisions still requiring Phase 0 evidence").

That evidence came in with PR #3855, the idle-host write benchmark of 30 September 2026:

  • Command cost. A materialized screening save issued 25 commands inside a multi-document snapshot transaction. A source-only save issued 2. Every cell failed the under-10% p95 gate, by 46% to 1,283%.
  • Contention. Every save writes shared per-project statistics documents: the control row, the project-scope current row, the publication guard and the outbox slots. At ten reviewers on different Studies, 399 of 1,000 materialized submissions exhausted their retries; source-only had none.

Decision

Replace the same-transaction point update for every screening- and annotation-dependent family with the design in async-point-fold-design.md. It sits behind the default-off flag materializedProjectStatisticsFold and a durable per-project FoldMode.

  1. Single-document pending entries.
  2. The save. A point save appends a compact pending entry (the classified transition plus any invalidation intents) to the Study in its existing single-document write. It writes no statistics document and uses no statistics transaction.
  3. The fold. A leased per-project worker in project-management folds entries into the rows, in batches of at most 32, in a transaction only it runs.
  4. Reads. Readers serve stored rows plus pending entries from one pinned snapshot, so values are exact before the fold.
  5. Rebuilds publish authoritative(B) - pending(B).
  6. Worker outages never fail a save: readers fall back past age and size thresholds.
  7. Storage-version tripwire. A fold-mode project's control carries StorageVersion = global + 1000 + FoldProtocolVersion. Every pre-fold binary already treats that as an incompatible project, so it fails closed. Rows and history never carry a fold marker. A cluster-gitops allowlist guard, bracketed by the durable narrow gate, covers the paths that do not check storage versions during a rollback.
  8. Protocol compatibility (N-1). Entries, the control stamp, heartbeats and fleet membership carry a fold protocol version.
  9. From the production baseline P0, a binary at N reads, folds and rebuilds N and N-1 entries, writes only the stamped protocol, and quarantines older entries.
  10. The stamp advances by an owned compare-and-set once every live member declares N.
  11. Only additive changes use the N-1 window. A derivation-changing change is a breaking bump that needs a reset of the affected families.

Owner decisions recorded (Chris, 30 September and 1 October 2026):

Decision
a MVP scope. All screening- and annotation-dependent families migrate in one MVP. ReviewerAnnotation and QuestionAnswers stay Stale-on-save (invalidation intents) and are served authoritatively
b Write gate. p95 overhead under 10% or at most +2 ms, and zero statistics-caused save failures or exhaustion at 1, 2, 5 and 10 reviewers, in same-Study and different-Study cells
c Version bump. The fold bumps the Study's Audit.Version when it removes entries
d Reviewer-tracking mode. It is read just before the write. A future mode-switch owner must add a writer grace period
e Per-stage reviewer tracking is deferred to #3876
f Rollback. The runbook order is followed and the allowlist GitOps guard, which existing binaries honour, is applied
g Staging pilot. Staging-only enable after slice 2, behind a positive IsStaging() allow setting. Production waits for the full MVP
h N-1 support from production P0. Between MVP slices the staging pilot resets instead

Consequences

  • The hot path. Point saves keep today's source-level shape: one Study write plus the existing non-statistics documents. Statistics add only two read-only control reads and a larger document. Gate (b) is evaluated by the benchmark's new fold arm.
  • New durable state:
  • Study.PendingStatistics and StatisticsFoldSequence;
  • control FoldMode and protocol fields;
  • per-protocol fold-worker heartbeats on the global control;
  • pmProjectStatisticsFleetMember;
  • pmProjectStatisticsFoldQuarantine (30-day TTL);
  • a partial index on pmStudy, built as a separate production operation.
  • History. History stays free of pending sets. Copies need an empty pending set, and authoritative builds commit an observation barrier.
  • Rollback requires the documented runbook. A rebuild of every family is required after any guard window.
  • Classifier fixes after production activation require a breaking protocol bump and a reset of the affected families.
  • Server version. MongoDB 4.4 or later is required.
  • Retrying a lost Study upsert. Once a fold can bump a Study's version between a writer's read and its save, a cached writer's version-guarded upsert can lose with E11000. The two retry shapes are not interchangeable:
  • StudyUpsertConflictRetry is for a whole-aggregate save that recomputes its change from a fresh uncached read. It reloads, re-applies and saves again, gated on the fold flag. A reload that differs only by the fold's own bump is retried without charge, up to a small cap. Use it for new cached writers (presence, idle-session and liveness consumers).
  • ReservationChangeSave.TrySaveAsync is for a reservation change that must commit with its statistics half in one transaction. It surfaces the lost race as ReservationChangeConflictException for its caller's own retry. Keep reservation and capacity paths on it; StudyUpsertConflictRetry also accepts that exception, so it can wrap such a call.

References