Screening source-write development benchmark

ProjectScreeningWriteBenchmark measures the existing capacity-save source path with materialized writes off against the six-argument source-plus-statistics transaction with writes enabled. Both use real MongoDB replica-set source storage. The enabled path includes the durable-mode snapshot read, operation/receipt admission and real screening writer/coordinator. Each request owns a separate repository/cache and reloads and reclassifies after an optimistic conflict, with the API's three-attempt limit (ReviewController.TrySaveScreeningAsync, maxAttempts = 3, immediate reload with no delay). The harness uses GetAsync for its known surviving fixtures where the controller uses GetOrDefaultAsync and also handles deletion, capacity and duplicate-operation outcomes. It does not replay the mutable aggregate inside a transaction retry.

Current ReviewController also resolves durable tracking mode on each attempt and chooses the plain save path when tracking is off. This benchmark deliberately retains the capacity-save repository overloads with tracking off; its three-attempt limit matches the controller, but its measured path and command counts do not represent that current HTTP branch. Reviewer membership/authorization, controller mode resolution and event dispatch are outside this fixture. Integrating reviewer maintenance code does not enable its flags in this ProjectScreening-only benchmark.

The minimum corpus, PS-WRITE-01, contains one screening project and ten studies. Each matrix cell uses 1, 2, 5 or 10 reviewers submitting to the same study or separate studies. Later rounds alternate inclusion/exclusion decisions by the same reviewers. The artifact's corpus counts describe maximum matrix dimensions, not a claim that every cell has ten active reviewers. This is deliberately not a large-project PS-DS-02 benchmark. Scale values other than 1 are refused.

Reproduce

From the PR worktree root:

SYRF_STATS_DATASET=PS-WRITE-01 SYRF_STATS_ITERATIONS=100 SYRF_STATS_WARMUP=10 \
SYRF_STATS_RESULTS_DIR=/tmp/syrf-screening-write-results \
dotnet test src/libs/project-management/SyRF.ProjectManagement.Mongo.Data.Tests/SyRF.ProjectManagement.Mongo.Data.Tests.csproj \
  --filter FullyQualifiedName~ProjectScreeningWriteBenchmark

Command counting is opt-in. Add SYRF_STATS_COUNT_ROUNDTRIPS=1 to the command above for the command breakdown; leave it unset for a no-subscriber control timing. Without the dataset variable, ordinary CI skips this benchmark. The existing replica-set fixture owns and removes its mongo:8.0 container. Existing shared-runner resource limits apply when SYRF_TEST_JOB_KEY is set; the artifact records that policy and the runtime/server environment. No deployed database, credentials or live flags are needed.

What the harness now records

Following the write-overhead diagnosis (screening-write-overhead-diagnosis.md, #3255), three things changed in what this harness reports. The artifact schema is extended, never renamed, so an existing reader of Durations, Notes or the *_outcomes keys is unaffected.

• Command counting is opt-in via SYRF_STATS_COUNT_ROUNDTRIPS. A CommandStartedEvent subscriber forces the driver to materialise every command document, and the materialised arm issues many times more commands than the source-only arm, so the recorder's own cost is asymmetric between the two arms being compared. Running once without it gives a no-subscriber control timing. When it is off, commands and commit_attempts read not_counted rather than zero, and no RoundTrips record is emitted.
• Commands are broken down by driver command name, in the artifact's existing RoundTrips field, so a reduction is attributable to a specific command class rather than only to a batch total. The per-name counts are windowed exactly as the total is, so they sum to commands.
• Per-attempt command counts are recorded in the *_commands_per_attempt note, for the single-reviewer cells only. With two or more reviewers the submissions run concurrently against one shared counter, so a per-attempt delta would include another reviewer's commands; the artifact says so rather than publishing a figure it cannot support.

The duplicate pre-check's assertion also moved outside the timed window. ResolveByReceiptAsync itself stays inside it, because production runs the same pre-check before opening a transaction.

Evidence and interpretation

The existing BenchmarkResults format records p50/p95 submission duration and repository-attempt duration. Submission time includes source reload, envelope/receipt reads, classification and retries. It excludes seeding, assertion queries, HTTP middleware, assignment, event dispatch and background work. The source-only baseline performs no statistics admission reads. Candidate/baseline ordering alternates by reviewer count. Each variant has its own warmup.

Every batch checks exact persisted decisions against successful submissions, and enabled batches also check exact authoritative/projection parity, operation receipt/delta presence, committed projection revision advancement and the coalesced outbox's client revision. Exhausted retries are reported as failures to save. A repeated decision can save source without a classified delta; these saves are counted separately and correctly have no statistics receipt/delta. There is no silent exception-to-success conversion. Driver command monitoring counts all commands during submissions, commit attempts and UnknownTransactionCommitResult failures; it excludes seeding and post-batch assertion queries. There is no injected unknown-commit failure in this timing run.

The p95 overhead gate is strictly less than 10%. Its result is emitted for each matrix cell, never asserted as a CI correctness test on a shared development host. All-submission latency includes exhausted requests, so a lower p95 with a worse success rate cannot establish a usable performance improvement. The artifact includes saved, conflict, exhausted and no-delta counts so that tradeoff remains visible. Maximum serialized summary bytes are also recorded.

This slice keeps active-reviewer tracking off and enables only ProjectScreening maintenance. It does not establish reviewer-family maintenance overhead, reservation/assignment throughput, annotation write cost, non-capacity API latency, large-project capacity, live rollout consistency or production acceptance. Those required programme measurements remain ordered follow-up evidence. Existing deterministic replica-set tests separately prove the non-capacity transaction conflict response and rollback.

Development result: 5 September 2026

The 100-iteration artifact records 10 warmup batches per variant. All exact correctness assertions passed. Every p95 overhead gate failed. These measurements leave the performance acceptance gate open.

Target	Reviewers	Baseline p95 ms	Materialized p95 ms	Overhead	Baseline saved	Materialized saved
Same study	1	8.57	64.64	654%	100/100	100/100
Same study	2	11.24	47.56	323%	200/200	117/200
Same study	5	20.35	44.77	120%	300/500	141/500
Same study	10	28.90	44.55	54%	329/1000	146/1000
Different studies	1	6.19	48.65	686%	100/100	100/100
Different studies	2	6.14	94.55	1441%	200/200	200/200
Different studies	5	7.71	80.96	950%	500/500	350/500
Different studies	10	21.89	134.89	516%	1000/1000	598/1000

The different-study result exposes contention on shared project statistics despite independent source records: 1,406 optimistic conflicts and 402 exhausted submissions in its ten-reviewer candidate cell. No unknown-commit failure was observed. Summary documents remained at most 1,982 bytes in this small corpus. The artifact's no_delta_saves field counts successful source saves without a statistics transition, so it includes every source-only baseline save; it is not a count of unchanged source data.

Required continuation #3255 owns the bounded write-path performance and contention improvement before claiming activation readiness. Increasing retries alone would trade exhausted requests for more load and latency; it does not establish the required overhead threshold. Re-run this matrix after the chosen change and add the required reviewer-family and active-tracking variants as their writers become available. The present slice does not silently expand maintenance or alter the existing three-attempt source contract to make its numbers pass.

Dependency integration measurement: 13 September 2026

After merging main and reviewer-maintenance prerequisite #3297 at 0803a4ec5, the opt-in smoke run (3 measured batches, 1 warmup) passed. A separate 30-batch artifact then ran all eight cells with five warmup batches per variant against disposable MongoDB 8.0.28 on .NET 10.0.0. Each cell still has one project and ten studies, and only ProjectScreening is enabled. The measured total is 1,080 submissions per variant; warmup is excluded. Every source/projection, receipt/delta and outbox correctness assertion passed. No unknown-commit failure was observed.

All eight overhead gates failed again. Thirty batches are a modest local measurement, not a large-project, current-controller, staging or production acceptance result. The historical 100-iteration artifact above remains unchanged. This run records that dependency revision; it does not establish a comparable host-controlled trend against September 5.

Target	Reviewers	Baseline p95 ms	Materialized p95 ms	Overhead	Baseline saved	Materialized saved
Same study	1	11.10	84.44	660.79%	30/30	30/30
Same study	2	24.68	57.83	134.28%	60/60	36/60
Same study	5	26.49	52.90	99.70%	90/150	46/150
Same study	10	30.73	60.30	96.22%	97/300	44/300
Different studies	1	8.40	62.83	647.97%	30/30	30/30
Different studies	2	8.46	93.91	1010.16%	60/60	60/60
Different studies	5	11.36	96.04	745.78%	150/150	105/150
Different studies	10	14.60	86.96	495.69%	300/300	176/300

Baseline exhausted 263 of 1,080 submissions after 1,005 total optimistic conflicts; materialized exhausted 553 after 1,883 conflicts. Exhaustion is a failed save, never counted as success. During measured submissions the driver recorded 4,649 baseline commands versus 28,104 materialized commands, including 527 materialized commit attempts. Those command counts exclude assertion queries and seeding. The higher command count and contention are observed diagnostic evidence; this slice introduces no unmeasured partitioning or write-protocol changes to chase the threshold.

Reproduce this sample with the command above using SYRF_STATS_ITERATIONS=30 and SYRF_STATS_WARMUP=5. Focused integration validation additionally passed 67 tests covering durable mode saves, optimistic contention and reviewer materialization; with no dataset variable the benchmark was skipped as intended. Performance follow-up remains #3255, and rollout stays gated by the failed write-overhead criterion and the separate live acceptance requirements.

The later integration of #3297 at 266a8dac3 includes Sonar fixes, legacy null-list recovery and a real Project-document admission write for reviewer maintenance. The recorded artifact predates those fixes. Reviewer maintenance remains disabled in this benchmark; measuring its additional write and contention requires a separately enabled workload before rollout acceptance. No fresh performance measurement is claimed by this integration.

Paired run 2026-09-15 (loaded host)

main at 0fe1716f6 and #3475 at 6546bb25d were run back to back, same host, same minute, SYRF_STATS_ITERATIONS=30 SYRF_STATS_WARMUP=5 SYRF_STATS_COUNT_ROUNDTRIPS=1, MongoDB 8.0.28 on .NET 10.0.0. Artifacts: main arm, #3475 arm.

Both the pre-#3475 main arm and the post-#3475 candidate arm ran on the same loaded host; neither was an idle-host fixture. Load average was 20-45 with roughly 130 CI containers running throughout both arms. This measurement therefore fails the hygiene rule this programme set for itself — "run unchanged main and the candidate back to back on the same idle host" — in one specific respect: the p95 columns below are indicative only and no ratio derived from them may be quoted. The command, conflict, exhaustion and saved counts are deterministic work counts rather than timings and are the reliable part of this run.

Cell	Commands (main → #3475)	Conflicts	Exhausted	Saved	Materialised p95 ms	Source-only p95 ms
same_study 1	900 → 630	0 → 0	0 → 0	30 → 30	118.72 → 56.53	58.55 → 12.62
same_study 2	1390 → 1090	82 → 80	25 → 25	35 → 35	95.67 → 53.62	13.29 → 17.18
same_study 5	2812 → 2553	343 → 343	111 → 111	39 → 39	84.33 → 56.52	15.73 → 26.41
same_study 10	5196 → 4974	779 → 770	251 → 246	49 → 54	58.99 → 71.41	41.99 → 30.72
different_studies 1	900 → 630	0 → 0	0 → 0	30 → 30	47.91 → 42.95	6.44 → 8.68
different_studies 2	2612 → 1860	58 → 60	0 → 0	60 → 60	103.98 → 75.32	38.86 → 5.90
different_studies 5	5090 → 3779	193 → 193	44 → 43	106 → 107	103.06 → 93.50	7.31 → 5.85
different_studies 10	9296 → 7142	434 → 428	125 → 118	175 → 182	94.07 → 98.10	9.94 → 10.15

What this establishes

• The command reductions are real and match the unit measurement. The uncontended cells fall 900 -> 630 over 30 submissions, i.e. 30 -> 21 commands per save, exactly the figure ProjectScreeningWriteRoundTripBudgetTests asserts. The candidate artifact's per-command-name breakdown for that cell is find 390, update 120, insert 60, findAndModify 30, commitTransaction 30 — 13 finds, 4 updates, 2 inserts, 1 findAndModify and 1 commit per save — and its per-attempt note records 16 commands inside the repository save, leaving 5 before the transaction.
• The contended cells drop by more in absolute terms, because a conflicting attempt also does less wasted work: 2612 -> 1860 at two different-study reviewers, 5090 -> 3779 at five, 9296 -> 7142 at ten.

What this does not establish

• A4's contention effect is not demonstrated. Conflicts and exhaustion are essentially unchanged (58->60 / 0->0; 193->193 / 44->43; 434->428 / 125->118). Ordering the contended writes first was expected to cut the work a loser does, not the number of losers, and that is what the counts show — a loser still loses. The small movements in saved (175 -> 182 at ten different-study reviewers, 49 -> 54 at ten same-study) are within the noise of a loaded host and are not claimed as an improvement.
• No latency claim. Materialised p95 is lower in six of the eight cells and higher in two, but the source-only p95 of the same cell moved by as much as 6.6x between arms (different_studies 2: 38.86 ms on main against 5.90 ms on the candidate) purely from host load. That is why the artifacts' own *_p95_gate overhead percentages are not reproduced here: under this load they measure the scheduler, not the change.
• Every one of the eight <10% overhead gates still fails, in both arms. Nothing here moves the performance acceptance gate.

At the time, an idle-host re-run was owed before any rollout claim; it was run on 2026-09-30 (see Idle-host run 2026-09-30 below) and the gate still fails. The defensible statement about this change remains the one the command counts support: nine fewer round trips per conflict-free save, and proportionally less wasted work per losing attempt.

Idle-host run 2026-09-30

main at 65ae1099b, MongoDB 8.0.28 on .NET 10.0.0, PS-WRITE-01 at SYRF_STATS_ITERATIONS=100 SYRF_STATS_WARMUP=10, exactly the Reproduce command above. Three sequential runs from one scratch worktree, one build:

• (a) timing, no command counting: artifact.
• (b) command count, SYRF_STATS_COUNT_ROUNDTRIPS=1: artifact.
• © repeat of (a), for run-to-run variance: artifact.

Reviewer maintenance is not an arm and there is no env toggle for it. The harness fixes ActiveReviewerTrackingEnabled = false and toggles only MaterializedProjectStatisticsWrites between the two arms, so no run of this benchmark measures enabled reviewer maintenance. STATUS asks for it; that measurement still needs a separately enabled workload and is not claimed here.

Host conditions

The machine is also the CI runner host (48 CPUs). The doc's idle threshold is load per CPU at most 0.5.

Run	Load average at start	Peak 1-minute load during/after	Peak load per CPU	Docker containers running
(a) timing	4.42	7.88	0.16	56-58
(b) counting	7.53	10.75	0.22	57-58
© repeat	7.78	14.85	0.31	57

Load stayed below the threshold in every snapshot, so no waiting was needed. About 57 CI containers were present throughout but mostly idle; the load-average figures are the evidence, not a claim of an empty host. The 15 September runs above had load 20-45.

Results (run a)

Overhead is materialized p95 over source-only p95 minus one. The gate is strictly under 10%.

Target	Reviewers	Source-only p50 ms	Source-only p95 ms	Materialized p50 ms	Materialized p95 ms	Overhead	Gate
Same study	1	4.08	7.76	37.51	61.21	689%	FAIL
Same study	2	5.75	11.12	31.40	43.85	294%	FAIL
Same study	5	10.99	19.07	25.63	43.95	131%	FAIL
Same study	10	18.87	30.20	28.80	44.24	46%	FAIL
Different studies	1	4.79	5.82	35.40	39.31	575%	FAIL
Different studies	2	4.90	5.94	42.82	81.20	1268%	FAIL
Different studies	5	4.70	5.92	43.89	81.84	1283%	FAIL
Different studies	10	5.15	13.88	43.54	84.29	507%	FAIL

Run-to-run variance (overhead per cell in runs a, c and the counting run b, whose subscriber adds its own asymmetric cost): same study 689/611/596, 294/293/289, 131/128/116, 46/51/53; different studies 575/662/766, 1268/1426/1438, 1283/1366/1167, 507/1088/570. The verdict is stable; the individual percentages are not, especially the source-only p95 in the low-millisecond cells (a 1 ms shift in a 5-6 ms baseline moves the ratio by 15-20%). Do not quote a single percentage to more than order of magnitude.

Outcomes and command counts

Saved, conflicts and exhaustion are from run a; commands are from the counting run b (100 batches, so submissions per cell are 100 times the reviewer count).

Target	Reviewers	Source-only saved	Materialized saved	Source-only conflicts / exhausted	Materialized conflicts / exhausted	Source-only commands (per submission)	Materialized commands (per submission)
Same study	1	100/100	100/100	0 / 0	0 / 0	200 (2.0)	2500 (25.0)
Same study	2	200/200	116/200	100 / 0	268 / 84	700 (3.5)	4143 (20.7)
Same study	5	302/500	142/500	896 / 198	1125 / 358	3300 (6.6)	9172 (18.3)
Same study	10	310/1000	151/1000	2376 / 690	2612 / 849	7724 (7.7)	17669 (17.7)
Different studies	1	100/100	100/100	0 / 0	0 / 0	200 (2.0)	2500 (25.0)
Different studies	2	200/200	200/200	0 / 0	198 / 0	400 (2.0)	7600 (38.0)
Different studies	5	500/500	350/500	0 / 0	646 / 150	1000 (2.0)	15513 (31.0)
Different studies	10	1000/1000	601/1000	0 / 0	1401 / 399	2000 (2.0)	28782 (28.8)

No unknown-commit failure occurred and every parity, receipt and outbox assertion passed. Exhaustion is a failed save; a lower materialized p95 in a contended cell does not offset its lower success rate.

Verdict: FAIL, on an idle host

All eight cells fail the under-10% p95 gate, with overhead from 46% (ten reviewers, same study) to 1,283% (five reviewers, different studies). The host met the doc's idle definition, so unlike the 15 September paired run these timings are acceptance-grade evidence, and they say the gate is not met. Only the single-reviewer cells are free of contention; they fail by 575-689%, so the cost is structural per-save work and not contention alone.

Different-study cells regress the most in relative terms: source-only saves never conflict there (0 conflicts at every reviewer count), while the materialized arm exhausts 150 of 500 submissions at five reviewers and 399 of 1,000 at ten because every save contends on the shared per-project statistics rows, which drops materialized success to 60% of submissions at ten reviewers.

Cost drivers

Uncontended (one reviewer), the materialized save issues 25 commands against 2 (find + findAndModify) source-only. Names are from the counting run's breakdown; the collection order comes from ProjectScreeningWriteRoundTripBudgetTests output at the same commit.

Command	Source-only per save	Materialized per save	Added
find	1	16	+15
update	0	4	+4
insert	0	2	+2
aggregate	0	1	+1
commitTransaction	0	1	+1
findAndModify	1	1	0

This is 25 commands per save, up from the 21 recorded for #3475 on 15 September: the four added commands are the bounded ledger admission probes (aggregate and find on pmProjectStatisticsRevision) and the two pmProjectStatisticsReceiptRetirement reads (receipt retirement guards), all merged after that measurement. In the table these four are the aggregate row's +1 and three of the find row's +15 (one revision probe plus the two retirement reads); the rest of the table was already present in the 21-command count. Under contention the materialized arm also issues one abortTransaction per lost attempt (for example 1,396 aborts at ten different-study reviewers), each preceded by the full read set again.

Top three added command types per save: find (+15), update (+4), insert (+2). The 15 extra reads dominate: the traced order shows pmProjectStatisticsGlobalControl read 3 times (pre-transaction, envelope authority capture, in-transaction), pmProjectStatisticsControl, pmProjectStatisticsSourceOperationReceipt, pmProjectStatisticsReceiptRetirement and pmProjectStatisticsPublicationGuard each read twice (once for the envelope/pre-check, once again inside the transaction), plus two pmProjectStatisticsRevision reads.

Optimisation candidates (not implemented)

Each is grounded in the traced sequence above and needs its own correctness review, since the duplicate reads exist to close races between the pre-check and the transaction.

1. Read pmProjectStatisticsGlobalControl once per submission, not three times. Hold the snapshot captured in BeginOperationAsync and re-validate it inside the transaction with the write's own version guard (or one read whose result the transaction reuses) instead of three separate finds.
2. Fold the pre-transaction duplicate pre-check into the transaction. ResolveByReceiptAsync and receipt-retirement preflight read the receipt and retirement collections, and the transaction reads both again. A single in-transaction read, with the duplicate result returned from the transaction, removes roughly four finds per save at the cost of opening a transaction for a duplicate.
3. Combine the Control and PublicationGuard checks into the same read as the receipt admission (one query against a single guard document per project, or one $lookup) instead of one find per guard family.
4. Merge the two update writes on Control and PublicationGuard, and the read-then-update on the NotificationOutbox, into a single upsert (the outbox already uses a coalesced slot). The 4 updates and 2 inserts are the writes the transaction has to keep, so the saving here is the paired read, not the write.
5. Address contention separately from round trips. In the different-study cells every save writes the same pmProjectStatisticsCurrent and Control rows, so conflicts and exhaustion (399 of 1,000 at ten reviewers) do not fall with command count. Options: shard the statistics row by study bucket, or accumulate deltas in the already-written revision/ledger rows and fold them asynchronously. This is the design question behind #3255 and #3251, not a local tweak.

Even a large round-trip reduction is unlikely to reach under 10% while a save must run a multi-document transaction against a source path that is a single findAndModify; the gate as written may need Chris's explicit decision, as STATUS already notes.

Limits

• One project, ten studies, capacity-save path only. Not the current controller path, not HTTP, not staging or production.
• Reviewer maintenance and active-reviewer tracking are off; their overhead remains unmeasured.
• The source-only p95 in low-millisecond cells is noisy, so percentages differ by up to 2x between identical runs (see above). The pass/fail verdict does not.