Skip to content

Allocation query performance validation

This supplies reproducible evidence for ledger item 9 in STATUS.md, subject to the acceptance contract. It does not enable allocation, change an environment database or define a new latency SLA. Representative deployment qualification remains distinct from a synthetic run on a shared development host.

Smallest useful measurement slice

AllocationQueryPerformanceTests is an opt-in test in src/libs/project-management/SyRF.ProjectManagement.Mongo.Data.Tests. It exercises the actual repository next-study, assigned-study page and administrator progress methods, and separately measures derived-plan construction. No production application code or query design changes are included in this measurement slice.

The test creates a new isolated MongoDB 8 Testcontainers instance. It accepts no database connection string and does not read deployment connection settings. It skips before creating the fixture unless explicitly enabled; its fixture is disposed in finally and Testcontainers resource reaping remains enabled. No shared Docker cleanup or environment database operation is part of the command.

Reproducible commands

Run from this PR's worktree root, not the main checkout. Docker and the repository's .NET 10 toolchain are required. Ordinary test runs skip this benchmark; existing CI routing is unchanged.

SYRF_ALLOCATION_PERF=1 \
SYRF_ALLOCATION_PERF_STUDIES=10000 \
SYRF_ALLOCATION_PERF_REVIEWERS=10 \
SYRF_ALLOCATION_PERF_ITERATIONS=10 \
dotnet test src/libs/project-management/SyRF.ProjectManagement.Mongo.Data.Tests/SyRF.ProjectManagement.Mongo.Data.Tests.csproj \
  --configuration Release --filter FullyQualifiedName~AllocationQueryPerformanceTests \
  -m:2 -v:quiet --logger 'console;verbosity=detailed'

Use the same command with 100,000 studies and 50 reviewers for the larger case. For a quick instrumentation check use 100 studies, two reviewers and three iterations. Inputs are bounded: 100–100,000 studies, 2–50 reviewers and 3–30 timed iterations. These are harness resource bounds, not newly imposed production feature limits. Run cases sequentially to avoid self-contention. Opt-in runs read the assembly's actual MSBuild configuration and reject every configuration other than Release (including Debug, Local, Docker and missing metadata) before starting MongoDB, so development-build timings cannot accidentally be published as representative application performance.

Corpus and assertions

  • Deterministic study/reviewer/stage identifiers, two reviews per study and valid equal shares (including deterministic remainder basis points).
  • The requested number of primary-project studies plus 10% separate-project noise. Each study carries a 4,096-character compressible synthetic abstract. This is document-size ballast, not a claim to reproduce production annotation payloads or compression ratios.
  • Saved work on 35% of studies: 30% have one completed or incomplete ordinary session and 5% have two completed sessions at capacity. Five percent are excluded; five percent have a reservation. Owner/status sequences are decoupled so the measured reviewer has both completed and incomplete work and reservations. These percentages describe the primary-project corpus.
  • The actual repository index initializer creates the current configured study index catalogue; the result prints index names/keys. The harness does not invent an alternative index design.
  • Independently computed assigned/completed/in-progress totals must match every administrator row. The measured reviewer's counts must match the administrator row and its page must contain at most 25 studies. Existing allocation Mongo regression tests remain the broader semantic authority for failure, exclusion, reservation, legacy and work-in-progress-limit behavior.
  • Initial cases have no maximum in-progress limit; the final named limit case sets it to one, asserts no new work is available and permits only incomplete saved-work links.

Metrics and interpretation

Each operation emits JSON timing samples, first-call time, warm median, nearest-rank p95/max and mean process allocation bytes. The median averages the middle pair for even sample counts. There is one first call and one additional untimed warm-up before the timed samples.

The first call is not cold-storage latency: seeding has already warmed database/OS caches, and driver/JIT warm-up may be included. Ten samples provide descriptive measurements, not a stable tail-latency estimate or an SLA result. Process allocation counts include background driver/runtime work; Mongo resident memory is a process snapshot, not per-query peak memory.

Database profiling is disabled for timed samples and enabled only for a separate diagnostic execution in the disposable database. Its plan summary, examined document/key counts, response bytes and diagnostic duration are recorded. Entries are matched to the exact collection and execution time window; truncated large command bodies cannot accidentally match an earlier query. Missing or ambiguous diagnostics fail the run instead of reporting stale evidence.

Reported version, processor count, corpus size, data/index bytes and index catalogue accompany the results. Record the Git commit, host/container constraints and concurrent host activity alongside the captured log. Repository-only timings exclude HTTP/authentication, configuration validation, network latency and browser rendering; plan construction is reported separately.

Shared-host observations — 2026-09-05

The corrected Release harness passed its 10,000-study / 10-reviewer case against repository commit d14f66c046e7bed4f5ddc0d2cc3a27b0e38742ff. The captured raw observations include all samples, exact index definitions and the harness SHA-256. Earlier Debug/instrumentation runs are discarded.

PR review subsequently hardened only the pre-fixture build-configuration guard to inspect actual assembly metadata rather than infer Release from the absence of DEBUG. The captured runs were explicitly built with --configuration Release; their original harness hashes are retained. Corpus generation, measured operations and timing/diagnostic code are unchanged.

The run used .NET 10.0.0 and MongoDB 8.0.28 on Juniper (Intel Xeon Platinum 8260, 24 cores / 48 logical processors), with no benchmark-specific CPU/memory cap. Concurrent development and CI were active; observed host load averages were approximately 65/58/40. This is a noisy shared host, not a controlled production-equivalent environment. The corpus occupied 63,679,990 data bytes and 2,695,168 index bytes, including the separate-project noise.

Operation Warm median (ms) Sample p95/max (ms) Diagnostic documents examined
Derived plan construction 33.7 42.2 Not a database query
Legacy next study 85.6 105.6 10,000
Allocated next study 53.6 74.0 1,988
Reviewer page and counts 321.9 369.6 10,000
Administrator progress 1,423.6 1,777.1 10,000
Stale reviewer page 235.5 244.2 10,000
Next study with empty buckets 78.2 107.7 10,000
Reviewer at in-progress limit 258.0 292.8 10,000

The allocated next-study query used the project/bucket index. Reviewer/admin reads and the empty-bucket query scanned the project's study documents. Plan construction allocated about 30.8 MB per sample in the test process; this is not retained memory. These observations identify admin aggregation, empty-work scans and derived-plan allocation as follow-up candidates, not proof that a particular optimization or replacement storage model is necessary.

The 100,000-study / 50-reviewer case also passed against the same source and harness; its raw observations retain all ten timed samples. It occupied 636,909,890 data bytes and 24,813,568 index bytes. Shared-host load increased during this run (an observed snapshot reached approximately 121/86/66), so the two cases are not a controlled scaling comparison.

Operation Warm median (ms) Sample p95/max (ms) Diagnostic documents examined
Derived plan construction 381.2 538.9 Not a database query
Legacy next study 1,223.5 1,362.0 100,000
Allocated next study 81.1 93.4 3,997
Reviewer page and counts 2,879.5 3,395.5 100,000
Administrator progress 56,427.1 87,934.6 100,000
Stale reviewer page 3,443.9 3,651.0 100,000
Next study with empty buckets 562.8 667.3 100,000
Reviewer at in-progress limit 2,543.7 3,560.4 100,000

Administrator progress is a material rollout concern: its separate profiled execution took 98 seconds and scanned the entire project. Plan construction allocated approximately 125 MB per sample in the test process. These are observations under substantial contention, not production predictions. There were no test failures or Mongo query errors in either case.

The local go/no-go conclusion is insufficient evidence for production activation. Repeat under production-like resources with an agreed applicable latency budget; investigate the administrator aggregation first if these costs persist. Preserve the count/eligibility regressions when changing queries. Empty-work short-circuiting and derived-plan memoization are subsequent measured candidates, not silently implemented optimizations.

Budget and correction — 2026-09-07

The repository owner set the applicable budget on 2026-09-07: administrator allocation-progress read warm p95 <= 2,000 ms at 100,000 studies / 50 reviewers, measured on this harness profile. Reviewer-facing my-studies paging must not regress and is reported at the same profiles. This is the first declared numeric budget for this read; the 2026-09-05 section above deliberately declared none.

Diagnosis

GetAllocationProgressAsync built the whole project's study stream once and then added four to five $facet branches per reviewer (assigned, completed, inProgress, available, and limitInProgress when the in-progress limit applies). Every branch is an independent $match + $count over the entire facet input, so 50 reviewers meant 200-250 full re-evaluations of the project's studies inside one aggregation. The profiled plan (IXSCAN { ProjectId: 1 }, 100,000 documents examined) hides this: docsExamined counts the single index scan and fetch, not the per-branch re-processing, which is why the recorded evidence showed a normal-looking plan against a 98-second query.

The arithmetic matches the measurements. The reviewer page runs the same prefix with six branches and took 2,084.6 ms warm median at 100,000 studies; the administrator read with 250 branches took 50,901.2 ms in the same run - about 200 ms per additional branch either way.

Change

The administrator read now classifies each study once in a single $project (its bucket's owning reviewers, whether it is excluded, whether the stage is already sufficiently allocated, and the reviewers holding ordinary sessions or slot reservations), expands it only to the reviewers it can possibly affect, and $groups by reviewer. The number of passes over the corpus no longer depends on the reviewer count.

Semantics are unchanged. Every constant in the new expressions is produced by the same serializer the rendered query filters use (legacy GUID binary subtype, enum representation, nullable agreement ratio), legacy tallies keep the TotalAllocatedSessionCount -> NumberOfCandidateSessions fallback that the atomic capacity guard uses, a stale or out-of-domain WorkloadShareBucket still resolves to no owner, and the existing regression tests assert that every administrator row equals the same reviewer's own page counts. The single-reviewer my-studies page keeps its existing facet pipeline: its branch count is already fixed, and a measured 10,000-study comparison showed the grouped shape was slower there.

Method

Both branches were measured on the same host on 2026-09-07, back to back, with the harness, fixture generator and commands unchanged from the 2026-09-05 protocol above. Baseline runs used a detached worktree at ddf9371dde4a8088c289f1ae2ccf9204271a9681 (the branch point); corrected runs used c63b432dbb65ee9ff4370f502a8fb10f7cf76ca0. The harness file is byte-identical in both (SHA-256 19e1f6a3...). Each run was started only while the one-minute load average was below the host's 48 logical processors; the observed load averages are recorded in each evidence file. The host remained shared with concurrent development and CI, so these are still shared-host observations, not environment qualification.

Raw observations, including all ten timed samples per operation, the exact index catalogue and the profiler entries: baseline 10k/10, grouped 10k/10, baseline 100k/50, grouped 100k/50. The 2026-09-05 files are retained unchanged.

Before and after

100,000 studies / 50 reviewers (warm median / warm p95, ms):

Operation Before After Change
Administrator progress 50,901.2 / 56,100.1 2,843.7 / 3,475.7 17.9x / 16.1x faster
Reviewer page and counts 2,084.6 / 2,639.1 2,022.8 / 2,103.5 no regression
Stale reviewer page 1,941.7 / 2,036.8 2,348.5 / 3,047.1 within run-to-run noise
Reviewer at in-progress limit 2,105.1 / 2,828.8 2,114.2 / 2,964.8 unchanged
Allocated next study 75.4 / 91.1 76.0 / 91.8 unchanged
Legacy next study 742.8 / 994.0 635.2 / 958.5 unchanged
Next study, empty buckets 401.4 / 411.7 417.5 / 439.2 unchanged
Derived plan construction 153.0 / 164.6 145.8 / 199.3 unchanged (not a query)

10,000 studies / 10 reviewers (warm median / warm p95, ms):

Operation Before After Change
Administrator progress 1,145.9 / 1,260.5 340.6 / 457.1 3.4x / 2.8x faster
Reviewer page and counts 245.5 / 290.3 268.7 / 313.3 within run-to-run noise
Stale reviewer page 229.8 / 248.3 265.0 / 341.0 within run-to-run noise
Reviewer at in-progress limit 317.6 / 387.3 257.5 / 288.0 unchanged

First call versus warm, and the separately profiled diagnostic execution, for the administrator read (the first call is not cold storage latency - seeding has already warmed the caches):

Profile First call (ms) Diagnostic query (ms) Plan Documents / keys examined Response bytes Mean process allocation
100k/50 before 56,492.1 51,278 IXSCAN { ProjectId: 1 } 100,000 / 100,000 13,554 21.9 MB
100k/50 after 3,510.2 2,928 IXSCAN { ProjectId: 1 } 100,000 / 100,000 5,936 1.8 MB
10k/10 before 1,276.3 1,067 IXSCAN { ProjectId: 1 } 10,000 / 10,000 2,834 8.1 MB
10k/10 after 735.8 350 IXSCAN { ProjectId: 1 } 10,000 / 10,000 1,296 1.7 MB

The plan and examined counts are unchanged by design: the correction removes repeated in-pipeline work, not the index selection. The client-side allocation per administrator call falls by about 12x at 100,000 studies because the driver no longer materializes 250 facet result arrays.

Result against the budget: not met

Warm p95 at 100,000 studies / 50 reviewers is 3,475.7 ms against a 2,000 ms budget - 1,475.7 ms over, or 1.74x the budget. Warm median (2,843.7 ms) is also above it. The budget is not lowered and no threshold is declared passed. Reviewer paging did not regress at either profile.

Remaining risk and the next step

The residual cost is per-document, not per-reviewer. Per-study cost is about 26 microseconds at both 10,000 and 100,000 studies, so the read is now linear in corpus size and flat in reviewer count, but it still fetches every study document in the project. A plain indexed find over the same corpus (next-empty-buckets, 417.5 ms for 100,000 documents) puts the scan floor at about 4 microseconds per document, so roughly 2.4 of the 2.8 seconds is aggregation-expression evaluation over 100,000 fetched documents. Removing pipeline stages does not help: an intermediate variant with one extra $project measured 2,940.6 / 3,418.5 ms, statistically the same.

The next step, and the only one measured evidence supports, is to stop fetching documents that cannot change the answer:

  1. Take the bucket-derived part of assigned and available from a covered per-bucket count on the existing { ProjectId: 1, WorkloadShareBucket: 1 } index, with no document fetch.
  2. Restrict the full classification to studies that can deviate from that baseline - those that are excluded, or that hold a tally, session or slot reservation for the stage - and correct the histogram with it.

Step 2 needs the $or to be an index union. The activity arm is already covered by the existing { ProjectId: 1, "ExtractionInfo.SessionTallies.StageId": 1, ... } index; the exclusion arm would need a new pmStudy index on { ProjectId: 1, "ScreeningInfo.InclusionInfo.SufficientlyExcluded": 1 }. On this fixture that subset is roughly 40-45% of the corpus, which projects to about 1.5 s and would meet the budget - but it degrades back towards the present cost for a project in which every study has been reviewed, and it adds a production index build on the largest collection plus count reconciliation across two reads. That is a separately reviewed change, not an extension of this one.

Until it lands, the administrator progress read is bounded and no longer scales with reviewer count, but the 100,000-study / 50-reviewer profile remains outside the declared budget, and this remains local diagnostic evidence on a shared host rather than environment qualification or activation authority.

Covered bucket counts measured and rejected — 2026-09-22

The 2026-09-07 next step was built, measured and rejected in PR #3604. It misses the budget at the target profile and makes a fully reviewed project about 50% slower. The administrator read stays the single-pass classification. The budget remains not met. The last subsection names the approach this evidence supports instead.

What was built

  1. A covered histogram: study count per WorkloadShareBucket from the existing { ProjectId: 1, WorkloadShareBucket: 1 } index, with no document fetch. This is the baseline: every study counts as assigned to its bucket's owners, and as available to them on an active stage.
  2. The existing classification, run only on studies that can deviate from that baseline: studies holding a tally for the stage, and studies marked sufficiently excluded under some threshold. The excluded ones were found through a new partial pmStudy index on { ProjectId: 1, "ScreeningInfo.InclusionInfo.SufficientlyExcluded": 1 } (that is the field path in code). Each deviating study also reported its baseline share, which was subtracted so its classified counters replaced it. Counts were clamped at zero in case the two reads disagreed.

The reads rely on a tally invariant: every stage session or slot reservation implies a SessionTally for that stage. The aggregate derives tallies on save, and the atomic reservation pipeline seeds one. A read-only check of syrf_staging found no violations, but it holds only 95 studies (21 with stage activity), so that is weak evidence. Production was not queried.

The first version used one rooted $or over both arms. At 10,000 studies its SUBPLAN stage cost about 363 ms in explain, against about 86 ms for the classification itself, so it was split into two disjoint single-index reads run concurrently. The $ne on the tally path is the exact complement of the tally arm.

Method and harness changes

The harness gained a fully reviewed profile (SYRF_ALLOCATION_PERF_FULLY_REVIEWED=1: every primary study holds two completed sessions and there are no reservations). Other changes:

  • every timing sample records the host load average;
  • the profiler reports every command an operation issues;
  • each measured aggregate is re-run under explain: executionStats.

The administrator classification command embeds a 10,000-entry bucket-owner table, which the profiler truncates. For that command the harness explains the pipeline the repository builds instead.

Each profile ran as a back-to-back pair on Juniper: the same Release harness (SHA-256 fe9ebf18…), with repository code at b9f8bad14 (baseline) and at the candidate commit. The rooted-$or pair used harness 572a9245…. A run started only once the one-minute load average was below 48 and no other dotnet build/dotnet test process was running. Recorded start/end loads were 13-44 on 48 logical processors. MongoDB 8.0.28, .NET 10.0.

Baselines moved about ±15% between sessions (100k/50 baseline: 2,415.8 / 3,086.1 ms, 2,128.2 / 2,481.4 ms and 2,123.5 / 2,372.2 ms), so only differences larger than that count as real. This is shared-host diagnostic evidence, not environment qualification.

Results (warm median / warm p95, ms; ten timed samples each)

Profile Baseline Covered buckets (rejected, 57e365c71) Change
100k / 50 2,415.8 / 3,086.1 1,847.3 / 2,391.8 24% / 22% faster; p95 over budget
100k / 50, fully reviewed 3,018.9 / 3,351.4 4,603.3 / 4,907.5 52% / 46% slower
10k / 10 295.3 / 423.2 311.2 / 409.2 unchanged
10k / 10, rooted $or (29488c53d) 284.1 / 329.3 336.9 / 353.0 19% slower median

Diagnostic explain for the rejected read at 100k/50:

Read Plan Keys / documents examined Stage time
Histogram IXSCAN { ProjectId: 1, WorkloadShareBucket: 1 }, covered 100,000 / 0 about 50 ms
Tally arm IXSCAN { ProjectId: 1, ExtractionInfo.SessionTallies.StageId: 1, … } 40,000 / 40,000 FETCH 33 ms; classification $project 1,171 ms
Excluded arm IXSCAN on the new partial index 5,000 / 5,000 (0 returned) under 10 ms

The fully reviewed profile shows the same pattern at full scale. The tally arm classified all 100,000 studies, and its $project took 3,534 ms, against 2,133 ms for the baseline's ProjectId scan of the same corpus.

Why it cannot meet the budget. Fetching fewer documents is cheap: FETCH costs 33 ms for 40,000 documents. The cost is classifying each document that is fetched, about 29 µs per study in the classic engine. Every study with stage activity must be classified, so a read that classifies per study costs roughly 29 µs × studies with activity. That grows to the whole project as it is reviewed. At 100,000 studies the fully reviewed case therefore stays near or above 3 seconds for any such design, and this one made it worse.

Folding in the #3329 review notes

  • Null bucket set means every bucket: now one documented helper, OwnsEveryBucket. It runs once in C#, so the query is unchanged.
  • Unrequested group rows from former reviewers: narrowing each study's session holders to the requested reviewers ($setIntersection with the requested list) was measured and rejected:
Profile Baseline Narrowed (d79a7e53c)
100k / 50 2,128.2 / 2,481.4 2,380.2 / 2,712.9
100k / 50, fully reviewed 3,085.9 / 3,373.3 4,056.9 / 4,322.0
10k / 10 282.6 / 433.4 284.9 / 337.0

The fully reviewed difference (about 30%) is well outside the run-to-run spread. Building and discarding a few extra group rows is cheaper, and AdminProgress_IgnoresSessionsFromReviewersOutsideTheRequestedSet already pins that those rows are never reported.

What ships in #3604

The shipped read is the single-pass classification. Its query is unchanged, confirmed back to back against the baseline:

Profile Baseline Shipped (bcb7d38b1)
100k / 50 2,123.5 / 2,372.2 2,049.9 / 2,367.7
100k / 50, fully reviewed 3,024.3 / 3,314.5 2,988.9 / 3,358.3
10k / 10 284.4 / 496.8 274.6 / 294.0

Plan: IXSCAN { ProjectId: 1 }, one command, 100,000 keys and 100,000 documents examined at 100k/50. Reviewer paging is unchanged (1,701.8 / 2,023.3 ms at 100k/50).

Also shipped: a Testcontainers parity matrix. It checks every count the administrator endpoint returns against each reviewer's own page across 32 combinations of:

  • capacity mode;
  • excluded-study visibility;
  • stage activity;
  • in-progress limit;
  • annotation target.

The matrix corpus covers exclusions under the current and an old threshold, capacity, legacy tallies and arrays, reservations, reconciliation and other-stage sessions, out-of-domain buckets, partial, every-bucket, null, empty and saved-only shares, and a former reviewer's sessions. Separate tests cover the stale plan, the null-share invariant, a fully reviewed stage and an empty project. The tests passed before and after every candidate.

Raw evidence (all samples, index catalogues, profiler entries, explain stage trees, load):

Exploratory variants (not reproducible from committed code)

These ran on a temporarily modified harness, with loads of 13-90 as noted. They are indicative only.

Variant Profile Result
ProjectId scan filtered to deviating studies before classifying (no new index) 100k/50, load ~65 about 1.8 s, against about 3.3 s for the full scan in the same run
Same filter, returning slim per-study rows for classification in C# 100k/50, load ~65 about 1.46 s before any C# classification
$match → $group on reviewer-independent study facts, classified in C# against the plan 100k/50, load ~13 1,844 ms median
Same 100k/50 fully reviewed, load ~25 3,287 ms median

The facts-$group variant's counts matched the current read, but it ran in the classic engine (explain version 1). None of the variants brings the fully reviewed case near 2 seconds.

Not met. On this harness, no read that classifies studies at query time reaches warm p95 ≤ 2,000 ms at 100k/50 across both the mixed and the fully reviewed corpus. The shipped read measured 2,367.7 ms p95 (mixed) and 3,358.3 ms p95 (fully reviewed).

The evidence supports materialized per-reviewer allocation counters: counters maintained at write time, so the administrator read costs O(reviewers) instead of O(studies with activity). The writes involved are session save and completion, slot reservation and release, exclusion recalculation, and plan publication. The FEAT-024 materialized-statistics machinery (fencing, rebuild, verified mode) is the existing pattern to reuse.

That is a separately designed and reviewed change, not an extension of this one. Until it lands, the administrator read is unchanged from #3329: linear in corpus size, flat in reviewer count, and outside the declared budget at 100,000 studies. This remains local diagnostic evidence on a shared host, not environment qualification or activation authority.

Release boundary

The synthetic measurement is reproducible local evidence, not completion of production-scale qualification or permission to enable allocation. No latency budget has been declared passed.

Before environment activation, repeat with an appropriately representative study/reviewer/ saved-session distribution and production-like resource/index conditions. Compare allocated and legacy next-study behavior, and include assigned/admin reads, stale and empty work cases. The source handoff specifies no numeric latency budget, so do not declare a fabricated SLA passed. Record applicable existing service budgets or the remaining budget decision, any query timeouts/resource failures, and a rollout go/no-go conclusion supported by measurements.

Memoization or query changes should respond to measured bottlenecks. Materialized allocation plans, a new dashboard and history-aware rebalancing are not prerequisites for collecting this evidence and are not silently included in this PR. Keep the feature flag default-off while required lifecycle, authorization or rollout evidence remains outstanding.