Skip to content

Screening-only Project Overview cutover

The Phase 5.1 boundary is one project-screening endpoint and one independently reversible Project Overview consumer. Stage and membership consumers continue to use their existing contracts.

The Project Overview screening totals widget now acquires a project-route SignalStore resource when mounted and releases it on route exit. Direct entry to Project Overview starts its own authorized read; moving to a stage releases these totals until a project-level consumer needs them again. The store shares one request across simultaneous consumers, clears on viewer, project, flag or disclosure changes, retains same-scope totals across transient failures, and ends polling after a 401/403/404. This endpoint does not expose a server revision, so scope cancellation and response project matching protect against late or mismatched responses; server monotonicity is not claimed. Personal reviewer progress and legacy global summary rows remain separate follow-up cutovers.

The separate Project Overview summary-count slice (#3786) moves only the displayed search, imported-study and membership metadata counts into a route-provided SignalStore. It projects the existing authorized project-details/SearchDto response, so direct entry, project SignalR updates and reconnect reloads use the same source and preserve imported-file count semantics. The page acquires the projection while mounted and clears it on route exit, project/viewer change or loss of Project.View. It does not add a statistics endpoint, take ownership of paginated table totals or change the SearchPopulation consumer flag. No new flag is needed for this local state ownership change; the existing project read and its flag-off fallback remain authoritative.

GET /api/projects/{projectId}/screening-stats returns ProjectScreeningStats. The endpoint requires project view authorization and ViewScreeningProgressGraph, including explicitly authorized public nonmembers before consulting either statistics source. Both checks, and the caller handed to either source, use the caller's effective application groups (claim groups plus investigator application roles), as the authorization handlers do (#3642). Its dedicated materializedProjectStatisticsProjectOverview flag defaults to false and requires the Pages kill switch, global serving and the screening family in the runtime catalogue. The existing adapter also requires writes, serving, the family and the explicit project allowlist, and the reader enforces durable freshness and compatibility within its snapshot.

The per-reviewer screening leaderboard this page also renders is not part of this endpoint's response, but it is subject to the same two permissions and is shaped before it leaves the server, on both statistics paths, by MembershipScreeningVisibilityShaper (#3083). A caller without ViewScreeningProgressGraph receives no membership screening rows, so the leaderboard is absent rather than empty; a caller with it but without ViewScreeningProgressGraphDecisions receives every row's counts under per-request "Reviewer A" labels with no reviewer id, their own row flagged. The feature README holds the tier table. The web page renders what it is sent and decides nothing about visibility itself.

The authoritative delegate is lazy. A Fresh materialized response invokes no study aggregation. Every authoritative response, including a disabled-consumer request, opens a pinned read-only Mongo snapshot. It reads the current Project without the repository aggregate cache, rechecks Project.View and ViewScreeningProgressGraph permissions (including explicitly authorized public nonmembers), and reads global/project controls before running the facets in that same session. Inclusion and definition rewrite fences, the legacy inclusion-job flag and durable-mode disagreement return HTTP 503 with a typed reason; revoked permissions return 403. An unpinnable snapshot also returns 503. No refusal executes screening facets. These admission reads are required source-consistency work even while the consumer flag is off; performance comparisons must include them. A reader refusal never authorizes use of the request's cached Project settings. Fallback executes the original screening totals and screening-tally facets only, with the original mapper and formulas. The original broad FullStats query remains unchanged and is the independent parity oracle. This endpoint does not substitute one field into a previously computed broad response.

Read-request benchmarks must compare identical project corpora, report aggregate commands/facets, p50/p95, and distinguish Fresh, fallback and disabled-consumer requests. Existing reviewer-statistics requests and broad legacy SignalR server aggregation remain separate costs. The Overview refresh bound also adds periodic requests; whole-page acceptance must include those costs rather than claiming all page aggregation has disappeared. No staging or production flag activation is authorized by this implementation or its local tests.

Deployment and runtime activation boundary

The project-management mutation host does not read the API runtime override store (#3360). Activation therefore requires consistent static write/family configuration across every API and mutation-host replica, durable mode admission, backfill and parity evidence. This consumer does not solve runtime maintenance propagation or authorize a live activation. API-only write/family overrides are rejected, and previously saved overrides are ignored in favor of the immutable deployed values. Runtime serving can stop reads, but cannot activate beyond deployed serving readiness.

materializedProjectStatisticsPages is a master kill switch for both this dedicated consumer and existing full-stats substitution (#3361). Turning it off restores authoritative-only full-stats and removes the screening-only Overview consumer, even when its dedicated flag remains on. The dedicated flag can independently roll back Overview. The screening-only endpoint remains callable while either consumer flag is off and then uses its guarded authoritative query.

The browser's project guard waits for its initial runtime flag snapshot before deciding whether to request FullStats. Transient failures retry twice at 250ms intervals; each attempt is limited to three seconds. A sustained outage retains deployed defaults after that bounded wait so project navigation remains available. This degraded path may request legacy FullStats and must not be counted as a successful cutover in performance evidence. Ordinary successful cold-start runtime activation skips the broad request, including when the deployed browser consumer default is false.

Route-owned Screening Overview state

The Screening Overview route provides one NgRx SignalStore. Its mounted component acquires demand and releases it on route exit; Angular may retain a route injector, so the final release also unsubscribes from scope changes, polling and invalidations and resets the structured state. The store holds the current project/caller/permission scope, coherent response, last good response, refresh status, errors and accepted revisions as shared signals. Project, caller, graph access, decision-tier and membership-visibility changes clear the previous response synchronously before another authorized read. Transient failures retain last good data only within that unchanged scope; refusals clear it and end refreshes until the scope changes. The Pages and Screening Info flags still restore the legacy consumer. SignalR remains an invalidation that triggers authorized HTTP refetch, with reconnect and 30-second missed-message recovery; direct snapshot transport is deferred to #3777.

This is a thin Screening Overview cutover. Project Overview's separate screening totals component still owns its input-driven polling loop, and its own-progress state has a separate cutover. They should move to route-owned statistics state in subsequent, independently reviewable work; this Screening Overview slice does not claim whole-page or all-family SignalStore completion.

Overview behavior and rollback

With the dedicated flag enabled, Project Overview shows total studies, sufficiently screened studies and remaining studies to members admitted by the existing project route who have project-view and graph permission. The API also supports explicitly authorized public nonmembers, but the existing browser route rejects ProjectMemberStatus.NotMember before Overview mounts; this slice preserves that route boundary. Public Overview navigation is a separate, permission-scoped follow-up #3461, including checks that other project routes remain membership-only. Its existing reviewer screening and annotation progress continues to load through the reviewer endpoint. Overview no longer requests broad FullStats when both page gates are on. This is the bounded latency path for #3311; the existing full-stats endpoint retains its equality-gated shadow mode and still aggregates. A child-route guard loads that unchanged contract when users navigate to existing screening, stage or other project consumers, including after first entering through Overview. Turning the flag off removes the new surface, cancels its requests, and restores legacy statistics loading on the current Overview.

The separate component never marks membership/stage statistics loaded and does not accept unsolicited broad response updates as a newer screening snapshot. It requests the current screening answer on entry and when an authorized project subscription receives ProjectStatisticsChanged. With materializedProjectStatisticsSignalR enabled, each API instance runs a bounded leased outbox pass once per second. The sink awaits RabbitMQ publication before advancing the Mongo outbox watermark. A unique temporary consumer queue per API instance fans the publication out to all pods; a shared competing-consumer queue would miss viewers on other pods. Each pod sends to all of its subscribed, currently authorized viewers, regardless of who made the change. Delivery rechecks project membership and investigator roles with uncached reads and rejects disabled users.

Notifications contain string-encoded revisions, never counts. The browser deduplicates each invalidation identity and refetches the existing authorized HTTP endpoint. Bursts during a fetch retain one trailing refresh. A successful project subscription (including reconnect) refetches; a 30-second poll recovers missed events and works with notification delivery disabled. Requests time out after 15 seconds; transient failures retain same-scope counts with an out-of-date warning, while terminal 401/403/404 refusals clear them. Leaving the view cancels reads and timers. Hidden browser tabs may throttle timers.

Stage review uses the same invalidation transport when materializedProjectStatisticsSignalR is enabled. Relevant screening and annotation family notifications, visibility invalidations, and subscription acknowledgements trigger the existing retrieveStageReviewerStats action. Its existing HTTP endpoint enforces Project.View and derives personal counters from the authenticated caller, never a viewer ID supplied by a notification. Family bursts are coalesced for 100 ms; a 30-second timer recovers missed notifications while the stage view is mounted and authenticated. Flag-off, sign-out, and view destruction stop this refresh stream. The existing reviewer-statistics effect remains the only request/response path for these refreshes; no FullStats broadcast or replacement counter store is introduced.

The Stage progress dialog observes the opener's live selector values, so an already-open dialog updates along with the header. Personal totals can differ between reviewers: another reviewer's decision can change availability without incrementing the current viewer's screened count. The dialog reactivity is an unconditional display correctness fix within the existing redesigned stage UI; background refresh remains behind the existing SignalR consumer flag.

Fan-out is deliberately limited to one message per API pod to bound uncached authorization reads. A global event can delay project events; raising concurrency is a measured scaling follow-up. A socket send failure does not skip later viewers: delivery attempts continue, then the message failure is surfaced to the transport. Clients deduplicate any redelivery and polling recovers missed notifications.

The outbox watermark means broker acceptance, not acknowledgement by every browser. Failed publication leaves the slot pending. Pod queues are temporary: disconnected browsers recover through subscription/refetch and polling. No new legacy FullStats broadcasts are produced. The API alone overrides the shared disabled sink, after the production registry; other hosts do not schedule dispatch. The independent SignalR flag defaults off and is its rollback switch.

Before enabling on an existing pilot, account for the existing outbox backlog ceiling: a slot older than 15 minutes or beyond the attempt ceiling marks its family non-servable when drained. Do not bypass that safeguard. After delivery recovers, rebuild affected families and run a fresh parity comparison before validating materialized reads. Notification rollback leaves maintenance and page selection unchanged; the bounded HTTP refresh continues.

Local correctness proof

The real Mongo replica-set suite compares all screening-only results to the original full-facet oracle, including empty corpora and threshold/ratio boundary profiles. Endpoint tests cover lazy materialized reads, fallback, disabled-consumer behavior, nonmembers and active members without graph permission. Tests exercise Pages/dedicated-flag combinations, consumer rollback and initial runtime snapshot readiness. HTTP tests prove the parity admin route returns403 for nonadministrators without querying either existing or absent projects. Browser tests exercise real router navigation from Overview to legacy screening/stage consumers, flag-off loading, preserved reviewer requests, rendering, retry and subscription cancellation. Angular template compilation also checks the complete application. Reproducible before/after performance evidence is a separate artifact; these correctness tests do not assert timing improvement on a shared host.

Reproduce the focused checks from the worktree root:

dotnet test src/libs/project-management/SyRF.ProjectManagement.Mongo.Data.Tests/SyRF.ProjectManagement.Mongo.Data.Tests.csproj --filter 'FullyQualifiedName~ProjectScreeningQueryAdapterParityTests|FullyQualifiedName~ProjectStatisticsAuthorizationContextSourceTests'
dotnet test src/services/api/SyRF.API.Endpoint.Tests/SyRF.API.Endpoint.Tests.csproj --filter 'FullyQualifiedName~ProjectScreeningEndpointTests|FullyQualifiedName~RuntimeFeatureFlag'

From src/services/web, run:

pnpm exec ngc -p tsconfig.build.json --noEmit
pnpm exec ng test --watch=false --include='src/app/project/project-overview/project-screening-totals.component.spec.ts' --include='src/app/core/services/project/project-statistics-route.guard.spec.ts' --include='src/app/core/services/project/project-guard.service.spec.ts' --include='src/app/core/services/runtime-feature-flags.service.spec.ts'

Rollout also requires #3371: its administrative mode-transition surface opens the durable fleet and project narrow gates after backfill. Until that prerequisite is merged and its runbook completed, keep this consumer disabled; do not automatically admit a project. Parity requires writes and screening-family maintenance enabled, but global serving may remain off. The authoritative aggregation receives the caller cancellation token through the terminal Mongo query.

Runtime maintenance protection

API runtime overrides cannot change materializedProjectStatisticsWrites or any metric-family maintenance gate. The provider captures immutable deployed values at construction and reapplies them to both its effective lookup and the shared FeatureFlags singleton on every refresh, including partial snapshots. Stored overrides are normalized to the same deployed values; attempts to mutate these gates or automatically enable them through a consumer dependency are refused. This keeps API source writers consistent with the project-management host, which reads static deployment configuration.

Runtime serving may be stopped without stopping maintenance, and unrelated partial refreshes preserve that stop. Runtime activation cannot exceed deployed serving readiness. Existing page, Overview, SignalR and export consumer switches retain their runtime behavior. Changing maintenance requires coordinated deployment values across mutation hosts; no new rollout flag or live activation is added.

Regression coverage includes every maintenance gate in both directions, saved legacy overrides, mutable-singleton/configuration isolation and serving kill-switch persistence. See review 5190938570.

Administrator workflow

Application administrators can open Admin Console → Statistics Pilot. The screen lists existing projects admitted by deployment configuration and reports the API's observed maintenance/consumer flags and durable fleet/project gates. It does not claim that an API observation verifies another host. The calculation section explicitly separates Build statistics from Start maintenance and rebuild. Prepare the statistics service first if needed, then build, start maintenance and compare with source data. The application-use section controls Project Overview screening totals separately. The UI shows refusals and incomplete baselines without advancing to a success state. Operations are serialized; opening the page changes nothing.

In staging, Use precomputed totals on Project Overview rechecks parity before enabling only Pages and ProjectOverview through the existing audited runtime flag API. Writes, Serving and Screening must already be effective. These consumer flags are global; other projects still use authoritative fallback. The preset records previous overrides in the browser tab's session storage. Restore previous Project Overview settings restores only its changes, in reverse dependency order, using revision checks; a later edit stops automatic restoration rather than overwriting it. Keep the tab until restoration is complete. A partially applied preset retains its completed changes for restoration.

Stop maintenance and precomputed reads invokes the durable project gate independently of the write/serve switches. Disabling stops ongoing maintenance and fences serving immediately; repeat after the displayed quarantine time to complete Disabled. The screen leaves the fleet gate intact because a fully disabled fleet cannot be re-enabled by this administration surface. Reopening a project always rebuilds after the new epoch and requires a fresh parity check before applying the page preset.

Durable project selection, unified writer-host runtime overrides and reusable beta releases are tracked in the rollout administration plan. The initial screen does not silently bypass the existing deployment authority or enroll additional projects.

When Use precomputed totals on Project Overview is disabled, the pilot displays the reason beside the button. A passing comparison enables activation once the request finishes and Serving is on. Refresh status clears both the comparison and its displayed result; compare again after refreshing. A saved activation attempt must be restored before retrying. If restoration reports that flags changed later, review those changes in Feature Flags before proceeding.

Screening Info charts and reviewer counts — PR #3586

The separate default-off materializedProjectStatisticsScreeningInfo flag migrates the existing Screening Info page's pie and leaderboard together. It requires Pages, serving, writes, project admission and the project/membership screening families; annotation families and stage existence are not prerequisites.

GET api/projects/{projectId}/screening-overview-statistics reuses the coherent overview reader. It captures the permitted reviewer roster, screening definitions and counters in one snapshot, serves all required Fresh rows together, or recomputes the entire permitted response in a new pinned source snapshot. Graph-hidden sections stay null, signed availability stays signed, and roster/selection/response bounds never silently truncate reviewers. Administrator/all versus restricted/own visibility remains the existing contract plus catalogue permissions.

The reviewer table follows the same #3083 tiers as the Stage Overview bundle and the legacy full-stats routes, because every row leaves through MembershipScreeningVisibilityShaper:

  • without View screening progress graph: no pie and no rows;
  • with the graph permission alone: every row anonymised ("Reviewer A", no investigator id, a surrogate membership id for peers), the caller's own row flagged isSelf; peer rows for this tier come from the guarded source read, since the materialized peer-identity rule refuses identified peer rows without the decisions permission;
  • with View screening progress graph decisions as well: rows named exactly as computed.

The response carries no annotation list, so an anonymised row cannot be joined to an identified one. The controller passes the effective application groups (claim groups plus investigator application roles), so a role-held grant decides the tier.

The page uses local Angular signal state, coalesced authorized refetch on screening-family, visibility/global and reconnect events, and a 30-second recovery poll. A caller whose permission report lacks the graph permission makes no request. A definitive 401/403/404 ends polling and invalidation subscriptions until the scope changes; transient failures keep retrying, and a refresh keeps the last coherent snapshot on screen until its replacement arrives. Identity changes cancel requests; invalid scope, failed reads or regressing revisions hide prior values. Initial route loading skips FullStats for this enabled page only. Explicit disabled/non-pilot responses and flag rollback restore legacy loading; errors do not bypass guarded fallback. The leaderboard stays inside the page's area-chart display switch in both modes. That switch's area chart is currently inert (its screening history selector returns no series), and this consumer does not feed it.

This is a current-count consumer. The existing observed history endpoint remains separate. Deployment, cross-pod browser verification, measured latency/aggregation reduction and soak acceptance remain open; this implementation does not enable any environment flag.