Queued-work ledger (M5a inventory, M5b decisions)

Milestone M5 of the implementation plan binds queued work to persisted authority. This ledger lists every job family that runs in the background, who produces it, what it is checked against today, and what M5a changed. The last section records the M5b classification per family, which Chris decided under design item P9 on 2026-10-02, and what implements it.

How the M5a gate works and how to operate it: delegated jobs (M5a). The queue and exchange names are in the broker security contract.

Contents

• Inventory
• What M5a changed
• Deployment order
• P9: decided by Chris, 2026-10-02

Inventory

Verified on main at 280b835fc (2026-10-02). "Checked today" means the authority applied before the job runs, with every flag at its default.

Delegated job families (M5a gates them)

Family	Producer and admission point	Transport	Consumer	Checked today	Job key (M5a)
Reference import	API POST api/projects/{id}/searches returns a presigned upload and publishes ISearchUploadStartedEvent (policy ImportSearch). The S3 notifier publishes ISearchUploadSavedToS3Event after the upload.	PM saga SearchImportJobStateMachine sends IStartParsingReferenceFileCommand as a MassTransit job	ReferenceFileParseJobConsumer (PM), 4 retries	Only at the API request. The saga and the parse job never recheck.	Search ID
Bulk study update	API getSignatureForBulkStudyUpdate (policy BulkUpdateStudies) saves a BulkStudyUpdateJob with StartedByInvestigatorId and ExecutionVersion. The S3 notifier submits the job.	SubmitJob<IStartBulkStudyUpdateJobCommand>	StartBulkStudyUpdateJobConsumer (PM), 4 retries	API request only. BulkStudyUpdateRequireCurrentExecutionVersion (default off) rejects jobs created before the fixed execution version; it is a compatibility floor, not authority. With BulkStudyUpdateAtomicApply (ADR-020) the job is version 2 and is handed, after the same gate, to RunBulkStudyUpdateOperationConsumer (IRunBulkStudyUpdateOperationCommand).	File ID
Batch risk of bias	API POST …/searches/{searchId}/rob/calculate (policy CalculateRob, fixed in #3060) saves a RiskOfBiasJob, then submits with that job ID.	SubmitJob<IStartRobProcessingJobCommand>	RobProcessingJobConsumer (PM), no retry, 2 h timeout	API request only. No actor is stored on the job.	MassTransit job ID (= PM RoB job ID)
Screening inclusion recalculation	API POST …/screening/settings (policy Edit) commits the agreement mode and starts the calculation; a domain event makes the API send the command.	Send IUpdateStudyScreeningStatsCommand	UpdateStudyScreeningStatsConsumer (PM), no retry	API request only. UpdatedBy in the message is logged, never checked.	Message correlation ID

Other queued work (not gated in M5a, with the reason)

Family	Producer → consumer	Current contract	Why M5a leaves it alone
Bulk PDF upload, processing and cleanup	API upload session → S3 notifier → IProcessBulkPdfUploadCommand → PDF agent (ARRNC) → PM single writer (BulkPdfUploadClaim/Progress/Finalize), notifier-authority consumers, receipt drain, multipart sweep	Durable session, processing claim, cleanup obligations and quarantine fences (bulk PDF upload)	Its production semantics must not change without the owner (stop condition); cleanup is a system obligation by design. Classified below.
Scheduler contract (Quartz service)	Quartz hosts the MassTransit message scheduler and the job-service sagas for the PM job queues. It fires ISearchUploadTimeoutExpiredEvent, review-session IMarkSessionIdle/IRemoveIdleSession/IRemoveSuspendedSession/ICheckConnectionLiveness, daily statistics, and statistics maintenance, drift check and repair	System maintenance. Nothing scheduled carries a person's authority.	No delegated action is scheduled today. A scheduled message is delivered with the envelope the producer wrote, so a delegated job scheduled later keeps its key and is gated like any other execution.
Search-import saga steps	SearchImportJobStateMachine activities, SearchImportJobErrorConsumer, SearchUploadSavedFaultConsumer	Bookkeeping on the import record (create, file received, failed, completed)	They record what already happened. The data-changing step is the parse job, which is gated.
Committed-fact fan-out (API)	ProjectStatisticsChangedConsumer, ActivityClaimRevokedConsumer	Recipients rechecked at delivery since M4b	Committed facts are still processed; fresh disclosures re-authorize recipients (realtime delivery).
Statistics maintenance (PM)	Daily statistics, maintenance, drift check, repair consumers	System-owned, default-off scheduled work	No person's authority.
Review-session maintenance (PM)	Idle, suspended-session and liveness consumers	Release capacity only	They only remove or release; nothing is granted.
Data export	API DataExportController	Synchronous HTTP plus the export-job SignalR stream (M4b)	Not queued.
Dead or unconsumed contracts	IStartSearchUpdateCommand (a request client is injected into StudyController but never used; the broker contract records an unaccounted consumer U), IStartStudyUpdateCommand, IUpdateStudyParsingProgressCommand, ILivingSearchEnabledEvent/ILivingSearchDisabledEvent (published, no in-repo consumer), IStudyGivenSyRFIDEvent, IAnnotationAddedEvent, IReviewerAddedEvent, IAnnotationsReconsiledEvent (no in-repo publisher or consumer)	None	No authority may ever be derived from them. Retire them once B0 accounts for consumer U.

What M5a changed

#3894, dark behind DelegatedWorkAdmissionEnforced (default off, project-management only, not runtime-overridable).

• Binding, version 1. DelegatedWorkBinding records family, job key, target project, resource, action (fixed per family: ImportSearch, BulkUpdateStudies, CalculateRob, Edit), whose authority (DelegatedActor with an actor, or SystemObligation with a named contract and no actor), admission time and policy version.
• Persisted, server-side admission. DelegatedWorkAdmissionService checks current authority, then writes the binding to pmDelegatedWorkAdmission (majority, journaled). The actor comes from the producer's authenticated request; nothing in the message carries authority. The API producers call it since M5a-2 (below).
• Recheck at every execution and retry. The four consumers above call the gate before any effect. It reads the admission linearizably and decides through the existing IAuthorityEvaluator (the actor's lifecycle) and the M3a project evaluator (active membership and the family's action), on the same uncached readers. Every decision is appended to the admission.
• Denial is audited and terminal. The denial is written before the consumer acts. Restoring access later does not revive the job. A denial is never retried (each job definition ignores it); the job faults through its family's existing path.
• Quarantine, not drop. A message with no admission (legacy or unclassified), a malformed or newer-version admission, or a message whose family, project or resource differs from its admission is stored whole in pmDelegatedWorkQuarantine and acknowledged. A redelivery adds an occurrence, not an entry; nothing retries it. The quarantine is a Mongo collection, so the broker topology and permissions are unchanged.
• System obligations exist only by approval. P9 (Chris, 2026-10-02) approves two contracts, registered in PM and the API as DelegatedWorkP9Obligations.Approved: bulk-study-update-rollback.v1 (ADR-020 decision D8: only restoring before-images and releasing locks of an admitted version 2 bulk study update; no producer writes a binding naming it) and screening-inclusion-consistency.v1 (the screening recalculation, admitted by its producer since #3929). An actor is never guessed for one.
• Unavailable never executes. An unreadable admission or authority throws a retryable exception. The family's existing retry policy applies; no retry policy means the job faults.

Deployment order

Consumers deploy first and need nothing from producers. With the flag off they behave exactly as before.

1. This PR: PM consumers understand the binding (dark).
2. Producer PR (M5a-2, #3898): the API writes the admission in the same request that creates the job, before the job can start, and the screening producer sets the correlation ID. The write never blocks or fails the request: a denied or unavailable admission is logged and counted, and the job is still enqueued (quarantined only once the consumer flag is on). Admissions are additive Mongo documents and the correlation ID is an envelope field, so an older PM ignores both. There is no message-contract change, and therefore no ordering hazard that promotion would need to guarantee. The S3 notifier needs no change: the API knows the search and file IDs when it signs the upload.
3. Flag on: only after every PM replica runs this code, producers write admissions, the in-flight legacy backlog has drained or been classified, and P9 is decided (it was, on 2026-10-02). Turning it on while producers write nothing quarantines every new job. That is fail-closed, but it is an outage of those features. Turn on bulkStudyUpdateAtomicApply first: only a version 2 bulk update rolls back when it is stopped (P9 policy 1); a version 1 job retried after a partial apply and then denied keeps what it had applied. Turn on batchRiskOfBiasAtomicApply first too: without it a RoB run is checked only when it starts, and its save is not all or nothing.

What a held or refused job leaves behind, with the flag on (P9, below, decides each family's outcome):

• Quarantined jobs publish nothing and touch no PM record. A quarantined reference import stays parsing: the saga's only timeout covers the upload, not the parse. A quarantined bulk update keeps its current status. The MassTransit job itself completes.
• A denied reference import faults its job, so the saga's existing fault path marks the import failed and deletes the search's Studies (proven over a real broker in M5b). Denied bulk-update and batch RoB jobs fault without touching their PM record. A version 2 bulk update denied mid-run rolls back (ADR-020 D5/D8).
• A denied or unavailable screening recalculation throws before the consumer's error handling, so the project stays CalculatingInclusionInfo and records no project error. This is why P9 (below) makes it an admitted obligation: since #3929 it is denied only if its project no longer exists, and a requester losing access no longer stops it. A recalculation admitted before #3929 still names its requester and is rechecked as such. Its endpoint has no retry policy, so the message goes to the error queue on the first attempt; the "never retried" rule for denials is enforced by the job definitions, and any retry policy added to this endpoint must also ignore DelegatedWorkDeniedException.

P9: decided by Chris, 2026-10-02

Status: decided. P9 was deferred until M5 ("job owners approve durable-obligation exceptions; absent actor does not imply authority"). Chris decided it on 2026-10-02 and is the owner of every family below. Everything stays behind default-off flags (DelegatedWorkAdmissionEnforced, plus bulkStudyUpdateAtomicApply and batchRiskOfBiasAtomicApply); with them off, behaviour is unchanged.

Policy.

1. Data-changing background jobs are atomic. When one is stopped because its requester's permissions changed, it rolls back: it leaves no partial results. The same holds for cancellation and failure.
2. Tidy-up and consistency jobs finish, whoever asked for them: screening inclusion recalculation, bulk-PDF clean-up, and the bulk-update rollback (bulk-study-update-rollback.v1, ADR-020 D8).
3. Owner: Chris, for every family.

Family	Decision	How it is met	State
Reference import	Stops; leaves nothing.	Already all-or-nothing inside an attempt: fatal errors and cancellation delete the search's Studies (reference-library import). A denial at the M5a gate, on the first attempt or on a retry, faults the job; the saga's existing fault path (FailSearchImportJob) marks the import failed and deletes every Study of that search, including any a dead attempt saved. A retry happens only after the gate's unavailable answer or a process death, because the consumer turns every other failure into a fault event.	Proven (#3921): ReferenceImportDenialBrokerTests (real RabbitMQ, replica set, job service and saga); cleanup pinned by ProjectManagementServiceStatisticsFenceTests. No code change was needed.
Bulk study update	Stops; rolls back.	Version 2 jobs (ADR-020, bulkStudyUpdateAtomicApply): the D5 recheck before locking and between apply batches denies, audits the denial on the admission, and rolls back under bulk-study-update-rollback.v1. A version 1 job is not atomic: turn on bulkStudyUpdateAtomicApply before DelegatedWorkAdmissionEnforced (see deployment order).	Proven (#3921): BulkStudyUpdateExecutorTests drive the production D5 recheck over a persisted admission on a replica set; a revocation between batches rolls back exactly and stays denied after access returns.
Legacy in-flight bulk study updates	Classify from the trusted StartedByInvestigatorId.	The M5a gate backfills an admission for a bulk update that has none, from its server-written StartedByInvestigatorId (MongoBulkStudyUpdateLegacyClassifier, one linearizable pmProject read; policy version delegated-work.m5b.legacy-backfill.1), then rechecks it like any other admission. A job with no recorded actor, or not in the message's project, is still quarantined; an unreadable record is unavailable, never an allow.	Implemented, dark (#3929).
Batch risk of bias	Stops; rolls back.	Behind the new default-off batchRiskOfBiasAtomicApply: the save runs as an operation with a lease, a generation, the search's slot and per-study before-images. Authority is rechecked before every batch; a denial, cancellation or failure restores every study this run wrote to its exact prior result (batch-risk-of-bias-rollback.v1, restore only), leaving any study another run has since written alone; recovery rolls back a run whose process died. No study lock is needed: RiskOfBiasInfoV2 is written only by batch RoB runs, never by people (all-or-nothing save).	Implemented, dark (#3931). The AI RoB tool is suspended; this is dormant-path correctness and nothing is reactivated.
Screening inclusion recalculation	Finishes (consistency job).	Admitted as the system obligation screening-inclusion-consistency.v1: it needs only the project to exist, never the requester's current authority. Deploy PM before turning DelegatedWorkAdmissionEnforced on: a PM without the approval denies such an admission.	Implemented, dark (#3929): real RabbitMQ test, the recalculation finishes after its requester is removed.
Bulk PDF cleanup, quarantine fences, receipt drain, multipart sweep	Finishes (tidy-up job), contract bulk-pdf-cleanup.v1.	Nothing to change: M5a does not gate these consumers, so they already finish.	Decided; no code.
Bulk PDF processing (stop at claim)	Stays with the bulk-PDF owners.	Unchanged.	Not in M5.
Scheduled maintenance (statistics, review sessions, upload timeout)	System-owned; no delegated admission.	Unchanged.	Not applicable.

Already decided for version 2 bulk study updates (ADR-020, 2026-10-02): D5 (recheck between batches) and D8 (approve bulk-study-update-rollback.v1, restricted to restore-and-release for an admitted operation).