ADR-017: Deactivating the Bulk PDF notifier authority once uploads exist

Status

In-Review. Supersedes the "Turning the authority off" placeholder in Activate the Bulk PDF notifier authority, which recorded only the mechanics of enabled: false and revokedClientIds and said no procedure existed.

Context

ADR-015 designed activation of the notifier cleanup authority and the durable-capture reconciler. Activation is a zero-only contract: the hosts accept the authority only against a proof that the environment holds no Bulk PDF state at all (BulkPdfZeroUsageProof.IsRecordedEmptyActivationProof, src/libs/kernel/SyRF.SharedKernel/Settings/BulkPdfZeroUsageProof.cs:32-43). That contract is one-directional by construction: it says how to start from nothing and says nothing about stopping once objects, release records and receipts exist.

The activation how-to therefore carried a placeholder instead of a procedure, and .claude/rules/pdf-agent.md pointed at it. Meanwhile the surrounding machinery kept growing — the admission-refusal path (#3517), the receipt-drain producer (#3556), the retry-authority window (#3555), the zero-usage proof counters (#3557) — and each added a state that a naive "flip enabled to false" would strand. This ADR designs the missing direction.

The pressure is concrete. Staging is the first environment that will hold real uploads under an active authority. The moment one completed upload exists, the environment can no longer produce a passing zero-usage proof, so an operator who turns the authority off has no supported way to turn it back on, and every object whose cleanup obligation has not yet been discharged is held by a reconciler that can no longer reach an authority.

Decision

The invariant

Two statements must hold at every instant of every procedure below. Everything else in this ADR exists to serve them.

1. An object with a cleanup obligation never becomes deletable without its receipt. A completed upload stamps CompletedObjectCleanupRequiredAt (BulkPdfUploadJob.RequireCompletedObjectCleanup, src/libs/project-management/SyRF.ProjectManagement.Core/Model/ProjectAggregate/BulkPdfUploadJob.cs:303-312), and from then on IsSafeForDeletion is IsTerminal && CleanupAcknowledgedAt.HasValue (same file, :286-288). Only the notifier authority can set CleanupAcknowledgedAt (AcknowledgeBulkPdfCleanupConsumer, src/services/project-management/SyRF.ProjectManagement.Endpoint/Consumers/BulkPdfNotifierAuthorityConsumers.cs:671-675). Project deletion and bounded history pruning apply the same fence (ProjectManagementService.cs:1136, Project.cs:1508,1527). Disabling the authority does not break this invariant — it suspends it, indefinitely, which is why a pause is safe and a deactivation is not.
2. Authority is never re-granted to a different owner while state exists. Ownership is the CleanupAuthorityId, not the OpenIddict client ID. A registered release is invisible to any other authority (GetBulkPdfReleaseStateConsumer answers Found = false, BulkPdfNotifierAuthorityConsumers.cs:178-190) and acknowledgement refuses a foreign owner (:490-493). Introducing a new cleanupAuthorityId for an environment that already holds release records therefore orphans every one of them, permanently. Client rotation must keep the cleanupAuthorityId constant.

Three situations, not one

Situation	Uploads	Authority	Clients	Durable capture	Reversible?	Needs absence proof
(a) Pause	false	stays true	unchanged	enabled: true, publicationPaused: true	Yes, fully	No
(b) Rotate / revoke a client	unchanged	stays true	next-gen added, then old revoked	unchanged except reconciler credentials	Yes, per step	No
© Full deactivation	false	false last	all revoked	enabled: false	No (see "Reactivation")	Yes

"Turning it off" in casual use almost always means (a). © is a one-way door and must not be chosen because (a) was not considered.

(a) Pause — stop admitting uploads, keep cleanup converging

The only change is the upload flag.

featureFlags:
  bulkPdfUpload: false            # the only value this procedure changes
bulkPdfNotifierAuthority:
  enabled: true                   # unchanged — do not touch
durableCapture:
  enabled: true                   # unchanged
  publicationPaused: true         # optionally raised, to stop agent hand-off

Order: set featureFlags.bulkPdfUpload: false, let ArgoCD sync, and change nothing else.

What this produces:

• The API stops admitting new sessions. Admission is refused on the flag alone; initiate and retry already return 503 bulk_pdf_cleanup_authority_unavailable in the inverse mismatch (BulkPdfUploadController.cs:113), and RuntimeFeatureFlagsController reports bulkPdfUpload off to browsers (RuntimeFeatureFlagModels.cs:183-188), so the UI hides the feature.
• Already-admitted sessions keep heartbeating, completing, cancelling and abandoning, and every Project Management consumer keeps converging. This is the deliberate shape chosen in #3517 after a startup throw crash-looped staging: BulkPdfUploadEnablement logs and refuses, and never throws (src/libs/webhostconfig/SyRF.WebHostConfig.Common/Extensions/BulkPdfUploadEnablement.cs:51-60,67-71).
• The reconciler keeps registering cleanup, holding the exact version to its deadline, deleting it, acknowledging and retiring the hold through the outbox — the whole RetentionWaiting path is independent of publication (BulkPdfScheduledReconciler.cs:373-407).
• Raising durableCapture.publicationPaused: true additionally stops the object-ready hand-off to the agent while a release is still Uploaded (BulkPdfScheduledReconciler.cs:800-815) without touching the terminal-cleanup path (:835-848). Use it when the agent, not the upload feature, is the thing being paused.
• The pause does not hold an Aborting release (a cancel or abandon that landed after capture but before the hand-off). The reconciler still sends it through the object-ready command (:817-834), because Project Management's cleanup-only branch is the only writer that registers its cleanup authority and terminalises the job in one transaction, and that branch never sends agent work (BulkPdfUploadObjectReadyConsumer.cs:99,137-212). Holding it deadlocked staging upload 5ba76ac4 in Aborting (2026-09-25).

A pause is unbounded in time and costs nothing to hold. There is no reason to escalate to © in order to "tidy up".

STOP. Never set bulkPdfNotifierAuthority.enabled: false as part of a pause. With the authority off every notifier route answers a typed 503 bulk_pdf_notifier_authority_unavailable (BulkPdfNotifierAuthorityController.cs:29,43,58,135, AuthorizeCaller, ADR-017 gap 3, syrf#3589) instead of processing the call. The reconciler's API client recognises that problem code and raises BulkPdfNotifierAuthorityPausedException (BulkPdfScheduledReconciler.cs:45,201); the worker stops the rest of the invocation's authority calls with one log line and releases every already-leased row back to its previous state on the normal retry delay (:349-357), and the hold is retained forever. That is fail-closed and loses no data, but it converges nothing, and an environment left there accumulates held objects indefinitely.

(b) Client rotation and revocation without data loss

This implements the rotation sequence ADR-015 states at L424-431. It is the procedure for a leaked or expiring secret, and it does not require a new zero-usage proof: once activationRecordedAt is committed the recorded receipt authorises startup indefinitely (BulkPdfZeroUsageProof.cs:50-63).

1. Add the next-generation client alongside the current one, with the same cleanupAuthorityId and the same environmentRoot:

bulkPdfNotifierAuthority:
  enabled: true
  clients:
    - clientId: bulk-pdf-notifier-staging       # current
      clientSecret: { secretName: bulk-pdf-notifier-staging, key: clientSecret }
      environmentRoot: staging
      cleanupAuthorityId: 7b0c6f55-4d4c-4d41-9a55-2f2a1b8d7c01
    - clientId: bulk-pdf-notifier-staging-2     # next generation
      clientSecret: { secretName: bulk-pdf-notifier-staging-2, key: clientSecret }
      environmentRoot: staging
      cleanupAuthorityId: 7b0c6f55-4d4c-4d41-9a55-2f2a1b8d7c01   # IDENTICAL
  revokedClientIds: []

Two clients sharing one cleanupAuthorityId is explicitly legal: the API groups by CleanupAuthorityId and requires only that each group names one root (src/services/api/SyRF.API.Endpoint/Auth/BulkPdfNotifierApiOptions.cs:25-27), and Identity mirrors the rule (IdentityHostOptions.cs:270-274). TryResolveAuthority maps either client ID to the same authority (BulkPdfNotifierApiOptions.cs:57-59), so mid-rotation both credentials address the same release records.

A different cleanupAuthorityId violates invariant 2 and is the one mistake in this procedure that cannot be undone by editing values back.

1. Install the new client ID and secret into the reconciler only (durableCapture.reconciler.clientId, clientSecretName, src/services/s3-notifier/.chart/values.yaml:93-95). Leave the old client admitted.

2. Prove both read routes and all three cleanup-write operations on the new credential before revoking anything — release state, binding resolve, cleanup register, cleanup acknowledge, hold retire.

3. Revoke the old client in one change: move its ID from clients[] into revokedClientIds[]. The two lists are mutually exclusive and a config naming an ID in both is refused at startup by the API (BulkPdfNotifierApiOptions.cs:23-24), by Identity's option validation (IdentityHostOptions.cs:262-263) and again by the seeder (OpenIddictClientSeeder.cs:300-303) — so the move is atomic by construction.

On the next Identity start the revoked OpenIddict client is deleted (OpenIddictClientSeeder.cs:350-357). That loop sits outside the Enabled block, and the revokedClientIds env block renders regardless of enabled (src/charts/syrf-common/env-mapping.yaml:900-909), so revocation works while paused. The API independently refuses a revoked client's still-valid token (BulkPdfNotifierApiOptions.cs:52-54) — that is the deny fence ADR-015 L428-430 requires until the maximum already-issued token lifetime has elapsed.

1. Keep the revoked ID in revokedClientIds permanently. Nothing prunes it, and its only cost is one env var.

© Full deactivation — only after a notifier absence proof

Full deactivation is permitted only when ADR-015's absence proof (L575-576) holds for the exact environment root, extended with the local state the same contract implies:

Must be zero	Where it lives	How it is read
Active release records (ProcessingCapable, CleanupOnly)	pmBulkPdfUploadReleaseRecord	Mongo count filtered on State
Storage bindings	pmBulkPdfUploadStorageBinding	Mongo count
Quarantine fences	pmBulkPdfUploadQuarantineFence	Mongo count
Embedded jobs with an unacknowledged obligation	pmProject.BulkPdfUploadJobs	Mongo aggregate (see "Gaps")
Active holds (Pending/Leased/Published/RetentionWaiting/ReceiptRetirementPending/RejectedPreHold)	DynamoDB capture ledger	scripts/bulk-pdf-zero-usage-proof.sh --absence-proof --hold-table <name> (activeHoldCount)
QuarantinedHold items	DynamoDB capture ledger	same script (quarantinedHoldCount)
Retirement outboxes	DynamoDB capture ledger	same script (retirementOutboxCount)
Bulk PDF queue messages, visible and in flight	RabbitMQ	rabbitmqctl list_queues
Object versions and incomplete MPUs under the reserved prefix	S3	aws s3api list-object-versions / list-multipart-uploads

Ordering — each step must be verified before the next begins:

1. Pause exactly as in (a): featureFlags.bulkPdfUpload: false. Nothing else changes yet.
2. Let cleanup converge. Every terminal job's retry authority must lapse before its cleanup can even be registered: RetryAuthorityExpiresAt = terminalAt + 30 days, exclusive boundary (BulkPdfUploadJob.cs:35,347-358,379-383; RegisterBulkPdfCleanupConsumer refuses while HasRetryAuthority(now), BulkPdfNotifierAuthorityConsumers.cs:126-129). Then each object waits out its snapshotted retention deadline anchored to RegisteredAt (BulkPdfScheduledReconciler.cs:437-443). A full deactivation therefore cannot complete in less than 30 days after the last terminal upload. Plan for it; do not try to shorten it.
3. Let receipts drain (optional for the proof). Retained audit receipts no longer count toward releaseRecordCount (gap 6, syrf#3601), so the absence proof does not wait for them to expire; they are purged on their own schedule while the drain stays on. With BulkPdfReceiptDrain:Enabled on, the bounded producer discovers retained receipts oldest-first (BulkPdfReceiptDrainProducer.RunOnceAsync, src/services/project-management/SyRF.ProjectManagement.Endpoint/Services/BulkPdfReceiptDrainProducer.cs:154-209) and the consumer proves exact-key outbox absence, history pruning and receipt expiry together before any purge (BulkPdfReceiptDrainReconciler.cs:62-116). Receipt retention is StagingReceiptRetentionDays ≥ 90 / PreviewReceiptRetentionDays ≥ 7 (BulkPdfNotifierAuthorityConsumers.cs:20-35).
4. Take the absence proof for the exact environment root, in one window. Every counter must read zero.
5. Revoke every notifier client: move all clients[].clientId into revokedClientIds[], leaving clients: []. Keep enabled: true for this step, so the API and Identity stay in the configuration shape they validated. Let Identity restart and delete the OpenIddict clients.
6. Hold the deny fence for at least the maximum already-issued token lifetime before claiming credential absence (ADR-015 L428-430).
7. Disable the authority: bulkPdfNotifierAuthority.enabled: false. The zeroUsageProof and clients blocks stop rendering; revokedClientIds keeps rendering and must be kept.
8. Disable durable capture: durableCapture.enabled: false, and leave the EventBridge rule in its rendered DISABLED state (src/services/s3-notifier/.chart/templates/durable-capture.yaml:94). Set BulkPdfReceiptDrain:Enabled off last: a disabled pass queries and sends nothing (BulkPdfReceiptDrainProducer.cs:156-161) and a disabled consumer answers "not drained" rather than faulting (BulkPdfReceiptDrainReconciler.cs:44-61).

Reactivation after a full deactivation

There is no designed "resume". Reactivation is an ordinary first activation and needs a fresh, complete zero-usage proof. releaseRecordCount counts only release records that still carry cleanup authority — every BulkPdfUploadReleaseRecordState except CleanupReceipt (scripts/bulk-pdf-zero-usage-proof.sh, addressed by gap 6 below). A retained CleanupReceipt row is a non-authorizing audit record (ADR-015) and does not block reactivation, so reactivation no longer waits for the receipt-retention window (90 days staging, 7 days preview) to elapse on top of the 30-day retry window — only the active states need to reach zero. A fresh proofId is mandatory; reusing a previous proof is forbidden (see below).

Explicitly forbidden

• Deleting receipts or embedded history by hand. The receipt is the only evidence that an object was retired under durable authority; without it the job is not deletable (BulkPdfUploadJob.cs:286-288) and the absence proof is a fiction.
• TTL or age-based purge of holds, tombstones, outboxes or receipts. ADR-015 L484-486: "DynamoDB TTL is never teardown or absence proof." Active holds and QuarantinedHold items carry no TTL by design (L472-473).
• Reusing a zero-usage proof. The proof records counts and timestamps only and is not bound to an environment, which is precisely why the whole configuration must name a single root (BulkPdfNotifierApiOptions.cs:28-33). A reused proofId would activate a root that already holds jobs or objects.
• Changing cleanupAuthorityId or environmentRoot on a live configuration. See invariant 2.
• Deleting an OpenIddict notifier client outside revokedClientIds. The seeder recreates any client still listed in clients[] on every start (OpenIddictClientSeeder.cs:341-346), and a hand-deleted client leaves no API-side deny fence.
• Escalating to © to clear a stuck hold. A stuck hold is a STOP condition, not a reason to remove the authority that could still discharge it.

Operator checks and STOP conditions

Before and between steps:

# Counters 1-7 for one environment root, read-only, with a ready-to-paste YAML block.
scripts/bulk-pdf-zero-usage-proof.sh --env staging --dry-run       # print the exact commands
scripts/bulk-pdf-zero-usage-proof.sh \
  --env staging --bucket syrfapp-uploads-staging \
  --mongo-uri "$MONGODB_URI" --db syrf_staging \
  --rabbit-vhost syrf-staging --rabbit-exec "kubectl -n rabbitmq exec rabbitmq-0 --"

Before step 4 ("Take the absence proof"), use --absence-proof --hold-table <name> to add the three DynamoDB counters this table requires (active holds, QuarantinedHold items, retirement outboxes) to the same read-only run, instead of counting the capture ledger by hand. Capture tombstones and poisoned rows print alongside them for information only — see "Where the DynamoDB counters come from" in the script. The command STOPs (non-zero exit) if any zero-usage counter, active hold, QuarantinedHold item, or outbox is non-zero:

scripts/bulk-pdf-zero-usage-proof.sh \
  --env staging --absence-proof --hold-table <durableCapture.tableName> --dry-run  # print the exact commands
scripts/bulk-pdf-zero-usage-proof.sh \
  --env staging --absence-proof --hold-table <durableCapture.tableName> \
  --bucket syrfapp-uploads-staging \
  --mongo-uri "$MONGODB_URI" --db syrf_staging \
  --rabbit-vhost syrf-staging --rabbit-exec "kubectl -n rabbitmq exec rabbitmq-0 --"

Release records by state, and jobs still owing a receipt (Mongo, read-only — note every GUID is CSUUID, so use CSUUID("…") if you filter by id):

db.pmBulkPdfUploadReleaseRecord.aggregate([
  { $match: { EnvironmentRoot: "staging" } },
  { $group: { _id: "$State", n: { $sum: 1 } } }
])                                   // ProcessingCapable / CleanupOnly must be 0

db.pmProject.aggregate([
  { $unwind: "$BulkPdfUploadJobs" },
  { $match: { "BulkPdfUploadJobs.CompletedObjectCleanupRequiredAt": { $ne: null },
              "BulkPdfUploadJobs.CleanupAcknowledgedAt": null } },
  { $count: "owing" }                // must be absent (0)
])

Queue health, which the seven counters do not cover:

rabbitmqctl -p syrf-staging list_queues name messages messages_unacknowledged \
  | grep -i bulkpdf                  # every *_error queue must be 0

STOP immediately, and do not proceed to the next step, when any of these is true:

• Any *BulkPdf*_error queue is non-empty. No UseMessageRetry is configured on the notifier endpoints, so any command that does fault is dead-lettered rather than retried. A non-empty error queue means a cleanup request was lost and the matching hold is stranded. Since syrf#3576 a disabled authority is no longer one of those faults — the consumers respond "not processed" instead (gap 4 below) — so anything in an error queue now is a real failure.
• Any capture row is in QuarantinedHold. The reconciler parks a row there when the held version is absent before cleanup (BulkPdfScheduledReconciler.cs:387-394) or when binding resolution is ambiguous (:360-370). Quarantine is an operator-attention state and is never cleared by waiting.
• Any counter is non-zero after the retention windows have demonstrably elapsed. This means convergence has stopped, not that more time is needed.
• The proof window (observedAt … expiresAt, at most one hour, BulkPdfZeroUsageProof.cs:36) closed before the values were committed. Take a new proof with a new proofId; never back-date one.

Rolling back a partial activation

A partial activation is an ordering mistake, not a data-loss event, provided it is corrected before the first upload completes. Each shape has a defined behaviour:

Shape	Observed behaviour	Correct move
Identity on, API off	Identity seeded the clients; every notifier route answers 503 bulk_pdf_notifier_authority_unavailable (BulkPdfNotifierAuthorityController.cs:29, syrf#3589), the reconciler's API client raises BulkPdfNotifierAuthorityPausedException (BulkPdfScheduledReconciler.cs:45,201) and the worker releases the lease back to its previous state (:349-357). Nothing is deleted or acknowledged.	Either complete the activation on the API, or roll Identity back by moving the seeded client IDs into revokedClientIds. No state exists, so no proof is consumed.
API on, PM off	The API forwards; PM answers with its result contract's "not processed" shape flagged AuthorityDisabled and logs one line per message (syrf#3576), and the API returns 503 bulk_pdf_notifier_authority_unavailable (syrf#3589). Nothing faults and nothing is dead-lettered; nothing is registered, acknowledged or purged either.	Roll forward — enable the authority on PM. The reconciler reissues each request on its own schedule, so no redrive is needed, and there is no _error queue to drain.
PM on, API off	No route is reachable; the reconciler holds. Harmless.	Complete the API side or revert PM.
Uploads on, authority off	Both hosts start (#3517). BulkPdfUploadEnablementStartupCheck logs RefusalMessage naming both keys (BulkPdfUploadEnablement.cs:40-45,67-71); admission is refused 503 bulk_pdf_cleanup_authority_unavailable; browsers see the flag off.	Either set featureFlags.bulkPdfUpload: false or complete the authority activation. Both directions are safe; the invariant is held by admission refusal, not by the flag value.
Authority off on API, on in Identity's revokedClientIds	Revocation renders regardless of enabled (env-mapping.yaml:900-909), so the clients are deleted as intended.	This is the supported combination for revoking while paused.

Sequence for staging

1. Confirm the environment is in the documented activated shape, and record the current proofId and activationRecordedAt in the change description.
2. Pause: featureFlags.bulkPdfUpload: false. Sync. Confirm the API reports the flag off and bulk_pdf_cleanup_authority_unavailable on initiate.
3. Wait for the last terminal job's retry window (30 days) plus its retention deadline. Re-run the counters weekly; each run must be monotonically decreasing.
4. Confirm BulkPdfReceiptDrain:Enabled is on. Do not wait for receipt retention: retained audit receipts are excluded from releaseRecordCount (syrf#3601) and are reported only as retainedAuditReceiptCount.
5. Take the absence proof. All counters zero, no QuarantinedHold, no non-empty _error queue.
6. Revoke every client (clients: [], all IDs in revokedClientIds), keeping enabled: true. Sync. Confirm Identity deleted the OpenIddict clients and the API rejects their tokens.
7. Hold the deny fence for the maximum issued token lifetime.
8. bulkPdfNotifierAuthority.enabled: false, then durableCapture.enabled: false, then BulkPdfReceiptDrain:Enabled off. Sync. Confirm all three hosts start.
9. Record in syrf#3168 that the environment is deactivated and that reactivation needs a fresh proof.

Consequences

Easier: an operator can pause Bulk PDF uploads safely and indefinitely with a one-value change, and can rotate a leaked notifier credential without touching release ownership. Both were previously undocumented and both were previously likely to be attempted by disabling the authority, which strands cleanup.

Harder — deliberately: full deactivation is slow (30-day retry window, then the retention window, then the receipt window) and one-way in practice. That cost is the price of invariant 1; the alternative is a supported path to deleting an object whose receipt never arrived.

Accepted: an environment paused indefinitely keeps a small amount of DynamoDB and Mongo state and an idle reconciler schedule. That is cheaper than any mechanism that could age it out.

Gaps in the current code

Each of these is a real limitation found while writing this ADR, not a design choice. None is worked around above; procedures stop where the code stops.

1. No script produces the ADR-015 absence proof. Addressed (syrf#3577): scripts/bulk-pdf-zero-usage-proof.sh --absence-proof --hold-table <name> adds the DynamoDB counters — activeHoldCount (Query on the StateBucket/DueAtEpoch GSI, StateBucket = "ACTIVE" filtered to exclude QuarantinedHold), quarantinedHoldCount (same Query filtered to QuarantinedHold), and retirementOutboxCount (StateBucket = "OUTBOX", ItemType = "HoldRetirementOutbox") — to the same read-only run as the seven zero-usage counters, and STOPs (non-zero exit) if any of them, or any zero-usage counter, is non-zero. captureTombstoneCount/poisonedRowCount print for information via a table Scan (neither state carries the GSI's StateBucket attribute, so no index query can reach them) and never gate the STOP condition, matching ADR-015 L575-576.
2. BulkPdfReceiptDrain has no GitOps values key (being closed by syrf#3560). The options are bound from the BulkPdfReceiptDrain configuration section (SyRF.ProjectManagement.Endpoint/Program.cs:384-389) but the section appears nowhere in src/charts/syrf-common/env-mapping.yaml or any chart, so the only way to set BulkPdfReceiptDrain:Enabled, Interval, BatchSize or TableName in a deployed environment is a raw SYRF__BulkPdfReceiptDrain__* environment variable. Step 8 above cannot currently be expressed as a values change. Follow-up: add a bulkPdfReceiptDrain section to env-mapping.yaml for project-management.
3. A disabled authority is indistinguishable from a missing release. Addressed (syrf#3589). Every notifier route used to return 404 via TryGetAuthority (BulkPdfNotifierAuthorityController.cs:29,43,58,135) whether the authority was disabled or the caller simply had no mapped client, and the reconciler treated a paused authority as a transient error. AuthorizeCaller now checks Enabled before resolving the caller, so a disabled authority answers a typed 503 bulk_pdf_notifier_authority_unavailable — mirroring BulkPdfUploadController.RejectUnavailableCleanupAuthority's bulk_pdf_cleanup_authority_unavailable shape — on every route, whether the authority is disabled here or Project Management answered AuthorityDisabled (gap 4). An unmapped caller and an unknown release both keep answering exactly as before: 404 and a normal Found: false/zero-match 200 respectively, since neither is a paused authority. BulkPdfScheduledReconciler's API client recognises the 503's problem code and stops the rest of that invocation's authority calls with one log line, releasing every already-leased row back to its previous state on the normal retry delay instead of retrying each one and counting it as a row failure.
4. Project Management's notifier consumers dead-letter when the authority is off. Addressed by syrf#3576. RequireEnabled threw and no retry policy was configured for those endpoints, so a partial deactivation lost the command to an _error queue. The seven consumers now do what BulkPdfReceiptDrainReconciler already did: log one line per message and respond with their own result contract's "not processed" shape, carried by a new additive AuthorityDisabled field on each I*Result. An environment root the authority never serves (production, or unparseable) still fails closed — that is a disagreement between the caller and the host, not a paused authority. HTTP semantics were unchanged at the time (the API mapped a disabled response to the same 404 gap 3 described); gap 3 above closes that.
5. The receipt drain is not gated on the notifier authority. Addressed by syrf#3576. The drain was gated only on BulkPdfReceiptDrain:Enabled, so receipt purging continued while the authority was off, removing the evidence an absence proof relies on. RunOnceAsync now also requires BulkPdfNotifierAuthority:Enabled and skips the pass with one log line otherwise (BulkPdfReceiptDrainProducer.cs), so the Step 8 flag ordering is enforced rather than merely asked for. A skipped pass removes nothing: every retained receipt is rediscovered by the ordinary candidate query once the authority is enabled again. BulkPdfReceiptDrainReconciler now reads the same BulkPdfNotifierAuthority:Enabled option and answers its own "not drained" shape when it is off, closing the in-flight race where commands the producer queued during its last enabled pass would otherwise still be drained or purged after the authority is turned off: both the producer and the consumer are gated.
6. The zero-usage proof cannot distinguish a terminal receipt from live state. Decided and addressed (syrf#3601, Chris, 2026-09-22): retained audit receipts must not count toward the zero-usage proof, so reactivation after a full deactivation is not blocked for the whole receipt-retention window. releaseRecordCount previously ran countDocuments({}) over the whole pmBulkPdfUploadReleaseRecord collection (scripts/bulk-pdf-zero-usage-proof.sh (count_collection)), and BulkPdfZeroUsageProof bound that one unqualified counter (BulkPdfZeroUsageProof.cs:20,37), so reactivation was blocked for the full receipt-retention window even when nothing was actually held. The script now filters on State (BulkPdfUploadReleaseRecordState, BulkPdfUploadReleaseRecord.cs:285-290, persisted as its plain Int32 value — no [BsonRepresentation] attribute and no enum convention pack are registered anywhere in this repo) and excludes CleanupReceipt (= 2), the retained, non-authorizing audit record: a receipt can never regain cleanup authority (MarkCleanupOnly throws once State == CleanupReceipt) and ADR-015 treats an audit receipt as neither cleanup authority nor container capacity. The excluded count still prints, as an informational retainedAuditReceiptCount line that never gates the proof. --absence-proof reads the same filtered counter, via one shared helper, so its meaning (zero active release records) is unchanged.
7. No recorded expiry for the revocation deny fence. Decided (Chris, 2026-09-22, syrf#3610): revoked notifier client IDs stay in revokedClientIds permanently. Nothing prunes the list and no expiry is recorded, by design. Keeping an ID listed costs one short config string, while removing it too early could let an already-issued token outlive the fence (ADR-015 L428-430) or let the same client ID be recreated with authority it should not have. Never remove an entry; a rotated client always gets a new ID.

Evidence

Every behavioural claim above is taken from these files at 6df9d1fc2.

Claim	File:line
Deletability requires the cleanup receipt once the obligation exists	src/libs/project-management/SyRF.ProjectManagement.Core/Model/ProjectAggregate/BulkPdfUploadJob.cs:286-288
The obligation is stamped at completion	same file :290-291,303-312
Only acknowledgement sets CleanupAcknowledgedAt	same file :314-323; …/Consumers/BulkPdfNotifierAuthorityConsumers.cs:671-675
Project deletion and history pruning apply the same fence	…/Core/Services/ProjectManagementService.cs:1136; …/Core/Model/ProjectAggregate/Project.cs:1508,1527
Retry authority is 30 days, exclusive boundary, derived for legacy rows	BulkPdfUploadJob.cs:35,347-358,379-383
Cleanup registration refuses while retry authority remains	BulkPdfNotifierAuthorityConsumers.cs:122-129
Acknowledgement refuses a foreign owner or a non-CleanupOnly record	BulkPdfNotifierAuthorityConsumers.cs:490-493
A registered release is invisible to another authority	BulkPdfNotifierAuthorityConsumers.cs:175-190
A disabled authority makes every PM notifier consumer respond "not processed" rather than throw (changed by syrf#3576; it threw at 6df9d1fc2)	BulkPdfNotifierAuthorityConsumers.cs TryResolveEnvironment and the seven Consume guards
Receipt retention floors (90 staging / 7 preview)	BulkPdfNotifierAuthorityConsumers.cs:16-35
Two clients may share one cleanupAuthorityId; all clients share one root	src/services/api/SyRF.API.Endpoint/Auth/BulkPdfNotifierApiOptions.cs:25-33
A revoked ID cannot also be active	BulkPdfNotifierApiOptions.cs:23-24; …/identity/…/Configuration/IdentityHostOptions.cs:262-263; …/Services/OpenIddictClientSeeder.cs:300-303
The API refuses a revoked client's token, and any token while disabled	BulkPdfNotifierApiOptions.cs:40-59
Identity deletes revoked clients on every start, regardless of enabled	OpenIddictClientSeeder.cs:102-104,350-357
Identity re-seeds any client still listed in clients[]	OpenIddictClientSeeder.cs:341-346
Identity re-checks the proof on every start while enabled	OpenIddictClientSeeder.cs:295-299; IdentityHostOptions.cs:234-237
revokedClientIds renders while enabled is false	src/charts/syrf-common/env-mapping.yaml:900-909
Uploads-on/authority-off refuses admission instead of crashing	src/libs/webhostconfig/SyRF.WebHostConfig.Common/Extensions/BulkPdfUploadEnablement.cs:10-21,31-45,63-74
The 503 admission refusal and the browser-visible flag	…/api/…/Controllers/BulkPdfUploadController.cs:113; …/RuntimeFeatureFlags/RuntimeFeatureFlagModels.cs:169,183-188
A disabled authority answers a typed 503, distinct from an unmapped caller's 404 (syrf#3589)	…/api/…/Controllers/BulkPdfNotifierAuthorityController.cs:29,43,58,135,160-176 (AuthorizeCaller, AuthorityResponse)
The reconciler fails closed on any HTTP error and re-leases; a paused authority additionally stops the rest of that invocation's authority calls with one log line, without counting it as a row failure	…/s3-notifier/…/BulkPdfScheduledReconciler.cs:45,201,341-370
Publication pause stops agent hand-off but not cleanup	BulkPdfScheduledReconciler.cs:800-815 vs :757-791,835-848
An Aborting release is handed to the cleanup-only branch regardless of the pause	BulkPdfScheduledReconciler.cs:817-834; …/Consumers/BulkPdfUploadObjectReadyConsumer.cs:99,137-212
The retention deadline is anchored to the first RegisteredAt	BulkPdfScheduledReconciler.cs:841-846
An absent held version quarantines instead of acknowledging	BulkPdfScheduledReconciler.cs:387-394
Capture states, including QuarantinedHold, and the pause default	…/s3-notifier/…/BulkPdfDurableCapture.cs:13-24,181,214-215
The EventBridge rule ships DISABLED; setupRoleAuthorized ships false	src/services/s3-notifier/.chart/templates/durable-capture.yaml:94; .chart/values.yaml:66-87
The drain producer decides nothing; the consumer proves outbox absence, pruning and expiry	…/Services/BulkPdfReceiptDrainProducer.cs:139-209; …/Consumers/BulkPdfReceiptDrainReconciler.cs:62-116
A disabled drain responds rather than faults	BulkPdfReceiptDrainReconciler.cs:44-61
Drain candidates are acknowledged receipts, undrained or expired	…/Mongo.Data/Repositories/BulkPdfUploadReleaseRecordRepository.cs:494-516
The drain binds from configuration with no chart values key	…/project-management/SyRF.ProjectManagement.Endpoint/Program.cs:384-389 (and absent from env-mapping.yaml)
Proof freshness, the one-hour bound and the recorded activation receipt	src/libs/kernel/SyRF.SharedKernel/Settings/BulkPdfZeroUsageProof.cs:32-63
releaseRecordCount counts only active (non-CleanupReceipt) release records; retained receipts print separately and never gate (gap 6, syrf#3601)	scripts/bulk-pdf-zero-usage-proof.sh (count_active_release_records, count_retained_audit_receipts)
BulkPdfUploadReleaseRecordState persists as a plain Int32 (no [BsonRepresentation], no enum convention pack registered anywhere in this repo)	BulkPdfUploadReleaseRecord.cs:285-290; verified empirically against the pinned MongoDB.Bson 3.10.0
The proof script reads Mongo, S3 and RabbitMQ in zero-usage mode, plus the DynamoDB hold table in --absence-proof mode (syrf#3577)	scripts/bulk-pdf-zero-usage-proof.sh (count_*, dynamo_count)

External: ADR-015 L118-124 (no blanket age-delete, 24-hour orphan convergence), L424-431 (rotation sequence and the teardown deny fence), L472-486 (no TTL on active holds; TTL is never absence proof), L550-580 (the cleanup-receipt fence and the notifier absence proof).

Related

• Activate the Bulk PDF notifier authority
• Bulk PDF durable capture evidence
• ADR-012: Bulk PDF upload — hybrid pipeline
• ADR-015: Host every Bulk PDF agent on ARRNC API