Skip to content

Activate the Bulk PDF notifier authority

The Bulk PDF notifier authority (BulkPdfNotifierAuthority) is default-off. This guide lists the chart values an environment sets in cluster-gitops to turn it on, the zero-usage proof it requires, and the order in which it and featureFlags.bulkPdfUpload are enabled.

Changing these values is a GitOps change to environment configuration. Nothing in the SyRF repository activates an environment, and a merged PR does not prove one is ready. The other activation gates in the durable-capture evidence and in syrf#3168 still apply.

What each chart renders

Values key API Project Management Identity
bulkPdfNotifierAuthority.enabled yes yes yes
bulkPdfNotifierAuthority.zeroUsageProof.* yes yes yes
bulkPdfNotifierAuthority.clients[] clientId, environmentRoot, cleanupAuthorityId no same, plus clientSecret by Secret reference
bulkPdfNotifierAuthority.revokedClientIds[] yes no yes

While enabled is false the charts render no SYRF__BulkPdfNotifierAuthority__* variable, so the hosts keep their disabled defaults. The one exception is revokedClientIds, which renders regardless of enabled: Identity deletes revoked notifier clients on every start, and revoking is how an activated environment is turned back off.

Rendering fails if an enabled client omits any field. Identity never takes a literal client secret: each client names a Kubernetes Secret and the key inside it.

Prerequisites

  • The API has a complete OpenIddict introspection client: identityService.baseUrl (an absolute http(s) URL), identityService.audience, identityService.clientSecretName and, if not syrf-api, identityService.clientId. With the authority enabled, the API refuses to start and names every missing setting.
  • bulkPdfStorage.environmentRoot (or the preview root derived from syrf.prNumber) is the root the clients will name. Every client in one deployment uses the same single root, and production is rejected.
  • A Kubernetes Secret holds each notifier client secret for Identity. Its name and key go in the values; the secret material never does.
  • featureFlags.bulkPdfUpload is still false.
  • Retry authority expires. A terminal Failed/Infected/Abandoned/Cancelled upload keeps its one-shot retry authority for BulkPdfUploadJob.RetryAuthorityWindow (30 days) from the terminal instant and no longer. This is an activation prerequisite (syrf#3168): without it an abandoned failed upload would block its own cleanup registration, hold its release against the reconciler forever, and keep its row out of history pruning. There is no chart value — the window is a domain invariant of the cleanup contract, so every environment agrees on when an abandoned object becomes collectable.

Values to set

Set the same enabled and zeroUsageProof on all three services. Set clients on API and Identity (Identity also needs clientSecret).

bulkPdfNotifierAuthority:
  enabled: true
  clients:
    - clientId: bulk-pdf-notifier-staging            # OpenIddict client the s3-notifier uses
      clientSecret:                                  # Identity chart only
        secretName: bulk-pdf-notifier-staging
        key: clientSecret
      environmentRoot: staging                       # single root for the whole deployment
      cleanupAuthorityId: 7b0c6f55-4d4c-4d41-9a55-2f2a1b8d7c01
  revokedClientIds: []
  zeroUsageProof:
    proofId: 0f4d2a9e-3c1b-4b8e-9d7a-5e6f7a8b9c0d   # new GUID per proof
    observedAt: "2026-09-16T09:00:00Z"
    expiresAt: "2026-09-16T09:45:00Z"               # later than observedAt, at most 1 hour after it
    activationRecordedAt: "2026-09-16T09:10:00Z"    # see "Record activation" below
    releaseRecordCount: 0
    storageBindingCount: 0
    quarantineFenceCount: 0
    queueMessageCount: 0
    embeddedJobCount: 0
    storageObjectCount: 0
    multipartUploadCount: 0

These values become SYRF__BulkPdfNotifierAuthority__Enabled, SYRF__BulkPdfNotifierAuthority__Clients__<n>__{ClientId,ClientSecret,EnvironmentRoot,CleanupAuthorityId}, SYRF__BulkPdfNotifierAuthority__RevokedClientIds__<n> and SYRF__BulkPdfNotifierAuthority__ZeroUsageProof__<Field>.

Project Management only

Two more chart follow-ups from syrf#3168, both PM-only and independent of enabled above:

# Receipt drain reconciler (BulkPdfReceiptDrainReconciler). tableName/interval/batchSize render
# only once enabled.
bulkPdfReceiptDrain:
  enabled: true
  tableName: bulk-pdf-cleanup-hold-staging   # the exact environment's DynamoDB hold table; reuses
                                              # S3Settings' AWS credentials and region (#3094 T12)
  interval: "00:10:00"                       # how often the bounded producer discovers candidates
  batchSize: 100                             # receipts one discovery pass may schedule

# Receipt retention (BulkPdfNotifierAuthorityOptions). Renders unconditionally with defaults equal
# to the C# option defaults, so only set these to change the retention window away from default.
bulkPdfNotifierAuthority:
  receiptPolicyVersion: v1                   # bump only alongside a coordinated policy migration
  receiptRetentionDays:
    staging: 90                              # non-preview BulkPdfEnvironmentRoot; must stay >= 90
    preview: 7                               # preview BulkPdfEnvironmentRoot; must stay >= 7

bulkPdfReceiptDrain.enabled is necessary but not sufficient. A discovery pass also requires bulkPdfNotifierAuthority.enabled: true: with the authority off the producer logs one line and purges nothing, because a purged receipt is deleted evidence and the notifier absence proof rests on that evidence (ADR-017). Turning the authority off therefore stops the drain on its own, so a pause can never quietly consume the receipts a later proof needs. Nothing is lost by the skip — every retained receipt is rediscovered by the ordinary candidate query once the authority is enabled again.

The zero-usage proof

The proof shows that the environment holds no Bulk PDF state the authority could take over. The hosts accept only a proof in which every counter is explicitly zero. An omitted counter does not render and binds as unset, which fails the proof, so every counter must be present.

Observe each count for the exact environment root, all within one window that starts at observedAt:

Counter What it counts
releaseRecordCount Active (non-receipt) release records held by Project Management — every BulkPdfUploadReleaseRecordState except the retained CleanupReceipt audit state, which is non-authorizing and printed separately as an informational retainedAuditReceiptCount (never gates; ADR-017 gap 6)
storageBindingCount Bulk PDF upload storage bindings
quarantineFenceCount Bulk PDF upload quarantine fences
queueMessageCount Messages on the Bulk PDF notifier queues, visible and in flight
embeddedJobCount Bulk PDF upload jobs embedded in projects
storageObjectCount Object versions under the environment root's reserved Bulk PDF prefix
multipartUploadCount Incomplete S3 multipart uploads under that prefix

scripts/bulk-pdf-zero-usage-proof.sh collects all seven counters for one environment, read-only, and prints a ready-to-paste zeroUsageProof: block with a new proofId, observedAt, and expiresAt (45 minutes later):

scripts/bulk-pdf-zero-usage-proof.sh \
  --env staging \
  --bucket syrfapp-uploads-staging \
  --mongo-uri "$MONGODB_URI" \
  --db syrf_staging \
  --rabbit-vhost syrf-staging \
  --rabbit-exec "kubectl -n rabbitmq exec rabbitmq-0 --"

--env staging fills in the bucket, database, and vhost defaults shown above; every other --env (production, or previews/pr-N) requires them explicitly, since only the staging naming convention is documented here. Use --dry-run to print the exact mongosh/aws/ rabbitmqctl commands the script would run without running them. It never runs against a real environment on its own — you choose the target with --mongo-uri/--bucket/--rabbit-vhost.

If any count is not zero, do not activate. The authority is a zero-only contract: an environment that already holds Bulk PDF state needs a separately designed migration. The hosts reject any non-zero counter. Render each counter through an exact integer (… | int64 | quote in _env-blocks.tpl) — Helm decodes every YAML number as float64, and without that conversion a large counter renders in scientific notation that the .NET long? binder rejects outright.

The hosts check the proof at startup:

  • expiresAt must be later than observedAt and at most one hour after it.
  • Startup succeeds while the proof is fresh (observedAt ≤ now < expiresAt), or at any later time once activationRecordedAt is recorded.

Record activation

Upload admission needs a recorded activation, not just a fresh proof. Set activationRecordedAt to a time that is:

  • at or after observedAt,
  • before expiresAt, and
  • not in the future when the services start.

Commit it while the proof is still fresh. The recorded proof then survives restarts after expiresAt. If the window closes before activation is recorded, take a new proof with a new proofId.

Order of operations

  1. Confirm the prerequisites, with featureFlags.bulkPdfUpload still false.
  2. Take the zero-usage proof and commit the bulkPdfNotifierAuthority values for API, Project Management and Identity, with activationRecordedAt inside the proof window. Let ArgoCD sync.
  3. Check that API, Project Management and Identity start, and that Identity seeded the notifier clients.
  4. Only then set featureFlags.bulkPdfUpload: true in a separate change.

Do not reverse this order. With uploads enabled and the authority disabled, API and Project Management still start (a startup throw crash-looped staging, syrf#3517) and refuse new-session admission instead: the API returns 503 bulk_pdf_cleanup_authority_unavailable and reports bulkPdfUpload off to browsers. Completed uploads record cleanup obligations that only the authority can discharge, and once uploads exist the zero-usage proof can no longer pass.

Turning the authority off

The designed procedure is ADR-017: Deactivating the Bulk PDF notifier authority once uploads exist. Read it before changing any value below — it distinguishes three situations that are usually confused, and only one of them is reversible in practice.

Pause (stop admitting uploads, keep cleanup converging) — the usual case:

  1. Set featureFlags.bulkPdfUpload: false. Change nothing else.
  2. Optionally raise durableCapture.publicationPaused: true to stop the agent hand-off while leaving terminal cleanup running.
  3. Confirm the API reports bulkPdfUpload off and refuses initiate with 503 bulk_pdf_cleanup_authority_unavailable.

Leave bulkPdfNotifierAuthority.enabled: true. Disabling it during a pause makes every notifier route answer a typed 503 bulk_pdf_notifier_authority_unavailable (syrf#3589), so the reconciler recognises the paused authority, stops the rest of that invocation's authority calls and releases every leased object back to its previous state on the normal retry delay — it holds every object forever and converges nothing, and it stops the receipt drain as well. Project Management answers those commands rather than dead-lettering them (syrf#3576), so nothing is stranded in an _error queue — but the answer says "not processed", which converges nothing either, and the API maps that answer to the same typed 503. Both BulkPdfReceiptDrainProducer and BulkPdfReceiptDrainReconciler read bulkPdfNotifierAuthority.enabled and gate on it, so a receipt queued by the producer's last enabled pass is still answered "not drained" rather than purged if the reconciler processes it after the authority is turned off.

Rotate or revoke a client (leaked or expiring secret) — no new proof needed:

  1. Add the next-generation client to clients[] with the same cleanupAuthorityId and the same environmentRoot. A different cleanupAuthorityId orphans every existing release record permanently.
  2. Install the new client ID and secret into durableCapture.reconciler only.
  3. Prove both read routes and all three cleanup-write operations on the new credential.
  4. Move the old client ID from clients[] to revokedClientIds[] in one change. Identity deletes the OpenIddict client on its next start and the API refuses its remaining tokens.
  5. Keep the revoked ID listed permanently (ADR-017 gap 7, decided). Never remove an entry from revokedClientIds; a rotated client always gets a new ID.

Fully deactivate — one-way, and only after a notifier absence proof:

  1. Pause as above.
  2. Wait out every terminal job's 30-day retry authority window, then each object's retention deadline.
  3. Keep BulkPdfReceiptDrain on so retained receipts are purged on their own schedule. There is no need to wait for the receipt retention window: audit receipts do not count toward releaseRecordCount (syrf#3601). The authority is still enabled: true at this point, which the drain requires; step 7 is what stops it.
  4. Prove zero active release records, zero storage bindings, zero quarantine fences, zero embedded jobs with an unacknowledged cleanup obligation (pmProject.BulkPdfUploadJobs aggregate in ADR-017 "Operator checks"), zero active holds, zero QuarantinedHold items, zero retirement outboxes, zero queue messages and zero objects. Run scripts/bulk-pdf-zero-usage-proof.sh --absence-proof --hold-table <durableCapture.tableName> for the exact environment root — it prints the zeroUsageProof: and notifierAbsenceProof: blocks together and STOPs (non-zero exit) on any non-zero counter above; do not count the DynamoDB capture ledger by hand. STOP on any non-zero counter, any QuarantinedHold, or any non-empty *BulkPdf*_error queue.
  5. Move every client ID into revokedClientIds with clients: [], keeping enabled: true.
  6. Hold that deny fence for the maximum issued token lifetime.
  7. Only then bulkPdfNotifierAuthority.enabled: false, then durableCapture.enabled: false, then BulkPdfReceiptDrain:Enabled off. The drain already stopped purging when enabled: false landed, so the last step is tidy-up rather than the thing that protects the evidence.

Reactivation afterwards is an ordinary first activation and needs a fresh zero-usage proof with a new proofId, which cannot pass until every terminal receipt has also been purged. Never delete receipts or history by hand, never age-purge holds or outboxes, and never reuse a proof.

The mechanics behind these steps: enabled: false stops rendering the clients and proof, and listing notifier client IDs in revokedClientIds makes Identity delete those OpenIddict clients and the API refuse their tokens. A revoked client ID cannot also be an active client. revokedClientIds renders whether or not enabled is set, which is what makes revoking while paused work.