Skip to content

Unified feature flags: authority, propagation and administration

1. Decision requested and delivery boundary

Proposal: retain a versioned, generated catalog of defaults and constraints; make MongoDB the authority for runtime policy; distribute complete, revisioned snapshots to backend consumers using change streams plus reconciliation. Browsers receive evaluated flags through the API and revision invalidations over SignalR. They never connect to MongoDB.

This is a temporary planning document for review, not an approved architecture or permission to implement, deploy, change flags, or activate production. The current production restriction remains in force. No additional feature flag is needed for this documentation-only change. A future provider cutover needs an explicit deployment-controlled migration switch with an independent rollback path; it must not depend on the provider it selects.

Smallest usable first release

Deliver non-production global Boolean policy with one generated catalog, one accepted defaults version, atomic overrides/audit, a shared immutable backend snapshot, change-stream recovery, browser bootstrap/refresh correctness, and a minimal status view in the existing flag administration page. Prove the provider in API and Project Management using an existing runtime-safe consumer after its operation boundaries are audited. Do not migrate statistics maintenance or reviewer-capacity modes merely to demonstrate propagation.

Acceptance criteria:

  1. A reviewed administrator change survives an API restart; all participating API/PM replicas and connected browsers converge to the same policy identity and values without deployment.
  2. Conflicting writes produce a conflict, and a dependency change is applied only as part of the exact reviewed plan, in one revision with its audit entry.
  3. Cold start, watch establishment races, disconnect, invalid resume token, out-of-order responses, and a missed notification recover without silently reverting an override to a permissive default.
  4. The administrator can distinguish saved policy, local application, fleet convergence, stale/unknown consumers, and page-reload requirements.
  5. Ordinary legacy behaviour is preserved per migrated flag. Durable statistics, capacity, membership and upload obligations retain their own enforcement.
  6. Production remains read-only; no targeting, beta programme, percentage rules, general policy language, audit export or new flag-admin application is needed for this release.

Shortest critical path: pin catalog/default parity → extract shared evaluator and snapshot → shadow-read in API/PM → enable one safe vertical slice with recovery and existing admin UI → prove two replicas per host plus browser failure scenarios → review broader migration. Recovery and minimum consistency diagnostics are intrinsic correctness work; rich analytics and advanced workflows are follow-ups.

Coordination with statistics

Read-only inspection identified PR #3523, feat(stats): operate the screening pilot from the admin console, at head 16f6d025ab4f527499dc7c41901fbaae602bc477. It owns pilot status, fleet initialization, backfill, project activation/rebuild, parity, disable and revision-checked consumer presets. Its status reflects the API's observed gates, not writer-host proof. Its follow-ups are #3524–#3526; inspect their current scope before creating execution issues here.

This proposal neither duplicates that UI nor changes its deployment allowlist, presets, staging operations or durable controls. Later generic administration should link to that domain workflow. “Project targeted” never means “projection enrolled and Fresh.” The separate Codex task SyRF: Finish materialised project statistics was inspected read-only; no instructions were sent and no pilot worktree was edited.

Confirmed product requirement (Chris, 2026-09-15): both statistics controls must ultimately be manageable at runtime. Keep their meanings and the explicit initial-build action separate:

Administrator control Ultimate runtime behaviour Current staging-pilot limitation
Calculate and maintain statistics Start or stop ongoing calculation/maintenance through a coordinated runtime domain transition across every mutation host, preserving source-only invalidation and durable epochs. Maintenance/write and family prerequisites remain deployment-configured. The current API override mechanism cannot supply working runtime maintenance control across API and PM. Present this as a deployment prerequisite, not an operable runtime toggle.
Use precomputed statistics in the application Independently enable or disable eligible application consumers at runtime, subject to maintenance, enrollment, Freshness and compatibility checks. Maintenance may remain On while application use is Off. Existing runtime consumer switches/presets work only inside the deployment-enabled pilot and its durable readiness constraints. Runtime policy cannot currently activate deployment-disabled global serving.
Build initial statistics Explicitly request initial backfill/build when required; report progress and completion separately from either toggle. Retain the pilot's explicit build/backfill action. Enabling maintenance or application use must not imply the initial build has run or succeeded.

Runtime maintenance is therefore required later work, not a permanent startup-only exception or an optional enhancement. The broad shared-provider change belongs to this planning stream; the immediate pilot must describe the interim limitations honestly without claiming that the full runtime capability has shipped.

2. Evidence and limits

Audit performed on Juniper, with container /home/chris/workspace/syrf, main checkout main/, and wt-resolved document worktree pr/pr3527.plan-unified-runtime-feature-flags-mgljx8/. Current owner-approved pr/ layout supersedes the original delegation's .worktrees/ wording. The wt config show command misleadingly reported defaults; inspection of the installed configuration loader confirmed that .worktreerc.local resolves the approved parent, and wt new returned that exact layout. The tool selected its configured feat/ branch prefix; no manual worktree was created. Base source commit: 3afae3c07d4ca367f34afe84d6a1dc5e67d6ce9b. The source links below are repository-relative; conclusions refer to that base, not later merges.

Read-only staging Mongo sampling found runtimeFeatureFlagOverrides, _id: global, revision 61, updated 2026-09-15 11:24:36.849 UTC. Recent projected audit rows were revisions 59–61 for themeToggle, ending On. The state includes overrides for proportional allocation, PDF/UI features and statistics gates. In particular, stored statistics Off values are not evidence of effective Off: current maintenance normalization uses deployment values. No identities or free-text audit comments were retrieved. This sample confirms persistence and recent matching records, not complete audit integrity or all-pod consistency.

Production Mongo and live pod settings were not inspected. Local GitOps files were read at checkout HEAD 6c521c7c6a5c61573f2e10c39b943c9cb000cfac; these are desired-config observations, not proof of applied Helm precedence or deployed images. No credentials, runtime overrides, deployment settings or application files were changed. No additional model/billing policy was altered and no budget block was reported by the tools.

Current implementation map

Surface Evidence Behaviour and architectural consequence
Catalog/generation env mapping, generator YAML generates browser types/defaults/parser/selectors, env JSON and chart defaults. Runtime metadata/dependencies remain separately declared in C#. Several key aliases differ, e.g. showContactUsForm → featureFlags.contactUsForm, graph2data → Graph2DataEnabled. Preserve aliases during migration.
API catalog RuntimeFeatureFlagCatalog, catalog tests 54 explicit Boolean definitions; absent API configuration becomes false. The parity test checks selected YAML sections, not all operational Boolean settings. activeReviewerTrackingEnabled is generated from fullStackFeatureFlags but excluded from that runtime catalog/test selection.
Defaults shared FeatureFlags, API appsettings, API chart, web chart YAML defaults are usually false; signalRActive and apmEnabled are true. Shared SignalRActive defaults true. API appsettings gives MaxInProgressSessions=true, while API/PM chart values contain false. This demonstrates multiple default surfaces; it does not establish the rendered winner.
Backend binding SyrfConfigureServices Binds FeatureFlags once and registers a mutable singleton. It validates bulk-upload/deletion compatibility. No shared live policy provider is registered here.
Mongo state/audit RuntimeFeatureFlagStore One global override document, separate audit collection, transaction around state replacement and audit insert. Revision check is optional when expected revision is absent. No explicit transaction concern options at this call site; verify client defaults before claiming majority durability. Production both refuses mutation and ignores stored overrides.
Dependency safety resolver, store above Validates graph, requires exact confirmed dependency set, normalizes invalid loaded combinations Off, and pins induced unreviewed changes Off. Preserve these behaviours, including negative prerequisites.
API propagation provider/refresh service, Program Startup fetch then 30-second polling. Failures retain last values, initially deployed values. Applies selected values individually to the mutable shared singleton. Lower revisions rejected; equal revisions accepted. Snapshot merge preserves missing keys. No epoch or defaults/catalog digest.
API access controller Anonymous GET returns descriptors and effective values; PUT checks real administrator groups, environment and reason. Commit applies locally, then best-effort broadcasts revision. Broadcast failure does not undo the write. No audit-history endpoint in this controller.
Browser bootstrap main.ts, AppConfigService Fetches default/environment JSON, merges/parses, then fetches runtime snapshot with ten-second timeout before lifecycle initialization. On failure uses deployed values. Pre-bootstrap apply does not seed the later service's revision counter.
Browser refresh RuntimeFeatureFlagsService, SignalRService Immediate fetch plus five-minute timer and revision hints; three-second timeout with two retries. Counter starts at -1; equal revision accepted; missing keys use captured deployed values. SignalR reconnect has a separate observable, but flag service does not subscribe to it. Post-initial failure actually retains current state; a comment referring only to deployed defaults understates this.
UI consumers FeatureToggleService, selectors, generated selectors Mixture of generated selectors, feature adapters, project conditions and lifecycle reads. A flag is not the whole eligibility predicate. Legacy stageStudiesTable$ is hard-coded false rather than catalog-driven. Audit direct/static reads before promising hot activation.
Existing flag admin component, template Search, categories, overridden filter, deployed/override/effective columns, per-row dialog, dependency confirmation and reload markers already exist. UI comment is optional because it synthesizes a non-empty reason when omitted; API still requires a reason. No targeting, convergence ledger or full audit browser.

Every deployable service and shared consumer

Consumer Observed flag path Migration boundary
API Runtime provider used by feature-gated authorization, review/allocation controllers, submission and statistics adapter; shared singleton still used by hub/domain services Extract into shared libraries without making PM depend on the API assembly. Capture one immutable decision snapshot per operation. Authorization remains separate.
Project Management Startup singleton; statistics adapter; liveness/idle/suspension consumers receive flags; bulk study update consumer checks current execution version No runtime-store refresh registration found. Inventory includes message consumers and shared repositories, not just HTTP routes. Preserve old-job rejection and already-admitted work.
Shared PM libraries StageReviewService, StudyRepository, allocation reads, authoritative statistics queries, ProjectStatisticsProductionRegistry, durable-mode gate Mutable flags are read inside broader domain protocols. Replacing a singleton is insufficient if a constructor captured a value or a transaction rereads a changing flag.
Quartz Program, job/saga configuration No direct common FeatureFlags/runtime provider use found in C# search of this service. Schedules and broker wiring are operational config. Require explicit enrollment only if a future job actually evaluates a feature; do not manufacture a watcher for “all services.”
Identity IdentityFeatureFlags, host options, authorization controller, admission middleware and ClaimPending page IdentityClaimRecovery bound at startup outside shared catalog; guidance only, never admission or identity mapping. Keep startup-owned until an Identity-specific review proves live change safe. No new broad Mongo authority credentials by default.
PDF agent No common FeatureFlags/runtime provider use found in service C# ARRNC-hosted execution and storage/authority boundaries remain operational configuration. Browser/API admission changes cannot stop cleanup or accepted jobs. Later runtime controls require a scoped service API or existing approved transport, not automatic Mongo access.
S3 notifier/reconciler durable capture, scheduled reconciler Environment-bound capture enablement, publication pause and process mode are separate controls; no common runtime provider. Lambda cannot rely on a perpetual in-process change stream while suspended. Keep startup config; later fetch a scoped snapshot per invocation with bounded caching if required.
Web JSON/bootstrap, runtime API, SignalR, store/selectors Only public/evaluated results belong in the browser; operational, targeting and audit details need separate authenticated admin endpoints.

Source search covered src/services and src/libs for FeatureFlags, runtime provider and direct flag reads, with tests excluded for the production-consumer pass. This is a static inventory, not proof that every dynamic configuration lookup or external worker has been exercised. P1 must add a machine-checked consumer manifest and fail CI on newly unclassified reads before any consumer migration. Pin known reads and explicit lifecycle exceptions; a new read must name its host, owner, activation boundary and migration classification.

Environment and catalog gaps to resolve

The catalog inventory lists each schema FeatureFlags entry, aliases, default and service routing, including deliberate runtime-catalog exceptions.

  • Local GitOps staging API values explicitly set bulk upload, deletion lifecycle and theme On. Staging web also sets AF2 and integrated PDF viewer On, among other UI flags, while newQuestionManagement is false there. Production web has explicit legacy UI/telemetry values, including LogRocket. Service-local configuration can therefore differ even when an override revision matches.
  • Inventory final rendered values across chart defaults, shared/environment values, service values, preview metadata, extra env, appsettings and local bootstrap JSON before migration. Record source provenance and hashes, not a credentials-bearing environment dump. Never choose “API wins” or “latest deployment wins” to silently resolve disagreement.
  • The source comments saying all statistics gates have no consumers are stale; live consumer classes exist. Likewise, an old guide's short reload list omits integratedPdfViewer and zonelessChangeDetection; the actual catalog includes both.
  • signalRActive changes the browser bootstrap path, but ApplyBackendFlags does not update the shared backend SignalRActive. Reviewer tracking is an intentionally separate, deployment-sensitive contract. P5 explicitly owns this discrepancy: before migrating signalRActive, audit hub/service initialization and reconnect behavior, then either implement safe backend activation or classify it as restart-bound. Report consumer-specific desired and active values instead of implying one global immediate effect; do not add a mutable singleton assignment as a substitute for that audit.
  • API AddSignalR configuration inspected here has no Redis SignalR backplane registration. AddStackExchangeRedisCache elsewhere is session caching, not a hub backplane. Controller-local broadcasts alone do not prove other replicas' clients receive hints. Reconciliation is already essential.

3. Target authority model

Catalog versus mutable policy

Keep one source-controlled catalog, preferably extending the existing YAML schema rather than introducing another handwritten inventory. Generate .NET and TypeScript definitions plus chart/bootstrap adapters. Per flag declare: stable key and aliases; Boolean type; compiled fallback; owner and description; default; exposure; consuming services; dependencies; allowed scopes; activation class; stale-state behaviour; and retirement status. Include compatibility/schema versions and a canonical digest. Unknown keys, conflicting aliases and dependency cycles fail catalog validation.

“Unified defaults” means the same catalog and accepted baseline across consumers, not identical production/staging behaviour. Environment baseline differences must be explicit and versioned once. Proposed long-term model: publish an immutable validated catalog/default bundle, then atomically point an environment policy at its accepted digest. GitOps deploys binaries and proposes baseline versions; an explicit reconciliation command activates the baseline and records the diff/audit. Pods never reseed, overwrite or advance the baseline on startup. Runtime overrides win over that accepted baseline until cleared.

For the first release, keep existing deployed baseline values and require their per-consumer digests to agree for every migrated key; this avoids a simultaneous default-authority migration. A subsequent baseline-import PR replaces the process-local baseline for those keys. A clear-override action always previews the baseline it will reveal.

State identity and storage

Proposed environment policy envelope:

environmentId, storeEpoch, revision, schemaVersion,
catalogDigest, defaultsDigest, policyDigest,
overrides, killSwitches, updatedAtUtc, mutationId

The unique database/environment identity is deployment-pinned. storeEpoch identifies a lineage, not sortable time. revision is a monotonically increasing Int64 within that lineage; serialize it as a decimal string at API boundaries to avoid JavaScript integer precision loss. Digests use a specified canonical serialization and include aliases/default version where semantically relevant.

For global Booleans, retain one bounded current document and the existing audit collection. A transaction updates state, advances revision and appends audit with explicit majority write concern and snapshot transaction reads. Add unique revision/epoch and mutation-id indexes after checking legacy data. P1 makes expected revision mandatory on every existing mutation route (including reset/dependency retries), preserving typed conflict handling; old clients must fetch and echo it rather than receive a blind-write compatibility bypass. P3 adds an idempotency key to resolve lost success responses; reuse with different payload is rejected. Keep transaction callbacks free of broadcast or other external side effects. See MongoDB transaction guarantees.

Legacy initialization (P2/P3 prerequisite; completed by P4 baseline import): a document with no storeEpoch is recognized as legacy/uninitialized by new readers, not as an acknowledged restore and not as an automatically accepted new lineage. Shadow readers may evaluate it for parity with an explicit Legacy status; they cannot advertise authoritative Ready. An explicit reviewed migration command freezes legacy writers, checks the exact source revision and normalized-policy digest, and transactionally assigns a new lineage identifier plus the accepted baseline identity, preserving overrides and the numeric revision. It appends an immutable migration receipt with source audit references. Legacy audit records remain unmodified; new unique indexes apply only to records carrying the new identity fields, after duplicate checks. Old audit rows need no invented mutation IDs. Repeated migration with the same receipt is idempotent; a changed source aborts. Only after validating that receipt may enrolled readers accept the initial lineage and become Ready. A subsequent missing epoch or changed lineage is quarantined. Never let an ordinary cold read or first administrator write silently initialize the authority.

Later targeting policy grows into immutable bundles plus one atomic current pointer if bounded singleton limits become inadequate. Readers load and validate the entire referenced bundle before swapping. Never splice rules from different revisions. Deleted overrides mean baseline; missing current state in an initialized environment means unavailable/corrupt, not revision zero or “all defaults.” Only a reviewed initialization/import creates a first state.

Rollback writes a new forward revision with reviewed prior policy content. It never decrements revision or deletes history. Database restore requires a controlled new lineage, verified environment binding and refreshed consumers; an unexpected epoch or revision regression quarantines the provider until the restore is acknowledged. Store epoch cannot detect a same-lineage backup restore by itself. The runbook must rotate lineage after restore, and baseline health checks must detect lower revisions. No automatic “accept whatever epoch arrived last.”

Read and write APIs

  • Preserve the current endpoint for legacy global Boolean clients during transition, including value:null and exact dependency confirmation; add fields compatibly. New mutations require revision/plan identity while old callers remain restricted to the old policy surface.
  • A new evaluated endpoint returns only browser-exposed decisions, context identity, envelope version, freshness and activation information. Anonymous bootstrap gets an intentionally public subset. Authenticated project results require validated membership/permissions.
  • Separate administrative catalog/policy, preview-plan, commit-plan, audit-history and convergence endpoints. Preview is side-effect free. A plan digest binds requested change, complete dependency/induced-change diff, current revision, catalog/default versions and actor scope. Changed context requires re-preview, not an automatic broader retry.
  • Expose ETags bound to policy and evaluation context. If a request asks for a minimum revision and a replica is behind, fetch from authority or return a typed retryable response; never label older state current.
  • A committed mutation response means saved, not fleet applied. Notification failure cannot turn a known committed write into an apparent failed write. Unknown commit outcome is resolved by mutation ID before retry.

4. Propagation and failure recovery

Backend design

flowchart LR
  A[Administrator] --> API[Policy write API]
  API --> DB[(Mongo state and audit)]
  DB --> AW[API watcher and reconciler]
  DB --> PW[PM watcher and reconciler]
  AW --> AS[Immutable API snapshot]
  PW --> PS[Immutable PM snapshot]
  AS --> H[Local SignalR revision hint]
  H --> B[Browser]
  B --> E[Evaluated flags API]
  AS --> E

Use one watcher per participating process/environment, not per request or flag. The shared package owns validation, revision comparison, immutable snapshot publication and evaluation; its Mongo adapter owns transport. Initial direct watchers avoid adding a central relay/broker failure mode. Each API watcher notifies its local hub clients after applying a revision; if a backplane is added later, elect/deduplicate notification fan-out to avoid amplification.

The API controller remains a fast local invalidation path, but the Mongo watcher repairs commit-before-notify crashes. The stream is a wake-up mechanism, not event sourcing or proof of exactly-once delivery. Mongo streams require replica sets or sharded clusters and available history for resumption. updateLookup may return a later state than the original event; use full snapshot loading and its own revision instead of interpreting it as an exact historical image. MongoDB change-stream semantics

Race-free bootstrap protocol

  1. Resolve and verify environment/database identity; validate compiled catalog and activation compatibility. State starts Uninitialized.
  2. Establish a collection watch before reading current state. Ensure the cursor has actually been opened, not merely allocated lazily; obtain its starting position and start consuming/coalescing notifications. Preserve invalidation/drop events when filtering.
  3. Read the full majority-committed state and accepted baseline/bundle, validate its envelope/digests and normalize dependencies. Publish one immutable snapshot atomically. For a multi-document future bundle, use the immutable pointer protocol or a coherent snapshot transaction.
  4. Drain buffered invalidations and refetch until the requested watermark is satisfied. Coalesce to “refresh needed” and highest observed revision; a notification during an in-flight fetch must schedule a subsequent fetch, never get lost when that fetch clears a Boolean.
  5. Mark Ready only after a valid state is loaded and the watch-or-reconciliation mode is known. Open request/message admission for enrolled feature paths. Notifications after this handoff still schedule refresh normally.

An acceptable alternative starts from a verified Mongo operation time then watches from that time; it must demonstrate overlap and reject a timestamp outside retained history. Do not implement read-then-watch without overlap. Pin and test the actual driver alias/version: current store and provider references already involve different Mongo driver namespaces. The C# driver documentation describes WatchAsync, cancellation and resumption options; validate them against the repository's locked package version during implementation.

Steady-state invariants

  • Swap a complete snapshot reference, not individual properties. A request or message captures it once. Transaction-sensitive paths continue checking their durable mode within the domain transaction. Never update process-global context for a user's evaluation.
  • Within an epoch, lower revisions are ignored; equal revision/equal digest is idempotent; equal revision/different digest is a fault. A defaults/catalog change requires a new coherent envelope, not reuse of an old override revision with different semantics.
  • A processed resume token advances only after its invalidation is durably covered by an applied snapshot or retained pending work. An in-memory token is sufficient for same-process reconnect; after restart always perform full bootstrap. Persisting only a token without the associated snapshot is unsafe. No persistent checkpoint infrastructure is needed for the MVP.
  • Keep opaque resume tokens opaque. Reconnect with the same stream options/pipeline, bounded exponential backoff and jitter. Invalid/expired token triggers a new watch-before-read full resync. Invalidation/drop additionally triggers lineage/state validation; startAfter is a technical resume option, not authorization to trust a replacement collection. MongoDB resume behaviour
  • Reconcile full current state periodically even with a healthy stream. Suggested initial bounds: 30 seconds with jitter for backend reconciliation, five minutes for browser fallback, healthy propagation objective p95 under two seconds. These are proposed test/operational targets, not current measurements or hard guarantees during partitions.

Failure matrix

Event Required outcome
Mongo unavailable at cold start No silently permissive fallback for enrolled risky capabilities. Serve basic shell/legacy-safe paths where possible; return typed unavailable for paths requiring policy. Consumer admission remains stopped where correctness requires policy. Do not globally fail all service liveness and cause restart loops.
Watch unavailable, snapshot reads work Degraded polling mode, visible status and bounded freshness. No claim of instant kill-switch application.
Brief outage after valid snapshot Retain last known good within per-flag staleness policy. Existing baseline is not authoritative recovery for an active override.
Maximum policy age exceeded Harmless presentation may retain state; risky new admission falls back to its declared safe path or refuses. “Off” is not universally safe: disabling a protection can be dangerous. Preserve durable in-flight work.
Missed/duplicate/out-of-order event Coalesce, reread whole state, compare envelope; reconciliation repairs missed invalidation. No replay of mutations from notification payloads.
Malformed policy or unknown required schema Keep last valid state marked degraded within policy bounds; deny incompatible enablement and alert. Never partially install a bundle.
API receives new state but client is on another pod That pod's watcher sends local hint; reconnect/fallback fetch also repairs. No reliance on the writing pod's Clients.All reaching the fleet.
DB rollback/restore/delete/recreate Reject regression or unexpected lineage, require controlled recovery and full resync. Never reset client counters to accept an arbitrary response.
Audit insert fails Transaction does not publish the policy. If commit result is uncertain, query mutation receipt; no second independent mutation.
New incompatible binary joins Report unsupported catalog/capability; do not enable features requiring it. Existing compatible paths remain usable where possible.

Browser bootstrap and ongoing refresh

Load only endpoint/environment/public bootstrap configuration from JSON. Resolve public flags before lifecycle initialization with a finite timeout and classified safe fallback; then seed the Angular service with the exact same envelope and active snapshot. Fetch authenticated/project-context flags after session resolution and before showing targeted entry points. Reevaluate on login, logout, impersonation changes, project switch and beta preference changes.

On first connection, reconnect, online/focus return and any newer revision hint, refetch. Also poll without SignalR so turning signalRActive Off cannot disable recovery forever. SignalR automatic reconnect does not retry an initial start failure; explicitly handle both paths. Register listeners before connecting and fetch after connection to close the subscribe/fetch race. SignalR client guidance

Maintain a request generation for auth/project context. A slow response for project A must never overwrite project B, even when A has a larger policy revision. Discard results with mismatched environment/context/schema; accept equal revision only with matching digest. Global policy revision alone cannot version membership changes or beta preference changes, so include a context version/hash and reauthorize on the server. Use private/no-store responses for personalized evaluations; never CDN-cache them by revision alone.

Expose desired versus active lifecycle values. Do not rewrite the apparent active value for a reload-bound subsystem while it is still running the old configuration. Show a reload notice and preserve unsaved review data; no forced refresh or form destruction. Push carries no targeting rules, identities, audit data or complete policy; clients fetch their own authorized results.

5. Safe runtime changes and justified exceptions

Class Proposed rule Examples and qualifications
Immediate display Update reactive selector after snapshot swap Theme, navigation visibility, non-destructive display controls; verify all static reads and preserve dirty forms.
Next operation Capture at request/message admission New-feature entry points and allocation decisions after domain audit. Completion/retry of admitted work follows recorded contract, not a fresh global Boolean.
Page reload Persist desired now, apply lifecycle on reload Existing catalog: integrated PDF viewer, SignalR, devMode, zoneless, Sentry, APM, LogRocket. Some can later support teardown/restart, but that is separate implementation work. Telemetry disable may suppress sends immediately where supported without claiming full teardown.
Coordinated domain transition Provider distributes intent; domain transaction establishes authority Statistics maintenance/families, serving modes, reviewer capacity. Freshness/epochs/writer compatibility remain mandatory; fleet cache agreement alone is insufficient.
Startup/deployment only Show why and link to owning runbook; reject runtime mutation Credentials, identity issuer/signing keys, storage environment root, broker/DI topology, worker mode, executable/schema writer floors, notifier authority enablement. Not every Boolean is a feature flag.

Statistics compatibility is specifically governed by materialized-stats rules and runtime activation constraints. Maintenance remains deployment-pinned in the MVP. Runtime serving can currently stop reads only within a deployment-enabled pilot. Future hot maintenance requires all mutation hosts to enroll, compatibility floors to be met and durable mode transitions to remain atomic. Source-only invalidation must continue after a family is turned Off.

The target supports maintenance On / application use Off, and both Off with safe legacy reads. Application use On while maintenance is unavailable must be refused or fall back safely. A runtime request to stop maintenance must coordinate the serving restriction before accepting a state that could serve unmaintained data; it must explain any dependent change in the reviewed plan. Neither control substitutes for the explicit initial build or its Freshness proof.

The same principle preserves disabled memberships after feature rollback and bulk-upload cleanup after admission closes. A kill switch stops new exposure/admission or selects a safe fallback; it does not erase accepted obligations, reopen denied access or undo irreversible data changes.

6. Targeting, beta and gradual rollout (ordered follow-ups)

Evaluation context

Use a typed context with environmentId, verified SyrfUserId, separately resolved InvestigatorId, authorized projectId, and service/operation capability. User and investigator are distinct identities; do not infer an InvestigatorId from a GUID-shaped token subject or email. Ambiguous mappings cannot match targeted enablement. Background work gets trusted project/operation context and does not impersonate the user who originally enabled a flag.

Use an OpenFeature-compatible evaluator boundary if it reduces coupling, but keep SyRF's authorization, policy and domain controls outside the SDK. OpenFeature defines a targeting key and context merge rules; it does not supply this proposed precedence or data model. OpenFeature evaluation context

For project-wide shared behaviour, evaluate by project, never individual membership: two investigators must not maintain different projection formats for one project. User targeting is appropriate for private UI experiences. Declare allowed scope/grain per flag. A project administrator may manage that project's eligible pilot preferences only under a separately granted permission; they cannot mutate global defaults or override operational constraints.

Explicit precedence proposal

Evaluate in this order, returning both value and bounded reason code:

  1. Authorization, environment policy, supported binary/catalog, required domain readiness and prerequisites restrict all enablement.
  2. Environment/global or project emergency force-off dominates every allow rule and beta preference. For protective controls define the safe-state semantics explicitly rather than assuming false.
  3. Explicit applicable deny/exclusion dominates overlapping allows. Conflicting rules at the same priority are rejected on save, not resolved by database order.
  4. For flags permitting individual scope: explicit user setting, then investigator setting, then project setting. Explicit project exclusions still win through step 3. Project-only flags ignore/reject individual scope rules.
  5. Eligible beta opt-in plus any rollout cap, then deterministic cohort rules, then environment override, then accepted default. A flag's policy must specify whether opt-in is required; opting out of an opt-in-only beta excludes the user even from a generic percentage cohort.
  6. Recheck dependencies in the same evaluation context; a targeted child cannot run when its parent is false for that project/user. Dependency plans must not broaden a parent to everyone merely to enable one child.

Distinguish “set default Off” from “force Off for everyone”; the former permits targeted enablement, the latter does not. Show this difference clearly in the editor. No allow can bypass permission, cohort eligibility or a durable domain gate. No arbitrary script expressions in the first targeting release.

Beta preference lifecycle

Store explicit per-feature opt-in/out, subject type/id, programme/policy version and timestamp. Offer a user-facing Beta Features page with eligibility, behaviour, limitations, current availability and withdrawal. Do not copy this preference into authorization roles. A project-wide beta needs project-owner authority and a domain readiness workflow where applicable. Withdrawal prevents new beta admission at the documented safe boundary; in-flight work remains recoverable. Deletion/deactivation and identity remapping need a defined preference retention/migration policy.

Deterministic percentage rollout

Specify a versioned algorithm shared by evaluators, such as SHA-256 of a length-prefixed UTF-8 tuple (environmentId, cohortSaltVersion, rolloutGroup, subjectType, canonicalSubjectId), using a defined unsigned prefix modulo 10,000; enable when bucket is below integer basis points. No language-native hash, random-per-request choice, IP or email. Known test vectors must match .NET and any other server implementation. Browsers normally consume server decisions rather than duplicate hashing.

Keep the same salt/group while increasing 1% → 5% → 25% → 100%, so the cohort grows stably. Related flags may explicitly share a rollout group; otherwise cohorts are independent. Percentage applies to the eligible population, not an exact headcount; small project sets may be unrepresentative. Missing stable identity does not fall back to randomness for shared behaviour. Unleash's documented stickiness/group concept supports this design principle; the proposed SHA-256 protocol is a SyRF choice, not Unleash compatibility. Unleash stickiness

Pause/ramp/rollback is a reviewed new policy revision. Automatic metric-driven rollback and statistical experimentation are optional later work, not a prerequisite for safe manual cohort rollout.

7. All-pod consistency and complete administration UX

Minimum diagnostics required for the MVP

Every participating process reports service, instance/pod UID, process-start ID, image/version, environment, supported catalog, accepted defaults digest, epoch/revision/policy digest, applied time, last successful authoritative check, watch state, last error class, stale status and pending activation. Report value/digest discrepancies per consuming service; do not leak project/user identifiers in metrics labels.

Compare observed consumers with an authoritative expected workload/instance inventory. Missing heartbeats are unknown/missing, not healthy and not silently removed from the denominator. Deployment replacement needs UID-aware retirement and a bounded grace period. Ephemeral jobs report at invocation/admission; browsers are an unbounded population and cannot be part of an “all pods acknowledged” guarantee. Fleet acknowledgement proves convergence at an instant, not distributed transactional consistency or every possible future pod's state.

Initial read-only admin view: saved revision, responding API revision, required hosts applied/expected, last check, stale/unknown/mismatch, reload-required count where observable, and link to per-host details. Without authoritative expected inventory, label the result “observed replicas” and do not claim all-pod proof or permit a transition requiring it.

Full administration workflow

  1. Environment selection: unmistakable environment banner and production read-only policy; selected project is separate from environment. Search by friendly name/key/owner/category; filter overridden, targeted, stale, deprecated and restart-required flags.
  2. Flag detail: purpose/owner, lifecycle, supported services, baseline provenance, override/rules, desired/effective/active values, dependencies, last editor and change time. Explain blocked effective values in plain language.
  3. Edit: Default / On / Off plus separate emergency action; scope selection restricted by catalog and permission. Later add target selector, exclusions, beta eligibility, percentage/group preview and expiry where supported. Validate identities without enumerating unauthorized users/projects.
  4. Preview: show exact before/after, dependency additions and induced Off pins, affected scope estimate with uncertainty, reload/domain-transition consequences and baseline revealed by clearing. A simulator lets authorized admins evaluate a specified user/investigator/project with an explanation; it grants no impersonation or mutation authority.
  5. Commit: require policy-appropriate reason; retain synthesized descriptions only where the approved non-production policy permits them. Bind confirmation to revision/plan digest. Conflict preserves the user's draft and offers a fresh diff. A lost response is “checking result,” not automatic repeat.
  6. Result: Saved → Applying → Converged, or Degraded/Partially observed/Action required. Show a concrete rollout deadline and the missing hosts. Offer refresh and a reviewed compensating change; do not claim rollback completed until observed.
  7. History: paginated/filterable audit with before/after, scope, actor and impersonated identity, reason, request/mutation ID, catalog/default versions and outcome. Restore previews a forward compensating change; it never overwrites intervening edits blindly. Protected export is a later slice with retention/redaction policy.
  8. Beta/cohort experience: self-service preference UI plus a project-specific enrollment link; explain eligible but unavailable, opted out, rollout not reached, or paused. Link statistics users to the existing Statistics Pilot workflow for backfill/parity, rather than embedding a second operator console.

Use the existing Angular Material admin shell. Keyboard focus, labelled controls, non-colour status, accessible errors/live status and preserving draft edits are acceptance criteria for each UI slice. Do not expose storage internals, resume tokens or hashes to ordinary end users; keep technical diagnostics in administrator detail views.

Permissions, audit and operational ownership

Preserve real-actor administrator checks under impersonation. Split capabilities conceptually into read catalog, inspect targeting, simulate, change environment policy, change authorized project policy, emergency stop, and inspect/export audit. Exact role mappings are a review decision; default to current global administrators for the MVP and deny new capabilities to everyone else. Browser hints are not authorization tokens. Cookie-authenticated mutation must retain the application's antiforgery protections and same-origin/CORS policy.

Mongo readers receive least-privilege state/bundle read plus change-stream access, not audit writes or arbitrary collections. API write authority is separate from read-side adapters. Prevent ordinary direct database edits through operational access controls; any approved repair is a controlled audited command with revision increment and schema validation. Audit insertion is atomic with state, but the current application-level log is not inherently tamper-proof against a database administrator. Stronger export/retention controls need explicit requirements.

Production enablement is a separate policy decision and PR updating the current feature-flag rules/runbook. Until then, the existing feature-flag policy still says production GitOps is authoritative and runtime overrides are non-production only. This proposal's target architecture is not an exception to that rule.

8. Alternatives and tradeoffs

Alternative Advantage Reason to choose/defer
Extend 30-second polling to all consumers Smallest transport change, simple recovery Useful intermediate/shadow phase and fallback. Does not meet prompt propagation objective alone; still needs coherent defaults, revisions and domain safety.
One watcher per process, full snapshot refresh No new infrastructure; API/PM independently recover Recommended initial target. Connections/read load grow with pods; measure and size pools. Scope to one environment policy collection, not every domain collection.
Central watcher + RabbitMQ fan-out Fewer Mongo streams, existing broker Later if scale warrants. Needs durable outbox, broadcast semantics per instance, relay failover and snapshot reconciliation. A competing-consumer queue is insufficient because every pod must observe policy.
All services synchronously call flag API Central evaluation/defaults Adds request latency and API dependency to PM/worker availability. Suitable for scoped external/serverless consumers, not every hot-path flag check.
GitOps-only values Strong review/history, familiar rollback Retain for infrastructure/startup and current production. Requires deployments for runtime product controls and can drift per service.
OpenFeature with Unleash/another control plane Existing targeting/admin capabilities; portable evaluation seam Evaluate before building advanced targeting. Requires deployment, identity/authorization integration, audit guarantees and migration of SyRF-specific dependencies/durable modes. SDK adoption alone supplies no authority or propagation.
Redis SignalR backplane Cross-pod connection distribution Not a durable policy store. May be appropriate for broader realtime architecture, but not needed solely for flag hints if each API watcher notifies local clients. Existing Redis session cache is unrelated. Microsoft scaling guidance

9. Migration and independently reviewable PR roadmap

Each PR must state its flag decision, update docs with actual results, and leave a usable rollback path. These are proposed issue-sized slices, not newly created issues or a promise of simultaneous delivery.

Order Reviewable outcome Acceptance / rollback
P0 — this document Agree scope, authority, exceptions and precedence Review only. Main/application/deployment behaviour unchanged.
P1 — generated contract and write safety Generate .NET catalog/defaults/metadata alongside TS; machine-checked consumer manifest with CI failure on newly unclassified reads; explicit lifecycle exceptions; require expected revision on existing mutation routes Preserve current values/aliases and dependency results; pin drift and missing-revision/conflict tests. No runtime authority change; retain old adapter during parity comparison. This phase can ship mandatory CAS and the manifest as a first independent correctness PR.
P2 — shared snapshot seam Extract evaluator/storage adapter; API+PM shadow snapshots and minimum diagnostic endpoint Compare against current paths; no hot maintenance changes. Roll back by disabling shadow registration.
P3 — usable propagation slice Global non-production Boolean CAS/audit, watchers, bootstrap/full resync/reconciliation; enroll audited runtime-safe API/PM path, browser refresh and existing admin status Two API/two PM replicas plus browser pass failure matrix; no lost override after restart. Provide deployment-only RuntimeFeatureFlags:ProviderMode=Legacy|Shadow|Shared (proposed configuration, not a runtime catalog flag). Legacy selects the existing API RuntimeFeatureFlagProvider and startup-bound PM FeatureFlags; Shadow compares without changing reads. Before returning from Shared to Legacy, prove effective-value parity for every enrolled host/key against the active revision and deployed defaults; otherwise reject rollback and first write a reviewed forward policy revision. Never register both paths as writers to the same consumer. This is the first runtime MVP.
P4 — authoritative baseline import Reviewed rendered-default/override migration into accepted baseline envelope Dry-run diff must preserve existing outcomes. Freeze old writes during handoff; single writer afterward. Restore via new forward revision, not reseeding.
P5 — finish safe consumers Migrate one host/flag family at a time, including activation metadata, captured reads and the named signalRActive backend/browser discrepancy No new direct mutable singleton reads; fixture parity for each path. signalRActive requires a lifecycle/reconnect audit and consumer-specific desired/active diagnostics before enrollment. Startup exceptions remain explicit. Domain-sensitive migration requires separate proof and owner review.
P5a — required statistics runtime controls Deliver Calculate and maintain statistics and Use precomputed statistics in the application as separate runtime controls using the shared provider and existing durable domain transitions; retain explicit Build initial statistics action All mutation hosts observe maintenance transitions; application use can stop while maintenance continues; stopping maintenance cannot leave unsafe serving active. No toggle silently starts or claims completion of initial build. Integrate with the pilot UI rather than duplicate it.
P6 — audit/admin depth Read-only history, explanations, preview/restore, richer fleet comparison Authorization/pagination/conflict/accessibility tests. Link #3523 and its follow-ups; do not recreate them.
P7 — project targeting Project-scoped allow/deny for a suitable private pilot, evaluated by API No cross-project leakage, shared-project decisions consistent, domain enrollment separate. Disable targeted policy via forward revision.
P8 — individual targeting and beta User/investigator context, opt-in lifecycle and self-service UI Identity ambiguity, impersonation, opt-out, project switch and preference invalidation tests. No authorization changes.
P9 — gradual rollout Stable basis-point cohorts, exclusions, pause/ramp/rollback and explanation Golden vectors, grain compatibility and overlap/precedence tests. Manual ramp first.
P10 — production policy review Explicit production permissions, operational recovery exercise and rollout approval Current non-production restriction remains until separately accepted. No automated promotion of staging overrides.

Migration mechanics

  1. Export only policy/config data and record exact source commits/images plus actual rendered per-service defaults; capture override state/audit lineage. Do not include secrets or personal audit fields in a general migration artifact.
  2. Classify aliases and catalog exceptions; preserve protective values and distinguish configured/effective values. Resolve service disagreements explicitly with owners. A generated default change must not silently alter production.
  3. Shadow-evaluate new against old per host/context. Categorize deliberate normalization differences separately from defects. Record minimal capability/writer floors before admitting new policy fields.
  4. Import with a dry-run diff and expected source revision. If concurrent administration changes it, abort/recompute. Use a brief write freeze for authority handoff rather than two independent writers. Preserve legacy audit references and immutable migration receipt.
  5. Activate the new reader only for enrolled keys/hosts. Older binaries must reject unsupported policy enablement, or be retired before it can be saved. Versioned endpoints keep old browsers on their supported subset; old clients never edit targeting rules they cannot display.
  6. Before rolling back the provider, translate active policy into equivalent legacy behaviour or disable unsupported rules through a reviewed forward revision. Refuse a binary rollback that would discard a critical override, re-enable a stopped feature or bypass a durable mode.
  7. Delete old copies only after every consuming release has migrated and rollback windows close. Retire flags with owner-approved cleanup of policy, generated definitions, UI and tests; retain audit meaning for removed keys.

10. Verification plan and risks

Existing useful tests cover catalog parity, provider propagation, dependency normalization, controller authorization/revision errors, browser refresh/admin behaviour, statistics activation constraints and shared configuration compatibility. The store and refresh service are excluded from coverage in source; this audit did not run application tests or claim they establish real Mongo failover semantics.

Required new verification, attached to the corresponding implementation PR:

  • Contract: generated .NET/TS/default/Helm parity across all catalog sections; alias preservation; malformed graph, negative dependency, unknown scope and unsupported schema; mandatory CAS; complete diff binding; idempotency with changed payload.
  • Real Mongo replica-set integration: two concurrent administrators, state/audit atomicity, commit-response loss, primary stepdown, startup write race, cursor invalidation/history loss, full resync, empty initial state, restore regression and same-revision/different-digest corruption. Use isolated data; never toggle staging to run automated tests.
  • Fleet: two API and two PM processes, terminate writer after commit before notify, partition one reader, scale a new pod, expire a heartbeat, change baseline version and introduce an incompatible binary. Verify expected inventory includes missing consumers and no false convergence.
  • Domain: statistics epochs/Fresh guards, source-only invalidation on disable, reviewer durable mode and capture-once semantics, disabled membership persistence, upload admission versus convergence workers. Existing pilot UI remains independently usable.
  • Statistics control contract: independently operate runtime maintenance and application use after P5a; reject unsafe use without maintenance; preserve maintenance when use is disabled; verify stop-maintenance coordination under a stale writer; show initial-build required/in-progress/failed/complete independently. Before P5a, verify the pilot labels deployment prerequisites separately from its working runtime consumer controls.
  • Browser: bootstrap response revision seeded into service; old responses cannot overwrite; reconnect/initial connection failure, transport disabled, retry exhaustion, anonymous/authenticated transition, impersonation, rapid project switch, reload-bound desired/active split and unsaved form preservation.
  • Targeting/security: no cross-user/project cache reuse; identity resolution ambiguity; overlapping denies/allows; beta opt-out; percentage 0/100 boundaries and cross-language vectors; anonymous/admin endpoint separation; audit visibility and reason requirements.
  • Operational exercise: controlled lineage rotation after restore, diagnosis of stale pods and safe kill response. Measure propagation latency/read load under realistic fleet size before tightening SLOs. A partition means bounded/fail-safe degradation, never a promise of universal instantaneous changes.
Risk Required mitigation / phase
A generic runtime switch bypasses domain invariants Keep statistics/capacity/startup exceptions; require domain admission proof in P5.
Drift is hidden behind a shared revision Include defaults/catalog/policy digest and expected fleet inventory in P2–P4.
Cold fallback undoes an emergency override Classified stale/cold-start behaviour in P3; no blanket permissive defaults.
New control plane becomes its own failure dependency Basic shell/legacy-safe paths survive; migration selection and recovery are deployment-owned.
Target rules leak identity or affect wrong project Server-derived context, authorized API projection and context-keyed cache in P7–P8.
One document grows without bound Explicit size/rule limits; immutable bundle/pointer migration before limits, not premature event sourcing.
Changing flags strands accepted work Separate admission and completion semantics; durable operation contracts in P5.
Review expands the MVP indefinitely Only correctness/regression/major-security blockers enter P1–P3; record optional ideas in later rows/issues.

11. Open decisions for review

Recommended defaults are stated so review can change them without blocking this planning deliverable.

  1. Production: keep current read-only restriction through P9; separately approve runtime production policy in P10. Is broader production use ultimately wanted?
  2. Baseline authority: accept versioned Git-generated bundles through a reviewed reconciliation command, with no per-pod reseeding. Who owns baseline activation and conflict resolution?
  3. First live backend flag: choose after operation-level audit; do not use statistics maintenance, reviewer mode or irreversible job execution as the demonstration. Candidate scope must genuinely exercise API/PM without changing domain contracts.
  4. Freshness: proposed 30-second reconciliation and per-flag stale limits need measured capacity/availability input. Which admissions must stop immediately through durable controls rather than tolerate propagation lag?
  5. Identity and beta: approve distinct user/investigator/project scopes, deny-first semantics, opt-in-only treatment and preference retention. Is investigator-level rollout needed independently from account-level UI rollout?
  6. Project authority: decide which project roles can opt into eligible betas and whether owner consent is needed for shared-project behaviour; no default delegation of global admin permissions.
  7. Operator audit: keep current synthesized non-production reason versus require a human explanation; choose retention and audit-export requirements. Tamper-evident storage is separate from atomic application audit.
  8. Scope of lifecycle exceptions: keep current page-reload list and Identity/notifier startup boundaries initially; prioritize later teardown/restart work only for concrete user need.
  9. Managed platform versus custom targeting: reevaluate an existing control plane before P7–P9. Do SyRF's dependency/domain constraints justify maintaining a rules editor and evaluator?
  10. Fleet inventory owner: use existing deployment/health ownership to identify expected replicas; decide how ARRNC and invocation-based workers report capabilities without pretending they are continuously connected pods.

12. Research notes

Official sources were checked on 2026-09-15. MongoDB docs establish transport/transaction constraints; OpenFeature and Unleash inform portable context and stable cohorts; Microsoft documents client recovery and scale-out. The authority envelope, precedence, failure classes, bootstrap handoff, roadmap and UI are proposed SyRF design decisions, not guarantees supplied by those libraries. Use the linked sources at the relevant sections and recheck exact installed versions when implementing.

See also the current operating guide, environment generator architecture, and statistics plan. These describe existing policy/domain obligations; this In-Review document does not supersede them.

Document validation

./docs/scripts/validate-docs.sh --verbose completed successfully with 63 existing warnings; no broken links were found. The two proposal files' YAML and relative links were also checked directly, and git diff --check passed. Application tests and deployments were not run for this documentation-only change. Main remained clean at the final local check.

13. First implementation: partial P1/P2 with one scoped API cutover

PR #3585 implements mandatory expected revision on the existing writer, a reviewed C# typed-consumer manifest checked by CI, shared immutable legacy override observations in API/PM, and local administrator diagnostics. Its independent deployment opt-in can move only the existing API disableMembership admission gate to that shared reader. PM observes/evaluates the same saved overrides but keeps every domain consumer unchanged. The operating guide defines cold-start, stale/failure, recovery and rollback behavior.

This is a usable first migration seam, not completion of the first runtime MVP acceptance criteria. The P3 authority envelope and change-stream handoff, generated shared catalog/default parity, baseline import, fleet inventory and domain-safe PM behavioral migration remain required ordered follow-ups. The legacy content digest deliberately excludes defaults and does not advertise authoritative Ready or cross-host effective-value parity. No new statistics-maintenance, reviewer mode or bulk-job execution switch is introduced. The default mode preserves existing production and non-production behavior.

Local validation includes a real isolated Mongo replica-set test with separate Lamar API/PM registrations, a committed override, complete reset and a restarted PM reader; immutable capture, stale/unknown recovery, revision regression and same-revision corruption tests; the existing API flag and route-gate tests; and consumer-manifest regression checks. These establish the tested seam, not a deployed multi-replica fleet or browser-to-PM domain-behavior proof.