Run the application-authority fixture

The application-authority fixture (milestone M0a of the application authority transition) is an isolated, native, two-replica stack for proving authorization behaviour with real sessions and tokens. It is test-only and opt-in: no CI workflow runs it. Since M1a it can also run both API replicas in any application-authority mode (--application-roles-mode). Since M1b the lane runs enforced by default and proves role grant and revoke; since M1c it also proves the lifecycle-wide last-administrator guard; since M2a it also proves that every other application capability follows the same mode contract (see the claim-consumer ledger).

bash e2e/authority/run.sh

Run it from any worktree root. The command builds the services it needs, starts the stack, seeds synthetic accounts, runs the specs and tears the stack down again. Expect about 3 minutes on a cold build and under a minute with --skip-build.

Option	Effect
--skip-build	Reuse this worktree's existing Release binaries of API, PM, Identity and the seeder.
--keep	Leave the stack running afterwards (for debugging). Stop it with bash e2e/authority/teardown.sh.
--application-roles-mode enforced\|shadow\|off	Mode for both API replicas. The default is enforced since M1b, because the lane proves the target contract; off is the deployed default and proves it is unchanged. Sets FeatureFlags__SyrfOwnedApplicationRoles and FeatureFlags__SyrfOwnedApplicationRolesEnforced, and (M1c) the same pair for Identity. After the specs, the runner prints each replica's count of logged authority decisions and disagreements.
-- <args>	Pass the remaining arguments to Playwright, for example -- --grep refresh.

What it runs

Component	Where	Notes
Front door	https://localhost:<front>	The single public origin, like the deployed ingress. Identity registers its BFF callback and Swagger redirect here. Routes each request to replica A or B named by the x-authority-replica header or authority-replica cookie; a request naming neither gets 421. Every response carries x-authority-served-by.
API replica A and B	http://127.0.0.1:<api>, <api-second>	Identical configuration: BFF enabled on provider openiddict, sessions in the shared Valkey, per-request native introspection. The /api/e2e-auth/login bypass and the development picker are off.
Identity	https://localhost:<identity>	The real Identity host outside Development, so its full production configuration contract applies. Throwaway CA, TLS, signing and data-protection material per run.
Project Management	http://127.0.0.1:<pm>	Started first: Identity's readiness requires the PM collections.
API Mongo relay	127.0.0.1:<quartz> (e2e/authority/mongo-gate.mjs)	The API replicas, and only they, reach MongoDB through this loopback relay, on the slot's otherwise unused Quartz port (#3859). A spec pauses it (setApiMongoGate('paused'), a SIGUSR1 to its pid) to make the PM store stop answering the API mid-request while Identity, and so token introspection, keeps working; SIGUSR2 resumes it. Paused bytes are held, never dropped.
MongoDB rs0, RabbitMQ, Valkey	containers	Single-member replica set (mongo:8, as the ordinary E2E stack), rabbitmq:3.13-management, and the digest-pinned Valkey image of the syrf-valkey chart. No Docker volumes: data lives in tmpfs.

The synthetic accounts (admin, member, suspended, pm-admin, m1a-admin, m1b-root, m1b-target, m1c-a, m1c-b, all @authority-e2e.invalid) are created through Identity's own UserManager by e2e/authority/tools/AuthoritySeeder, with passwords generated for each run. A first BFF sign-in creates each PM investigator record, exactly as in production.

Isolation and safety

• Own stack identity. The lane runs as a run-scoped stack, E2E_STACK_ID=authority-pr<N> (or authority-<hash> outside a pr<N>.<slug> worktree), through e2e/scripts/ports.sh. It therefore gets its own port slot and reservation, Compose project syrf-e2e-authority-…, network, databases, cookie names and state directory /tmp/syrf-e2e-authority-…, and never adopts this worktree's ordinary E2E stack. See the local multi-stack runbook.
• Loopback only. Every endpoint is checked for loopback before any process starts, the seeder refuses a non-loopback MongoDB, and published container ports bind to 127.0.0.1.
• No developer settings. Each service starts under env -i with an explicit variable list and an empty private HOME, so shell variables, the .NET user-secrets store and ~/.config/syrf cannot reach it. Every offset-0 default in appsettings.e2etest.json that belongs to the ordinary E2E stack (Mongo, RabbitMQ, LocalStack, mock OIDC) is overridden to this stack's slot.
• Verified before mutation. The first spec step checks that MongoDB, RabbitMQ and Valkey are published by this stack's own labelled containers on the manifest ports, that MongoDB is the rs0 primary, that Identity is the local issuer, and that both replicas' BFF login redirects to it. Nothing is written before those checks pass.
• TLS stays verified. Node trusts only the run's throwaway CA (NODE_EXTRA_CA_CERTS); Chromium accepts the fixture's certificate by its SPKI pin rather than ignoring certificate errors.
• Exact teardown. teardown.sh stops the lane's processes by pid file (as process groups), removes containers of the syrf-e2e-authority-… project only after each one's org.syrf.e2e.worktree label proves ownership, removes the network, releases the reservation and deletes the state directory. It then verifies that no process, container, volume, network or listening port of the stack remains, and fails otherwise. It never touches the shared syrf-quartz-dev container, the ordinary E2E stack or another worktree's stack, and it is idempotent. run.sh calls it on every exit unless you pass --keep.
• No secrets in output. Passwords and tokens stay in the 0600 state directory; Playwright traces, screenshots and video are off; the front door never logs query strings.

Reading the results

The run passes when every baseline test passes and every characterization fails as expected.

Spec	Tests	Expected today
authority-baseline.spec.ts	Local-stack guard; native BFF sign-in on A and on B, each honoured on both replicas; BFF refresh on A and on B; direct native bearer (Identity-issued PKCE token, introspected per request) on A and B; SignalR negotiate + JSON handshake on A and B with the BFF session and with a bearer token	Pass (9)
authority-m1a-application-roles.spec.ts	ListUsers for an Identity-claim administrator with no PM role, and for a deactivated PM administrator, on A and B with sessions and a bearer token. M2a (same file, sharing its two sign-ins): the email-template, batch-administration and flag-management reads, a flag mutation and a no-recipient administrative send, on A and B with the session and a bearer token, for the claim without the PM role, for the PM role, and after revoking it; the self-capabilities response agrees with every answer. M3a (same sign-ins): the caller, holding the PM Administrator role and the administrator claim, creates a private project and reads its members on A and B; removing their membership by a majority write denies the next request on A and B (enforced only). #3859 outage (same sign-ins): with the PM Administrator role and the claim, ListUsers bearer requests to A and B are sent while the API's Mongo relay is paused for 12 s	Pass in every mode. Outage: enforced answers 503 authority-unavailable, Retry-After: 5, within 8 s, before the store returns; off/shadow answer 200 once it returns (the decision is unchanged); all answer 200 afterwards. (M1a 2; M2a 2 in off/shadow, 3 in enforced): the claim decides in off/shadow (the capabilities read is 404); in enforced the claim alone gets 403 everywhere, the PM role is authorized (the mutation reaches validation, 400, and is never applied), and the revoke denies the next request on A and B
authority-m1b-role-administration.spec.ts	M1b acceptance through PUT /api/investigators/{id}/application-roles only. Bootstrap administrator through the real operator tool (SyRF.ProjectManagement.ApplicationRoleBootstrap): dry run, apply refused without --confirm-database, apply, idempotent re-run. Grant, then revoke, observed on A and B with old BFF sessions, a direct token and a refreshed session; the revoked actor loses role administration at once; audit records committed. No positive cache (alternating grants and revokes). Revision conflict and idempotent retry. A2: an administrator who is not a project member gets 403 on /api/projects/{id}/investigators. Four rounds of concurrent removal of the last two administrators, by each other and by themselves	Pass (5) in enforced (1 skipped). In off/shadow: pass (1): the endpoints answer 404 on A and B and no audit record exists (5 skipped)
authority-m1c-lifecycle-guard.spec.ts	M1c acceptance with the real Identity. The last active administrator cannot be removed by DELETE /api/account/profile (A and B: 409 last-active-administrator), by Identity's admin delete without a reservation or with a forged operation ID (409 deactivation-not-reserved), or by a self-revoke; nothing is revoked or reserved. Concurrent deletions of the last two administrators: one 204 (reserved, finalized, Identity account gone, old token refused), the other 409. A deletion racing a role removal of the other administrator: exactly one takes effect. Consumes m1c-a/m1c-b; the third administrator is member, granted by API	Pass (3) in enforced (1 skipped). In off/shadow: pass (1): deletion unchanged, no reservation written (3 skipped)
authority-m4a-suspension.spec.ts	M4a acceptance. A PM-only suspension (no Identity call) refuses the old BFF session, a session refreshed before it, a direct bearer token and a socket negotiate on A and B with 403 subject-suspended, and removing it admits the next request. Under a PM-only suspension Identity (which gets the same applicationSuspension value) refuses the BFF refresh (401, the session ends) and a new authorization code (access_denied) (#3896). The API suspension: self-suspension 409; suspend 200 with withdrawal: completed and a settled application-suspension reservation, replayed on B; afterwards the old session, token and refresh are refused. Restore 200, twice 409 not-suspended; the old session and token stay refused and a new sign-in works on A and B. Three password sign-ins (m4a-admin, m4a-target twice)	Pass (3) in enforced (1 skipped). In off/shadow: pass (1): the endpoint answers 404 and a stored suspension refuses nothing (3 skipped)
authority-r1-characterization.spec.ts	R1 gaps, below	In enforced (default): all three gaps are closed and pass (the deactivation gap by M4a subject admission). In off/shadow: expected failure (3)

Each R1 characterization asserts its preconditions and the majority-acknowledged mutation normally, records the commit boundary (operationTime) and every status as test annotations, and checks that each post-commit answer is a real authorization answer (200/401/403). Only then does it call test.fail() and assert the R1 target, so a broken fixture is a real failure, never a hidden one.

Characterization	Observed today	Flips to passing in
Administrator role removed from PM and the Identity copy: existing BFF sessions (from A and B), a refreshed session and a direct bearer token still list users on A and B	200 everywhere with claims deciding (off/shadow)	Closed in enforced (M1a boundary, flipped as the lane default by M1b): 403 everywhere. Deployed environments close it at the M6 cutover
PM Deactivated investigator: existing sessions and token still read /api/projects on A and B	200 everywhere in off/shadow	Closed in enforced by M4a (#3888): applicationSuspension follows enforced mode in this lane, and its subject admission refuses a deleted, suspended or pending-deletion subject (403 everywhere). Deployed environments close it when they enable the flag after M6
PM Administrator role alone (no Identity group) does not grant ListUsers on A or B: the policy matches the case-sensitive claim group administrator	403 everywhere with claims deciding	Closed in enforced (M1a/M1b): 200 everywhere

When a milestone closes a gap, Playwright reports "expected to fail, but passed" and the run fails: delete that test's markExpectedGap(...) call in the same PR. The two ListUsers characterizations use markListUsersGap(...), which treats the gap as closed only when the run enforces PM authority on both replicas; they stay recorded as expected failures in off/shadow, which is what deployed environments run until M6. Results are written to e2e/test-results/authority/results.json.

Resource cost

Measured on the maintainer's host (2026-09-30), steady state after the specs:

Component	Memory
API replica A / B	~340 MB each
Project Management	~320 MB
Identity	~260 MB
MongoDB (data in tmpfs)	~355 MB
RabbitMQ	~170 MB
Front door	~70 MB
Valkey	~5 MB
Stack total	~1.9 GB, plus Chromium during the specs (~2.3 GB peak)

Time: about 3 minutes from a cold checkout (image pulls and a Release build of four projects with two build workers), about 50 seconds with --skip-build; the specs themselves take about 40 seconds in enforced mode (M1c).

Troubleshooting

• A service did not become ready. run.sh prints the last log lines. With --keep, all logs stay in /tmp/syrf-e2e-authority-…/*.log; after a failed run without --keep they are copied to a printed temporary directory before teardown.
• "Invalid email or password" after repeated runs against a kept stack. Identity's password-attempt protection allows five password checks per account per five minutes. A normal run stays inside that budget; start a fresh stack instead of re-running specs against a kept one.
• A sign-in stalls on the login page (the test times out in signInThroughBff). Identity also admits only 20 password attempts per source per five minutes (PasswordAttemptStore.SourceLimit), and every sign-in in this lane comes from the same loopback address. A full enforced run makes about 17; the 21^st is refused with the ordinary "Invalid email or password" page and no log line, which looks like a stall. New specs must reuse bearer tokens or existing sessions rather than add password sign-ins; count them in identity.log (POST /Account/Login). The lane now enforces this itself: signInThroughBff draws from lib/sign-in-budget.ts (18 sign-ins per five minutes in the single Playwright worker) and fails with a message naming the limit instead of stalling. Production limits are unchanged.
• A sign-in stalls after the password is accepted (#3893). identity.log shows POST /Account/Login answered 302, and then nothing from the browser: no /connect/authorize, no /api/auth/callback on the API. This host is the shared CI runner. When other jobs create or remove Docker networks, Chromium can see a host network change and drop the in-flight redirect (net::ERR_NETWORK_CHANGED). It happened in off, shadow and enforced runs alike. The application-authority mode does no work on the sign-in path: Identity reads only whether the mode is enforced, and the BFF login and callback never call the authority gate. signInThroughBff (lib/bff-sign-in-recovery.ts) waits at most 30 seconds for the callback. After a dropped or stalled navigation it resumes once by opening /api/auth/login again. Identity's cookie from the accepted password normally completes the flow without a second password attempt. It logs [authority] BFF sign-in on replica … resumed once: <cause>, and a second failure fails the test with the cause instead of waiting for the 120-second test timeout.
• Teardown refuses a container. A container with this stack's project label but another owner label exists. Inspect it; do not remove it by name.
• P3 read benchmark. The linearizable-read benchmark (three-member replica set, failover, deadline) is a separate stack, not part of this fixture: bash e2e/authority/benchmark/run.sh, results in P3 benchmark.
• Guard tests only: bash e2e/authority/test-guards.sh, node --test e2e/authority/front-door.test.mjs, node --test e2e/authority/sign-in-budget.test.mjs, node --test e2e/authority/sign-in-recovery.test.mjs and node --test e2e/authority/mongo-gate.test.mjs need no stack and also run in CI through .github/scripts/test-e2e-concurrency.sh.