Skip to content

BFF session store (Valkey)

The API's BFF keeps server-side sessions in Redis. Outside Development it refuses to start with BFF enabled unless it has a Redis connection string (BffAuth:Redis:Configuration) and a key prefix (BffAuth:Redis:KeyPrefix). This page covers the Valkey instances that provide Redis for staging, the S30 rehearsal, PR previews and production (slice S08A). BFF is disabled in every environment. Nothing connects to these instances until a later, separately reviewed switch sets bffAuth.enabled: true.

Topology

Chris approved this topology on 2026-09-22. It replaces the earlier plan for separate staging and rehearsal instances.

Instance Namespace Used by ACL user → key pattern Persistence Memory request / limit
valkey-nonprod valkey-nonprod staging, S30 rehearsal, every PR preview staging → staging:*, rehearsal → rehearsal:*, preview → pr-* none 32Mi / 128Mi
valkey-production valkey-production production production → production:* AOF on a 1Gi standard-rwo PVC 64Mi / 192Mi

The shared instance isolates environments the same way RabbitMQ uses vhosts. Each environment class has its own ACL user, and that user can touch only keys under its prefix. The default user is disabled. All previews share the preview user, and each preview uses the prefix pr-<N>:. That gives only soft isolation between previews, which is acceptable because previews hold only synthetic data.

How the prefix is applied

BffAuth:Redis:KeyPrefix (chart value bffAuth.redis.keyPrefix) becomes the outermost part of the StackExchange.Redis cache InstanceName, and that cache puts the instance name in front of every key it touches. A session key therefore looks like this:

staging:syrf:bff:v1:<binding-hash>:<session-hash>
└──┬───┘└─┬─┘└──────────── SessionKeySpace ────────┘
 prefix  app

The prefix is validated at startup (BffRedisKeyPrefix). It may contain lowercase letters, digits and hyphens, must start with a letter or digit, must end with its only :, and may be at most 64 characters. Because the only colon is the last character, no key under one prefix can begin with another prefix. That holds even for pairs like pr-1: and pr-12:. The prefix is required when BFF is enabled outside Development, Local and E2E. Development, Local and E2E may leave it empty and keep the old syrf: layout.

Server configuration (SyRF chart src/charts/syrf-valkey)

  • Image: the official valkey/valkey image (BSD-3-Clause), pinned by multi-arch index digest in values.yaml. To update it, change image.tag and image.digest together, then run helm unittest src/charts/syrf-valkey.
  • ACL: an init container builds /acl/users.acl in an in-memory emptyDir from one mounted Secret per user. The file contains only the SHA-256 of each password, never the password. The chart refuses user names, key patterns and command rules outside a closed character set, and refuses rules that would grant every command, key or channel.
  • Command rules (aclUserRules): -@all +@connection +@hash +@keyspace -@dangerous -scan -randomkey -dbsize -move. The ASP.NET Core Redis cache needs only HMGET/HSET/HMSET, EXPIRE and DEL. The rules also block SCAN, RANDOMKEY, DBSIZE, INFO, KEYS, CONFIG, FLUSH*, Pub/Sub and scripting, because those would reveal or affect other environments' keys. Without INFO, the cache cannot detect the server version, logs that once, and falls back to HMSET.
  • Memory: maxmemory 64mb with maxmemory-policy volatile-ttl. Every session key has a TTL (its absolute expiry), so under memory pressure the sessions closest to expiring are evicted first. That disrupts users the least. allkeys-lru would evict an active user's session as readily as a stale one. An idle server uses about 4 MiB. A session is at most a few KiB, so 64 MiB holds thousands. In production, maxmemory is kept at a third of the 192Mi limit because an AOF rewrite forks the process, and its copy-on-write pages count outside maxmemory.
  • Pod: one replica with the Recreate strategy. It runs as a non-root user with a read-only root filesystem, no capabilities, no service-account token and TCP probes. The probes use TCP because the disabled default user cannot PING without a password.
  • NetworkPolicy: ingress on port 6379 is allowed only from syrf-api pods in the permitted namespaces, and all egress is denied. The cluster does not enforce NetworkPolicy today. GKE has addonsConfig.networkPolicyConfig.disabled: true and does not use Dataplane V2. The policies record intent, and the enforced boundary is ACL authentication plus key patterns. Turning on enforcement is a camarades-infrastructure change and is tracked separately.
  • Persistence (production): appendonly yes with appendfsync everysec on a PVC annotated argocd.argoproj.io/sync-options: Delete=false,Prune=false. Deleting the Argo Application therefore does not delete the persisted sessions.
  • Encryption in transit: not enabled. All traffic stays inside the cluster. Adding TLS with a cert-manager certificate is a follow-up.

Credentials (cluster-gitops)

No credential is stored in git or created by hand.

  1. The valkey-acl-password ESO ClusterGenerator, defined in argocd/local/argocd-secrets, generates 48 alphanumeric characters. The value is alphanumeric because , and = are separators in a StackExchange.Redis connection string.
  2. plugins/local/valkey-<instance>/resources/acl-credentials.yaml creates one CreatedOnce ExternalSecret per ACL user, for example valkey-staging-password with key password, in the Valkey namespace. Each user also gets a read-only ServiceAccount whose Role may get only that one Secret, and a kubernetes-provider ClusterSecretStore named <namespace>-<user>. The store's conditions admit only that user's consumer namespaces.
  3. Each consumer namespace gets a namespace-local bff-redis Secret with key connectionString. This is the API chart's existing bffAuth.redis.secretName contract.
  4. staging: an externalSecrets entry in plugins/local/extra-secrets-staging/values.yaml, read from the valkey-nonprod-staging store.
  5. production: an externalSecrets entry in plugins/local/extra-secrets-production/values.yaml, read from the valkey-production-production store.
  6. previews: the bff-redis-preview ClusterExternalSecret in plugins/local/valkey-nonprod/resources/ creates bff-redis in every namespace labelled syrf.org.uk/environment: preview, read from the valkey-nonprod-preview store. The preview ApplicationSet passes bffAuth.redis.secretName=bff-redis and bffAuth.redis.keyPrefix=pr-<N>:.
  7. rehearsal (S30): the valkey-nonprod-rehearsal store admits only the namespace syrf-rehearsal. S30 adds that namespace's bff-redis ExternalSecret and sets keyPrefix: "rehearsal:". It must never use the staging credential or prefix. Prepared (not merged) in camaradesuk/cluster-gitops#1188 (plugins/local/extra-secrets-rehearsal); the rehearsal API is the only BFF consumer with bffAuth.enabled: true. preflight.sh --mode isolated-rehearsal fails if the store, the user=rehearsal connection string or the prefix drift.

The connection string has the form valkey-nonprod.valkey-nonprod.svc.cluster.local:6379,user=<user>,password=<generated>,abortConnect=false. The staging and production API values already set bffAuth.redis.secretName: bff-redis and a keyPrefix, and keep bffAuth.enabled: false explicitly. The chart renders no BFF environment variables while BFF is disabled, so these values change nothing in the rendered API until the enable switch.

Rotating a password

CreatedOnce values are never regenerated while their Secret exists, so rotation is a reviewed GitOps change that introduces a new Secret name rather than an in-place edit:

  1. In cluster-gitops, rename the user's password ExternalSecret and its target (for example valkey-staging-password → valkey-staging-password-v2), the matching Role resourceNames, the passwordSecret.name in the instance's values.yaml, and the consumer's remoteRef.key.
  2. Argo syncs: ESO generates the new value. The renamed passwordSecret.name changes the Helm-rendered volume secretName in the pod spec, and that spec change is what rolls the Valkey Deployment. The ACL is rendered only by the init container at pod start. The consumer bff-redis Secret picks up the new value within an hour (refreshInterval), and API pods read it on their next restart.

Never change a password Secret's contents in place under the same name. Kubernetes does not restart a pod when a mounted Secret changes, and the init container does not re-run, so Valkey would keep the old password while the consumer bff-redis Secret refreshes to the new one and the API starts failing authentication.

For the non-production instance a Valkey restart clears all sessions, because it has no persistence. For production, the AOF keeps sessions across the restart.

Verifying locally

helm unittest src/charts/syrf-valkey
helm template valkey-nonprod src/charts/syrf-valkey -f <cluster-gitops>/plugins/local/valkey-nonprod/values.yaml
dotnet test src/services/api/SyRF.API.Endpoint.Tests --filter "FullyQualifiedName~BffRedisKeyPrefixTests"

BffRedisKeyPrefixTests runs the real RedisCache over a recording connection. It proves that every key the session store reads, writes, refreshes or removes carries the prefix, and that two environments sharing one server never address the same key.