Required Kubernetes Secrets for SyRF Services

Overview

This document lists all Kubernetes secrets required for SyRF services to run in the cluster. These secrets must be created in the syrf-staging namespace before services can start.

GCP Configuration Reference

Important: All GCP resources for SyRF are in project camarades-net (region europe-west2).

For complete infrastructure details — project/region values, DNS zones, GCP Secret Manager and the External Secrets Operator gcpsm-secret-store ClusterSecretStore — see SyRF Cloud Infrastructure Reference. GKE cluster configuration itself is documented in the camarades-infrastructure repository and published at docs.syrf.org.uk/infrastructure/cluster-config/.

External Services Behind These Secrets

All credentials are stored in GCP Secret Manager and synced into Kubernetes by the External Secrets Operator. This table maps each external service to the Kubernetes secret that carries its credentials; the per-secret YAML shapes follow below.

Service	Details	K8s Secret
Auth0	Domain: syrf.eu.auth0.com, Tenant: syrf	auth0
MongoDB Atlas	Cluster0 (M20, production), Preview (staging + PRs), Region: AWS eu-west-1	mongo-db (API, PM, Quartz)
MongoDB Atlas (Identity)	Non-Development Identity reads its own database via mongodb-identity and needs a separate read-only mongodb-project-management secret; its chart does not consume mongo-db	mongodb-identity, mongodb-project-management
Elasticsearch	Elastic Cloud - search indexing + APM	elastic-db
S3	Bucket: syrfapp-uploads (user file uploads)	aws-s3
SES	Email from noreply@syrf.org.uk. Production only (the default transport everywhere unless an environment opts into SMTP)	aws-ses (API, PM, Quartz); Identity uses ses-credentials (keyId, accessKey, fromAddress)
Mailpit (non-production only)	Shared, authenticated mail catcher for staging, rehearsal and previews, used where emailTransport.provider: smtp (never production). See below and non-production email	mailpit-smtp-credentials (syrf-staging today; API and Identity read the optional keys password and fromAddress; the chart default name is smtp-credentials)
Google OAuth (Identity)	Google sign-in for the Identity service (clientId, clientSecret; reused Auth0-era client)	google-oauth
Sentry	Error tracking and performance	sentry
Elastic APM	Application Performance Monitoring	elastic-apm
ROB AI (mothballed)	Retired MapsGroup proof of concept; no runtime consumer	Existing rob-api-credentials retained externally; not required
Google Sheets	Data export (service account auth)	google-sheets
RabbitMQ	In-cluster message broker, receives S3 Lambda events	rabbit-mq

Non-production email (Mailpit)

Production always sends through AWS SES (aws-ses / ses-credentials above). Staging, the S30 rehearsal environment and PR previews instead share one authenticated Mailpit instance, deployed and owned entirely in cluster-gitops (plugins/local/mailpit, chart charts/mailpit) — there is no SyRF-owned chart for it. SyRF services send to it through the configuration-selected SMTP transport (emailTransport.provider: smtp, see non-production email); an environment uses it only once its cluster-gitops values opt in.

• Web UI / login / per-environment message tags / credential shape: cluster-gitops docs/how-to/shared-mailpit-non-production-email.md.
• The syrf-staging namespace has a mailpit-smtp-credentials Secret. The SyRF charts read only password and fromAddress from it (emailTransport.smtp.secretName). Host, port, username and TLS mode are plain Helm values under emailTransport.smtp.
• In SMTP mode the API still uses its aws-ses Secret: it renders every templated message through SES TestRenderEmailTemplate (nothing is sent through SES). That SES identity therefore needs ses:TestRenderEmailTemplate, and must be in the account and region that hold the stored templates.
• Rehearsal and preview credentials follow the identical shape but are not yet wired (no consumer, no rehearsal namespace yet) — see the cluster-gitops doc for the exact pattern to add when that lands.

Critical Note on Secret Management

DO NOT commit secrets to Git. These secrets should be managed via:

1. External Secrets Operator (recommended) - syncs from Google Secret Manager
2. Manual kubectl creation (temporary/development only)
3. Sealed Secrets (alternative GitOps-friendly approach)

API Service Secrets

The API service requires 14 secrets:

Authentication & Authorization

1. auth0

apiVersion: v1
kind: Secret
metadata:
  name: auth0
  namespace: syrf-staging
type: Opaque
stringData:
  clientSecret: "<AUTH0_CLIENT_SECRET>"

2. swagger-auth

apiVersion: v1
kind: Secret
metadata:
  name: swagger-auth
  namespace: syrf-staging
type: Opaque
stringData:
  clientSecret: "<SWAGGER_CLIENT_SECRET>"

3. public-api

apiVersion: v1
kind: Secret
metadata:
  name: public-api
  namespace: syrf-staging
type: Opaque
stringData:
  apiKey: "<PUBLIC_API_KEY>"

Databases

4. mongo-db

apiVersion: v1
kind: Secret
metadata:
  name: mongo-db
  namespace: syrf-staging
type: Opaque
stringData:
  username: "<MONGODB_USERNAME>"
  password: "<MONGODB_PASSWORD>"

5. elastic-db

apiVersion: v1
kind: Secret
metadata:
  name: elastic-db
  namespace: syrf-staging
type: Opaque
stringData:
  username: "<ELASTICSEARCH_USERNAME>"
  password: "<ELASTICSEARCH_PASSWORD>"

6. dev-postgres-credentials

apiVersion: v1
kind: Secret
metadata:
  name: dev-postgres-credentials
  namespace: syrf-staging
type: Opaque
stringData:
  postgresql-password: "<POSTGRES_PASSWORD>"

Message Queue

7. rabbit-mq

apiVersion: v1
kind: Secret
metadata:
  name: rabbit-mq
  namespace: syrf-staging
type: Opaque
stringData:
  password: "<RABBITMQ_PASSWORD>"

AWS Services

8. aws-s3

apiVersion: v1
kind: Secret
metadata:
  name: aws-s3
  namespace: syrf-staging
type: Opaque
stringData:
  keyId: "<AWS_ACCESS_KEY_ID>"
  accessKey: "<AWS_SECRET_ACCESS_KEY>"

9. aws-ses

apiVersion: v1
kind: Secret
metadata:
  name: aws-ses
  namespace: syrf-staging
type: Opaque
stringData:
  keyId: "<AWS_ACCESS_KEY_ID>"
  accessKey: "<AWS_SECRET_ACCESS_KEY>"

9a. smtp-credentials (non-production only, when emailTransport.provider: smtp)

Both keys are optional: without password the client authenticates with an empty password (Mailpit with MP_SMTP_AUTH_ACCEPT_ANY), and without fromAddress the sender defaults to noreply@syrf.org.uk. The SMTP username is not secret; it is the per-environment emailTransport.smtp.username value.

apiVersion: v1
kind: Secret
metadata:
  name: smtp-credentials
  namespace: syrf-staging
type: Opaque
stringData:
  password: "<MAILPIT_SMTP_PASSWORD>"
  fromAddress: "<SENDER_ADDRESS>"

External Services

10. google-sheets

apiVersion: v1
kind: Secret
metadata:
  name: google-sheets
  namespace: syrf-staging
type: Opaque
stringData:
  serviceAccountEmail: "<SERVICE_ACCOUNT_EMAIL>"
  password: "<SERVICE_ACCOUNT_PASSWORD>"
  key-cert.json: |
    <GOOGLE_SERVICE_ACCOUNT_JSON_KEY>

11. rob-api-credentials — historical, not required

The MapsGroup AI proof of concept is mothballed. SyRF no longer consumes this secret; do not provision it for new environments. Existing credentials and infrastructure are untouched by the code retirement. See the retirement boundary.

Historical shape (for inventory only):

apiVersion: v1
kind: Secret
metadata:
  name: rob-api-credentials
  namespace: syrf-staging
type: Opaque
stringData:
  open-ai-api-key: "<OPENAI_API_KEY>"
  maps-api-key: "<MAPS_API_KEY>"

Monitoring & Error Tracking

12. elastic-apm

apiVersion: v1
kind: Secret
metadata:
  name: elastic-apm
  namespace: syrf-staging
type: Opaque
stringData:
  secretToken: "<ELASTIC_APM_SECRET_TOKEN>"
  serverUrl: "<ELASTIC_APM_SERVER_URL>"

13. sentry

apiVersion: v1
kind: Secret
metadata:
  name: sentry
  namespace: syrf-staging
type: Opaque
stringData:
  dsnUrl: "<SENTRY_DSN_URL>"

Project Management Service Secrets

PM service requires the same secrets as API (shared configuration).

Identity Service Secrets

Statistics parity evidence client (Identity, optional)

Off by default; required only while an environment runs the FEAT-024 automated parity capture.

Secret	Key	Consumed as	Where it lives
syrf-statistics-evidence	clientSecret	SYRF__StatisticsParityEvidence__ClientSecret (Identity)	Kubernetes Secret in the Identity namespace, created through cluster-gitops' usual secret path (External Secrets / GCP Secret Manager)

It renders only when the Identity values set statisticsParityEvidence.enabled: true (statisticsParityEvidence.clientSecret.{secretName,key} default to the table above). With the secret configured (trimmed, at least 32 characters; a shorter one is logged and treated as off), Identity seeds the confidential client-credentials client syrf-statistics-evidence (a fixed id) whose only scope is statistics:parity:read; without it — or with a blank or CHANGEME value in any case — Identity seeds nothing and deletes every registration holding that scope, so removing the secret revokes the credential once Identity restarts (already-issued tokens stay valid for up to an hour). The API needs no secret or setting for it.

The token reads GET /api/admin/project-statistics/{projectId}/parity for allowlisted projects only and is rejected on every other route. See the staging proof runbook for the threat model and capture procedure. Generate a long random value; never reuse another client's secret.

Statistics operator client (Identity, optional)

Off by default; staging only.

Secret	Key	Consumed as	Where it lives
syrf-statistics-operator	clientSecret	SYRF__StatisticsOperator__ClientSecret (Identity)	Kubernetes Secret in the Identity namespace, generated in-cluster by an ExternalSecret (CreatedOnce) in cluster-gitops

It renders only when the Identity values set statisticsOperator.enabled: true (statisticsOperator.clientSecret.{secretName,key} default to the table above). The same rules as the evidence client apply (trimmed, at least 32 characters, blank/CHANGEME/short means off and revokes every registration holding the scope). Identity then seeds syrf-statistics-operator (a fixed id) whose only scope is statistics:operate, accepted by the API on the statistics administration operations alone (pending index, fold status/enable/disable/reset, per-family backfill/rebuild, parity read). See the staging proof runbook.

Quartz Service Secrets

Quartz requires a subset of the API secrets:

• mongo-db
• rabbit-mq
• elastic-apm (optional)
• sentry (optional)

Web Service Secrets

Web service (Angular) typically doesn't require secrets for startup, but may need:

• Configuration for API endpoints (via ConfigMap, not Secret)

Quick Deploy All Secrets (Development Only)

WARNING: Only use this for local development. For production, use External Secrets Operator.

# Create all secrets from a secrets.env file
kubectl create secret generic auth0 \
  --from-literal=clientSecret="${AUTH0_CLIENT_SECRET}" \
  -n syrf-staging

kubectl create secret generic mongo-db \
  --from-literal=username="${MONGODB_USERNAME}" \
  --from-literal=password="${MONGODB_PASSWORD}" \
  -n syrf-staging

# ... repeat for all secrets

Using External Secrets Operator (Recommended)

Prerequisites

1. External Secrets Operator installed in cluster
2. Google Secret Manager configured
3. Workload Identity set up for ESO

Example ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: mongo-db
  namespace: syrf-staging
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcpsm-secret-store
    kind: ClusterSecretStore
  target:
    name: mongo-db
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: syrf-staging-mongodb-username
  - secretKey: password
    remoteRef:
      key: syrf-staging-mongodb-password

Verification

Check if all required secrets exist:

# List all secrets in staging namespace
kubectl get secrets -n syrf-staging

# Expected secrets for API service:
# - auth0
# - swagger-auth
# - public-api
# - mongo-db
# - elastic-db
# - dev-postgres-credentials
# - rabbit-mq
# - aws-s3
# - aws-ses
# - google-sheets
# - rob-api-credentials (historical only; no runtime consumer)
# - elastic-apm
# - sentry

Troubleshooting

Pod stuck in "Pending" state

Symptom: Pod shows FailedMount events

Check:

kubectl describe pod <pod-name> -n syrf-staging | grep -A 5 "Events:"

Common errors:

• secret "google-sheets" not found - Create the missing secret
• secret "mongo-db" not found - Create the missing secret

Service fails to start after secrets are created

Check pod logs:

kubectl logs <pod-name> -n syrf-staging

Common issues:

• Incorrect secret key names (must match what service expects)
• Invalid credentials (wrong password, expired tokens)
• Missing permissions (IAM, database access)

Security Best Practices

1. Never commit secrets to Git
2. Use External Secrets Operator for production
3. Rotate secrets regularly (at least every 90 days)
4. Use RBAC to limit access to secrets
5. Encrypt secrets at rest (enabled by default in GKE)
6. Audit secret access via GCP Cloud Audit Logs

Next Steps

1. For Production: Set up External Secrets Operator

2. See: External Secrets Operator documentation

3. For Development: Create secrets manually using kubectl create secret

4. Migration from Jenkins X: Extract secrets from old cluster using kubectl get secret -o yaml

GitHub Actions Repository Secrets

The following secrets must be configured in the GitHub repository settings (Settings > Secrets and variables > Actions) for CI/CD workflows to function correctly.

PR Preview Workflow Secrets

The PR preview workflow (.github/workflows/pr-preview.yml) requires these secrets for resource cleanup and MongoDB Atlas integration:

GCP Workload Identity (Required for cleanup jobs)

Secret Name	Description	Example
GCP_WORKLOAD_IDENTITY_PROVIDER	Workload Identity provider URI	projects/123456789/locations/global/workloadIdentityPools/github/providers/github
GCP_SERVICE_ACCOUNT	GCP service account email	github-actions@camarades-net.iam.gserviceaccount.com

These secrets enable the cleanup job to:

• Authenticate to GKE cluster
• Delete RabbitMQ vhosts for closed PRs
• Manage preview environment resources

MongoDB Atlas (Required for PR previews)

Secret Name	Description	Example
ATLAS_PROJECT_ID	MongoDB Atlas project ID	abc123def456

This secret is required for generating AtlasDatabaseUser manifests for PR preview databases.

Statistics parity evidence (optional)

Secret Name	Description
SYRF_STATISTICS_EVIDENCE_CLIENT_SECRET	Environment secret of the main-only statistics-evidence environment, not a repository secret. The same value as the Identity syrf-statistics-evidence Secret; read from the environment by scripts/fetch-statistics-parity.py. Set only while an automated FEAT-024 parity capture runs, and delete it with the Identity Secret afterwards.

Statistics operator (optional)

Secret Name

Description

SYRF_STATISTICS_OPERATOR_CLIENT_SECRET

Environment secret of the main-only statistics-operator environment, not a repository secret. The same value as the staging Identity syrf-statistics-operator Secret; used only by the Statistics Operator workflow (scripts/statistics-operator.py). Copy it after the cluster-gitops sync with kubectl -n syrf-staging get secret syrf-statistics-operator -o jsonpath='{.data.clientSecret}' \| base64 -d \| gh secret set SYRF_STATISTICS_OPERATOR_CLIENT_SECRET --env statistics-operator -R camaradesuk/syrf, which prints nothing. Environment setup: runbook.

Setting Up Workload Identity Federation

1. Create Workload Identity Pool (if not exists):

gcloud iam workload-identity-pools create "github" \
  --project="camarades-net" \
  --location="global" \
  --display-name="GitHub Actions Pool"

1. Create Provider:

gcloud iam workload-identity-pools providers create-oidc "github" \
  --project="camarades-net" \
  --location="global" \
  --workload-identity-pool="github" \
  --display-name="GitHub" \
  --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository" \
  --issuer-uri="https://token.actions.githubusercontent.com"

1. Grant Access to Service Account:

gcloud iam service-accounts add-iam-policy-binding \
  "github-actions@camarades-net.iam.gserviceaccount.com" \
  --project="camarades-net" \
  --role="roles/iam.workloadIdentityUser" \
  --member="principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/github/attribute.repository/camaradesuk/syrf"

1. Get Provider URI:

gcloud iam workload-identity-pools providers describe "github" \
  --project="camarades-net" \
  --location="global" \
  --workload-identity-pool="github" \
  --format="value(name)"

1. Add secrets to GitHub:
2. Navigate to Settings > Secrets and variables > Actions
3. Add GCP_WORKLOAD_IDENTITY_PROVIDER with the provider URI
4. Add GCP_SERVICE_ACCOUNT with the service account email

Related Documentation

• GitOps Deployment Guide
• External Secrets Operator
• GCP Workload Identity Federation