CI/CD Workflow Overview¶

Purpose¶

This document explains the complete structure and execution flow of the ci-cd.yml GitHub Actions workflow, which handles versioning, building, and deploying all SyRF services.

Workflow File¶

Location: .github/workflows/ci-cd.yml

Trigger: Push to main branch, or manual workflow dispatch

Main release queue¶

Main releases share the workflow-level main-service-release concurrency group from version calculation through artifact publication and successful service tagging (#3692). The next run therefore fetches the previous release's tags before computing versions. Since queue order is not guaranteed to match commit order, a fail-closed remote-main-tip guard also runs before change detection: superseded commits emit no-release outputs, and the current main run releases their accumulated changes. A superseded manual dispatch fails explicitly instead: re-run it on current main with the same force-rebuild inputs. An ordinary push cannot replace an operator's recovery request. Failed or malformed remote lookups fail the job. Tags are still created only after successful publication; a calculation-only job lock would not cover this critical section.

The GitHub multi-run queue uses queue: max and disables cancellation. Up to 100 runs may wait; additional runs are cancelled by GitHub if that limit is reached. This intentionally reduces main release throughput to prevent overlapping version allocation. Test-only dispatches and non-main no-op dispatches have unique groups and do not block releases. Preview workflows are unchanged.

docs-rebuild.yml shares the same repository-wide lock because cross-repository rebuilds publish into the stable docs image/tag namespace too. It is main-only and checks out current main with full history after acquiring the lock, rather than building a queued event's older SyRF SHA. External docs can change without a new SyRF commit, so the rebuild advances the patch version past any already-published candidate before pushing images. It never overwrites a stable docs version merely because GitVersion returned the preceding run's tag. The main workflow uses the same allocator for its docs output, including forced rebuilds after an external rebuild at the same commit. Its Docker image, git tag and promotion all consume that output. Aggregation is unchanged. Before rebuilding, the external workflow also requires docs/ (including its chart), mkdocs.yml and requirements.txt to match the last reachable stable docs tag. If a main docs change is awaiting release, the external run fails with retry instructions instead of tagging it and suppressing the main pipeline's promotion.

Run bash .github/scripts/test-main-release-concurrency.sh to verify the queue contract and its negative fixtures. This protects concurrent releases; it does not redefine recovery from a manually cancelled/failed publication or change the existing retry contracts.

Complete Workflow Structure¶

┌─────────────────────────────────────────────────────────────────┐
│ 1. detect-changes                                               │
│    Path-based filtering determines which services changed       │
│    Outputs: api_changed, pm_changed, quartz_changed,            │
│             web_changed, s3_notifier_changed                    │
└─────────────────────────────────────────────────────────────────┘
                              │
    ┌─────────────────────────┼─────────────────────────┐
    │                         │                         │
    ▼                         ▼                         ▼
┌────────────────┐  ┌─────────────────┐  ┌─────────────────────────┐
│ 2. version-*   │  │ 2.5 test-dotnet │  │ 2e. version-s3-notifier │
│  (parallel)    │  │  (parallel)     │  │   (if s3_notifier       │
│                │  │                 │  │    changed)             │
│ version-api    │  │ - Run xUnit     │  │                         │
│ version-pm     │  │ - Upload        │  │ Uses GitVersion         │
│ version-quartz │  │   coverage      │  │ Outputs: version, sha   │
│ version-web    │  │                 │  └────────────┬────────────┘
│                │  ├─────────────────┤               │
│ Uses GitVersion│  │ 2.5 test-web    │               │
│ Outputs:       │  │  (parallel)     │               │
│  version, sha  │  │                 │               │
└───────┬────────┘  │ - Run Vitest    │               │
        │           │ - Upload        │               │
        │           │   coverage      │               │
        │           └────────┬────────┘               │
        │                    │                        │
        │    ┌───────────────┴───────────────┐        │
        │    │ BLOCKS BUILD IF TESTS FAIL    │        │
        │    └───────────────┬───────────────┘        │
        │                    │                        │
        ▼                    ▼                        ▼
┌───────────────────────────────────────┐  ┌─────────────────────────┐
│ 3. build-and-push-images              │  │ 4. deploy-lambda        │
│    (Docker services)                  │  │    (S3 Notifier only)   │
│                                       │  │                         │
│ - Build .NET images                   │  │ - Build Lambda package  │
│ - Push to GHCR                        │  │ - Deploy to AWS Lambda  │
│ - Tags: version, version-sha          │  │ - Upload package        │
│                                       │  │                         │
│ REQUIRES: test-dotnet AND test-web    │  │                         │
│           to pass (or be skipped)     │  │                         │
└─────────────────┬─────────────────────┘  └───────────┬─────────────┘
                  │                                    │
                  ├────────────┬───────────────────────┘
                  │            │
       ┌──────────▼────────┐   │
       │ 5. promote-to-    │   │
       │    staging        │   │
       │  (Docker only)    │   │
       │                   │   │
       │ - Clone gitops    │   │
       │ - Update values   │   │
       │ - Create PR       │   │
       └──────────┬────────┘   │
              │           │
              └─────┬─────┘
                    │
     ┌──────────────▼────────────────────┐
     │ 6. create-and-push-tags           │
     │    (Runs AFTER all builds)        │
     │                                   │
     │ ONLY creates tags if both:        │
     │ 1. Versioning succeeded           │
     │ 2. Build/deployment succeeded     │
     │                                   │
     │ Tags: {service}-v{version}        │
     │ NO ORPHANED TAGS                  │
     └──────────────┬────────────────────┘
                    │
     ┌──────────────▼────────────────────┐
     │ 7. create-lambda-release          │
     │    (S3 Notifier only)             │
     │                                   │
     │ - Download Lambda package         │
     │ - Create GitHub Release           │
     │ - Attach package as artifact      │
     └───────────────────────────────────┘

Job Details¶

1. detect-changes¶

Purpose: Determine which services need building based on changes since last tag

Implementation: Uses tag-based detection script (.github/scripts/detect-service-changes.sh)

How it works:

For each service, find the last matching git tag (e.g., api-v8.21.0)
Compare source files against that tag using git diff
Check if Docker image exists in GHCR for that version
Determine action: build, retag, skip

Source Paths Checked:

api: src/services/api/**, src/libs/kernel/**, src/libs/mongo/**, etc.
pm: src/services/project-management/**, plus shared libraries
quartz: src/services/quartz/**, plus dependencies
web: src/services/web/**
user_guide: user-guide/**
docs: docs/**, mkdocs.yml
s3_notifier: src/services/s3-notifier/**, src/libs/s3-notifier/**

Outputs per service:

{service}_changed: Boolean (true if source or chart changed)
{service}_needs_build: Boolean (true if Docker build required)
{service}_needs_retag: Boolean (true if only chart changed, can retag)

Manual Override: Use workflow dispatch with force_rebuild_* options to bypass detection.

See ADR-007: Tag-Based Change Detection for details.

2. version-* Jobs (Parallel Execution)¶

Purpose: Calculate semantic version for each changed service

Jobs: version-api, version-pm, version-quartz, version-web, version-s3-notifier

Trigger: Only runs if corresponding service changed

Process:

Checkout code with full git history (fetch-depth: 0)
Install GitVersion 6.x
Run GitVersion with service-specific config: src/services/{service}/GitVersion.yml
Output calculated version and SHA

Outputs:

version: Semantic version (e.g., 1.2.3)
sha: Short commit SHA

GitVersion Config:

Mode: ContinuousDeployment
Tag prefix: {service}-v (e.g., api-v, s3-notifier-v)
Conventional commits: feat:, fix:, chore:

2.5. test-* Jobs (Test Phase)¶

Purpose: Run automated tests before Docker builds to catch issues early

Jobs: test-dotnet, test-web

Trigger: Only runs if corresponding services changed

test-dotnet¶

Trigger: If API, PM, Quartz, or S3 Notifier changed

Process:

Checkout code
Setup .NET 10.x
Restore dependencies (dotnet restore syrf.sln)
Build solution (dotnet build --configuration Release)
Run tests with coverage (dotnet test with XPlat Code Coverage)
Upload test results and coverage artifacts

Timeout: 10 minutes

Artifacts:

dotnet-test-results-{sha}: TRX test result files
dotnet-coverage-{sha}: Cobertura XML coverage files

test-web¶

Trigger: If web service changed

Process:

Checkout code
Setup Node.js 20
Install dependencies (pnpm install --frozen-lockfile)
Run Vitest tests with coverage (pnpm exec ng test --no-watch --coverage)
Upload coverage artifacts

Timeout: 5 minutes

Artifacts:

web-coverage-{sha}: Coverage reports (lcov, cobertura, text)
web-test-results-{sha}: JUnit XML test results

Coverage Thresholds (configured in vitest.config.ts):

Statements: 50%
Branches: 40%
Functions: 50%
Lines: 50%

Tests fail if coverage drops below these thresholds.

Build Gate¶

The build-docker job requires tests to pass:

needs: [detect-changes, version, build-web-artifacts, test-dotnet, test-web]
if: |
  (needs.test-dotnet.result == 'success' || needs.test-dotnet.result == 'skipped') &&
  (needs.test-web.result == 'success' || needs.test-web.result == 'skipped')

If tests fail, Docker builds are blocked and deployment does not proceed.

See How to: Run Tests for local testing instructions.

3. build-and-push-images¶

Purpose: Build Docker images for all .NET and Angular services

Depends On: version-api, version-pm, version-quartz, version-web

Trigger: If ANY version job succeeded

Process:

Setup Docker Buildx
Login to GHCR (GitHub Container Registry)
For each successfully versioned service:
Build Docker image from Dockerfile
Push to ghcr.io/camaradesuk/syrf-{service}
Tag with: {version}, {version}-sha.{sha}, latest
Use GitHub Actions cache for layer caching

Services Built:

API: .src/services/api/SyRF.API.Endpoint/Dockerfile
PM: ./src/services/project-management/SyRF.ProjectManagement.Endpoint/Dockerfile
Quartz: ./src/services/quartz/SyRF.Quartz/Dockerfile
Web: ./src/services/web/Dockerfile

Permissions: contents: read, packages: write

4. deploy-lambda¶

Purpose: Build and deploy S3 Notifier to AWS Lambda

Depends On: version-s3-notifier

Trigger: If version-s3-notifier succeeded

Process:

Setup .NET 10.0
Install Amazon.Lambda.Tools globally
Restore dependencies
Build Lambda package: dotnet lambda package
Output: s3-notifier-{version}.zip
Configuration: Release
Framework: net10.0
Deploy to AWS Lambda: dotnet lambda deploy-function
Function name: syrfAppUploadS3Notifier
Region: eu-west-1
Runtime: dotnet10
Tags: Version, SHA, Repository
Upload package as GitHub artifact (90 day retention)

AWS Secrets Required:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

5. promote-to-staging¶

Purpose: Automatically create PR to cluster-gitops repository to deploy services to staging

Depends On: build-and-push-images, all version-* jobs (except s3-notifier)

Trigger: If build-and-push-images succeeded AND at least one Docker service versioned successfully

Process:

Checkout cluster-gitops repository using GITOPS_PAT
For each successfully built Docker service:
Update environments/staging/{service}.values.yaml
Replace image.tag with new version using sed
Create pull request using peter-evans/create-pull-request action
Branch: promote/staging-{run-number}
Title: "Promote services to staging"
Body: List of updated services with versions and source run link
Labels: staging, automated, gitops
Auto-delete branch after merge
Output summary to GitHub Actions step summary

Services Updated (Docker only):

API: environments/staging/api.values.yaml
PM: environments/staging/project-management.values.yaml
Quartz: environments/staging/quartz.values.yaml
Web: environments/staging/web.values.yaml

Note: S3 Notifier (Lambda) is NOT promoted via GitOps - it deploys directly to AWS Lambda

Immutable image tags: for the eight Docker services, service.imageTag is written as the immutable {version}-sha.{shortsha} tag (the same tag _docker-build.yml and the "Retag image" step already push alongside the mutable {version} tag), not the mutable {version} tag. When several merges to main produce the same GitVersion number (for example a run of non-bumping commits), a promotion that wrote the bare {version} tag could roll pods without changing the image reference, and imagePullPolicy: IfNotPresent then let a node keep serving its previously cached build for that tag — this is what happened to staging web on 2026-09-07, where two different builds ran under syrf-web:7.115.0 and the Angular app crashed with chunk 404s. The sha-suffixed tag always resolves to exactly one manifest, so every replica of a rollout serves the same build. s3-notifier is the one exception: it has no Docker image, and its chart derives the Lambda package's S3 key from image.tag (lambda-packages/s3-notifier-v{image.tag}.zip), so its imageTag stays the bare {version}. The promotion PR table displays this collected imageTag for each service, so Lambda rows show the package version without a Docker SHA suffix.

GitHub Secret Required:

APP_ID: GitHub App ID for cluster-gitops authentication
APP_PRIVATE_KEY: GitHub App private key for cluster-gitops authentication
See setup-gitops-github-app.md for configuration instructions

Example PR Body:

## Automated Staging Promotion

This PR updates the staging environment with the latest service versions from the CI/CD pipeline.

**Source Run**: [123](https://github.com/camaradesuk/syrf/actions/runs/123)
**Source Commit**: abc123def456

### Services Updated

- API: `v8.21.0`
- Web: `v5.0.1`

### Deployment

Once merged, ArgoCD will automatically sync the changes to the staging cluster.

6. create-and-push-tags¶

Purpose: Create git tags ONLY for successful builds/deployments

Depends On: build-and-push-images, deploy-lambda, all version-* jobs

Trigger: always() (runs even if some jobs failed)

Process:

Checkout code with full history
Configure git user as github-actions[bot]
For each service:
Check BOTH conditions: a) Version job succeeded b) Build/deployment job succeeded
If both true: Create annotated tag {service}-v{version}
If either false: Skip (no tag created)
Push all created tags to remote

Tag Logic (Docker Services):

if [ "${{ needs.version-api.result }}" = "success" ] &&
   [ "${{ needs.build-and-push-images.result }}" = "success" ]; then
  git tag -a "api-v${VERSION}" -m "Release API v${VERSION}"
fi

Tag Logic (Lambda):

if [ "${{ needs.version-s3-notifier.result }}" = "success" ] &&
   [ "${{ needs.deploy-lambda.result }}" = "success" ]; then
  git tag -a "s3-notifier-v${VERSION}" -m "Release S3 Notifier v${VERSION}"
fi

Summary Output: Shows which tags were created and which builds failed

6. create-lambda-release¶

Purpose: Create GitHub Release for S3 Notifier Lambda with package artifact

Depends On: create-and-push-tags, version-s3-notifier, deploy-lambda

Trigger: If all three dependencies succeeded

Process:

Download Lambda package from GitHub artifact
Create GitHub Release using softprops/action-gh-release
Tag: s3-notifier-v{version}
Name: S3 Notifier v{version}
Body: Deployment details (function name, region, runtime, commit)
Attach Lambda package (.zip file)
Not a draft or prerelease

Permissions: contents: write

GitOps Promotion Workflow¶

The promotion workflow creates a declarative deployment pipeline:

CI builds and pushes Docker images to GHCR
Promotion job creates PR to cluster-gitops repository
Human reviews and merges the PR (manual approval gate)
ArgoCD detects change and syncs to Kubernetes cluster
Services deployed to staging environment

This approach provides:

Audit trail: All deployments tracked via git history
Rollback capability: Revert PR to rollback deployment
Manual approval: Team can review before deployment
Declarative state: GitOps repository represents desired cluster state

Key Design Principles¶

1. No Orphaned Tags¶

Problem: Don't want git tags pointing to non-existent artifacts

Solution: Tags are created ONLY after successful builds/deployments

Example:

If API build fails → No api-vX.Y.Z tag created
If Lambda deployment fails → No s3-notifier-vX.Y.Z tag created
User sees clear indication in workflow summary

2. Partial Success Handling¶

Scenario: API builds successfully, PM build fails

Behavior:

API: ✅ Docker image pushed, ✅ tag created
PM: ❌ No Docker image, ❌ No tag created
Workflow continues (doesn't fail completely)
Summary shows which succeeded and which failed

3. Artifact Retention¶

Docker Images: Stored in GHCR indefinitely (until manually deleted)

Lambda Packages:

GitHub Artifacts: 90 days retention
GitHub Releases: Indefinite (attached to release)

4. Parallel Execution¶

Version Jobs: Run in parallel (5 jobs simultaneously)

Build Jobs: Sequential within job but parallel between services

Benefit: Faster overall workflow execution

Workflow Execution Examples¶

Example 1: API and PM Changed¶

1. detect-changes
   └─ api_changed=true, pm_changed=true

2. version-api (parallel with version-pm)
   └─ version=1.2.3, sha=abc123

3. version-pm
   └─ version=2.3.4, sha=abc123

4. build-and-push-images
   ├─ Build API → ghcr.io/camaradesuk/syrf-api:1.2.3
   └─ Build PM → ghcr.io/camaradesuk/syrf-pm:2.3.4

5. promote-to-staging
   ├─ Update api.values.yaml → tag: "1.2.3"
   ├─ Update project-management.values.yaml → tag: "2.3.4"
   └─ Create PR to cluster-gitops: "Promote services to staging"

6. create-and-push-tags
   ├─ Create tag: api-v1.2.3
   └─ Create tag: pm-v2.3.4

Example 2: S3 Notifier Changed¶

1. detect-changes
   └─ s3_notifier_changed=true

2. version-s3-notifier
   └─ version=1.0.5, sha=def456

3. deploy-lambda
   ├─ Build package: s3-notifier-1.0.5.zip
   ├─ Deploy to syrfAppUploadS3Notifier
   └─ Upload artifact

4. create-and-push-tags
   └─ Create tag: s3-notifier-v1.0.5

5. create-lambda-release
   ├─ Download s3-notifier-1.0.5.zip
   └─ Create GitHub Release with package

Example 3: Build Failure¶

1. detect-changes
   └─ api_changed=true

2. version-api
   └─ version=1.2.4, sha=ghi789

3. build-and-push-images
   └─ Build API → FAILS (Docker error)

4. create-and-push-tags
   └─ NO TAG CREATED (build failed)

Summary shows:
- ❌ API: v1.2.4 (Docker build failed - no tag created)

Monitoring and Debugging¶

View Workflow Run¶

Go to Actions tab in GitHub
Select Auto Version with GitVersion workflow
Click on specific run

Check Job Status¶

Green checkmark: Job succeeded
Red X: Job failed
Yellow circle: Job skipped or in progress

Check Tag Creation¶

# Fetch latest tags
git fetch --tags

# List service tags
git tag -l "api-v*"
git tag -l "s3-notifier-v*"

Check Docker Images¶

# List images in GHCR
gh api /user/packages/container/syrf-api/versions

# Or use Docker
docker pull ghcr.io/camaradesuk/syrf-api:latest

Check Lambda Deployment¶

# Get Lambda function info
aws lambda get-function \
  --function-name syrfAppUploadS3Notifier \
  --region eu-west-1

# Check Lambda tags
aws lambda list-tags \
  --resource arn:aws:lambda:eu-west-1:ACCOUNT:function:syrfAppUploadS3Notifier

Check GitHub Releases¶

Visit: https://github.com/camaradesuk/syrf/releases

Look for releases with tag s3-notifier-v*

Troubleshooting¶

Tag Not Created¶

Check:

Did version job succeed? (Check version-{service} job)
Did build/deploy job succeed? (Check build-and-push-images or deploy-lambda)
Both must be green for tag to be created

Fix: Fix the failing build/deployment job and rerun workflow

Docker Build Fails¶

Common Causes:

Syntax error in Dockerfile
Missing dependencies
Build context issues
Network/GHCR connectivity

Fix: Check build logs in build-and-push-images job

Lambda Deployment Fails¶

Common Causes:

Missing AWS credentials
Incorrect IAM permissions
Lambda function doesn't exist
Network issues

Fix: Check deploy-lambda job logs, verify AWS secrets

Release Not Created¶

Prerequisite: Tag must be created first

Check:

Does s3-notifier-v{version} tag exist?
Did create-lambda-release job run?
Check job logs for errors

Promotion PR Not Created¶

Common Causes:

Missing or invalid GITOPS_PAT secret
build-and-push-images job failed
No Docker services built successfully (e.g., only Lambda changed)
Token doesn't have repo scope
Token doesn't have access to cluster-gitops repository

Fix:

Check promote-to-staging job logs in GitHub Actions
Verify APP_ID and APP_PRIVATE_KEY secrets exist and are accessible
Ensure GitHub App has proper permissions (Contents: Read/Write, PRs: Read/Write)
Verify at least one Docker service (API/PM/Quartz/Web) built successfully
See setup-gitops-github-app.md for GitHub App configuration

Promotion PR Created But Values Not Updated¶

Symptom: PR exists but image tags in values files unchanged

Common Causes:

sed command pattern mismatch
Values file structure changed
Branch conflicts in cluster-gitops

Fix:

Check PR diff in cluster-gitops repository
Verify values file structure matches expected format:

image:
  tag: "1.2.3"  # This line must exist and match pattern

Close conflicting PRs or branches in cluster-gitops

Staging Environment¶

Overview¶

Staging is a pre-production environment that receives automated deployments from the CI/CD pipeline after successful builds on main branch.

Staging vs Production¶

Aspect	Staging	Production
URL Pattern	`*.staging.syrf.org.uk`	`*.syrf.org.uk`
Namespace	`staging`	`production`
Promotion	Automatic (PR auto-merges)	Manual (requires review)
Database	`syrftest` ⚠️	`syrftest`
RabbitMQ Vhost	`staging`	`/` (default)
Quartz Schema	`[staging]`	`[production]`
Replicas	1 per service	2+ per service
Resources	Reduced	Full allocation

Staging shares the production database (syrftest). This is a known issue and represents a significant risk.

Implications:

Changes in staging can affect production data
Staging tests run against live user data
Schema migrations must be backwards-compatible
No isolated testing of data-destructive operations

Planned resolution: See MongoDB Testing Strategy for the plan to create syrf_staging database.

Staging Deployment Flow¶

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Staging Deployment Flow                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  1. Push to main                                                            │
│            ↓                                                                │
│  2. CI/CD builds Docker images                                              │
│            ↓                                                                │
│  3. Images pushed to GHCR                                                   │
│     ghcr.io/camaradesuk/syrf-{service}:{version}                            │
│            ↓                                                                │
│  4. promote-to-staging job creates PR to cluster-gitops                     │
│     Updates: syrf/environments/staging/{service}/config.yaml                │
│            ↓                                                                │
│  5. PR validation passes (YAML lint, etc.)                                  │
│            ↓                                                                │
│  6. PR auto-merges (no human approval required for staging)                 │
│            ↓                                                                │
│  7. ArgoCD ApplicationSet detects config.yaml change                        │
│            ↓                                                                │
│  8. ArgoCD regenerates Application with new targetRevision                  │
│            ↓                                                                │
│  9. ArgoCD syncs: pulls Helm chart, deploys to staging namespace            │
│            ↓                                                                │
│  10. PostSync hook creates GitHub commit status (argocd/deploy-staging)     │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Staging Version Files¶

Staging service versions are stored in the cluster-gitops repository:

cluster-gitops/syrf/environments/staging/
├── namespace.yaml              # Environment metadata
├── staging.values.yaml         # Environment-specific Helm values
├── api/
│   ├── config.yaml            # chartTag: "8.21.0" (updated by CI/CD)
│   └── values.yaml            # Service-specific overrides
├── project-management/
│   ├── config.yaml
│   └── values.yaml
├── quartz/
│   ├── config.yaml
│   └── values.yaml
└── web/
    ├── config.yaml
    └── values.yaml

Verifying Staging Deployment¶

Check deployment status:

# Via ArgoCD CLI (requires cluster access)
argocd app get syrf-staging-api

# Via kubectl
kubectl get deployments -n staging
kubectl get pods -n staging

# Check running versions
kubectl get deployment syrf-api -n staging -o jsonpath='{.spec.template.spec.containers[0].image}'

Check commit status in GitHub:

Navigate to the commit on main branch
Look for argocd/deploy-staging status check

Production Promotion¶

Overview¶

Production deployments require manual approval via PR merge in the cluster-gitops repository.

Promotion Flow¶

┌─────────────────────────────────────────────────────────────────────────────┐
│                        Production Promotion Flow                             │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  1. After successful staging promotion (PR merged)                          │
│            ↓                                                                │
│  2. promote-to-production job automatically triggers                        │
│            ↓                                                                │
│  3. Job copies opted-in service configs from staging to production          │
│     syrf/environments/staging/{service}/config.yaml                         │
│                      ↓ copy values ↓                                        │
│     syrf/environments/production/{service}/config.yaml                      │
│     Services without an existing production config are skipped             │
│            ↓                                                                │
│  4. Updates envName from "staging" to "production"                          │
│            ↓                                                                │
│  5. Creates PR to cluster-gitops                                            │
│     Labels: requires-review, automated                                      │
│            ↓                                                                │
│  6. Workflow completes (green checkmark)                                    │
│     PR is NOT auto-merged                                                   │
│            ↓                                                                │
│  ┌─────────────────────────────────────────────────────────────────────────┐│
│  │ 7. MANUAL GATE: Administrator reviews and merges PR                    ││
│  │    - Review what services are being updated                            ││
│  │    - Verify staging testing is complete                                ││
│  │    - Coordinate deployment timing                                      ││
│  └─────────────────────────────────────────────────────────────────────────┘│
│            ↓                                                                │
│  8. After merge, ArgoCD syncs to production namespace                       │
│            ↓                                                                │
│  9. PostSync hook creates GitHub commit status (argocd/deploy-production)   │
│            ↓                                                                │
│  10. PostSync hook creates GitHub Release (production only)                 │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

A service is eligible for this step only when its reviewed syrf/environments/production/{service}/config.yaml already exists. Adding a staging-only service does not create a production configuration or deployment; production enablement requires a separate reviewed GitOps change first. If no effective production config changes remain—because no services are eligible or every opted-in config already matches staging—the job reports a successful no-op and creates no production PR.

Production Checklist¶

Before merging a production promotion PR:

Staging deployment verified working
Critical user flows tested in staging
No known regressions or blockers
Database migrations are backwards-compatible
On-call team aware of deployment
Deployment timing appropriate (avoid peak hours)

Rollback Procedure¶

Quick rollback (preferred):

In cluster-gitops, find the previous production promotion PR
Copy the old chartTag values from git history
Create a new PR reverting the config.yaml changes
Merge the revert PR
ArgoCD will sync the previous versions

Alternative - via ArgoCD:

# List application history
argocd app history syrf-production-api

# Rollback to previous revision
argocd app rollback syrf-production-api {revision-number}

Note: ArgoCD rollback is temporary - the next sync will revert to git state. Always fix in git for permanent changes.

Environment Comparison¶

Complete Environment Matrix¶

Aspect	Preview	Staging	Production
Trigger	PR + `preview` label	Push to `main`	Manual PR merge
Namespace	`pr-{number}`	`staging`	`production`
URL	`pr-{n}.syrf.org.uk`	`staging.syrf.org.uk`	`syrf.org.uk`
MongoDB	`syrf_pr_{number}`	`syrftest` ⚠️	`syrftest`
Quartz Schema	`[preview_{number}]`	`[staging]`	`[production]`
RabbitMQ	`pr-{number}`	`staging`	`/`
Lambda	`*-pr-{number}`	(shared prod)	`syrfAppUploadS3Notifier`
S3 Prefix	`preview/pr-{n}/`	(shared prod)	`Projects/`
Auto-cleanup	Yes (on PR close)	No	No
GitHub Deployment	Yes	No (commit status)	No (commit status + Release)
Replicas	1	1	2+
Resources	Minimal	Reduced	Full
Data	Seed/Snapshot	Production ⚠️	Production

Data Isolation Status¶

Environment	Database	Isolation	Risk Level
Preview	`syrf_pr_{number}`	✅ Full	Low
Staging	`syrftest`	❌ Shares with production	High
Production	`syrftest`	N/A (is production)	N/A

Planned improvements: Staging will get its own syrf_staging database - see MongoDB Testing Strategy.

GitOps GitHub App Setup: setup-gitops-github-app.md - Required for promotion automation
Sentry Integration: setup-sentry-integration.md - Web service sourcemaps
S3 Notifier Service: src/services/s3-notifier/README.md
CI/CD Decisions: docs/decisions/ADR-001-ci-cd-approach.md
Dependency Map: docs/architecture/dependency-map.yaml
CLAUDE.md: Main developer reference guide

Maintenance¶

Adding a New Service¶

Create service in src/services/{new-service}/
Add GitVersion.yml to service folder
Add Dockerfile (or Lambda configuration)
Update detect-changes job with new path filter
Add version-{new-service} job
Add to either build-and-push-images or create new deployment job
Add to create-and-push-tags logic
Test with a commit to the new service

Modifying Tag Logic¶

Tags are created in create-and-push-tags job.

To change tag format or conditions:

Edit .github/workflows/ci-cd.yml
Find create-and-push-tags job
Modify bash script in "Create and push tags" step
Test thoroughly before merging

Changing Lambda Function Name¶

If Lambda function name changes:

Update deploy-lambda job: dotnet lambda deploy-function {NEW_NAME}
Update documentation
Update AWS IAM policies
Consider migration strategy for existing function

CI/CD Workflow Overview¶

Purpose¶

Workflow File¶

Main release queue¶

Complete Workflow Structure¶

Job Details¶

1. detect-changes¶

2. version-* Jobs (Parallel Execution)¶

2.5. test-* Jobs (Test Phase)¶

test-dotnet¶

test-web¶

Build Gate¶

3. build-and-push-images¶

4. deploy-lambda¶

5. promote-to-staging¶

6. create-and-push-tags¶

6. create-lambda-release¶

GitOps Promotion Workflow¶

Key Design Principles¶

1. No Orphaned Tags¶

2. Partial Success Handling¶

3. Artifact Retention¶

4. Parallel Execution¶

Workflow Execution Examples¶

Example 1: API and PM Changed¶

Example 2: S3 Notifier Changed¶

Example 3: Build Failure¶

Monitoring and Debugging¶

View Workflow Run¶

Check Job Status¶

Check Tag Creation¶

Check Docker Images¶

Check Lambda Deployment¶

Check GitHub Releases¶

Troubleshooting¶

Tag Not Created¶

Docker Build Fails¶

Lambda Deployment Fails¶

Release Not Created¶

Promotion PR Not Created¶

Promotion PR Created But Values Not Updated¶

Staging Environment¶

Overview¶

Staging vs Production¶

⚠️ CRITICAL: Database Sharing¶

Staging Deployment Flow¶

Staging Version Files¶

Verifying Staging Deployment¶

Production Promotion¶

Overview¶

Promotion Flow¶

Production Checklist¶

Rollback Procedure¶

Environment Comparison¶

Complete Environment Matrix¶

Data Isolation Status¶

Related Documentation¶

Maintenance¶

Adding a New Service¶

Modifying Tag Logic¶

Changing Lambda Function Name¶