Skip to content

Query BFF auth metrics in Google Managed Prometheus

The API records BFF authentication telemetry through the OpenTelemetry meter SyRF.API.BffAuth. This guide covers how those metrics reach Google Managed Prometheus (GMP), how to switch the path on per environment, and how to query it from the command line, from scripts/auth-migration/query-auth-telemetry.sh and from the ARRNC Grafana.

How it works

flowchart LR
    API["syrf-api pod<br/>:8080 app, :9464 /metrics"] -->|scrape every 60s| C["GMP collector<br/>(gmp-system DaemonSet)"]
    C --> M["Cloud Monitoring<br/>project camarades-net"]
    M -->|PromQL, Google OAuth| S["query-auth-telemetry.sh"]
    M -->|PromQL, service account| G["ARRNC Grafana<br/>Google Cloud Monitoring data source"]
  • Exporter. With PrometheusMetrics:Enabled set, the API adds the OpenTelemetry Prometheus exporter to its metrics pipeline and listens on a second port (PrometheusMetrics:Port, default 9464). /metrics answers only on connections that arrived at that port. The check uses the connection's local port, not the Host header, so a request through the Ingress cannot reach it. Any other path on the metrics port returns 404. A metrics port equal to an application port is refused twice: the chart fails to render and the host fails at startup. Code: src/services/api/SyRF.API.Endpoint/Telemetry/PrometheusScrapeEndpoint.cs.
  • Chart. prometheusMetrics.enabled: true in the API chart values:
  • sets SYRF__PrometheusMetrics__Enabled and SYRF__PrometheusMetrics__Port, generated from src/charts/syrf-common/env-mapping.yaml;
  • adds a container port named metrics, which is not added to the Service;
  • renders a GMP PodMonitoring named syrf-api-metrics. It selects the Deployment's own app label and scrapes metrics every scrapeInterval, default 60s.
  • Scraping. GMP's existing collector DaemonSet does the scraping, so no new pods run and no GCP write permission is needed. GMP adds project_id, location, cluster, namespace, job and instance labels.
  • Ingestion filter. keepMetricsRegex (default syrf_.*) keeps only SyRF's own instruments: BFF auth and FEAT-024 project statistics. The runtime and ASP.NET Core instruments are still served on /metrics but GMP does not ingest them. GMP bills per ingested sample, so widen the filter deliberately.

Metric names and labels

Default Prometheus translation (UnderscoreEscapingWithSuffixes), pinned by PrometheusScrapeEndpointTests:

Instrument Prometheus series
syrf.bff.auth.operations (counter, {request}) syrf_bff_auth_operations_total
syrf.bff.auth.callback.duration (histogram, ms) syrf_bff_auth_callback_duration_milliseconds_bucket / _sum / _count

Labels: provider (auth0 | openiddict | mock | disabled), deployment_environment (from the instrument tag deployment.environment; development | preview | staging | production | unknown), operation, outcome, status_class, plus otel_scope_name and the GMP target labels above. While BFF auth is off (bffAuth.enabled: false, the current state everywhere) no BFF operations run, so no BFF series exist. Scraping still works and up is 1.

Enable it for an environment

The switch is environment configuration in cluster-gitops, not a feature flag. Enabling it changes no request handling:

# cluster-gitops: syrf/environments/<env>/api/values.yaml
prometheusMetrics:
  enabled: true

The environment's AppProject (argocd/projects/syrf-<env>.yaml) must allow monitoring.googleapis.com/PodMonitoring in namespaceResourceWhitelist, otherwise the Argo sync fails. Previews are not enabled and their AppProject does not allow the kind.

Verify read-only after the sync:

kubectl -n syrf-staging get podmonitoring syrf-api-metrics -o jsonpath='{.status.conditions}'
kubectl -n syrf-staging get pod -l app=api-staging-syrf-api \
  -o jsonpath='{.items[0].spec.containers[0].ports}'

To turn it off, set enabled: false (or remove the block). Argo prunes the PodMonitoring and the pod stops listening on the metrics port.

Query from the command line

The GMP query API is the Prometheus HTTP API at https://monitoring.googleapis.com/v1/projects/camarades-net/location/global/prometheus and needs a Google OAuth token for an identity with roles/monitoring.viewer:

curl -fsS -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  'https://monitoring.googleapis.com/v1/projects/camarades-net/location/global/prometheus/api/v1/query' \
  --data-urlencode 'query=up{namespace="syrf-staging",job="syrf-api-metrics"}'

Rollout telemetry windows

query-auth-telemetry.sh appends /api/v1/query to --metrics-endpoint. It takes the token from a file descriptor, so the token never appears in argv, the environment or the evidence file:

exec 7< <(gcloud auth print-access-token)
scripts/auth-migration/query-auth-telemetry.sh \
  --mode bff-window --provider auth0 --environment staging \
  --start 2026-10-01T00:00:00Z --end 2026-10-02T00:00:00Z \
  --metrics-endpoint https://monitoring.googleapis.com/v1/projects/camarades-net/location/global/prometheus \
  --metrics-token-fd 7 \
  --baseline /abs/path/baseline.json \
  --output /abs/path/window.json
exec 7<&-

METRICS_TOKEN_FD is the environment-variable equivalent. Without a token descriptor the script sends no Authorization header. That is what an authenticating proxy on http://127.0.0.1:<port> expects.

Query from the ARRNC Grafana

The ARRNC Grafana runs outside GCP, so it uses Grafana's built-in Google Cloud Monitoring data source (stackdriver). It supports a PromQL query type against GMP and authenticates with a key for the read-only service account arrnc-grafana-reader (roles/monitoring.viewer). The account is created by camaradesuk/camarades-infrastructure Terraform. We chose this over Google's datasource-syncer, which needs a CronJob and a Grafana admin token to refresh a Prometheus data source every few minutes.

Data source settings (Grafana API POST /api/datasources, or provisioning):

name: GMP camarades-net
uid: gmp-camarades-net
type: stackdriver
access: proxy
jsonData:
  authenticationType: jwt
  defaultProject: camarades-net
  clientEmail: arrnc-grafana-reader@camarades-net.iam.gserviceaccount.com
  tokenUri: https://oauth2.googleapis.com/token
  universeDomain: googleapis.com
secureJsonData:
  privateKey: <private_key from the service-account key JSON>

The key is created once by an operator (gcloud iam service-accounts keys create) and lives only in Grafana's encrypted secureJsonData, never in git or Terraform state. To rotate it, create a new key, update the data source, then delete the old key.

Example panel queries (query type PromQL, project camarades-net):

sum by (operation, outcome) (rate(syrf_bff_auth_operations_total{deployment_environment="staging"}[5m]))
histogram_quantile(0.95, sum by (le) (rate(syrf_bff_auth_callback_duration_milliseconds_bucket{deployment_environment="staging"}[15m])))
up{job="syrf-api-metrics"}