Rotating OIDC Client Secrets Without Downtime

An OIDC client secret is a single shared string between your identity provider and your portal — rotate it the naive way and every in-flight login fails the instant the old value stops being accepted. This guide implements the dual-secret overlap pattern so a rotation produces zero failed authentications.

The technique matters because the OIDC client secret authenticates your portal to the IdP on every token exchange. It is the highest-blast-radius credential covered by Secrets Management & Rotation, and rotating it carelessly takes down the login path for the whole organization. This is the operational companion to your OIDC & SSO Configuration, which defines how that secret is consumed at runtime.

Prerequisites

The zero-downtime pattern only works if four things are true: the IdP accepts two active secrets at once, the secret lives in versioned Vault storage, the portal reads it from a rendered template rather than a literal, and you hold an admin token to manage credentials.

Prerequisites for overlap rotation Multi-secret IdP support, versioned Vault storage, config from a rendered template, and an admin token enable overlap. IdP: two secrets Vault KV v2 config from template admin token Overlap possible zero downtime
If the IdP caps you at one active secret, overlap is off the table — fall back to blue/green.
  1. IdP support for multiple active secrets. Okta, Auth0, Microsoft Entra ID, and Keycloak >= 24 all allow two valid client secrets on one application registration simultaneously. Confirm your tenant permits this before relying on overlap.
  2. Vault KV v2 at portal-kv from the secrets engine baseline, holding the current secret at portal-kv/oidc/portal.
  3. Portal config sourced from Vault, not from a static clientSecret literal. The portal must read ${OIDC_CLIENT_SECRET} from a rendered Vault Agent template.
  4. IdP admin API token with scope to manage application credentials, itself stored in Vault at portal-kv/oidc/idp-admin — never inline.
  5. Backstage >= 1.20.0 (or equivalent middleware) configured with tokenEndpointAuthMethod: client_secret_post.

Exact Configuration

The pattern: create a second secret, deploy it, then retire the first — never overwriting the value the live fleet is still using. During the overlap window both secrets are valid, so no in-flight login can hit an invalid credential.

Dual-secret overlap rotation timeline A second secret is registered and deployed while the first stays valid, then the first is retired after the fleet migrates. 1. Register 2nd 2. Write KV 3. Roll fleet 4. Retire old overlap window — both secrets valid (≥ 15 min) old secret only deleted after every pod is on the new one
The old secret is retired only at the end; deleting it early is the one mistake that causes downtime.

1. Register a second client secret at the IdP

# Requires Okta API; token read from Vault, never inline
ADMIN_TOKEN=$(vault kv get -field=token portal-kv/oidc/idp-admin)
NEW_SECRET=$(curl -s -X POST \
  "https://${IDP_TENANT_DOMAIN}/oauth2/v1/clients/${OIDC_CLIENT_ID}/credentials/secrets" \
  -H "Authorization: SSWS ${ADMIN_TOKEN}" \
  -H "Accept: application/json" | jq -r '.client_secret')

Both the old and the new secret are now valid at the IdP. No portal change has happened yet, so logins are unaffected.

2. Write the new secret as a new KV version

KV v2 retains the previous version, giving you an instant rollback target.

# Requires Vault >= 1.16.0
vault kv put portal-kv/oidc/portal client_secret="${NEW_SECRET}"
vault kv get -format=json portal-kv/oidc/portal | jq '.data.metadata.version'

3. Roll the fleet onto the new secret

Trigger a rolling restart so each pod’s Vault Agent re-renders the template with the new version. The overlap window means pods on the old secret and pods on the new secret both authenticate successfully during the rollout.

# Requires Kubernetes >= 1.27
kubectl rollout restart deployment/portal-backend -n platform
kubectl rollout status deployment/portal-backend -n platform --timeout=180s

# vault-agent template (excerpt) — Requires vault-k8s >= 1.4.0
vault.hashicorp.com/agent-inject-secret-oidc: "portal-kv/data/oidc/portal"
vault.hashicorp.com/agent-inject-template-oidc: |
  {{- with secret "portal-kv/data/oidc/portal" -}}
  OIDC_CLIENT_SECRET={{ .Data.data.client_secret }}
  {{- end }}

4. Retire the old secret after the overlap window

Only once every pod reports ready on the new secret — verify before deleting — remove the old credential at the IdP. Keep the overlap at least as long as your longest token exchange plus rollout time; 15 minutes is a safe default.

# Deactivate then delete the superseded secret by its IdP secret id
curl -s -X DELETE \
  "https://${IDP_TENANT_DOMAIN}/oauth2/v1/clients/${OIDC_CLIENT_ID}/credentials/secrets/${OLD_SECRET_ID}" \
  -H "Authorization: SSWS ${ADMIN_TOKEN}"

Validation

The rotation is provable at each stage: two secrets active during overlap, every pod rendering the new version, a live token exchange returning 200, and exactly one secret remaining after retirement.

Four rotation validation checkpoints Two secrets during overlap, all pods on the new version, a 200 token exchange, and one secret after retirement. 2 secrets active during overlap All pods new rendered version Token 200 live exchange 1 secret left after retire
Confirm every pod is on the new version before running step 4 — that check is what makes the retirement safe.
# 1. Confirm two secrets were active during overlap (run before step 4)
curl -s "https://${IDP_TENANT_DOMAIN}/oauth2/v1/clients/${OIDC_CLIENT_ID}/credentials/secrets" \
  -H "Authorization: SSWS ${ADMIN_TOKEN}" | jq 'length'
# Expect: 2

# 2. Confirm every pod renders the new KV version
kubectl get pods -n platform -l app=portal-backend -o name | while read p; do
  kubectl exec "$p" -n platform -c portal -- cat /vault/secrets/oidc | grep -c OIDC_CLIENT_SECRET
done
# Expect: 1 from every pod

# 3. End-to-end token exchange with the live secret
curl -s -o /dev/null -w "%{http_code}\n" -X POST "https://${IDP_TENANT_DOMAIN}/oauth2/v1/token" \
  -d "grant_type=client_credentials&client_id=${OIDC_CLIENT_ID}&client_secret=${NEW_SECRET}&scope=openid"
# Expect: 200

# 4. After step 4, confirm only one secret remains
curl -s "https://${IDP_TENANT_DOMAIN}/oauth2/v1/clients/${OIDC_CLIENT_ID}/credentials/secrets" \
  -H "Authorization: SSWS ${ADMIN_TOKEN}" | jq 'length'
# Expect: 1

Edge Cases & Troubleshooting

The failures cluster around three timing mistakes: retiring the old secret too early, a pod that never re-rendered, and config that mismatches the IdP. The diagram maps each symptom to its timing fix.

Rotation failure symptoms and fixes invalid_client spikes, stale pods, and config mismatch each map to a re-add, re-render, or config fix. invalid_client spike pods on old value curl works, portal fails re-add old, finish rollout force re-render restart match client_secret_post
An invalid_client spike almost always means step 4 ran before every pod finished migrating — re-add and wait.
Symptom Root Cause Resolution
invalid_client spikes mid-rollout Old secret deleted before all pods migrated Re-add the old secret at the IdP, finish the rollout, then retire it
Some pods still authenticate with the old value Vault Agent cached the prior KV version Confirm vault.hashicorp.com/agent-inject re-render fired; force kubectl rollout restart again
IdP rejects creating a second secret Tenant caps clients at one active secret Use a short maintenance window with a blue/green deployment instead of overlap
New secret works in curl but not in portal Wrong tokenEndpointAuthMethod Ensure IdP registration and portal config both use client_secret_post
Rotation succeeds but next login loops Stale session cookies signed under old config Bump the session cookie key alongside the rotation, or set a short maxAge

Frequently Asked Questions

Why not just overwrite the secret and restart everything at once?

A hard cutover means there is a window where the IdP has the new secret but some pods still present the old one, producing invalid_client for those requests. The overlap pattern keeps both secrets valid until the fleet has fully migrated, so no request ever hits an invalid credential.

How long should the overlap window be?

At least the time for a full rolling restart plus your longest token lifetime, so any in-flight refresh completes against a still-valid secret. Fifteen minutes covers most deployments; extend it if your rollout or token TTLs are longer.

What if my IdP only allows one client secret?

Fall back to a blue/green deployment: stand up a parallel registration with the new secret, shift traffic at the load balancer, and decommission the old registration once drained. This trades a second app registration for the overlap capability.