Rotating OIDC Client Secrets Without Downtime

An OIDC client secret is a single shared string between your identity provider and your portal — rotate it the naive way and every in-flight login fails the instant the old value stops being accepted. This guide implements the dual-secret overlap pattern so a rotation produces zero failed authentications.

The technique matters because the OIDC client secret authenticates your portal to the IdP on every token exchange. It is the highest-blast-radius credential covered by Secrets Management & Rotation, and rotating it carelessly takes down the login path for the whole organization. This is the operational companion to your OIDC & SSO Configuration, which defines how that secret is consumed at runtime.

Prerequisites

IdP support for multiple active secrets. Okta, Auth0, Microsoft Entra ID, and Keycloak >= 24 all allow two valid client secrets on one application registration simultaneously. Confirm your tenant permits this before relying on overlap.
Vault KV v2 at portal-kv from the secrets engine baseline, holding the current secret at portal-kv/oidc/portal.
Portal config sourced from Vault, not from a static clientSecret literal. The portal must read ${OIDC_CLIENT_SECRET} from a rendered Vault Agent template.
IdP admin API token with scope to manage application credentials, itself stored in Vault at portal-kv/oidc/idp-admin — never inline.
Backstage >= 1.20.0 (or equivalent middleware) configured with tokenEndpointAuthMethod: client_secret_post.

Exact Configuration

The pattern: create a second secret, deploy it, then retire the first — never overwriting the value the live fleet is still using.

1. Register a second client secret at the IdP

# Requires Okta API; token read from Vault, never inline
ADMIN_TOKEN=$(vault kv get -field=token portal-kv/oidc/idp-admin)
NEW_SECRET=$(curl -s -X POST \
  "https://${IDP_TENANT_DOMAIN}/oauth2/v1/clients/${OIDC_CLIENT_ID}/credentials/secrets" \
  -H "Authorization: SSWS ${ADMIN_TOKEN}" \
  -H "Accept: application/json" | jq -r '.client_secret')

Both the old and the new secret are now valid at the IdP. No portal change has happened yet, so logins are unaffected.

2. Write the new secret as a new KV version

KV v2 retains the previous version, giving you an instant rollback target.

# Requires Vault >= 1.16.0
vault kv put portal-kv/oidc/portal client_secret="${NEW_SECRET}"
vault kv get -format=json portal-kv/oidc/portal | jq '.data.metadata.version'

3. Roll the fleet onto the new secret

Trigger a rolling restart so each pod’s Vault Agent re-renders the template with the new version. The overlap window means pods on the old secret and pods on the new secret both authenticate successfully during the rollout.

# Requires Kubernetes >= 1.27
kubectl rollout restart deployment/portal-backend -n platform
kubectl rollout status deployment/portal-backend -n platform --timeout=180s

# vault-agent template (excerpt) — Requires vault-k8s >= 1.4.0
vault.hashicorp.com/agent-inject-secret-oidc: "portal-kv/data/oidc/portal"
vault.hashicorp.com/agent-inject-template-oidc: |
  {{- with secret "portal-kv/data/oidc/portal" -}}
  OIDC_CLIENT_SECRET={{ .Data.data.client_secret }}
  {{- end }}

4. Retire the old secret after the overlap window

Only once every pod reports ready on the new secret — verify before deleting — remove the old credential at the IdP. Keep the overlap at least as long as your longest token exchange plus rollout time; 15 minutes is a safe default.

# Deactivate then delete the superseded secret by its IdP secret id
curl -s -X DELETE \
  "https://${IDP_TENANT_DOMAIN}/oauth2/v1/clients/${OIDC_CLIENT_ID}/credentials/secrets/${OLD_SECRET_ID}" \
  -H "Authorization: SSWS ${ADMIN_TOKEN}"

Validation

# 1. Confirm two secrets were active during overlap (run before step 4)
curl -s "https://${IDP_TENANT_DOMAIN}/oauth2/v1/clients/${OIDC_CLIENT_ID}/credentials/secrets" \
  -H "Authorization: SSWS ${ADMIN_TOKEN}" | jq 'length'
# Expect: 2

# 2. Confirm every pod renders the new KV version
kubectl get pods -n platform -l app=portal-backend -o name | while read p; do
  kubectl exec "$p" -n platform -c portal -- cat /vault/secrets/oidc | grep -c OIDC_CLIENT_SECRET
done
# Expect: 1 from every pod

# 3. End-to-end token exchange with the live secret
curl -s -o /dev/null -w "%{http_code}\n" -X POST "https://${IDP_TENANT_DOMAIN}/oauth2/v1/token" \
  -d "grant_type=client_credentials&client_id=${OIDC_CLIENT_ID}&client_secret=${NEW_SECRET}&scope=openid"
# Expect: 200

# 4. After step 4, confirm only one secret remains
curl -s "https://${IDP_TENANT_DOMAIN}/oauth2/v1/clients/${OIDC_CLIENT_ID}/credentials/secrets" \
  -H "Authorization: SSWS ${ADMIN_TOKEN}" | jq 'length'
# Expect: 1

Edge Cases & Troubleshooting

Symptom	Root Cause	Resolution
`invalid_client` spikes mid-rollout	Old secret deleted before all pods migrated	Re-add the old secret at the IdP, finish the rollout, then retire it
Some pods still authenticate with the old value	Vault Agent cached the prior KV version	Confirm `vault.hashicorp.com/agent-inject` re-render fired; force `kubectl rollout restart` again
IdP rejects creating a second secret	Tenant caps clients at one active secret	Use a short maintenance window with a blue/green deployment instead of overlap
New secret works in curl but not in portal	Wrong `tokenEndpointAuthMethod`	Ensure IdP registration and portal config both use `client_secret_post`
Rotation succeeds but next login loops	Stale session cookies signed under old config	Bump the session cookie key alongside the rotation, or set a short `maxAge`

Frequently Asked Questions

Why not just overwrite the secret and restart everything at once?

A hard cutover means there is a window where the IdP has the new secret but some pods still present the old one, producing invalid_client for those requests. The overlap pattern keeps both secrets valid until the fleet has fully migrated, so no request ever hits an invalid credential.

How long should the overlap window be?

At least the time for a full rolling restart plus your longest token lifetime, so any in-flight refresh completes against a still-valid secret. Fifteen minutes covers most deployments; extend it if your rollout or token TTLs are longer.

What if my IdP only allows one client secret?

Fall back to a blue/green deployment: stand up a parallel registration with the new secret, shift traffic at the load balancer, and decommission the old registration once drained. This trades a second app registration for the overlap capability.

Up: Secrets Management & Rotation
Section: Authentication, RBAC & Security Governance
OIDC & SSO Configuration