Secrets Management & Rotation

Internal developer portals accumulate long-lived secrets — OIDC client secrets, catalog API tokens, database credentials, signing keys — and every one of them is a standing liability until it is centralized, short-lived, and rotated on a schedule.

This capability sits at the core of Authentication, RBAC & Security Governance. The goal is to remove secrets from config files and CI variables entirely, broker them through a vault, issue them with the shortest viable lifetime, and rotate them without a human ever pasting a credential into a terminal. The reference stack below uses HashiCorp Vault 1.16 as the broker, but the patterns map cleanly onto AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault.

Secret brokering and rotation flow A portal authenticates to a vault with a workload identity, receives a short-lived lease, and the vault rotates the upstream credential on expiry. Portal workload (K8s service account) Vault broker leases + policies Database dynamic creds IdP / OIDC client secret API tokens KV v2 store authn rotate on lease expiry
The workload never holds a static secret; it exchanges its identity for a short-lived lease the vault rotates upstream.

Prerequisites & Environment Baseline

Before brokering a single credential, confirm the runtime and trust anchors are in place:

  1. Vault server: HashiCorp Vault >= 1.16.0, unsealed, with audit devices enabled and a TLS listener. The portal must reach it over HTTPS on 8200.
  2. Workload identity: A Kubernetes service account per portal component, used for the Vault Kubernetes auth method, so no static Vault token is ever stored. Confirm kubectl auth can-i create tokenreviews succeeds for the Vault server’s service account.
  3. Vault Agent or CSI driver: vault-k8s >= 1.4.0 (agent injector) or the Secrets Store CSI driver >= 1.4.0 to render leased secrets into the pod without application code changes.
  4. Engine access: Permission to enable the database, kv-v2, and transit secrets engines on the Vault namespace dedicated to the portal.
  5. Aligned identity: A completed OIDC & SSO Configuration so human operators authenticate to Vault through the same IdP rather than holding root tokens.
# Requires Vault CLI >= 1.16.0
export VAULT_ADDR="https://${VAULT_HOST}:8200"
vault status -format=json | jq '{sealed, version, cluster_name}'
# sealed must be false before proceeding

Step-by-Step Configuration & Plugin Architecture

1. Enable and scope the secrets engines

Provision one mount per concern so policies stay narrow. Map each role to a portal component, never to a person.

# Requires Vault >= 1.16.0
vault secrets enable -path=portal-db database
vault secrets enable -path=portal-kv -version=2 kv

# Dynamic Postgres credentials, 1h lease, 24h max
vault write portal-db/config/catalog \
  plugin_name=postgresql-database-plugin \
  allowed_roles="catalog-ro" \
  connection_url="postgresql://{{username}}:{{password}}@${DB_HOST}:5432/catalog?sslmode=require" \
  username="${VAULT_DB_ROOT_USER}" \
  password="${VAULT_DB_ROOT_PASSWORD}"

vault write portal-db/roles/catalog-ro \
  db_name=catalog \
  default_ttl="1h" max_ttl="24h" \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"

2. Bind the workload identity with a least-privilege policy

The policy grants read on exactly the paths this component needs and nothing else. Scope this to the same ownership boundaries you define in Role-Based Access Control Setup, so the credential a service can mint never exceeds the access its owners hold in the portal.

# portal-catalog.hcl — Requires Vault >= 1.16.0
path "portal-db/creds/catalog-ro" {
  capabilities = ["read"]
}
path "portal-kv/data/catalog/*" {
  capabilities = ["read"]
}

vault policy write portal-catalog portal-catalog.hcl
vault write auth/kubernetes/role/portal-catalog \
  bound_service_account_names=portal-catalog \
  bound_service_account_namespaces=platform \
  policies=portal-catalog ttl=1h

3. Inject leased secrets without code changes

The Vault Agent injector renders credentials into the pod and re-renders them before each lease expires, so the application reads from a file and stays oblivious to rotation.

# deployment.yaml — Requires vault-k8s >= 1.4.0
metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "portal-catalog"
    vault.hashicorp.com/agent-inject-secret-db: "portal-db/creds/catalog-ro"
    vault.hashicorp.com/agent-inject-template-db: |
      {{- with secret "portal-db/creds/catalog-ro" -}}
      POSTGRES_USER={{ .Data.username }}
      POSTGRES_PASSWORD={{ .Data.password }}
      {{- end }}

4. Store static-but-rotatable secrets in KV v2

Some secrets — third-party API tokens, the OIDC client secret — cannot be made dynamic. Keep them in KV v2 (which versions every write) and rotate them on a cron, an operation covered end to end in Rotating OIDC Client Secrets Without Downtime.

# Requires Vault >= 1.16.0
vault kv put portal-kv/catalog/github token="${GITHUB_APP_TOKEN}"
vault kv metadata get -format=json portal-kv/catalog/github | jq '.data.current_version'

Validation & Health Checks

# 1. Confirm a dynamic credential is issued and leased
vault read -format=json portal-db/creds/catalog-ro | jq '{lease_id, lease_duration, username: .data.username}'
# Expect: lease_duration 3600, a generated username like v-kubernet-catalog-ro-...

# 2. List active leases to confirm rotation churn is healthy
vault list sys/leases/lookup/portal-db/creds/catalog-ro | wc -l

# 3. Verify the injected secret rendered inside the pod
kubectl exec deploy/portal-catalog -n platform -c portal -- cat /vault/secrets/db | grep -c POSTGRES_USER
# Expect: 1

Maintenance & Lifecycle Management

  • Rotate the root credential Vault uses to mint dynamic creds with vault write -force portal-db/rotate-root/catalog. After this, even an operator cannot reproduce the root password.
  • Force-revoke a compromised lease tree with vault lease revoke -prefix portal-db/creds/catalog-ro, which drops every issued database role immediately.
  • Track rotation events by forwarding Vault’s audit device to your SIEM; pair this with Audit Logging & Compliance so every issue, renew, and revoke is attributable.
  • Upgrade path: Vault minor upgrades are rolling on the standby nodes first; run vault operator raft list-peers to confirm quorum before stepping the active node.
  • Metrics: alert on vault.expire.num_leases growth (lease leak) and vault.token.create_with_namespace spikes (credential abuse).

Common Pitfalls & Mitigation Strategies

  • Secrets in CI variables or app-config.yaml. Root cause: convenience during bootstrap. Fix: replace with ${ENV_VAR} placeholders fed by Vault Agent and add a gitleaks pre-commit gate.
  • Lease TTL longer than the pod lifetime. Root cause: copying defaults. Fix: set default_ttl shorter than the deployment’s rolling-restart cadence so credentials churn naturally.
  • One Vault policy shared across components. Root cause: skipping per-workload roles. Fix: one service account, one role, one policy — mirror the ownership model from your permission design.
  • No max TTL on dynamic roles. Root cause: omitted max_ttl. Fix: always cap max_ttl so a renewed lease cannot live indefinitely.
  • Root-token break-glass left active. Root cause: forgotten bootstrap token. Fix: revoke the initial root token after configuring auth methods; regenerate via unseal keys only during incidents.

Frequently Asked Questions

Should the OIDC client secret be a dynamic secret or a KV entry?

OIDC client secrets are minted by the identity provider, not by Vault, so they cannot be dynamic. Store them in KV v2 and rotate them with a scheduled job that registers a new secret at the IdP, writes the new version to Vault, and retires the old one after an overlap window.

How short should database credential TTLs be?

Make the default TTL shorter than your normal pod lifetime — one hour is a common starting point — so credentials are recycled by ordinary restarts. Set max_ttl to bound the worst case, typically 24 hours, after which renewal is refused and a fresh credential must be issued.

Can we adopt this without rewriting the portal application?

Yes. The Vault Agent injector or the Secrets Store CSI driver render leased secrets into files or environment variables, so the application reads them exactly as it read static config. The only code change is pointing config at the rendered path.