Aller au contenu principal

Secret and Key Rotation

This page is a verifiable operations procedure for rotating credentials that STOA depends on at a site. It expands the checklist in Security Hardening into controls you can run before, during, and after a rotation, plus a deliberate rollback path. Day-2 install and upgrade remain in Installation and Upgrade. Offline install is covered separately (Offline Installation).

Scope of this guide
  • In scope: classifying what to rotate, recording a change window, applying dual-credential / grace where the product supports it, verifying with site-local checks, and rolling back when a control fails.
  • Out of scope: inventing a new secret manager product; consumer-facing API key UX details (see API Key Rotation); application data-at-rest encryption key re-wrap without downtime ([confirm when CAB-2268 ships] — do not claim zero-downtime app-key rotation until that feature is released and documented).
  • Secrets in this document: placeholders only (registry.example.com, stoa-system, ${TOKEN}). Never paste production values into tickets, PRs, or chat.

The hosted public platform was shut down on 2026-05-23; operators run these steps on their own infrastructure (ADR-037).

1. What you rotate​

FamilyTypical ownersDual / gracePrimary methodDeep guide
Consumer / subscription API keysTenant adminYes (grace period hours)Control Plane API / PortalAPI Key Rotation
Consumer mTLS certificatesTenant adminYes (grace_period_hours)POST .../certs/rotateConsumer Onboarding
Webhook delivery secretsTenant / integratorNo automatic dual window — coordinate cutoverPATCH webhook configWebhooks
Database passwordPlatform operatorNo — plan short reconnectSecret store → K8s Secret → restartSecurity Hardening
Keycloak client secretsPlatform operatorNo — update IdP then clientsKeycloak Admin + configKeycloak
Gateway admin / service tokensPlatform operatorPrefer dual token then drop oldK8s Secret + restartThis page §4
Registry pull credentialsPlatform operatorPrefer dual then drop oldImagePullSecretsThis page §4
TLS certificates (edge / ingress)Platform operatorcert-manager renewalcert-manager / site PKISecurity Hardening
Application encryption key (data at rest)Platform operator[confirm][confirm procedure after CAB-2268]Track CAB-2268

Recommended cadence (align with your policy; defaults from hardening): database, Keycloak clients, and gateway tokens about every 90 days; TLS via automated renewal; API keys on demand or every 90 days for production consumers.

2. Pre-rotation controls (do these first)​

Record the answers in the change ticket before you touch production.

  1. Inventory — Name the secret family, where the current value lives (secrets backend / K8s Secret / IdP), and every consumer of that value (Deployments, CronJobs, external callers).
  2. Blast radius — List namespaces and services that will restart or reconnect. Prefer one family per change window.
  3. Dual-credential plan — If the product supports a grace period (API keys, consumer certs), set grace_period_hours explicitly. If it does not, define an ordered cutover (new secret live → update all consumers → revoke old) and a maximum time-to-revoke.
  4. Backup / restore point — Confirm a recent successful backup (Backup and Recovery) and note the Helm revision (helm history stoa-platform -n stoa-system) in case config rollback is needed.
  5. Communication — Notify tenants that own the credential when the rotation is consumer-visible (API keys, mTLS). Platform-only secrets stay on the ops channel.
  6. Offline / air-gapped sites — Ensure the new secret material is already on the site media path; do not open outbound Internet to fetch credentials (Offline Installation).

Stop if any control is unanswered. Do not rotate “to see what happens”.

3. Rotation patterns​

3.1 Consumer API keys (product grace)​

Follow API Key Rotation end-to-end:

  1. POST /v1/subscriptions/${SUB_ID}/rotate-key with an explicit grace_period_hours.
  2. Store the new_api_key once (shown only once).
  3. Deploy the new key to callers while both keys work.
  4. Confirm traffic moved (subscription rotation_count / last_rotated_at, gateway metrics).
  5. After grace expiry, old key must fail; clear gateway cache only if immediate invalidation is required (see that guide).

Control after step 4: a probe with the new key returns HTTP 2xx on a known route; a probe with the old key still succeeds only inside the grace window.

3.2 Consumer mTLS certificates​

curl -X POST "${STOA_API_URL}/v1/consumers/${TENANT_ID}/${CONSUMER_ID}/certs/rotate" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "new_certificate_pem": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
    "grace_period_hours": 48
  }'

During grace, both certificates are accepted. Distribute the matching private key through the tenant's own secret path — never through this documentation site.

Control: handshake with the new cert succeeds; handshake with the old cert succeeds only until grace ends.

3.3 Platform secrets via the secrets backend → Kubernetes​

Pattern from Security Hardening:

Secrets backend (OpenBao or HashiCorp Vault) → Kubernetes Secret → Pod env / volume → restart

The secrets backend is reached through the Vault HTTP API: OpenBao by default, or a HashiCorp Vault you already run (ADR-075).

Operator sequence:

  1. Write the new value in the secrets backend (or your site secret store). Keep the old value readable until post-controls pass.
  2. Sync or apply the Kubernetes Secret in stoa-system (name is site-local — do not publish real names).
  3. Restart only the Deployments that consume that Secret (kubectl rollout restart deployment/<name> -n stoa-system).
  4. Watch rollout: kubectl rollout status deployment/<name> -n stoa-system.
  5. Run §5 verification controls.
  6. Only then revoke or delete the old value in the secrets backend.

Never commit the new secret to Git, Helm values.yaml, or ConfigMaps.

3.4 Dual token then drop (gateway admin / pull secrets)​

When the runtime accepts only one active token at a time:

  1. Create token B alongside token A in the secret store.
  2. Update the Secret so the workload prefers B but can fall back to A if your chart supports dual keys; otherwise update all replicas in one controlled rollout.
  3. Verify with §5.
  4. Remove A from the store and from any external registry ACL.

If the chart has no dual-key field, treat this as a short maintenance window and keep Helm rollback ready (Upgrade).

3.5 Application encryption key​

Zero-downtime re-wrap of the application encryption key is tracked as product work (internal ticket CAB-2268). Until that lands and is documented here:

  • Do not rotate the data-at-rest key with an ad-hoc kubectl edit.
  • Prefer restoring from backup over a half-applied re-encrypt.
  • Mark any site runbook that claims zero-downtime app-key rotation as stale until CAB-2268 is Done and this section is updated.

4. Change window checklist (operator)​

Copy into the ticket; tick in order.

  • Pre-controls (§2) recorded
  • Family and dual/grace plan chosen (§1 / §3)
  • Backup age and Helm revision noted
  • New material generated and stored (never in the ticket body)
  • Rotation command or Secret apply executed
  • Rollout / grace window started
  • Verification controls (§5) green
  • Old credential revoked or expired
  • Ticket closed with time stamps (start, verify, revoke)

5. Verification controls​

Run these on the site network (no Internet required).

ControlHowPass
Workload readykubectl get pods -n stoa-systemDesired replicas Ready
Control Plane healthSite health endpoint or kubectl probe used in your install guideHTTP 2xx / Ready
Auth with new credentialCall a known API with the new API key, cert, or tokenSuccess
Auth with old credentialSame call with the old credentialFails after grace / revoke; still OK only inside documented grace
No secret leakagekubectl get secret <name> -n stoa-system -o yaml reviewed only on a secure console; ticket contains no secret valuesTicket clean
LogsPlatform logs show expected reconnect / rotation events, not crash loopsStable

Fail any control → go to §6 before revoking the old credential.

6. Rollback​

Rollback means restore service with the previous credential, not “force the new one”.

  1. Stop revoking — If the old credential is still valid (grace or dual), leave it valid.
  2. Restore Secret — Re-apply the previous Kubernetes Secret from the secret store history (secrets backend version history / sealed backup). Do not invent a third value mid-incident.
  3. Restart — kubectl rollout restart on the affected Deployments; wait for Ready.
  4. Helm — If the rotation shipped as a values change, helm rollback stoa-platform <REVISION> -n stoa-system (Upgrade).
  5. API key / cert grace — If callers were updated to a bad new key, point them back to the old key while grace remains; open a new rotation later with a fresh new key.
  6. Data-at-rest — If an unsupported app-key change was attempted, restore from backup (Backup and Recovery) rather than improvising decrypt tooling.
  7. Record — Note which control failed, keep the new material quarantined, schedule a follow-up window.