"Rotate Everything" Is Not a Plan: The Secret-Rotation Drill You Never Ran

At 13:58 UTC on October 30, a hosting platform put a line on its status page that most engineers hope they never have to write: "Assume all environment variables have been decrypted. Disable/rotate all keys."

The company was Northflank. Users had started reporting that their AI API keys were being used by someone else. Over the following hours the guidance hardened: rotate secrets on workloads, in secret groups, in global secrets; disable public access to add-ons; check your AI provider billing for activity you didn't cause.

That instruction is correct. It is also, for most teams, unexecutable in a useful timeframe — and that is the part worth writing about. The breach is the event. The rotation is the response. Almost nobody has measured the second one.

What the timeline actually shows

The public updates are unusually blunt, and the sequence matters:

• 11:43 UTC — a threat-intelligence alert from Wiz lands.
• 11:44 UTC — engineers begin reviewing it.
• 12:02 UTC — first network policy patch deployed.
• 12:50 and 13:05 UTC — additional patching.
• 13:58 UTC — customers told to assume every environment variable is decrypted.
• 15:12 UTC — self-service secret rotation enabled for Postgres and MongoDB.
• 23:38 UTC — users confirm unauthorised use of their AI API keys.
• 01:04 UTC the next day — audit logs enabled for all customers, including free users.

Read the last line twice. Audit logging was switched on roughly thirteen hours after the alert, and after the instruction to rotate. Which means that for most of the incident, the honest answer to "which secrets were actually read?" was we can't tell you. And when you can't scope a breach, the only defensible scope is everything.

That is why "assume all" is not overcautious language. It's the statement you make when your logs can't narrow the set. The blanket instruction is a symptom of a visibility gap that existed long before the attack.

Rotation is a deployment, not an edit

Here's the mental model that breaks teams. People hear "rotate the keys" and picture a settings page: paste a new value, hit save, done. But a secret is not a value sitting in one place. It's a value that has been copied into everywhere that needs it.

For a single database password in a mid-sized production system, the copies typically include: the database itself, the app's environment variables, a Kubernetes secret or secret group, the CI pipeline that runs migrations, a backup job, an analytics worker someone deployed last spring, a developer's local .env file, and possibly a connection string baked into a container image built six months ago. Changing the password is one action. Making all of those agree is a deployment with an ordering problem and a rollback plan.

Concretely, there are three questions you need to answer before you touch anything:

1. What is the dependency closure? Every consumer of this secret, including the ones that aren't in your orchestrator.
2. Can both values be valid at once? If the issuer supports two active credentials, you can rotate with zero downtime. If it doesn't, you're choosing a maintenance window.
3. What is the longest lag between rotation and a consumer picking it up? A pod that rolls in 30 seconds is easy. An hourly cron, a batch job, or a vendor's cached session is not.

The ordering problem nobody writes down

Rotate in the wrong order and you don't get a security incident, you get an outage on top of one. Two rules cover most cases:

For credentials you issue (database passwords, internal API tokens), the order is: create the new credential alongside the old, roll every consumer, verify traffic, then revoke the old one. The window where both are valid is your safety margin. Size it to your slowest consumer, not your fastest.

For signing keys (JWTs, webhooks, anything a third party verifies), the order inverts in one crucial way: the verifier must accept the new key before you start signing with it. Your dual-key window has to be longer than your longest-lived token plus your deployment lag. Pick a 15-minute window and you'll log out every user with a one-hour session, which is survivable — until it happens during the incident.

The failure I see most often isn't a hard crash. It's the silent tail: the cron job that runs at the top of the hour and fails because its credential died, no alert fires, and you discover it at month-end when a report is missing rows. Hard failures page you. Soft failures wait.

Why "everything" is where plans die

Even a well-run team usually finds that rotation splits into tiers with wildly different costs:

• Tier 1 — machine credentials in your own platform. Fully automatable. If you can't rotate these in minutes, that's a tooling problem, not a security problem.
• Tier 2 — third-party API keys. A dashboard click each. Fast, but rate-limited by human attention and by how many dashboards exist.
• Tier 3 — signing keys and OAuth client secrets. Need coordination with consumers, sometimes with a partner. Hours, not minutes.
• Tier 4 — the ones that aren't in the platform at all. A laptop, a CI variable, an SFTP account at a vendor that requires a support ticket, a Lambda someone deployed in 2023 for a demo that became load-bearing.

Tier 1 is where the tooling is. Tier 4 is where the time goes. In a typical sweep, the last 15% of secrets consume most of the wall clock — and some of them simply cannot be rotated without a human at another company.

So the number that matters is not "how many secrets do we have." It's how many distinct humans must click something, and how many of them are not on this call. That's the ceiling on your response time, and it has nothing to do with your secrets manager.

The drill: 90 minutes, once a quarter

"Evacuate the building" works as an instruction because there are exits, drills, and a known duration. "Rotate everything" works the same way or not at all. Build the muscle before you need it:

1. Announce it. A fire drill that looks like an incident burns trust and on-call goodwill. Say the date, say the window, say what might break.
2. Enumerate first, rotate second. Time how long it takes just to produce the list of every secret production uses, with an owner. Many teams discover the list doesn't exist and the answer is "infinite." That's a valid drill result.
3. Rotate 100% of Tier 1, and one Tier 3. The Tier 3 rotation is the interesting one — it's where the coordination cost lives.
4. Score four numbers: minutes to full enumeration, minutes to rotate all of Tier 1, number of services that broke, and number of secrets with no named owner. Publish them where engineers can see them.
5. Rotate the cheapest secret in production every quarter, forever. Not for security — for memory. A rotation path that isn't exercised rots and takes the runbook with it.

A first drill usually fails. That's the return on investment. You found out on a Tuesday afternoon instead of at 14:00 UTC on a day when someone else's breach set your schedule.

The trade-offs you should name out loud

Rotating more than necessary has real costs, and pretending otherwise makes the plan sound unserious. Invalidating sessions logs out every user. Rotating a webhook secret breaks integrations until the other side updates. Rotating a partner credential may need a change request and a support ticket measured in days. Some secrets genuinely cannot be rotated on demand, and no amount of policy changes that.

So put a rotation cost column next to every secret in your inventory. Cheap ones get automated and rotated on a schedule so the muscle exists. Expensive ones get a documented paper drill and an accepted-risk note with a name on it. The dangerous category is the secret that everyone assumes is cheap until the day it isn't.

And fix the visibility gap before you need it. The reason October 30 became a blanket "assume all" is that nobody could produce a shorter list. Audit logs, access records, and a queryable answer to "who read this secret, and when" are cheaper than a full-fleet rotation ordered under pressure.

Key Takeaways

• "Rotate everything" is a symptom. When you can't scope what was read, everything is in scope — that's a logging gap, not a security posture.
• Rotation is a deployment. Treat it as a dependency-closure problem with an ordering rule and a rollback, not a settings change.
• Order matters more than speed. Create new, roll consumers, verify, then revoke. Size the overlap window to your slowest consumer.
• The tail is the cost. Count the humans outside your team who must act — that's your real response time.
• Rehearse or guess. A 90-minute quarterly drill with four published numbers will tell you more than any policy document.
• Name the expensive secrets. Some credentials cost days of coordination. Write that down with an owner instead of discovering it mid-incident.