2026-11-09 · 9 min read

Your Cron Job Is a Power Grid Problem: The Cost of Agreeing on When

Flat isometric illustration on a near-black background showing many small devices firing pulses at once into a tall indigo spike, beside the same devices staggered into a smooth low violet wave with one green node.

Every sixty seconds, the frequency on Ireland's power grid sags by a few millihertz. Then it recovers. Then it does it again. The same dip shows up in Germany, Australia, Malaysia and the United States — at the same instant, on grids that share nothing but a clock.

A millihertz is a thousandth of a hertz. Twenty of them is 0.04% of a 50 Hz grid, which is invisible to you and meaningless to a toaster. It still took researchers at Forschungszentrum Jülich, working with the German transmission operator Tennet, years to trace where it came from. TechCrunch's account of the finding is worth your time, because the answer isn't really about power engineering. It's about what happens when a lot of independent systems quietly agree on when to do something.

The culprit was a battery check

The researchers started where you'd expect: hydropower turbines, whose vibrations can resonate through a grid. That got them partway but didn't explain the clockwork regularity. Resonances come and go. This one arrived every sixty seconds, forever.

So they installed more than 80 power-quality meters around their own campus and watched. The signal peaked on the transformer serving the battery storage system. Every minute.

Stationary batteries — the uninterruptible power supplies in data centers, the grid-scale packs, the little one under your desk — check their state of charge on a fixed interval. They're internet-connected, so their clocks are synchronized to within milliseconds. When one checks its charge, the load bump is nothing. When a few hundred thousand do it in the same second, it becomes a measurable event on a national grid.

The trend line is not subtle. On Ireland's grid, which by 2022 was carrying data centers at 18% of total load, the minute-by-minute dip roughly quadrupled from 3 millihertz in 2016 to 12 millihertz in 2021. Some recent dips exceed 20 millihertz. The grid is engineered to survive swings of plus or minus 200, and operators would rather keep it inside 100. So this is not a crisis. It's headroom being eaten by a scheduling decision nobody made on purpose.

The failure had no owner

Here's the part that should make you uncomfortable: not one device in that fleet was misconfigured. Each battery did exactly what its spec said. Each was cheap to check. Each, alone, was correct.

The bug lived in the aggregate. It existed only when you added up ten thousand correct decisions that happened to share a timestamp. No single vendor's test suite would ever have caught it, because no single vendor owned the sum.

You have that bug. Probably several.

Where shared clocks bite you

The pattern is always the same: a periodic action, a fleet of actors, and a time source they all trust. In infrastructure, that's NTP. In your application, it's often something dumber — the moment a deploy finished, or the moment a cache entry was written.

  • Cron at the top of the hour. Four hundred workers running 0 * * * * don't do four hundred hours of work spread evenly. They do four hundred units of work in the same second, then idle for 3,599 more.
  • Cache TTLs set at deploy time. Every key written during a rollout gets the same expiration. An hour later, all of them miss at once and your database gets a spike it interprets as an attack.
  • Retry loops with a fixed interval. One dependency blips, every client waits exactly one second, and the recovery gets hammered by a synchronized wall of retries that looks much worse than the original failure.
  • Token refresh on a fixed cadence. Every instance booted at the same time refreshes at the same time, so your auth service sees a flat line punctuated by cliffs.
  • Telemetry flushes. Fifty thousand devices posting every ten seconds at :00, :10, :20 — a reader that could handle 200 requests per second gets 50,000 in a 400-millisecond window.

None of these are exotic. All of them are survivable at small scale, which is exactly why they ship.

Why your load test won't find it

Two reasons, and both matter for how you fix it.

First, load tests usually start at t=0 with every virtual user synchronized, then ramp. That means the spike is baked into the test from the first second, so it looks like normal behaviour and you tune capacity to survive it rather than smoothing it away. You end up buying hardware to absorb a scheduling artifact.

Second, the damage is proportional to fleet size, and your staging environment has three replicas. The per-actor cost is trivial — one query, one HTTP call, one battery check. You can't feel it until you have thousands of actors, and by then it's in production and nobody suspects the schedule.

The math is the useful part. If you have N jobs per period, each holding a resource for S seconds, spread over a window W, peak concurrency is roughly N × S / W. With 400 hourly jobs at 8 seconds each, firing together means 400 concurrent database connections. Spread over a 600-second window, the same work peaks around 6. Same workload. Two orders of magnitude less capacity required.

Three places to put the random number

"Add jitter" is the advice; where you add it determines whether you can still debug anything.

1. Phase jitter (stable offset — the default)

Give each client a fixed offset derived from its identity, not from a random number generator. Something like hash(client_id) modulo the period. Clients spread out, but a given client always lands in the same slot, so you can look at one machine and reason about it. This is what you want for cron jobs, polling intervals and scheduled syncs. It's also what RandomizedDelaySec does in systemd timers, except systemd picks per-run; hashing your own identity is more debuggable.

2. Per-run jitter (randomized interval — for retries)

For anything that runs in response to a failure, use full jitter: pick a random delay between zero and the backoff cap, not the cap plus a random nudge. This is the well-known fix for retry storms, and it works because the clients are already desynchronized by the time they're failing. Add a cap so a bad dependency can't keep you in backoff forever.

3. Deadline-preserving jitter (when the output must land by a wall-clock time)

Billing runs at 00:00. Payroll closes on the 25th. You cannot randomize those. What you can do is separate the boundary from the work: take a consistent snapshot at the exact boundary, then let every worker process its slice over the following hour, whenever it gets scheduled. The number is correct because the snapshot was; the load is smooth because the processing wasn't.

What breaks when you add jitter

Be honest about the cost, because it's real.

  • "Runs at midnight" assumptions. Half your pipelines probably assume that a job at 00:00 sees yesterday's complete data. Once jobs land anywhere in a window, you need a watermark — an explicit marker of what data is final — instead of a clock reading.
  • Dashboards get noisy at boundaries. Hour-over-hour comparisons against a moving edge look worse than they are. Bucket by completion time, not schedule time.
  • Ordering gets weaker. If job A must run before job B, jitter can invert them. Use an explicit dependency, not a two-minute gap.
  • Jitter hides capacity problems. If the flattened load is still above what your database can take, you've converted a spike into a plateau — a slower, duller outage. Smoothing buys you time to fix the real limit; it doesn't remove it.

And the fix is worth doing anyway, because the alternative is waiting for vendors. Fewer than 3% of IoT devices run their latest firmware. The battery makers can ship a randomized schedule in a patch, and most of those patches will sit uninstalled for years. The researchers' conclusion was blunt: the mystery is solved, the problem will persist.

A twenty-minute audit

Pull your schedules and sort them by how many actors fire at once, not by how expensive each one is.

  1. List every recurring job. Note the period and how many instances run it.
  2. For each, ask: if all of them fired in the same second, what would break? If the answer is "nothing," move on. If it's "the primary," you found one.
  3. Compute peak concurrency with the N × S / W formula. Compare it to what your smallest production node can absorb.
  4. Add stable phase jitter sized so peak concurrency sits near average concurrency — and make the window at least a few seconds wide so sub-second clock alignment doesn't survive.
  5. Replace every wall-clock dependency downstream with a data watermark.

The grid researchers' recommendation was one line of code: don't check the battery exactly on the minute. That's the whole fix. It's unglamorous, it's cheap, and nobody ships it, because the failure it prevents is invisible until the day it's expensive.

Key Takeaways

  • Individually harmless periodic actions become a single large event when the actors share a clock. The bug lives in the aggregate and has no owner.
  • The same pattern is behind cron spikes, cache stampedes, retry storms and telemetry floods — and it grows with fleet size, not with load.
  • Peak concurrency is roughly N × S / W. Spreading 400 eight-second hourly jobs over ten minutes cuts concurrency from 400 to about 6.
  • Prefer stable, hash-based phase jitter so individual clients stay debuggable; use per-run randomization for retries; preserve hard deadlines by splitting the boundary from the work.
  • Smoothing doesn't raise your ceiling. If the flattened load still exceeds capacity, you've just made the outage quieter and longer.
  • Replace wall-clock assumptions downstream with explicit data watermarks before you turn jitter on.

[ Call to action ]

Ready to scale your engineering team?

Tell us the roles you need to fill and we'll get back within 24 hours.