Snapshots Aren’t Backups: A CTO Playbook for Ransomware‑Proof Recovery

The week another national land registry was wiped and a US healthcare vendor lost “significant” data, one truth got louder: if your “backups” live where your production IAM can touch them, they’re not backups. They’re bargaining chips.

You don’t need a bigger lecture on ransomware. You need a recovery plan that survives a root‑level breach without drama, restores fast enough to save the business, and costs a single‑digit percentage of your infra bill. This post gives you that plan.

The problem: snapshots are negotiable, backups are not

Ransomware crews don’t start by encrypting files. They start by deleting recovery paths. If your recovery path is EC2/EBS snapshots, a compromised admin token can purge them in minutes. Even cross‑region replication won’t save you if the same federation chain can delete there too. Ditto for persistent volumes in Kubernetes, database autosnapshots tied to the same control plane, and “weekly exports” sitting in an S3 bucket your app role can write to.

Two recent headlines should be sobering to any CTO: an attacker wiped a country’s land registry database; a US vendor serving thousands of hospitals reported large data theft and disruption. In both cases, continuity hinged on whether backups were immutable, isolated, and drill‑proven. Most aren’t.

The decision framework: score your recovery posture on five axes

Before you buy anything, assess your recovery posture across five axes. You need all five to be green.

1) Mutability: Can an attacker with prod admin creds change or delete your backups?

• Acceptable: S3 Object Lock (compliance mode) or GCS Bucket Lock with retention. Azure Immutable Blob with Time‑based Retention.
• Not acceptable: Snapshots or objects deletable by the same IAM principals that operate prod, or by any federated SSO path used day‑to‑day.

2) Isolation: Are backups in a separate blast radius?

• Acceptable: Separate cloud account/project/tenant with Service Control Policies (SCPs) preventing policy or lock changes, separate admin chain, separate KMS root, separate MFA and hardware tokens.
• Not acceptable: Same account, same KMS, or any break‑glass that a compromised IdP can assume.

3) Key independence: Can you decrypt if your primary KMS is down or burned?

• Acceptable: Dedicated KMS keys in the backup account, with key material and admins not shared with prod. Optionally escrowed in an HSM or split knowledge scheme.
• Not acceptable: Backup data encrypted solely under the same KMS keys used by prod.

4) Recoverability SLOs: Do you have a measured RPO/RTO?

• Acceptable: Defined and tested RPO ≤ 15 minutes for primary OLTP data; RTO ≤ 4 hours for a full stack recovery of your critical path service.
• Not acceptable: “We think we can restore in a day.” If you haven’t timed it, it doesn’t exist.

5) Coverage: Is everything you run and rely on recoverable?

• Acceptable: Databases, object stores, infra state, container images, secrets, CI artifacts, and SaaS dependencies (GitHub, Slack, Google Workspace, Notion) with export automation.
• Not acceptable: Only databases or only VM snapshots.

The reference architecture: ransomware‑resilient backups on public cloud

This pattern assumes AWS, but the same primitives exist on GCP and Azure.

Core pattern: Double‑account, object‑locked, versioned backups

1. Create a dedicated “Backup” account with its own root, MFA, and hardware security keys. No federation from your corporate IdP for root access. Apply SCPs that disallow changes to Object Lock settings, KMS key deletion, and role modifications except from a controlled break‑glass path.
2. Provision S3 buckets with Object Lock enabled in compliance mode, versioning on, and a default retention window (e.g., 30 days). Governance mode is not enough; compliance mode prevents even root from removing retention until expiry.
3. Use cross‑account replication from a “Staging Vault” bucket in your prod account to the Object‑Locked buckets in the Backup account. Replication should be one‑way and initiated by a dedicated replication role in the backup account, not from prod. The destination bucket applies Object Lock on write.
4. Encrypt with a backup‑account KMS key (SSE‑KMS) that has no grants to prod roles. Store key admins in a separate security group with hardware tokens and out‑of‑band emergency contacts.
5. Tier storage: 30 days in S3 Standard‑IA for fast restore; monthly fulls in Glacier Deep Archive for 12 months. Object Lock applies across versions and storage classes.

Data movers: App‑aware, incremental, verifiable

• Postgres: Use pgBackRest or WAL‑G to ship base backups and WAL to the Staging Vault, then replicate to the Locked bucket. Configure continuous archiving to target an “append‑only” prefix a prod app role cannot list or delete. Run daily restore‑verify jobs that spin up a temp instance, replay WAL to a target timestamp, run CHECKPOINT, and do a logical sanity query set.
• MySQL: XtraBackup full + incremental, binlog shipping, and daily restore verification.
• MongoDB: Continuous Oplog capture with periodic mongod snapshot and replay; validate collection counts and hashes after restore.
• Blob stores: Don’t mirror the whole bucket hourly. Generate content‑addressed bundles with manifests and hashes (e.g., zstd‑compressed tarballs per partition/day) so integrity is verifiable and restore is parallelizable.
• Infra state: Terraform state remote backend with versioning and object lock, plus periodic snapshots to the Locked bucket. Keep AMIs or image snapshots for golden base OS in the backup account.
• Secrets: Nightly export and re‑encryption of Vault/Secrets Manager items into the backup account under the backup KMS. On restore, rotate everything.

SaaS dependencies: automate the boring exports

• GitHub: Mirror repos to read‑only backup remotes in the backup account or a neutral Gitea mirror; export issues and PR metadata weekly.
• Slack: Enterprise export API for public and private channels; apply legal retention policy. Store exports under Object Lock.
• Google Workspace/Microsoft 365: Use Vault/Compliance exports for mail and Drive; monthly for non‑regulated, weekly for regulated teams.
• CI/CD: Snapshot build cache and essential artifacts; keep Docker image digests and a copy of critical images in the backup registry.

Numbers that matter: size, speed, and cost

Let’s ground this with a sane startup profile:

• Primary OLTP Postgres: 4 TB
• Object data: 12 TB
• Infra/config/secrets/other: 4 TB equivalent
• Total primary: ~20 TB

Assume 2% daily change rate across datasets and 40% compression/dedup on backups.

• Storage footprint: ~8 TB base full + ~400 GB/day incrementals. 30‑day hot window ≈ 8 TB + (0.4 TB × 30) = ~20 TB in S3 Standard‑IA.
• Monthly hot storage cost (us‑east‑1): ~20 TB × $0.0125/GB ≈ $250. Double it for cross‑account replication ≈ $500. Add requests/retrieval cushion: call it $700–$900/month.
• Cold archive: Keep one monthly full in Glacier Deep Archive for 12 months: 8 TB × 12 ≈ 96 TB × ~$0.001/GB ≈ $100/month.
• Restore throughput: Target 1 TB/hour per database and 2–3 TB/hour for object data parallel pulls. That puts your 4 TB DB back in 4 hours; blobs in ~4–6 hours.

Bottom line: a ransomware‑resilient posture for this profile is well under 1–2% of a $50–100K/month cloud bill. The real cost is the engineering discipline to drill it.

Runbooks that actually work (and how to prove it)

Define SLOs before you write steps

• RPO: ≤ 15 minutes for OLTP, 1 hour for blobs.
• RTO: ≤ 4 hours to user‑visible service, ≤ 24 hours for long‑tail data.

Restoration order of operations

1. Containment: Lock IdP, rotate break‑glass. Freeze prod accounts. Preserve forensics: memory, disks, and logs to a WORM evidence bucket separate from your backup buckets.
2. Rebuild the foundations first: Identity (Okta/AD), network primitives, KMS in the clean account.
3. Secrets and images: Stand up Vault/Secrets Manager from backup. Rotate keys, tokens, and database passwords immediately post‑restore.
4. Databases: Restore base, replay WAL/binlogs to RPO target, validate schema, counts, and hashes. Promote read‑only, then switch to primary.
5. Blob/object stores: Parallel restore by partition/day. Validate manifests/hashes.
6. App/services: Deploy from golden images; strictly block unknown AMIs and unsigned containers.
7. Traffic: Gradual cutover behind a feature flag or read‑only mode first, then full write traffic.

Drills cadence and metrics

• Monthly: 2‑hour partial restore drill (DB subset + a service) during business hours. Success = RPO/RTO hit, data validated, runbook steps marked with time stamps.
• Quarterly: Full‑day, red‑team‑assisted table‑top + live restore into an isolated VPC. Include SaaS export imports. Success = users can log in and perform a real transaction.
• Track: Median/95th restore time, verification failures, missing dependencies, and surprises. Your board packet should show a recovery SLO you actually meet.

Anti‑patterns to kill this quarter

• “The provider has us”: EBS snapshots, RDS automated backups, and GKE PV snapshots are in the same control plane. They’re convenient restores, not ransomware defenses.
• Shared KMS: Encrypting backups with the same keys or same admins as prod means one blast radius.
• “We’ll rely on versioning”: S3 versioning without Object Lock is deletable by an admin.
• Hot‑only posture: A 30‑day hot window without cold archive leaves you exposed to long‑dwell tampering.
• Unverifiable blobs: Random object copies without manifests and hashes make integrity claims performative.

What about offline?

True air‑gap is still the gold standard. For most startups, it’s overkill operationally. A pragmatic middle ground:

• Quarterly offline snapshot: Export quarterly fulls to encrypted LUKS volumes on removable SSDs, store in secure offsite storage with chain‑of‑custody logs. This is your meteor‑strike copy.
• Cloud‑air‑gap variant: A backup account with Object Lock + SCPs, no standing network peering to prod, and admin auth that’s not federated from your corporate IdP. It’s not a literal air gap, but it breaks most breach chains.

Legal and forensics you’ll wish you had

• WORM evidence: Write audit logs, EDR telemetry, and snapshots of compromised systems to a separate WORM bucket with 1‑year retention. Keep it logically and administratively isolated from your backup volume.
• 72‑hour readiness: Many regimes require incident notification in 72 hours. Pre‑bake templates, contacts, and a fact‑based checklist of what you can assert (retention windows, encryption at rest/in transit, object lock IDs).
• Third‑party coverage: Document which vendors you rely on for continuity and what their WORM/immutability posture is. Include clauses for backup immutability and recovery drills in MSAs.

Open source, agents, and incremental everything

Two trends worth acknowledging from this week’s tech chatter. First, incremental computation libraries are getting good enough to make continuous verification cheap. Apply that mindset to backups: compute rolling checksums and verification queries incrementally instead of in big weekend jobs. Second, the AI tooling wave makes it tempting to defer ops discipline to “agents.” Don’t. Use agents to generate restore manifests and validate records, but keep the authority to delete or unlock out of their hands. In a breach, agency belongs to humans.

Who owns this and how to resource it

• Single throat to choke: Assign a Staff+ SRE as Recovery Owner. Their scorecard is RPO/RTO SLOs and drill pass rate.
• Budget: Plan for 1–2% of cloud spend. Most of that is storage; the rest is drill time.
• Operational hours: Use a nearshore pod to run monthly drills and keep the runbooks fresh. With 6–8 hours of US overlap, drills don’t wreck your on‑call rotation, and you still get coverage for early‑morning incidents.

Practical starting points for the next 30 days

1. Turn on Object Lock in a new, dedicated backup account. If you can’t, create a new bucket; you can’t retro‑enable it on existing ones.
2. Separate KMS in that backup account and remove all prod principals. Document the break‑glass.
3. Wire one data mover end‑to‑end: e.g., pgBackRest base + WAL to Staging Vault, replicated to Locked bucket. Prove a 4‑hour restore on a 500 GB subset.
4. Export a SaaS you rely on (GitHub or Slack) and store it under Object Lock. Time it.
5. Book your first drill on the calendar and invite legal. If it’s not scheduled, it’s not real.

Trade‑offs and reality checks

• Immutability vs. agility: Object Lock in compliance mode will bite you when you make mistakes. That’s the point. Use staging buckets to catch errors before locking.
• Cost vs. RTO: Glacier is cheap but slow. Keep a 30‑day hot window in IA to hit your RTO; push older to Deep Archive.
• Security vs. operability: Separate accounts and keys mean more ceremony. Write the ceremony down and practice it, or you’ll invent shortcuts during an incident.
• People vs. tools: The best architecture won’t save you if the only person who knows the restore steps is on vacation. Cross‑train and record screen‑capture run‑throughs.

If you do nothing else

If you only do one thing this quarter, make one dataset truly immutable and test a timed restore. You’ll learn more in that one drill than in ten tabletop exercises. And you’ll never again confuse snapshots with backups.

Key Takeaways

• Snapshots are not backups. If prod IAM can delete your recovery path, so can an attacker.
• Use Object Lock/Bucket Lock in a separate account with separate KMS keys and SCPs.
• Set and measure RPO ≤ 15 minutes and RTO ≤ 4 hours for your critical path.
• Drill monthly (partial) and quarterly (full). If it’s not tested, it doesn’t exist.
• Back up SaaS dependencies, not just infra: GitHub, Slack, Google/Microsoft, CI.
• Target 1–2% of cloud spend for resilient backups; the rest is discipline.
• Assign a Recovery Owner and use a nearshore pod to keep runbooks fresh with 6–8 hours overlap.