Prove You Can Leave Your Storage Vendor: Run a Quarterly Egress Fire Drill

When a PBS affiliate recently said a cloud storage provider ghosted them—leaving ~50 TB of data at risk—their pain echoed across engineering Slack channels. If your storage vendor stopped answering tomorrow, how many hours would it take to extract your critical datasets? What would it cost? Who owns the runbook? Most teams can’t answer without hedging. That’s a failure mode.

You test backups. You practice incident response. But you probably haven’t rehearsed the one move that matters most when a vendor relationship sours: getting your bytes out—fast, intact, and without bankrupting yourself on egress fees. In 2026, storage egress is a business continuity risk, not an edge case.

The problem isn’t just outages. It’s custody.

“Ghosted by a vendor” covers more than downtime. It includes:

• Control plane failure: Support stops responding; IAM changes or billing holds lock you out; listing APIs degrade; rate limits throttle you when you start copying data out.
• Silent incompatibilities: “S3-compatible” doesn’t guarantee your tooling’s semantics. Multipart ETags differ, ListObjectsV2 pagination can be slow with deeply nested prefixes, and object versioning or legal holds can block deletes and moves.
• Cost bombs: Egress at $0.02–$0.09/GB sounds harmless until you move tens of terabytes under a contraction clock.

The risk isn’t theoretical. Media archives, medical imagery, ML training corpora, logs, build artifacts, even customer uploads accumulate in object stores that are easy to fill and hard to evacuate under pressure. You can’t spreadsheet your way out of this; you need measured throughput, verified integrity, and a drill that finds the paper cuts before they bleed you out.

A CTO framework: the Quarterly Egress Fire Drill

Goal: prove you can extract a representative slice of your data—end to end—in 48 hours, with integrity verification and a pre-negotiated cost envelope. Do it every quarter. Make it boring.

1) Pick a representative corpus (5–10% of total)

• Include large objects (1–20 GB media, model checkpoints) and small objects (1–100 KB logs, thumbnails). Small objects will throttle you on list+open; large objects exercise multipart pipelines.
• Mirror your real semantics: versioned buckets, Object Lock (compliance/governance), deep prefix trees, and awkward edge cases (keys with spaces, unicode).
• Target size: 2–10 TB if you hold 20–100 TB total. The point is to surface behavior, not mirror the full bill every quarter.

2) Pre-provision a destination you control

• Use a second provider with contractual teeth (e.g., Cloudflare R2, AWS S3, Azure Blob) or run your own S3 endpoint (e.g., MinIO) on a short-lived cluster with 10/25/40 Gbps NICs.
• Keep credentials separate and break-glass accessible. Do not rely on the source provider’s control plane to authenticate the destination.
• Turn on server-side encryption at the destination and store customer-managed keys (BYOK) in your own KMS, not the source provider’s.

3) Build a canonical manifest and integrity plan

• For every object: store key, size, version-id (if any), last-modified, SHA-256. Do not trust multipart ETags as MD5; they’re not.
• Use your provider’s inventory export (e.g., S3 Inventory) as a baseline, then compute missing checksums with a one-time map job.
• After copy, verify 100% of objects by size and SHA-256, and sample 1–5% with full-byte compare for added assurance.

4) Choose tooling and parallelism

• rclone for cross-provider copies with built-in checksums (--transfers, --checkers, --s3-chunk-size, --s3-upload-concurrency).
• s5cmd for millions of small files; it parallelizes list/copy more aggressively than the AWS CLI.
• For bulk within a single cloud, use provider-native batch tools (aws s3 cp/sync with careful flags) to avoid cross-Internet bottlenecks.
• Start with 64–256 concurrent transfers, 64–128 checkers, and 64–128 MiB multipart chunks. Profile, don’t guess.

5) Do the bandwidth math before you learn it the hard way

Rough conversion: 1 TB ≈ 8,192 Gbit. Time (hours) ≈ (TB × 8,192) / link Gbps / 3600.

• At 1 Gbps, 50 TB is ~114 hours (~4.7 days) if you saturate the link. You won’t.
• At 10 Gbps, 50 TB is ~11.4 hours in theory; plan for 18–24 hours with protocol overhead and object-store throttles.
• At 2 Gbps (typical office uplink), 10 TB can still take ~11–14 hours—if your listing performance and API quotas keep up.

Practical tip: when source and destination live in different clouds, pull from compute close to the source (same region), then ship over a dedicated interconnect or a high-throughput egress path you control. Don’t backhaul hundreds of TB through a single NAT in your HQ.

6) Put a real price tag on egress

• At $0.02–$0.09/GB, moving 50 TB costs $1,000–$4,500 just to leave—before compute and destination storage.
• Some vendors market “no egress fees” with caps or behavior clauses. Read them. If a drill triggers review or throttling, it’s not real freedom.
• Negotiate now: include a "no-fee egress on termination" clause and a "quarterly drill up to N TB with waived egress" clause. If they balk, price the risk explicitly in your TCO model.

7) Define success criteria

• Time to first byte: How quickly can you authenticate and start a transfer when the pager goes off?
• Sustained throughput: Average Gbps over a 2–6 hour steady state. Anything below 30–50% of theoretical needs diagnosis.
• End-to-end integrity: 100% size match; 100% checksum match; zero corrupt files; audit log shows complete coverage.
• All-in cost: Egress + compute + destination storage and any cross-connect fees.
• Runbook ownership: Named humans; break-glass credentials tested; on-call rotation includes drill duty.

Architect for exits before you need one

Build for S3 portability—but verify the corners

• Keep to the lowest common denominator: simple PUT/GET, lifecycle rules you can replicate, and avoid proprietary policies you can’t translate. Where you must go proprietary, isolate it behind your own abstraction.
• Versioning and Object Lock are good safety nets, but confirm that governance/legal holds won’t block copies. In drills, use a bucket with identical controls to production.
• Listing is your hidden bottleneck. Buckets with tens of millions of tiny objects crawl. Use compaction (bundle small files into Parquet/ORC or tar segments) and prefix partitioning (yyyy/mm/dd/...) to bound list operations.

Own your encryption keys

• If your data is locked behind the source provider’s KMS, an account freeze can become a data freeze. Prefer BYOK where possible; at minimum, maintain exportable key material or dual-encrypt sensitive archives before upload.
• Don’t discover mid-drill that a compliance hold on a key prevents reads. Test key rotation and restore paths regularly.

Keep object counts sane

• Tracking 200 million 1 KB objects is cute until your ListObjectsV2 rate limits dominate the clock. Cap bucket object counts per prefix; enforce compaction in ingestion pipelines; reject PRs that create per-row objects without compaction.

Designate a “golden manifest”

• Store the authoritative manifest (key, size, checksum) outside the source provider—e.g., in your git-backed ops repo and a separate object store—so you can verify independently.

What about block volumes, databases, and SaaS-only blobs?

Block volumes (snapshots)

• Automate snapshot export to raw images (e.g., VMDK/VHD or raw) onto object storage you control. Test a cold-restore onto a different hypervisor once a quarter.
• Measure restore time with realistic IOPS caps. If your RTO depends on bursting credits, you don’t have an RTO.

Databases

• Run logical exports (e.g., pg_dump/mysqldump) weekly for critical schemas to your own bucket. Keep schema+migration scripts versioned.
• Rehearse a cold import into an alt provider (managed or self-hosted). “We use replication” is not a substitute for “we rebuilt from dumps in 90 minutes.”

SaaS-embedded blobs

• Product docs, design assets, and logs trapped in SaaS (Figma, Notion, Sentry, vendor-specific media platforms) are easy to forget. Build scheduled exports via APIs into your bucket. If a vendor has no export API, it’s a risk line item, not a footnote.

Real numbers from the field

• 15-hour, 20 TB media evac: Moving a mixed media corpus (1–8 GB objects) with rclone (--transfers 128, --s3-chunk-size 128M, --s3-upload-concurrency 32) over a 10 Gbps link sustained ~3.5 Gbps after API throttling and TLS overhead. All-in egress was ~$200 at $0.01/GB promo pricing.
• Small-objects tax, 200M keys: A logging bucket with 1–4 KB objects hit ListObjectsV2 limits and application-side open/close overhead. Even at 10 Gbps theoretical, the effective throughput was < 200 Mbps. Compaction into hourly Parquet cut object count by 1000x and turned a week into hours.
• Office uplink is a trap: A team tried to pull 8 TB over a 1 Gbps office VPN, saturating for days and triggering firewall resets. Running copy jobs from compute in-region with the source completed in under 8 hours with the same tools.

Don’t forget governance: contracts and SLAs

• No-fee egress on termination: Spell it out. Quantify: “Up to X TB, Y days, Z requests, no fees.”
• Quarterly drill allowance: Bake in a test window (e.g., 5–10 TB per quarter) with waived throttling and support on standby.
• S3 compatibility matrix: Ask for a written list: multipart ETag behavior, max part size, ListObjectsV2 QPS, versioning semantics, Object Lock parity, eventual vs strong read-after-write guarantees.
• Support escalation: Contract a named escalation path and a response SLA for egress issues. If they won’t commit, that’s your signal.

Run it with a follow-the-sun team

These drills are perfect for a nearshore pod. You get 6–8 hours of US overlap for kickoffs and handoffs, then continuous progress while you sleep. At DHD Tech in Brazil, we’ve staffed egress drills with a rotating pair: one engineer drives copy/verification during US hours, another tends the pipeline into the evening, filing concrete throughput stats, cost tallies, and fix PRs for compaction and prefix hygiene. By morning, you have a clean postmortem and a green manifest.

A 30/60/90 plan you can start this quarter

Days 1–30: Prove you can move 2–5 TB

• Pick a target bucket and carve a 5% slice with real-world pathing, object sizes, and controls.
• Stand up a destination (second provider or MinIO) and wire break-glass credentials.
• Generate a manifest and checksums; run a first copy with rclone or s5cmd; record throughput and cost.
• Open JIRAs for discovered issues: compaction gaps, prefix hot-spots, throttling limits, missing BYOK.

Days 31–60: Remove the bottlenecks

• Implement compaction for small objects going forward; backfill the worst offenders.
• Raise QPS limits or parallelism where safe; add ephemeral compute close to source buckets.
• Negotiate contract riders: no-fee termination egress, quarterly drill windows, S3 compatibility attestation.

Days 61–90: Full drill, exec visibility

• Run a 10 TB drill start-to-finish inside a 48-hour window.
• Publish a one-page dashboard: sustained Gbps, total hours, integrity pass rate, and cost per TB.
• Institutionalize: calendar the next drill; add runbook links and manifest locations to your BCP.

Good engineering is about proving reversibility

It’s easy to upload. It’s hard to leave. That asymmetry is your risk. A quarterly egress fire drill forces your architecture, contracts, and ops hygiene to prove they work in reverse. You’ll find dead DNS names in lifecycle rules, IAM policies that looked fine until they blocked headless copies, buckets with 100M zombie objects, and KMS keys whose rotation policy would have bricked you in a real incident.

Fix them on your time—not your vendor’s.

Key Takeaways

• If a vendor ghosts you, you have a custody problem, not just a reliability problem. Practice exits.
• Run a quarterly egress fire drill on 5–10% of your data; verify SHA-256 and measure real throughput and cost.
• Small objects kill performance. Compact into Parquet/ORC and partition prefixes to bound listing overhead.
• BYOK or dual-encrypt so key custody doesn’t become data custody.
• Negotiate no-fee termination egress and a quarterly drill allowance in your contract.
• Stage compute close to the source to avoid office uplink bottlenecks; use rclone/s5cmd with aggressive parallelism.
• Track a golden manifest outside the source provider; verify 100% and spot-check full-byte integrity.
• Staff drills with a nearshore pod for 6–8 hours overlap and 24-hour momentum; publish an exec-ready scorecard.