Turn Off Half Your Topics: A 30‑Day Audit for Pub/Sub, Kafka, and SNS

You probably have a topic right now that nobody needs. It’s still hot, still billing, still paging on-call during a retry storm—and zero business value reaches a user. A widely shared dev story this month about turning off Pub/Sub and nobody noticing wasn’t a fluke. In our nearshore audits for US startups and scale-ups, we routinely find 20–40% of event traffic is dead weight: zombie topics, redundant fanouts, accidental analytics, and “just-in-case” streams that nobody owns.

If you run Pub/Sub, Kafka (self-managed or Confluent), SNS/SQS, or Kinesis, you’re paying twice: money and noise. The money is obvious. The noise is worse: cascading retries, lag alarms, and a distributed systems tax on every deploy. The fix isn’t a six-month rewrite. It’s a 30-day, low-risk circuit-breaker audit with hard gates and brownouts. Here’s the playbook.

Why this matters now

• Event infra scaled faster than your ownership model. It’s trivial to publish; it’s rare to deprecate.
• Vendors reward fanout. One event to three subscriptions is 3x deliveries and 3x failure surfaces.
• AI-driven features amplify event volume (clickstreams, traces, embeddings) and hide waste in “analytics.”
• Recent industry chatter (“we turned off Pub/Sub...”) showed what we see in the field: no blast, big savings.

At a typical Series B–D product with 80–200 topics/streams, 150–600 consumer groups, and a few high-throughput producers, we see:

• 10–30% topics with no effective consumers (no acks, no materialized effect, or consumer offline >30 days).
• 5–10% analytics fanouts that duplicate batch jobs or warehouse CDC.
• 3–5% notification/event mirroring to multiple buses for “portability” that never materialized.
• 1–3% outright policy violations (PII in payloads, cross-region replication without a DPA).

Cleaning this up cuts 15–35% of monthly messaging spend and reduces incident pages by ~25–40% because your system stops retrying into a void.

The audit lens: Five questions per stream

Don’t start with costs; start with effects. For every topic/stream/subscription tuple, answer:

1. What changes for a user if this stream pauses for 24 hours? Name the screen, API, or SLA. If you can’t, it’s suspect.
2. Can the consuming system recover from missing events? If the answer is “we can recompute from the source of truth,” this is a brownout candidate.
3. Is this data available via a cheaper path? Warehouse table, CDC, periodic batch, direct service call, or edge cache.
4. Who owns it? If ownership is a Slack handle and not a team with an on-call rotation, it’s debt.
5. What is the redundancy multiplier? Fanout count × delivery attempts × replays. Large multipliers deserve re-justification.

Measure before you move: the seven metrics that expose zombies

Export these per topic/stream and per subscription/consumer group for the last 30 days:

• Ingest rate (messages/s, bytes/s) and delivery rate per consumer.
• Consumer liveness (last ack timestamp, average ack delay, percent time consuming).
• Lag (offsets for Kafka; oldest unacked age for Pub/Sub/SQS).
• Retry/backoff counts and DLQ volume.
• Duplicate ratio (message keys/events seen >1 across consumer groups within N minutes).
• Schema churn (versions/month) and schema violations (bad payloads per 10k messages).
• Cost proxy: deliveries × average payload bytes (egress), storage hours × partitions/retention, cross-region replication bytes.

Plot three distributions: topics with zero recent acks, topics with consumer lag growing but no incidents opened, and topics whose total delivered bytes > 3× published bytes. That last one uncovers expensive, multi-subscription fanouts where nobody can explain the extra two consumers.

The 30‑day plan: low-risk, reversible, and visible

Days 1–7: Inventory, tag, and trace

• Inventory sources and sinks. Emit a producer header on every message for 30 days: x-stream-owner, x-purpose (customer-impacting, analytics, internal cache), x-pii (none, pseudonymous, sensitive), and x-criticality (tier 0–3).
• Map effects. For each consumer group, name the user-facing effect: “orders page shows up-to-date status,” “billing emails,” “internal dashboard.” No effect, no protection.
• Enable shadow tracing. Sample 1–5% of messages and correlate to downstream API calls, DB writes, or UI telemetry. If streams don’t correlate, they’re likely dead.
• Set cost baselines. Even if your vendor pricing is opaque, compute a relative score: deliveries × bytes × replications. You don’t need dollar precision to choose targets.

Days 8–14: Classify and draft the kill list

• Classify streams: Tier 0 (user/blocking), Tier 1 (user/degraded OK), Tier 2 (internal), Tier 3 (analytics/backfill-only).
• Identify zombies: zero acks in 30 days, or consumers with liveness < 5% and no incident history.
• Find redundant fanouts: same payload published to multiple buses (e.g., SNS and Kafka) “just in case.” Pick one. Leave a minimal bridge if you truly need both.
• Spot payload inflation: large blobs (100–500 KB) in events used for a single field downstream. Replace with IDs and fetch-on-read in Tier 1+.
• Brownout candidates: Tier 1 or 2 streams where consumers can recompute or tolerate staleness for 24h. Mark for circuit-breaker trials.

Days 15–21: Put in the safety rails

• Introduce per-subscription circuit breakers. Feature-flagged drop at ingress or subscription delivery. Default to fail-open on Tier 2–3 (drop messages), fail-closed on Tier 0.
• Add replay insurance. Mirror raw events to cheap object storage (e.g., GCS/Amazon S3) for 7–14 days with a compacted JSONL or Parquet log. If a kill breaks something, rehydrate.
• Publish deprecation notices inside the stream itself (a control message every N minutes) and in your internal comms. If nobody screams, that’s a sign.
• Observability snapshots. One dashboard per candidate: ingest/deliveries/lag/retries, last effect timestamp, and a big owner name.

Days 22–30: Brownouts, cuts, and consolidation

• Run brownouts: 1-hour drop at 5–10% of traffic in business hours for Tier 2–3. Then 4-hour at 100% for Tier 3 off-peak. Watch support tickets, SLOs, and dashboards.
• Delete or disable zombies (no acks, no effect). No partial measures. Document deprecation with an end date and contact.
• Consolidate fanouts. Route analytics-only consumers through a single bus and batch into the warehouse. Kill direct fanouts to three analytics sinks.
• Shrink retention on hot paths with strong source-of-truth backing. If you can replay from DB or object storage, you don’t need 7 days of backlog in Kafka.
• Right-size partitions. If 80% of partitions are nearly idle, halve them. For Pub/Sub and SQS, reduce parallelism where it inflates request costs with no latency win.

What savings look like (realistic ranges)

Messaging pricing is a labyrinth. You don’t need exact dollars to make the decision. Use these back-of-the-envelope ranges to set expectations:

• Delivery volume is the biggest lever. If you have 200M events/month and average 2.5 subscriptions, you process 500M deliveries. Killing one redundant subscription on 30% of topics can cut 15–25% of total deliveries immediately.
• Payload size matters. Dropping average payload from 20 KB to 3–5 KB (IDs, not blobs) reduces egress and storage by ~70–85% for those streams. If only 30% of your traffic is bloated, you still save ~20% overall egress.
• Retention is stealth spend. Cutting Kafka retention from 7 days to 48 hours on hot paths (while mirroring to object storage) can trim 40–60% of broker storage. For managed Kafka, that’s direct bill reduction. For self-hosted, it’s fewer disks and fewer “Kafka is out of space” pages.
• Partitions and connections drive operational overhead. Reducing partition count 30–50% on underutilized topics removes rebalancing storms and cuts CPU by double digits.

Across companies spending anywhere from a few thousand to tens of thousands per month on messaging (Confluent, Pub/Sub, SNS/SQS, Kinesis), we see 15–35% cost reduction in 30–60 days with this audit, plus a measurable drop in incident volume. The operational win often dwarfs the invoice delta.

Governance that keeps waste from regrowing

Turning things off is the easy part. Keeping them off takes three habits.

1) Make ownership in-band

• All producers must set x-stream-owner to a team alias that’s on-call. Broker policies reject messages without it.
• All consumers must register a x-criticality tag and a data classification. No tag, no subscription.
• Topics without an owning team auto-expire in 90 days unless renewed.

2) Put brownouts into CI/CD

• Every new stream ships with a circuit-breaker flag and a 1-hour brownout test in staging that validates user-facing SLOs stay green or degrade within defined bounds.
• Block deploys for streams that fail the brownout test without an approved Tier 0 exemption.

3) Stop pretending all events are real-time

• Define two lanes: operational (sub-1s latency, Tier 0–1) and analytical (minute-scale latency, Tier 2–3). Default new analytics to batch or micro-batch via your warehouse ingestion (e.g., CDC + incremental models).
• Audit quarterly: any analytical consumer on your operational bus must earn an exception or move.

Architectural trade-offs you should acknowledge

• Event-sourcing vs. event-driven. If you’re using the log as your source of truth, don’t slash retention or partitions blindly. Your replay story is your uptime story.
• Cross-service coupling. Replacing fat events with IDs can reintroduce synchronous lookups. That’s fine for Tier 1–3; for Tier 0 paths under strict latency SLOs, keep minimal state in the event (hashes, version IDs) and cache smartly.
• Analytics latency. Moving analytics off the hot bus may shift dashboards from “real-time” to “near real-time” (seconds to minutes). Ask which decisions truly need sub-second freshness. Most don’t.
• Security and compliance. Mirroring to object storage for replay is cheaper, but you must apply the same access controls, encryption, and retention policies—or better. The compliance team should sign off.

Common failure modes (and how to avoid them)

• Silent consumers. A service reads but drops on the floor. Detect via “effect beacons”: if a consumer writes to a DB/table, emit a periodic heartbeat event with the last applied offset. No beacon = no effect.
• Shadow duplication. Two teams publish the same event with slightly different schemas. Fix with a registry, contracts, and clear ownership. Duplicate publishers are a smell; merge or deprecate.
• Brownout blindness. You ran a brownout during a quiet period and declared victory. Schedule at least one trial in peak hours for Tier 1–2 before you kill.
• Optimizing the wrong bus. Some teams obsess over Kafka while 60% of cost sits in Pub/Sub fanouts or SNS HTTP deliveries. Measure all buses; optimize the biggest offender first.

What to automate next

• Topic scorecards. Nightly job tags topics with a debt score (no owner, low liveness, high fanout, high retries) and opens tickets automatically after thresholds.
• Budget guards. Per-bus budgets that trigger automatic brownouts for Tier 3 streams when spend L7 exceeds forecast by 20%+.
• Schema linting. CI hook that rejects payloads over a size threshold or containing disallowed fields (e.g., raw PII) unless a waiver is present.
• Consolidation bots. For duplicative analytics consumers, propose a single egress to the warehouse with a data contract diff and a migration window.

Where nearshore fits

If your team is at capacity, this is a tidy, bounded engagement for a nearshore pod: 4–6 weeks, 2–3 engineers embedded, one SRE lead, and a product liaison to validate user impact. Expect 6–8 hours overlap with US time zones from Brazil, weekly demos, brownout reports, and a concrete “cut list” with rollback plans. It’s the kind of work that pays for itself in the first quarter—because zombie topics don’t argue with spreadsheets.

The quiet benefit: fewer pages and faster deploys

When you remove waste, you reduce rebalancing storms, backpressure chains, and idempotency edge cases that only show up at 2 a.m. Fewer consumers mean fewer places to leak PII, fewer IAM policies to manage, and a smaller blast radius for inevitable bad deploys. You’re not just saving money; you’re buying clarity.

Key Takeaways

• You’re likely running 20–40% zombie or redundant event traffic. Measure effects first, not dollars.
• In 30 days, with circuit breakers and brownouts, you can safely delete or consolidate a third of your topics.
• Focus on deliveries, payload size, retention, and partitions—those are your biggest levers.
• Keep waste from regrowing with in-band ownership tags, CI brownouts, and two-lane (operational vs. analytical) governance.
• The operational win (fewer incidents, clearer ownership) often beats the invoice reduction.