OpenTelemetry Isn’t Going Well at Your Company. Fix It in 90 Days.

Your observability bill doubled last quarter and nobody can find the trace for the one request you actually care about. Meanwhile, engineers still grep logs while billions of spans flow into a vendor you don’t want to be married to. If you’ve read the recent "OTel isn’t going well" threads, you’re not alone. The problem isn’t OpenTelemetry. It’s how you adopted it.

This is a 90‑day, SLO‑driven rescue plan that cuts cost by 50–70%, stops cardinality explosions, and makes traces measurably useful in incident response. It works for monoliths, microservices, serverless, and edge functions. It’s battle-tested across US SaaS at 100–10,000 RPS and polyglot stacks.

Why Your OTel Rollout Went Sideways

• You shipped everything, raw. Head-sampling at 1% and pushing spans straight to a vendor is how you get the illusion of adoption and the reality of pain. Tail events don’t make it. Hot paths are underrepresented. Your “baseline” is noise.
• Cardinality detonated your time series. Someone added user_id, email, or request_path as labels. Ten labels with 100 values each is 10^10 theoretical series. Your metrics backend melted, so you turned off exemplars or downsampled into uselessness.
• Context propagation is broken in the places that matter. HTTP is fine. Your gRPC gateway, message bus, cron, and serverless triggers are not. Parent-child links get cut; baggage carries PII; exemplars have no home.
• No semantic governance. service.name drifts across languages. Teams invent their own attribute keys. Your service graph is a shrug emoji.
• Logs are a landfill, not a pipeline. Everything goes hot; nothing is structured; request_id doesn’t bridge logs, metrics, and traces. On-call relies on tribal knowledge and luck.

The North Stars (Decide These First)

OpenTelemetry is just plumbing. Decide why you’re plumbing it.

1. Incident response SLOs: You want p50 MTTR under 30 minutes and first useful signal in under 5. Every decision in this plan is scored against those SLOs.
2. Data budget: Cap to a number you can defend to your CFO. As a rule of thumb: 0.1–0.3% of revenue or $20–$50 per host per month. If you’re paying 2–3x that, you are subsidizing entropy.
3. Golden paths: Pick 2–3 journeys that pay the bills. These get richer traces, 100% error retention, and exemplars. Everything else gets probabilistic treatment.

Architecture That Doesn’t Fight You

There are many correct choices. These work with today’s reality:

• Always put an OpenTelemetry Collector in the middle. Collectors are your control plane: sampling, attribute scrubbing, cardinality guards, and routing. Run them as a DaemonSet in k8s and as a sidecar/VM at the edge. See OpenTelemetry Collector.
• Tail-based over head-based sampling for prod user traffic. Keep 100% of errors and slow traces. Sample normal traces down to a budget. Use tail_sampling with per-rule budgets and a global circuit breaker.
• Exemplars wire metrics to traces. Without exemplars you just built two islands. Use span-to-metrics to populate RED (Requests, Errors, Duration) and attach exemplars for top charts.
• Logs are a two-tier pipeline. Hot (7–14 days) structured, request-scoped logs; Cold (90–365 days) object storage with cheap search-on-read. Don’t pay hot prices for compliance retention.
• Choose storage you can afford to keep. Vendor or self-host is a business choice. If you self-host, a pragmatic stack is Grafana Tempo for traces, Prometheus/Mimir for metrics, and ClickHouse or Loki for logs. Storage cost: ~20–30% of managed vendors at scale, with ops you must own.

Concrete Numbers (So We’re Not Hand-Waving)

Let’s say you run 10,000 RPS across ten services. Median trace is 10 spans. A minimal span averages 400–800 bytes when batched. At 100% sampling, you’re writing 10,000 × 10 × 600 B ≈ 60 MB/s, or ~5 TB/day of traces alone. That’s a budget crater.

• Tail-sample to 5% (keep 100% of 5xx + p95 latency): ~250 GB/day traces.
• Drop high-cardinality attributes you don’t need (user_id, request_path with params): 30–50% reduction in span size.
• Batching + compression on the Collector typically yields another 20–30% cut on the wire.

Net: from 5 TB/day to ~100–150 GB/day without losing what on-call needs. That’s a 97% reduction in stored volume with better incident signal.

The 90‑Day Plan

Days 0–30: Stop the Bleeding and Set Guardrails

1. Inventory and map. List every workload emitting telemetry by runtime, version, library, and destination. Capture what’s actually on: metrics, traces, logs, profilers. Expect surprises.
2. Insert a centralized Collector layer. Route all telemetry to regional Collectors. Enable processors: batch, memory_limiter, attributes (PII scrubbing), k8sattributes (if on k8s), and transform for semantic fixes. Only the Collectors talk to vendors or storage.
3. Standardize service identity. Enforce service.namespace, service.name, and service.version across languages. Break builds that don’t comply. Without identity, nothing else matters.
4. Kill baggage-by-default. Baggage leaks PII and explodes cardinality. Disable it globally. Whitelist minimal, non-PII keys if you truly need it.
5. Attribute allowlist, not denylist. For spans and metrics, ship only approved keys. Start with: http.method, http.route (templated), http.status_code, db.system, db.operation, messaging.system, net.peer.name (quantized). Drop anything user-scoped.
6. Cap head sampling to something sane. Until tail-based is live, run head sampling at 10–20% on high-traffic services and 100% on low-traffic or admin APIs. Keep 100% on canaries and internal staging.
7. Stand up a Golden Path dashboard. Instrument 2–3 money paths end-to-end. Build RED metrics and attach exemplars. Add a “Find Trace” panel that works in 2 clicks during incidents.

Days 31–60: Make Traces Answer Questions

1. Turn on tail-based sampling. In the Collector, add rules: keep if status_code ≥ 500; keep if latency ≥ p95; keep if release.version is new; else probabilistic 5%. Set per-rule and global budgets with a backpressure policy (shed normal traffic first).
2. Propagate context through the weird parts. Audit and fix W3C TraceContext across gRPC, message buses, and serverless triggers. For messaging, write traceparent into message headers and create links between producer and consumer spans. For cron/batch, start new traces and carry correlation IDs into logs.
3. Exemplars everywhere that matters. Configure span-to-metrics so latency, error, and throughput charts attach trace IDs. Ensure your metrics backend supports querying exemplars by click.
4. Establish cardinality SLOs. Enforce: no metric with a label cardinality over 1,000 unique values/day; no new label keys without a design review; no high-card keys on hot metrics. Add CI checks that query your backend for series churn and fail PRs that exceed thresholds.
5. Restructure logs. Make request_id a first-class field. Log JSON lines with a fixed schema. Route hot logs (7–14 days) to fast search; everything else ships to object storage with a query layer. Add log drop rules for chatty DEBUG and health checks at the Collector or Vector/Fluentd level.
6. Build the first incident runbook powered by OTel. A checklist that starts in a metrics dashboard, jumps to exemplars, opens the right trace, and fetches correlated logs by request_id. Time it. Aim for under 5 minutes to “first useful signal.”

Days 61–90: Lock Costs, Prove Outcomes, and Automate

1. Put cost guardrails in code. Set per-service budgets for traces (e.g., max 2 GB/day), metric series (max N active series), and logs (GB/day). Wire autoscaling or backpressure in the Collector to shed non-critical data when budgets are hit.
2. Run a game day. Break something benign (increase latency on a leaf service). Time from alert to root cause with and without exemplars. Expect a 2–3× faster diagnosis when configured right. If you don’t see it, your sampling or exemplars need tuning.
3. Codify semantic conventions. Document which OTel semantic conventions you follow. Provide language-specific snippets and linters. Add a pre-merge checklist.
4. Right-size your retention. Traces: 7 days hot, 30 days warm if you do RCA frequently. Metrics: 18 months with downsampling. Logs: 7–14 days hot, 90–365 days cold. If you can’t justify the extra day, don’t pay for it.
5. Decide vendor vs self-host for the next year. If you can’t run storage reliably, buy it. If your volume is predictable and >200 GB/day, self-hosting Tempo/Mimir/ClickHouse can be 20–30% of vendor costs. The trade-off is SRE headcount. A focused nearshore pod can bridge the gap at 20–30% lower labor cost than US rates with 6–8 hours overlap time zones.
6. Publish the Observability Contract. One page: SLOs, budgets, approved attributes, sampling rules, service identity rules, and runbooks. Treat changes as ADRs.

Trade-offs You Should Embrace (Not Avoid)

• Sampling is not a sin. You cannot afford 100% traces at scale. You don’t need them. What on-call needs is “keep the bad and the weird.” Tail-based sampling with SLO alignment does exactly that.
• Logs are not a replacement for traces, and vice versa. Traces tell you timing and causality; logs tell you facts. Correlate them with request_id and exemplars. If you can’t click from a spike to the trace, you didn’t finish the job.
• Don’t chase perfect semantic coverage. 80% consistent conventions now beats 100% theoretical compliance never. Freeze keys, then iterate.
• Self-hosted observability is not free. Yes, storage cost drops dramatically. No, it won’t run itself. Budget 0.25–0.5 FTE SRE per 1 TB/day of ingest for care and feeding if you want SLOs you can bet an on-call week on.

What Good Looks Like by Day 90

• On-call starts in one place. A single RED dashboard with exemplars that you can click into traces and then into correlated logs without jumping tools or tabs.
• Costs are inside a guardrail, not a hope. You know daily GB by telemetry type, how sampling changes it, and when budgets will trigger backpressure.
• Trace coverage is intentional. 100% of 5xx kept; p95 slow traces kept; normal requests sampled 2–10% depending on service criticality. You can answer “what happened” and “why now.”
• Cardinality is tamed. No metric with >1,000 unique label values/day. No PII in attributes. High-cardinality data lives in logs, not on hot metrics.
• Propagation works across the boundaries that used to fail. You can follow a user journey across HTTP → queue → worker → serverless without the trace falling apart.

Common Pitfalls (And How to Dodge Them)

• “We’ll fix sampling later.” Later never comes. Put tail-based sampling into the Collector by Day 60 with explicit rules and budgets.
• Letting per-team autonomy override service identity. You can decentralize a lot. Not this. Break builds that don’t set service.name correctly.
• Mixing raw, user-scoped labels into hot metrics. If you need per-user analytics, that’s a data warehouse or ClickHouse problem. Keep per-user anything out of Prometheus/Mimir.
• Chasing 100% auto-instrumentation. Auto helps, but business spans on golden paths require manual work. Give teams a 1–2 hour recipe for “add a span around this use case.”
• Shipping dev/test noise to prod backends. Route non-prod to a separate pipeline. Better: drop most of it. Observability should mirror production reality, not unit tests.

A Note on Compliance and Privacy

OTel is a PII magnet if you’re careless. Treat attribute governance as a privacy control:

• Drop user-identifiers at the edge. If a use case truly requires them, hash or bucket, and document the retention and access policy.
• Scrub secrets in the Collector. Add processors that redact bearer tokens, API keys, and emails before storage. Never rely on app teams to “remember.”
• Audit. Monthly queries for PII patterns in attributes and logs. If you can’t prove scrubbing worked, it didn’t.

Tooling Checklist You Can Use Tomorrow

• Collector processors: batch, memory_limiter, tail_sampling, attributes (drop/redact), transform (normalize attributes), k8sattributes, spanmetrics, probabilistic_sampler (for low-traffic services), resourcedetection.
• Dashboards: Per-service RED; global service graph; per-journey SLOs with error budgets; “Find Trace” with filters for service.name, http.route, status_code, and release.version.
• Runbooks: 1) Start at SLO panel. 2) Click exemplar. 3) Inspect trace critical path. 4) Jump to logs for request_id. 5) Capture timeline. 6) Decide rollback vs mitigate.

OpenTelemetry is the right bet. The early pain isn’t a verdict; it’s a signal that you installed pipes without deciding what you wanted to flow through them. Give your teams a north star, put the Collector in charge, and budget by SLOs, not vibes. Ninety days from now you’ll have lower costs, faster incident response, and a system on-call engineers actually use.

Key Takeaways

• Decide your SLOs and budgets first; wire every OTel choice to them.
• Put the OTel Collector in the middle and move to tail-based sampling by Day 60.
• Enforce identity and semantics: service.name, service.namespace, service.version are non-negotiable.
• Use exemplars to bridge metrics-to-traces; make “Find Trace” a 2-click action.
• Cap cardinality: allowlist attributes, kill baggage-by-default, and add CI checks.
• Restructure logs into hot structured + cold cheap; correlate via request_id.
• Lock costs with per-service data budgets and Collector backpressure.
• Expect 50–70% cost reductions and 2–3× faster diagnosis once tuned.