Stop Letting NaNs Into Production: A CTO’s Numerical Hygiene Playbook

NaNs are not dramatic. They don’t blue‑screen your server or blow up the logs. They quietly slip through JSON, poison your rankings, and make your A/B stats lie. In 2026—when you’re mixing FP8/BF16 inference, quantized embeddings, and third‑party model calls—NaNs are the bug class that will degrade your product without anyone noticing until revenue moves.

Hacker News recently resurfaced a great reminder with “Two Case Studies of NaN.” The gist: NaNs are contagious. They propagate across computations and rarely surface as an exception. If you don’t design for numerical hygiene, they’ll slip through your AI, your services, and your warehouse. Let’s fix that.

Why you almost never see NaNs (even when they’re there)

• JSON hides them. The JSON spec forbids NaN and Infinity. Many encoders serialize NaN to null; some emit the literal NaN and pray; many decoders reject the payload. Either way, you lose signal or silently sanitize.
• Most languages don’t trap on IEEE‑754 exceptions by default. Division by zero, invalid sqrt, catastrophic exp—none of them raise unless you ask. On GPUs, you can’t trap at all in production runtimes.
• Your metrics pipeline flattens them. Charting layers implicitly coerce NaN to zero or drop the point. Your pretty dashboard is lying to you.

Meanwhile, a single NaN in a 768‑dimensional embedding will often make cosine similarity NaN, which many ANN libraries treat as worst‑possible score or skip entirely. That’s not a crash; that’s a subtle recall cliff. We’ve seen teams claw back 5–10% retrieval recall by eliminating a <0.1% NaN rate in embeddings.

Where NaNs come from in modern stacks

• Mixed precision inference/training. FP16/FP8 saturate fast. Softmax on large logits overflows; layer norm with near‑zero variance divides by ~0; cast back to FP32 yields Inf or NaN.
• Normalization and statistics. Zero‑variance batches, log of non‑positive values, sqrt of negative numbers, or sum of empty arrays.
• Third‑party API results. Model providers sometimes return NaN probabilities or scores on edge inputs. Your SDK likely coerces or drops them without telling you.
• Parser and ETL drift. CSVs with "" mapped to float; NumPy and Go can both produce NaN on bad parses if you’re permissive. Downstream features inherit the poison.
• Fast‑math compilers. -ffast-math and equivalent flags promise speed by relaxing IEEE guarantees. They also permit algebraic transforms that amplify invalid operations.
• Quantization/dequantization errors. Asymmetric scales near zero plus aggressive clipping can make downstream denominators practically zero.

The real cost of one NaN in the wrong place

• Rankers and retrieval. One NaN inside a vector makes cosine similarity NaN; the candidate might drop out or rank last. If 0.05% of queries carry a NaN vector, that’s 500 bad results per million queries. At 3% CTR lift from correct top‑3, you’re giving up real money.
• Tool calling for LLMs. A NaN in a function score can short‑circuit the selector; your agent “decides” to skip a tool, or the planner degenerates. These aren’t visible as 5xx—they look like the model got “dumb.”
• Databases and analytics. In Postgres, NaN is a valid float value. It sorts oddly and can break dedup/aggregations in surprising ways. If you’re not filtering with isfinite(x), you will ship nonsense dashboards.
• Pricing and bids. Any scoring model that feeds a bid multiplier or discount engine will propagate NaN to a default; if your default is lenient, congratulations—you’re overpaying or overd iscounting.

A CTO playbook: detect, contain, and then eliminate

Don’t start by rewriting kernels. Start by turning NaNs from ghosts into events your org can see and route around. Here’s a layered plan with concrete, low‑overhead steps.

Layer 1: Wire and storage hygiene

• Use binary protocols that preserve float bits. Prefer Protobuf or Arrow Flight for AI data planes. JSON makes NaNs invisible or someone’s problem. If you must use JSON to the browser, down‑cast or encode floats as base64 when precision matters and validate on the server boundaries.
• gRPC/HTTP interceptors that scan payloads. For vectors and score arrays, a single pass is cheap. A modern CPU core can stream 20–40 GB/s; scanning a 3 KB embedding at 1,000 RPS is ~3 MB/s—0.01 core. On find, increment a counter, attach model/version tags, and either sanitize (nan_to_num) or fail fast by route.
• Database constraints. In Postgres, add CHECK (isfinite(x)) on float columns that must never store NaN/Inf. For arrays, enforce in triggers or at write‑path services. Rejecting bad rows early is cheaper than repairing warehouses later.

Layer 2: Model‑level stability

• Stable softmax and normalization. Always subtract max(logits) before exp; accumulate in FP32 even when weights are FP16/FP8; add eps (e.g., 1e‑5) to denominators in layer/batch norm.
• Clamp and sanitize at model outputs. Use functions like torch.nan_to_num(x, nan=0.0, posinf=1e9, neginf=-1e9) at the final stage for embeddings and scores. Clamping at the boundary beats propagating NaN into ranking systems.
• Disable unsafe fast‑math in hot paths. Compilers and graph engines offer flags that sacrifice NaN safety for speed. Keep them off for numerically sensitive ops; the speed gain is often 0–3% and not worth the risk.
• Quantization with guardrails. Calibrate with real tail inputs; enable per‑channel scales; verify that dequant produces finite values under extreme inputs. Run unit tests with adversarial ranges.

Layer 3: Runtime guards in your services

• Language‑level checks. In Python/NumPy: np.isfinite asserts at boundaries; set numpy.seterr(all='raise') in CI and tests to force failures. In Go: math.IsNaN, math.IsInf checks at deserialization and before persistence. In Java/Kotlin: Double.isFinite.
• ONNX Runtime/TensorRT debug gates (non‑prod first). Enable model‑level NaN propagation checks in staging runs to find the layer where NaNs are born. For PyTorch training, torch.autograd.set_detect_anomaly(True) in debug builds will catch the source; for inference, wrap critical ops and assert finite tensors.
• Contracts, not comments. Add schema‑level contracts to Protobuf: dedicate a message type for Embedding that is validated at construction; do not pass raw repeated float32 around your fleet.

Layer 4: Observability and policy

• Define a NaN SLO. Example: “Fewer than 1e‑7 non‑finite values per output element, per model, per day.” Track as a counter with labels: model, version, layer, feature, route.
• Fail‑closed vs sanitize—decide explicitly. For anything touching money, identity, or security policies, fail the request and surface a retriable error. For ranking/retrieval, sanitize to safe defaults and record a high‑cardinality sample for replay.
• Automated fallbacks. If NaN rate exceeds threshold for a model version or route, circuit‑break to a slower but stable path (e.g., CPU inference or last‑known‑good version) until a human reviews.
• Store tail samples for 30 days. Keep 100–1,000 replayable exemplars per model/version with exact inputs and seeds to reproduce and fix. They are small (<100 KB each) and priceless during incidents.

Concrete implementation patterns (by stack)

Python/PyTorch services

• Wrap all model outputs with a small utility that asserts finiteness, clamps to ranges, and emits a metric. Keep the per‑request overhead to microseconds by vectorizing checks.
• In preprocessing, convert user inputs to float with strict parsing. Reject or impute; never let an invalid parse become NaN silently.
• In training and eval, run a nightly “numerical fuzz” job: sample extreme inputs, add small noise, sweep batch sizes; fail the job if any tensor reports non‑finite values post‑layer norm/softmax/attention.

Go microservices

• At API boundaries, scan float slices with math.IsNaN/math.IsInf. For embeddings: pre‑allocate and branchless‑scan with SIMD where available; the bottleneck is memory bandwidth, not CPU.
• In gRPC interceptors, add a single pass over known float fields. On detection, annotate context with a reason and route to a sanitizer fallback.
• When marshalling to JSON (browser clients), explicitly map NaN/Inf to null or safe sentinel values and set a header flag. Downstream services must treat “sanitized” as degraded QoS.

Postgres, warehouses, and analytics

• Use CHECK (isfinite(col)) on float columns. For arrays, consider a write‑path function that rejects any non‑finite element.
• Queries: always filter metrics with isfinite(value) and decide how to treat non‑finite—exclude from aggregations or bucket separately.
• If you must store raw model outputs with NaNs for forensics, do it in a quarantine table with retention and no joins to core analytics.

ONNX Runtime and TensorRT

• Prefer FP16 with FP32 accumulation on sensitive ops (softmax, layer norm). Validate your engine by sweeping input ranges around your real 99.9th percentile.
• Add a post‑engine plugin that scans outputs for finiteness and emits a structured log on violations. Keep it off in the hot path for ultra‑latency SLAs; keep it on for canaries and batch jobs.
• Avoid fusing kernels that hide the source of NaNs until you’ve validated numerics; then re‑enable fusions selectively.

Governance: make numerical hygiene a contract, not a best practice

• Numerical acceptance tests in CI. Ship synthetic test vectors that exercise zeros, denormals, extreme logits, and empty tensors. Assert “no non‑finite anywhere.” This takes minutes and prevents weeks of incident chasing.
• Property‑based testing for math helpers. For any function doing log/exp/division/sqrt, generate adversarial inputs and assert finiteness and monotonicity properties.
• Model promotion checklists. Before promotion, require: (a) finiteness scan over 1M representative outputs; (b) range histograms for key tensors; (c) plan for sanitize vs fail‑closed per call site.
• Ownership maps. Name a single team that owns “numerical SLOs” across models and services. If everyone owns it, no one does.

Performance trade‑offs and how to keep overhead tiny

Yes, checking floats costs something. But it’s cheap if you place it right and vectorize.

• Wire‑scan cost. Scanning a 3 KB embedding at 1,000 RPS is ~3 MB/s. A single core streams 20–40 GB/s. That’s ~0.01 core for wire‑level guarantees—rounding error.
• Model‑level clamping. torch.nan_to_num is memory‑bandwidth bound; on a 768‑dim tensor, it’s single‑digit microseconds on CPU/GPU. Do it once at the leaf.
• Metrics cardinality. NaN counters are high‑cardinality by nature. Cap labels to model/version/route/layer. Sample violators to logs for details.
• Fail‑fast vs sanitize. This is a product decision. For money/security, fail. For ranking, sanitize to stable defaults and degrade gracefully. Document which applies where.

Your 90‑minute, 2‑day, and 30‑day plan

In the next 90 minutes

• Add a single finiteness scan to your top embedding/ranking response and emit a metric (model, version, route). Light up a dashboard panel that shows daily NaN rate.
• In Postgres, add isfinite filters to your most‑important analytics queries. You will be surprised.

In the next 2 days

• Introduce CHECK (isfinite(x)) on at least one high‑value float column and fix the first few violations at the source.
• Wrap model outputs with nan_to_num or equivalent and set sane clamps. Re‑run A/Bs on retrieval or ranking—watch recall and CTR move.
• Add a gRPC/HTTP interceptor that scans known float payloads and counts violations. Route “bad” responses through a sanitizer.

In the next 30 days

• Codify a NaN SLO, wire it into on‑call, and give the owning team a budget to fix root causes.
• Stand up a nightly numerical fuzz job against each promoted model with adversarial inputs and batch sizes. Promotion blocks if any non‑finite values appear.
• Decide, document, and implement fail‑closed vs sanitize policies route‑by‑route. Audit call sites touching pricing, identity, and security.

Why this matters now

AI stacks in 2026 are numerically aggressive by design: smaller dtypes, tighter latency budgets, heavier vector traffic. NaNs are the predictable byproduct—and they’re the easiest bug class to eradicate with fast wins. The overhead is negligible; the payoff is immediate: cleaner rankings, trustworthy analytics, fewer “the model got dumber” tickets.

In a world where regulators are eyeing safety and reliability, a NaN hygiene program is also a governance story. You can look your board—and your customers—in the eye and say: we don’t just have experiments; we have numerical contracts. That’s the bar now.

Key Takeaways

• NaNs rarely crash your systems; they silently corrupt them—especially in retrieval, ranking, and analytics.
• JSON hides NaNs; use Protobuf/Arrow for AI data planes and scan payloads at boundaries.
• Add Postgres CHECK (isfinite(x)) and range clamps at model outputs for immediate wins.
• Define a NaN SLO, decide where to fail‑closed vs sanitize, and automate fallbacks when rates spike.
• The performance cost of finiteness scans is tiny (≈0.01 core per 1,000 RPS of 3 KB vectors); the ROI is large.