Your RAG Is Flaky Because Retrieval Isn’t Deterministic: A CTO Playbook

If your RAG answers swing between great and garbage for the same question, stop blaming the model. Nine times out of ten, the culprit is nondeterministic retrieval. That dev.to post making the rounds — “Your RAG Eval Isn’t Flaky. Your Retrieval Is Non-Deterministic.” — is right. Your top‑k isn’t stable, your index isn’t versioned, and your ranking tie‑breakers aren’t deterministic. You are evaluating a moving target.

This post gives you a CTO‑level playbook to fix it. We’ll define where nondeterminism creeps in, what to instrument, and how to lock down your retrieval path so the same query against the same corpus returns the same context — today, tomorrow, and after your next index build.

Where Retrieval Nondeterminism Hides (It’s More Places Than You Think)

RAG pipelines drift because many “small” components are probabilistic, parallel, or time‑dependent. A few common offenders:

• Index churn: Incremental updates shuffle graph neighborhoods (HNSW), recluster centroids (IVF), or change deletion maps. Two minutes apart, same query, different neighbors.
• Approximate search variability: HNSW and IVF search are path‑dependent. ef_search, nprobe, beam width, thread scheduling — all change candidate sets.
• Non‑stable sort: If you sort by score only, ties produce random order across threads/hosts. Your LLM sees a different first 2‑3 chunks and answers differently.
• Tokenizer/segmenter drift: Updating the tokenizer or chunker changes chunk boundaries. The semantic unit that used to rank #3 now spans two chunks and drops out of top‑k.
• Recency boosting via now(): If you boost by “freshness” using wall‑clock time, the same query, hours apart, gets different rankings.
• Floating point noise: Mixing quantization schemes (FP32 index, INT8 queries), different BLAS backends, and thread counts can subtly reorder borderline scores.
• Preprocessing roulette: Casefolding, Unicode normalization, diacritic stripping (critical for Portuguese/Spanish), stopword lists — tiny changes alter embeddings and candidates.

The downstream effect isn’t subtle. In the wild, we’ve seen top‑5 Jaccard similarity between two runs of the same query range from 0.55 to 0.9 depending on index settings. At 0.6, your eval harness will scream “model drift” while the LLM is innocent.

The Goal: Retrieval Stability You Can Measure

You can’t govern what you don’t measure. Define a metric and an SLO:

• Retrieval Stability Index (RSI): For a fixed corpus snapshot, run each query N times and compute the average Jaccard similarity of the top‑k result sets (or position‑aware overlap if you care about order). Target RSI ≥ 0.90 for production Q&A. For compliance‑sensitive flows, push to ≥ 0.95.
• Stability SLO: 95% of production queries must have RSI ≥ 0.9 across 3 independent runs against the same snapshot.

Then build your pipeline so you can actually hit that SLO.

A CTO Playbook to Make Retrieval Deterministic

1) Treat Your Corpus as Immutable, Versioned Artifacts

• Content addressing: Assign every document a content hash (e.g., SHA‑256). Store a manifest mapping doc_id → content_hash, language, and effective_from timestamps.
• Append‑only updates: Updates create a new doc version. Old versions remain queryable until the next snapshot is cut and activated. This prevents mid‑query drift.
• Snapshot discipline: All queries hit a named corpus snapshot (e.g., index_2026‑07‑12T19:00Z). Snapshots are immutable and roll back atomically.

Result: When someone complains “answers changed,” you can correlate to a snapshot change, not a ghost in the machine.

2) Canonical, Deterministic Chunking

• Pin the tokenizer: Choose a tokenizer and version (e.g., cl100k_base vX) and don’t auto‑upgrade. Tokenizer upgrades are index migrations, not hotfixes.
• Fixed stride, fixed policy: Use fixed token windows (e.g., 512 tokens with 64 overlap) or a language‑aware policy — but lock it in. Changing overlap from 64 to 80 is a new corpus version.
• Normalize before chunking: Unicode NFKC, casefold, standard punctuation treatment, and consistent diacritic policy. For Portuguese/Spanish, decide whether you embed accent‑stripped variants or not. Be consistent.

Pro tip: Don’t generate “smart” chunks with an LLM if you care about determinism. If you must, store the chunker prompt, model version, and sampling seed in the manifest and treat changes as migrations.

3) Deterministic Embeddings

• Pin the embedder: Model name, provider, and checksum go into the index manifest. Changing from text‑embedding‑3‑small to text‑embedding‑3‑large is a new corpus snapshot — full stop.
• Normalize consistently: Always L2‑normalize either at write time or query time, not both. Write the policy down.
• Quantization alignment: If the index uses quantization (e.g., INT8), apply the same transform to query vectors. Mixed precision is a common source of tie‑breaker chaos.

Store a digest (e.g., xxh3) of the float vector before quantization alongside the stored vector. It gives you a sanity check that you didn’t silently change preprocessing.

4) ANN Indexes with Seeds, Not Vibes

Approximate nearest neighbor methods are fast because they skip work. They’re reproducible if you take control.

• HNSW (FAISS, Qdrant, Milvus): Fix M and efConstruction, set a build seed if the library supports it, and avoid concurrent mutation. At query time, set efSearch explicitly. Don’t rely on provider “auto” modes.
• IVF (FAISS, Milvus, pgvector ivfflat): Train centroids on the exact same sample, with a fixed seed and parameters (nlist). For pgvector, train lists once per snapshot; don’t let auto‑train rerun opportunistically.
• Exact indexes for small K: If your collection is ≤ 5M vectors and throughput allows, use exact search (e.g., FAISS IndexFlatIP/L2 or Qdrant exact mode) for high‑risk workloads. It’s 4‑10x slower but maximally stable.

Whichever engine you pick, export index parameters into the snapshot manifest. Never “tune in prod” without creating a new snapshot.

5) Stable Ranking and Tie‑Breakers

• Stable sort: Rank by (score DESC, doc_id ASC, chunk_offset ASC). Write it down and enforce it everywhere (service, SDK, SQL).
• No hidden randomness: Disable “autocut,” “sampling,” or “hybrid blending” modes that drop near‑ties. If you must hybridize (BM25 + vector), fix alpha and term weights.
• Deterministic re‑rankers: If you use cross‑encoders to re‑rank, use greedy decoding or set a fixed seed with temperature 0, and pin the model version.

6) Time Is a Parameter, Not a Global

• As‑of queries: If you do recency boosting, provide an as_of timestamp with the query and include it in the cache key. Don’t call now() inside the ranking function.
• Time‑windowed corpora: For fast‑moving data, build daily or hourly shards and route queries by a deterministic policy (e.g., last 30 days + policy docs). Shard boundaries are part of the snapshot.

7) Single‑Writer, Snapshot‑Reader Architecture

• Build off the hot path: Create the index in isolation, validate it, then atomically swap readers to the new snapshot.
• Zero interleaving writes: Don’t allow background merges or compactions that change topologies mid‑day. If your vendor forces compaction, timebox it and treat it as a snapshot rotation.

8) Retrieval Caching You Can Trust

• Cache keys that matter: Query text digest, as_of, top‑k, re‑ranker version, corpus snapshot id. If any differ, it’s a miss.
• Cache the set, not the text: Cache canonical result IDs and positions; rehydrate content from the snapshot. This catches accidental text mutation.

9) Bilingual Reality (Portuguese and Spanish Matter)

Latin American support data isn’t English‑only, and diacritics matter. If you operate in Portuguese or Spanish:

• Pick a normalization strategy: Either index both accent‑preserving and accent‑stripped variants or standardize on one. Mixing ad hoc rules injects silent drift.
• Language‑aware chunking: Sentence segmentation in Portuguese/Spanish (e.g., handling abbreviations like “Sr.”) needs a pinned rule set. Don’t inherit locale rules from a host OS.
• Cross‑lingual embeddings: If you use multilingual embeddings, pin the version, because coverage and subword vocabularies evolve more aggressively than English‑only models.

Instrument It: What to Log and Compare

You can’t improve RSI without visibility. At minimum, log these for every query:

• Query fingerprint: Digest of normalized query text and language code.
• Corpus snapshot id: Name and manifest hash.
• Embedding and index versions: Model id, tokenizer id, ANN parameters.
• Candidate set and scores: Top‑k IDs and raw scores before any re‑ranking.
• Re‑ranker version and final order: The last 10 rows of the ladder are where ties live; you need to see them.

Build a nightly job that replays a 1,000‑query canary set against the active snapshot and calculates RSI. Alert if RSI drops below your SLO. If you’re brave, publish the RSI on your internal status page like an error budget.

Tool‑Specific Guardrails (Without Vendor Worship)

FAISS

• Exact: IndexFlatIP/L2 is deterministic if your BLAS backend and thread count are fixed.
• IVF: Train once per snapshot with fixed seed and training set; fix nlist and nprobe.
• HNSW: Fix M, efConstruction, efSearch. Build the index single‑threaded for maximum reproducibility if you can afford it.

Qdrant

• HNSW knobs: Set m, ef_construct, and ef explicitly. Disable auto‑tuning. Consider exact mode for small collections.
• Quantization: If you enable scalar quantization, apply the same to query vectors and freeze the codec config per snapshot.

Milvus

• Index types: IVF_FLAT, HNSW, IVF_PQ — in all cases fix index params and collection schema version.
• Segment compaction: Schedule compaction during off‑hours and treat as a snapshot swap. Don’t leave it ad hoc.

pgvector

• ivfflat training: Train lists with a fixed seed and a pinned training sample. Set ivfflat.probes explicitly. Re‑train only on snapshot cutover.
• Exact search: For ≤ 1M rows or latency‑insensitive workloads, use exact to eliminate ANN variance.

Weaviate

• Hybrid determinism: Fix alpha for hybrid BM25/vector; avoid autoCut if you care about stable top‑k.
• Module versions: Pin re‑ranker and text2vec module versions. Upgrades equal new snapshots.

Trade‑Offs You Should Acknowledge Up Front

• Latency vs stability: Exact search can be 4–10x slower than ANN. A common compromise is ANN to 50 candidates + deterministic cross‑encoder re‑rank to 10.
• Freshness vs determinism: Append‑only snapshots mean new knowledge lands on a schedule (hourly/daily), not instantly. If “up‑to‑the‑second” matters, route a small freshness shard explicitly and document that those answers can vary.
• Cost of discipline: Pinning models, tokenizers, and index params slows feature work. The payback is faster debugging and credible evals. If you can’t reproduce a bad answer, you can’t fix it.

A Simple Governance Model That Works

Treat your index like a build artifact.

• Snapshot manifest: JSON file with fields for corpus digest, tokenizer version, embedder id, preprocessing rules, index type + params, re‑ranker id, and time policy. Check it into Git.
• Change review: Any change to manifest fields goes through PR review with an RSI diff and an eval diff.
• Promotion gates: Canary RSI ≥ 0.9 and no regression on a domain‑specific QA set. If it fails, it doesn’t ship.

What “Good” Looks Like in Numbers

On typical SaaS support corpora (50k–500k chunks):

• RSI: 0.93–0.97 across three runs with HNSW ef_search fixed at 128.
• Latency: 35–70 ms ANN search p95 on CPU for top‑50 candidates; 90–140 ms with exact search on 500k vectors; add 30–60 ms for cross‑encoder re‑rank p95 on a T4/A10.
• Update cadence: Hourly snapshots for changelogs; daily for docs; weekly for policy/legal.

Your numbers will differ, but if your RSI is stuck at 0.7, you’re not unlucky — your pipeline is.

Why This Matters to You as a CTO

Deterministic retrieval lets you do real engineering on top:

• Trustworthy evals: You can attribute gains to changes instead of noise. A +3 point lift means something.
• Lower support risk: When answers swing, support tickets spike. Stability in top‑k cuts variance in answer quality, even when the LLM is stochastic.
• Regulatory posture: You can show auditors exactly which corpus and index version produced an answer. That’s the difference between “we think” and “we know.”
• Nearshore leverage: A disciplined pipeline is portable. You can replicate the whole stack with a Brazilian nearshore pod and expect the same behavior — same manifest, same RSI, same SLOs.

Getting Started This Month

1. Week 1: Add snapshot IDs to your retrieval service and log top‑k IDs + scores. Build the RSI job on last week’s traffic.
2. Week 2: Freeze tokenizer, embedder, and chunking. Cut your first immutable snapshot. Switch queries to reference it explicitly.
3. Week 3: Fix ANN params and tie‑breakers. Disable any “auto” tuning. Re‑cut snapshot and measure RSI.
4. Week 4: Introduce a cross‑encoder re‑ranker with deterministic settings. Set your RSI SLO and wire alerts.

After that, move snapshot builds off the hot path, add promotion gates, and stop taking “model drift” heat for a retrieval problem you can control.

Key Takeaways

• RAG flakiness is usually retrieval nondeterminism, not LLM mood swings.
• Define and track an RSI metric; target ≥ 0.9 stability for production Q&A.
• Version everything: corpus, tokenizer, embedder, index type/params, re‑ranker, and time policy.
• Use deterministic chunking and stable sort tie‑breakers; treat tokenizer upgrades as migrations.
• Fix ANN parameters and seeds; consider exact search for high‑risk flows.
• Make time explicit with as_of queries; stop calling now() in ranking.
• Adopt single‑writer, snapshot‑reader architecture and cache canonical IDs, not text.
• For Portuguese/Spanish corpora, pin normalization and segmentation rules to avoid silent drift.
• Govern with a manifest and promotion gates; no snapshot ships without an RSI/eval diff.