Move Your Embeddings to the Client: A CTO Playbook for Private, Fast Search

You are still POSTing user text to a third‑party embeddings API. It worked in 2024. In 2026, it’s lazy, slow, expensive, and increasingly non‑compliant. The community just shipped a 7 MB embedding model that runs in the browser (WASM). Small, local models are gaining traction exactly where the network is unreliable. The writing is on the wall: a big slice of semantic search and RAG preprocessing belongs on the client.

This isn’t a purity play. It’s a pragmatic redesign that reduces inference spend, tightens privacy posture, and drills latency down to “feels instant.” If you’re leading an engineering team at a startup or scale‑up, you need a clear decision framework for when to move embeddings to the edge and how to roll it out safely.

What Changed Since 2024

• Models shrank. A new wave of tiny embedding models (for example, a ~7 MB WASM model highlighted on Hacker News) now runs fully offline in the browser. You no longer need a 200–500 MB download to get acceptable vectors.
• Runtime maturity. WebAssembly SIMD, threads, and WebGPU lifted ceilings. On mid‑range laptops, embedding a paragraph no longer feels like compiling Chromium.
• Offline is normal again. You’ve seen it with FOSS offline maps and the broader “small models in unreliable networks” trend. Users expect core features to work with the Wi‑Fi off.
• Privacy heat rose. Device IDs, silent telemetry, and cross‑app tracking are in the headlines. Shipping raw text to embedding endpoints is harder to justify to counsel, auditors, and enterprise customers.

When Client‑Side Embeddings Make Sense

Not all retrieval is created equal. Move the right 60% to the client and leave the rest alone.

Good candidates

• User‑scoped search and RAG. Notes, tickets, docs, chats, CRM snippets, code snippets. Think 1–50k items per user, not a global corpus.
• Zero‑knowledge or end‑to‑end encrypted data. If your servers shouldn’t read the content, don’t ask them to embed it.
• Privacy‑sensitive domains. Health, finance, HR. Even if you have legal basis, minimizing exposure is powerful in enterprise sales.
• Latency‑sensitive UX. Type‑ahead, “as you paste” suggestions, quick‑open search. Sub‑100 ms wins are obvious to end users.
• Intermittent connectivity. Field work, public transit, air travel, emerging markets. “Works offline” is worth NPS points.

Bad candidates (keep server‑side)

• Global search across users/tenants. Aggregations, popularity signals, and abuse defenses generally need centralization.
• Strict centralized audit requirements. If every query and index update must be centrally logged and reproducible, client‑only flows complicate life.
• Very large corpora per user. If a single user routinely searches 1M+ items, you will hit browser storage and memory walls.

The Cost and Latency Math (With Round Numbers)

CTOs care about dollars and milliseconds. Here is the quick mental model:

• Embedding API cost. Major providers price embeddings by tokens processed. Even at low single‑digit dollars per million tokens, indexing user content adds up. A single kilobyte averages a few hundred tokens. Millions of items across your base equals five to six figures annually, and that’s before re‑embedding for model upgrades.
• Round‑trip latency. A network call plus provider inference is typically 200–800 ms in the happy path; worse on mobile. Local embeddings often execute in the 10–70 ms range for a short paragraph on a 2023–2025 laptop, longer on low‑end phones. But it’s predictable and works offline.
• Storage footprint on device. A 384‑dimensional vector at int8 is ~384 bytes. For 10k items, raw vectors are ~3.8 MB. Add an approximate‑nearest‑neighbor index (HNSW/IVF) and some metadata and you are still in the tens of megabytes, which is fine for modern browsers and mobile apps if you manage cache and quotas.
• Server egress. Stop shipping user text to embed and you cut egress on every ingestion event. If your clients are outside your region, this is non‑trivial.

The win is not just unit economics. It’s that you can afford to embed on every keystroke, every paste, every edit, because you removed the network tax.

A Decision Framework You Can Apply This Quarter

1) Scope the content and quality target

• Corpus size per user: 1k, 10k, 100k, 1M items. Browser and mobile are happy up to tens of thousands if you’re careful. Six figures is pushing it without sharding.
• Text length: short titles vs. multi‑paragraph bodies. Small local models do better on short to medium text. For long docs, embed chunks client‑side and stitch server‑side if needed.
• Quality bar: define recall@K and MRR targets on a representative test set. If server vectors give recall@10 of 0.90, accept 0.85 locally with client‑side re‑rank? Decide now.

2) Select a model and runtime by device class

• Browsers: start with a tiny WASM embedding model (the 7 MB class) for universal coverage. Where WebGPU is available, optionally offer a larger 100–200 MB model behind a “High Accuracy” toggle for power users.
• Android: target NNAPI with a quantized model via TensorFlow Lite/ONNX; Android 17’s on‑device AI stack is finally reliable for this workload.
• iOS: ship a Core ML version and run on the Neural Engine when available; fall back to CPU for background indexing.

Lock the vector dimension and normalization scheme. Mismatched dimensions or cosine vs. dot errors will burn a sprint later.

3) Choose an indexing strategy

• Small corpora (<10k items): brute‑force cosine L2 on optimized arrays is fine. Use Web Workers and SIMD.
• Medium corpora (10k–200k): HNSW (hierarchical graphs) or IVF (inverted file) built client‑side or shipped as a prebuilt shard. hnswlib has WASM ports; measure memory overhead.
• Hybrid: server performs coarse filtering (by metadata/time), client does exact or graph‑based rerank on a few thousand candidates. This preserves privacy while keeping queries snappy.

4) Add privacy and governance upfront

• Don’t phone home raw text. Telemetry should be timing and counters, not content. If you need search analytics, aggregate on‑device and report differentially private summaries.
• Treat vectors as sensitive. Vectors can leak semantics. Encrypt at rest on device where possible; namespaced per tenant. If you sync vectors, encrypt them end‑to‑end or rotate on model upgrade.
• Model versioning. Tag every vector with a model_id and normalize_version. Never mix vectors across incompatible models. Roll forward with background re‑embedding.

5) Ship fallbacks by capability

• Capability detect: on boot, measure available compute, memory, and storage quota. Place the user into one of three bands: Local‑Fast, Local‑Okay, or Remote.
• Remote fallback: keep your server embedding path alive for low‑end devices and locked‑down browsers. Make it configurable per tenant.
• Graceful degradation: if a device runs hot or low on battery, back off background indexing and rely on metadata filters plus last‑known vectors.

Three Reference Architectures

1) Client‑only, sync metadata

Use when: zero‑knowledge positioning, personal data vaults, high privacy requirements.

• All text stays local. Client computes vectors, builds a local ANN index, and persists to IndexedDB/SQLite (mobile).
• Server syncs only item IDs, timestamps, and optional encrypted blobs. Search runs entirely on device.
• Trade‑off: no cross‑device search without encrypted vector sync; higher initial download for model.

2) Hybrid rerank

Use when: medium corpora, multi‑device UX, compliance needs a server trail.

• Client embeds queries and items. Server stores encrypted vectors for sync and coarse candidate selection per query (e.g., by project/time/owner).
• Client downloads top N candidates’ vectors, performs exact cosine rerank, and renders results. Raw text need not leave the device.
• Trade‑off: you must operate a vector sync service and encrypt metadata well.

3) Server coarse + client embeddings

Use when: you have existing server search infra you cannot replace this quarter.

• Server answers metadata‑only queries quickly (keyword, filters). Client embeds just the user’s context and query, reranking the server’s top N hits locally.
• Trade‑off: quality depends on server recall; meta leakage risk remains if filters are too coarse.

Quality Pitfalls and How to Avoid Them

• Domain mismatch. Tiny general‑purpose models may underperform in code, medical, or legal text. Consider lightweight domain adaptation or a curated synonym dictionary client‑side.
• Chunking strategy. For long docs, chunk size, stride, and title‑boosting matter more than squeezing another 1% from the model. Standardize and test.
• Index drift. Background edits must re‑embed and update the ANN graph atomically. Use a write‑ahead log and versioned index segments.
• RAG hallucinations. Don’t let local retrieval silently feed junk to your generator. Attach provenance and enforce answerable‑by‑context checks before model calls.

Storage and Performance Engineering in the Browser

• Where to store: IndexedDB for structured blobs, OPFS (Origin Private File System) for large binary shards in modern Chromium‑based browsers, SQLite via WASM for predictable performance inside Electron/Tauri or mobile.
• Compression: int8 vectors cut memory 4x vs. float32 with minor recall impact. Product quantization can bring another 2–4x if you can tolerate small accuracy loss.
• ANN memory: HNSW loads edges into memory; plan for 2–3x vector size overhead when fully resident. Limit concurrency and warm subsets by namespace.
• Workers: always move embedding and ANN work to Web Workers. Keep the UI thread under 16 ms frame budget.
• Thermals: gate background indexing by battery and CPU temperature where available. On mobile, index when charging or recently active.

Compliance, Contracts, and Sales Enablement

If you sell to enterprises, client‑side embeddings are a gift to your security questionnaire flow.

• Data residency: “No user content leaves the device for search” preempts half the residency debate.
• DPIAs and DSRs: minimizing processing of personal data server‑side reduces your exposure in privacy impact assessments and data subject requests.
• Vendor risk: you remove a third‑party embeddings provider from the “sub‑processor” list for this feature. That unblocks procurement faster.

Rollout Plan: 30 / 60 / 90 Days

Day 0–30: Prove it

• Pick one user‑scoped feature (e.g., workspace search) and implement a client‑only path behind a feature flag.
• Integrate a tiny WASM model and measure cold start, per‑paragraph embed time, and recall@10 vs. your server baseline on a 1k‑query test set.
• Define capability bands and implement a remote kill switch and server fallback.
>

Day 31–60: Productionize

• Add local ANN for corpora over 10k items. Persist to OPFS/IndexedDB with versioned segments.
• Instrument telemetry without content: timings, failure codes, model version, device band. Route to a separate privacy‑governed sink.
• Write the security note: vectors are sensitive, model versions are pinned, encryption at rest where supported, and E2EE if you sync vectors.

Day 61–90: Scale it

• Roll out to 25–50% of users on capable devices. Watch crash rates, cold starts, and recall deltas per tenant.
• Offer an optional “High Accuracy” model for WebGPU‑capable desktops. Keep the small model as default.
• Plan your next migration: client‑side re‑rank for your global search, or on‑device summarization for top hits.

Trade‑Offs You Should Acknowledge in the Room

• Accuracy: A 7 MB model won’t match a 500 MB frontier embedder on niche domains. But with good chunking and rerank, users may not notice. Test with your data, not Twitter’s.
• Complexity: You’re introducing a second retrieval stack and capability detection. Keep it behind a thin abstraction and be fanatical about model versioning.
• Storage quotas: Browsers are better than they used to be, but quotas and eviction policies vary. Design for partial indexes and fast rebuilds.
• Device diversity: Expect a 10x spread in throughput. Your product needs to feel good on both a two‑year‑old Android and a new M‑series Mac.

Why This Matters for Nearshore Teams (And Your Roadmap)

Client‑side embeddings let distributed teams ship real user‑visible speed without expensive infra changes. For US‑LATAM product orgs, there’s a bonus: building robust Rust‑to‑WASM pipelines, on‑device indexes, and hybrid retrieval layers is the kind of systems work senior nearshore engineers excel at, with 6–8 hours of overlap for tight iteration. You’ll spend time on product UX instead of another vector cluster.

The Bottom Line

You don’t have to move all retrieval to the client. But if you aren’t moving any of it, you are paying more than you need to, waiting longer than users will, and carrying more privacy risk than your sales team can defend. Start with one user‑scoped feature. Ship a tiny model. Measure. Then expand.

Key Takeaways

• Small embedding models now run in the browser; a ~7 MB WASM model is enough for many user‑scoped searches.
• Move embeddings client‑side for user‑scoped, privacy‑sensitive, latency‑critical features; keep global search server‑side.
• Expect vectors around 384 bytes per item at int8; 10k items fit comfortably on device with a modest ANN index.
• Tag every vector with a model version; never mix incompatible embeddings; encrypt vectors at rest and during sync.
• Use capability detection to band devices and keep a server fallback; instrument telemetry without content.
• Hybrid architectures (server coarse, client rerank) preserve privacy while avoiding quality cliffs.
• Plan a 90‑day rollout: prove it, productionize it, then scale with optional higher‑accuracy models on capable hardware.