Every few quarters, a privacy headline promises the moon. This month’s version: “Google is making private AI practical with homomorphic encryption.” If you’re a CTO with PII, PHI, or payments in your model inputs, that sentence hits like a compliance cheat code. Encrypt everything, compute anyway, sleep better.
Reality check: homomorphic encryption (HE) is finally crossing from insider crypto talks into engineering backlogs. But it’s not a silver bullet for chatbots or complex inference. In 2026, HE is a precision tool. Used well, it can eliminate an ugly class of breach scenarios and simplify your legal posture. Used broadly, it’ll light your infra bill on fire and crater your latency.
What just changed — and what didn’t
Research from large vendors and open-source efforts (e.g., Google’s compiler work to target HE runtimes, plus steady progress in OpenFHE, Microsoft SEAL, and Zama’s Concrete) means two practical things for you:
- The “toy-only” era is over: Small real models (linear/logistic, tiny MLPs, shallow CNNs) can run under HE with second-to-minute latency if you constrain parameters and operations. That was not true at this level five years ago.
- The speed gap is still brutal: Expect 10^2 to 10^4 overhead versus plaintext compute, depending on scheme and circuit depth. Transformers remain in “academic demo” territory, not production SLAs.
Translation: you can ship HE for narrow, high-value use cases where your business wins by keeping the server blind. You can’t bolt HE on your RAG copilot and keep a 300–500 ms TTI.
Decide with a threat-model first, not a benchmark first
HE is about who you’re hiding from during compute, not just “privacy is good.” Here’s the short ladder.
Who are you trying to blind?
- Curious cloud admin / rogue SRE: HE helps. Data stays encrypted at rest, in transit, and during compute on your servers.
- Compelled disclosure / cross-border access: HE helps if you keep keys client-side or off the compute boundary. You need counsel to bless this.
- Malicious vendor / compromised hyperscaler node: HE helps by default, but side-channels and metadata remain.
- Compromised client device: HE doesn’t help. The plaintext originates at the edge.
- Insider with key access: If your ops team can reach keys, HE may be theater. Make the key boundary explicit.
If you’re not targeting at least the first two categories, enclaves or standard hardening might be the better 80/20.
What’s shippable in 2026 (and what isn’t)
Good bets (90-day pilots are realistic)
- Encrypted scoring over numeric feature vectors: Fraud risk, AML flags, underwriting scores, offers/eligibility, PII similarity scores, ad-lift models. Think 20–200 features; polynomial/logistic form.
- Private set operations via HE-backed approximate arithmetic: De-duplication, overlap checks with partners without revealing raw identifiers (when you can encode as embeddings and operate additively).
- Threshold-able decisions: Return a score or a yes/no. Avoid heavy argmax over large classes; keep post-processing local.
Maybe, with compromises
- Tiny CNNs for classification: You’ll replace non-polynomial ops (ReLU, max-pool) with HE-friendly polynomials. Expect accuracy drops and seconds-to-minutes latency.
- Gradient-boosted trees: Possible if you convert to arithmetic circuits, but circuit depth balloons. Often easier to distill to a small MLP and use that under HE.
Not yet (don’t do this)
- Interactive LLM chat: Latency and multiplicative depth make this unrealistic for production. If someone tells you “we run transformers fully homomorphically in real time,” get a second opinion.
- Multi-tool agent plans under HE: Too many branchy, non-polynomial operations. Use enclaves or on-device for the sensitive steps instead.
Pick the right scheme for the job
You don’t need to become a cryptographer, but you do need to know the three families you’ll be asked about.
- CKKS (approximate arithmetic): Great for real-valued vectors and ML scoring. You trade precision for performance. Most “encrypted regression/MLP” demos use CKKS.
- BFV/BGV (exact integer arithmetic): Useful when correctness requires discrete math, but performance is worse for typical ML workloads.
- TFHE/CGGI (gate-by-gate boolean): Extremely flexible, painful for deep nets. Shines in circuits with lots of control flow but modest depth.
Rule of thumb: If your plaintext model is linear or a tiny MLP over floats, start with CKKS. Keep circuit depth minimal to avoid frequent bootstrapping (the expensive “refresh” step).
Latency, throughput, and cost — the only numbers that matter
You will see eye-popping microbenchmarks. Ignore them. Model your own workload with representative feature distributions and circuit depths.
Back-of-the-napkin planning numbers
- Overhead: 100×–10,000× versus plaintext is a sane planning window today. Start modeling at 1,000×; celebrate if you beat it.
- Per-request latency: A small logistic model under CKKS can often land in single to low double-digit seconds on a high-end CPU, or sub-5s with an optimized GPU path. Anything deeper jumps quickly.
- Throughput: Batch aggressively. HE loves SIMD-style packing of many features into a single ciphertext; your per-request cost drops as you fill the slots.
Cost model example
Suppose you score 100k users/day with a 64-feature logistic model. Plaintext: a single modest CPU instance can do this in milliseconds per request, say low three figures in monthly infra. Under CKKS at 1,000× overhead, you’re likely looking at a few powerful CPU nodes or a GPU-backed service, pushing this into low five figures monthly. Is five figures to blind your servers to PII worth it? If it lets you partner with a bank who otherwise says no, the answer is probably yes.
Architectures that work without derailing your roadmap
Pattern 1: HE for the sensitive core, plaintext for the rest
Encrypt features client-side, send to a dedicated HE scoring service, get a single scalar back. Everything else (identity, logging, personalization) stays plaintext. You’ve cut the blast radius of compromise by keeping the critical join — user attributes to decision — off-limits to the server.
- Key detail: Keep private keys off the compute boundary. The scoring service must never see decryption keys.
- Where HE helps: Legal and vendor-risk teams love this. Even a full database exfil yields only encrypted features.
Pattern 2: Split trust — HE plus enclaves
Use enclaves (TDX/SEV-SNP) for heavy pre/post steps and HE for the final computation. For example, tokenization and normalization inside an enclave, then homomorphic scoring, then enclave again for thresholding or auditing.
- Why: Enclaves are fast but expand the trust base to hardware vendors and attestation services. HE minimizes that trust without paying full HE cost everywhere.
Pattern 3: Hybrid edge
Derive embeddings on-device, encrypt them, and ship only the vector. Homomorphic scoring runs server-side; client interprets the score locally. You shed raw PII while keeping model control centralized.
Key management is where projects live or die
HE lets you compute without decrypting, but someone still holds decryption keys. Treat that boundary as a product requirement, not a footnote.
- Client-held keys: Best privacy posture. Keys live in mobile secure enclaves or browser storage guarded by a platform keystore. You need escape hatches for recovery and rotation.
- Partner-held keys: In B2B integrations, each partner encrypts with their public key and decrypts results locally. You never touch plaintext.
- Server-held keys: Generally defeats the point unless keys are tiered and physically separate from compute with strict policy enforcement and HSMs.
Whichever you pick, write it down as a policy. Auditors will ask, and your SREs will need the runbook when a rotation goes wrong.
What does compliance actually get out of this?
Lawyers care about definitions. If you can credibly argue “no plaintext ever existed server-side for the sensitive join,” breach-notification obligations can change materially. This is jurisdiction-specific. Your counsel may treat homomorphically processed data as outside certain scopes if keys never cross the compute boundary and you log proof (e.g., attestation of encryption before ingress, cryptographic parameters, and absence of decryption paths in production).
Don’t overpromise. Metadata (IP, timestamps, payload sizes) still leaks. You’ll still need DPA terms, DPIAs, and sane retention. But your negotiating position with regulated partners will be better.
Tooling you can actually use
- OpenFHE: Modern successor to PALISADE/HElib lineage, supports CKKS/BFV/BGV and GPU acceleration paths via community projects. C++ core with Python bindings.
- Microsoft SEAL: Battle-tested library (CKKS/BFV). No official GPU path, but mature and well-documented. Good for first pilots.
- Zama Concrete: TFHE-focused with Rust ergonomics. Great for boolean circuits; expect work if your model is float-heavy.
- Compiler/tooling: Keep an eye on MLIR/HEIR-style projects that lower ML graphs to HE-friendly circuits. Useful to estimate circuit depth and bootstrap counts even if you don’t deploy them yet.
For the browser, pair server libraries with a thin client encryption layer. You’ll likely ship a WebAssembly module to package features into ciphertexts. Budget real time to harden this, including origin isolation and anti-tamper for keys.
Pitfalls we keep seeing in pilots
- Choosing the wrong activation: Your favorite nonlinearity isn’t HE-friendly. Use low-degree polynomials (e.g., Chebyshev approximations) and accept the accuracy hit.
- Noise budget exhaustion: Deep circuits die mid-flight. Shallow wins. Budget your bootstraps explicitly.
- Side-channels via branching: If the server can infer secrets from whether you took path A or B, you’ve lost. Keep compute uniform where possible.
- Logging plaintext: It sounds stupid until a “temporary debug” log ships. Wrap your HE surface in a minimal API and lock it down.
- No batch discipline: Pack feature vectors. Empty slots are wasted money.
Your 90-day plan
Week 0–2: Pick one decision
- Choose a single scalar outcome: risk score, eligibility, or similarity.
- Define a latency budget: e.g., p95 ≤ 5s. If you need sub-second, don’t start with HE.
- Freeze a feature set: 32–128 numeric features is a good target.
Week 2–6: Build the railings
- Stand up a separate scoring microservice with a minimal API: encrypt(features) → ciphertext, score(ciphertext) → ciphertext, decrypt(ciphertext) → score.
- Prototype in SEAL or OpenFHE with CKKS. Validate against a plaintext baseline to within your tolerance.
- Ship a client encryption module (mobile and/or web) with key storage design reviewed by security.
- Bench with realistic payloads: batch sizes, p50/p95/p99 latencies, and throughput saturation.
Week 6–10: Prove it’s not theater
- Move keys off compute. If keys stay server-side, write down why and what compensating controls exist.
- Add operational SLOs: rate limits, backpressure, and clear failure modes.
- Run a tabletop breach drill: assume full database exfil of encrypted features; verify you can credibly claim “no plaintext existed.”
- Loop in legal and a design partner customer. This is a selling point; use it.
Build vs. buy vs. nearshore
Two truths. First, there aren’t many experienced HE engineers available in the Bay Area, and the ones who exist are incredibly expensive. Second, you don’t need a PhD to ship a scoped pilot, but you do need engineers who think like cryptographers: parameter discipline, side-channel awareness, and comfort with strange performance profiles.
The pragmatic approach we see working: staff a pod of 4–6 with one senior applied-crypto engineer, two ML/DS folks, one platform/backend, and one SRE. Expect 6–10 weeks to reach a production pilot if you keep scope tight. If you lack crypto depth in-house, nearshore partners with applied-crypto and Rust/C++ chops can be productive quickly and give you the time-zone overlap you need (Brazil gives you 6–8 hours with US teams) without the sticker shock of rare onshore talent.
What success looks like
Success is not “we encrypted all of inference.” Success is being able to look your board, your largest enterprise prospect, and a regulator in the eye and say: “For this decision that touches sensitive PII, our servers never see plaintext. If compromised, the attacker gets ciphertext. Here is our latency SLO, our key boundary, and our audit evidence.”
That’s shippable in 2026. And it’s worth real money.
Key Takeaways
- HE is production-viable today for narrow, scalar decisions over numeric features; it’s not ready for low-latency chat or complex agent loops.
- Start with CKKS for small linear/MLP models; keep circuit depth and nonlinearity low.
- Budget a 1,000× overhead and design for batching; celebrate any wins beneath that.
- Architect for key separation: client- or partner-held keys change your legal posture.
- Combine HE with enclaves for practical performance while minimizing trust.
- Stand up a separate scoring surface, prove correctness vs. plaintext, and run a breach tabletop.
- Staff a 4–6 person pod; a focused pilot is a 6–10 week effort, not a platform rewrite.