Ban JSON from Your AI Data Plane: Protobuf and Arrow Beat Pretty Strings

JSON made your microservices possible. It’s also quietly strangling your AI stack. If your embeddings, tool calls, and agent logs still travel as pretty strings, you’re paying a 2–4x tax in bandwidth and CPU on every call. That’s money, latency, and SLO budget you don’t get back.

This month’s “Protobuf for Python, without compromises” drop on Hacker News is the nudge you needed. The protobuf-py effort removed a lot of Python footguns and performance cliffs. Translation: the last good excuse to keep shipping JSON inside your AI data plane just died.

What the JSON Tax Looks Like in 2026

Embeddings

Take a typical 1,536‑dimension float32 embedding (6,144 bytes raw). If you serialize those as a JSON array of numbers, each value often costs 8–12 ASCII bytes plus punctuation. At a conservative 9 bytes per value, that’s ~13.8 KB per vector. If you base64 the raw floats instead (a common hack), base64 inflates by exactly 33%: 6.0 KB becomes ~8.2 KB.

So the same vector is:

• 6.1 KB as binary (float32)
• 8.2 KB as base64 (binary-in-text)
• ~13.8 KB as JSON numbers

Call it a 2.3x penalty JSON vs binary. Multiply across 10 million embeddings per day (not crazy if you run RAG for 100 QPS): you’re moving ~77 GB of unnecessary bytes every day. At $0.05–$0.09/GB egress, that’s $3.8–$6.9K/month burned on air—before CPU costs.

Tool Calls and Agent Traces

Agent frameworks love to stuff everything into JSON: tool inputs, outputs, intermediate artifacts. This is developer-friendly, not machine-friendly. Hand a 150 KB JSON blob to Python’s parser and you’re paying dozens to hundreds of microseconds of CPU per hop. Protobuf/FlatBuffers decoders are typically 3–5x faster than Python JSON parsing in real apps, and they don’t allocate strings for every comma-separated number.

Logs and Metrics

Even logs get ugly. Teams stream token logs and semantic events as JSON over SSE or WebSocket. Great for a browser. Terrible between services. It’s common to see 5–10x more bytes on the wire than the actual information content, and all of it has to be parsed before it’s useful.

Decision Framework: What Goes Where

There isn’t one hammer. Use the right wire format per lane of your AI data plane. The test: can you move, decode, and index the payload in less than 100 microseconds per 100 KB on your median node, with headroom? If not, change the format.

Lane 1: Tool Calls and Agent Actions (RPC)

• Pick: gRPC with Protobuf.
• Why: Contracted, versionable, HTTP/2 multiplexing, streaming, battle‑tested in polyglot stacks. With the modern Python bindings (protobuf‑py), you avoid the reflection/GC potholes that historically hurt.
• Notes: Use oneof for variant tool outputs, reserve field numbers for evolution, and stick to primitive numeric types where possible. Keep blobs out-of-band (object store) and pass content‑addressable references instead.

Lane 2: Embeddings and Batch Feature Vectors (Bulk)

• Pick: Apache Arrow for in‑memory/IPC and Arrow Flight for network transport; Parquet for storage.
• Why: Columnar, zero‑copy, SIMD‑friendly. Arrow packs a 1,536‑dimensional float column in tight memory and ships it without turning floats into text. Flight gives you backpressure-aware streaming over gRPC without reinventing batch semantics.
• Notes: Keep batch sizes in the 64 KB–4 MB range for throughput without head‑of‑line blocking. Favor float16/bfloat16 where acceptable; document precision trade‑offs.

Lane 3: Realtime Streaming to Browsers (Edge)

• Pick: Keep JSON or JSON Lines for Server‑Sent Events to browsers. Compress with gzip or zstd and cap event size.
• Why: Browsers like text. Don’t fight the platform at the edge. Do force all server‑to‑server streams onto a binary format.
• Notes: If you must stream binary to the browser, use WebTransport or WebSocket with ArrayBuffer. Expect wider compatibility issues than SSE.

Lane 4: Internal Caches and Queues

• Pick: Use binary values in Redis/Valkey. For Kafka/Pulsar, use Protobuf or Avro with schema registry.
• Why: JSON values in hot caches blow up memory and CPU. Schema‑aware topics avoid consumer drift and let you evolve fields without manual reprocessing.

The Hard Numbers (and a Quick ROI Check)

Here’s a back‑of‑the‑napkin calculation for a typical mid‑scale RAG system:

• Workload: 120 QPS average, each request writes one 1,536‑dim float embedding to a feature store and publishes one tool‑call trace of ~50 KB.
• JSON baseline: Embedding ~13.8 KB + trace 50 KB = ~63.8 KB/request. That’s ~7.6 MB/s or 657 GB/day across services.
• Binary refactor: Embedding 6.1 KB + trace 15 KB (protobuf, lightly compressed) = ~21.1 KB/request. ~2.5 MB/s or 218 GB/day.
• Savings: ~439 GB/day. At $0.05–$0.09/GB egress: $657–$1,183/month in egress alone. CPU savings usually dwarf this: JSON parsing and GC reduction across your fleet can free 5–20% of CPU on hot services, which at current instance pricing is real money.

Note how quickly this compounds in multi‑region or fan‑out topologies.

Schema Design: Don’t Recreate Your JSON Spaghetti in Protobuf

You don’t win by swapping curly braces for varints. You win by imposing shape.

• Separate blobs from metadata: Put big binary arrays (embeddings, images) in dedicated fields or streams. Keep metadata tight and typed.
• Use oneof for variants: Tool results should be an algebraic type, not a JSON object with ad‑hoc keys. For example, result can be one of: search_result, file_write, http_call.
• Version forward, not backward: Add fields with new numbers. Never repurpose a field number. Use feature flags or capability negotiation, not guesswork.
• Choose numeric precision intentionally: float16/bfloat16 halves bandwidth against float32. For cosine similarity and ANN, that’s usually fine. Document where it isn’t.
• Define limits: Max array lengths, max string lengths. Enforce at decode. Your AI data plane is part of your security perimeter.

Browser Reality Check: JSON at the Edge, Binary Inside

“But our TypeScript front end…” Sure. Keep JSON at the boundary. Use gRPC‑Web or Connect for browser compatibility if you want strongly typed APIs; they can speak JSON on the wire to the browser while your server speaks Protobuf internally.

The rule: no JSON past the edge tier. Front ends talk JSON or JSON Lines to the edge; all inter‑service links are binary. If you must pass embeddings to a browser (rare), send them as ArrayBuffer in a WebSocket or WebTransport, not as base64 strings in JSON.

Tooling Maturity in 2026: The Excuses Are Gone

• Python: The newer protobuf‑py layers are faster and less magical. They play well with type checkers and avoid the GC churn older wrappers caused. Apache Arrow’s Python bindings are stable and production‑grade; Arrow Flight works with asyncio.
• TypeScript/Go/Java: First‑class compilers and gRPC libraries. Buf‑based workflows enforce API breaking‑change rules in CI.
• FlatBuffers and Cap’n Proto: If your use case needs in‑place reads and ultra‑low latency, consider these. The trade‑off is steeper learning curve and rougher multi‑language ergonomics than Protobuf. Use them for hot paths, not everywhere.
• Schema registries: Confluent‑style registries aren’t just for Avro. Keep a repo of .proto files with lint rules and generated docs; treat it like source of truth.

Migration Plan: 30, 60, 90 Days

Day 0–30: Measure and Box the Problem

• Instrument payloads: Log wire sizes and parse times per route, per service. Sample 1–5% of traffic if you must. Tag with content type.
• Pick two guinea pigs: One RPC (a busy tool‑call endpoint). One bulk job (embedding ingest).
• Decide formats: gRPC+Protobuf for the RPC; Arrow Flight for the bulk job. Agree on field naming and versioning policy in writing.

Day 31–60: Build Dual‑Stack

• Generate types: Land .proto files in a shared repo. Generate language bindings as part of CI (ts‑proto for TS, protobuf‑py or protoc‑gen‑mypy for Python).
• Implement adapters: For the chosen endpoints, produce and consume both JSON and Protobuf in parallel. Gate by header or versioned path.
• Wire‑compat tests: Create golden binary fixtures and round‑trip tests across languages. Add a fuzz pass for decoders with size caps to catch parser bombs.
• Operational hooks: Emit metrics: decode_failures, decode_latency_p99, payload_ratio (json_bytes/binary_bytes), and per‑route bandwidth.

Day 61–90: Flip, Then Spread

• Edge policy: Enforce “JSON only at the edge.” Block JSON on internal links with Envoy or service mesh policy (content‑type allow list).
• Kill base64: For any binary payloads that remain in text transports, replace base64 blobs with object store references or binary streams.
• Rollout schedule: Migrate one hot path per week. Expect 20–40% bandwidth reductions per service hop and visible CPU headroom.
• Cost check: After 30 days, compare egress and CPU before/after. Reinvest freed CPU in latency SLOs or instance downscaling.

Operational Guardrails

• Observability: Parse errors must be first‑class incidents. Alert on any spike. Keep per‑format histograms of decode times and sizes.
• Backpressure: Use HTTP/2 stream limits and Flow Control in gRPC. For Arrow Flight, cap inflight batches per peer.
• Security: Treat your deserializers like you treat your SQL clients: fuzz them, cap them, and keep them off the request thread pool if you can.
• Data retention: Keep binary payloads short‑lived. Persist only business facts. For analytics, convert to Parquet with explicit schema and TTL.

Trade‑Offs You Should Stare At

• Debuggability: You lose the “just curl it” ease. Counter with CLI tools to pretty‑print Protobuf and Arrow, and make them part of your developer images.
• Browser friction: gRPC‑Web and Connect clean up a lot, but you’ll still maintain JSON at the outer boundary. That’s fine—keep it there.
• Team learning curve: Your engineers will need to learn schema evolution and field numbering discipline. It’s worth it; enforce with linters and CI diffs.
• Third‑party APIs: Some vector DB vendors still default to JSON arrays. Where possible, use their binary/batch APIs. Otherwise, convert at the edge of your system, not in hot internals.

Edge Cases: When JSON Is Good Enough

• Small, rarely called endpoints: JSON is perfectly fine for a settings read or a one‑off admin tool.
• Public APIs with broad client ecosystems: Keep JSON for reach, and explicitly cap payloads. Offer a binary variant for partners.
• Human‑in‑the‑loop logs: If a payload is only ever read by a human in a browser, prioritize legibility and compress aggressively.

A Simple Rule to Institutionalize

Binary inside; JSON outside. Put it on a poster. Make a CI rule that blocks new JSON fields on internal endpoints. Add a dashboard tile that shows your JSON‑to‑binary traffic ratio drifting down week over week. And celebrate the first time you delete 20% of a service fleet without hurting an SLO.

The Internet’s elders got us to a world of interoperable text. The next decade of AI is about throughput, latency, and unit economics. That means contracts you can evolve and bytes that pull their weight. You don’t have to rewrite the world—just stop paying a tax you don’t need to.

Key Takeaways

• JSON inflates embeddings and traces 2–4x vs binary, and base64 adds a strict 33% overhead. This is real money and latency.
• Use gRPC+Protobuf for tool calls, Arrow/Flight for embeddings and batch features, and binary values in caches/queues.
• Keep JSON at the browser edge; enforce binary inside the mesh. Migrate lane by lane with dual‑stack endpoints.
• Instrument payload size and decode latency; aim for under 100 microseconds per 100 KB on the median node.
• Expect 20–40% bandwidth savings per hop and 5–20% CPU headroom on hot services after migration.
• The ecosystem is ready in 2026: modern protobuf‑py, mature Arrow, and strong TS/Go/Java tooling remove prior excuses.