2026-07-30 · 10 min read

Stop Pulling Weights Live: Build a Private Model Registry Before the Next HF Incident

If you wouldn’t apt-get from a random mirror in production, why are your inference pods curling weights from the open internet at boot?

After the latest Hugging Face break-in was dissected in the news, it should be obvious: your model artifact pipeline is a supply chain. Treat it like one. Availability is reason enough—security seals the deal. Rate limits, deleted repos, or a tampered tokenizer can take down your agent platform just as surely as a database outage.

This is a CTO playbook for building a private, signed, policy-governed model registry. It’s not another academic ode to SLSA. It’s what to stand up in 90 days so your agents don’t fall over the next time a public model hub hiccups.

First, the ugly math: cold starts and egress are killing you

Let’s say you deploy a quantized 70B model for retrieval-augmented generation. Numbers you can bank on:

Base FP16 weights: ~140–160 GB. Quantized GGUF for production: 30–50 GB.
Tokenizer + vocab + merges + safetensors/gguf index + LoRA adapters: +1–3 GB.
Autoscaling from 0 to 10 replicas? First deploy pulls ~0.3–0.5 TB.
Cloud egress from object storage or cross-AZ traffic often runs $0.02–$0.09/GB. That’s $6–$45 per cold scale event—before you serve a single token.

More important: if your nodes pull directly from a public hub, you’ll hit per-IP rate limits, 429s, or timeouts. Add 10–20 minutes to warm each pod with 50 GB at 100–200 MB/s effective throughput under real-world contention. Those 10–20 minutes are SLO violations waiting to happen.

Threat model: what actually goes wrong

Swapped artifacts: A compromised repository serves a modified tokenizer.json that silently degrades or hijacks behavior. You won’t notice until your evals drift.
Pickle landmines: Legacy PyTorch checkpoints with pickled code paths can execute arbitrary code at load. If your loader “helpfully” supports it, you gave RCE to whoever controls the artifact.
Quantization toolchain drift: Tiny changes in k-quant or activation ordering can yield different perplexity and tool-use rates, even if the filename matches.
License sprawl: A “non-commercial” LoRA slides into production because someone copied a model_id from a notebook.
Availability collapse: A popular model gets yanked or renamed; pods stall on startup; your autoscaler churns.

These are not hypothetical. They are the same classes of failures you already learned to prevent for containers and OS packages—now repeated for model artifacts.

Decision framework: do you need a private registry?

You don’t need to be OpenAI to justify one. If any of these are true, you do:

You run production inference where cold-starts matter (SLOs on first-token latency or request per second).
You’re in regulated sectors (finance, healthcare, education) or serve minors. Auditability matters.
Your monthly egress on weights exceeds $500 or your team re-pulls artifacts multiple times per week.
You ship agents that call tools. The blast radius of a tampered checkpoint is not “just worse text.” It’s money movement or ticket updates.

When can you wait? Proof-of-concepts with a single toy model, no autoscaling, and no user data. Once you see the first 429 from a public hub during a deploy, you’re already late.

Architecture: a model registry that fits your stack

Transport: use the registry you already understand

OCI registry as the backbone: Harbor, Artifactory, or ECR/GCR/ACR. They support immutability, replication, RBAC, and Notary v2 signatures. You can store arbitrary artifacts via ORAS: push safetensors, GGUF, tokenizers, LoRAs as first-class blobs.
Object store mirror for bulk: Back the registry with S3/GCS/Azure Blob for cheap storage and multi-region replication. Keep the registry index small; offload big layers to object storage with signed URLs.

Format policy: ban pickle, standardize metadata

Allowed formats: safetensors for frameworks, GGUF for llama.cpp-class inference. Disallow any artifact that requires Python pickle to load.
Model SBOM: For each model version, embed a JSON manifest listing base weights hash, tokenizer hash, quantization tool version, LoRA parent hash, license, and intended-usage tags. Store this as an OCI artifact attached to the weight digest.

Signing and provenance: no signature, no run

Sign every artifact with Sigstore Cosign or Notary v2. Store public keys in your KMS or use keyless OIDC with your CI as the identity. Attach SLSA provenance (in-toto) describing how the artifact was built or transformed.
Binary Authorization at deploy: Admission controllers reject unsigned or untrusted-digest models. This is table stakes for containers; apply it to models.

Import pipeline: one way in, many ways out

Importer service: A dedicated job with the only egress to public hubs. It pulls by immutable reference (commit SHA or tag digest), converts to approved formats in a hermetic container, computes digests, generates SBOM + attestation, and pushes to your registry.
Policy gate: The importer enforces license allow-lists and format policy. If license = non-commercial and env = prod, reject. If format = pickle, reject.
Quarantine staging: New imports land in a staging project. CI runs evals and security scans (e.g., scan tokenizer merges, vocab shifts, and config diffs). Promotion to prod is a deliberate action that re-signs under a prod key.

Runtime verification: fast and ruthless

InitContainers verify signatures and digests before weights touch the GPU. If verification fails, the pod never becomes ready.
Local NVMe cache with LRU: Keep hot weights on node-local SSDs and validate digests before use. This cuts warm-up by minutes and slashes east-west traffic.
Multi-region replication with pre-warming: Replicate artifacts to the target region’s registry and pre-warm caches during deploy windows.

Observability and audit

Artifact-level telemetry: Emit counters keyed by model_digest, tokenizer_digest. If perplexity or tool-use rates drift, you can prove you’re running the same bits.
Access logs: Every pull is attributable to a service account and environment. You’ll know which pod ran which digest at which time.

The developer experience: don’t make it painful

One command to import: Provide a CLI: modelctl import hf://org/model@sha -o my-registry/models -p staging. It prints the internal URI and attestation IDs.
Deterministic promotion: modelctl promote models/llama-3-70b@sha -e prod. Same digest, new signature, visible to prod namespaces.
Self-serve catalog: A web UI listing approved models, hashes, licenses, and example code snippets for loaders across Python, Rust, Go.

Policy that bites: examples you can borrow

Use OPA/Gatekeeper or your platform’s admission controls. Translate principle to code:

Block unsigned or untrusted digests

Rule: Only allow models with a Cosign signature by org=ml-platform and with provenance type=model-importer@v1.

Outcome: A stray hard-coded HF URL in a deployment manifest fails admission.

Enforce license allow-list

Rule: If env=prod, license in [Apache-2.0, MIT, Meta-Llama, BigCode OpenRAIL].

Outcome: Non-commercial or research-only licenses never clear promotion to prod.

Disallow pickle-loading code paths

Rule: Loader images used in prod must pass a static scan proving no import of torch.load with unpickler enabled.

Outcome: Even if a dev tries to sneak in a convenience loader, it won’t deploy.

Cost and performance: yes, this saves money

Storage: 2 TB of weights on S3/GCS runs ~$46–$50/month at standard tiers. That’s less than a single day of production egress during a chaotic deploy.
Egress: Pulling from a regional registry inside your VPC/AZ is pennies. Eliminating repeated cross-region pulls easily saves $500–$2,000/month at modest scale.
Cold-starts: Pre-warmed NVMe caches cut 10–20 minutes of warm-up per replica to under 2 minutes, depending on I/O and checksum speed. That’s the difference between a graceful deploy and a PagerDuty weekend.

Rollout plan: 30/60/90 days

Days 1–30: Make it stop hurting

Stand up a managed or self-hosted OCI registry (Harbor/Artifactory/ECR) with immutability.
Build the importer job that fetches a single critical model, converts to safetensors/GGUF, and signs with Cosign (key in KMS).
Wire an initContainer to verify signatures and digests before model load. Roll this to your busiest service.
Block direct egress to public hubs from production namespaces; only the importer has access.

Days 31–60: Put policy in the loop

Add license allow-list and pickle ban to the importer. Fail imports that violate policy.
Create staging and production projects in the registry. Require promotion (and a second signature) to go prod-visible.
Attach an SBOM and provenance (in-toto/SLSA) to every artifact. Start logging pulls with service-account identity.
Introduce NVMe node-local caches with LRU and checksum-on-use.

Days 61–90: Make it boring

Replicate the registry to all production regions. Pre-warm caches during low-traffic windows.
Expose a CLI and internal catalog UI. Integrate with your model router and feature flags.
Set binary authorization on the cluster: no signature, no run. Add canaries that intentionally try to pull public URLs and alert if they succeed.
Run a game day: revoke a signer key, pull an artifact, attempt an unsigned deploy, and verify that everything fails closed.

What about fine-tunes and adapters?

Treat LoRAs and merged checkpoints exactly like base weights:

Adapters are code: They change behavior materially. Sign them. Track their parent digest. Enforce that adapters can only target approved parent digests.
Merged checkpoints get a new identity: Never tag a merged artifact with the base model’s ID. New digest, new SBOM, new evals.
Dataset artifacts: If you host training data deltas or synthetic corpora, store them as versioned artifacts with the same signing and policy gates. You’ll need this for reproducibility and audits.

Multi-cloud and edge: don’t strand your bytes

Vendor lock-in avoidance: OCI and ORAS mean you can mirror between ECR/GCR/ACR/Harbor. Keep the import pipeline declarative so you can recreate mirrors quickly elsewhere.
Edge/On-device: If you ship quantized models to mobile or IoT, front them with a private CDN, still backed by the registry as origin. Sign the payload and verify on device before first run.

Organizational reality: who owns this?

Platform/Infra owns the registry, signing keys, admission policies, and importer service.
ML Platform owns format policies, SBOM schema, eval gates, and model catalog.
Security owns key management policy, incident response, and game days.

If you’re short-staffed, a nearshore pod can build this in parallel with your core team. Expect 6–8 hours/day overlap, a 4–6 week first milestone for the importer and verification path, and another 4–6 weeks to harden policy and multi-region replication.

What about “managed model hubs” and vendors?

Great—use them as upstreams, not production dependencies. The rule is simple: One way in, many ways out. Your importer pulls from vendors, normalizes and signs the bytes, and your runtime consumes only what your platform has vouched for. If a vendor rotates a URL or a license overnight, your runtime shouldn’t even notice.

The skeptic’s corner: is this overkill?

You already do this for containers. Models are bigger, touchier, and higher blast-radius. The Hugging Face incident made headlines; the quieter reality is availability, cost, and drift. A private model registry turns “hope the internet is up” into “deploy is a local file copy.”

Key Takeaways

Public hubs are great upstreams, terrible production dependencies. Build a private, signed, policy-governed model registry.
Use an OCI registry plus ORAS for transport, safetensors/GGUF for formats, and Sigstore/Notary for signatures.
Enforce “no signature, no run” with admission controls; ban pickle; allow-list licenses; attach SBOM and provenance.
Importer service is the only egress to public hubs. It converts, signs, scans, and promotes artifacts across staging → prod.
Expect 10–20 minute warm-ups to drop to under 2 minutes with local caches; egress savings often exceed $500/month.
Roll out in 90 days: stand up the registry, add policy, replicate regions, and run game days to ensure it fails closed.