Getting model weights into an air-gapped enclave without breaking accreditation

Air-gapped work has a particular rhythm. Everything takes longer, nothing is reversible, and the question that ends up mattering is not “can we get this in” but “can we do it again next quarter without a fresh approval battle”.

What actually has to happen#

Outside the boundary: fetch the weights, check the licence permits what you intend to do with them, scan the artefact, generate an SBOM for everything travelling with it, hash it, and sign a manifest that lists every file and its digest.

Across the boundary: physical media, under change control, with a documented custody chain.

Inside: verify the signature before anything else touches the artefact, and fail closed if it does not verify. Admit it to an internal registry. Only then deploy to the serving tier.

Nothing crosses back except records generated inside the enclave.

That sequence is not complicated. Getting an organisation to accept it as a standing process rather than a one-off exception is the work.

The licence check is not a formality#

Model licences vary considerably in what they permit, and several of the terms that matter (acceptable use, redistribution, whether outputs may train other models, whether the licence can be revoked) are exactly the terms that a legal review inside a regulated organisation will want in writing before deployment.

Do this before the transfer, not after. We have watched a fully-approved artefact sit unused for six weeks while legal worked through terms that could have been read on day one.

Provenance is the thing you are really building#

Eighteen months later, somebody will ask which model version produced a specific output. If the answer requires archaeology, you have a problem that no amount of good engineering afterwards will fix.

What makes that answerable: immutable artefact identifiers carried from the outside registry to the inside one; deployment records that tie an artefact digest to a time window and a serving endpoint; and a trace format that records the model identifier on every inference, not just at deploy time. The last one is easy to skip and expensive to retrofit.

Updates are the part people under-plan#

An enclave with no update path becomes an enclave running a model with a known defect and no way to replace it.

Plan the cadence before the first transfer. Batch updates so each crossing carries the model, its dependencies, base images and security patches together. The crossing is the expensive step, not the payload. Keep the previous artefact resident so rollback does not require another transfer. And rehearse the rollback, in the enclave, before you need it.

Things that have caught us out#

Dependency closure is larger than you think. A serving container’s transitive dependency set includes things that resolve at build time from the internet. Build outside, transfer the built image, and verify inside, do not attempt to build in the enclave from a partial mirror.

Tokenisers and configs travel separately and get forgotten. A model that loads with a mismatched tokeniser produces output that looks plausible and is subtly wrong. Include them in the manifest and verify them with the weights.

GPU driver and CUDA versions have to be planned as part of the same crossing. An artefact built against a newer runtime than the enclave has will fail at load, and the fix is another transfer cycle.

Clock skew breaks signature validation. Enclaves without external time sources drift. This produces a signature failure that looks like tampering and causes an incident review. Establish an internal time source early.

The part that is genuinely hard#

Everything above is process, and process can be written down. The hard part is organisational: an air-gapped deployment involves a security team, a platform team, a change board and usually an accreditation authority, and each of them will ask a version of the same question at a different time.

The single most useful thing we do on these engagements is write the transfer procedure as a document those four groups review together, once, before the first crossing. It takes about two weeks and saves a quarter.