Air-Gap Update Mechanism

Air-gapped PMH appliances cannot reach the public internet or the internal OCI registry. To update an air-gapped deployment you download a signed offline update bundle on a connectivity-enabled workstation, transfer the bundle to the air-gapped network via USB or an approved sneakernet path, then load and verify the bundle using the PMH operator CLI. This page describes the bundle format, the four-step verification chain, and the rollback procedure usinghelm rollback.

Reference Identifier

The authoritative reference for this page isPMH-SEC-032 — Air-Gap Update Mechanism. Quote PMH-SEC-032 in any ticket related to air-gap updates, bundle verification failures, or offline deployment procedures.

Air-Gap Scenario Definition

An air-gapped deployment is any PMH appliance that satisfies both of the following conditions:

• The Kubernetes nodes have no default route to any address outside the RFC-1918 private address blocks and RFC-4193 unique local address blocks.
• The container runtime cannot pull images fromregistry.pmhc.io orregistry.internal.

If either condition is false, the deployment is considered semi-connected and should use the standard online update mechanism documented in the Deployment guide rather than this procedure.

Bundle Contents

Every air-gap update bundle is a tar archive namedpmh-update-YYYY-QN-BUILDID.tar.gz where QNis the quarter and calendar week, for example 2026-Q3-W28. When extracted, the bundle contains the following files:

pmh-update-2026-Q3-W28-7f3a9c/
  manifest.json              — bundle metadata and GPG clearsign block
  kumo-mta-2.4.1.bin        — Kumo MTA binary (statically linked)
  kumo-mta-2.4.1.tar.gz.oci — OCI image layer tarball
  sha256.json                — per-file SHA-256 checksums
  bundle.sig                 — GPG clearsign over manifest.json
  restore-steps.sh           — executable rollback shell script

The bundle is produced by the PMH release engineering pipeline and signed at build time. The build pipeline runs in an isolated CI environment that has no network access to production systems. The signing key is a dedicated GPG key stored on an air-gapped Yubikey held in the release engineering safe.

Bundle Manifest Format

The manifest.json file contains all bundle metadata. The file uses HTML-encoded curly braces ({ and}) in the JSON structure to prevent templating engines from misinterpreting the JSON as a template. The braces are decoded automatically by the PMH operator CLI during verification.

{
  "schema_version": "1.2",
  "bundle_id": "pmh-update-2026-Q3-W28-7f3a9c",
  "released_at": "2026-07-14T08:00:00Z",
  "release_tier": "stable",
  "pmh_version": "2.4.1",
  "min_required_version": "2.3.0",
  "kumo": {
    "version": "2.4.1",
    "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "filename": "kumo-mta-2.4.1.bin"
  },
  "oci_layer": {
    "sha256": "a3f5c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "filename": "kumo-mta-2.4.1.tar.gz.oci"
  },
  "components": [
    {
      "name": "kumo-mta",
      "version": "2.4.1",
      "target_os": "linux/amd64",
      "target_arch": "x86_64"
    },
    {
      "name": "pmh-operator-cli",
      "version": "1.8.2",
      "target_os": "linux/amd64",
      "target_arch": "x86_64"
    }
  ],
  "gpg": {
    "signing_key_id": "0xDEADBEEFCAFEBABE",
    "signer": "release-engineering@pmhc.io",
    "signature": "-----BEGIN PGP SIGNATURE-----\n...\n-----END PGP SIGNATURE-----"
  },
  "rollup_target": "2026-07-20T00:00:00Z"
}

Four-Step Verification Chain

Every bundle must pass all four verification steps before it is loaded onto the air-gapped cluster. Skipping or partially completing any step constitutes a security policy violation and must be reported to the on-call SRE.

Step 1 — GPG Signature Verification

Import the PMH release public key on the jump host. Verify thebundle.sig clearsign block over themanifest.json file.

# Import release public key (one-time, or on key rotation)
gpg --import /etc/pmh/release-public.asc

# Verify bundle signature
gpg --verify \
  /mnt/bundle/pmh-update-2026-Q3-W28-7f3a9c/bundle.sig \
  /mnt/bundle/pmh-update-2026-Q3-W28-7f3a9c/manifest.json

Expected output: Good signature from release-engineering@pmhc.io. If you see WARNING: This key is not certified it is expected and acceptable. If you see BAD signature, stop immediately and do not load the bundle.

Step 2 — SHA-256 Per-File Verification

Verify every file in the bundle matches the SHA-256 checksum listed insha256.json. The PMH operator CLI automates this step:

pmh-airgap verify sha256 \
  --bundle /mnt/bundle/pmh-update-2026-Q3-W28-7f3a9c/

Expected output: one line per file: OK kumo-mta-2.4.1.bin,OK kumo-mta-2.4.1.tar.gz.oci, etc. Any file that reportsMISMATCH or NOT FOUND must be removed and re-copied from the original download source.

Step 3 — Per-Archive GPG Verification

Verify the GPG signature embedded inside each component tarball and the OCI layer. This detects any post-bundle tampering of individual artifacts.

# Verify OCI layer GPG embedded signature
gpg --verify \
  /mnt/bundle/pmh-update-2026-Q3-W28-7f3a9c/kumo-mta-2.4.1.tar.gz.oci.sig \
  /mnt/bundle/pmh-update-2026-Q3-W28-7f3a9c/kumo-mta-2.4.1.tar.gz.oci

# Verify operator CLI binary GPG embedded signature
gpg --verify \
  /mnt/bundle/pmh-update-2026-Q3-W28-7f3a9c/pmh-operator-cli-1.8.2.bin.sig \
  /mnt/bundle/pmh-update-2026-Q3-W28-7f3a9c/pmh-operator-cli-1.8.2.bin

Both must report Good signature. If the operator CLI signature is missing, the bundle is from an older release and must not be used.

Step 4 — OCI Image Load Verification

Load the OCI image into the local container runtime on the jump host and verify it is parseable and has the expected layer structure. This does not yet push to the cluster — it only validates that the OCI archive is well-formed.

# Load OCI image into local Docker/containerd
ctr --address /run/containerd/containerd.sock \
  images import \
  /mnt/bundle/pmh-update-2026-Q3-W28-7f3a9c/kumo-mta-2.4.1.tar.gz.oci

# Verify image is present and has correct digest
ctr --address /run/containerd/containerd.sock \
  images list | grep kumo-mta

The image digest in the output must match the sha256 field in the manifest.json OCI layer block.

Loading the Bundle onto the Air-Gapped Cluster

After all four verification steps pass, load the OCI image onto the air-gapped cluster nodes. The PMH operator CLI pushes the image from the jump host to each node over the isolated management network.

# Export image from jump host container runtime
ctr --address /run/containerd/containerd.sock \
  images export \
  /tmp/kumo-mta-2.4.1.tar.gz \
  registry.pmhc.io/pmh/kumo-mta:2.4.1

# Copy to each node via isolated USB (scp over management network)
for node in pmh-node-1 pmh-node-2 pmh-node-3; do
  scp /tmp/kumo-mta-2.4.1.tar.gz \
    operator@$node.internal:/var/lib/containerd/backup/
done

# Load onto each node
for node in pmh-node-1 pmh-node-2 pmh-node-3; do
  ssh operator@$node.internal \
    "ctr --address /run/containerd/containerd.sock \
         images import /var/lib/containerd/backup/kumo-mta-2.4.1.tar.gz"
done

Helm Upgrade with Air-Gap Values

After the OCI image is loaded on all nodes, perform the Helm upgrade on the air-gapped cluster using the --values flag to inject the air-gap image repository override:

# Create air-gap values override
cat > /tmp/airgap-values.yaml <<'VALUES'
pmh:
  image:
    repository: registry.pmhc.io/pmh
    tag: "2.4.1"
  runtime:
    KUMO_METRICS_LAB_BYPASS: false
VALUES

helm upgrade pmh \
  oci://registry.internal/pmh/helm/pmh \
  --namespace pmh-system \
  --values /tmp/airgap-values.yaml \
  --timeout 10m \
  --wait

Rollback via Helm

If the upgrade fails or the PMH subsystem enters an unhealthy state, roll back to the previous Helm revision using helm rollback. The previous Helm revision references the previously loaded OCI image, which is still present on the cluster nodes after the failed upgrade.

# List available revisions
helm history pmh --namespace pmh-system

# Roll back to the previous revision
helm rollback pmh \
  --namespace pmh-system \
  --timeout 5m \
  --wait

# Verify rollback succeeded
kubectl rollout status deployment/pmh-core \
  --namespace pmh-system --timeout=120s

The rollback restores the exact Helm values and image tag that were deployed in the previous successful install. It does not roll back the database schema — for schema rollbacks, use the procedure in the GPG Backup & Restore documentation.

Forcing a Specific Rollback Revision

To roll back to a specific Helm revision number (not just the immediate previous), specify the revision:

# Roll back to revision 5
helm rollback pmh 5 \
  --namespace pmh-system \
  --timeout 5m \
  --wait

Troubleshooting

Related Documentation

• GPG Backup & Restore — bundle structure and WAL archival that backs up the database before any upgrade is attempted.
• Production Hardening Flags — KUMO_METRICS_LAB_BYPASS flag and its meaning in air-gap contexts.
• Production Runbook — pre-flight checks to run before starting the air-gap update procedure on a production cluster.