GPG Backup & Restore

Every PMH appliance runs a scheduled GPG-signed backup every six hours. Backups include the PostgreSQL base dump, all Write-Ahead Log (WAL) segments generated since the last base backup, a machine-readable manifest, and a Bourne shell restore script. All artifacts are encrypted with AES-256 and signed with a detached GPG signature. This page documents the backup architecture, bundle structure, the six-step restore drill you must perform quarterly, and the failure handling matrix.

Reference Identifier

The authoritative references for this page arePMH-SEC-031 — GPG Backup & Restore andPMH-SEC-032 — WAL Archive Mode. Quote both identifiers when opening a backup-related security or operations ticket.

Encryption and Signing

Algorithm Suite

PMH backups use the following cryptographic suite, enforced by thepmh.backup.encryption Helm values block:

• Cipher: AES-256-CBC (OpenSSL aes-256-cbc)
• Key derivation: PBKDF2 with 100 000 iterations, SHA-512
• Signing: RSA-4096 GPG detached signature
• Key storage: Master key stored in HashiCorp Vault atsecret/pmh/backup/master; passphrase stored separately atsecret/pmh/backup/passphrase

Configuration Values

# Helm values — pmh.backup.encryption block
pmh:
  backup:
    encryption:
      cipher: aes-256-cbc
      key_bits: 4096
      pbkdf2_iterations: 100000
    gpg:
      recipient: backup@pmh.internal
      sign_key: 0xABC123DEADBEEF
    schedule:
      interval_hours: 6
      retention_days: 30
      wal_mode: archive

WAL Archive Mode

When WAL_ARCHIVE_MODE is set to archive (the production-default), PostgreSQL ships every WAL segment to the S3-compatible backup bucket as soon as it is closed. The base backup runs every six hours and the WAL segments captured between base backups are replayed on restore to achieve point-in-time recovery (PITR) with a maximum target recovery point objective (RPO) of six hours.

WAL segments are named using the PostgreSQL 16-byte hexadecimal segment ID:

000000010000000000000001
000000010000000000000002
...
0000000100000000000000FF

Each segment is uploaded as wal/YYYY-MM-DD/HHMMSS/0000000100000000000000XXinside the backup bucket. The restore procedure replays WAL segments in lexicographic order, stopping at the requested recovery point.

Schedule and Retention

Bundle Structure

Each six-hour backup cycle produces a bundle directory on the local/var/lib/pmh/backup/staging/ filesystem before uploading to S3. The staging directory structure is as follows:

/var/lib/pmh/backup/staging/YYYY-MM-DD/HHMMSS/
  manifest.json              — machine-readable metadata
  base_backup.tar.gz.gpg    — encrypted base dump
  base_backup.sig           — GPG signature over base_backup.tar.gz
  wal_segments.tar.gz       — compressed WAL segments since last base backup
  wal_segments.sig          — GPG signature over wal_segments.tar.gz
  bundle.sig                — GPG clearsign block over manifest.json
  restore-steps.sh          — executable Bourne shell restore script

manifest.json Format

The manifest is a JSON document signed with the bundle GPG key. It references all artifact SHAs, the WAL segment range, the encryption parameters, and the intended restore target.

{
  "schema_version": "1.0",
  "backup_id": "2026-07-17T120000Z",
  "tenant_id": "tenant-acme",
  "created_at": "2026-07-17T12:00:00Z",
  "pg_version": "16.3",
  "wal_range": {
    "start": "000000010000000000000001",
    "end":   "00000001000000000000001F"
  },
  "base_backup": {
    "filename": "base_backup.tar.gz.gpg",
    "sha256": "a3f5c...",
    "size_bytes": 524288000
  },
  "wal_archive": {
    "filename": "wal_segments.tar.gz",
    "sha256": "b7d9e...",
    "size_bytes": 104857600,
    "segment_count": 31
  },
  "encryption": {
    "cipher": "aes-256-cbc",
    "pbkdf2_iterations": 100000,
    "iv_hex": "c0ffe..."
  },
  "signature": "-----BEGIN PGP SIGNATURE-----..."
}

Six-Step Restore Drill

Perform this drill on the first Tuesday of every quarter, or whenever theWAL_ARCHIVE_MODE or encryption keys are rotated. The drill environment is a isolated Kubernetes namespace (pmh-restore-drill) that is torn down automatically after the drill completes. Total drill time is approximately 45 minutes.

Step 1 — Pull the Latest Bundle

Using the PMH backup CLI (pmh-backup pull), download the most recent base backup bundle and its associated WAL segments into the drill namespace staging directory.

pmh-backup pull \
  --tenant tenant-acme \
  --target /var/lib/pmh/backup/restore-drill/ \
  --profile pmh-restore-drill \
  2&>1 | tee /tmp/restore-drill-step1.log

Expected output: Pull complete. 6 artifacts, 629 MB total. SHA-256 verification: PASS.

Step 2 — Verify GPG Signatures

Import the backup public key from Vault and verify both the base backup signature and the WAL archive signature.

# Import public key
gpg --import /etc/pmh/backup/pubring.gpg

# Verify base backup
gpg --verify \
  /var/lib/pmh/backup/restore-drill/base_backup.sig \
  /var/lib/pmh/backup/restore-drill/base_backup.tar.gz.gpg

# Verify WAL archive
gpg --verify \
  /var/lib/pmh/backup/restore-drill/wal_segments.sig \
  /var/lib/pmh/backup/restore-drill/wal_segments.tar.gz

Both gpg --verify commands must reportGood signature from backup@pmh.internal with no warnings. Any warning or failure must stop the drill immediately.

Step 3 — Decrypt Base Backup

Decrypt the base backup using the master key retrieved from Vault at runtime. The passphrase is injected via an environment variable sourced from the Vault agent sidecar, never written to disk.

VAULT_ADDR=https://vault.internal:8200 \
vault read -field=passphrase secret/pmh/backup/passphrase >/tmp/bk_passphrase

openssl aes-256-cbc -d \
  -in /var/lib/pmh/backup/restore-drill/base_backup.tar.gz.gpg \
  -out /var/lib/pmh/backup/restore-drill/base_backup.tar.gz \
  -pass file:/tmp/bk_passphrase \
  -pbkdf2 -iter 100000

sha256sum /var/lib/pmh/backup/restore-drill/base_backup.tar.gz

Confirm the SHA-256 matches the value in manifest.json. Wipe the passphrase file immediately after use:

shred -u /tmp/bk_passphrase

Step 4 — Extract and Prepare WAL Replay

tar -xzf /var/lib/pmh/backup/restore-drill/wal_segments.tar.gz \
  -C /var/lib/pmh/backup/restore-drill/

# Verify WAL segment filenames are contiguous
ls /var/lib/pmh/backup/restore-drill/wal/ \
  | sort -u \
  | awk 'NR==1{prev=$0;sub(/[^0-9].*/,"",prev)} \
         $0!=prev+1{print "GAP at " prev "->" $0; exit 1} \
         {prev=$0}' \
  || { echo "WAL gap detected — ABORT"; exit 1; }

Step 5 — Execute Point-in-Time Restore

Start a temporary PostgreSQL instance in recovery mode, replaying WAL segments up to the timestamp encoded in the bundle ID. The drill simulates a restore to T-15 minutes from the backup time.

recovery_time="2026-07-17T11:45:00Z"

pg_ctlcluster 16 pmh-restore start \
  --without-gss-enc \
  --恢復-target-lsn=$recovery_time \
  -o "--wal-dir=/var/lib/pmh/backup/restore-drill/wal/ \
      --restore-command=restore_command.sh \
      --recovery-target-time=$recovery_time"

# Wait for recovery to complete
sleep 30
pg_ctlcluster 16 pmh-restore status

Step 6 — Verify Data Integrity

# Verify row counts match expected values in manifest
psql -h localhost -U pmh_admin -d pmh -c \
  "SELECT COUNT(*) FROM credits.ledger;" \
  | grep -E '^[0-9]+$'

# Verify WAL range end LSN matches manifest
psql -h localhost -U pmh_admin -d pmh -c \
  "SELECT pg_last_wal_replay_lsn();"

Expected: row counts match manifest.json seeded values, andpg_last_wal_replay_lsn() falls within thewal_range.end segment range. If either check fails, capture the full output and escalate to the SRE on-call before proceeding.

Failure Handling Matrix

Key Rotation

GPG master keys and AES passphrases must be rotated annually or immediately after any potential compromise. The rotation procedure is:

1. Generate new GPG key pair on the HSM (never on a networked machine).
2. Update secret/pmh/backup/master in Vault.
3. Update secret/pmh/backup/passphrase in Vault.
4. Run a manual backup: pmh-backup run --full.
5. Verify the new bundle passes the restore drill steps 2 and 3 above.
6. Archive (do not delete) the old key material with a 7-year retention label in the HSM audit log.

Related Documentation

• Production Hardening Flags — WAL_ARCHIVE_MODE flag and associated risk levels.
• Air-Gap Update Mechanism — backup bundles are used as the source of truth for air-gap restore scenarios.
• Incident Response Playbook — escalation path when a backup failure is detected during an incident.