Production Runbook

This runbook defines the operational procedures for running a PMH appliance in production. It covers the pre-flight checklist you must complete before any deployment, the health checks for each subsystem, the eight critical metrics with their alert thresholds, the Helm deployment command, and the rollback procedure. Read this runbook in full before your first production deployment and keep it open during every subsequent deployment window.

Reference Identifiers

The authoritative references for this page are:

• PMH-SEC-025 — Production Hardening Flags
• PMH-SEC-030 — Health Check Protocol
• PMH-SEC-034 — Critical Metrics and Alerting

Pre-Flight Checklist

Complete every row in the checklist before initiating a Helm deployment. Check items may be performed in parallel by separate engineers; each item must be signed off individually in the deployment log.

Component Health Checks

After every deployment, verify each of the four core subsystems independently. Run these checks in the order listed. Do not proceed past a failing check without resolving it or escalating to the on-call SRE.

Kumo MTA (port 8081)

# Kumo shaping health
kubectl exec -n pmh-system \
  $(kubectl get pod -n pmh-system -l app=pmh-core -o jsonpath='{.items[0].metadata.name}') \
  -- curl -sf http://localhost:8081/health/shaping

# Expected output:
# {"status":"ok","queue_depth":1247,"uptime_seconds":86400}

# Kumo delivery health
kubectl exec -n pmh-system \
  $(kubectl get pod -n pmh-system -l app=pmh-core -o jsonpath='{.items[0].metadata.name}') \
  -- curl -sf http://localhost:8081/health/delivery

# Expected output:
# {"status":"ok","deliveries_total":48291,"bounces_total":1204}

Caddy Reverse Proxy (port 9080)

# Caddy admin API
curl -sf http://localhost:9080/admin/config/load

# Expected: {"mode":"load","status":"success"}

# TLS certificate expiry (must be > 30 days)
echo | openssl s_client -servername pmh.example.com \
  -connect pmh.example.com:443 2>/dev/null \
  | openssl x509 -noout -dates

# Expected: notBefore and notAfter fields, notAfter > 30 days from today

Astro Web Application (port 4321)

curl -sf http://localhost:4321/health | python3 -c \
  "import sys,json; d=json.load(sys.stdin); \
   assert d['status']=='ok', f'Unexpected: {d}'; \
   print('Astro health OK')"

PostgreSQL (port 5432)

# Replication slot lag
psql -h localhost -U pmh_admin -d pmh -c \
  "SELECT slot_name, pg_size_pretty(pg_wal_lsn_diff( \
     pg_current_wal_lsn(), restart_lsn)) AS lag \
   FROM pg_replication_slots;"

# Autovacuum progress
psql -h localhost -U pmh_admin -d pmh -c \
  "SELECT relname, last_autovacuum, last_autoanalyze \
   FROM pg_stat_user_tables \
   ORDER BY last_autovacuum DESC LIMIT 5;"

Eight Critical Metrics

These eight metrics must be monitored at all times during a production deployment window. If any metric crosses its threshold, pause the deployment and consult the on-call SRE before proceeding.

Helm Deployment Commands

Use the following command template for all production deployments. Replace the --set values with the values appropriate for the current release. Never use --force in production unless explicitly directed by the on-call SRE during an incident.

# Pre-flight: ensure namespace exists and context is correct
kubectl config use-context pmh-prod-us-east-1
kubectl create namespace pmh-system --dry-run=client -o yaml | kubectl apply -f -

# Dry-run the upgrade (no changes made)
helm upgrade pmh \
  oci://registry.internal/pmh/helm/pmh \
  --namespace pmh-system \
  --set pmh.version=2.4.1 \
  --set pmh.runtime.MIB_CHECK_ONLY=false \
  --set pmh.runtime.ALLOW_DEV_SEED=false \
  --set pmh.runtime.ENFORCE_TLS12_MIN=true \
  --set pmh.runtime.CREDIT_ENFORCEMENT=true \
  --set pmh.backup.encryption.cipher=aes-256-cbc \
  --dry-run \
  --debug 2&1 | less

# Live deployment
helm upgrade pmh \
  oci://registry.internal/pmh/helm/pmh \
  --namespace pmh-system \
  --set pmh.version=2.4.1 \
  --set pmh.runtime.MIB_CHECK_ONLY=false \
  --set pmh.runtime.ALLOW_DEV_SEED=false \
  --set pmh.runtime.ENFORCE_TLS12_MIN=true \
  --set pmh.runtime.CREDIT_ENFORCEMENT=true \
  --set pmh.backup.encryption.cipher=aes-256-cbc \
  --timeout 15m \
  --wait \
  --atomic \
  --cleanup-on-fail

The --atomic flag causes Helm to automatically roll back to the previous successful release if the upgrade fails. The--cleanup-on-fail flag ensures failed test pods are removed before the rollback begins.

Monitoring the Deployment

# Watch pod status during rollout
kubectl get pods -n pmh-system -l app=pmh-core -w

# Watch Helm release status
helm status pmh -n pmh-system

# Follow Kumo logs during deployment (one replica at a time)
REPLICA=0
kubectl exec -n pmh-system \
  pmh-core-6878d94f5c-ltz9x -c kumo-mta -- \
  tail -f /var/log/pmh/kumo.log 2&1 | \
  while read line; do
    echo "[$REPLICA] $line"
  done

Rollback Procedure

Rollback is initiated automatically when --atomic is set and the upgrade fails. For manual rollbacks, or if the atomic rollback fails, use the following procedure.

# Step 1: List the Helm revision history
helm history pmh -n pmh-system

# Step 2: Roll back to the previous (known-good) revision
helm rollback pmh \
  --namespace pmh-system \
  --timeout 10m \
  --wait

# Step 3: Verify the rollback pod is ready
kubectl rollout status deployment/pmh-core \
  --namespace pmh-system --timeout=180s

# Step 4: Verify credit ledger is accessible
curl -sf http://localhost:9080/internal/credits/last_sync \
  | python3 -c "import sys,json; d=json.load(sys.stdin); \
     assert d.get('last_sync_at'), 'Credit sync endpoint not responding'; \
     print('Credit sync endpoint OK')"

# Step 5: Re-run the component health checks (Kumo, Caddy, Astro, PostgreSQL)
# See the Component Health Checks section above

Post-Deployment Verification

After a successful deployment, run the following verification steps within 30 minutes of the rollout completing.

1. Confirm queue_depth is below 5 000 and trending flat or down.
2. Confirm bounce_rate is below 2% over the last 15 minutes.
3. Confirm the last backup bundle uploaded to S3 was created after the deployment completed.
4. Confirm no new FBL or suppression entries were added by the upgrade.
5. Close the deployment window in PagerDuty and log the Helm revision number, the timestamp, and the operator name.

Related Documentation

• Production Hardening Flags — the nine flags that control runtime behaviour and their risk levels.
• GPG Backup & Restore — backup schedule and restore drill procedure.
• Incident Response Playbook — escalation path when a critical metric alert fires during a deployment.
• Air-Gap Update Mechanism — update procedure for air-gapped deployments.