Incident Response Playbook

This playbook defines how the PostMTA Hybrid operations team detects, triages, contains, and recovers from production incidents. It covers the four severity levels, the detection sources and monitoring paths, the triage procedure, the containment steps to stop active bleeding, the customer notification templates for each severity, the recovery procedure, the postmortem template, and the on-call rotation structure. Every operator must be familiar with this playbook before taking their first on-call shift.

Reference Identifiers

The authoritative references for this page are:

• PMH-SEC-030 — Health Check Protocol
• PMH-SEC-031 — GPG Backup & Restore
• PMH-SEC-037 — Customer Communication SLA
• PMH-SEC-038 — Postmortem and Root Cause Analysis

Severity Definitions

Severity (SEV) levels define the impact scope and the response time SLA. The on-call engineer assigns the initial SEV when the incident is declared. The SEV may be escalated or de-escalated as the impact becomes clearer during triage. Changing the SEV requires the on-call SRE lead to be informed.

Detection Sources

Incidents are detected through one or more of the following paths. The primary detection sources are monitored 24/7 by the PagerDuty alerting pipeline. Secondary sources require manual review.

Triage Procedure

Triage must be completed within 15 minutes of the first alert. The goal of triage is to determine the scope and impact — which tenants are affected, which component is the root cause, and whether the incident is still worsening. Do not attempt to fix anything during triage; only observe and document.

Scope Determination

1. Identify the blast radius.Run the following query against the PostgreSQL credit ledger to count affected tenants by their credit enforcement status:

   psql -h localhost -U pmh_admin -d pmh -c \
     "SELECT \
        tenant_id, \
        credit_balance, \
        credit_enforcement, \
        updated_at \
      FROM credits.ledger \
      WHERE credit_enforcement = true \
        AND credit_balance < 1000 \
      ORDER BY updated_at DESC \
      LIMIT 20;"

2. Identify the failing component.Run the component health checks from the Production Runbook for each subsystem in sequence: Kumo MTA, Caddy, Astro, PostgreSQL. The first subsystem that does not return status: ok is the likely root cause.

   # Check Kumo shaping health
   kubectl exec -n pmh-system \
     $(kubectl get pod -n pmh-system -l app=pmh-core -o jsonpath='{.items[0].metadata.name}') \
     -- curl -sf http://localhost:8081/health/shaping
   
   # Check Caddy admin
   curl -sf http://localhost:9080/admin/config/load
   
   # Check Astro health
   curl -sf http://localhost:4321/health

3. Check the deployment timeline.If a recent Helm deployment coincides with the incident start time, the incident is deployment-related and the rollback procedure from the Production Runbook must be initiated immediately.

   helm history pmh -n pmh-system --max=5
   # Compare REVISION, DEPLOYED, and STATUS columns to the incident start time

4. Document the incident.Open a new incident document in the PMH incident log (Google Doc template linked in PagerDuty). Record: incident start time, SEV, initial scope, affected tenants, suspected root cause. This document is the single source of truth for the duration of the incident.

Impact Assessment

Determine if the incident is actively worsening by checking the queue depth trend over the last 15 minutes:

# Query queue depth every 30 seconds for 2 minutes
for i in 1 2 3 4; do
  ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  depth=$(kubectl exec -n pmh-system \
    $(kubectl get pod -n pmh-system -l app=pmh-core -o jsonpath='{.items[0].metadata.name}') \
    -- curl -sf http://localhost:8081/admin/queue/depth 2>/dev/null \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['depth'])")
  echo "$ts $depth"
  sleep 30
done

If the queue depth is growing by more than 1 000 messages per minute, the incident is worsening and containment must begin immediately, even if triage is not yet complete.

Containment

Containment stops the active bleed. There are three containment phases: stop-the-bleeding, isolate, and protect backups.

Stop-the-Bleeding

These actions are taken in the first five minutes regardless of root cause to prevent further customer harm. They can be taken by the primary on-call without SRE lead approval.

1. Throttle outbound volume to 10% of normal.

   # Set shaping rate to 10% of baseline
   kubectl exec -n pmh-system \
     $(kubectl get pod -n pmh-system -l app=pmh-core -o jsonpath='{.items[0].metadata.name}') \
     -- curl -sf -X POST http://localhost:8081/admin/shaping/rate \
        -H "Content-Type: application/json" \
        -d '{"rate_percent": 10}'
   
   # Confirm shaping is active
   kubectl exec -n pmh-system \
     $(kubectl get pod -n pmh-system -l app=pmh-core -o jsonpath='{.items[0].metadata.name}') \
     -- curl -sf http://localhost:8081/admin/shaping/status

2. Enable CREDIT_ENFORCEMENT if not already enabled.If the incident involves credit over-spending, ensure the enforcement flag is blocking new sends:

   helm upgrade pmh oci://registry.internal/pmh/helm/pmh \
     --namespace pmh-system \
     --set pmh.runtime.CREDIT_ENFORCEMENT=true \
     --timeout 3m --wait

3. Close the tenant API intake.If the Astro web application is still responding, disable new campaign creation at the application layer to prevent new message injection:

   kubectl exec -n pmh-system \
     $(kubectl get pod -n pmh-system -l app=pmh-astro -o jsonpath='{.items[0].metadata.name}') \
     -- curl -sf -X POST http://localhost:4321/internal/admin/campaigns/pause-all

Isolate

Isolate the failing component to prevent cascade failure into healthy components.

• If Kumo MTA is the root cause: scale Kumo to a single replica to reduce resource contention, then drain the queue:

  kubectl scale deployment/pmh-core -n pmh-system --replicas=1
  kubectl exec -n pmh-system \
    $(kubectl get pod -n pmh-system -l app=pmh-core -o jsonpath='{.items[0].metadata.name}') \
    -- curl -sf -X POST http://localhost:8081/admin/queue/drain

• If PostgreSQL is the root cause: block all writes at the Caddy layer except the WAL archival connection:

  kubectl exec -n pmh-system \
    $(kubectl get pod -n pmh-system -l app=pmh-caddy -o jsonpath='{.items[0].metadata.name}') \
    -- caddy adapt --config /etc/caddy/Caddyfile \
       --adapter caddyfile 2>&1 | \
       grep -A5 "handle_path /internal/credits"

Protect Backups

Verify the WAL archival process is still running and that the most recent backup bundle was uploaded successfully. If WAL archival has stopped, the RPO is at risk and the SRE lead must be notified immediately.

# Check pg_stat_archiver last archive time
psql -h localhost -U pmh_admin -d pmh -c \
  "SELECT last_archived_wal, last_archived_time, \
          archive_error \
   FROM pg_stat_archiver;"

# If last_archived_time is more than 10 minutes ago, escalate immediately
psql -h localhost -U pmh_admin -d pmh -c \
  "SELECT now() - last_archived_time AS archive_age \
   FROM pg_stat_archiver;"

Customer Notification Templates

Notify affected customers within the SLA response time for the declared SEV. Use the template that matches the SEV. Replace the bracketed variables before sending. All notifications must be sent from support@pmhc.ioand CC ops@pmhc.io. Do not send notifications until the incident document has been created and the on-call SRE lead has been briefed.

SEV-1 — Total Outage (Template)

Subject: [URGENT] PostMTA Service Outage — [TENANT_NAME] — [INCIDENT_ID]

Dear [CUSTOMER_NAME],

We are experiencing a total service outage affecting your PostMTA account
([TENANT_ID]) starting at [INCIDENT_START_TIME] UTC.

What we know:
- The outage was detected at [DETECTION_TIME] UTC.
- Our engineering team is actively responding with SRE leadership engaged.
- A full root cause analysis will be published after service is restored.

Current status: [RESTORING / INVESTIGATING]
Estimated resolution: [ETA] UTC

We will update you every 30 minutes until service is restored.

We sincerely apologise for the impact this is having on your operations.
If you have any questions, reply to this email or call our 24/7 support line.

— PostMTA Operations Team
Incident reference: [INCIDENT_ID]
Support: support@pmhc.io | +1-800-PMH-OPS

SEV-2 — Major Degradation (Template)

Subject: [IMPORTANT] PostMTA Performance Degradation — [TENANT_NAME]

Dear [CUSTOMER_NAME],

We are currently experiencing elevated bounce rates and reduced delivery speed
for your PostMTA account ([TENANT_ID]).

What we know:
- Bounce rate is currently [BOUNCE_RATE]% (threshold: 5%)
- Affected sending domain(s): [DOMAIN_LIST]
- Cause: [BRIEF_DESCRIPTION — e.g. "upstream API rate limiting" or "database replica lag"]

Our team is actively working to resolve this. We currently estimate
full service restoration by [ETA] UTC.

Interim recommendations:
- Hold non-time-sensitive campaigns until [ETA]
- We are throttling outbound volume to 10% to prevent further bounce impact

We will update you in 60 minutes or when the situation changes.

— PostMTA Operations Team
Incident reference: [INCIDENT_ID]
Support: support@pmhc.io

SEV-3 — Minor Degradation (Template)

Subject: PostMTA Advisory — Elevated Queue Depth — [TENANT_NAME]

Dear [CUSTOMER_NAME],

We have observed an elevated queue depth for your account ([TENANT_ID])
and wanted to proactively notify you.

Current queue depth: [QUEUE_DEPTH] messages
Expected queue drain time: [ETA] based on current throughput

No action is required from your side. We are monitoring the situation
and will update you if it escalates.

If your delivery SLAs are at risk, please reply to this email and we
will prioritise your tenant.

— PostMTA Operations Team
Support: support@pmhc.io

Recovery Procedure

Recovery begins after containment is stable and the blast radius is known. Recovery has three parallel workstreams: queue replay, database restore, and health verification.

Queue Replay

After the root cause is resolved, replay messages from the in-memory Kumo queue and the PostgreSQL outbox table. Messages that were suppressed due to credit enforcement are automatically replayed after the credit balance is restored.

# Resume normal shaping rate
kubectl exec -n pmh-system \
  $(kubectl get pod -n pmh-system -l app=pmh-core -o jsonpath='{.items[0].metadata.name}') \
  -- curl -sf -X POST http://localhost:8081/admin/shaping/rate \
     -H "Content-Type: application/json" \
     -d '{"rate_percent": 100}'

# Flush the outbox table
psql -h localhost -U pmh_admin -d pmh -c \
  "SELECT pmh.flush_outbox(max_batch_size => 5000);"

# Verify queue depth is declining
kubectl exec -n pmh-system \
  $(kubectl get pod -n pmh-system -l app=pmh-core -o jsonpath='{.items[0].metadata.name}') \
  -- curl -sf http://localhost:8081/admin/queue/depth

Database Restore

If the incident required a point-in-time restore from the GPG backup bundle, follow the restore procedure in the GPG Backup & Restore documentation. The on-call DBA must approve any restore operation. After restore, run the data integrity checks (Step 6 of the restore drill) before declaring the database healthy.

# Initiate PITR restore to T-15 minutes from incident start
recovery_target="2026-07-17T11:45:00Z"
pg_ctlcluster 16 pmh restore \
  --waldir=/var/lib/pmh/wal/restore \
  --recovery-target-time=$recovery_target \
  --recovery-target-action=promote

Health Verification

After recovery, re-run all eight critical metrics checks from the Production Runbook. All eight must be within healthy thresholds before the incident is declared resolved.

# Run all critical metric checks
for metric in queue_depth bounce_rate complaint_rate credit_balance WAL_lag \
              replica_lag Caddy_error_rate autovacuum_stall; do
  python3 /usr/local/bin/pmh-metric-check --metric=$metric \
    --threshold-file /etc/pmh/metrics/thresholds.yaml \
    || { echo "FAIL: $metric"; exit 1; }
done
echo "All 8 critical metrics: PASS"

Postmortem Template

A postmortem is required for every SEV-1 and SEV-2 incident. It must be published within 72 hours of the incident being declared resolved. The postmortem is a blameless document; its purpose is systemic improvement, not individual accountability.

Postmortem Document Structure

INCIDENT POSTMORTEM
====================
Incident ID:        [INCIDENT_ID]
Date:               [DATE]
Duration:           [START] – [END] UTC ([TOTAL_DURATION])
Severity:           [SEV-1 / SEV-2]
Lead Investigator:  [NAME]
Reviewers:          [SRE_LEAD], [ENG_MANAGER]

1. SUMMARY
   One-paragraph description of what happened, the customer impact,
   and the resolution.

2. TIMELINE (UTC)
   [TIME]  Alert fired
   [TIME]  Primary on-call acknowledged
   [TIME]  SEV-1 declared
   [TIME]  Root cause identified
   [TIME]  Containment actions applied
   [TIME]  Customer notifications sent
   [TIME]  Recovery initiated
   [TIME]  Service restored
   [TIME]  Incident declared resolved

3. ROOT CAUSE
   Technical description of the failure chain.
   What was the triggering event?
   What was the amplifying factor?

4. CONTRIBUTING FACTORS
   - Factor 1
   - Factor 2

5. RESPONSE EFFECTIVENESS
   - What went well?
   - What was slow or blocked?

6. ACTION ITEMS
   | Action | Owner | Due Date | Ticket |
   |--------|-------|----------|--------|
   | Add WAL lag alert at 15s | [name] | [date] | PMH-XXXX |
   | Improve backup verification | [name] | [date] | PMH-YYYY |

7. CUSTOMER IMPACT SUMMARY
   Tenants affected: [N]
   Messages delayed: [N]
   Billing: [description]

8. PREVENTIVE MEASURES
   List of systemic improvements to prevent recurrence.

On-Call Rotation

The PMH operations team uses PagerDuty for on-call scheduling. The rotation is weekly, starting Monday at 00:00 UTC. The on-call schedule is:

Handing Off On-Call

# Confirm the incoming on-call has accepted the PagerDuty schedule
pd schedules list --name "PMH Primary On-Call" | grep current

# Verify alerts are routing to the incoming on-call
pd incidents list --status triggered --output json \
  | python3 -c "import sys,json; \
      active=[i for i in json.load(sys.stdin) if i['status']=='triggered']; \
      assert len(active)==0, f'Has unresolved incidents: {active}'; \
      print('No open incidents. Safe to hand off.')"

Handoff is complete when the incoming on-call acknowledges the PagerDuty schedule and confirms there are no open incidents. Document the handoff time in the #pmh-ops Slack channel.

Related Documentation

• Production Runbook — component health checks and rollback procedure referenced in this playbook.
• GPG Backup & Restore — WAL archival, backup bundle verification, and PITR restore procedure for database recovery.
• Production Hardening Flags — CREDIT_ENFORCEMENT flag and other flags referenced during containment steps.
• Air-Gap Update Mechanism — helm rollback procedure referenced when rolling back a failed upgrade during an incident.