DISASTER_RECOVERY.md (21122B)
1 # Disaster Recovery Runbook 2 3 Last updated: 2026-03-21 4 5 --- 6 7 ## Overview 8 9 This cluster has three layers of backup: 10 11 | Layer | Tool | Scope | Frequency | Retention | Storage | 12 | --------------------- | --------------------------------------------- | ------------------------------------- | ------------------------ | ---------- | ----------------------------------- | 13 | Application + Volumes | Velero (Kopia) | All namespaces + hcloud-volumes | 4× daily (0,6,12,18 UTC) | 14 days | `mtrnord-talos-velero` S3 bucket | 14 | Longhorn Volumes | Longhorn backup (incremental, full every 14d) | All Longhorn volumes (group: default) | Daily 02:00 UTC | 14 backups | `mtrnord-longhorn-backup` S3 bucket | 15 | Longhorn Config | Longhorn system-backup | Longhorn settings + metadata | Daily 05:00 UTC | 7 backups | `mtrnord-longhorn-backup` S3 bucket | 16 | PostgreSQL WAL | CNPG barman-cloud | postgres-cluster | Continuous + scheduled | 30 days | `mtrnord-talos-pg-backup` S3 bucket | 17 18 > **Key constraint**: hcloud-volumes CSI does NOT support snapshots. Velero uses Kopia (file-system copy into S3) for those volumes, which requires applications to be quiesced or tolerant of slightly inconsistent snapshots. 19 20 --- 21 22 ## Scenario 1: Single Application Recovery (Velero Kopia) 23 24 Use this when one app's data is corrupt or accidentally deleted. 25 26 ### Quick reference 27 28 ```bash 29 # List available backups 30 velero backup get 31 32 # See what's in a backup 33 velero backup describe cluster-backup-0000-20260228000000 --details 34 35 # Restore a single namespace (full — recommended, see gotchas below) 36 velero restore create my-restore \ 37 --from-backup cluster-backup-0600-20260228060000 \ 38 --include-namespaces=<namespace> \ 39 --existing-resource-policy=none 40 41 # Monitor kopia (file) restore progress 42 kubectl -n velero get podvolumerestore -o wide 43 44 # Restore to a different namespace (test without overwriting live data) 45 velero restore create --from-backup cluster-backup-0600-20260228060000 \ 46 --include-namespaces=myapp \ 47 --namespace-mappings myapp:myapp-restored 48 ``` 49 50 ### Critical: how Velero Kopia volume restore actually works 51 52 Kopia does NOT restore data directly into a PVC. Instead: 53 54 1. Velero restores the **pod** (with an injected `restore-wait` init container) 55 2. The node-agent writes backup data into the PVC while the init container waits 56 3. Only after kopia completes does the main container start 57 58 **Consequence:** restoring only PVCs/PVs (`--include-resources=persistentvolumeclaims,persistentvolumes`) creates an empty PVC shell — no data is written. **Always include pods** or restore the full namespace. 59 60 ### Correct procedure for a namespace with a running Deployment 61 62 If Flux has already reconciled the namespace (Deployment + RS exist at current replicas), a naive restore will fail silently: the RS kills the restored pod before kopia can run. 63 64 **Step 1: Suspend Flux to prevent it from fighting the restore** 65 66 ```bash 67 flux suspend kustomization apps 68 ``` 69 70 **Step 2: Delete the Deployment and all ReplicaSets in the namespace** 71 72 This is necessary because `--existing-resource-policy=none` (Velero default) will skip existing resources, leaving the RS at `replicas=0`, which immediately deletes the restored pod. 73 74 ```bash 75 kubectl -n <namespace> delete deploy <name> 76 kubectl -n <namespace> delete rs --all 77 kubectl -n <namespace> delete pvc <name> # also delete the (possibly empty/corrupt) PVC 78 ``` 79 80 **Step 3: Run the restore** 81 82 ```bash 83 velero restore create <restore-name> \ 84 --from-backup <backup-name> \ 85 --include-namespaces <namespace> \ 86 --existing-resource-policy=none 87 ``` 88 89 Velero will restore the Deployment (with `replicas=1` from backup), RS (desired=1), pod (with kopia init container), and PVC. Since the RS wants exactly 1 pod and Velero provides exactly 1, the RS will not kill the pod. 90 91 **Step 4: Monitor kopia progress** 92 93 ```bash 94 # Watch bytes written 95 kubectl -n velero get podvolumerestore -o wide -w 96 97 # Check pod init container status 98 kubectl -n <namespace> get pods -w 99 # Pod should be in Init:0/1 while kopia runs, then 0/1 while app starts 100 ``` 101 102 **Step 5: After data confirmed, resume Flux** 103 104 ```bash 105 # Verify data is present (app starts, or run a debug pod to inspect PVC) 106 kubectl -n <namespace> logs <pod> 107 108 # Resume Flux — it will reconcile the Deployment back to the state in git 109 flux resume kustomization apps 110 ``` 111 112 ### Stuck restore: finalizer deadlock 113 114 If a restore gets stuck in `InProgress (Deleting)` due to a failed kopia operation (e.g., the pod was killed before kopia could write anything): 115 116 ```bash 117 # Force-remove the finalizer — safe if no kopia operations were in flight 118 kubectl -n velero patch restore <restore-name> \ 119 --type=json -p='[{"op":"remove","path":"/metadata/finalizers"}]' 120 ``` 121 122 This is safe when the restored pod was deleted immediately (kopia never started) — there is no kopia state to clean up. 123 124 ### Backup consistency: databases being written during backup 125 126 Velero Kopia takes a file-system snapshot of a live volume. For databases like RocksDB (continuwuity/conduwuit), the backup may capture an inconsistent state if: 127 128 - The database was in recovery mode during backup 129 - The CURRENT file was written pointing to a MANIFEST that was later renamed/replaced before the snapshot completed 130 131 **Symptom:** restored pod fails with `IO error: No such file or directory: /data/MANIFEST-XXXXXX` 132 133 **Fix:** try an older backup (e.g., 06:00 instead of 18:00) taken when the database was in a stable state. Check backup timestamps vs. when the problem started. 134 135 ```bash 136 # List all backups with timestamps 137 velero backup get 138 139 # Inspect which backup point to use 140 velero backup describe <backup-name> | grep Created 141 ``` 142 143 --- 144 145 ## Scenario 2: PostgreSQL Point-in-Time Recovery 146 147 CNPG continuously archives WALs to S3. This allows recovery to any point within the 30-day retention window. 148 149 ### 2a. Recover to latest (e.g. after accidental table drop) 150 151 ```bash 152 # Stop services that use postgres (reduce connections) 153 # Then edit cnpg-cluster.yaml to add a recovery bootstrap: 154 155 bootstrap: 156 recovery: 157 source: pg-s3-backup 158 recoveryTarget: 159 targetTLI: latest # or use targetTime for PITR 160 161 externalClusters: 162 - name: pg-s3-backup 163 plugin: 164 name: barman-cloud.cloudnative-pg.io 165 parameters: 166 barmanObjectName: hetzner-base-backup 167 serverName: pg-cluster-v2 # original WAL archive path 168 ``` 169 170 Also change `plugins.parameters.serverName` to something new (e.g. `pg-cluster-v2-restored`) to avoid "expected empty archive" error. 171 172 ### 2b. Point-in-Time Recovery (PITR) 173 174 ```yaml 175 bootstrap: 176 recovery: 177 source: pg-s3-backup 178 recoveryTarget: 179 targetTime: "2026-02-27T20:00:00Z" # RFC 3339, adjust as needed 180 ``` 181 182 ### 2c. Important lessons from past migrations 183 184 - **DO NOT use `bootstrap.recovery.backup.name`** with the barman-cloud plugin — causes "missing Azure credentials" error. Always use `externalClusters` + `source`. 185 - **Set a different `serverName` in `plugins`** from the recovery `serverName` — otherwise WAL archiver fails with "expected empty archive". 186 - **If timeline mismatch error**: add `targetTLI: latest` or use `targetTime` before the timeline switch. 187 - **Deleting the CNPG Cluster object ALSO deletes the PVCs** by default. Before deleting for migration, verify `deletionPolicy` or set it to `retain`. 188 - After recovery: remove the `bootstrap` and `externalClusters` sections from `cnpg-cluster.yaml` once the cluster is healthy. 189 190 ### 2d. Full postgres recovery procedure 191 192 ```bash 193 # 1. Scale down apps using postgres 194 flux suspend kustomization apps 195 196 # 2. Delete the existing (broken) cluster 197 kubectl -n postgres-cluster delete cluster pg-cluster-v2 198 199 # 3. Edit gitops/infrastructure_talos/configs/cnpg-cluster.yaml: 200 # - Add bootstrap + externalClusters as above 201 # - Set storage.storageClass: hcloud-volumes (or longhorn if migrated) 202 # - Set plugins.parameters.serverName: pg-cluster-v2-restored 203 204 # 4. Commit + push and let Flux apply 205 git add infrastructure_talos/configs/cnpg-cluster.yaml 206 git commit -m "recovery: restore pg-cluster-v2 from S3 backup" 207 git push 208 209 # 5. Watch recovery 210 kubectl -n postgres-cluster get cluster pg-cluster-v2 -w 211 kubectl -n postgres-cluster logs -l cnpg.io/cluster=pg-cluster-v2 -f 212 213 # 6. Once Ready: resume apps 214 flux resume kustomization apps 215 216 # 7. Cleanup: remove bootstrap/externalClusters from cnpg-cluster.yaml and commit 217 ``` 218 219 --- 220 221 ## Scenario 3: Longhorn Volume Recovery 222 223 ### Longhorn Recurring Jobs 224 225 Backup target: `s3://mtrnord-longhorn-backup` (Hetzner Object Storage HEL1) 226 227 **Current configuration:** 228 229 | Job name | Type | Schedule | Group | Retain | Concurrency | Notes | 230 | --------------------- | ----------------- | ------------------- | ------- | ------ | ----------- | ---------------------------------------------------------------------------------------- | 231 | `volume-backup` | `backup` | `0 2 * * *` (02:00) | default | 14 | 2 | Volume data → S3. Primary recovery source. Incremental, full every 14 days. | 232 | `post-backup-cleanup` | `snapshot-delete` | `0 3 * * *` (03:00) | default | 2 | 2 | Enforces max 2 snapshots per volume after backup runs. Prevents copy/move failures. | 233 | `system-backup` | `system-backup` | `0 5 * * *` (05:00) | — | 7 | — | Longhorn config/metadata backup. `volume-backup-policy: if-not-present`. | 234 | `filesystem-trim` | `filesystem-trim` | `0 4 * * *` (04:00) | default | — | 2 | Reclaim space from deleted files. Runs between backup (02:00) and system-backup (05:00). | 235 236 **Global Longhorn settings:** 237 238 - Max snapshots per volume: **5** (hard ceiling, monitoring before raising — `snapshot-delete` retain=2 enforces the soft limit, leaving 3 slots for system snapshots during replica rebuilds) 239 - Backup target: `s3://mtrnord-longhorn-backup` (Hetzner Object Storage HEL1) 240 241 **Why no `snapshot` or `snapshot-cleanup` job:** 242 243 - No `snapshot` job: with a low global snapshot limit, an hourly retain=24 would immediately hit the ceiling. Velero (6h) + Longhorn backup (daily) provide sufficient recovery points without in-cluster snapshots. 244 - No `snapshot-cleanup`: redundant when `backup` job does pre-backup cleanup and `snapshot-delete` enforces the count hard limit. 245 246 ### 3a. Restore a single Longhorn volume from backup 247 248 **Via Longhorn UI (easiest):** 249 250 1. Go to Longhorn UI → Backup 251 2. Find the volume backup 252 3. Click Restore → enter a name for the restored volume 253 4. Once restored, create a PVC pointing to the new volume or update the app's PVC 254 255 **Via kubectl:** 256 257 ```bash 258 # List available backups 259 kubectl -n longhorn-system get backups.longhorn.io 260 261 # Restore by creating a Volume CR pointing to the backup 262 kubectl apply -f - <<EOF 263 apiVersion: longhorn.io/v1beta2 264 kind: Volume 265 metadata: 266 name: restored-volume 267 namespace: longhorn-system 268 spec: 269 fromBackup: "s3://your-bucket?backup=backup-name&volume=volume-name" 270 numberOfReplicas: 2 271 size: "10Gi" 272 EOF 273 274 # Then create a PVC that binds to it (via Longhorn UI or static PV/PVC) 275 ``` 276 277 ### 3b. Restore from Longhorn system-backup 278 279 System backups capture the entire Longhorn state (volumes + settings). 280 281 ```bash 282 # List system backups 283 kubectl -n longhorn-system get systembackups.longhorn.io 284 285 # Restore a system backup (this restores Longhorn settings + volumes) 286 # Do this in Longhorn UI: Settings → System Backup → Restore 287 # OR via CR: 288 kubectl apply -f - <<EOF 289 apiVersion: longhorn.io/v1beta2 290 kind: SystemRestore 291 metadata: 292 name: restore-from-system-backup 293 namespace: longhorn-system 294 spec: 295 systemBackup: <system-backup-name> 296 EOF 297 298 kubectl -n longhorn-system get systemrestores -w 299 ``` 300 301 > **Warning**: System restore overwrites current Longhorn settings. Only use for full Longhorn recovery, not single-volume restore. 302 303 ### 3c. Restore Longhorn volume via Velero CSI snapshot 304 305 If Velero captured a CSI snapshot (via the `longhorn-velero-vsc` VolumeSnapshotClass): 306 307 ```bash 308 velero restore create --from-backup cluster-backup-0600-20260228060000 \ 309 --include-namespaces=<namespace> \ 310 --wait 311 ``` 312 313 Velero recreates the VolumeSnapshot and Longhorn restores the volume from it automatically. 314 315 ### 3d. Which method to use? 316 317 | Situation | Best method | 318 | ------------------------------- | -------------------------------------- | 319 | Single volume, recent data loss | Longhorn UI restore (3a) | 320 | Need data from >4 days ago | Velero restore (3c) — 14-day retention | 321 | Total Longhorn state loss | Longhorn system-backup restore (3b) | 322 | App namespace fully deleted | Velero namespace restore (Scenario 1) | 323 324 --- 325 326 ## Scenario 4: Full Cluster Recovery (Total Loss) 327 328 Use this when the entire cluster is gone and you need to rebuild from scratch. 329 330 ### Prerequisites 331 332 - Terraform state intact (in Hetzner Cloud or backed up) 333 - Access to S3 buckets: `mtrnord-talos-velero` and `mtrnord-talos-pg-backup` 334 - Age private key (stored separately — see below) 335 - All secrets in gitops repo are SOPS-encrypted — need age key to decrypt 336 337 ### Step 1: Rebuild infrastructure 338 339 ```bash 340 cd cluster2025-talos/cloud 341 terraform apply # recreates cloud VMs, network, firewall, kubeconfig 342 343 # For Proxmox nodes: re-apply Talos machineconfig 344 cd cluster2025-talos/proxmox 345 terraform apply -var-file=proxmox.tfvars 346 ``` 347 348 ### Step 2: Bootstrap Flux 349 350 ```bash 351 # Flux bootstrap will re-deploy all controllers and apps from gitops repo 352 flux bootstrap github \ 353 --owner=MTRNord \ 354 --repository=gitops \ 355 --branch=main \ 356 --path=clusters/talos_cluster 357 ``` 358 359 Or using your existing bootstrap method. 360 361 ### Step 3: Provide age key for SOPS decryption 362 363 ```bash 364 # The age private key must exist as a secret in flux-system 365 kubectl -n flux-system create secret generic sops-age \ 366 --from-file=age.agekey=/path/to/age.agekey 367 ``` 368 369 Without this, Flux cannot decrypt any SOPS-encrypted secrets (postgres credentials, velero credentials, etc.). 370 371 ### Step 4: Wait for core infrastructure 372 373 ```bash 374 # Wait for Velero, Longhorn, cert-manager, CNPG to be Ready 375 flux get kustomization 376 flux get helmrelease -A 377 kubectl -n velero get pods 378 kubectl -n longhorn-system get pods 379 ``` 380 381 ### Step 5: Restore from Velero 382 383 ```bash 384 # Find the most recent backup 385 velero backup get 386 387 # Full cluster restore (all namespaces) 388 velero restore create full-restore \ 389 --from-backup cluster-backup-0000-20260228000000 \ 390 --exclude-namespaces=velero,longhorn-system,kube-system,flux-system,cert-manager \ 391 --wait 392 393 # Flux will already manage velero/longhorn/etc — exclude those to avoid conflicts 394 ``` 395 396 ### Step 6: Restore PostgreSQL 397 398 After Velero restores the `postgres-cluster` namespace, the CNPG operator will see the Cluster CR but the PVCs may not exist. If so, follow Scenario 2d above to recover from S3 WAL archives. 399 400 If Velero successfully restored the PVCs (hcloud-volumes Kopia backup), CNPG should recover automatically once the operator is running. 401 402 ### Step 7: Verify 403 404 ```bash 405 # Check all pods running 406 kubectl get pods -A | grep -v Running | grep -v Completed 407 408 # Check postgres 409 kubectl -n postgres-cluster get cluster pg-cluster-v2 410 411 # Check apps 412 flux get ks apps 413 ``` 414 415 --- 416 417 ## Key Backup Locations 418 419 | What | Where | Path | 420 | ----------------------------------------- | --------------------------- | ------------------------------------------------------------ | 421 | Velero backups (all namespaces + volumes) | Hetzner Object Storage HEL1 | `mtrnord-talos-velero` bucket | 422 | Postgres WAL archives | Hetzner Object Storage HEL1 | `mtrnord-talos-pg-backup/pg-base-backup/pg-cluster-v2/` | 423 | Postgres scheduled base backups | same bucket | `mtrnord-talos-pg-backup/pg-base-backup/pg-cluster-v2/base/` | 424 | Longhorn volume backups | Hetzner Object Storage HEL1 | `mtrnord-longhorn-backup` bucket | 425 | Longhorn system backups | same bucket | `mtrnord-longhorn-backup` bucket | 426 | GitOps repo | GitHub | MTRNord/gitops | 427 | Terraform state | Hetzner Cloud S3 / local | cluster2025-talos/cloud/terraform.tfstate | 428 | Age private key | Local machine | `~/.config/sops/age/keys.txt` or `age.agekey` | 429 430 > **CRITICAL**: The age private key is the master key for all cluster secrets. Store it in a password manager (Bitwarden, etc.) in addition to the local file. Without it you cannot decrypt any secret in the cluster. 431 432 --- 433 434 ## Recovery Decision Tree 435 436 ``` 437 Something is broken 438 │ 439 ├── Single app data loss / corruption 440 │ └── Velero restore of that namespace (Scenario 1) 441 │ 442 ├── PostgreSQL data loss / corruption 443 │ ├── Minor (table drop, bad migration) 444 │ │ └── CNPG PITR to before the event (Scenario 2) 445 │ └── Major (cluster deleted, storage gone) 446 │ └── Full CNPG recovery from S3 (Scenario 2d) 447 │ 448 ├── Longhorn volume corrupted 449 │ └── Restore from Longhorn backup or Velero CSI snapshot (Scenario 3) 450 │ 451 └── Total cluster loss 452 └── Rebuild Terraform → Bootstrap Flux → Velero restore → CNPG restore (Scenario 4) 453 ``` 454 455 --- 456 457 ## Known Gotchas 458 459 - **hcloud-volumes Kopia backup consistency**: Kopia copies files from live pods. For databases other than postgres (which uses CNPG's own WAL-based backup), backups may be inconsistent if the app is writing during backup. Consider pre-backup hooks or accepting slight inconsistency. For RocksDB databases (conduwuit/continuwuity), a backup taken during recovery mode may reference a MANIFEST file that doesn't exist in the snapshot — try an earlier backup in that case. 460 - **Velero Kopia requires pods to write PVC data**: Kopia injects a `restore-wait` init container into restored pods to write data. Restoring only PVCs creates empty shells. Always restore the full namespace or explicitly include pods. 461 - **RS at replicas=0 kills restored pods**: If the Deployment/RS already exist in the cluster at `replicas=0` (e.g., Flux reconciled before restore), the RS deletes the kopia pod before data can be written. Fix: delete the Deployment and all RSes first, so Velero restores them fresh from backup at `replicas=1`. 462 - **Velero restore finalizer deadlock**: A restore stuck in `InProgress (Deleting)` has a `restores.velero.io/external-resources-finalizer` that blocks deletion. Remove with `kubectl -n velero patch restore <name> --type=json -p='[{"op":"remove","path":"/metadata/finalizers"}]'` — safe when kopia never started. 463 - **Suspend Flux before disaster recovery**: Flux reconciles frequently. If not suspended, it will override restored Deployments (e.g., reset replicas to what's in git) or create RSes that fight with Velero. Always run `flux suspend kustomization apps` before a restore, and `flux resume kustomization apps` after. 464 - **Velero Schedule CRDs**: Velero CRDs (incl. `Schedule`) are installed by the HelmRelease (infra-controllers). Schedules themselves live in infra-configs which runs after controllers — this is why they're in `infrastructure_talos/configs/velero-schedules.yaml` not in the velero controller directory. 465 - **node-agent PodSecurity**: Kopia's `node-agent` DaemonSet requires `hostPath` volumes. The `velero` namespace must have `pod-security.kubernetes.io/enforce: privileged`. 466 - **Longhorn backup disk space**: Longhorn snapshots are space-expensive during generation. Prefer Velero (Kopia) for volume backups where possible. 467 - **Longhorn snapshots taken during recovery/expansion are unsafe**: A snapshot taken while a volume was being expanded or used for database recovery may not be usable. Prefer Velero backups taken at scheduled times when the app was quiescent. 468 - **startupProbe for databases with variable startup time**: For apps like conduwuit/continuwuity that open a RocksDB database (startup time varies by DB size and WAL replay), use a `startupProbe` instead of `initialDelaySeconds`. startupProbe gives a large budget (e.g., `failureThreshold: 18, periodSeconds: 10` = 3 minutes) without delaying health signaling once the app is actually ready. Kubernetes does NOT run liveness/readiness probes until all init containers AND the startupProbe succeed — so kopia data is fully written before the app is probed.