README.md (4548B)
1 # Midnightthoughts GitOps 2 3 GitOps configuration for Kubernetes on [Talos Linux](https://www.talos.dev/) using [Flux CD](https://fluxcd.io/). 4 5 ## Stack 6 7 | Component | Technology | 8 |---|---| 9 | OS | [Talos Linux](https://www.talos.dev/) | 10 | GitOps | [Flux CD](https://fluxcd.io/) | 11 | Cloud | [Hetzner Cloud](https://www.hetzner.com/cloud) (CAX nodes) + Hetzner LBs | 12 | CNI | [Cilium](https://cilium.io/) | 13 | Ingress | [Envoy Gateway](https://gateway.envoyproxy.io/) | 14 | Storage | [Longhorn](https://longhorn.io/) (block) + Hetzner Object Storage (S3) | 15 | Database | [CloudNativePG](https://cloudnative-pg.io/) (PostgreSQL 18) | 16 | Secrets | [SOPS](https://getsops.io/) + [age](https://age-encryption.org/) | 17 | Backups | [Velero](https://velero.io/) (4× daily) | 18 | Monitoring | VictoriaMetrics + Grafana + Loki | 19 | Auth | [Authentik](https://goauthentik.io/) | 20 | Updates | [Renovate](https://docs.renovatebot.com/) | 21 | TLS | [cert-manager](https://cert-manager.io/) + Let's Encrypt | 22 23 ## Repository Structure 24 25 ``` 26 ├── clusters/talos_cluster/ # Flux bootstrap & Kustomizations 27 ├── infrastructure_talos/ # Controllers (CNI, storage, cert-manager, monitoring) 28 │ ├── controllers/ # Longhorn, Velero, CloudNativePG, etc. 29 │ └── configs/ # Cluster config (CNPG cluster, issuers, secrets) 30 ├── apps/ 31 │ └── talos_cluster/ # All deployed applications 32 ├── .github/workflows/ # CI: validation, security scans, docs deployment 33 └── scripts/ # Helper scripts 34 ``` 35 36 ## Quick Start 37 38 ```bash 39 # Generate Talos config 40 talosctl gen config cluster-2025 https://CONTROL_PLANE_IP:6443 41 talosctl apply-config --insecure --nodes CONTROL_PLANE_IP --file controlplane.yaml 42 talosctl bootstrap --nodes CONTROL_PLANE_IP 43 talosctl kubeconfig --nodes CONTROL_PLANE_IP 44 45 # Install Flux 46 flux check --pre 47 kubectl apply -k clusters/talos_cluster/flux-system 48 49 # Setup SOPS (generate a NEW key — never reuse an existing one) 50 age-keygen -o age.key 51 kubectl create secret generic sops-age --namespace=flux-system --from-file=age.agekey=age.key 52 # Update .sops.yaml with your public key 53 # Store age.key somewhere secure (password manager, not in this repo) 54 55 # Trigger reconciliation 56 flux reconcile kustomization flux-system --with-source 57 ``` 58 59 ## Working with Secrets 60 61 ```bash 62 # Encrypt a new secret 63 sops -e -i secret.yaml 64 65 # Edit an encrypted secret 66 sops secret.yaml 67 68 # View decrypted (don't commit output) 69 sops -d secret.yaml 70 ``` 71 72 The `encrypted_regex` in `.sops.yaml` controls which fields are encrypted. All secrets are encrypted with age using a shared cluster key stored in the `sops-age` Kubernetes secret. 73 74 ## Common Operations 75 76 ```bash 77 # Force reconcile after a push 78 flux reconcile kustomization flux-system --with-source 79 80 # Check status of all Flux resources 81 flux get all -A 82 83 # Show recent Flux errors 84 flux logs --level=error --all-namespaces 85 86 # Rollback: revert the commit and push 87 git revert HEAD && git push 88 ``` 89 90 ## Backups 91 92 Velero runs 4× daily (00:00, 06:00, 12:00, 18:00 UTC) backing up all cluster resources and Longhorn volumes to Hetzner Object Storage. CloudNativePG WAL archiving provides continuous PostgreSQL backup to a separate S3 bucket. 93 94 ```bash 95 # Check backup status 96 kubectl get backup.velero.io -n velero --sort-by='.metadata.creationTimestamp' 97 98 # Trigger manual backup 99 velero backup create manual-$(date +%Y%m%d-%H%M) --include-namespaces '*' 100 ``` 101 102 ## CI/CD 103 104 | Workflow | Trigger | Purpose | 105 |---|---|---| 106 | `validate.yaml` | PR / push to main | Manifest validation, security scans (gitleaks, trivy, kubescape) | 107 | `docs.yml` | Push to main (*.md changes) | Build and deploy MkDocs to GitHub Pages | 108 | `build-continuwuity.yaml` | Manual | Custom Continuwuity image build | 109 110 Run validation locally: 111 ```bash 112 ./scripts/validate.sh 113 ``` 114 115 ## Documentation 116 117 ```bash 118 # Install doc dependencies (once) 119 make docs-install 120 121 # Preview locally at http://127.0.0.1:8000 122 make docs 123 124 # Build to verify (outputs to site/) 125 make docs-build 126 ``` 127 128 Add a `README.md` to any app directory under `apps/talos_cluster/<app>/` and it will automatically appear in the docs navigation. 129 130 ## Troubleshooting 131 132 ```bash 133 # Flux 134 flux check 135 flux logs --all-namespaces 136 137 # App not reconciling 138 kubectl describe kustomization <name> -n flux-system 139 kubectl describe helmrelease <name> -n <namespace> 140 141 # Check pod 142 kubectl describe pod <pod> -n <namespace> 143 kubectl logs <pod> -n <namespace> --previous 144 145 # Talos node health 146 talosctl health --nodes <NODE_IP> 147 talosctl logs -n <NODE_IP> 148 ```