Blog
49 articles on infrastructure, automation, and cloud.
All
AI Ansible Azure Backup Compliance DNS Debugging GitOps Hardware Homelab Kubernetes MikroTik Monitoring NFS Networking Proxmox Redis Review SSO Security Terraform Vault Velero ZFS
2026
- Self-Hosted SSO for 25 Services: Authelia OIDC on Kubernetes06 Aug 2026How a single Authelia instance protects Proxmox, PBS, Grafana, ArgoCD, Headscale, and 20+ web apps with OIDC — CNPG-managed Postgres, Redis sessions, hmac_secret in Vault, and a Traefik ForwardAuth middleware that gates every request.
- My Terraform Runner Destroyed Itself Mid-Apply02 Aug 2026Atlantis was running inside k3s, managing the same Proxmox VMs it ran on. When bpg/proxmox issued a qmshutdown for a non-live-update attribute, it killed the Atlantis pod that was executing the apply. Here's the structural hazard and the fix.
- The ZFS ARC Freeze: How a Marginal PSU Killed My Entire Homelab30 Jul 2026My Proxmox host froze under load three times in one week. The root cause wasn't software — it was a marginal PSU triggering TDP throttling, which exposed a ZFS ARC deadlock. Here's the full chain from symptom to fix.
- How a Single Volume Was 65% of My Velero Backup and What I Almost Excluded Instead26 Jul 2026The nfs-provisioner's root-mount volume was backing up the entire shared NFS export as one 33.4GB blob every night — redundant because every app's PVC was already backed up separately. The investigation almost went wrong when the first hypothesis pointed at the wrong disk.
- Best Mini PC for Homelab 202625 Jul 2026A hands-on comparison of the 5 best mini PCs for homelab use in 2026 — covering Intel N100, Ryzen 5, and Ryzen 7 platforms for Proxmox, K3s, and always-on workloads.
- Best NAS for Home Backup 202625 Jul 2026A hands-on comparison of the 5 best NAS devices for home backup in 2026 — covering Synology, QNAP, Terramaster, and UGREEN for personal and homelab use.
- MikroTik vs Ubiquiti for Home Network 202625 Jul 2026MikroTik RB5009 vs Ubiquiti UniFi Dream Router — a detailed head-to-head comparison covering VLANs, VPN, security, Terraform automation, and real-world homelab use.
- Why a Cloud Backup Sync Was Failing Daily for Six Weeks23 Jul 2026A daily offsite backup cron job had a 99% failure rate for over a month. It looked like a credential or network problem. It was neither - Google Drive's API throttles hard on exactly the storage format Proxmox Backup Server uses, and a 24-hour cron window was never going to be enough.
- Zero NetworkPolicies on Vault: How I Found the Biggest Gap in My Cluster and a GitOps Tracking Bug That Hid It19 Jul 2026Vault is the trust root every ExternalSecret reads from. Its namespace had zero NetworkPolicies — any pod in the cluster could reach it. The investigation also uncovered a class of ArgoCD drift that silently ignores merged changes.
- 310 Restarts in 21 Days: CNPG's Silent PodMonitor Failure and the Leader-Election Trap17 Jul 2026CloudNativePG's auto-generated PodMonitor was missing a single label — Prometheus never scraped it. The same I/O fragility that causes etcd timeouts was triggering leader-election failures, restarting the operator 310 times in 21 days. Here's how I traced both root causes.
- Beszel: Lightweight Host Monitoring That Doesn't Deserve Its Own Server16 Jul 2026Why I replaced a heavyweight monitoring stack for host-level metrics with a single container, how Kyverno caught my first deploy before it hit production, and why internal-only dashboards don't need a public DNS record.
- Inside My Homelab: The Hardware Behind Every Article on This Blog15 Jul 2026A full tour of the physical rack, compute, and networking gear powering my K3s cluster, Proxmox hosts, and everything I write about here — what I run, why I picked it, and what I'd change.
- Kubernetes Health Probes: The Host Header Trap That Restarts Healthy Pods15 Jul 2026Adding health probes to 20+ workloads taught me that kubelet sends the Pod IP as the Host header — and apps with host-validation reject it. Here's the full sweep, the gotcha that caught me, and the probe patterns that actually work.
- Migrating Atlantis to an LXC Accidentally Made It Fully Public14 Jul 2026Moving Atlantis from Kubernetes to a dedicated LXC involved repointing the Cloudflare Tunnel. The new tunnel pointed straight at the LXC's IP, bypassing Traefik and Authelia entirely. Atlantis has no auth of its own. For roughly 18 hours, anyone with the URL had full plan/apply access to the infrastructure repo.
- Renovate OOMKilled Three Times: Why the Fix Wasn't More Memory14 Jul 2026Two GiB wasn't enough, so I bumped to 3 GiB. Still OOMKilled. Bumped to 4 GiB. Still OOMKilled. The real fix wasn't memory at all — it was Terraform hash concurrency. Here's how I isolated the actual spike.
- Zero NetworkPolicies on the Database Namespace: The Gap That Let Any Pod Reach Authelia's Postgres13 Jul 2026The database namespace holding Authelia's session storage had no network restrictions. Any pod in the cluster could reach it. Here's the audit that found it, the traffic pattern that shaped the fix, and the namespace I deliberately left unrestricted.
- Discord Voice Choppy? It Was Bufferbloat — Fixed with 51 Lines of Terraform12 Jul 2026Discord voice was robotic for people hearing me. Confirmed clean over mobile data — home network path. Zero QoS on the WAN interface meant a 50 Mbit upload ceiling was easy to saturate. Here's the investigation, the fix, and why PCQ per-flow fairness matters.
- The Disaster Recovery Runbook Nobody Had Actually Run11 Jul 2026DISASTER-RECOVERY.md documented the Proxmox Backup Server restore procedure in detail. It had never been executed end-to-end. Running it for the first time - against a scratch VM, alongside the still-running original - found a network conflict the documentation never mentioned.
- The .gitleaks-baseline.json That Suppressed Live Production Secrets10 Jul 2026A gitleaks baseline file is supposed to suppress known-false-positive findings. It turned out to be suppressing the live rpc_secret and admin_token for a production Garage S3 cluster - unrotated since the commit that introduced them to a public repo months earlier.
- Full Observability on k3s: kube-prometheus-stack + Loki + Grafana OIDC04 Jul 2026Deploy a production-grade monitoring stack on bare-metal k3s: Prometheus, Loki with Garage S3 storage, Promtail on edge nodes via Ansible, SNMP monitoring for MikroTik, and Grafana SSO via Authelia OIDC - all GitOps-managed.
- Redis Killed Nextcloud and Nobody Noticed for Hours01 Jul 2026Redis running without a PVC still has persistence enabled by default. When it can't write RDB snapshots to a read-only rootfs, it doesn't crash - it silently refuses all writes. Here's how that turned into a full Nextcloud outage and why the logs pointed at the wrong thing first.
- HA DNS for Homelab: Unbound + AdGuard Home + Keepalived on Raspberry Pi27 Jun 2026A two-node recursive DNS stack with ad filtering, automatic config sync, and transparent failover - fully managed with Ansible.
- I Hardened Pod securityContext and Broke 9 Containers in Production25 Jun 2026capabilities.drop: [ALL] and runAsNonRoot: true passed schema validation cleanly. Within minutes of merge, nine containers - including both Postgres instances backing Paperless and Nextcloud - were down. Here's the failure analysis, why a manual kubectl fix got silently undone, and the lesson for any blanket securityContext change.
- kubectl Said Everything Was Correct. Traefik 404'd Anyway.25 Jun 2026Migrating Jellyfin off k3s onto a GPU-passthrough LXC meant pointing a Service at an external IP. The EndpointSlice looked completely correct via kubectl - Service existed, endpoint listed right - but Traefik 404'd every request. A second, unrelated gotcha surfaced in the same migration: a PVC silently shared by reference across two unrelated files.
- ArgoCD Gotchas: Cache Staleness and the SharedResourceWarning Nobody Explains22 Jun 2026kubectl apply succeeds, the field reverts within seconds, and there's no error anywhere. Two ArgoCD debugging patterns that hit the same homelab three times in one day: repo-server cache staleness reverting live edits, and two Applications silently fighting over the same resource.
- I Ran Gitleaks Against My Own Repo and Found 12 Real Secrets22 Jun 2026A full-history gitleaks scan of a homelab repo that had been running for months turned up 12 distinct plaintext secrets - including an OIDC signing key. Here's the scanning setup, the baseline strategy that doesn't block on pre-existing leaks, and the remediation plan.
- My Firewall Had 77 Rules. Terraform Knew About 22 of Them.22 Jun 2026Multiple rounds of 'reconstruct the firewall' work each added a fresh generation of rules without removing the old one. Because RouterOS evaluates rules in order and stops at the first match, the oldest, broadest generation was silently winning over the newest, narrower one - undoing a security tightening that looked complete in Terraform.
- Hardening Unattended Raspberry Pi Edge Nodes: Watchdog, fail2ban, nftables, and the Mistakes That Take Down DNS22 Jun 2026Two Raspberry Pis run DNS for an entire network with no one watching them most of the time. A hardware watchdog, fail2ban, an additive nftables host firewall that doesn't conflict with Docker, log size caps, and an alerting path that works even when the rest of the monitoring stack is down.
- k3s Backup Without the Complexity: Velero + Garage S3 on Longhorn20 Jun 2026Replace MinIO with Garage - a single 50MB binary - as the Velero backup target. Full daily cluster backups with Longhorn volume snapshots, deployed via ArgoCD.
- External Secrets Operator + HashiCorp Vault: GitOps Secret Lifecycle in Kubernetes18 Jun 2026Kubernetes Secrets are base64-encoded, not encrypted. Moving secrets out of the cluster into Vault - and syncing them back via External Secrets Operator - gives you rotation, audit logging, and compliance without changing how applications consume secrets.
- How a 1 GiB Memory Limit Took Down My Entire k3s Cluster18 Jun 2026A single misconfigured resource limit triggered a cascade: OOMKill on the control-plane, load average of 90, 1.2M DNS queries per day, and kubelet reporting the wrong allocatable memory. Here's the full post-mortem.
- Kyverno: Supply Chain Security as Admission Control on Kubernetes18 Jun 2026Most Kubernetes clusters accept any container image, any privilege level, and any resource configuration by default. Kyverno lets you enforce policies at admission time - before anything runs. Here's how to build a supply chain security baseline with Audit-first rollout.
- IPv6 NAT66 Behind a FritzBox: The RouterOS 7 Bug That Broke WiFi Clients18 Jun 2026Set up IPv6 on MikroTik behind a FritzBox with CGN: ULA prefix, NAT66 masquerade - plus the RouterOS 7 bug that broke WiFi clients. Full setup and fix.
- SLO Burn-Rate Alerting with Prometheus: Beyond Threshold Alerts18 Jun 2026Most teams alert when availability drops below a threshold. Burn-rate alerting tells you how fast you're spending your error budget - so you page on trajectory, not just current state. Here's how to implement the Google SRE Workbook approach on a bare-metal k3s cluster.
- Self-Hosted Tailscale Control Plane: Headscale on k3s with Authelia OIDC13 Jun 2026Deploy Headscale on a bare-metal k3s cluster with Longhorn persistence, Traefik ingress, and Authelia OIDC authentication - fully GitOps-managed via ArgoCD.
- Wildcard TLS on K3s with cert-manager and Cloudflare DNS0122 May 2026 (updated)HTTP-01 challenges fail behind a default-deny firewall. Switching cert-manager to Cloudflare DNS01 wildcard validation gives every Traefik IngressRoute a *.yourdomain.com certificate without opening port 80 to the internet.
- GitOps on K3s: Managing a Complete Homelab with ArgoCD20 May 2026How to manage an entire Kubernetes homelab - MetalLB, Traefik, Longhorn, Authelia, and more - as a Git repository using ArgoCD's App-of-Apps pattern.
- Bare-Metal LoadBalancer on K3s: MetalLB + Traefik with ArgoCD18 May 2026How to get a real external IP on a bare-metal Kubernetes cluster using MetalLB L2 mode, and wire it up with Traefik for automatic HTTPS - fully GitOps-managed with ArgoCD.
- NIS2 Article 21 in Azure: Implementing Network Security Controls with Terraform17 May 2026A technical deep-dive into the network security requirements of NIS2 Article 21 and how to implement them in Azure using Terraform - with concrete code, not legal theory.
- Zero-Trust RAG: Defeating the Shared Private Link Deadlock in Azure Terraform16 May 2026How to programmatically approve Azure AI Search Shared Private Links using AzAPI, and why your AI architecture will fail an audit without proper Identity Chaining.
- Enterprise Homelab: K3s, Authelia & Longhorn on Proxmox with Terraform16 May 2026How to build a production-grade Kubernetes homelab with K3s, Authelia SSO, Longhorn storage, and ArgoCD - and the five painful mistakes that will cost you hours if you don't know about them.
- Breaking the Loop: Solving Circular Dependencies in Azure Firewall Routing07 May 2026How to implement Azure Firewall Forced Tunneling in Terraform without triggering cycle errors, and why a simple 0.0.0.0/0 route will instantly break your Windows VMs.
- Architecting an Enterprise-Grade Homelab: My Ansible Master Playbook06 May 2026Take a tour of a fully automated, segmented, and highly available homelab architecture orchestrated entirely via Ansible and GitOps.
- Automating MikroTik WireGuard VPN with Role-Based Access via Terraform05 May 2026Deploy a WireGuard VPN on MikroTik using Terraform. Learn how to implement role-based network access, isolating mobile devices from full admin laptops.
- Automating MikroTik Bridge VLAN Filtering & Proxmox Trunks with Terraform04 May 2026Master MikroTik's notoriously complex Bridge VLAN Filtering. Learn how to automate dynamic VLAN matrices, Proxmox trunk ports, and edge devices using Terraform.
- Surviving Azure Policies: Zero-Trust Hub & Spoke with Terraform03 May 2026How to build an enterprise-grade Azure network architecture that blocks internet traffic by default and survives aggressive DeployIfNotExists (DINE) policies - without breaking your CI/CD pipeline.
- Implementing a Zero-Trust MikroTik Firewall with Terraform03 May 2026Learn how to enforce strict VLAN isolation, fast-track traffic, and build a default-deny firewall for MikroTik RouterOS using Infrastructure as Code.
- Deploying Gemma 4 26B on Proxmox: IaC Setup with Terraform, Ansible & AMD iGPU02 May 2026A complete guide to automating a local AI stack on Proxmox LXC using Terraform and Ansible, including Open-WebUI and AMD Radeon Vega iGPU workarounds.
- Hardening Azure Acmebot for ISO 27001 & NIS2 Compliance01 May 2026A deep dive into architecting a Zero-Trust Let's Encrypt automation using Terraform, Azure Private Link, and VNet Integration.