Build Logs from the Lab
Real engineering stories. Kernel panics at midnight, RAID failures on Christmas, and the occasional moment where everything just works. No tutorials. No "10 tips." Just the raw notes.
The 463-Message Saga
I spent 4 days and 463 ChatGPT messages trying to get Docker and Traefik working. Day 3 alone was 238 messages. But when that curl finally returned 200 OK at 11 PM on day 4, I may have scared the neighbors.
The RAID That Almost Ate Christmas
Four HGST drives. One dying Synology. USB docks that kept disconnecting. The week between Christmas and New Year's became a crash course in mdadm, LVM, Btrfs, and why you should never trust USB for data recovery.
The Crash Recovery & The Lab Engine Goes Live
System crash at 00:30, crash recovery, fleet package drift discovered, lab engine deployed to production, UX fixed, and multi-container support planned. 4 parallel sessions, 7 critical infrastructure issues resolved.
The Kernel That Panicked Every Three Minutes
Server rebooting every 1-3 minutes. Couldn't stay up long enough to investigate. Turned out K3s pods were crash-looping so hard they destabilized the kernel, and Ubuntu's default panic setting auto-rebooted before I could catch it.
[object Object] — The Three-Headed Hydra That Ate My Admin Panel
Nine hours debugging why Cloudflare Pages rendered [object Object] instead of HTML. Three independent sources of process. One monkey-patched Response constructor. Zero regrets.
The 812GB Hiding in Plain Sight
I spent hours trying to migrate a VM between hypervisors. Kernel panics, graphics corruption, UEFI nightmares. Then I ran fdisk and discovered 812GB of unallocated space on my main drive. Sometimes the solution isn't fixing the problem — it's finding a better problem.
25 Bugs, One Night
12.5 hours, 25+ bugs across 4 components, deployed to every node before sunrise. The v2.5.1 overnight marathon.
The Tunnel With Two Heads
My websites were flipping between working and broken at random. Same URL, same moment, different results. Turns out I had two cloudflared instances fighting over the same tunnel — and Cloudflare was helpfully load-balancing between them.
The Thunderbolt That Killed DNS (and SSH, and My Confidence)
Plugged in a Thunderbolt ethernet adapter for faster NAS transfers. Lost DNS. Lost SSH. Three root causes, a CorpDNS detour, and a lesson about hot-pluggable networking on Unraid.
The Reboot Loop That Blamed the Wrong Code
When Alpha-Centauri started rebooting every 90 seconds, I was convinced my build swarm code had achieved sentience and was trying to escape. Spoiler: it was innocent.
The Race Condition That Ate My Binaries
Drones were deleting their packages before the orchestrator could validate them. Also: a Docker container crash-looping because it was looking for SSH keys that don't exist in the new architecture.
Three Rsync Bugs In One Day
A 7-hour debugging session uncovered three separate rsync bugs: missing timeouts, an invalid SSH flag, and uploading 3GB instead of one package. Also built a CLI tool because I was tired of SSHing everywhere.
The AudioBooks Folder That Ate Itself Three Times
When you lose your Claude context mid-cleanup and discover your Unraid server has 3.5TB of audiobooks nested three folders deep with 3,582 empty placeholder folders for good measure.
The Day I Learned OpenRC Isn't Systemd
KDE Plasma crashed and wouldn't give me a terminal. After the reboot, I discovered why: half the services I needed weren't running because OpenRC doesn't auto-start things the way systemd does. Also, /run/ is empty after every reboot and nobody told me.
The GRUB That Forgot Everything
Deleted a corrupted GRUB. Now /etc/grub.d/ was empty. os-prober couldn't see Windows or CachyOS. NVIDIA parameters were wrong. Found the working config in a backup file I didn't know existed.
These are just the highlights. There are 90+ more entries in the full journal.
Browse All Journal Entries