Building Argo OS: 100 Days Into Custom Linux
Sometimes you look at a perfectly functional openSUSE installation and think: “This is too easy. I should make my life harder.”
That’s not exactly how it started, but close.
I’ve been running a homelab for over 12 years now. Started with seedbox scripts, moved through ESXi (2016-2019), landed on Proxmox, and accumulated massive storage across Unraid and Synology units. My main workstation is a 12-year-old i7-4790K that I built on the high end back in the day—32GB RAM, NVMe storage, and an RTX 4070 Ti that I upgraded to recently.
The machine still runs great. Linux gave it new life compared to Windows. But Gentoo compilation times? Those were killing me. An update could take anywhere from a few hours to a few days depending on what needed rebuilding. That takes away the fun of having a computer, period.
I wanted the optimization of Gentoo (-march=native is a hell of a drug) but I didn’t want to spend my life watching packages compile. I wanted to design and customize my own system—make it mine—without the time tax.
This is the story of Argo OS: a Gentoo-based distribution I built over nights and weekends starting October 2025. It’s part technical documentation, part disaster log, and part therapy session.
The Journey at a Glance
This is a 4-part series documenting the entire build:
| Part | What Happened | Key Lesson |
|---|---|---|
| Part 1: First Principles (this page) | Started with ext4. Lost everything. Rebuilt with Btrfs. | Never trust a filesystem without snapshots. |
| Part 2: The Qt6 Crisis | A routine update broke KDE. Rolled back in 2 minutes. | Snapshots aren’t just backups—they’re undo buttons. |
| Part 3: Cloud & Code | Local backups don’t survive house fires. Built encrypted cloud sync. Accidentally wrote a package manager. | Scope creep is real, but sometimes useful. |
| Part 4: The Hybrid Vision | Realized Gentoo is great for hardware, Nix is great for dotfiles. Combined them. | Use the right tool for each layer. |
Supporting technical guides:
- Btrfs & Snapper Complete Guide — The snapshot system in detail
- The Build Swarm — How I distribute compilation across 66 cores
- apkg: The Teaching Package Manager — The wrapper that explains itself
Part 1: First Principles & The Foundation
October 22, 2025
The Architecture Vision
Before I ran a single emerge, I mapped out the ecosystem. This wasn’t going to be just a desktop install. It was going to be a distribution.
┌─────────────────────────────────────────────────────────────┐
│ THE ARGO OS ECOSYSTEM │
├─────────────────────────────────────────────────────────────┤
│ │
│ BINHOST (Compilation) DRIVER (Desktop) │
│ ├── Compiles packages ├── Never compiles │
│ ├── Hosts binary repo ├── Pulls binaries │
│ └── Borrows idle time └── Daily use │
│ │ ▲ │
│ └──────── binpkgs ─────────┘ │
│ │
│ GOLDEN IMAGE (VM) BACKUP (Cloud) │
│ ├── Bare-bones Stage 3 ├── Encrypted Btrfs sends │
│ ├── SSH-ready baseline ├── Google Drive via rclone │
│ └── Clone in 30 seconds └── Off-site protection │
│ │
└─────────────────────────────────────────────────────────────┘The idea: make compilation every other computer’s problem. Borrow idle time from machines that would otherwise be doing nothing. Build once, deploy everywhere.
First Contact: The VM Foundation
I started safely. Spin up a VM, install Gentoo. Standard procedure.
For the uninitiated, installing Gentoo involves booting a minimal live environment, formatting disks manually, unpacking a “Stage 3” tarball (a minimal root filesystem), and then compiling the kernel and bootloader yourself. It takes me about 5 hours to get to a working TTY with SSH.
The Initial Specs:
- Host: openSUSE Tumbleweed with KVM/QEMU
- VM: gentoo-desktop
- Storage: qcow2 disk image
- Target:
x86-64-v3(Haswell optimized) - Init System: OpenRC (not systemd—I understand OpenRC)
- Filesystem:
ext4
Wait. ext4?
Yes. And that was Mistake #1.
The make.conf That Started It All
Every Gentoo system lives and dies by /etc/portage/make.conf. This file controls how every package gets compiled.
# /etc/portage/make.conf (October 2025 - VM version)
# CPU optimization - compile for this exact machine
COMMON_FLAGS="-O2 -pipe -march=native"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
# Parallel compilation - use all 8 vCPUs
MAKEOPTS="-j8"
# What we want in our packages
USE="X gtk qt5 qt6 kde plasma wayland -systemd -gnome pulseaudio pipewire"
# Video drivers for VM
VIDEO_CARDS="vmware virtio"
# Accept ~amd64 keywords for fresh packages
ACCEPT_KEYWORDS="~amd64"
# The critical line - create binary packages automatically
FEATURES="buildpkg binpkg-multi-instance"The key decisions:
-march=native: Compiles for your specific CPU. Maximum performance. The catch: binaries won’t work on different CPUs. I’d learn to use -march=haswell later for broader compatibility across my fleet.
MAKEOPTS="-j8": Parallel compilation. Uses all available cores. Dramatically speeds up builds—but on my 4790K, “dramatically” still meant hours for big packages.
FEATURES="buildpkg": Every time Portage compiles a package, it also creates a binary tarball in /var/cache/binpkgs/. I wasn’t just installing software; I was building a repository.
-systemd: No systemd. OpenRC is simpler, I understand it, and it doesn’t try to manage my life.
The Compilation Marathon
Days 1-7. October 22-27, 2025.
Day 1-2: Base system
GCC. Glibc. Binutils. The building blocks. Hours of watching text scroll by.
Day 3-4: KDE Plasma desktop
emerge --ask kde-plasma/plasma-meta4-6 hours of compilation on a VM with 8 vCPUs. This is where I started questioning whether I should just use Arch.
Day 5-6: Essential applications
Firefox (compiled with PGO), vim, git, development tools.
Day 7: Configuration and polish
Theme tweaks, keyboard shortcuts, SSH configuration.
The Statistics:
- Installed packages: 1,571
- Package database files: 55,782 files in
/var/db/pkg/ - Compilation time: 40-50 hours total
- System storage: ~35GB
- Binary packages created: 1,124
.gpkg.tarfiles, 28.3GB
By October 27th, it was perfect. A lean, mean, optimized machine.
The Crash (October 27)
I shut down the VM. I went to bed.
The next morning, I booted it up.
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)I stared at the screen. Rebooted. Same error.
The Autopsy
I mounted the qcow2 image from the host:
sudo modprobe nbd
sudo qemu-nbd --connect=/dev/nbd0 /var/lib/libvirt/images/gentoo-desktop.qcow2
sudo mount /dev/nbd0p3 /mnt/gentooRan fsck:
sudo fsck.ext4 -y /dev/nbd0p3The output was horrifying:
/dev/nbd0p3: recovering journal
Clearing orphaned inode 262245 (uid=0, gid=0, mode=0100644, size=0)
Clearing orphaned inode 262244 (uid=0, gid=0, mode=0100644, size=0)
...
/dev/nbd0p3: ***** FILE SYSTEM WAS MODIFIED *****Some freak host-system write error had mangled the ext4 superblock. fsck “recovered” it—by deleting corrupted files. Half of /var/db/pkg was gone.
What /var/db/pkg Means
In Gentoo, /var/db/pkg is everything. It’s the database of what’s installed.
The critical files at risk:
/var/db/pkg/- 55,782 files tracking every installed package/var/lib/portage/world- The list of explicitly installed packages/etc/portage/- All configuration (weeks of USE flag optimization)
If you lose /var/db/pkg, you have a zombie OS. The system runs, but you can’t update, remove, or install anything. Portage has no idea what’s on the disk.
50 hours of work. Gone.
The Backup That Saved Everything
Pure luck.
I had automated VM backups running on my NAS. Daily at 2:00 AM. 7-day retention.
There was a snapshot from October 27th at 2:00 AM—taken 6 hours before the corruption.
The backup wasn’t planned for this disaster. But it saved 40+ hours of work.
Backups you don’t have are useless. Backups you do have are priceless.
The Pivot: Embracing Btrfs
I sat there staring at the error message, and I realized: If this was openSUSE, I would just type snapper rollback and be drinking coffee by now.
I nuked the VM.
We were starting over. But this time, we were doing it right.
The New Filesystem Layout
I adopted the openSUSE partition strategy—complex, but it allows snapshotting the system without rolling back personal files.
The Partition Map:
| Device | Size | Filesystem | Mount Point |
|---|---|---|---|
| /dev/vda1 | 512MB | FAT32 | /boot/efi |
| /dev/vda2 | 8GB | swap | [swap] |
| /dev/vda3 | 241.5GB | Btrfs | / |
The Subvolume Architecture:
# Create the subvolume structure
mount /dev/vda3 /mnt/gentoo
cd /mnt/gentoo
btrfs subvolume create @
btrfs subvolume create @home
btrfs subvolume create @snapshots
btrfs subvolume create @var-cache
btrfs subvolume create @var-log| Subvolume | Mount | Purpose |
|---|---|---|
@ | / | Root filesystem—frequently snapshotted |
@home | /home | User data—separate snapshot schedule |
@snapshots | /.snapshots | Snapper storage (excluded from snapshots) |
@var-cache | /var/cache | Package cache—exclude from snapshots |
@var-log | /var/log | Logs—persist through rollback for debugging |
Why separate /var/log? If your system crashes and you roll back to a snapshot before the crash, you want the logs from the crash to persist. Otherwise, you can’t debug what happened.
Mount Options Matter
The /etc/fstab for Btrfs:
UUID=xxx / btrfs defaults,subvol=@,compress=zstd:3,noatime,discard=async,space_cache=v2 0 1
UUID=xxx /home btrfs defaults,subvol=@home,compress=zstd:3,noatime 0 2
UUID=xxx /.snapshots btrfs defaults,subvol=@snapshots 0 2
UUID=xxx /var/cache btrfs defaults,subvol=@var-cache,noatime 0 0
UUID=xxx /var/log btrfs defaults,subvol=@var-log,noatime 0 0What each option does:
compress=zstd:3— Transparent compression. 20-30% space savings. Negligible CPU overhead.noatime— Don’t update file access times. Fewer writes, better SSD life.discard=async— SSD TRIM support without blocking operations.space_cache=v2— Faster free space tracking.
Snapper: The 2-Minute Rollback
Snapper is openSUSE’s snapshot management tool.
emerge sys-fs/btrfs-progs app-backup/snapper sys-boot/grub-btrfsThe OpenRC challenge: Snapper expects systemd timers. Gentoo uses OpenRC. No problem—cron works fine:
# /etc/cron.hourly/snapper-timeline
#!/bin/bash
/usr/bin/snapper -c root create --cleanup-algorithm timeline --description "timeline"# /etc/cron.daily/snapper-cleanup
#!/bin/bash
/usr/bin/snapper -c root cleanup timeline
/usr/bin/snapper -c root cleanup numberThe killer feature: automatic package snapshots.
I added Portage hooks that create snapshots before and after every emerge:
# /etc/portage/bashrc.d/snapper-pre.sh
if [[ ${EBUILD_PHASE} == "setup" ]]; then
snapper -c root create --description "Before emerge ${CATEGORY}/${PN}-${PVR}"
fiNow every package installation is bracketed by snapshots. Break something? Roll back in 2 minutes.
GRUB Integration
grub-btrfs makes every snapshot bootable from the GRUB menu:
emerge sys-boot/grub-btrfs
grub-mkconfig -o /boot/grub/grub.cfgNow GRUB shows:
Gentoo Linux (current system)
Advanced options
├─ Gentoo Linux (Snapshot 1) - Fresh installation
├─ Gentoo Linux (Snapshot 2) - Before emerge firefox
├─ Gentoo Linux (Snapshot 3) - After emerge firefox
└─ ...You can test a snapshot before committing. Boot in, verify everything works, then decide.
The First Stability Milestone (November 15)
By mid-November, I had the system on bare metal.
We had:
- KDE Plasma 6: Running on proprietary NVIDIA drivers (RTX 4070 Ti)
- Sound: Pipewire (after fighting with ALSA for 3 days)
- Browsers: Firefox (compiled with PGO)
- Snapshots: Hourly timeline, pre/post package, GRUB bootable
And the test that mattered:
snapper create -d "Before breaking things"
rm -rf /lib/modules
# System is now dead. No kernel modules.
reboot
# Select snapshot from GRUB
# Desktop loads.It worked. I had achieved the dream: An indestructible rolling release.
What I Learned
After the first month:
ext4 Is Fine Until It Isn’t
Traditional filesystems don’t have snapshots. One corruption event, one bad update, one rm -rf—and you’re rebuilding from scratch.
Btrfs snapshots aren’t a luxury. For a source-based distro where updates can break anything, they’re a requirement.
Binary Packages Are Essential
Nobody should compile KDE Plasma more than once. Build it on a dedicated machine, serve the binary, install in seconds.
| Package | Compile Time | Binary Install | Savings |
|---|---|---|---|
| KDE Plasma | 12 hours | 45 minutes | 94% |
| LibreOffice | 2 hours | 2 minutes | 98% |
| Firefox | 45 minutes | 30 seconds | 99% |
The Golden Image Is a Baseline, Not a Clone
My “golden image” isn’t a fully-loaded system. It’s a bare-bones Stage 3 with SSH ready. Cloning takes 30 seconds. From there, I pull binaries from the binhost to customize it.
This keeps the golden image simple and the binhost as the source of truth for what packages look like.
Test Your Backups
A backup you’ve never tested is not a backup. It’s a hope.
I now intentionally break things just to verify rollback still works.
What Comes Next
In Part 2, the system breaks again—this time from a routine update. A Qt6/elogind mismatch brought down KDE Plasma. But this time, recovery took 2 minutes instead of 50 hours.
The difference? Snapshots.
This is Part 1 of a 4-part series. Continue to Part 2: The Qt6 Crisis.