Building Argo OS: 100 Days Into Custom Linux

Sometimes you look at a perfectly functional openSUSE installation and think: “This is too easy. I should make my life harder.”

That’s not exactly how it started, but close.

I’ve been running a homelab for over 12 years now. Started with seedbox scripts, moved through ESXi (2016-2019), landed on Proxmox, and accumulated about 252TB of storage across Unraid and Synology units. My main workstation is a 12-year-old i7-4790K that I built on the high end back in the day—32GB RAM, NVMe storage, and an RTX 4070 Ti that I upgraded to recently.

The machine still runs great. Linux gave it new life compared to Windows. But Gentoo compilation times? Those were killing me. An update could take anywhere from a few hours to a few days depending on what needed rebuilding. That takes away the fun of having a computer, period.

I wanted the optimization of Gentoo (-march=native is a hell of a drug) but I didn’t want to spend my life watching packages compile. I wanted to design and customize my own system—make it mine—without the time tax.

This is the story of Argo OS: a Gentoo-based distribution I built over nights and weekends starting October 2025. It’s part technical documentation, part disaster log, and part therapy session.

The Journey at a Glance

This is a 4-part series documenting the entire build:

Part	What Happened	Key Lesson
Part 1: First Principles (this page)	Started with ext4. Lost everything. Rebuilt with Btrfs.	Never trust a filesystem without snapshots.
Part 2: The Qt6 Crisis	A routine update broke KDE. Rolled back in 2 minutes.	Snapshots aren’t just backups—they’re undo buttons.
Part 3: Cloud & Code	Local backups don’t survive house fires. Built encrypted cloud sync. Accidentally wrote a package manager.	Scope creep is real, but sometimes useful.
Part 4: The Hybrid Vision	Realized Gentoo is great for hardware, Nix is great for dotfiles. Combined them.	Use the right tool for each layer.

Supporting technical guides:

Btrfs & Snapper Complete Guide — The snapshot system in detail
The Build Swarm — How I distribute compilation across 62 cores
apkg: The Teaching Package Manager — The wrapper that explains itself

Part 1: First Principles & The Foundation

October 22, 2025

The Architecture Vision

Before I ran a single emerge, I mapped out the ecosystem. This wasn’t going to be just a desktop install. It was going to be a distribution.

┌─────────────────────────────────────────────────────────────┐
│                    THE ARGO OS ECOSYSTEM                     │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  BINHOST (Compilation)      DRIVER (Desktop)                 │
│  ├── Compiles packages      ├── Never compiles               │
│  ├── Hosts binary repo      ├── Pulls binaries               │
│  └── Borrows idle time      └── Daily use                    │
│         │                          ▲                         │
│         └──────── binpkgs ─────────┘                         │
│                                                              │
│  GOLDEN IMAGE (VM)          BACKUP (Cloud)                   │
│  ├── Bare-bones Stage 3     ├── Encrypted Btrfs sends        │
│  ├── SSH-ready baseline     ├── Google Drive via rclone      │
│  └── Clone in 30 seconds    └── Off-site protection          │
│                                                              │
└─────────────────────────────────────────────────────────────┘

The idea: make compilation every other computer’s problem. Borrow idle time from machines that would otherwise be doing nothing. Build once, deploy everywhere.

First Contact: The VM Foundation

I started safely. Spin up a VM, install Gentoo. Standard procedure.

For the uninitiated, installing Gentoo involves booting a minimal live environment, formatting disks manually, unpacking a “Stage 3” tarball (a minimal root filesystem), and then compiling the kernel and bootloader yourself. It takes me about 5 hours to get to a working TTY with SSH.

The Initial Specs:

Host: openSUSE Tumbleweed with KVM/QEMU
VM: gentoo-desktop
Storage: qcow2 disk image
Target: x86-64-v3 (Haswell optimized)
Init System: OpenRC (not systemd—I understand OpenRC)
Filesystem: ext4

Wait. ext4?

Yes. And that was Mistake #1.

The make.conf That Started It All

Every Gentoo system lives and dies by /etc/portage/make.conf. This file controls how every package gets compiled.

# /etc/portage/make.conf (October 2025 - VM version)

# CPU optimization - compile for this exact machine
COMMON_FLAGS="-O2 -pipe -march=native"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"

# Parallel compilation - use all 8 vCPUs
MAKEOPTS="-j8"

# What we want in our packages
USE="X gtk qt5 qt6 kde plasma wayland -systemd -gnome pulseaudio pipewire"

# Video drivers for VM
VIDEO_CARDS="vmware virtio"

# Accept ~amd64 keywords for fresh packages
ACCEPT_KEYWORDS="~amd64"

# The critical line - create binary packages automatically
FEATURES="buildpkg binpkg-multi-instance"

The key decisions:

-march=native: Compiles for your specific CPU. Maximum performance. The catch: binaries won’t work on different CPUs. I’d learn to use -march=haswell later for broader compatibility across my fleet.

MAKEOPTS="-j8": Parallel compilation. Uses all available cores. Dramatically speeds up builds—but on my 4790K, “dramatically” still meant hours for big packages.

FEATURES="buildpkg": Every time Portage compiles a package, it also creates a binary tarball in /var/cache/binpkgs/. I wasn’t just installing software; I was building a repository.

-systemd: No systemd. OpenRC is simpler, I understand it, and it doesn’t try to manage my life.

The Compilation Marathon

Days 1-7. October 22-27, 2025.

Day 1-2: Base system

GCC. Glibc. Binutils. The building blocks. Hours of watching text scroll by.

Day 3-4: KDE Plasma desktop

emerge --ask kde-plasma/plasma-meta

4-6 hours of compilation on a VM with 8 vCPUs. This is where I started questioning whether I should just use Arch.

Day 5-6: Essential applications

Firefox (compiled with PGO), vim, git, development tools.

Day 7: Configuration and polish

Theme tweaks, keyboard shortcuts, SSH configuration.

The Statistics:

Installed packages: 1,571
Package database files: 55,782 files in /var/db/pkg/
Compilation time: 40-50 hours total
System storage: ~35GB
Binary packages created: 1,124 .gpkg.tar files, 28.3GB

By October 27th, it was perfect. A lean, mean, optimized machine.

The Crash (October 27)

I shut down the VM. I went to bed.

The next morning, I booted it up.

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

I stared at the screen. Rebooted. Same error.

The Autopsy

I mounted the qcow2 image from the host:

sudo modprobe nbd
sudo qemu-nbd --connect=/dev/nbd0 /var/lib/libvirt/images/gentoo-desktop.qcow2
sudo mount /dev/nbd0p3 /mnt/gentoo

Ran fsck:

sudo fsck.ext4 -y /dev/nbd0p3

The output was horrifying:

/dev/nbd0p3: recovering journal
Clearing orphaned inode 262245 (uid=0, gid=0, mode=0100644, size=0)
Clearing orphaned inode 262244 (uid=0, gid=0, mode=0100644, size=0)
...
/dev/nbd0p3: ***** FILE SYSTEM WAS MODIFIED *****

Some freak host-system write error had mangled the ext4 superblock. fsck “recovered” it—by deleting corrupted files. Half of /var/db/pkg was gone.

What `/var/db/pkg` Means

In Gentoo, /var/db/pkg is everything. It’s the database of what’s installed.

The critical files at risk:

/var/db/pkg/ - 55,782 files tracking every installed package
/var/lib/portage/world - The list of explicitly installed packages
/etc/portage/ - All configuration (weeks of USE flag optimization)

If you lose /var/db/pkg, you have a zombie OS. The system runs, but you can’t update, remove, or install anything. Portage has no idea what’s on the disk.

50 hours of work. Gone.

The Backup That Saved Everything

Pure luck.

I had automated VM backups running on my NAS. Daily at 2:00 AM. 7-day retention.

There was a snapshot from October 27th at 2:00 AM—taken 6 hours before the corruption.

The backup wasn’t planned for this disaster. But it saved 40+ hours of work.

Backups you don’t have are useless. Backups you do have are priceless.

The Pivot: Embracing Btrfs

I sat there staring at the error message, and I realized: If this was openSUSE, I would just type snapper rollback and be drinking coffee by now.

I nuked the VM.

We were starting over. But this time, we were doing it right.

The New Filesystem Layout

I adopted the openSUSE partition strategy—complex, but it allows snapshotting the system without rolling back personal files.

The Partition Map:

Device	Size	Filesystem	Mount Point
/dev/vda1	512MB	FAT32	/boot/efi
/dev/vda2	8GB	swap	[swap]
/dev/vda3	241.5GB	Btrfs	/

The Subvolume Architecture:

# Create the subvolume structure
mount /dev/vda3 /mnt/gentoo
cd /mnt/gentoo

btrfs subvolume create @
btrfs subvolume create @home
btrfs subvolume create @snapshots
btrfs subvolume create @var-cache
btrfs subvolume create @var-log

Subvolume	Mount	Purpose
`@`	`/`	Root filesystem—frequently snapshotted
`@home`	`/home`	User data—separate snapshot schedule
`@snapshots`	`/.snapshots`	Snapper storage (excluded from snapshots)
`@var-cache`	`/var/cache`	Package cache—exclude from snapshots
`@var-log`	`/var/log`	Logs—persist through rollback for debugging

Why separate /var/log? If your system crashes and you roll back to a snapshot before the crash, you want the logs from the crash to persist. Otherwise, you can’t debug what happened.

Mount Options Matter

The /etc/fstab for Btrfs:

UUID=xxx  /           btrfs  defaults,subvol=@,compress=zstd:3,noatime,discard=async,space_cache=v2  0 1
UUID=xxx  /home       btrfs  defaults,subvol=@home,compress=zstd:3,noatime  0 2
UUID=xxx  /.snapshots btrfs  defaults,subvol=@snapshots  0 2
UUID=xxx  /var/cache  btrfs  defaults,subvol=@var-cache,noatime  0 0
UUID=xxx  /var/log    btrfs  defaults,subvol=@var-log,noatime  0 0

What each option does:

compress=zstd:3 — Transparent compression. 20-30% space savings. Negligible CPU overhead.
noatime — Don’t update file access times. Fewer writes, better SSD life.
discard=async — SSD TRIM support without blocking operations.
space_cache=v2 — Faster free space tracking.

Snapper: The 2-Minute Rollback

Snapper is openSUSE’s snapshot management tool.

emerge sys-fs/btrfs-progs app-backup/snapper sys-boot/grub-btrfs

The OpenRC challenge: Snapper expects systemd timers. Gentoo uses OpenRC. No problem—cron works fine:

# /etc/cron.hourly/snapper-timeline
#!/bin/bash
/usr/bin/snapper -c root create --cleanup-algorithm timeline --description "timeline"

# /etc/cron.daily/snapper-cleanup
#!/bin/bash
/usr/bin/snapper -c root cleanup timeline
/usr/bin/snapper -c root cleanup number

The killer feature: automatic package snapshots.

I added Portage hooks that create snapshots before and after every emerge:

# /etc/portage/bashrc.d/snapper-pre.sh
if [[ ${EBUILD_PHASE} == "setup" ]]; then
    snapper -c root create --description "Before emerge ${CATEGORY}/${PN}-${PVR}"
fi

Now every package installation is bracketed by snapshots. Break something? Roll back in 2 minutes.

GRUB Integration

grub-btrfs makes every snapshot bootable from the GRUB menu:

emerge sys-boot/grub-btrfs
grub-mkconfig -o /boot/grub/grub.cfg

Now GRUB shows:

Gentoo Linux (current system)
Advanced options
├─ Gentoo Linux (Snapshot 1) - Fresh installation
├─ Gentoo Linux (Snapshot 2) - Before emerge firefox
├─ Gentoo Linux (Snapshot 3) - After emerge firefox
└─ ...

You can test a snapshot before committing. Boot in, verify everything works, then decide.

The First Stability Milestone (November 15)

By mid-November, I had the system on bare metal.

We had:

KDE Plasma 6: Running on proprietary NVIDIA drivers (RTX 4070 Ti)
Sound: Pipewire (after fighting with ALSA for 3 days)
Browsers: Firefox (compiled with PGO)
Snapshots: Hourly timeline, pre/post package, GRUB bootable

And the test that mattered:

snapper create -d "Before breaking things"
rm -rf /lib/modules
# System is now dead. No kernel modules.
reboot
# Select snapshot from GRUB
# Desktop loads.

It worked. I had achieved the dream: An indestructible rolling release.

What I Learned

After the first month:

ext4 Is Fine Until It Isn’t

Traditional filesystems don’t have snapshots. One corruption event, one bad update, one rm -rf—and you’re rebuilding from scratch.

Btrfs snapshots aren’t a luxury. For a source-based distro where updates can break anything, they’re a requirement.

Binary Packages Are Essential

Nobody should compile KDE Plasma more than once. Build it on a dedicated machine, serve the binary, install in seconds.

Package	Compile Time	Binary Install	Savings
KDE Plasma	12 hours	45 minutes	94%
LibreOffice	2 hours	2 minutes	98%
Firefox	45 minutes	30 seconds	99%

The Golden Image Is a Baseline, Not a Clone

My “golden image” isn’t a fully-loaded system. It’s a bare-bones Stage 3 with SSH ready. Cloning takes 30 seconds. From there, I pull binaries from the binhost to customize it.

This keeps the golden image simple and the binhost as the source of truth for what packages look like.

Test Your Backups

A backup you’ve never tested is not a backup. It’s a hope.

I now intentionally break things just to verify rollback still works.

What Comes Next

In Part 2, the system breaks again—this time from a routine update. A Qt6/elogind mismatch brought down KDE Plasma. But this time, recovery took 2 minutes instead of 50 hours.

The difference? Snapshots.

This is Part 1 of a 4-part series. Continue to Part 2: The Qt6 Crisis.