ArgoBox
ArgoBox Infrastructure & Automation
user@argobox:~/journal/2026-02-06-monitoring-in-60-minutes
$ cat entry.md

Monitoring Stack in 60 Minutes

[Friday, February 6, 2026] MOOD: LOG
○ NOT REVIEWED

Monitoring Stack in 60 Minutes

Date: 2026-02-06 Duration: ~1 hour (plus some bonus side quests) Target: Altair-Link (10.42.0.199) Services Deployed: 7

The Plan

Quick session. Deploy a full monitoring stack to Altair-Link. Prometheus, Grafana, Loki, cAdvisor, Smokeping, Healthchecks, Promtail. All containers, all Docker, one compose file.

Should take an hour. (Narrator: it mostly did.)

19:00 — Old Containers in the Way

First docker compose up -d failed. Old containers from a previous attempt had the same names.

Error response from daemon: Conflict. The container name "/prometheus" is already in use

Right. These were from a test run weeks ago that I never cleaned up.

docker rm -f prometheus grafana loki cadvisor smokeping healthchecks promtail

Seven docker rm -f commands. Seven ghosts cleared. Tried again.

19:10 — Port Shuffle

cAdvisor wanted port 8080. Something else was already on 8080.

ss -tlnp | grep 8080
# gateway-Altair already bound to 8080

The build swarm gateway. Not moving that. Bumped cAdvisor to 8082.

Final port layout on Altair-Link:

Prometheus       :9090
Grafana Enhanced :3006
Loki             :3100
cAdvisor         :8082
Smokeping        :8081
Healthchecks     :8084
Promtail         (no external port, pushes to Loki)

Everything came up clean. Grafana connected to Prometheus and Loki on first try. Dashboards loading. Metrics flowing.

19:35 — The npm Side Quest

Unrelated, but I noticed npm was missing on Capella-Outpost. Tried to run a quick npm install for a side project and got:

bash: npm: command not found

Checked the nodejs package. Installed, yes — but from the binhost. And the binhost build had USE="-npm". So I had nodejs without npm. A JavaScript runtime that can’t manage packages. Very useful.

The fix: rebuild nodejs locally with the npm USE flag.

echo 'net-libs/nodejs npm' >> /etc/portage/package.use/nodejs
emerge -1 net-libs/nodejs

On a binary-only Gentoo system. Where the entire point is that we don’t compile locally. Where I’ve built an entire distributed build swarm specifically to avoid compiling on workstations.

nodejs took 12 minutes. I watched the compiler output scroll by and appreciated the irony.

19:55 — Stack Verified

Back on Altair-Link. All seven services stable:

  • Prometheus scraping 4 targets, all UP
  • Grafana serving dashboards on :3006, data sources connected
  • Loki ingesting logs from Promtail
  • cAdvisor reporting container metrics on :8082
  • Smokeping pinging 6 endpoints, latency graphs building
  • Healthchecks monitoring 3 cron jobs
  • Promtail tailing /var/log and Docker container logs

Full observability for Altair-Link and every service running on it. One compose file. One hour.

Bonus: USB Housekeeping

While waiting for nodejs to compile (the irony never stops), reformatted an old OPNsense USB drive. It had a FreeBSD partition table that KDE refused to mount.

wipefs -a /dev/sdb
mkfs.vfat -F 32 /dev/sdb1

Reformatted to FAT32. Configured KDE automount in System Settings so removable drives mount to /run/media/commander/ on insert. Small thing, but I was tired of sudo mount every time I plugged in a USB drive.

Seven monitoring services deployed in an hour. One nodejs compiled out of spite. One USB drive returned to civilization.