user@argobox:~/journal/2023-09-30-the-463-message-saga
$ cat entry.md

The 463-Message Saga

○ NOT REVIEWED

The 463-Message Saga

Date: September 27-30, 2023 Duration: 4 days Messages: 463 (just ChatGPT — Claude got another 25) Issue: Everything about Docker and Traefik Result: Modern containerized infrastructure


The Setup

September 2023. I’d been running services the old-fashioned way — installed directly on hosts, manual port management, praying nothing conflicted. It worked. Mostly.

Then I discovered Traefik. Automatic service discovery. SSL termination. Clean URLs instead of http://192.168.1.50:8443. The dream.

How hard could it be?


Day 1: Optimism (114 messages)

Morning: Read the docs. Felt confident.

Afternoon: First docker-compose up. Container started. Dashboard appeared. “This is easy,” I thought.

Evening: Nothing routed correctly. Services unreachable. Traefik labels ignored.

Night: Stack Overflow. Reddit. More ChatGPT. Nothing worked.

Day 1 emotional arc: Confident → Confused → Frustrated → “I’ll figure it out tomorrow”


Day 2: Network Hell (83 messages + Claude)

The containers were running. Traefik was running. Nothing could talk to each other.

docker network ls
# traefik-public exists

docker network inspect traefik-public
# All containers attached

curl http://service.domain.local
# Connection refused

I tried ChatGPT. I tried Claude. I tried both simultaneously, comparing answers.

The images wouldn’t even pull reliably:

Error response from daemon: Get https://registry-1.docker.io/v2/:
net/http: request canceled while waiting for connection

Network instability. DNS issues. Docker Hub throttling. All at once.

Day 2 emotional arc: Determined → Exhausted → “Maybe containers aren’t for me”


Day 3: The 238-Message Day

Two hundred and thirty-eight messages. In one day.

Looking back, I understand why. I was learning:

  • Docker networking models (bridge, host, overlay)
  • Traefik dynamic configuration
  • Container DNS resolution
  • Inter-container communication
  • Cloudflare API integration
  • Cronjob automation

All at once. Every answer revealed three new questions.

10:00 AM: "How do I expose a container port?"
10:15 AM: "Why can't containers see each other?"
10:30 AM: "What's the difference between expose and ports?"
10:45 AM: "Why does DNS work sometimes and not others?"
11:00 AM: "How do Traefik labels work?"
11:15 AM: "Why are my labels being ignored?"
...
11:00 PM: "Is any of this actually working?"

The ruTorrent migration alone took 50 messages. Cloudflare DDNS was another 40.

Day 3 emotional arc: Information overload → Gradual understanding → “Wait, I think I see it now”


Day 4: The Breakthrough (225 messages + Claude)

Still broken. But I was close. I could feel it.

The final obstacles:

Obstacle 1: “Network already exists”

Error: network traefik-public already exists

Docker Compose kept trying to create a network that already existed.

# The fix - tell Compose the network is external
networks:
  traefik-public:
    external: true

Obstacle 2: Firewall

pfSense was blocking Docker traffic. The containers were up, the routing was configured, but packets were dying at the firewall.

Obstacle 3: Service Ports

Traefik needs to know which port inside the container to route to. I’d been specifying the wrong one for three days.

labels:
  - "traefik.enable=true"
  - "traefik.http.routers.app.rule=Host(`app.domain.com`)"
  - "traefik.http.services.app.loadbalancer.server.port=8080"  # THIS ONE

The Moment

Day 4. Approximately 11 PM. Message 460-something.

Everything aligned:

  • Docker network external: ✓
  • Traefik labels correct: ✓
  • Firewall rules added: ✓
  • DNS resolving: ✓
  • Cloudflare DDNS updating: ✓

I typed the curl command, fully expecting another failure:

curl http://rutorrent.domain.com
HTTP/1.1 200 OK
Server: nginx
...

I stared at the terminal.

200 OK.

It worked.

I may have yelled. My daughter definitely thought I’d lost it. “Did your computer do something?”

“It finally worked,” I said.

“Okay,” she said, and went back to Roblox. Fair enough.


What I Built

Before:

Services installed directly on hosts:
- Service1 on Host1:8080
- Service2 on Host2:3000
- Service3 on Host3:9000

Access: http://host-ip:port

After:

All services in containers:
- service1.domain.com → Traefik → container1
- service2.domain.com → Traefik → container2
- service3.domain.com → Traefik → container3

Access: https://service.domain.com (auto-SSL)

What I Learned

Technical

  1. Docker networks are weird. Bridge, host, overlay — they all behave differently, and none of them work the way you expect the first time.

  2. Traefik labels are picky. One typo and your service is invisible. The enable flag is easy to forget.

  3. Firewalls don’t know about containers. Docker creates its own network subnets. Your firewall needs explicit rules for them.

  4. UEFI variables don’t transfer. This became relevant later, but the lesson started here: VMs carry invisible state.

Process

  1. 463 messages is a lot. But also: 463 messages means I didn’t give up. Sometimes persistence is the skill.

  2. Use multiple AI sources. ChatGPT for rapid iteration, Claude for deep explanations. They have different strengths.

  3. Break time helps. Four days meant four nights of sleep. Fresh perspectives matter.

  4. Document everything. This journal exists because I learned to write things down.


The Architecture

Internet


pfSense (Firewall)


┌─────────────────────────────────────┐
│ Docker Host                          │
│                                       │
│  ┌─────────────────────────────────┐ │
│  │ Traefik (Reverse Proxy)          │ │
│  │  - Port 80 (HTTP)                │ │
│  │  - Port 443 (HTTPS)              │ │
│  │  - Auto service discovery        │ │
│  │  - SSL/TLS termination           │ │
│  └──────────────┬──────────────────┘ │
│                 │                     │
│    ┌────────────┼────────────┐       │
│    ▼            ▼            ▼       │
│  ┌────┐      ┌────┐      ┌────┐     │
│  │App1│      │App2│      │App3│     │
│  └────┘      └────┘      └────┘     │
│                                       │
│  Network: traefik-public              │
└───────────────────────────────────────┘


Cloudflare (DDNS)

Impact

This wasn’t just about Docker. This was the moment my infrastructure went from “thing I manage” to “thing I build.”

Before: Traditional sysadmin. Install software, configure services, hope nothing breaks.

After: Container-first thinking. Everything isolated, portable, reproducible.

The 463 messages weren’t wasted. They were tuition.


Files Created

FilePurpose
docker-compose.yml (Traefik)Reverse proxy configuration
docker-compose.yml (services)Per-service containers
/etc/docker/daemon.jsonDocker DNS configuration
Cloudflare DDNS scriptAutomatic IP updates
Cronjob entryDDNS automation

What Came Next

This foundation enabled everything that followed:

  • More containerized services
  • Kubernetes exploration (K3s, 16 months later)
  • The build swarm (62 cores, distributed compilation)
  • This very blog

The 463-message saga was just the beginning.


Sometimes the conversations with the most messages represent the most important learning experiences.