Watchtower

What it is

Watchtower polls Docker registries for newer versions of every container running on a Docker host. Mine is configured monitor-only — it detects updates and sends notifications, but never pulls or restarts anything. Five instances total, one per Docker LXC, each watching only its local Docker daemon.

Why I run it

Two reasons it's monitor-only rather than auto-applying:

1. :latest is a surprise generator for stateful services. I learned this loudly during a password-manager upgrade where a point release shipped a UI rendering regression that didn't agree with my browser's cached state. Auto-updating Vaultwarden would have meant a black screen first thing in the morning, not "Watchtower says there's an update — let me read the release notes."
2. Knowing what's available matters as much as updating it. A weekly inventory of "what's drifted from upstream" is a useful homelab hygiene signal even when I don't intend to update everything.

The next layer up — the one I'm proud of — is the n8n + LLM pipeline that consumes Watchtower's notifications and gives them context. Raw "container X has a new image" isn't actionable. "Container X went from 1.42.3 to 1.43.0, here are the release notes, and the model rates this SAFE / REVIEW / WAIT based on whether the changelog mentions breaking changes" is.

How I use it

Each Docker LXC's Watchtower instance polls every 24 hours and emits a structured report (WATCHTOWER_NOTIFICATION_REPORT=true is critical — without it, the n8n parser only sees raw-text status lines and can't enumerate per-container updates reliably). Reports go to two places:

• A direct Discord webhook during validation windows.
• An n8n webhook that runs the advisor workflow.

The advisor workflow filters out startup pings and schedule heartbeats, parses the structured report, deduplicates by image hash, fetches GitHub release notes for known image-to-repo mappings, and sends the whole bundle to a local LLM (currently a 7B Qwen instruct model via LM Studio). The LLM returns a structured JSON verdict — SAFE / REVIEW / WAIT — and a brief rationale. That verdict gets posted as a Discord embed with a link to the release notes.

The Discord channel for advisories is deliberately separate from the channel that receives Grafana alerts and PBS reports. Volume management — I don't want a noisy update notification drowning out a backup failure.

Setup notes

• Per-LXC hostname identifier: each instance sets WATCHTOWER_NOTIFICATIONS_HOSTNAME to a unique string (docker-edge, docker-media, etc.) so the Discord notification tells me which LXC the update is on.
• Fork: I run nicholas-fedor/watchtower, an actively-maintained fork of the original containrrr/watchtower which went stale. Same compose interface, same env vars, plus modern shoutrrr support and better notification templates.
• Custom-built images: my Discord music bot is a locally-built image with no upstream registry. Watchtower would emit noisy "digest retrieval failed" errors for it every poll. WATCHTOWER_DISABLE_CONTAINERS: "<service>" skips it cleanly.
• Defensive enable=false labels: critical services (Vaultwarden, Plex, NPM, AdGuard) carry a com.centurylinklabs.watchtower.enable=false label as belt-and-suspenders. Even if monitor-only ever gets flipped off by accident, these four can't be touched.

Runbook

• Healthy looks like: 24-hourly polls happen on each LXC, the n8n workflow shows successful executions, Discord advisories arrive with the right hostname prefix.
• Discord URL format: the shoutrrr Discord transport uses a different component order from Discord's normal webhook URL. Keep the exact format in the private runbook; mixing the pieces up gives confusing 4xx errors with no obvious "auth failed" hint.
• No notifications even though Watchtower runs: check docker logs watchtower --tail=30 for the scan happening, verify WATCHTOWER_NOTIFICATION_URL is set, test the shoutrrr URL manually with docker run --rm containrrr/shoutrrr send -u ... -m test.
• Notifications from the wrong hostname: WATCHTOWER_NOTIFICATIONS_HOSTNAME is per-LXC. Verify each LXC's compose has the right value.
• Testing the advisor pipeline without waiting for a real update: POST a synthetic Watchtower payload to the n8n webhook with Found new image: lines and fake hashes. The pipeline runs end-to-end without needing an actual registry change.
• Where logs live: per-LXC docker logs watchtower, n8n's Execution history for the advisor workflow, Discord for the user-facing output.