NetBird incident on November 6, 2026: We are currently investigating reports of an issue affecting Management Service. Our team is working to determine the root cause and will pr Started 11:21 AM UTC, resolved ongoing (● 6m). Affected: Management Service.

WireGuard®-based overlay network and Zero Trust Network Access in one platform for reliable and secure connectivity

Over 14 days, Checkmk raised about 2,200 alerts for interface errors on `wt0`. That's the NetBird tunnel interface, and it was being held to error levels built for a copper NIC. I run one Checkmk server that watches every box I own, across more than one site. None of those boxes expose an agent port to the internet. Agent traffic rides the NetBird overlay, the same mesh I already use for everything else. My Checkmk settings assumed a LAN, though, and an overlay isn't one. I tried Beszel first, but it didn't have everything I wanted to monitor. The layout was great. I wanted more. ## One server pulling over the overlay The server is Checkmk 2.5 Community edition, running in Docker on mon-01. Three choices in the compose file matter: * The image tag is pinned, so it only moves when I move it. * It uses `network_mode: host`. * One named volume holds `/omd/sites`, which is the whole site. There's no auto update label on it either. Every monitored host runs the Checkmk agent and is a NetBird peer. The server pulls agent data from each host on port 6556, over the overlay. That path never touches the public internet, so there's nothing to forward and nothing to open on the edge router. ## Registering agents over the overlay The agent controller, `cmk-agent-ctl`, registers each host with the server and sets up TLS for the agent connection. To register, a host has to reach the server's REST API on 443 or 80 and its agent receiver, which sits on 8000 for the first site. Day to day, the data flows the other way: server to agent on 6556. That's the only mode Community edition has. Push mode is a Checkmk Ultimate feature. The only thing specific to the overlay is the address. Register against the server's NetBird name or its 100.64.x.x address, not a LAN address that only works from one site. Same for the host entries in Checkmk. Each one gets its overlay address, so the server reaches it the same way no matter where it lives. On the NetBird side, the policy only opens what each direction needs. The monitoring server's group can reach the agent port on the monitored hosts' group. The hosts can reach the server's web port and agent receiver port, for registration only. I'm not listing my exact rules, but that's the pattern. If there's a routed network behind a peer, the same idea applies to the resource instead of the host. I covered how resources work in the post on LAN only services over NetBird. ## The host firewall is what keeps the UI private With host networking, the site's web server listens on every interface of the host. Not just the overlay one. So the host firewall has to do the work Docker isn't doing. The pattern is simple: allow inbound traffic on the NetBird interface and drop the rest. Host networking adds no isolation of its own, so I count that firewall rule as part of the monitoring setup. If it gets loosened, or the container moves to a box with a public address, the UI goes with it. ## Three sources of noise and the rules that fixed them Once every host was in, the alerts started and didn't stop. I lived with the alerts for a few weeks. Then I dug into the deeper issues to make it stop reporting false positives. When I measured it, three things were going on. ### Interface errors on the tunnel Checkmk monitors every network interface, `wt0` included. My hosts had it on error levels of 0.01% for a warning and 0.1% for critical. That's reasonable for a physical NIC. On a WireGuard tunnel the numbers don't mean much, and they're where those 2,200 alerts came from. The fix is a network interfaces rule that raises the error levels to 50% warn and 90% crit, scoped to the tunnel's item only. On my hosts `wt0` always lands at the same item index, so the rule matches that index and leaves the real NICs alone. I check the scoping before trusting it. `cmk -D <host>` prints the effective parameters per item, so I can see the tunnel picked up the new levels and the physical interface didn't. ### One missed ping went straight to DOWN The host check template Checkmk generated on my site had `max_check_attempts` set to 1 and a check every minute. A single missed ping went to a hard DOWN state and sent a notification. There was no soft state buffer at all. Over 30 days I counted the host outages and how long each one lasted. 43% recovered inside 60 seconds. 75% recovered inside three minutes. That pattern fits overlay churn. A peer redoes its WireGuard handshake, or a connection falls back to a relay for a moment, then comes back. ### Ping levels sized for a LAN My hosts were running ping thresholds of 200 ms round trip for a warning and 500 ms for critical. Healthy hosts on my overlay spiked anywhere from about 540 ms to 3 seconds. I actually had a rule with looser ping levels already. It was tied to a `wan` host tag. Every host was tagged `lan`, so the rule had never applied to anything. Dead config. ### Tolerance rules at the root folder I replaced all of that with three rules at the root folder, with no conditions, so they hit every host: * host `max_check_attempts`: 4 * host `retry_interval`: 1 minute * ping levels: round trip 2,000 ms warn and 5,000 ms crit, packet loss 80% warn and 100% crit, 6 packets, 20 second timeout A host that's really gone now goes hard DOWN at about three minutes. A host that blinks for 40 seconds never pages me. Services got the same treatment: `max_check_attempts` 4 and a 1 minute retry. Agent data arrives once a minute, so that's the same three minute buffer. ### Checking the generated config A rule existing doesn't mean it applied. Checkmk writes the real monitoring config out to files, so I read those instead. Count the attempt values across every object: grep -hoE 'max_check_attempts +[0-9]+' \ /omd/sites/<site>/etc/nagios/conf.d/check_mk_objects.cfg | sort | uniq -c If some still read 1, the rule isn't matching them. The `wan` tag above is the kind of thing to look for. ### Two side effects of the retries On my site the discovery services check every 120 minutes, set by the periodic discovery rule. With 4 attempts, a discovery warning now takes about six hours to notify. I left it that way. And not every flapping service is an overlay blip. Before adding retries to something noisy, I look at its median outage length. If that's minutes or hours, retries won't help, and the real fix is the threshold. With all of it in place, I get a couple of alerts a week instead of dozens a day. That lets me focus on the real problems instead of learning to ignore the alerts entirely. ## Keeping it healthy through upgrades Upgrading the container is mostly painless. The documented path is to stop and remove the container, then start the new version on the same volume. With compose that means bumping the image tag and running `docker compose up -d`. On my server the site updated itself when the new container started. Four habits keep it that way. (The NetBird server has its own routine, in the NetBird server upgrade post.) ### Back up first, every time Checkmk's own update docs say to back up immediately before an update. I don't count on the old image tag to get me back. My rollback is the volume, so I stop the container and tar it before every upgrade. Major versions have one more rule. If the site isn't on the last patch of its current major version, update to that patch first, then jump to the new major. ### Move the agents in the same change I roll agents out with Ansible, and the playbook pins the agent version. That version changes in the same commit as the image tag. If the server moves and the playbook doesn't, the agent download URLs 404 and every host reads as out of date until the rollout runs. The server's own agent is the exception. I update that one by hand afterwards. ### Check the agent receiver after every upgrade After one upgrade the agent receiver didn't start, and it stayed down for five days. Nothing alerted. The server pulls agent data, so everything kept flowing. The only symptom was that new or reimaged hosts couldn't register. Now `omd status` is part of every upgrade. If it says "partially running", I find out which part. ### Discovery mode after an agent rollout After another rollout, two hosts lost every service they had. The playbook ran discovery in `fix_all` mode right after reinstalling the agent. Those two hosts answered empty while their agents restarted, and Checkmk saved them with zero services. I switched discovery to `new`, which only adds. Services that really disappear aren't removed automatically anymore. They show up on the host's discovery check and I clear them by hand. I'd take that trade over a host silently losing its monitoring. After any agent rollout I also compare service counts host by host. ## What I checked and what I didn't The tuning was done in September 2026. The server has been upgraded since, and the rules carried over. checkmk: 2.5.0p12 community edition (tuning), 2.5.0p17 (current) docs checked: checkmk 2.5.0 (latest), 2026-10-28 deployment: docker, network_mode host, one named volume agent mode: pull over tcp 6556 overlay: netbird (wireguard) Measured on my own server: * about 2,200 interface error alerts on `wt0` over 14 days * host outage durations over 30 days: 43% under 60 seconds, 75% under three minutes * healthy ping spikes from about 540 ms to 3 seconds * the agent receiver down for five days after an upgrade * the generated config reading 4 attempts after the root folder rules Not proven: that the short outages were handshakes and relay fallbacks. That's the likely cause on an overlay, but I didn't trace individual blips back to NetBird's logs. I also haven't checked whether the starting values my site ran with are Checkmk's shipped defaults or something inherited along the way. ## What I'd tell myself The settings weren't wrong. They were sized for hosts on the same switch. A 2 second ping over a tunnel is a normal Tuesday, and the rules should say so before the first host goes in. And after every upgrade, check that the thing watching everything is still running all of its parts. Names and addresses are changed. Everything else happened.

This article provides a guide to deploy Netbird on Debian VPS.

* WRITER 2026-11-04: backlog #18 resumed as the oldest stalled draft (standing order 2026-11-03). Source draft: agent workspace revenue/blog/daily/docker-outbound-ipv6/ (brief.md + post.md, written 2026-10-29). Slug docker-outbound-ipv6 is not in the live sitemap (checked 2026-11-04). * Theme fixes this pass: added topic networking, tldr (plain text, under 300 characters), feature_image_alt describing the image, tested-on block, and console fences with $ prompts. Tags now docker, ipv6, compose (networking moved to topic). Opening paragraph reworded so it does not repeat the tl;dr. * Console blocks carry commands only. The source note records no command output, so none is pasted. Do not add invented output. * Delivers on post #7 (docker-network-segmentation), which says the IPv6-only network is "its own post". Link it root-relative from the first section: /docker-network-segmentation/ * RESEARCHER 2026-11-04 (5 of 5 fetches, notes in researcher workspace revenue/blog/daily/docker-outbound-ipv6/research.md): verified network connect on a running container, compose external: true, implicit default dropped when a service lists networks, ip6tables on by default in current Engine, current Engine 29.9.3 (2026-11-03). Not verified: --ipv4=false release, ip6tables-default release, documented ULA masquerade. Changes made: (1) masquerade paragraph no longer claims "on by default since v29"; it now says current Engine enables ip6tables by default and keeps the NAT as observed on this host; (2) router advertisement caveat CUT (unverified). tldr checked, no change needed. * tested-on: Portainer and Watchtower versions are not in the note. Commander may add exact versions, otherwise leave as is. * Placeholders only: docker-01, v6-egress, fd12:3456:789a:1::/64, eth0, app-web, app-db, app-redis, example/web. No real prefix, network names, NIC, counts beyond "about a dozen", or stack numbers. * About 1,200 body words, short of the 1,500 target on purpose. * DESIGNER 2026-11-04: feature image done in designer workspace revenue/blog/daily/docker-outbound-ipv6/ (docker-outbound-ipv6.svg and docker-outbound-ipv6.png, 1200x675, slate/green terminal brand, no window chrome because the theme adds it). Image headline "Outbound IPv6, no restarts". Labels: docker-01, app-web, app-db, mail, v6-egress, fd12:3456:789a:1::/64, eth0, internet; all are in the post. Green marks only the outbound path. Lint: no IPv4 strings, no forbidden SVG elements, no blog domain. Checked at 600 px wide: nothing clipped or overlapping. feature_image_alt rewritten to describe the image. * GHOSTWRITER 2026-11-04: voice pass against renstur-voice.md v1. Opening now names Docker in the first sentence. Added the root-relative link to /docker-network-segmentation/ the writer asked for. Softened "Watchtower keeps a container's networks" to what was observed, since the researcher didn't verify it. Added a short "What I saw and what I didn't check" section (checklist rule 13) built only from facts already in the draft and research notes; no new experience invented, so no FILL slots. tldr reworded so it names docker network connect in plain text and ends on what I did; checked plain text, under 300 characters, not a copy of meta_description. No em dashes, no bold, title sentence case. Topic, tested-on block and all ## sections kept. * PRIVACY GUARD 2026-11-04: final gate CLEARED. Checked body, tldr, tags, tested-on, alt text, SVG and PNG. 0 IPv4 strings, 0 domains, 0 real hostnames or usernames; only placeholders (docker-01, v6-egress, fd12:3456:789a:1::/64, eth0, app-web, app-db, app-redis, mail, example/web). One edit: "Postfix" became "the mail server" so the post does not fingerprint the mail stack. Kept "about a dozen" and "June 2026" (generic). renstur-voice.md in these notes is internal and goes when the notes are removed. Every app container on my Docker host could reach the internet over IPv4 and nothing else. The host itself had a working global v6 address. I wanted the containers to have outbound IPv6 without restarting anything and without opening any of them up to inbound v6. In June 2026 I got there with one extra network and a `docker network connect` per container. docker engine: 29.x compose: compose spec, external networks portainer: stacks from on-disk compose files watchtower: image updates enabled ## What I wanted, and what I didn't The host already had a global IPv6 address on its uplink. The goal was egress only. Containers should be able to open connections out over v6. Nothing on the internet should be able to open one in. I also didn't want to touch the existing networks. The app containers already sat on their stack's default network with IPv4, and that was fine. Rebuilding that network with IPv6 turned on would have meant recreating every container on it. So the v6 side went on a separate network, and the containers got a second interface. In my post on splitting one Docker host into network tiers I said the IPv6-only network was its own post. This is it. ## An IPv6-only network with a ULA prefix Docker can create a network with no IPv4 at all. I gave it a ULA /64, so the container addresses are private and only the host's global address ever shows up on the internet. Create the network: $ docker network create --ipv6 --ipv4=false --subnet fd12:3456:789a:1::/64 v6-egress Pick your own ULA prefix. The `fd` range is meant to be generated randomly per site, so don't copy mine. On my host, Docker masqueraded outbound v6 traffic from that ULA subnet behind the host's global address, the same way it does for IPv4. That relies on ip6tables, which current Docker Engine turns on by default. If you've switched it off in `daemon.json`, this won't work until you turn it back on. ## Turning on forwarding for good The host has to forward v6 packets between the bridge and the uplink. I put it in a file of its own so it survives a reboot: net.ipv6.conf.all.forwarding=1 Load it without a reboot: $ sudo sysctl --system ## Attaching running containers No restarts here. `docker network connect` adds a network to a running container, and the container gets a new v6 interface on the spot: $ docker network connect v6-egress app-web I did that for every container in the app stack, about a dozen of them. Each one kept its IPv4 address on the stack network and picked up a second address from the ULA range. Then I took a few back off. The database and Redis containers don't need to reach the internet, so they don't get a route to it: $ docker network disconnect v6-egress app-db $ docker network disconnect v6-egress app-redis Check what a container ended up with: $ docker exec app-web ip -6 addr $ docker exec app-web ip -6 route You want an address from your ULA prefix and a default route through the network's gateway. To test egress, use whatever the image ships with. `curl -6` or `wget -6` against any site that answers over v6 will do. Plenty of slim images have neither, so test from one that does. ## The container I left off The mail container stayed IPv4 only, on purpose. If it had a v6 route, outbound mail could leave from the host's global v6 address. Receiving servers check that address. Without a matching reverse DNS record and an SPF record that covers it, that mail is a lot more likely to be rejected or filed as spam. So the mail server waits until the v6 address has rDNS and is in SPF. Until then it sends over IPv4, where both are already set up. If you run mail in a container, leave it off this network until you've done the same. ## What survives, and what doesn't The attachments held up better than I expected. They survived: * a host reboot * a Docker daemon restart * Watchtower pulling new images and recreating containers Every time Watchtower recreated a container, the second interface came back with it. A stack redeploy is different. Redeploying a stack from Portainer rebuilds the containers from the compose file. The compose file doesn't know about `v6-egress`, so the new containers come up without it. No error. The v6 interface is just gone. Anything attached by hand only lasts until the next redeploy. ## Making it permanent in compose The fix is to put the network in each service that should have it. The network already exists, so compose should treat it as external and not try to create it: services: web: image: example/web:latest networks: - default - v6-egress networks: v6-egress: external: true Keep `default` in the list. Once you name networks on a service, compose only attaches the ones you list. Drop `default` and the container is cut off from the rest of its stack. Leave the database, Redis and mail services alone. They stay off it. Then redeploy the stack. This time the redeploy is what adds the network, so the problem goes away instead of coming back. On my host, Portainer's stack editor reads the compose file from disk, so editing the file directly and redeploying from Portainer stuck. If your Portainer keeps the stack in its own database instead, make the edit in the Portainer editor or it'll get overwritten. ## What I saw and what I didn't check Observed on my host, Docker Engine 29.x, June 2026: * outbound v6 from the ULA subnet left NATed behind the host's global address * hand attachments survived a reboot, a daemon restart and Watchtower recreates * a Portainer stack redeploy dropped them without an error Not checked: which Engine release added `--ipv4=false` or made ip6tables the default, and whether Docker documents the ULA masquerade or I just got it. I also didn't read Watchtower's code to see how it carries networks over. I only watched it happen. ## What I'd tell myself Put the network in compose first and attach by hand second. The live attach was nice because nothing restarted. But it left me with a setup that only half existed on disk, and the next redeploy would have taken it away without saying anything.

netbird v0.82.0 released Release Notes for v0.82.0 Highlights 🛣 Latency-aware HA routing that holds stable routes Routing peers in an HA group with equal metrics are meant to be chosen by latency, bu...

My self-hosted NetBird reverse proxy was relaying everything. A media server I publish through it streamed fine, but every packet between the proxy and that server's peer went through NetBird's relay instead of a direct WireGuard path. ## What the relay was costing The setup is one small public VPS running the official self-hosted NetBird Docker stack. That's Traefik, the combined `netbird-server` (management, signal, relay and embedded STUN on 3478/udp), the dashboard, and NetBird's reverse proxy component, which publishes self-hosted services to the internet. With two remote streams going through the relay, the VPS sat at about 27% CPU. After the fix, with one remote stream of about 8.7 Mbps going direct, it was about 4%. `netbird-server` dropped under 1% because the relay had nothing to do. Not the same load, so not a clean benchmark. The relay going idle is the part I trust. There turned out to be two separate problems. ## A random WireGuard port The proxy's embedded NetBird client was running with `wg_port: 0`, which means a random port. Compose was publishing `51820/udp`. So even a perfect candidate would have pointed peers at a port nobody was listening on. Pin it in the proxy's env file: NB_PROXY_WG_PORT=51820 The release notes call the setting `--wg-port`. The env name above follows the same `NB_PROXY_` pattern and is what worked for me. That alone didn't get me off the relay. ## STUN on the same Docker host lies The proxy asks a STUN server what its public address is. In my case that STUN server was the embedded one in `netbird-server`, on the same Docker host. The request never left the box. The answer came back as the Docker bridge gateway address, `172.x.0.1`, instead of `203.0.113.10`. So the proxy never had a public (server-reflexive) candidate to offer. It asked its own house for its street address and got the room number. An external STUN server, queried from the same container, reported the public IP correctly, port preserved. That confirmed the hairpin. ## The coturn detour My first fix that worked was a STUN-only coturn on a second, separate public host. I firewalled it so only the VPS's address could reach it, and added it to the server's STUN list. Coturn here was only a STUN workaround, not NetBird's relay. The docs treat coturn as part of the older multi-container layout anyway. Two traps on the way. In my setup, setting `server.stuns` in the combined server's config turned the embedded STUN off completely. The log said so: STUN: disabled The advertised list became exactly what I put in `server.stuns`. I couldn't have embedded plus external. Second, the coturn 4.18 container crash-looped when I passed `--no-dtls`. Dropping the flag fixed it. ## Mapping the public address in v0.82.0 NetBird v0.82.0 (29 Oct 2026) added `NB_PROXY_EXTERNAL_IP_MAP` for the proxy's embedded client, the env form of `--external-ip-map`. The release notes say a proxy "in a container or behind NAT can advertise its public address and peers reach it directly instead of over relay." They also say it needs a fixed, forwarded `--wg-port`, and that it applies to single-account deployments. The format that worked for me is `<public-ip>/<container-ip>`: NB_PROXY_WG_PORT=51820 NB_PROXY_EXTERNAL_IP_MAP=203.0.113.10/172.x.0.40 Recreate only the proxy. The compose service is named `proxy`, the container `netbird-proxy`: $ docker compose up -d --no-deps proxy $ docker logs netbird-proxy 2>&1 | grep -o 'nat_external_ips: .*' nat_external_ips: [...] I retired the coturn workaround the same day. The embedded STUN is the only one again, every client shows one STUN server as Available, and the proxy is P2P to every peer it talks to. IPv6 takes a second, comma-separated element: NB_PROXY_EXTERNAL_IP_MAP=203.0.113.10/172.x.0.40,2001:db8::10/fdxx:x::40 Before that, the proxy only advertised its useless ULA address. IPv6 in Docker is set per network (`enable_ipv6: true` on the compose network), not in `daemon.json`, which is where I looked first. Once it was in, an IPv6-capable peer switched to a direct v6 path. One more trap: in my setup, if one element of the map fails to parse, NetBird drops the whole map. Re-check the candidates after every edit. ## Proving it actually went direct `netbird status -d` shows only the candidate pair ICE selected. Mine labelled it `prflx` before and after the fix, so it told me nothing. The client debug log is better. Raise the level on a peer: netbird debug log level debug Restart the proxy, then look for the candidates it sent: $ grep OnRemoteCandidate <client-log> | grep <proxy-key> | grep -o 'udp4 host [^ ]*' udp4 host 203.0.113.10:51820 That candidate is generated locally from the map. No STUN involved. A junk candidate with the bridge address still shows up from the hairpin STUN. It's harmless. Set the level back when you're done: netbird debug log level info The docs say a daemon restart resets it too. `netbird down` and `netbird up` don't reset it. Side effect of all the restarting: every proxy restart registers a new ephemeral proxy peer. Check for orphans afterwards and delete them through the API. In my setup a GET on a proxy-owned peer returned 422, but DELETE worked. ## How sure I am Observed: the relay going idle, the CPU drop, the bridge address in the hairpin STUN answer, the public candidate appearing in the debug log once the map was set, and the v6 peer going direct. Documented: the env variable, the fixed-port requirement and the single-account limit, all from the v0.82.0 release notes. The map format, the parse behaviour, the `server.stuns` behaviour and the 422 on proxy peers are only what I saw on 0.82.0 in October 2026. netbird: 0.82.0 (proxy and server) netbird-server: combined container, config.yaml reverse proxy: Traefik (official self-hosted stack) coturn: 4.18 (temporary STUN-only workaround, retired) checked: 2026-10-31 Names and addresses are changed. Everything else happened.

My Ansible playbooks run from Semaphore UI now. Weekly patching with a reboot, a nightly NetBird agent update, and a baseline that keeps hosts in line, all on cron, all with their output kept. Before that they ran when I SSHed somewhere and typed the command. Semaphore fixed that without turning into a project of its own. It's a web front end that runs `ansible-playbook` for you, keeps every run's log and fires templates on a schedule. It's one container plus a database. This is how mine is set up, and the traps I hit getting there. ## What Semaphore is made of The UI and the API use the same small vocabulary, so it's worth learning once. * A project holds everything else. I run one project for the fleet. * A repository is the git repo your playbooks live in. Before 2.20, Semaphore keeps a separate clone per template and pulls it before every task. * The key store holds SSH keys and become passwords. Inventories and repositories reference keys by entry, so a key never goes into a playbook. * An inventory is the host list. It can be typed inline or point at a file in the repository. * A variable group is a set of extra variables, environment variables and secrets handed to the run. Older builds and the API call it an environment. * A task template ties all of that to one playbook. A task is one run of a template. * A schedule is a cron expression attached to a template. Everything below is one of those seven things. ## Run it in Docker and pin the tag The install is a compose file with the Semaphore container and a database. The one change I'd make to any example you find: don't use the `latest` tag. I found that out the hard way, when the task output view turned into an endless loading skeleton and the logs filled with this: gorp: no fields [workflow_run_id workflow_node_id artifacts] in type Task Upstream has been pushing 2.18.x backports alongside 2.19.x releases. A 2.18 backport landed a couple of hours after a 2.19 release, `latest` went back to 2.18 on my host, and Watchtower pulled it that night onto a database already migrated by 2.19. A restart wouldn't have helped. The older binary doesn't know the new columns. The fix was to pin the exact version and take Semaphore away from Watchtower: services: semaphore: # Pinned on purpose. latest can move backwards between release lines. # Bump by hand, and never below the version the DB is migrated to. image: semaphoreui/semaphore:v2.19.16 There's no floating `v2.19` tag on Docker Hub, so pinning means bumping by hand. Also check Docker Hub before you bump. As of 3 November 2026, GitHub lists v2.19.18 as the latest release, but there's no v2.19.18 image yet. The newest image is v2.19.17. Check which schema version the database is on before you pick a tag. The migrations table keeps one row per applied migration, so ask for the newest: docker exec semaphore-db psql -U semaphore -c 'SELECT version FROM migrations ORDER BY upgraded_date DESC LIMIT 1;' Leave `-i` off that one if it ever goes into a script piped over `ssh ... bash -s`. With `-i`, psql eats the rest of the script from stdin. You can also see what an image is before you run it: docker run --rm --entrypoint semaphore semaphoreui/semaphore:v2.19.16 version On my build the suffix on the version string is the app commit. That's how I found out one newer point release reported the same commit on my host as the one before it, with refreshed OS packages. Take a database dump before any bump. It's the only undo you get: docker exec semaphore-db pg_dump -U semaphore semaphore | gzip > semaphore-db-$(date +%Y%m%d)-pre-bump.sql.gz ## Keep inventories in git, not in the UI Semaphore lets you type an inventory into a text box. I started that way and moved off it. Every inventory is now type `file`, pointing at a relative path in the repo like `ansible/inventory/fleet.ini`. The repo is the source of truth: I edit the file, push, and the next run pulls it. That gives every host change a review, a history and a diff. I have three inventories that overlap (patching, monitoring and the overlay peers), and they have to stay in step. When a host's SSH port changes, it changes in all three in one commit. I once considered a fourth inventory for a single template and dropped it. Three already need keeping in step. A host pattern needs no syncing. Before pushing an inventory change, I check how Ansible will parse it, inside the Semaphore container so it's the same Ansible the runs use: $ docker exec -i semaphore sh -c 'cat >/tmp/x.ini; ansible -i /tmp/x.ini patch-group --list-hosts' < ansible/inventory/fleet.ini hosts (3): app-01 web-01 mon-01 The `-i` on `docker exec` matters here. Without it stdin is dropped and you test an empty file. Two habits came out of this. First, add a new host after the last host line of its group, not at the end of the section. One of my groups ends with a comment that introduces the next group, and a host appended there looks like it belongs to the wrong one. Second, when reviewing a rename, check the address, not the label. A rename that leaves `ansible_host` untouched points the new name at the old machine. It reviews as cosmetic and nothing fails. The damage only shows up when a scheduled patch reboots the wrong box. ## Target hosts with a required survey variable Every template that can touch more than one host takes its target from a survey variable, and the play refuses to run without it: - name: System baseline hosts: "{{ target | mandatory }}" serial: 5 Survey variables reach Ansible as extra vars by default. The run dialog prompts for `target`, and I type a host, a group or `all`. Blank fails closed. I went this way because the built-in limit field didn't apply `--limit` on the build I was running at the time, which I wrote up in Semaphore's task limit does nothing. The survey variable works from the API too, which is how I trigger one-off runs from scripts. The task endpoint takes an `environment` field as a JSON string, and on my build that's where the survey values go: curl -s -X POST https://auto-01.example.lab/api/project/<project-id>/tasks \ -H "Authorization: Bearer $SEMAPHORE_TOKEN" \ -H "Content-Type: application/json" \ -d '{"template_id": <template-id>, "environment": "{\"target\":\"web-01\"}"}' One host never belongs in any pattern: the box Semaphore runs on. Ansible runs inside the container, so SSH to its own host arrives from the Docker bridge instead of the overlay. My host firewall only accepts SSH on the overlay interface. It times out. Every time. In my runs, one unreachable host marks the whole Semaphore task as an error, even when every real target succeeded. So any pattern that might expand to everything excludes the controller unconditionally: hosts: "{{ target | default('monitored') }}:!auto-01" The controller patches itself with systemd timers on the host instead. The box running the updates shouldn't be rebooting itself mid-run anyway. ## Schedules, history and the rerun trap Schedules are plain cron on a template, and Semaphore evaluates them in UTC unless you set `SEMAPHORE_SCHEDULE_TIMEZONE`. I left mine on UTC, and the repo README says so up front, because the one thing on the box that isn't UTC is Watchtower. I keep a window clear so a nightly agent update and an image pull never land on the same minutes. Every task keeps its full output, and the survey values it ran with sit in its `environment`. When a run fails, the answer is usually in that log by the time I look. One history behavior bit me early on. On my build, rerunning a failed task reuses the original task's commit, not the branch head. I pushed a fix and hit rerun 97 seconds later. The log fetched my new commit and then checked out the old one. Same bug, faithfully re-run. After pushing a fix, start a new run from the template. Don't rerun. To confirm what code a run used, the per-template clones live under `/tmp/semaphore` inside the container, which is the default temp path. Compare `git show origin/main:<path>` against the checked-out file. Be careful in there: in my setup those clones carry the repo token in the remote URL, so `git remote -v` prints a live credential. ## Secrets and two Ansible traps SSH keys and become passwords go in the key store. Everything else goes through variable groups. The plain variables in a group come back readable through the API, so I treat them as config that's better than git, not as a vault. Values on a group's Secrets tab are different: the API doesn't return them, and they reach Ansible in a Vault-encrypted file. Even so, anything that can't stand being read by someone with project access lives somewhere else. Two Ansible traps only showed up once Semaphore started running things fleet-wide. `run_once` plus a `when` guard is the first. `run_once` executes on the first host of each batch, so with `serial: 5` that's once per five hosts, and the `when` is evaluated against that host. My first host had no Docker, so a staging step guarded by "Docker exists" was skipped for that whole batch. Every host in it that did have Docker then failed looking for a file that was never fetched. Earlier runs had passed only because they were single-host runs against hosts with no Docker. Put `run_once` staging tasks at the top level, unguarded, and keep only the per-host work behind the condition. The second: if your playbooks reach hosts over a VPN overlay, upgrading the overlay agent cuts the connection Ansible is using. Fire that upgrade detached with `async` and `poll: 0`, then `wait_for_connection` before checking the result. The rest of what I do to keep one wedged host from stalling a run is in Stop Ansible hanging on a reachable host. ## What I'd tell myself The playbooks didn't get better. They just started running. Patching happens weekly, agent updates happen nightly, and when something breaks there's a log with a timestamp instead of my memory of a terminal I closed. Pin the image before the first update, not after the downgrade. Put inventories in git on day one. Make `target` required, exclude the controller, and start a new run after every fix. ## How sure I am Checked against Semaphore's docs, its API spec and Docker Hub: file inventories with a repo path, the pull before each task, survey variables as extra vars, the `environment` field on the task endpoint, UTC schedules, the missing floating `v2.19` tag, the `/tmp/semaphore` default, and Secrets not coming back over the API. The `run_once` batch behavior is in Ansible's docs. Observed on my own host and not in any docs I found: the `no fields` errors after the downgrade and their absence after pinning, limit-tagged runs going fleet-wide, the rerun checking out the old commit, one unreachable host failing the whole task, the token in the clone URL, and two point releases reporting the same commit. Not checked: why `latest` moved back to 2.18. Docker Hub only shows where tags point now, not where they pointed back then, so that part is my reading of push times. I also haven't checked whether the limit field is still broken on current releases. semaphore: v2.19.16 (Docker) ansible: 13.5.0 ansible-core: 2.20.10 python: 3.12.15 database: PostgreSQL Names and addresses are changed. Everything else happened.

Accessing your lab away from home shouldn't be complicated

This article provides a guide to deploy Netbird on Debian VPS.