Post cover image

August 27, 2026

The New Monitoring System Found the Old One’s dead body in Twenty Minutes

Subscribe here so you won’t miss future updates:

By Sigmund Brandstaetter CISSP, CCSP, CISM, OSCP, CEH

10 min read

OSINT and Cybersecurity Blog Stories around OSINT, Cybersecurity, Threat Intelligence, and other related topics.

How I replaced a Nagios deployment I forgot I had with Beszel, a tailnet-only hub on hardware I own, and a dead man's switch that watches the watcher.

None

Every fleet has a monitoring story, and most of them are embarrassing. Here is mine: fifteen VPS instances (9 still need enrollment as of this writing, but that is a one line command on each now that the core config and notification mechanism is in place), 2 self hosted Linux servers I depend on daily, and the day I deployed a new monitoring platform, it found a Nagios Core server running on my own hardware that I had completely forgotten existed. Better yet, every one of its NRPE clients had been failing at boot for months. The monitoring system was dead, and nothing was monitoring the monitoring system. Good Job Sigmund.

Why Beszel

I wanted basic fleet telemetry: CPU, memory, disk, network, service status. Free, self-hosted, and light enough that the agent does not compete with the workloads it watches. Yes, there are so many others, and as you already read, i tried before but lost interest 🕶️

The obvious candidates each had a catch. Netdata is superb per-host but its Cloud free tier caps concurrent visualization at 5 nodes, useless for the number i have to monitor, and running your own parent node costs real RAM per box. Grafana Cloud's free tier is genuinely generous at 10K active series, but a stock node_exporter emits several hundred series per host, so fifteen nodes sit right at the ceiling unless you spend your evening pruning collectors.

Beszel is a different animal. Hub and agent, both single Go binaries. The agent idles around 20 MB. The hub is PocketBase underneath, so you get a REST API, automatic S3 backups, and OAuth for free. It covers host metrics, per-partition disk and I/O, temperatures, S.M.A.R.T., Docker and Podman container stats, and, the sleeper feature, per-unit systemd service status. That last one is what found the dead body.

What it deliberately does not do: log aggregation, tracing, custom application metrics, PromQL. If you need those, you bolt Prometheus or Loki alongside. For fleet vitals, Beszel is the right size.

The architecture decision that matters

The interesting choice is not the tool, it is where the hub lives.

All fifteen VPS instances sit with one provider, different continents, but still. Host the hub there too and a provider-level incident takes out the fleet and the thing that tells you about the fleet in the same instant. So the hub went on-prem, on a spare Linux box on my desk, yes, its not very enterprise grade but it serves the purpose, with UPS being in place and all.

An on-prem hub behind residential NAT sounds like a port-forwarding nightmare. It is not, because I already run Tailscale. Again, yes, there are other alternatives, some will say much better ones, its a matter of preference I guess, so, no argument here. The hub binds exclusively to its tailnet address. Agents reach it over WireGuard. Nothing is exposed to the internet, nothing is exposed to the LAN, and there are no inbound firewall rules anywhere in the design.

That leaves one obvious hole: if the house loses power beyond the coverage of UPS, monitoring goes silent, and silence is indistinguishable from health. The fix is a dead man's switch, covered below. The short version is that the hub proves it is alive every minute to an external service, and the alert fires on absence rather than on failure.

None

Building the hub

Pick the quietest box you own. I wrote a small read-only assessment script that scores candidates on UPS presence, uptime, sleep-target status, Tailscale state, free resources and SMART health, then compared the summary lines. The winner was the box with 791 GB free and half the listening services of its rival. Boring hosts make good hubs.

Install is one script:

curl -sL https://get.beszel.dev/hub -o /tmp/install-hub.sh && chmod +x /tmp/install-hub.sh
sudo /tmp/install-hub.sh

Read it before you run it. It creates a dedicated user, installs to /opt/beszel, and writes a systemd unit. The unit binds to 0.0.0.0, which puts your dashboard on the LAN. Fix that with a drop-in rather than editing the vendor unit, so a future reinstall cannot silently undo you:

# /etc/systemd/system/beszel-hub.service.d/override.conf
[Service]
Environment="APP_URL=https://YOUR-HOST.YOUR-TAILNET.ts.net"
Environment="HEARTBEAT_URL=https://hc-ping.com/YOUR-UUID"
ExecStart=
ExecStart=/opt/beszel/beszel serve --http "YOUR_TAILNET_IP:8090"

The bare ExecStart= line is required. Without it systemd appends a second command instead of replacing the first, and the unit fails.

Verify with ss -lntp | grep 8090. You want the tailnet address and nothing on 0.0.0.0. Then create the admin account immediately. That first-run page is open to anyone who can reach the port, so do not wander off between starting the service and claiming it.

None

Two Tailscale settings that will save you at 3 a.m. in six months

First, key expiry. Tailscale node keys expire, 180 days by default and shorter if your tailnet is configured that way. Fifteen agents silently dropping off the tailnet next February is exactly the failure this whole build exists to prevent. Provision servers with tagged auth keys, which disables expiry by default, or disable it per machine in the admin console. Check the Machines page and confirm every fleet node shows expiry disabled.

Second, know your connection direction. Beszel supports two modes: the classic one where the hub dials each agent on port 45876, and a WebSocket mode where agents dial out to the hub using a universal token. Over a tailnet both work. The universal token flow is the one that scales, because agents self-register and you never touch the Add System dialog.

Rolling out agents

Enable the universal token in the hub (Settings, Tokens and Fingerprints, ephemeral persistence), then on each box:

sudo bash -c 'curl -sL https://get.beszel.dev -o /tmp/install-agent.sh && \
  chmod +x /tmp/install-agent.sh && \
  /tmp/install-agent.sh -k "HUB_PUBLIC_KEY" -t "TOKEN" -url "http://HUB_TAILNET_IP:8090" --auto-update false'

Two pitfalls from my own console history, offered so you can skip them. The hub URL flag is -url, not -u. Lowercase -u is uninstall, and the script will cheerfully tell you your URL is an invalid option. And if you paste the command with the placeholder key still in it, the agent installs fine and then crash-loops with failed to parse key, fixed by editing the unit rather than reinstalling.

I also keep auto-update off. Fifteen agents self-updating on their own schedule is not a scenario I want in monitoring infrastructure.

None

What it found on day one

This is the section that justifies the exercise.

Within minutes of the first agents connecting, one box showed a failed systemd unit. systemctl list-units --failed named it: nagios-nrpe-server, failed at every boot since at least July. The config told the rest of the story. NRPE was bound to a tailnet address and configured to accept polls from a specific host, which turned out to be the very box I had just installed the new hub on. Past me had built fleet monitoring over the tailnet with Nagios, and it had been dead for months because NRPE starts before tailscale0 comes up and cannot bind its address.

The server side was worse. Nagios Core, active and running, on the new hub's own host, with an Apache vhost serving its CGI interface. I had installed a second monitoring system next to the dead body of the first without knowing the first existed.

Hey, this may sound really stupid, but, who has not had their own mess ups every now and then?

The other find was disk. One box sat at 77 percent, flagged amber on the dashboard. Twenty minutes of du later: a 9.8 GB Suricata stats.log that nothing consumed (Suricata dumps every counter to it every 8 seconds by default, disable it in the outputs block), a 9.3 GB eve.json with no rotation, and 23 GB of EveBox database for an alert viewer I had stopped using when CrowdSec took over that job. One config line, one purge, one rm, and the box went from 77 percent to 32.

None None

None of this was invisible because it was hidden. It was invisible because nothing was looking.

Alerts that are actually tested

Thresholds live behind the bell icon on each system row. There is an All Systems tab with an overwrite checkbox for bulk application. One quirk worth knowing on 0.18.x: that tab is write-only. It pushes settings to every system and then resets its own toggles, which looks exactly like a failed save. Verify in the per-system tabs, or better, in the alerts collection via the PocketBase admin, before concluding it is broken.

My baseline: status offline for 2 minutes, disk above 80 percent for 10 minutes, everything else off until a week of baselines exists. CPU and load thresholds picked on day one are guesses, and guesses page you at 3 a.m.

Channels are global under Settings, Notifications. Email goes through the PocketBase backend (the /_/ admin, Mail settings), not the Beszel UI. Point it at a transactional provider on a domain you actually control SPF and DKIM for. Sending from a residential IP claiming your own domain is a deliverability grave. For push, Beszel routes through shoutrrr, so Telegram is a bot token and chat ID:

telegram://BOT_TOKEN@telegram?chats=CHAT_ID

The token goes in without the bot prefix, that prefix belongs to Telegram's API path, not your credential. And do not trust the Test URL button alone. There are reported cases of it claiming success while nothing arrives. The only test that counts is the real one: stop an agent, wait past the threshold, watch both channels fire, start it, watch both recoveries arrive. An untested alert channel is indistinguishable from a broken one.

None None

The dead man's switch

The hub watches the fleet. Nothing watches the hub, and the hub sits in a house with residential power. So the final piece inverts the logic: Beszel supports a HEARTBEAT_URL and POSTs a status summary to it on an interval. Point it at a Healthchecks.io check (free tier: 20 checks) and the external service alerts you when pings stop arriving.

Read that direction carefully, because it is the part people trip on: nothing ever connects inbound to the hub. The hub proves liveness outbound. The tailnet-only binding is never violated, and the question "how will they monitor my tailnet IP" answers itself: they do not, and never need to.

The heartbeat payload is better than a bare ping. It carries fleet state, total, up, down, so even the DOWN email tells you the last known status of every box before the lights went out.

Set the period from the actual ping cadence (Beszel pings every minute) with a grace window generous enough to absorb a patch reboot. And test it the same way as everything else: stop the hub, wait out the grace, confirm the alert, start it, confirm the recovery. Mine reported 2 minutes and 43 seconds of downtime, timestamped, from a service that has no idea where my hub is or how to reach it.

None None None

HTTPS without exposing anything

Last polish: the browser warning on a raw IP. Tailscale issues real Let's Encrypt certificates for tailnet hostnames. Enable HTTPS Certificates in the admin DNS page, then on the hub:

sudo tailscale cert YOUR-HOST.YOUR-TAILNET.ts.net
sudo tailscale serve --bg --https=443 http://YOUR_TAILNET_IP:8090

tailscaled terminates TLS on 443 and proxies to the hub. Agents keep using the plain tailnet IP and never notice. Valid certificate, name only resolves on the tailnet, nothing new exposed.

One disclosure to make with open eyes: issuing the cert logs the hostname in Certificate Transparency, permanently and publicly. A random tailnet name leaks little, but if you have ever built a CT timeline against a target, you know exactly what that record looks like from the other side. Decide accordingly.

If you run a filtering DNS client that owns the system resolver, tailnet names will not resolve through it. A bypass rule fails if there is no resolver behind it to fall through to; a redirect rule pointing the hostname at the tailnet IP works deterministically.

What this replaced, and what it costs

The final state: a hub on hardware I own, reachable only over the Tailscale network I already had, with a valid certificate. Agents on every box with zero inbound ports. Two alert channels on infrastructure independent of each other and of the fleet, both tested in both directions. An external dead man's switch, also tested in both directions, that knows nothing about my network. Systemd-level visibility that found a dead monitoring stack, a misconfigured IDS logging itself to death, and 44 GB of reclaimable disk in the first morning.

Total spend: zero. The hub idles under 25 MB of RAM. The agents are invisible.

The old Nagios stack had every capability this one has. What it lacked was anyone watching it, and a monitoring system nobody watches converges on the same value as no monitoring system at all, while costing you the false confidence. Test your alerts by breaking things on purpose. Watch the watcher. And run systemctl list-units --failed on a box you have not looked at in a while. You might meet a dead body of your own.

Tools referenced: Beszel, Tailscale, Healthchecks.io, shoutrrr.

Reach out if you have questions or comments or want to collaborate

Reach out on session at Session ID: 059db238ab37c3d92615c5cc24b694da29c598cc13e27886053722404118e14271

None

OSINTPH: Digital Forensics & Cybersecurity Consulting Open source intelligence, digital forensics, and cybersecurity consulting from Bangkok and Manila, working with clients…

FalconEye - Free OSINT Investigator's Toolkit Free self-hosted OSINT toolkit with 13 modules: crypto wallet tracer, phishing kit fingerprinting, domain intelligence…