September 26, 2026

Designing Edge Devices for Stable 24/7 Unattended Operation

Introduction

By Secure Edge Ops

12 min read

Introduction

I work on improving the security and reliability of edge/cloud environments used in medical systems, especially in environments where devices are expected to operate unattended for long periods.

Some of the main topics I focus on are:

  • recovery after power failures
  • retaining data during offline periods
  • detecting communication failures
  • applying security patches safely
  • maintaining long-term system stability

In the previous article, I discussed an important question:

How much of Azure IoT Edge and the container runtime should be fixed to a specific version, and how much should be allowed to update?

The policy I adopted was roughly:

Security updates
↓
Apply them
Patches within the same release generation
↓
Apply them under controlled conditions
Unvalidated major or generation changes
↓
Do not apply automatically

The goal was to balance two requirements that often conflict:

Security and operational stability

But version management alone is not enough to operate an unattended edge device 24 hours a day, 365 days a year.

In a real edge PC, many other things can go wrong:

System clock drifts
DNS configuration changes unexpectedly
Network routes are not restored correctly
Firewall or NAT state becomes inconsistent
A NIC does not recover properly
An update requires a reboot
Container runtime state changes
Only part of the system recovers after a power failure

So the design principle for this article is different.

Instead of trying to guarantee that failures never happen:

Design the system so that it can return to a known-good state when something goes wrong.

One of the main tools I used for this was:

systemd timer

1. Booting Successfully Is Not Enough

During development, we often verify something like this:

Power ON
↓
Ubuntu boots
↓
Docker starts
↓
Azure IoT Edge starts
↓
Application starts
↓
Cloud connection established

At that point, the system may appear complete.

For a device that runs unattended 24/7, however, that initial success is not the most important thing.

The real question is whether the system can return to the same working state:

One day later
One week later
One month later
After an update
After a network outage
After a power failure

That changed how I thought about the system.

Instead of focusing only on:

Creating the correct initial state

I also focused on:

Recreating the correct state periodically.

In long-term unattended operation, it is almost impossible to prevent every possible abnormal condition.

For example:

Temporary network failures
Changes on the router or DHCP side
Clock drift
Package updates
Stopped processes
Unexpected reboots
Power failures

Some of these events are outside the control of the edge PC itself.

So the design changed from:

Prevent every abnormal condition

to:

If an abnormal condition occurs,
return to the expected state during the next recovery process

Conceptually:

Normal State
             ↑
             │
          Verify
             │
        Repair if needed
             │
             │
     Temporary Failure

In other words:

Define the expected state, detect deviation from it, and restore it when necessary.

I applied this idea to each maintenance process.

2. Do Not Put Everything Into One Huge Maintenance Script

One approach I wanted to avoid was something like:

Every night at 01:00
Check everything
Update everything
Reconfigure the network
Restart everything
Reboot

Combining everything into one script may reduce the number of files that need to be managed.

But when something fails, troubleshooting becomes much more difficult.

For example:

Where exactly did the process fail?
Does one failed step stop everything after it?
Did the network process conflict with an update?
Did a reboot make it difficult to inspect the logs?

Instead, I separated maintenance processes according to their responsibilities.

For example:

NTP
NAT
DNS
IoT Edge updates
OS reboot checks
IoT Edge reboot checks
NIC recovery
Route repair
Firewall verification
Diagnostics

Each process has its own service and timer.

This makes the system easier to understand and easier to troubleshoot.

3. Separating Responsibilities with systemd Timers

For this system, I use standard Linux mechanisms:

systemd service
+
systemd timer

The basic relationship is:

.timer
↓
Starts at a defined time or condition
.service
↓
Performs the actual operation

For example:

mdlinkage-ntp-sync.timer
↓
mdlinkage-ntp-sync.service
↓
NTP synchronization

With this structure, I can check whether the timer itself is running:

systemctl status xxx.timer

And I can inspect logs for only the corresponding operation:

journalctl -u xxx.service

The important benefit is that it separates:

When something should run

from:

What the process actually does

That separation becomes extremely useful when debugging an unattended device.

4. The Maintenance Schedule

For this edge PC, the main maintenance processes are distributed across several hours.

00:15
Scheduled reboot
00:45
NTP synchronization
NAT verification
01:00
DNS verification and recovery
01:30
IoT Edge / Moby patch check
02:30
Check whether OS updates require a reboot
03:30
Check the reboot flag created by
IoT Edge / Moby updates
04:30
Verify and recover NIC state

In addition, immediately after boot, several network-related states are checked again:

Route
NAT
Firewall
NIC
Network configuration

The important point is:

These processes do not all run at the same time.

5. Why Maintenance Tasks Are Staggered

Imagine that all of the following occur simultaneously:

NTP synchronization
Package updates
DNS configuration
Container restart
Network reconfiguration
System reboot

Individually, every operation may be valid.

But when they happen at the same time, they can interfere with one another.

For example:

The network temporarily disconnects
The machine reboots during an update
The NIC restarts while DNS is being checked
The container runtime changes while container health is being verified

If something fails, troubleshooting also becomes difficult.

You are left asking:

Was it NTP?
DNS?
An update?
The network?
The reboot?

So the schedule looks more like this:

Process A
↓
Wait
↓
Process B
↓
Wait
↓
Process C

The key assumption is:

Maintenance processes themselves can become a source of failure.

This is why I deliberately separate them in time.

6. 00:15 — Scheduled Reboot

The system also performs a scheduled reboot every day at:

00:15

using a timer such as:

scheduled-reboot.timer

On an edge PC running continuously, temporary states can gradually accumulate.

For example:

Long-running application state
Runtime state changes
Temporary network stack issues
Unexpected process states

A scheduled maintenance window provides an opportunity to periodically return the entire system to a clean boot state.

But simply rebooting the device is not enough.

7. The System Must Recover Automatically After Reboot

A scheduled reboot becomes dangerous if the device does not automatically recover afterward.

For example:

Network does not return
Route is missing
Firewall configuration is not restored
NAT rules are missing
NIC remains down
IoT Edge does not start

In that case, the scheduled reboot itself becomes the cause of downtime.

That is why several recovery processes run after boot.

For example, a timer such as:

fix-route.timer

checks the routing table after boot.

If the expected route already exists:

OK

and the process exits.

If the route is missing, the script restores the expected state.

The logic is:

Check current state
↓
If healthy, do nothing
↓
If unhealthy, repair it

rather than:

Rewrite the configuration every time

This distinction is important.

8. Firewall and NAT Are Runtime States Too

The same idea applies to firewall and NAT configuration.

Configuring:

nftables

during the initial setup is not enough.

After power failures, network restarts, or other system events, I want to verify that the runtime state is still what I expect.

For this reason, I use recovery processes conceptually similar to:

ensure-nft-filter
ensure-nft-nat

The logic is:

Check the current rules
↓
Healthy
↓
Do nothing

or:

Check the current rules
↓
Missing or abnormal
↓
Restore the expected state

The principle here is:

Do not only verify configuration files. Verify what is actually active at runtime.

9. 00:45 — Synchronizing Time

Time synchronization is also important for an edge gateway.

Significant clock drift can affect:

TLS certificates
Log timestamps
Cloud communication
Troubleshooting
Comparing logs across multiple systems

During troubleshooting, I may need to compare:

Edge PC logs
Network logs
Cloud-side logs

If their clocks are inconsistent, reconstructing the sequence of events becomes much more difficult.

In this configuration, NTP synchronization is performed as a controlled:

one-shot

operation at defined times rather than relying only on a continuously running daemon.

The synchronization is also triggered after boot:

Boot
↓
Network Ready
↓
NTP Sync

This means that after a power failure, the device has an opportunity to correct its clock as part of the recovery sequence.

10. 01:00 — Periodically Checking DNS

DNS is another critical dependency for Azure IoT Edge.

The network itself may appear healthy:

IP connectivity
↓
OK
DNS resolution
↓
NG

But if DNS resolution fails, the device may still be unable to communicate with Azure IoT Hub.

From the user's perspective, the symptom may simply look like:

The network is connected,
but data is not reaching the cloud.

For this reason, DNS is not treated as something that is configured once and forgotten.

Its expected state is checked periodically.

It also helps to think about communication in layers:

Network
↓
Route
↓
DNS
↓
Cloud

Separating the layers makes troubleshooting easier.

11. 01:30 — A Dedicated Update Path for IoT Edge

At:

01:30

the device checks for IoT Edge and container runtime patches.

This is the update process discussed in the previous article.

Conceptually:

Check installed version
↓
Check candidate version
↓
Confirm that it remains within
the allowed release generation
↓
Dry-run
↓
Check dependencies
↓
Apply patch if conditions are satisfied
↓
Health check

This process does not run at the same time as OS updates or network maintenance.

I also use:

RandomizedDelaySec

so that if many edge PCs are deployed, they do not all begin maintenance at exactly the same moment.

For example:

Edge PC A
Edge PC B
Edge PC C
Edge PC D

As deployments grow, this becomes increasingly important.

A maintenance architecture designed for one device should not create a synchronization problem when the fleet grows.

12. 02:30 — Reboot Only When the OS Requires It

After certain Ubuntu updates, such as kernel or critical package updates, the system may create:

/var/run/reboot-required

However, I do not reboot immediately after the update.

The update process finishes first.

Then, at a separate time:

02:30

another process checks whether a reboot is actually required.

No reboot-required file
↓
Do nothing
reboot-required exists
↓
Reboot

In other words:

The update process and the reboot decision are separated.

This reduces unnecessary coupling between maintenance tasks.

13. 03:30 — Reboots Required by IoT Edge Updates Are Also Separated

The same principle is applied to IoT Edge and Moby updates.

If a patch requires a reboot, the update process creates a flag such as:

IoT Edge / Moby Update
↓
pending-iotedge-reboot

Then, at:

03:30

a separate timer checks the flag.

No flag
↓
Do nothing
Flag exists
↓
Check conditions
↓
Reboot

The logic also considers the OS-level:

reboot-required

state so that the system does not reboot multiple times within a short period unnecessarily.

The architecture therefore becomes:

Update
↓
Flag
↓
Maintenance Window
↓
Reboot

instead of:

Update
↓
Immediate Reboot

14. 04:30 — Verify the NIC Last

Near the end of the maintenance window, the system also checks the:

NIC

A timer such as:

nic-wakeup.timer

runs both:

After boot

and:

Daily at 04:30

For an edge gateway, the application may be healthy, but if the network interface is not functioning correctly, the system cannot perform its role.

I therefore treat these as separate layers:

Application
Container
Operating System
Network Interface

For example:

Application healthy
but
NIC unhealthy

still means the gateway as a whole is unhealthy.

15. Use Both Boot-Time Recovery and Scheduled Maintenance

Some timers run at a specific daily time.

Others also use:

OnBootSec

so that they can run shortly after the device starts.

Examples include:

NTP
NAT
Route
NIC
Firewall

This was important for handling unexpected power failures.

Imagine a power failure occurs at:

00:30

The device should not have to wait until the next day's scheduled maintenance to recover.

Instead:

Power Failure
↓
Power Restored
↓
Ubuntu Boot
↓
Network Ready
↓
Verify Route
↓
Verify Firewall / NAT
↓
Verify NIC
↓
Synchronize Time
↓
Application Recovery

In other words:

Boot itself becomes a recovery trigger.

16. Use Persistent=true Selectively

systemd timers also support:

Persistent=true

For example, if a task was scheduled for 01:00 but the PC was powered off at that time, systemd can run the missed task after the next boot.

That is useful for some processes.

But it does not make sense for every timer.

Consider a scheduled reboot.

If the device was powered off during yesterday's planned reboot window, there is little value in doing this immediately after the next boot:

Yesterday's reboot was missed
↓
Reboot immediately after startup

So I separate timers into:

Tasks whose missed execution
should be recovered after boot
and
Tasks whose missed execution
should simply be skipped

Again, the important question is not:

What systemd option should I enable?

The real question is:

Why does this timer exist, and what behavior is appropriate for that purpose?

17. Record Results, Not Just Execution

On an unattended device, nobody is watching the screen to confirm:

Success

So the result of every maintenance operation needs to be observable later.

With systemd, I can inspect logs per service:

journalctl -u mdlinkage-ntp-sync.service

And I can review timers with:

systemctl list-timers --all

This shows information such as:

When the timer last ran
When it will run next
Whether it is active

For example:

systemctl list-timers --all | grep -E \
'mdlinkage|mdl-|reboot-if'

can be used to review maintenance-related timers together.

18. Verify the Timers Themselves

Creating a timer file does not prove that the maintenance mechanism works.

I want to verify more than:

The timer file exists

I also check:

Is it enabled?
Is it active?
Is the scheduled time correct?
Did the corresponding service actually run?
Was the exit code successful?

For example, during acceptance checks I verify that entries such as:

00:15 Scheduled Reboot
00:45 NTP
00:45 NAT
01:00 DNS
01:30 IoT Edge Patch
02:30 OS Reboot Check
03:30 IoT Edge Reboot Check
04:30 NIC Wakeup

are actually registered at the expected times.

After boot, I also verify evidence that processes such as:

Route
NTP
NAT
Firewall
NIC

actually executed during the current boot session.

The principle is:

Do not only check whether the configuration exists. Check evidence that it actually ran.

19. Maintenance Operations Should Be Safe to Run Repeatedly

Another principle I applied to recovery processes is:

Running the process against
an already healthy system
should not break it

For example, if a route is already correct:

OK: route already correct

and the process exits.

Similarly, if the expected firewall table and rules are already healthy:

already healthy

and nothing is changed.

The timer therefore behaves like:

Timer runs
↓
Check current state
↓
Healthy
↓
Exit

or:

Timer runs
↓
Check current state
↓
Unhealthy
↓
Repair

rather than:

Timer runs
↓
Forcefully rewrite everything

This reduces the chance that the maintenance mechanism itself becomes a new source of failure.

But Scheduled Maintenance Cannot Detect Everything

With these mechanisms, the edge PC can verify many aspects of its own health.

For example:

Time
DNS
Route
Firewall
NAT
NIC
IoT Edge
Container Runtime
Reboot Requirements

But there is one important limitation.

What happens if the edge PC completely freezes?

Edge PC
↓
Freeze
↓
CPU / Kernel / Network stop

At that point, the device's own:

Timers
Health checks
Log collection
Recovery scripts

also stop.

This is why internal recovery alone is not sufficient.

20. Self-Recovery and External Monitoring Have Different Roles

I eventually divided the reliability architecture into two responsibilities.

The edge PC itself is responsible for:

Edge PC
↓
Maintain the expected state
↓
Recover automatically when possible

An external monitoring device is responsible for:

External Monitor
↓
Check from outside
whether the Edge PC is actually alive

The external monitor can observe the device through multiple paths, such as:

Ping
Heartbeat
Telemetry
rsyslog
netconsole

It is especially useful to separate an application-level heartbeat from lower-level signals such as kernel logging.

This makes it easier to distinguish between situations such as:

The application stopped,
but the kernel is still alive
Only the network stopped
The entire operating system froze
A kernel crash occurred

I will cover this external monitoring architecture in more detail in the next article.

21. Monitoring Should Not Destabilize the System Being Monitored

More monitoring is not always better.

Imagine repeatedly running commands such as:

docker inspect
smartctl
systemctl
powertop
many short-lived processes

every few seconds.

Monitoring itself may begin to affect:

CPU load
Power states
Disk access
Runtime behavior

This is particularly important when investigating freezes.

The monitoring process itself should not significantly change the conditions you are trying to observe.

For that reason, I divide telemetry collection according to the cost of collecting each type of information.

For example:

Every 5 seconds
↓
Lightweight heartbeat / telemetry
Every 60 seconds
↓
C-state / PSI / vmstat /
IRQ / Docker state
Every 10 minutes
↓
Network / filesystem /
NTP / SMART
Every hour
↓
More detailed SMART information

The principle is:

Collect the information you need, but avoid changing the system too much just by monitoring it.

22. Putting the Architecture Together

For this 24/7 unattended edge device, I think about reliability in three layers.

1. Prevention
Version management
Security updates
Scheduled maintenance
Staggered maintenance windows

2. Self-Recovery
Scheduled reboot
Route verification after boot
Firewall / NAT verification
DNS verification
NTP synchronization
NIC recovery
Conditional reboot

3. External Detection
Ping
Heartbeat
Telemetry
rsyslog
netconsole

The overall strategy is:

Make failures less likely
↓
If something fails, recover
↓
If recovery fails, detect it externally

Conclusion: 24/7 Reliability Does Not Come From One Powerful Mechanism

There is no single special technology making this system reliable.

Most of the mechanisms are standard Linux tools:

systemd
systemd timer
journal
nftables
Shell scripts
Health checks

The important part is how they are combined.

For every possible failure, I need to think about:

What type of failure is this?
At which layer should it be detected?
When should the system check for it?
How far should automatic recovery go?
At what point should responsibility move
to external monitoring?

That system-level design is much more important than any individual command or timer.

For unattended 24/7 operation:

Reliability is not only about keeping the system from stopping. It is also about being able to return to a known-good state — and detecting when it cannot.

Next: Detecting an Edge PC Freeze From the Outside

At this point, the edge PC itself has:

Scheduled maintenance
State verification
Self-recovery
Controlled reboot mechanisms

But if the entire edge PC freezes, all monitoring running inside the device stops as well.

So in the next article, I will introduce a different approach:

Using a separate Linux device to monitor the edge PC from the outside.

In my actual setup, I collect signals through multiple paths and intervals:

1-second Ping
5-second Heartbeat
5-second Telemetry
60-second detailed Telemetry
10-minute Health information
1-hour detailed SMART information
rsyslog
netconsole

The goal is not simply to collect as much information as possible.

Another important requirement is:

Keep the monitoring overhead low enough that the monitoring process itself does not interfere with freeze investigation.

The next article will focus on one question:

When the monitored device completely stops, what information needs to remain outside the device to help us understand what happened?