System Watchdog

The System Watchdog is an external self-healing monitor that runs independently of the Vloud engine. It checks critical services every 30 minutes and auto-restarts any that have stopped.

Why it’s external

The watchdog runs as a system cron job, not inside the engine. This means it can detect and restart the engine itself if it crashes. An internal worker couldn’t do that.

What it checks

Every 30 minutes, the watchdog checks 7 services:

Service	Check method
Vloud engine	curl health endpoint
Redis	redis-cli ping
Nginx	systemctl is-active
Job Worker	systemctl is-active
Scheduler	systemctl is-active
Postfix	systemctl is-active
Dovecot	systemctl is-active

Auto-healing

When a service is down, the watchdog tries to restart it:

1. systemd first (systemctl start <service>)
2. Direct start fallback (e.g., redis-server --daemonize yes)
3. Engine nohup fallback if systemd isn’t managing it

Logging

Every run is logged as a JSON line to /var/log/vloud/watchdog.log:

{
  "ts": "2026-05-23T04:57:40Z",
  "overall": "ok",
  "down_count": 0,
  "remediated_count": 0,
  "checks": [...]
}

The log auto-rotates at 5 MB.

Managing the watchdog

From the UI: Go to /system-watchdog in the admin panel.

• Run Now button triggers an immediate check
• Enable/Disable Cron toggles the 30-minute schedule
• History shows all past runs with expandable detail

From the CLI:

# Install the cron
bash /opt/vloud/scripts/vloud-watchdog.sh --install-cron

# Remove the cron
bash /opt/vloud/scripts/vloud-watchdog.sh --remove-cron

# Run once manually
bash /opt/vloud/scripts/vloud-watchdog.sh

Dashboard integration

The dashboard shows a System Watchdog tile in the control center strip:

• Healthy (green): all services running
• Needs Attention (amber): services were down but auto-fixed
• Critical (red): services still down after restart attempts