Recovery & safe mode

Safe mode: what it is

Safe mode is a degraded boot state where the engine starts but skips everything that isn’t auth + health-check + the safe-mode introspection API itself. No plugins register, no workers start, migrations don’t auto-apply. The dashboard renders enough surface for you to diagnose and exit; nothing else.

The whole point: if a regression in a plugin or a half-applied migration is breaking boot, you still have a working dashboard to fix it from.

What triggers safe mode

Auto-fallback

A counter at /var/lib/vloud/.boot-failures tracks consecutive crash-on-boot events. Each successful boot (engine reaches app.listen()) resets the counter to 0. Three consecutive failures auto-trip safe mode for the next boot, regardless of any flag.

That covers the worst case where you don’t even know to ask for safe mode: the engine is in a boot loop, systemd’s StartLimit eventually disarms it, and the next operator action lands you in a working dashboard.

Operator-initiated

Set VLOUD_SAFE_MODE=1 in /etc/vloud.env and restart, or use:

sudo vloud-rescue safe-boot

Useful before applying a risky update or when diagnosing suspected plugin issues — go in, do the diagnosis, exit when you’re done.

How to know you’re in safe mode

• Top of every page: yellow Safe mode active banner.
• GET /api/system/safe-mode returns { safe_mode: true, reason: "..." }.
• Most sidebar items disabled (the plugins that own them haven’t been registered).

Exiting safe mode

sudo vloud-rescue exit-safe-mode --yes

Clears the boot-failure counter, removes VLOUD_SAFE_MODE from /etc/vloud.env, and restarts the engine. The next boot is a regular full boot — if the underlying problem wasn’t fixed, the counter will start climbing again.

vloud-rescue CLI

Operator-side CLI for recovery operations. Installed to /opt/vloud/packages/server/dist/rescue/index.js and symlinked at /usr/local/bin/vloud-rescue by the bootstrap installer.

vloud-rescue safe-boot                 # request safe mode for next boot
vloud-rescue exit-safe-mode --yes      # clear safe mode + restart
vloud-rescue mark-migration-done <n>   # bypass a stuck migration
vloud-rescue rollback                  # interactive rollback to a previous slot
vloud-rescue status                    # what state am I in?

Each command is destructive in different ways. --yes is required on the exit / mark-done / rollback commands so you can’t fat-finger them in a hurry.

Migration failures

Migrations are applied automatically on engine boot. Each .sql file runs inside a transaction so partial state is impossible — either the migration applies fully or it rolls back. The applied state is tracked in the migrations table.

If a migration fails:

1. The boot fails with a clear error in journalctl.
2. After three consecutive boot failures, safe-mode auto-engages. You can now reach the dashboard and read the migration error.
3. Manually patch the schema (or roll back to a previous engine version that doesn’t expect that migration).
4. Tell the engine the migration is “done” without actually running it:

   sudo vloud-rescue mark-migration-done 234

5. Exit safe mode.

[!CAUTION] mark-migration-done is a footgun — only use it when you’ve manually applied the migration’s effects and the engine is stuck because the auto-applier failed partway. If you mark a migration done without applying its effects, the next migration that depends on it will fail anyway.

Boot-integrity refusal

After every successful update, the engine writes SHA256SUMS covering every file in dist/. Every subsequent boot re-hashes dist/index.js and compares against that file. If they don’t match, the engine exits with code 21 — interpreted as “something modified the dist tree out from under me”.

Most common cause: editing files in /opt/vloud/packages/server/dist directly. Don’t do that — the boot-integrity check exists precisely to catch surprise modifications that could be malicious (or just well-intentioned but broken).

Override (testing only): VLOUD_SKIP_INTEGRITY=1 in /etc/vloud.env. Don’t ship a production install with that set — it disables the canary’s last-line-of-defense check.

When all else fails: full re-install

The bootstrap installer is idempotent on re-run. Re-running it preserves your /etc/vloud.env (JWT secret, machine_id, install_id, trial state) and overwrites only /opt/vloud.

sudo systemctl stop vloud vloud-job-worker vloud-scheduler
sudo mv /opt/vloud /opt/vloud.broken-$(date +%s)
curl -fsSL https://install.vloud.app | sudo -E bash

Your database, JWT sessions, and license state survive. Tenant data is untouched.

Where logs live

What	Where
Engine stdout/stderr (live)	journalctl -u vloud -f
Engine logs (last 200 lines)	journalctl -u vloud -n 200 --no-pager
Job worker	journalctl -u vloud-job-worker
Scheduler	journalctl -u vloud-scheduler
Install log	/var/log/vloud/install.log
Tamper events	/var/lib/vloud/.tamper-events
Audit events	audit_events table; dashboard → Audit Timeline
System incidents	system_incidents table; dashboard → Security

Backups

The engine backs up tenant data (per-account tarballs) via the Backup engine you configure under Admin → Backups. It does not currently back up its own state (the vloud.db SQLite file, /etc/vloud.env, nginx site configs). For a recoverable engine state, script your own:

#!/usr/bin/env bash
set -euo pipefail
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
DEST=/root/vloud-state-$STAMP
mkdir -p "$DEST"
sudo systemctl stop vloud
sudo cp /opt/vloud/packages/server/vloud.db "$DEST/"
sudo cp /etc/vloud.env "$DEST/"
sudo cp -r /etc/nginx/sites-available/vloud-* "$DEST/" 2>/dev/null || true
sudo systemctl start vloud
sudo tar -czf "$DEST.tgz" -C /root "vloud-state-$STAMP"
sudo rm -rf "$DEST"

Cron this nightly. Restore: stop engine → restore files from the tarball → restart.

Pre-update precaution

Until automatic pre-update engine-state backup ships, run the above script before applying any update that lands new migrations. The blue/green canary protects you from broken binaries; a manual snapshot protects you from a broken migration that you only discover after the schema has already moved.