Five layers of uptime: linger, self-heal, spooling, watching, mesh · content/always_on.md
Always On — how qalnet stays up (and what to do when it blinks)
Date: 2026-09-11 · The question: "How can things be rigged so the qalnet is always on or on enough between systems?"
The answer is five layers, each catching what the previous misses. All are live.
Layer 1 — Boot without a human (✓ live)
- systemd user units (
qal-target.target): chain, loopd, relay, market, pub, bazaar, redeem, claim, id, pay, bridge —Restart=on-failure, 5s backoff. - Linger enabled (
loginctl enable-linger): units run at power-on without anyone logging in. tailscaledenabled at boot — the mesh IS the WAN.
Layer 2 — Self-healing (✓ live as of today)
qal-chain-heal.timer(every 5 min): probes the chain 3×; down → restart the unit; still down >90s → bounded regenesis (max 1/hour,scripts/chain_heal.sh) + Signal alert on every action.- Shadow-killer (
services_up.shv2): any stale process squatting a qal port that isn't the systemd owner gets killed — the red-team ops lesson, automated.
Layer 3 — Nothing is lost while it blinks (✓ live, proven today)
- Anchor spooling (qalpipe): chain down → anchors queue in
data/anchor_spool.jsonl; every new anchor attempt flushes the queue first. Proven end-to-end: stop chain → spool (1 queued) → chain back → next anchor flushes → spool 0, verify PASS. - Ledgers are append-only JSONL with atomic writes; identities derive from seeds — a dead process never loses money or history.
Layer 4 — Watched from outside (✓ live)
- Fleet monitor (
qal-monitor): probes all 14 endpoints every 60s; Signal alerts on state change (UP→DOWN and DOWN→UP, quiet re-alerts). The heartbeat filefleet_monitor_state.jsonis the machine-readable truth for any dashboard.
Layer 5 — Between systems (mesh redundancy)
- Relay #2 on cachyos-x8664 (systemd, Tailscale) — transport survives superlocal dying; clients
--relayspoll all, dedupe. - bb-mini installer (
deploy/bbmini/install.sh) — one command when that box powers on: relay #3 + provider + monitor. - Known weak spot (honest): the chain still finalises on one validator — mail and money-ledgers survive outages, but anchoring waits for the heal timer. The 4-validator Fleet milestone (installers ready for both minis) is the structural fix.
The degradation ladder (what "on enough" means)
| Failure | What still works | What waits | Recovery |
|---|---|---|---|
| Chain down (<5 min) | everything except new anchors | anchors (spooled) | unit restart by healer |
| Chain down (>90s) | everything; spool grows | anchors | bounded regenesis + alert |
| superlocal down | relay #2 mail; all docs/site | money rail, chain | reboot → linger brings all back |
| Tailscale DERP flap | local ops | cross-machine | warm-up + retry (tested) |
| GLM upstream down | assistant falls back to local KB | AI answers | automatic |
Runbook: scripts/services_up.sh (imposter-free start) · bash scripts/gen_devnet.sh (full regenesis) · journalctl --user -u qal-chain-heal (heal history).