Reliability & Self-Healing

An unattended system is only as good as its ability to notice and fix its own problems. Archon Merkaba's reliability layer has three levels: observe (monitors), decide (fingerprinting + registry), and act (tiered remediation) — with a human escalation path that is used deliberately, not by default.

#Observe: The Monitor Fleet

18+ independent monitors run on schedules on the cloud plane, each narrow by design:

• Resource monitors (CPU, disk, RAM, swap) with per-process attribution
• Trading-health monitors (daemon liveness, position risk, drawdown warnings)
• Data-freshness monitors (stale status files, stale candle data)
• Cron-health monitors (did scheduled jobs actually run?)
• Security monitors (threat scanning, secrets-in-tree, host artifact drift)
• Gateway watchdog (gateway + messaging + dashboard + lock cleanup + stuck sessions)

All monitor output is normalized onto a single event bus (append-only JSONL) with a shared alert-history stream — one place to answer "what has the system seen lately?"

#Drift Detection

Two dedicated detectors watch for change rather than failure:

• Host artifact drift — daily hashing of critical host-side files (restart scripts, systemd units, firewall rules). Any change alerts by chat-ops, distinguishing sanctioned deployments from surprises.
• Decision drift — an auditor reviews recent agent outputs for compliance with the Evidence Gate protocol (are claims supported by cited code/data?), catching quality regression in the agents themselves, not just the infrastructure.

#Decide: Fingerprints and the Registry

Every alert is reduced to a fingerprint — a canonical identity for a class of problem, extracted with strict source scoping so a trading-health signal can never be confused with a freshness signal. Fingerprints map to a remediation registry: a versioned JSON document pairing each known problem class with its canonical response, cooldowns, and safety flags.

#Act: Tiered Remediation

The registry classifies responses by blast radius:

The safety model is defense-in-depth:

• Global dry-run default — the executor simulates until explicitly enabled, and rollout proceeds one fingerprint at a time, ordered by blast radius, with a 24-hour observation window per activation.
• Shell allowlist — the executor can only run a short list of vetted commands.
• Cooldowns and daily caps — remediation storms are structurally impossible.
• Hard carve-outs — trading daemon state and risk configuration are tier-3 forever; no automation may touch them regardless of confidence.

Outcomes close the loop: every attempt is recorded to an outcomes database and audit mirror, the alerting layer skips issues the executor already handled, and a daily health audit by the self-healing agent reviews the whole pipeline. When a recent audit found the pipeline had been effectively idle (hundreds of attempts, all skipped by dry-run and caps), that finding itself became a remediation project — the monitoring monitors the remediator.

#Restart Discipline

Process management follows a hard protocol learned from incidents:

• One sanctioned restart path (a host-level safe-restart script with hash-aware deployment tracking and notifications) — enforced by a pre-tool-use hook that blocks ad-hoc restart commands.
• Single kill → verify stopped → single start → verify healthy. Never kill -9 first, never multiple start methods racing each other.
• Reboot resilience: autostart units are ordered so the daemon, its data feeds, the VPN, and the vault recover in dependency order — validated by an intentional reboot test rather than assumed.

#Escalation Path

When automation can't or shouldn't act, alerts flow to the founder by chat-ops with severity-appropriate routing. The design goal is a quiet channel: tier 0–2 handles the mechanical noise, drift detection catches the surprising, and a human sees only what genuinely needs judgment. Alert triage itself is agent-work — an on-call agent investigates each escalation and journals its findings, so repeated alerts get faster answers over time.