Automated remediation
18 policies · 5 automatic · mode observeA machine may run an action on its own only when the inventory declares that action "safe" — read-only, restartable, or reversible by doing nothing. Everything disruptive or destructive is proposed to a human with the exact command, its precondition and its undo. The ceiling is enforced by the validator, not by convention: a policy that binds a disruptive action in auto mode fails the build.
This console evaluates the policies and shows what would happen; it has never executed anything and cannot.
Execution belongs to scripts/jarvisos-remediate.mjs, which owns the audit ledger. Machine readable
at /api/v1/remediation.
Escalations
0 openNothing is waiting for a person. Escalations appear here when a fault has no policy, when the action needs a credential, spending, an irreversible change or hands on hardware, when the node that would run it is unreachable, or when automation tried and the fault did not clear.
Policies
Fault → declared action, with its bounds| Entity | On | Action | Mode | Bounds |
|---|---|---|---|---|
|
JarvisOS Worker (VIN)
worker-restart |
heartbeat-stale, self-reported |
Restart the worker on vin · safe |
auto |
2× · cooldown 5m · verify within
5m
no undo |
|
Offsite backup replica
offsite-sync-retry |
self-reported, heartbeat-stale |
Re-run the offsite sync job on nas · safe |
auto |
2× · cooldown 1h · verify within
45m
no undo |
|
Home1
home1-wake |
heartbeat-stale, no-signal |
Send Wake-on-LAN to the Home1 NIC on nas · safe |
auto |
2× · cooldown 10m · verify within
10m
no undo |
|
Postgres (primary)
postgres-primary-evidence |
heartbeat-stale, no-signal, self-reported |
Collect replication lag and WAL positions from both ends on postgres-replica · safe |
auto |
1× · cooldown 15m · verify within
2m
no undo |
|
Postgres (primary)
postgres-primary-fence |
heartbeat-stale, no-signal |
Fence the old primary so it cannot accept writes on home1 · destructive |
escalate |
irreversible, credentials
undo: unfence-primary |
|
Postgres (primary)
postgres-primary-promotion |
heartbeat-stale, no-signal |
Promote the streaming replica to primary on postgres-replica · destructive |
escalate |
irreversible
no undo |
|
JarvisOS Control Plane
control-plane-restart |
heartbeat-stale, self-reported |
Restart the control plane on its current host on home1 · disruptive |
propose |
irreversible
no undo |
|
JarvisOS Auth (SSO)
auth-restart |
heartbeat-stale, self-reported |
Restart the auth service on vin · disruptive |
propose |
irreversible
no undo |
|
Redis (cache / queue)
redis-restart |
heartbeat-stale, self-reported |
Restart Redis on home1 · disruptive |
propose |
irreversible
no undo |
|
WireGuard (Home1 <-> VIN)
wireguard-restart |
heartbeat-stale, self-reported |
Bounce the WireGuard interface on VIN on vin · disruptive |
propose |
irreversible
no undo |
|
Home1
home1-agent-restart |
self-reported |
Restart the JarvisOS agent on Home1 on home1 · disruptive |
propose |
irreversible
no undo |
|
VIN / VPS
caddy-config-reload |
self-reported |
Validate and reload the Caddy configuration on vin · disruptive |
propose |
irreversible
no undo |
|
NAS
nas-pool-evidence |
capacity/pool_health, self-reported |
Collect pool status and SMART data on nas · safe |
auto |
1× · cooldown 1h · verify within
5m
no undo |
|
NAS
nas-pool-space |
capacity/disk_pool | no action — a person decides | escalate |
irreversible, spend
no undo |
|
Public DNS
cloudflare-dns-escalation |
* | no action — a person decides | escalate |
credentials, vendor
no undo |
|
Google OAuth
google-oauth-escalation |
* | no action — a person decides | escalate |
vendor, credentials
no undo |
|
ISP Uplink
isp-uplink-escalation |
* | no action — a person decides | escalate |
vendor, physical
no undo |
|
GWSCLI / S3
gwscli-s3-escalation |
* | no action — a person decides | escalate |
spend, vendor, credentials
no undo |
Escalation channels
3/3 availableEvery gate is evaluated from state VIN already holds. Losing Home1 removes the actions that would have run there — they are refused as unreachable and escalated, naming what has to come back first — and removes the Home Assistant notification channel, which is declared best-effort and degrades to the local record. It does not remove the engine, the ledger, the policies, or any VIN-local remediation. Home Assistant is an input and a notifier here, never a gate.
-
Up
This console required
/remediation and /incidents on VIN, loopback only -
Up
Audit ledger required
remediation.jsonl in JARVISOS_STATE_DIR on VIN, append-only, 0600 -
Up
Home Assistant notification
HA notify service, reached through the control plane on Home1
Why it is bounded this way
Verification. An action is not a fix until the fault clears and holds. Every automatic attempt carries a verification window and a stability hold; if the entity is not ok by the end of the window the attempt is a failure, is rolled back where an undo exists, and escalates with the attempt record attached. Evidence-gathering steps declare verify.kind "action-succeeded": collecting replication state does not make anything better and must not claim to have.
Budgets. Four independent bounds, because a remediation loop with one bound has none: attempts per incident, a cooldown between them, a per-entity and estate-wide budget inside a rolling hour, and one action in flight at a time. Above all of them sits a circuit breaker — three failures inside an hour suspends every automatic action for thirty minutes and escalates instead. Faults younger than the settle window are left alone entirely, so a single missed heartbeat can never start anything.
Escalation. Escalation is not a notification. It carries the fault, the incident, the last known good state, every attempt and its outcome, what the loss costs in capability, and the exact next action with its precondition and its undo. A fault escalates when no policy covers it, when the action needs a credential, spending, an irreversible change or hands on hardware, when the node that would run it is unreachable, when attempts are exhausted, when an attempt ran and did not verify, or when the breaker is open.
Rollback. Delete config/remediation.json. The console reports the engine as unloaded, the /remediation page says so, and nothing else changes: incidents, recovery tasks, the dependency graph and the action catalogue do not depend on it. To disable automatic execution without losing the analysis, stop the timer that runs scripts/jarvisos-remediate.mjs — the console has never executed anything and cannot start.