JarvisOS HomeLab status
No signal

Data resilience

5 stores · 6/11 copies running · register proposed

Where every store this estate cannot afford to lose is written, where its copies are, how each one is fed, and what a schema change does to all of it. Recovery points are derived from the copies, never declared, and they are measured against what is running today — a copy that is planned protects nothing yet.

Meet their recovery point 3/4 counting only copies that run
Survive losing the site 1/4 4/4 once the design is running
Restore proven 1/4 a drill that passed, inside the review window
Tied to the home uplink 2 every off-site copy over one connection

Open gaps

Measured against what is running, not what is designed

Recovery matrix — by design

From the operator's side of each break

Derived from the copies below, never declared. Writable and auto failover need nobody; reads only means a copy here keeps serving while writes wait for a promotion; restore required means the only thing reachable is a backup.

Store Home1 is goneVIN (the VPS) is goneHome internet is downThe mesh is partitionedThe NAS is gone
JarvisOS Postgres Reads only Writable Writable Writable Writable
JarvisOS VIN journal Writable Lost Lost Lost Writable
JarvisOS sessions & queue (Redis) Restore required Writable Writable Writable Writable
Coolify's own Postgres Lost Writable Writable Writable Writable
Supabase (edge database) Writable Writable Restore required Writable Writable

Recovery matrix — today

Counting only copies that are actually running
Store Home1 is goneVIN (the VPS) is goneHome internet is downThe mesh is partitionedThe NAS is gone
JarvisOS Postgres Reads only Writable Writable Writable Writable
JarvisOS VIN journal Writable Lost Lost Lost Writable
JarvisOS sessions & queue (Redis) Restore required Writable Writable Writable Writable
Coolify's own Postgres Lost Writable Writable Writable Writable
Supabase (edge database) Writable Writable Restore required Writable Writable

Transports

What each copy crosses to get where it is going

Whether a path depends on the home uplink is walked through the inventory, not declared: the mesh tunnel does not mention the ISP uplink, it depends on it.

Path Crosses Home uplink What it is
Home LAN nothing that leaves the room no home1 to nas over the rack switch. Crosses nothing that leaves the room, which is exactly why it protects against nothing that happens to the room.
WireGuard mesh wireguard-tunnel yes home1 to vin over the tunnel. The tunnel depends on the ISP uplink, so anything sent this way stops when the home internet does — whether or not the copy says so.
Home uplink to the internet isp-uplink yes home1 or nas pushing straight out to a provider. One connection, one router, one ISP.
VPS network to the internet vin no vin pushing straight out over the provider's network. Shares no equipment with the house, which is the entire reason it exists in this register.
Vendor API from home isp-uplink yes home1 pulling from a provider's API over the home uplink.

The stores

Each store this estate cannot afford to lose, with both of its recovery points, every copy and what it takes to get it back.

JarvisOS Postgres fleet-fatal postgres

postgres-primary · No signal

Task state, the scheduler's leader lease, node identities and every record of what JarvisOS has done. Losing it does not degrade the estate, it forgets it.

Written by
single writer — one primary, promotion is a separate act, primary on home1
Recovery point — anywhere
30s today · promised 15m
Recovery point — losing the site
30s today · promised 1h
Copies
5 today, 6 by design, across fd-vps, fd-home-rack, fd-offsite
Last restore drill
2026-07-12 · pass in 22m
Home uplink
writes do not cross it; every off-site copy crosses it

Copies

Copy Kind Where Domain Path Behind by Use
pg-replica-vin streaming replica postgres-replica fd-vps off-domain mesh via home uplink 30s promotable serves reads
pg-wal-nas WAL archive nas-backups fd-home-rack home-lan 5m restorable
pg-base-nas snapshot nas-backups fd-home-rack home-lan 6h restorable
pg-offsite-home snapshot gwscli-s3-offsite fd-offsite off-domain home-egress via home uplink 1d restorable
pg-offsite-vin planned snapshot gwscli-s3-offsite fd-offsite off-domain vps-egress 6h restorable

Promotion — manual, from pg-replica-vin

  1. Confirm home1 is genuinely gone, not merely slow: the mesh is down and the console shows no heartbeat for 3 intervals.
  2. Fence: from VIN, drop the mesh route to home1 so a returning primary cannot be written to.
  3. Record the replica's last replayed LSN in the incident. That number is the data loss.
  4. pg_ctl promote on postgres-replica; wait for it to leave recovery.
  5. Repoint the JarvisOS control-plane standby unit on VIN at the local Postgres and start it.
  6. Announce read-write on VIN in the incident, with the LSN and the loss window.

Fenced by: home1 is fenced before the replica is touched: stop the JarvisOS control-plane unit, `systemctl stop postgresql`, and take the mesh route down from VIN. Only then does the replica get promoted. The old primary never returns as a primary — it is rebuilt as a replica of the new one.

Verified by: The console's own writes land: create and resolve a scratch incident, then confirm it survives a restart of the control-plane unit.

Costs: Everything committed on home1 and not yet replayed on the replica — 30s of writes at the declared lag, more if the mesh was already degraded when home1 died. The promotion records the LSN so the number is known rather than guessed.

Getting redundancy back: Redundancy is gone the moment the replica is promoted: there is one copy again. Within 24h, take a fresh base backup from the new primary on VIN to gwscli-s3-offsite, and rebuild home1 as a streaming replica of VIN when the site returns. The estate stays at reduced protection until both are done, and the incident stays open until then.

Restore — from pg-wal-nas

  1. Pick the target time. For corruption or a bad migration, that is the minute before it landed; for hardware loss it is 'latest'.
  2. Restore the newest pg-base-nas snapshot into a scratch data directory — never over the live one.
  3. Set restore_command against the WAL archive and recovery_target_time, then start Postgres and let it replay.
  4. Verify (below) before anything is repointed at it.
  5. Stop the control plane, swap the data directories, start Postgres, start the control plane.

Verified by: Row counts on the three tables that matter (tasks, node_identities, incidents) against the last known-good figures in the incident, plus `SELECT pg_is_in_recovery()` returning false and one write that survives a restart.

  • 2026-07-12 — pass, 22m, into scratch instance on home1, port 5433. Point-in-time restore to 24h earlier from pg-base-nas + pg-wal-nas. 22 minutes wall clock, of which 14 was WAL replay. Off-site copies were not exercised.

JarvisOS VIN journal estate-critical sqlite proposed

not in the inventory

Lets VIN keep working when the home uplink does not. Task results, telemetry and evidence are written here first and drained into Postgres when the mesh returns, so a home internet outage costs latency rather than work.

Not watched: Planned. Nothing runs on VIN to declare yet, so the inventory has no entity for it and the health engine cannot see it.

Written by
local-first — writes land locally and drain later, primary on vin
Recovery point — anywhere
none today (15m by design) · promised 15m
Recovery point — losing the site
none today (15m by design) · promised 15m
Copies
1 today, 2 by design, across fd-offsite, fd-vps
Last restore drill
never
Home uplink
writes do not cross it; no off-site copy crosses it

Copies

Copy Kind Where Domain Path Behind by Use
journal-offsite-s3 planned snapshot gwscli-s3-offsite fd-offsite off-domain vps-egress 15m restorable

Local-first writes

Drains into jarvisos-postgres. Queued writes survive a restart. Bounded at 4320m / 2048MB.

When both wrote: There is nothing to merge. Every row carries the node id that wrote it and a monotonic per-node sequence, and only jarvisos-worker on VIN writes them. Postgres is authoritative for ordering across nodes; the drain is an insert that is a no-op if the (node, sequence) pair is already there, so replaying the queue twice is harmless.

When it fills: Stop accepting new work and report the worker as degraded. Dropping the oldest entries would silently lose exactly the results this store exists to keep; refusing new work is visible and reversible.

Drain proven end to end: never

Restore — from journal-offsite-s3

  1. Restore the newest snapshot to /var/lib/jarvisos/journal.db on VIN.
  2. Start the worker with drain-only mode so it pushes the backlog before accepting new work.
  3. Watch the drain to zero, then let it take work again.

Verified by: Queue depth reaches zero and the same number of rows appears in Postgres. Re-running the drain must change nothing, which is the property the conflict rule promises.

No drill on record. Until one exists, the restore above is a hypothesis.

JarvisOS sessions & queue (Redis) estate-critical redis

redis · No signal

Session state for the SSO chokepoint and the short-lived job queue. Losing it logs everyone out and drops in-flight jobs; it does not lose anything that was ever committed.

Written by
single writer — one primary, promotion is a separate act, primary on home1
Recovery point — anywhere
none today
Recovery point — losing the site
none today
Copies
1 today, 1 by design, across fd-home-rack
Last restore drill
never
Home uplink
writes cross it; no off-site copy crosses it

Rebuilt, not restored

Start an empty Redis. Every user re-authenticates against Google on their next request; the job queue refills from the tasks table in jarvisos-postgres, which is authoritative for anything that had actually been accepted.

Costs: Everyone is logged out at once, and any job accepted but not yet started is re-derived from Postgres rather than resumed. Nothing committed is lost.

Coolify's own Postgres fleet-fatal postgres

not in the inventory

Every application, server, destination, environment variable and deployment history Coolify holds. Losing it does not stop what is already running — it stops anything from being deployed, redeployed, restarted or rolled back, including the deploy that would replace whatever just failed.

Not watched: It runs inside Coolify's compose stack on home1 and is not declared in the inventory, so nothing here probes it. Coolify is the only thing that knows whether it is healthy.

Written by
single writer — one primary, promotion is a separate act, primary on home1
Recovery point — anywhere
1d today · promised 1d
Recovery point — losing the site
none today (1d by design) · promised 1d
Copies
2 today, 4 by design, across fd-home-rack, fd-offsite
Last restore drill
never
Home uplink
writes do not cross it; no off-site copy crosses it

Copies

Copy Kind Where Domain Path Behind by Use
coolify-dump-local logical dump home1 fd-home-rack home-lan 1d restorable unencrypted
coolify-dump-nas planned logical dump nas-backups fd-home-rack home-lan 1d restorable
coolify-dump-offsite planned logical dump gwscli-s3-offsite fd-offsite off-domain home-egress via home uplink 1d restorable

Restore — from coolify-dump-nas

  1. Install the same Coolify version the dump came from. A newer one will migrate the schema on boot and the rollback stops existing.
  2. Stop the Coolify stack, drop and recreate its database, `pg_restore` the dump.
  3. Start Coolify and re-enter the credentials it does not hold in the database.
  4. Reconcile against the estate: `scripts/jarvisos-estate.mjs reconcile` must match the register before anything is deployed.

Verified by: Coolify lists the same applications and servers as the disposition register captured, and one throwaway redeploy succeeds.

No drill on record. Until one exists, the restore above is a hypothesis.

Supabase (edge database) estate-critical supabase-postgres

not in the inventory

The user-facing edge database: auth, row-level security and the tables the public apps read through PostgREST. It is where a browser writes; JarvisOS Postgres is where the estate's own record lives.

Not watched: Vendor-held. There is no host of ours to probe and no entity to declare; what we can see is what the API answers.

Written by
vendor-managed — the provider decides, and does not say how, held by Supabase (eu-central-1)
Recovery point — anywhere
1d today · promised 1d
Recovery point — losing the site
1d today · promised 1d
Copies
2 today, 3 by design, across fd-home-rack, fd-offsite, vendor-supabase
Last restore drill
never
Home uplink
writes do not cross it; every off-site copy crosses it

Copies

Copy Kind Where Domain Path Behind by Use
supabase-dump-nas logical dump nas-backups fd-home-rack off-domain vendor-api via home uplink 1d restorable
supabase-dump-offsite planned logical dump gwscli-s3-offsite fd-offsite off-domain home-egress via home uplink 1d restorable

Restore — from supabase-dump-nas

  1. Decide the target first: back into Supabase (a new project, then repoint the app) or into jarvisos-postgres (the exit plan, under time pressure — do not choose this during an incident unless Supabase itself is the incident).
  2. Create the target database and the roles the dump references. `anon` and `authenticated` do not exist outside Supabase and must be created by hand.
  3. pg_restore the dump. Expect the auth schema to fail against a plain Postgres; that is the part the exit plan calls out.
  4. Repoint the app's connection string and rotate the anon and service keys.
  5. Verify (below) before announcing.

Verified by: One row inserted through the app's own path, read back through PostgREST, and one row that RLS should hide staying hidden for an anonymous key.

No drill on record. Until one exists, the restore above is a hypothesis.

Supabase holds this

They decide: Failover, point-in-time recovery, the Postgres version, the maintenance window and how long any of that takes. None of it is visible from here and none of it can be tested from here.

Way off it: The schema is plain Postgres and every table used by the estate is reachable through a logical dump. Moving to jarvisos-postgres means: run both, dual-write from the app for one release, cut reads over, then stop writing to Supabase. What does not move for free is auth — GoTrue's user table and the RLS policies would have to be replaced by jarvisos-auth, which is the reason this is a plan and not a scheduled migration.

Migrations and their rollbacks

2/3 reversible

A migration is a write to every copy. Expand-contract survives a replica applying the old shape and a rollback that lands on the previous release; anything else has to say what it can be undone from and what that costs.

Add an owner column to tasks expand-contract reversible

jarvisos-postgres
  1. Expand: ADD COLUMN owner text NULL. Nothing reads it yet, and a nullable column is invisible to the running release.
  2. Backfill in batches of 1000 with a sleep between, so replication lag stays under the 30s the recovery point assumes.
  3. Deploy the release that writes and reads the column.
  4. Contract, one release later: SET NOT NULL once the backfill is proven complete.

Verified by: SELECT count(*) FROM tasks WHERE owner IS NULL returns zero before the contract step, and the previous release still boots against the expanded schema.

Rollback — inverse-migration

  1. DROP COLUMN owner

Costs at most: nothing · last executed: 2026-08-03

The shape of every migration to this store: each step is safe on its own, the old release keeps running against the new schema, and no step needs the replica to have caught up.

Drop the legacy heartbeats table destructive not reversible

jarvisos-postgres
  1. Confirm the newest pg-base-nas snapshot is under 6h old and that pg-wal-nas is current.
  2. Drain jarvisos-vin-journal to zero and hold the worker. A queue can be holding rows written by a release that predates this change, and they still have to land.
  3. RENAME the table to heartbeats_legacy_20260901 and leave it for one full retention period.
  4. DROP it only after that period, and only if nothing has asked for it.

Verified by: No error in the control-plane log referencing the table for a full retention period after the rename.

Rollback — restore from pg-wal-nas

  1. Point-in-time restore to the minute before the drop, into a scratch instance.
  2. pg_dump the single table out of the scratch instance and load it back into the live database.

Costs at most: 5m of writes · last executed: never

The rename-then-drop is what makes an irreversible change reversible for a retention period. The restore path exists for the case where someone skips the wait.

Journal schema v2 — add a node sequence expand-contract reversible

jarvisos-vin-journal
  1. Expand: ADD COLUMN node_seq integer NULL and a unique index on (node_id, node_seq) WHERE node_seq IS NOT NULL.
  2. New writes carry the sequence; queued rows written by the previous release do not, and drain unchanged.
  3. Contract only once the queue has drained to zero at least once under the new release.

Verified by: A disconnect-and-drain with rows written by both releases in the queue, ending at zero depth with no duplicate rows in Postgres.

Rollback — inverse-migration

  1. DROP INDEX, DROP COLUMN node_seq

Costs at most: nothing · last executed: never

A local-first store makes the expand-contract rule non-negotiable: at any moment the queue may hold rows written by a release that is no longer installed, and they still have to drain.

Accepted risks

Decisions on record, each with a review date

Notes

Structural facts and design not yet adopted

How to read this page

Source: config/datastores.json · machine-readable at /api/v1/data · narrative in docs/DATA-RESILIENCE.md.