Control-plane high availability
9 planes · 5 failure scenarios · design proposedThe JarvisAI/JarvisOS control plane, expressed as the functions that must keep working rather than the processes that provide them. Every survival claim on this page is derived from the topology below, never declared: see src/controlplane.js. Nothing here moves anything — this is a design artefact, like config/placement.json.
Open gaps
Rules this design breaks, and nobody has signed for-
Open gap: The dashboard reads a transitional telemetry source. cp-telemetryTelemetry ingest and dashboard data is measured by parsing provider logs. The dashboard should consume JarvisAI telemetry: a log format is not an interface, and it stops existing the moment the provider is unreachable.
Survival matrix — by design
From the operator's side of each breakDerived from the topology below, never declared. Serving and auto failover need nobody; degraded keeps a declared subset working; manual needs a promotion; break-glass means only the native CLI path is left.
| Control-plane function | Home1 is gone | VIN (the VPS) is gone | Home internet is down | The mesh is partitioned | The NAS is gone |
|---|---|---|---|---|---|
| Identity — session issue and verification | Degraded | Serving | Degraded | Serving | Serving |
| Scheduling — task dispatch and leases | Manual promotion 75s | Serving | Serving | Serving | Serving |
| Durable state — the one database | Manual promotion 21m | Serving | Serving | Serving | Serving |
| Service discovery and internal names | Serving | Serving | Serving | Serving | Serving |
| Configuration distribution | Serving | Serving | Serving | Serving | Serving |
| Ingress and health-based routing | Serving | Serving | Degraded | Serving | Serving |
| JarvisAI Proxy — model request routing | Serving | Serving | Degraded | Serving | Serving |
| Telemetry ingest and dashboard data | Serving | Serving | Serving | Serving | Serving |
| Operator console and runbooks | Serving | Serving | Serving | Serving | Serving |
Survival matrix — today
Counting only instances that are actually runningThe difference between these two tables is how much of the design is still on paper. It is here rather than in a status report because it is the number that decides what tonight's outage looks like.
| Control-plane function | Home1 is gone | VIN (the VPS) is gone | Home internet is down | The mesh is partitioned | The NAS is gone |
|---|---|---|---|---|---|
| Identity — session issue and verification | Degraded | Break-glass | Break-glass | Break-glass | Serving |
| Scheduling — task dispatch and leases | Break-glass | Serving | Serving | Serving | Serving |
| Durable state — the one database | Manual promotion 21m | Serving | Serving | Serving | Serving |
| Service discovery and internal names | Break-glass | Serving | Serving | Serving | Serving |
| Configuration distribution | Break-glass | Break-glass | Break-glass | Break-glass | Break-glass |
| Ingress and health-based routing | Serving | Break-glass | Break-glass | Break-glass | Serving |
| JarvisAI Proxy — model request routing | Break-glass | Break-glass | Break-glass | Break-glass | Break-glass |
| Telemetry ingest and dashboard data | Serving | Break-glass | Break-glass | Break-glass | Serving |
| Operator console and runbooks | Serving | Break-glass | Break-glass | Break-glass | Serving |
Failure scenarios
What is left, and who can still talk to whomHome1 is gone
The home server is down or unreachable: hardware, power, or a failed upgrade. The operator works from VIN over SSH. The NAS is still powered but has no compute of interest.
VIN (the VPS) is gone
The VPS is destroyed, suspended or unreachable. Home1 and the NAS are healthy and the home uplink works. Everything published to the internet goes with it.
Home internet is down
The ISP uplink is dead. Home1 and the NAS are healthy but cut off from every vendor and from VIN. VIN is healthy and still on the internet, but nobody at home can reach it. The operator is at home.
The mesh is partitioned
Both sites are healthy and both have internet, but the WireGuard tunnel between them is down. This is the scenario that punishes automatic promotion: each side can see the world and neither can see the other.
The NAS is gone
Bulk storage and the local backup tier are unavailable. Compute is untouched; recovery capability is not.
The planes
Each function that must keep working, with who leads it, how leadership moves, and what an operator types when nothing leads.
Identity — session issue and verification fleet-fatal
jarvisos-auth · No signalDecide whether a request is allowed. Every published route and every CLI call goes through this answer.
- Leadership
- stateless — nothing to elect
- State
- rebuildable in redis
- Routing
- health-checked via caddy
- Failover
- automatic · budget 30s
- Instances
- vin (active), home1 (active, planned)
- Failure domains
- fd-vps, fd-home-rack
- Telemetry
- jarvisai-telemetry
Why: Target: any instance can verify, because verification is a signature check. Today there is one process on VIN and losing it is losing identity.
Without it: Existing sessions keep verifying. New logins fail while Google or the session store is unreachable, and the break-glass token is the way in.
Failover, in order
- The router marks the instance unhealthy after 3 failed 5s probes.
- Traffic moves to the other verifier; no state moves, because there is none to move.
- A verified login is attempted against the surviving instance.
Verified by: A real login completes end to end, not just a 200 from /healthz.
Costs: None. Verification is stateless; only new-login capability depends on the store.
Break-glass — the native path
jarvisctl auth mint --offline --role operator --ttl 15m
On home1, vin. Gives you: A short-lived operator token accepted by every service, minted without Google, without Redis and without the auth process.
Does not give you: It cannot create users or change policy — only get one operator in.
Needs shell on the node; the offline signing key held in the operator's password manager (name only; never on a node with public ingress) · last proven: Quarterly game day: mint offline, use it, watch it expire.
Configuration: pull-replicated from jarvisos-config (auth policy, route ACLs, signing public keys) onto home1, vin. Each node keeps its last-good copy and serves from it indefinitely. Policy that cannot be refreshed is not policy that stops working.
- Home1 is gone: redis is on the other side of the break.
- VIN (the VPS) is gone: Keeps serving; nobody has to do anything.
- Home internet is down: google-oauth is unreachable from here: Existing sessions keep verifying. New logins fail while Google or the session store is unreachable, and the break-glass token is the way in.
- The mesh is partitioned: Keeps serving; nobody has to do anything.
- The NAS is gone: Keeps serving; nobody has to do anything.
Scheduling — task dispatch and leases fleet-fatal
jarvisos-control-plane · No signalDecide which worker runs which task, and make sure exactly one scheduler is deciding.
- Leadership
- lease held in the store it guards
- Lease
- 30s, renewed every 10s, takeover at 45s, fenced
- State
- durable in postgres-primary
- Routing
- health-checked via internal-dns
- Failover
- automatic · budget 75s
- Instances
- home1 (leader), vin (candidate, planned)
- Failure domains
- fd-home-rack, fd-vps
- Telemetry
- jarvisai-telemetry
Why: The lease is a row in the same database the scheduler writes to, so leadership and writability cannot land on different sides of a partition. Every dispatch carries the monotonic fencing token from that row; the database rejects a write carrying a token older than the current lease, so a scheduler that was paused for a minute cannot dispatch when it wakes.
Without it: Nothing new is dispatched. Running worker tasks continue and report on their own; the queue is durable, so work is delayed rather than lost.
Failover, in order
- The leader stops renewing (crash, host loss, or a partition from the database).
- The lease row expires after 30s; no candidate may touch it before 45s, which covers 5s of clock skew.
- The surviving candidate takes the row in one transaction and receives fencing token N+1.
- It replays unacknowledged dispatches; handlers are idempotent, so at-least-once is safe.
- Any write from the old leader carrying token N is rejected by the database, whether or not that leader knows it lost.
Verified by: A canary task is dispatched and observed to complete under the new token.
Costs: None of the scheduler's own. In-flight tasks are at-least-once and re-run.
Fenced by: The monotonic lease token, enforced in the database rather than by agreement.
Break-glass — the native path
jarvisctl task run --local <task-id> --fence-off
On home1, vin. Gives you: Runs one task in the foreground, from the local definition, with output on the terminal. No queue, no leader, no database.
Does not give you: It does not record the run and it takes no lease — never use it while a scheduler still holds one.
Needs shell on the node; the replicated task definition cache · last proven: Quarterly game day: stop both schedulers, run a real task by hand.
Configuration: pull-replicated from jarvisos-config (task definitions, worker pools, schedules) onto home1, vin. The local cache is what makes the break-glass path below possible: a task definition must be readable with the control plane down.
- Home1 is gone: postgres-primary is on the other side of the break.
- VIN (the VPS) is gone: Keeps serving; nobody has to do anything.
- Home internet is down: Keeps serving; nobody has to do anything.
- The mesh is partitioned: Keeps serving; nobody has to do anything.
- The NAS is gone: Keeps serving; nobody has to do anything.
Durable state — the one database fleet-fatal
postgres-primary · No signalHold everything that cannot be recomputed: tasks, leases, workspaces, audit.
- Leadership
- static primary — promotion is a human act
- State
- durable in postgres-primary
- Routing
- dns via internal-dns
- Failover
- manual · budget 21m
- Instances
- home1 (leader), vin (standby)
- Failure domains
- fd-home-rack, fd-vps
- Telemetry
- node-agent
Why: Deliberately not an election. With two compute domains any automatic promotion is a partition away from two primaries, and one logical database with two writers is unrecoverable here.
Without it: Reads continue from the replica. Writes stop until a promotion decision is taken.
Failover, in order
- Confirm the primary is genuinely gone, not merely unreachable from here.
- Fence it: stop the service and block its port, so it cannot return as a second writer.
- Read the replication lag. That number is the data-loss decision, and it is taken by a human, on purpose.
- Promote the replica.
- Flip db-primary.jarvis.internal to the new primary.
- Restart the scheduler candidates so they re-take the lease against the promoted store.
Verified by: A write succeeds against the promoted primary and the scheduler dispatches a canary task.
Costs: Everything inside the replication lag at the moment of promotion.
Fenced by: By hand, before promotion. There is no third domain to arbitrate this.
Break-glass — the native path
psql -h 127.0.0.1 -U jarvis jarvis # on the node holding the data
On home1, vin. Gives you: Direct SQL against whichever copy is reachable, including read-only inspection of the replica while deciding whether to promote.
Does not give you: It does not promote anything. Promotion is the runbook above, in order.
Needs shell on the node; the database password in /etc/jarvisos/config.env (name only) · last proven: Quarterly restore drill from the offsite copy.
- Home1 is gone: postgres-primary is on the other side of the break.
- VIN (the VPS) is gone: Keeps serving; nobody has to do anything.
- Home internet is down: Keeps serving; nobody has to do anything.
- The mesh is partitioned: Keeps serving; nobody has to do anything.
- The NAS is gone: Keeps serving; nobody has to do anything.
Service discovery and internal names estate-critical
internal-dns · No signalAnswer 'where is this service right now?' — including after a leader has moved.
- Leadership
- stateless — nothing to elect
- State
- rebuildable
- Routing
- dns via internal-dns
- Failover
- automatic · budget 30s
- Instances
- home1 (active), vin (active, planned)
- Failure domains
- fd-home-rack, fd-vps
- Telemetry
- node-agent
Why: Both resolvers answer authoritatively for the same generated zone. No election, because two identical answers are not a conflict.
Without it: Cached names keep resolving. New or moved names do not, which is why the TTL on elected records is short and everything else is long.
Failover, in order
- The resolver stops answering.
- Clients fall through to the second resolver already in their resolv.conf.
- The zone is unchanged, because it was replicated before the fault.
Verified by: A known name resolves to the expected address from a node in each partition.
Costs: None: the zone is configuration, not state.
Break-glass — the native path
jarvisctl discovery dump --hosts | sudo tee /etc/hosts.d/jarvis
On home1, vin, nas. Gives you: Freezes the current name→address map into /etc/hosts on one node, so a machine keeps working with no resolver at all.
Does not give you: It is a snapshot. Anything that moves afterwards must be re-dumped by hand.
Needs shell on the node; the replicated zone source · last proven: Quarterly game day: stop both resolvers, dump, keep working.
Configuration: pull-replicated from jarvisos-config (zone source) onto home1, vin.
- Home1 is gone: Keeps serving; nobody has to do anything.
- VIN (the VPS) is gone: Keeps serving; nobody has to do anything.
- Home internet is down: Keeps serving; nobody has to do anything.
- The mesh is partitioned: Keeps serving; nobody has to do anything.
- The NAS is gone: Keeps serving; nobody has to do anything.
Configuration distribution estate-critical proposed
not in the inventoryGet the same declared configuration onto every node, and keep it there when its source is gone.
- Leadership
- stateless — nothing to elect
- State
- rebuildable
- Routing
- node-local
- Failover
- none
- Instances
- home1 (active, planned), vin (active, planned)
- Failure domains
- fd-home-rack, fd-vps
- Telemetry
- node-agent
Why: Nothing to elect: each node pulls for itself and applies to itself. A push-based distributor would be a control plane needing its own control plane.
Without it: Every node serves its last-good configuration indefinitely and says so. Changes wait; nothing stops.
Break-glass — the native path
jarvisctl config apply --from /var/lib/jarvisos/config/last-good
On home1, vin, nas. Gives you: Re-applies the last known-good configuration from local disk with no network at all.
Needs shell on the node · last proven: Every boot: the same path is what a node uses when it cannot reach the source.
Configuration: pull-replicated from jarvisos-config git repository (mirrored on home1, backed up offsite) onto home1, vin, nas. Pull, verify, apply, and only then swap the last-good symlink. A config that fails verification is never applied, so a bad commit cannot take the estate down at 3am.
- Home1 is gone: Keeps serving; nobody has to do anything.
- VIN (the VPS) is gone: Keeps serving; nobody has to do anything.
- Home internet is down: Keeps serving; nobody has to do anything.
- The mesh is partitioned: Keeps serving; nobody has to do anything.
- The NAS is gone: Keeps serving; nobody has to do anything.
Ingress and health-based routing fleet-fatal
caddy · No signalGet an outside request to a healthy instance of the right service, and away from an unhealthy one.
- Leadership
- stateless — nothing to elect
- State
- rebuildable
- Routing
- health-checked via cloudflare-dns
- Failover
- automatic · budget 4m
- Instances
- vin (active), home1 (active, planned)
- Failure domains
- fd-vps, fd-home-rack
- Telemetry
- jarvisai-telemetry
Why: Two independent proxies, same route table from the config repo. Which one serves is a DNS answer, not an election.
Without it: Published names stop resolving to a working origin. Loopback and LAN access continue, and the SSH tunnel below is the way in.
Failover, in order
- Two consecutive origin health checks fail.
- The edge stops sending that origin traffic; in-flight requests drain for 30s.
- The surviving origin serves the same routes from the same replicated route table.
Verified by: A published route returns 200 through the edge, from a network outside the estate.
Costs: None.
Break-glass — the native path
ssh -N -L 8787:127.0.0.1:8787 operator@<node>
On home1, vin. Gives you: Direct loopback access to every service on a node without the proxy, without DNS and without forward-auth.
Does not give you: One node at a time, one operator at a time. It is not a way to keep serving users.
Needs shell on the node; the operator SSH key · last proven: Used routinely — this is how the recovery console is reached at all.
Configuration: pull-replicated from jarvisos-config (route table, forward-auth policy) onto home1, vin. Both origins must be able to serve every route from local config, or the second origin is decoration.
- Home1 is gone: Keeps serving; nobody has to do anything.
- VIN (the VPS) is gone: Keeps serving; nobody has to do anything.
- Home internet is down: cloudflare-dns is unreachable from here: Published names stop resolving to a working origin. Loopback and LAN access continue, and the SSH tunnel below is the way in.
- The mesh is partitioned: Keeps serving; nobody has to do anything.
- The NAS is gone: Keeps serving; nobody has to do anything.
JarvisAI Proxy — model request routing fleet-fatal proposed
not in the inventoryTake a model request, choose a healthy provider, enforce budget and policy, and return an answer or a typed failure — never a hang.
- Leadership
- stateless — nothing to elect
- State
- ephemeral
- Routing
- health-checked via caddy
- Failover
- automatic · budget 30s
- Instances
- vin (active, planned), home1 (active, planned)
- Failure domains
- fd-vps, fd-home-rack
- Telemetry
- jarvisai-telemetry
Why: Every instance is equivalent and holds nothing. This is the whole reason the proxy can be the front door: it is the one control-plane function with nothing to elect.
Without it: Cached and locally-servable responses continue. Every hosted provider call fails fast with a typed, logged error instead of hanging, so callers degrade rather than pile up.
Failover, in order
- An instance fails three consecutive 5s health checks.
- The router drains it over 20s and sends new requests to the other instance.
- Provider-level circuit breakers are rebuilt locally within one interval; nothing is transferred.
Verified by: A real completion round-trips through the surviving instance.
Costs: In-flight requests only, and those are retried by the caller with the same idempotency key.
Break-glass — the native path
jarvisctl ai call --direct --provider <name> --model <model> -
On home1, vin. Gives you: One request straight to one provider, bypassing the proxy, its policy and its budget entirely.
Does not give you: No budget enforcement, no telemetry, no fallback. It is for proving a provider works, not for doing work.
Needs shell on the node; a provider key named in /etc/jarvisos/config.env (name only; the value never appears in config or telemetry) · last proven: Quarterly game day, and any time a provider is suspected.
Configuration: pull-replicated from jarvisos-config (provider routing table, budgets, model policy) onto home1, vin. Provider keys are read from /etc/jarvisos/config.env on each node (names only in this file). The routing table is config; the credentials are not, and they never travel together.
- Home1 is gone: Keeps serving; nobody has to do anything.
- VIN (the VPS) is gone: Keeps serving; nobody has to do anything.
- Home internet is down: isp-uplink is unreachable from here: Cached and locally-servable responses continue. Every hosted provider call fails fast with a typed, logged error instead of hanging, so callers degrade rather than pile up.
- The mesh is partitioned: Keeps serving; nobody has to do anything.
- The NAS is gone: Keeps serving; nobody has to do anything.
Telemetry ingest and dashboard data degraded-tolerable proposed
not in the inventoryTurn what actually happened into something the dashboard and the incident log can read.
- Leadership
- stateless — nothing to elect
- State
- ephemeral
- Routing
- node-local
- Failover
- none
- Instances
- vin (active), home1 (active, planned)
- Failure domains
- fd-vps, fd-home-rack
- Telemetry
- provider-log-parsing
Why: Every node runs its own consumer. There is no aggregation tier to lose.
Without it: The dashboard shows the last snapshot each console holds, labelled stale. Judgement moves to the node's own logs.
Break-glass — the native path
jarvisctl telemetry tail --local
On home1, vin. Gives you: The raw local telemetry stream on a terminal, with no console, no dashboard and no ingest token.
Needs shell on the node · last proven: Quarterly game day.
- Home1 is gone: Keeps serving; nobody has to do anything.
- VIN (the VPS) is gone: Keeps serving; nobody has to do anything.
- Home internet is down: Keeps serving; nobody has to do anything.
- The mesh is partitioned: Keeps serving; nobody has to do anything.
- The NAS is gone: Keeps serving; nobody has to do anything.
Operator console and runbooks estate-critical proposed
not in the inventoryTell the operator what is broken and what to do about it — especially when everything else is broken.
- Leadership
- stateless — nothing to elect
- State
- none
- Routing
- node-local
- Failover
- none
- Instances
- vin (active), home1 (active, planned), nas (active, planned)
- Failure domains
- fd-vps, fd-home-rack
- Telemetry
- node-agent
Why: One console per node, each independent, each loopback-only. Deliberately not an inventory entity: the console must never be something it monitors.
Without it: The PWA serves its offline shell with the last cached snapshot, labelled stale, from the operator's phone.
Break-glass — the native path
less docs/RECOVERY.md
On home1, vin, nas. Gives you: Every runbook in plain text, readable with no console, no browser and no network.
Needs shell on the node, or the printed copy · last proven: Reviewed whenever a runbook changes; printed copy refreshed quarterly.
- Home1 is gone: Keeps serving; nobody has to do anything.
- VIN (the VPS) is gone: Keeps serving; nobody has to do anything.
- Home internet is down: Keeps serving; nobody has to do anything.
- The mesh is partitioned: Keeps serving; nobody has to do anything.
- The NAS is gone: Keeps serving; nobody has to do anything.
Notes
Structural facts and design not yet adopted-
Note: The design survives only because of instances that are not running yet. cp-identityIdentity — session issue and verification survives "VIN (the VPS) is gone", "Home internet is down", "The mesh is partitioned" only once the planned instance on home1 is actually running. Today that path is break-glass.
-
Note: The design survives only because of instances that are not running yet. cp-discoveryService discovery and internal names survives "Home1 is gone" only once the planned instance on vin is actually running. Today that path is break-glass.
-
Note: The design survives only because of instances that are not running yet. cp-configConfiguration distribution survives "Home1 is gone", "VIN (the VPS) is gone", "Home internet is down", "The mesh is partitioned", "The NAS is gone" only once the planned instances on home1, vin are actually running. Today that path is break-glass.
-
Note: The design survives only because of instances that are not running yet. cp-ingressIngress and health-based routing survives "VIN (the VPS) is gone", "Home internet is down", "The mesh is partitioned" only once the planned instance on home1 is actually running. Today that path is break-glass.
-
Note: The design survives only because of instances that are not running yet. cp-ai-routingJarvisAI Proxy — model request routing survives "Home1 is gone", "VIN (the VPS) is gone", "Home internet is down", "The mesh is partitioned", "The NAS is gone" only once the planned instances on vin, home1 are actually running. Today that path is break-glass.
-
Note: The design survives only because of instances that are not running yet. cp-telemetryTelemetry ingest and dashboard data survives "VIN (the VPS) is gone", "Home internet is down", "The mesh is partitioned" only once the planned instance on home1 is actually running. Today that path is break-glass.
-
Note: The design survives only because of instances that are not running yet. cp-consoleOperator console and runbooks survives "VIN (the VPS) is gone", "Home internet is down", "The mesh is partitioned" only once the planned instances on home1, nas are actually running. Today that path is break-glass.
How to read this page
- A plane "survives" a scenario only if it keeps working from the operator's side of the break with nobody typing anything. A documented manual promotion is a recovery, not a survival.
- This estate has two compute domains. A majority quorum needs three, so no durable state promotes itself. Every lease here is held inside the store it guards, which is the only election that stays correct without a third voter: whoever can write the lease is by definition the side that can write the data.
- The matrix is derived twice. "Today" counts only instances that are actually running; "by design" counts the plan as written. The gap between the two columns is the honest measure of how much of this design is still on paper.
- Routing is deliberately not part of the survival derivation. A caller inside a partition can always reach a local instance directly; whether the *published* route follows the leader is a separate question, checked by the routing rules instead.
Source: config/control-plane.json · machine-readable at
/api/v1/control-plane · narrative in
docs/CONTROL-PLANE-HA.md.