Injection

wan-disconnect · reversible · needs a person

Unplug the WAN cable at the router. No configuration is edited: a drill that changes a firewall rule to fake an outage has changed the estate it was measuring.

Undo: Plug it back in and confirm the tunnel re-establishes without a manual restart.

The router keeps serving the LAN throughout. Anyone in the house should still be able to reach the rack; if they cannot, that is the finding.

Steps

1. Record the baseline ▸ proposed
   Capture which capabilities answer, and the age of the newest off-site object, before anything is unplugged.
   Verify: A baseline exists and names the off-site object it measured from.
   Undo: none — reading is not a change
2. Disconnect the WAN ☝ a person
   Verify: The public hostnames stop resolving to a working origin from a phone on mobile data.
   Undo: Reconnect the WAN cable.
   Start the stopwatch here, not when somebody notices.
3. Confirm the rack is alive from inside the house ☝ a person
   Verify: The control plane answers on the LAN address while VIN reports it down.
   Undo: none — reading is not a change
   This is the step that distinguishes an isolated estate from a dead one. Skipping it is how the drill becomes theatre.
4. Check what the estate said about itself ▸ proposed
   Verify: An incident was opened, and its stated cause is the uplink rather than nine separate hosts.
   Undo: none — reading is not a change
5. Reconnect and watch it come back by itself ☝ a person
   Verify: Every capability returns without a manual restart, and the off-site sync catches up on its own schedule.
   Undo: none — this is the rollback

Stop the drill when

• The tunnel does not re-establish within 10 minutes of the cable going back in
  Stop the drill and treat it as an incident. Restarting WireGuard by hand is recovery, not drill data.
• A real fault is opened against anything outside the home rack during the window
  Reconnect immediately. Two faults at once means the measurement is worthless and the risk is not.

What happened

Never run, in either mode. Everything above it on this card is a prediction the dependency graph made and nobody has checked.

Injection

graceful-shutdown · reversible · needs a person

Shut both hosts down cleanly, NAS last and only with the pool healthy. Never during a resilver.

Undo: Power on the NAS first, wait for the pool to import cleanly, then home1.

The pool-health precondition is not advice. A NAS shut down mid-resilver is how a drill turns into the data loss it was rehearsing.

Steps

1. Record the off-site recovery point ▸ proposed
   The newest object in the off-site bucket, by name and timestamp. This is the state the estate would restart from.
   Verify: A specific object is named, not a bucket.
   Undo: none — reading is not a change
2. Refuse to start if the pool is not healthy ▸ proposed
   Verify: The pool is online, not degraded, and no resilver is in progress.
   Undo: none — this is a gate
3. Shut the room down ☝ a person
   Verify: Both heartbeats stop, and the console names the site rather than nine services.
   Undo: Power both hosts back on in NAS-first order.
4. Restore the crown jewels from the off-site copy, onto VIN ☝ a person
   Not from the NAS. The NAS is in the room this drill has just taken away, and a restore that reads from it has rehearsed nothing.
   Verify: A scratch database on VIN is built from the off-site copy and answers a known query.
   Undo: Delete the scratch restore. It never served traffic.
   This is the step the estate has never taken. Every restore drill on record so far read from the copy in the same room as the thing it protects.
5. Bring the room back ☝ a person restore-nas
   Verify: The pool imports clean, backups resume landing, and the off-site sync catches up.
   Undo: none — this is the rollback

Stop the drill when

• The pool is degraded or resilvering at the pool-check step checked automatically
  Do not start. Reschedule after the resilver completes.
• The off-site restore has not produced a readable dataset within 2 hours
  Stop, bring the room back, and file the failure. A restore path that does not work is the finding; grinding at it for six hours is not.

What happened

Never run, in either mode. Everything above it on this card is a prediction the dependency graph made and nobody has checked.

Injection

graceful-shutdown · reversible · needs a person

Shut home1 down cleanly. Not a power pull: a dirty stop adds a filesystem check to the recovery time and the drill would be measuring fsck.

Undo: Power on at the rack, or over IPMI where that exists, and bring services up in the order on the recovery page.

Steps

1. Read the replica's lag before anything else ▸ proposed
   This number decides whether the drill is a rehearsal or a data-loss event. It is read first and it is read again immediately before promotion.
   Verify: Lag is under the warn threshold and recorded.
   Undo: none — reading is not a change
2. Shut home1 down ☝ a person
   Verify: The heartbeat stops and an incident opens naming home1 as the root cause rather than six separate services.
   Undo: Power home1 back on.
3. Fence the old primary ☝ a person
   Before anything is promoted, make it impossible for home1 to accept a write when it comes back. Split-brain here is unrecoverable.
   Verify: The old primary cannot be written to from the control plane's credentials.
   Undo: Unfence once the drill ends and the rebuild plan is agreed.
4. Promote the replica on VIN ☝ a person promote-postgres-replica
   Verify: A write succeeds against the promoted primary and the control plane reconnects to it.
   Undo: There is none. Promotion is one-way until a fresh base backup is taken — which is the reason this step will never be automatic here.
   Destructive in the action catalogue. Automation may never run it, and this register cannot change that.
5. Bring up the standby control plane ▸ proposed failover-control-plane-to-vin
   Verify: The control plane accepts a task and places it on a worker.
   Undo: Stop the standby. It was not serving before the step.
6. Bring home1 back and rebuild, do not reattach ☝ a person restore-home1
   Verify: home1 is up, and the old database volume is rebuilt from a fresh base backup rather than reattached.
   Undo: none — this is the rollback
   Reattaching the old primary as-is is the single most expensive mistake available in this scenario.
7. Restart the internal resolver ⚙ automatic restart-internal-dns
   Verify: Internal names resolve from both sites.
   Undo: Stop the resolver. It was not answering before the step, so stopping it returns the estate to the state the step began in.

Stop the drill when

• Replication lag is above the critical threshold when promotion is due checked automatically
  Do not promote. Restore home1 instead and record the drill as aborted — promoting a lagging replica loses every transaction in the gap, and doing it on purpose during a drill is worse than doing it in an incident.
• home1 does not come back within 60 minutes
  The drill is over and an incident has started. Stop measuring and start recovering.

What happened

Never run, in either mode. Everything above it on this card is a prediction the dependency graph made and nobody has checked.

Injection

config-override · reversible · no person required

Point the auth service's OIDC discovery URL at an address that does not answer. No firewall rule, no DNS record and no key is touched — the estate's own policy forbids all three, and a drill that breaks that policy has changed the thing it was measuring.

Undo: Restore the discovery URL and confirm a fresh sign-in completes.

The override touches jarvisos-auth's configuration, which is inside the derived blast radius of this scenario — it is a thing the outage would break anyway, not collateral.

Steps

1. Record the session lifetime the estate actually issues ▸ proposed
   Not the configured value — the value on a token issued today. Those two disagree more often than anyone expects.
   Verify: An issued session's expiry is recorded.
   Undo: none — reading is not a change
2. Make the provider unreachable ☝ a person
   Verify: A fresh sign-in fails at the discovery step, not at the credential step.
   Undo: Restore the discovery URL.
3. Measure the grace period ▸ proposed
   How long the estate keeps working for people already inside it. That number is the degraded mode the identity register claims, checked.
   Verify: An existing session keeps working, and the moment it stops is recorded.
   Undo: none — reading is not a change
4. Restore the discovery URL ☝ a person
   Verify: A fresh sign-in completes without restarting anything else.
   Undo: none — this is the rollback

Stop the drill when

• The override cannot be reverted on the first attempt
  Stop and work it as an incident: the estate is now in a real identity outage of its own making.

What happened

Never run, in either mode. Everything above it on this card is a prediction the dependency graph made and nobody has checked.

Injection

container-stop · reversible · no person required

Stop the auth container on VIN. No configuration is edited and no key is touched.

Undo: Start the container and confirm a fresh sign-in completes end to end.

Steps

1. Establish one live session before injecting ▸ proposed
   Without a session established beforehand there is no way to tell whether existing sessions survive, which is the entire question.
   Verify: A session exists and is making successful requests.
   Undo: none — reading is not a change
2. Stop the auth service ☝ a person
   Verify: A fresh sign-in fails.
   Undo: Start the auth container.
3. Test the session established in the baseline ▸ proposed
   Verify: The result is recorded either way: a surviving session and a dead one are both valid answers, and only one of them matches what the identity register claims.
   Undo: none — reading is not a change
4. Confirm the recovery console still renders ▸ proposed
   This is the page an operator opens during an auth outage. If it needed auth, it would be unopenable exactly when it is needed.
   Verify: The console renders over loopback with the auth service stopped.
   Undo: none — reading is not a change
5. Restart auth ▸ proposed restart-auth
   Verify: A fresh sign-in completes and the control plane recovers without its own restart.
   Undo: none — this is the rollback

Stop the drill when

• The auth service does not come back on the first restart
  Stop the drill and work the incident. A key or store problem surfacing here is a real fault that the drill merely uncovered.

What happened

Never run, in either mode. Everything above it on this card is a prediction the dependency graph made and nobody has checked.

Injection

dataset-unmount · reversible · no person required

Unmount the backup dataset on the NAS. The pool is untouched and no object is deleted.

Undo: Remount the dataset and let the next scheduled backup land normally.

Only the on-site half can be injected. The off-site bucket belongs to a provider, so an executed run of this drill is always a partial rehearsal of the scenario it describes — and the register would rather say that than round it up to a pass.

Steps

1. Record the newest object in both copies ▸ proposed
   Verify: A named object and timestamp for the on-site copy and for the off-site one.
   Undo: none — reading is not a change
2. Unmount the backup dataset ☝ a person
   Verify: Writes to the dataset fail.
   Undo: Remount the dataset.
3. Wait for a scheduled backup and see whether it complains ▸ proposed
   The question is not whether the backup fails. It is whether anybody finds out, and when.
   Verify: The failure produced a signal that reached a person or an incident, with the delay recorded.
   Undo: none — reading is not a change
   A backup that fails silently is indistinguishable from one that succeeded, until the day it matters.
4. Remount and let it land ▸ proposed remount-nas-backups
   Verify: The next scheduled backup completes and the dataset holds a newer object than the baseline.
   Undo: none — this is the rollback
5. Re-run the off-site sync and confirm it catches up ⚙ automatic rerun-offsite-sync
   Verify: The off-site copy holds the object the on-site copy gained during the drill.
   Undo: Stop the sync. It copies and never deletes, so a stopped sync leaves the off-site copy exactly as it was.

Stop the drill when

• The pool reports a fault at any point during the drill checked automatically
  Remount immediately and stop. A storage fault during a storage drill is an incident wearing the drill's clothes.
• The drill has run 48 hours without a scheduled backup attempting the dataset
  Remount and file it: a backup schedule that does not fire is a bigger finding than the one this drill went looking for.

What happened

Never run, in either mode. Everything above it on this card is a prediction the dependency graph made and nobody has checked.

Injection

unit-stop · reversible · no person required

Stop the WireGuard unit on VIN. The peer configuration is not touched.

Undo: Start the unit and confirm the handshake completes without editing anything.

Steps

1. Record replication lag and the second path ▸ proposed
   Verify: Lag is under the warn threshold and the tailnet path answers before anything is stopped.
   Undo: none — reading is not a change
2. Stop the tunnel ☝ a person
   Verify: The handshake counter stops advancing.
   Undo: Start the WireGuard unit.
3. Prove the second private path still gets you in ▸ proposed
   Verify: An operator reaches home1 over the tailnet while WireGuard is down.
   Undo: none — reading is not a change
   If this fails, private-access is a single point of failure the graph does not know about, and that finding is worth more than the rest of the drill.
4. Bring the worker back once the link returns ⚙ automatic restart-worker
   Verify: The worker accepts a dispatched task and the control plane stops queueing.
   Undo: Stop the worker again. It was not serving before the step, so stopping it returns the estate to exactly the state the step began in.
   Safe in the action catalogue and undoable, so automation may run it inside a drill it did not start.
5. Restart the tunnel and let replication catch up ▸ proposed restart-wireguard
   Verify: Replication lag returns under the warn threshold without a resync.
   Undo: none — this is the rollback

Stop the drill when

• Replication lag exceeds the critical threshold and is still climbing 15 minutes after the tunnel returns checked automatically
  Stop the drill. A replica that cannot catch up is a data-resilience incident, not a link drill.
• The tailnet path is also down checked automatically
  Restart the tunnel immediately: the estate would have no private path in at all, which is not a state to explore on purpose.

What happened

Never run, in either mode. Everything above it on this card is a prediction the dependency graph made and nobody has checked.

Injection

unit-stop · reversible · no person required

Stop the Postgres unit on home1. The data directory is not touched.

Undo: Start the unit and confirm the replica reconnects without a resync.

Steps

1. Record lag and the newest WAL segment archived ▸ proposed
   Verify: Both numbers are recorded before anything stops.
   Undo: none — reading is not a change
2. Stop the database ☝ a person
   Verify: Writes fail and reads keep succeeding.
   Undo: Start the unit.
3. Check the control plane refuses work rather than swallowing it ▸ proposed
   A queue that accepts tasks it cannot persist is worse than one that returns an error, because the loss is silent and arrives later.
   Verify: A submitted task is rejected with an error, and nothing is acknowledged that was not written.
   Undo: none — reading is not a change
4. Start it again and confirm nothing needed restarting ☝ a person
   Verify: Writes succeed, the replica reconnects without a resync, and no dependent service needed a manual restart to recover.
   Undo: none — this is the rollback
   A dependent that needs a restart after its database returns is a finding against that dependent, not against the database.

Stop the drill when

• The replica begins to diverge or the archiver backs up checked automatically
  Start the primary immediately. This drill is not worth a resync.

What happened

• 2026-07-12 executed
  ◔ partial · homelab-operator · nothing timed

  This was the monthly restore drill recorded in config/datastores.json, not a failover drill. A dump was restored to a scratch database and verified against a known query.

  It proved the restore procedure works. It timed nothing: no stopwatch was started, so the 25 minute recovery objective is exactly as unmeasured as it was before.

  The restore read from nas-backups, which sits in the same room as the primary it protects.

  Finding: Every restore this estate has passed has read from a copy in the same room as the thing it protects. The off-site path — restoring onto VIN with nothing at home reachable — has never been walked. filed as HOST-009

  Finding: The run recorded an outcome and no durations. A drill that does not time itself cannot verify an RTO, which is most of what a drill is for. The runner now refuses to record an executed run without a measurement block. filed as HOST-013

Why it cannot be executed

• self-observing
  This console runs on vin, which the scenario takes down. Its own record of the drill would stop at the moment the drill began.

Injection

provider-stop · reversible · needs a person

Stop the instance from the provider console.

Undo: Start the instance and let the units come up in their declared order.

Steps

1. Put a watcher outside both sites ☝ a person
   A phone on mobile data, or anything off-fleet. Without it this drill has no observer at all.
   Verify: The external watcher records the exact second the published hostname stops answering.
   Undo: none — reading is not a change
2. Stop the instance ☝ a person
   Verify: The public hostname stops answering and the LAN-side services do not.
   Undo: Start the instance.
3. Record what noticed, and how ☝ a person
   The interesting answer is usually 'a person, eventually'. Writing that down is what makes the case for an off-fleet dead-man's switch.
   Verify: The first notification of the outage is named, with its source and its timestamp.
   Undo: none — reading is not a change
4. Start the instance and verify the edge ▸ proposed reboot-vin
   Verify: Published hostnames answer, sign-in completes, and the console renders without a manual restart of anything.
   Undo: none — this is the rollback

Stop the drill when

• The instance does not return within 15 minutes of being started
  Treat it as an incident and rebuild from the crown-jewels backup. The 15 minute RTO on VIN is a rebuild time, and this is the one chance to find out whether it is real.

What happened

• 2026-08-02 simulated
  ✓ passed · homelab-operator · nothing timed

  A walk of the model, not of the estate. It agreed with the dependency graph, which is the least surprising possible result — the simulation and the graph are the same arithmetic.

  It did establish the thing that keeps this drill from being executed: 86 of 162 capability points go, and every observer the estate owns is inside them.

  Finding: There is nothing off-fleet that would notice VIN stopped answering. The estate's only detector of a VIN outage is a process running on VIN. Until an external dead-man's switch exists, the detection time for this scenario is 'whenever a person happens to look'. filed as HOST-023

Why it cannot be executed

• no-injection
  There is no way to induce this fault. It belongs to somebody else, and the estate’s own policy forbids faking it by editing DNS, firewall rules or the Caddyfile. This drill is a simulation by construction.

Injection

none · not reversible · needs a person

There is no way to induce this. Faking it would mean editing DNS records or a firewall rule, and this estate's policy forbids both — a drill that breaks managed configuration to simulate an outage has caused one. So this drill is simulate-only, derived rather than declared, and the register says so on the page rather than quietly listing it beside the ones that can be run.

Steps

1. Record the TTL on every published record ▸ proposed
   The TTL is the length of the grace period. A five-minute TTL and a one-hour TTL are two entirely different outages.
   Verify: Every published name has its TTL recorded.
   Undo: none — reading is not a change
2. Find the next certificate renewal deadline ▸ proposed
   DNS-01 renewal needs the provider API. An outage that outlasts a renewal window turns a name-resolution problem into a TLS failure two days later, and nobody connects the two.
   Verify: The nearest renewal date is recorded, with the margin to it.
   Undo: none — reading is not a change
3. Walk the scenario against the model ▸ proposed
   Verify: The derived blast radius is recorded and compared against the last run's.
   Undo: none — simulation changes nothing
4. List every recovery step that needs this API ▸ proposed
   The output is a list of failovers that are unavailable during this outage. That list is the drill's real product.
   Verify: Each recovery action in the catalogue is marked as needing the provider control plane or not.
   Undo: none — reading is not a change

Stop the drill when

• Somebody proposes inducing this by editing a DNS record
  Stop. That is a change to managed configuration and it is not a drill. The simulation is the drill.

What happened

• 2026-08-02 simulated
  ✓ passed · homelab-operator · nothing timed

  Simulation only, and permanently so. The steps that produce this drill's real value — the TTL inventory, the certificate deadline and the dependency audit — are reading exercises that do not need the outage to be real.

  The dependency audit was not completed in this run, so the list of recovery actions that need the provider control plane does not exist yet.

  Finding: The VPS-loss scenario's degraded mode assumes published names can be re-pointed. This scenario removes exactly that ability. The two drills have never been considered together, and the combination is more likely than either alone if the cause is a billing or account action. filed as HOST-016

Why it cannot be executed

• no-injection
  There is no way to induce this fault. It belongs to somebody else, and the estate’s own policy forbids faking it by editing DNS, firewall rules or the Caddyfile. This drill is a simulation by construction.

Injection

none · not reversible · needs a person

There is no permitted way to induce this. Powering off vps1 would take down another operator's authentik, mail server and 13 published hostnames, none of which belong to this estate, to measure a loss this estate has already derived as zero. That is not a drill, it is an outage with a clipboard. If estate workloads are ever placed here, this injection becomes "stop the estate's containers only" and the drill becomes executable — that transition is the thing to watch for.

Steps

1. Confirm the second domain can execute work at all ▸ proposed
   Before asking what its loss costs, establish that it currently gives anything. Run the second-domain canary: a container on vps1 with --network none that must return the SHA-256 of a nonce it has never seen. Network-isolated on purpose — an answer proves the node produced it without help from VIN, home1, DNS or the internet.
   Verify: `jarvisos-independence.mjs canary --node vps1 --confirm` returns the expected digest, and the register records the receipt.
   Undo: none — the container is --rm and holds no state
2. Derive the blast radius before touching anything ▸ proposed
   Walk the dependency graph for the loss of vps1 and write down the predicted capability loss. If it is not empty, this drill has found something and the rest of it changes.
   Verify: The derived capability loss is recorded, and it matches what this drill declares it expects.
   Undo: none — reading is not a change
3. Ask what would notice ☝ a person
   Name the signal that would change if the node stopped existing, with its source. "A person, when they next open the page" is a valid and expensive answer.
   Verify: The first detector of a vps1 outage is named, or its absence is.
   Undo: none — reading is not a change

Stop the drill when

• Anybody proposes inducing this for real while another operator's workloads are on the node
  Stop. The injection is declared none for a reason, and it stays none until the estate owns something on this node that can be stopped on its own.

What happened

• 2026-08-17 simulated
  ✓ passed · jarvisos-worker · nothing timed

  The readiness step was executed for real against the live node; the injection was not, and may not be. So this is recorded as simulated with two steps actually run, rather than as an executed drill.

  Readiness passed: a --network none container on vps1 returned the correct SHA-256 of a fresh nonce in 281ms on a 4-vCPU host. The node executes work and needed nothing outside itself to do it.

  The derivation agreed with the declaration: losing vps1 costs no declared capability, because no declared capability touches it.

  A drill that costs nothing is measuring an asset nothing uses. The value of this run is the baseline: the next time it is run, a non-empty capability loss means the estate has genuinely started depending on its second domain.

  Finding: Nothing would notice. There is no JarvisOS agent on vps1 and no credential for its Coolify, so the estate's detection time for the loss of its own second failure domain is "whenever a person runs the probe". The console shows the node as unknown, and it would show exactly the same thing if the machine were destroyed. filed as HOST-025