New interactive logins fail for as long as the outage lasts. Every session already issued keeps working, and every refresh keeps working, because neither asks Google anything.

Credential: GOOGLE_CLIENT_SECRET in /etc/jarvisos/config.env on VIN — referred to by name and location only, never read into this register or any log line.

Fallback (planned): One local credential for one operator account, verified on the node with an argon2id hash and a second factor, usable only from the tailnet and only for routes marked break-glass. It exists so that a multi-hour Google outage does not also mean nobody can log in to fix anything. It is not implemented, and until it is, an IdP outage means no new logins at all.

Signs sessions and refresh records; every verifier checks that signature

Where

Private half at /etc/jarvisos/auth/keys/session-signing.<kid>.pem on each issuing node, mode 0600, owned by the auth service user. Public half published to each verifier as a JWKS file at /etc/jarvisos/auth/jwks.json. Excluded from every replicated storage tier by the rule in config/storage.json — live secrets do not go anywhere that replicates.

Held by

auth-vin-a (running), auth-vin-b (planned), auth-home1 (planned)

Rotation

every 90 days, 1d overlap · never performed

1. Generate the new keypair on the node that will issue with it. Never copy a private half between nodes over anything but the mesh, and never through a paste buffer.
2. Distribute the new public half to every verifier and confirm each one lists both key ids before anything is signed with the new one.
3. Switch issuing to the new key id. Verifiers now accept both.
4. Wait the overlap window — 24h, which is longer than the 12h refresh lifetime, so nothing signed with the old key is still alive at the end of it.
5. Remove the old public half from every verifier, then destroy the old private half.
6. Skipping the overlap is the emergency procedure, not the routine one: it is how everyone is signed out at once, and it is listed as such under revocation.

Authenticates this estate to Google when exchanging an authorization code

Where

GOOGLE_CLIENT_SECRET in /etc/jarvisos/config.env on VIN, mode 0600, root-owned. Its value appears in no register, no page, no log and no report.

Held by

auth-vin-a (running), auth-vin-b (planned), auth-home1 (planned)

Rotation

every 365 days, immediate overlap · never performed

1. Create a second client secret in the Google Cloud console. Google permits two to be active at once; that is the overlap window and it is the only reason this is not an outage.
2. Write the new value into /etc/jarvisos/config.env on each node holding it, then restart each auth instance one at a time and confirm a real login completes between restarts.
3. Delete the old secret in the console only after every instance has been restarted and proven.
4. Existing sessions are unaffected throughout: nothing on the verification path uses this credential.

Pass looks like: Fewer than 30 seconds of partial failures, and no request fails after the pool check has deregistered the dead instance. Sessions are not lost: nobody is logged out by this.

1. Confirm both instances are in the pool and both answer /healthz.
2. Open a published route in a browser holding a live session, and leave a one-request-per-second loop running against it from the tailnet.
3. docker kill the container serving auth-vin-a.
4. Record the number of failed requests in the loop and the wall-clock time until failures stop.
5. Restart the container and confirm it re-enters the pool.

The failure this drill covers is the one that actually happens. It is also the only one the current single-instance deployment cannot pass at all.

Pass looks like: In the target design: verification keeps working until the session expires, login fails immediately with a clear error, and no auth instance is deregistered by the readiness check. Today: everything fails at once, which is the result this drill exists to make undeniable.

1. With a live session in a browser, stop the session store.
2. Reload a published route repeatedly for the length of one session TTL.
3. Attempt a fresh login from a private window.
4. Restart the store and confirm both paths recover without a restart of the auth instances.

Pass looks like: Verification and refresh unaffected; login fails cleanly; nothing is deregistered. Any deviation means something on the hot path is calling the vendor.

1. Block egress to the provider's endpoints from the auth instances only. Do not touch the firewall on any other path, and revert with a scheduled command in case the session is lost.
2. Confirm existing sessions keep verifying and refreshes keep succeeding.
3. Confirm a fresh login fails with a message that names the upstream rather than a generic 500.
4. Confirm no instance leaves the pool.
5. Unblock and confirm login recovers with no restart.

This is the drill that proves the first rule in src/auth-ha.js rather than asserting it.

Pass looks like: In the target design: verification and login unaffected, because nothing on either path is at home. Today: everything published stops authenticating the moment home1 is unreachable.

1. From VIN, confirm which published routes still authenticate while home1 is powered off.
2. Repeat with home1 healthy but the mesh path to it blocked, which is the uplink case and behaves differently.
3. Record whether verification, refresh and login each survive, and for how long.
4. Bring the site back and confirm nothing needed a restart.

The two halves are one drill because the difference between them — a dead node versus a live but unreachable one — is exactly where a design that assumed 'down' silently fails.

Pass looks like: The operator has a working shell and a working status page within 15 minutes, with the SSO entirely absent. Nothing published is expected to work — VIN is the only public edge.

1. From the home side, with VIN treated as gone, reach home1 over SSH using the operator key only.
2. Start or reach the status console on 127.0.0.1 through that session and confirm it renders with no auth service running anywhere.
3. Confirm no step of this required a password manager, a browser, Google, or a page served through Caddy.
4. Record what remained unreachable, honestly: every published route is one of them.

This drill is about the operator's access, not the estate's availability. Conflating the two is how an estate ends up with a beautiful recovery plan nobody can open.

Pass looks like: A routine rotation logs nobody out. An emergency rotation logs everybody out within five minutes and leaves the system able to issue new sessions immediately.

1. Perform a full routine rotation following the declared procedure, holding a live session open throughout.
2. Confirm the held session survives the entire rotation, including the removal of the old public half after the overlap.
3. Then perform the emergency variant on a test key id: skip the overlap and confirm every session dies at once and a fresh login works immediately afterwards.
4. Record the wall-clock time of the second variant — that number is the global sign-out worst case, and it should replace the declared one.

Pass looks like: Every revocation bites inside 60 seconds, and the fail-static window is bounded and observable rather than indefinite.

1. Hold a live session in one browser and log out from another.
2. Poll a published route with the first session and record the time until it stops being served.
3. Repeat with the denylist store deliberately unavailable, and confirm the stale-tolerance bound behaves as declared: served for up to 60s, privileged routes refused.
4. Repeat for an administrator sign-out of another subject.

Until this passes, every revocation number in this register is an intention.