JarvisOS HomeLab status
No signal

Object storage and the S3 layer

3 endpoints · 3 buckets · 4/6 copies running · register partly-running

The object storage layer: which endpoint answers when an artifact or a backup is asked for, whether that answer is current, and who is allowed to delete it. The backup buckets are running today. The artifact plane — MinIO on the NAS with a read-through cache on VIN — is designed and not deployed, and this register says so in every number rather than describing the estate it hopes for.

Readable in every scenario 8/12 8 from a current, complete copy · 6/12 still writable
Off-site today 2/2 buckets that must survive the site and have a running copy outside it
Worst restore 1.5h from the best off-site copy · €0 of egress across every bucket
Proven off-site 0/2 1/2 restored from anywhere · 2/2 running edges verified

Open gaps

Measured against the copies that are running, not the ones designed

What still answers

Computed from the copies running today, and from who is left to ask

Each column is a real failure with a real observer: somebody has to be asking for the object, and where they are asking from decides which endpoints they can even reach. Served is a current, complete copy. Stale is complete and further behind than the objective allows. Partial is a cache: what it holds is current, and it does not hold everything.

Bucket Home rack offline
asked from vin
VPS offline
asked from home1
Home uplink down
asked from home1
Off-site provider gone
asked from vin
Database backups Served write: Read-only Served write: Writable Served write: Writable Served write: Writable
Configuration archive Served write: Read-only Served write: Writable Served write: Writable Served write: Writable
Build artifacts and layers No copy write: Read-only No copy write: Read-only No copy write: Read-only No copy write: Read-only

The endpoints

313 GB across all of them today · €0/month

An object store is not a disk: it has a name that resolves from some nodes and not others, a credential with a scope, and a set of S3 semantics that may or may not actually be implemented. All three are stated here, because each of them fails quietly.

MinIO on the NAS MinIO

nas-backups · No signal

The source of truth for every durable object in the estate: database backups, configuration archives and, once it exists, the artifact plane. It is on the NAS because that is the only place with room for the whole set, and it is not off-site because it is in the same room as the thing it protects.

Holds today
248 GB · 22.6k objects
By design
548 GB
Address
https://s3.nas.home.arpa path-style
Read / write
940 / 720 Mbps measured 2026-08-14
Cost
€0/month · owned-hardware · €0/GB out
Tier
t-nas-pool fd-home-rack

What it really supports: versioning object lock server-side lifecycle checksums (sha256) server-side replication

Reachable from Over What that path is
nas direct Loopback on the NAS itself, which is how the ILM and heal administration runs.
home1 internal-dns Over the rack LAN. Needs internal DNS to resolve s3.nas.home.arpa; the fallback is the literal address, which is in the runbook.
vin wireguard-tunnel → isp-uplink Over the mesh, which crosses the home uplink. That makes this path unavailable in exactly the scenario the off-site copy exists for.
Credential Scope Read from Held on
cred-nas-rw read-write JARVISOS_S3_NAS_RW · /etc/jarvisos/s3-nas-rw.env home1
cred-nas-ro read JARVISOS_S3_NAS_RO · /etc/jarvisos/s3-nas-ro.env vin
cred-nas-admin admin JARVISOS_S3_NAS_ADMIN · /etc/jarvisos/s3-nas-admin.env nas

Names and locations only. Nothing in this console reads a credential, so nothing in it can print one.

MinIO cache on VIN MinIO

vin · No signal

A read-through cache so a build on VIN does not cross the home uplink for every layer, and so something still answers for artifacts when the house is dark. It is a cache and nothing else: it is never the source of truth for anything, and losing it costs a slower build.

Holds today
0 MB of a 40 GB quota · 0 objects
By design
12 GB · 30% of quota
Address
https://s3.vin.internal path-style
Read / write
2000 / 2000 Mbps unmeasured
Cost
€0/month · included · €0/GB out
Tier
t-local-vin fd-vps

What it really supports: no versioning no object lock server-side lifecycle checksums (sha256) no server-side replication

Reachable from Over What that path is
vin direct Loopback on VIN. This is the path that matters: the cache exists for the node it is on.
home1 wireguard-tunnel → isp-uplink Over the mesh, for warming it deliberately after a release. Not a path anything depends on.
Credential Scope Read from Held on
cred-vin-cache-rw read-write JARVISOS_S3_VIN_RW · /etc/jarvisos/s3-vin-rw.env vin

Names and locations only. Nothing in this console reads a credential, so nothing in it can print one.

GWSCLI / S3 off-site GWSCLI/S3

gwscli-s3-offsite · No signal

The encrypted subset that survives the house. It holds the backup buckets and nothing reproducible, because paying to move bytes off-site that a build would recreate is buying a slower version of something we already have.

Holds today
65 GB of a 100 GB quota · 10.9k objects
By design
65 GB · 65% of quota
Address
https://s3.gwscli.example.net path-style
Read / write
200 / 40 Mbps measured 2026-07-12
Cost
€0/month · free-tier · egress unpriced
Tier
t-offsite-gwscli fd-offsite

What it really supports: no versioning no object lock no server-side lifecycle no checksums (none) no server-side replication

Reachable from Over What that path is
vin direct Straight out of the VPS's own uplink. This is the restore path: it does not touch the home rack, which is the point.
home1 isp-uplink The write path. Everything pushed off-site crosses the home uplink, which is why an uplink outage silently stops the estate advancing its off-site copy.
Credential Scope Read from Held on
cred-offsite-rw read-write JARVISOS_S3_OFFSITE_RW · /etc/jarvisos/s3-offsite-rw.env home1
cred-offsite-ro read JARVISOS_S3_OFFSITE_RO · /etc/jarvisos/s3-offsite-ro.env vin

Names and locations only. Nothing in this console reads a credential, so nothing in it can print one.

The buckets

One source of truth each, a derived distance from it to every other copy, and the arithmetic for pulling the best off-site copy back — in hours and in euros.

Database backups backup unique must survive the site

owned by homelab-operator · db-backups

Postgres base backups, WAL archives and the Supabase dumps. When this is gone, the estate's history is gone with it: there is no way to reconstruct a table from anything else that exists.

Off-site distance
1d behind the authority objective 1d
Restore from off-site
1.5h at 200 Mbps · unpriced egress objective 8h
Size today
240 GB · 27.5k objects
Last drill
2026-07-12 from c-db-nas — pass, 41m never from off-site
Copy Role Bucket / prefix Size Behind Protections
MinIO on the NAS source of truth jarvisos-backups/postgres/ 240 GB authority versions deletable dataset-level
GWSCLI / S3 off-site off-site replica jarvisos-offsite/restic/backups/ 62 GB 1d deletable client-side

Read order: MinIO on the NAS → GWSCLI / S3 off-site

Edge Route When Deletes Verified
e-db-offsite push on home1 c-db-nas → c-db-offsite nightly at 02:20 Europe/Amsterdam retain 2026-08-10 (pass)

Lifecycle

  • lc-db-nas-expire — expire after 190d on MinIO on the NAS server-side
    MinIO ILM (mc ilm rule add), applied on the NAS. Set beyond the 180-day retention on purpose: the retention command decides what to delete, and the lifecycle rule is only the backstop for what it misses.
  • lc-db-nas-multipart — abort-multipart after 2d on MinIO on the NAS server-side
    MinIO ILM. An interrupted 40 GB base backup leaves parts that no listing shows and that occupy the pool until something aborts them.
  • lc-db-offsite-forget — expire after 190d on GWSCLI / S3 off-site needs a client
    restic forget --prune, inside the same push that writes it, and its exit status fails the push. A server-side expiry rule here would not be retention, it would be repository corruption: the provider cannot tell a pack file that is still referenced from one that is not.
  • lc-db-offsite-multipart — abort-multipart after 3d on GWSCLI / S3 off-site needs a client
    `jarvisos-objectstore.mjs lifecycle` lists this as a rule the far end applies none of; the operator aborts stale parts by hand, quarterly. This endpoint runs no rules of its own and orphaned parts here are billed against the quota.

Integrity

  • MinIO on the NAS: mc admin heal on the bucket, then a sha256 manifest compare against the newest base backup, every 30d, full · last 2026-08-03 (pass)
  • GWSCLI / S3 off-site: restic check --read-data-subset=5%, every 30d, sampled 5% · last 2026-08-10 (pass)

How it comes back

  1. On VIN: source /etc/jarvisos/s3-offsite-ro.env — the read-only off-site key.
  2. Fetch the repository passphrase from wherever it is held offline. It is on no node, including this one.
  3. restic -r s3:https://s3.gwscli.example.net/jarvisos-offsite/restic/backups snapshots — confirm the newest snapshot is within the recovery point.
  4. restic restore latest --target /srv/restore/pg --verify
  5. Compare the restored base backup's sha256 against the manifest in the same snapshot before starting Postgres against it.
  6. Record the wall-clock duration in this register. An untimed restore leaves the recovery time an estimate.

Plus 45 minutes that are not transfer: finding the passphrase, decrypting, and verifying before anything is started against it.

Configuration archive backup unique must survive the site

owned by homelab-operator · config-archive

Coolify's own database dumps, the compose and environment files with credentials removed, and the Caddy configuration. It answers the question nobody plans for: what did this look like before the change nobody remembers making.

Off-site distance
1d behind the authority objective 1d
Restore from off-site
22m at 200 Mbps · unpriced egress objective 2h
Size today
8 GB · 6k objects
Last drill
never never from off-site
Copy Role Bucket / prefix Size Behind Protections
MinIO on the NAS source of truth jarvisos-backups/config/ 8 GB authority versions deletable dataset-level
GWSCLI / S3 off-site off-site replica jarvisos-offsite/restic/config/ 3 GB 1d deletable client-side

Read order: MinIO on the NAS → GWSCLI / S3 off-site

Edge Route When Deletes Verified
e-config-offsite push on home1 c-config-nas → c-config-offsite nightly at 02:40 Europe/Amsterdam retain 2026-08-10 (pass)

Lifecycle

  • lc-config-nas-expire — expire after 400d on MinIO on the NAS server-side
    MinIO ILM. A year of monthly points is what this archive is for, so the rule sits beyond the retention window rather than inside it.
  • lc-config-nas-multipart — abort-multipart after 2d on MinIO on the NAS server-side
    MinIO ILM.
  • lc-config-offsite-forget — expire after 400d on GWSCLI / S3 off-site needs a client
    restic forget --prune in the same push, exit status included.
  • lc-config-offsite-multipart — abort-multipart after 3d on GWSCLI / S3 off-site needs a client
    Listed by `jarvisos-objectstore.mjs lifecycle` and aborted by hand, quarterly, in the same sitting as the database repository.

Integrity

  • MinIO on the NAS: sha256 manifest compare over the whole prefix, every 30d, full · last 2026-08-03 (pass)
  • GWSCLI / S3 off-site: restic check --read-data (the repository is small enough to read all of it), every 30d, full · last 2026-08-10 (pass)

How it comes back

  1. On VIN: source /etc/jarvisos/s3-offsite-ro.env.
  2. Fetch the restic passphrase from offline storage.
  3. restic -r s3:https://s3.gwscli.example.net/jarvisos-offsite/restic/config snapshots
  4. restic restore latest --target /srv/restore/config --verify
  5. Read the restored env files before using them: credentials were removed on the way in, so anything that looks like one is a scrubbing failure and a finding.

Plus 20 minutes that are not transfer: finding the passphrase, decrypting, and verifying before anything is started against it.

Build artifacts and layers cache reproducible

owned by homelab-operator · artifact-cache

Build caches and container layers that a build on either node can pull instead of rebuilding. Losing all of it costs a slow afternoon, which is exactly why none of it is replicated off-site: paying to store bytes a `git clone` and a build would recreate is buying a slower version of what we already have.

Off-site distance
no off-site copy is fed objective 1d
Restore from off-site
nothing off-site to restore from objective 4h
Size today
0 MB · 0 objects
Last drill
never
Copy Role Bucket / prefix Size Behind Protections
MinIO on the NAS planned source of truth jarvisos-artifacts 300 GB est. authority deletable dataset-level
MinIO cache on VIN planned cache jarvisos-artifacts 12 GB est. nothing feeds it deletable none validates: etag

Read order: MinIO cache on VIN (cache) → MinIO on the NAS

Edge Route When Deletes Verified
e-artifacts-warm planned pull on vin c-artifacts-nas → c-artifacts-vin on demand, and after every release retain never

Lifecycle

  • lc-artifacts-expire — expire after 30d on MinIO on the NAS server-side
    MinIO ILM. A cache with no eviction is not a cache, it is a disk leak with good intentions.
  • lc-artifacts-multipart — abort-multipart after 1d on MinIO on the NAS server-side
    MinIO ILM.
  • lc-artifacts-vin-expire — expire after 14d on MinIO cache on VIN server-side
    MinIO ILM on VIN. Fourteen days rather than thirty, because this disk is rented and small: the cache's job is the current release, not the quarter.
  • lc-artifacts-vin-multipart — abort-multipart after 1d on MinIO cache on VIN server-side
    MinIO ILM on VIN.

Integrity

  • MinIO on the NAS: nothing verifies this copy
  • MinIO cache on VIN: nothing verifies this copy

How it comes back

  1. Nothing is restored. Every object here is rebuilt: `docker buildx build` against the source, and the layers repopulate as they are asked for.
  2. If the whole bucket is gone, the first build after that is slow and every build after it is not. That is the entire cost, and it is why nothing here is replicated off-site.

The access layer

5 client profiles

Which tool on which node talks to which endpoint, with which credential, at which scope. An object store has no "may put but not delete" permission, so a read-write key is a delete permission and is only issued to a node that actually writes.

Client Node Endpoint Credential Checksums Profile
cl-home1-mc mc home1 MinIO on the NAS cred-nas-rw read-write compared /etc/jarvisos/mc/config.json
cl-home1-restic restic home1 GWSCLI / S3 off-site cred-offsite-rw read-write compared /etc/jarvisos/restic-offsite.env
cl-vin-restore restic vin GWSCLI / S3 off-site cred-offsite-ro read compared /etc/jarvisos/restic-offsite-ro.env
cl-vin-rclone rclone vin MinIO on the NAS cred-nas-ro read compared /etc/jarvisos/rclone.conf
cl-nas-admin mc nas MinIO on the NAS cred-nas-admin admin compared /etc/jarvisos/mc/config.json

Accepted risks

Decisions on record, each against one copy and with a review date

Notes

Client-applied rules, sampled checks, unpriced egress and design not yet adopted

How to read this page

Source: config/objectstore.json · machine-readable at /api/v1/object-store · narrative in docs/OBJECT-STORAGE.md. Every recovery point here is derived from the replication schedules and every restore time from bytes over measured bandwidth; scripts/jarvisos-objectstore.mjs prints the commands that would verify them, and runs none of them.