# Offline Gentoo base recovery

`hydra-system` creates and restores a matched Btrfs root snapshot and the installed
`EFI/HydraOS` tree. Boot the trusted Hydra live ISO and run it as root with the
installed root and ESP unmounted. This command does not operate from the installed
OS. An installed `/etc/hydra/live-mode/active` file alone is insufficient: the
command also checks the live boot arguments, overlay root and squashfs mount.

The supported installation has `@root`, `@home` and `@snapshots`. `/boot`, kernel
modules, the Portage package database and world file belong to `@root`. Nested
subvolumes beneath `@root` are refused. Boot and fstab must select the filesystem
UUID plus `subvol=@root`, never a numeric root subvolume ID. `@home` and VM disks
stored there survive restoration; files stored elsewhere in `@root` roll back.

Record the Btrfs filesystem UUID and ESP PARTUUID at installation. From the live
ISO, inspect the plan before executing it:

```sh
hydra-system checkpoint --root-uuid ROOT_UUID --esp-partuuid ESP_PARTUUID --name before-update
hydra-system checkpoint --root-uuid ROOT_UUID --esp-partuuid ESP_PARTUUID --name before-update \
  --execute --confirm-root-uuid ROOT_UUID
hydra-system list --root-uuid ROOT_UUID
```

Commands emit JSON; the completed checkpoint has an `id` field. Boot the installed
system to perform ordinary Portage updates. To recover, boot the live ISO again:

```sh
hydra-system restore --root-uuid ROOT_UUID --checkpoint CHECKPOINT_ID
hydra-system restore --root-uuid ROOT_UUID --checkpoint CHECKPOINT_ID \
  --execute --confirm-root-uuid ROOT_UUID
```

Plans and list mount targets read-only; Btrfs inspection disables log replay.
Execution requires matching confirmation and verifies UUID/PARTUUID uniqueness,
ordinary partitions, inactive swap, no holders, and a single-device Btrfs layout.
A private mount namespace and a root-owned lock isolate the command's mounts.
No installed executable, network access or installed recovery program is needed.

Recovery restores only the recorded `EFI/HydraOS` directory. It does not change
partition tables, other EFI directories, `EFI/BOOT`, firmware entries or BootOrder.
Boot the restored system through its HydraOS firmware entry, or select
`EFI/HydraOS/grubx64.efi` in firmware. A whole-disk installation can also have a
removable fallback under `EFI/BOOT`; that fallback is deliberately not restored.
If an updated fallback can no longer load the restored root, select the owned
HydraOS loader explicitly. Automatic failed-boot recovery is not provided.

## Durable records and interrupted restores

The installer marker `/etc/hydra/storage.json` has exactly these fields:

```json
{
  "format": 1,
  "install_uuid": "installation UUID",
  "filesystem": "btrfs",
  "root_fs_uuid": "Btrfs UUID",
  "root_partuuid": "root GPT partition UUID",
  "esp_uuid": "FAT UUID (XXXX-XXXX)",
  "esp_partuuid": "ESP GPT partition UUID",
  "owned_efi_path": "EFI/HydraOS",
  "root_subvolume": "@root"
}
```

At the Btrfs top level, completed checkpoints are stored as:

```text
@snapshots/<checkpoint-UUID>/root             read-only Btrfs snapshot
@snapshots/<checkpoint-UUID>/efi/             owned EFI backup
@snapshots/<checkpoint-UUID>/checkpoint.json  completed descriptor, published last
```

The descriptor fields are `format`, `id`, `name`, `created`, `storage`,
`source_root_uuid`, `snapshot_uuid` and `efi_manifest`. `format` is integer 1;
`created` is a UTC ISO timestamp; `storage` is the exact marker above. Each manifest
entry records a relative `path`, `kind` (`dir` or `file`), numeric `mode` and
`mtime_ns`; files additionally record `size` and SHA-256 `sha256`. Backup paths,
bytes and metadata are rechecked before every restore. Links, hardlinks, special
files, ambiguous FAT names and extra checkpoint files are refused. Metadata files
must be regular, singly linked, owned by root and not group/world writable.

`@snapshots/restore.json` is the durable active transaction. It has exactly
`format`, `id` (transaction UUID), `checkpoint`, `checkpoint_digest`, `storage`,
`old_uuid`, `new_uuid`, `prepared_root`, `failed_root` and `phase`. The digest is
SHA-256 of the descriptor's canonical JSON (sorted keys, compact separators).
Phases are `preparing`, `prepared`, `root-retained`, `root-installed` and
`efi-restored`. `new_uuid` starts null; it is recorded once the writable snapshot
exists. Snapshot parent UUID and marker checks establish the source if power was
lost before that UUID was journaled. Directory fsync and filesystem sync delimit
transitions.

A writable copy is prepared as `@restore-<transaction-UUID>`. The previous root is
retained as `@failed-<transaction-UUID>` before the prepared root becomes `@root`.
Resume verifies the actual subvolume UUIDs at all three paths and accepts only the
three valid rename arrangements. The recorded phase alone never decides which
root to keep. After EFI restoration and verification, the transaction is archived
as `@snapshots/restored-<transaction-UUID>.json`. No failed roots or checkpoints are
automatically deleted.

Btrfs root replacement and FAT file copying are not one atomic operation. Power
loss can leave the system temporarily unbootable. Rerun the **same checkpoint ID**
from the live ISO with the same explicit confirmation; recovery validates its
records and resumes. Do not manually rename or delete recovery roots or journal
files. Unexplained UUID/path combinations are refused for offline inspection.
An incomplete checkpoint without its completed descriptor is never restorable.

These snapshots remain on the original disk. They are not external backups, and
there is no promise of online application consistency: checkpoint only while the
installation is offline. The unit suite uses a persistent fake Btrfs backend and
fresh processes to exercise journal and rename interruptions. Real Btrfs, FAT,
UEFI, Portage and disk-only boot acceptance must be verified separately in a fresh
VM before release; unit tests do not establish those claims.

Btrfs's [subvolume documentation](https://btrfs.readthedocs.io/en/latest/btrfs-subvolume.html)
explains snapshot boundaries and read-only snapshots; the command checks the
[read-only property](https://btrfs.readthedocs.io/en/latest/btrfs-property.html)
explicitly instead of inferring it from mount options.