Hydra-OS/docs/system-recovery.md

Offline Gentoo base recovery

hydra-system creates and restores a matched Btrfs root snapshot and the installed EFI/HydraOS tree. Boot the trusted Hydra live ISO and run it as root with the installed root and ESP unmounted. This command does not operate from the installed OS. An installed /etc/hydra/live-mode/active file alone is insufficient: the command also checks the live boot arguments, overlay root and squashfs mount.

The supported installation has @root, @home and @snapshots. /boot, kernel modules, the Portage package database and world file belong to @root. Nested subvolumes beneath @root are refused. Boot and fstab must select the filesystem UUID plus subvol=@root, never a numeric root subvolume ID. @home and VM disks stored there survive restoration; files stored elsewhere in @root roll back.

Record the Btrfs filesystem UUID and ESP PARTUUID at installation. From the live ISO, inspect the plan before executing it:

hydra-system checkpoint --root-uuid ROOT_UUID --esp-partuuid ESP_PARTUUID --name before-update
hydra-system checkpoint --root-uuid ROOT_UUID --esp-partuuid ESP_PARTUUID --name before-update \
  --execute --confirm-root-uuid ROOT_UUID
hydra-system list --root-uuid ROOT_UUID

Commands emit JSON; the completed checkpoint has an id field. Boot the installed system to perform ordinary Portage updates. To recover, boot the live ISO again:

hydra-system restore --root-uuid ROOT_UUID --checkpoint CHECKPOINT_ID
hydra-system restore --root-uuid ROOT_UUID --checkpoint CHECKPOINT_ID \
  --execute --confirm-root-uuid ROOT_UUID

Plans and list mount targets read-only; Btrfs inspection disables log replay. Execution requires matching confirmation and verifies UUID/PARTUUID uniqueness, ordinary partitions, inactive swap, no holders, and a single-device Btrfs layout. A private mount namespace and a root-owned lock isolate the command's mounts. No installed executable, network access or installed recovery program is needed.

Recovery restores only the recorded EFI/HydraOS directory. It does not change partition tables, other EFI directories, EFI/BOOT, firmware entries or BootOrder. Boot the restored system through its HydraOS firmware entry, or select EFI/HydraOS/grubx64.efi in firmware. A whole-disk installation can also have a removable fallback under EFI/BOOT; that fallback is deliberately not restored. If an updated fallback can no longer load the restored root, select the owned HydraOS loader explicitly. Automatic failed-boot recovery is not provided.

Durable records and interrupted restores

The installer marker /etc/hydra/storage.json has exactly these fields:

{
  "format": 1,
  "install_uuid": "installation UUID",
  "filesystem": "btrfs",
  "root_fs_uuid": "Btrfs UUID",
  "root_partuuid": "root GPT partition UUID",
  "esp_uuid": "FAT UUID (XXXX-XXXX)",
  "esp_partuuid": "ESP GPT partition UUID",
  "owned_efi_path": "EFI/HydraOS",
  "root_subvolume": "@root"
}

At the Btrfs top level, completed checkpoints are stored as:

@snapshots/<checkpoint-UUID>/root             read-only Btrfs snapshot
@snapshots/<checkpoint-UUID>/efi/             owned EFI backup
@snapshots/<checkpoint-UUID>/checkpoint.json  completed descriptor, published last

The descriptor fields are format, id, name, created, storage, source_root_uuid, snapshot_uuid and efi_manifest. format is integer 1; created is a UTC ISO timestamp; storage is the exact marker above. Each manifest entry records a relative path, kind (dir or file), numeric mode and mtime_ns; files additionally record size and SHA-256 sha256. Backup paths, bytes and metadata are rechecked before every restore. Links, hardlinks, special files, ambiguous FAT names and extra checkpoint files are refused. Metadata files must be regular, singly linked, owned by root and not group/world writable.

@snapshots/restore.json is the durable active transaction. It has exactly format, id (transaction UUID), checkpoint, checkpoint_digest, storage, old_uuid, new_uuid, prepared_root, failed_root and phase. The digest is SHA-256 of the descriptor's canonical JSON (sorted keys, compact separators). Phases are preparing, prepared, root-retained, root-installed and efi-restored. new_uuid starts null; it is recorded once the writable snapshot exists. Snapshot parent UUID and marker checks establish the source if power was lost before that UUID was journaled. Directory fsync and filesystem sync delimit transitions.

A writable copy is prepared as @restore-<transaction-UUID>. The previous root is retained as @failed-<transaction-UUID> before the prepared root becomes @root. Resume verifies the actual subvolume UUIDs at all three paths and accepts only the three valid rename arrangements. The recorded phase alone never decides which root to keep. After EFI restoration and verification, the transaction is archived as @snapshots/restored-<transaction-UUID>.json. No failed roots or checkpoints are automatically deleted.

Btrfs root replacement and FAT file copying are not one atomic operation. Power loss can leave the system temporarily unbootable. Rerun the same checkpoint ID from the live ISO with the same explicit confirmation; recovery validates its records and resumes. Do not manually rename or delete recovery roots or journal files. Unexplained UUID/path combinations are refused for offline inspection. An incomplete checkpoint without its completed descriptor is never restorable.

These snapshots remain on the original disk. They are not external backups, and there is no promise of online application consistency: checkpoint only while the installation is offline. The unit suite uses a persistent fake Btrfs backend and fresh processes to exercise journal and rename interruptions. Real Btrfs, FAT, UEFI, Portage and disk-only boot acceptance must be verified separately in a fresh VM before release; unit tests do not establish those claims.

Btrfs's subvolume documentation explains snapshot boundaries and read-only snapshots; the command checks the read-only property explicitly instead of inferring it from mount options.