Linux Rescue System

What Rescue is

Hetzner starts Rescue with PXE (network boot). Your NVMe or SATA drives sit idle. That is why Rescue can repair a broken system without writing to it. The environment is Debian-based. You can install extra tools with apt while Rescue is running.

Activate and boot

1. Open Robot → Servers → your server → Rescue.
2. Choose Linux, 64-bit. Add an SSH key if you have one. Activate.
3. Copy the temporary root password Robot shows you.
4. Reboot. If you still have SSH into the old OS:

root@server ~ # reboot

Connection to 203.0.113.10 closed by remote host.Connection to 203.0.113.10 closed.

That “closed by remote host” line is normal: the machine is restarting. If SSH is already dead, skip reboot and use Robot → Reset → hardware reset.

5. Wait about a minute, then SSH from your laptop:

you@laptop ~ % ssh root@YOUR.SERVER.IP# or, Rescue also listens on 222you@laptop ~ % ssh -p 222 root@YOUR.SERVER.IP

The authenticity of host '203.0.113.10 (203.0.113.10)' can't be established.ED25519 key fingerprint is SHA256:abCDefghIJKlmnopQRSTuvwxYZ0123456789abcd.Are you sure you want to continue connecting (yes/no/[fingerprint])? yesWarning: Permanently added '203.0.113.10' (ED25519) to the list of known hosts.root@203.0.113.10's password:Welcome to the Hetzner Rescue System.Hardware data:   CPU     : AMD EPYC 7502P 32-Core Processor   Memory  : 128 GB   Disk    : /dev/nvme0n1  512 GB   Disk    : /dev/nvme1n1  512 GBroot@rescue ~ #

Type yes for the host key, then the Rescue password from Robot. You should see Welcome to the Hetzner Rescue System and a prompt that says root@rescue, not your old hostname. If you still see root@Ubuntu-…, Rescue did not boot — activate it again and reset.

Look before you touch disks

Three commands, in this order. None of them change the disks. Read the sample, then match it to your screen.

lsblk — list block devices

lsblk = LiSt BLocK devices. A tree of disks, partitions, RAID, and LVM. In Rescue the MOUNTPOINT column is usually empty: nothing is mounted yet.

root@rescue ~ # lsblk

NAME            MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTnvme0n1         259:0    0 476.9G  0 disk├─nvme0n1p1     259:1    0   256M  0 part│ └─md0           9:0    0   255M  0 raid1├─nvme0n1p2     259:2    0     1G  0 part│ └─md1           9:1    0  1022M  0 raid1└─nvme0n1p3     259:3    0 475.7G  0 part  └─md2           9:2    0 475.6G  0 raid1    ├─vg0-root  253:0    0    50G  0 lvm    ├─vg0-swap  253:1    0     8G  0 lvm    └─vg0-var   253:2    0 417.6G  0 lvmnvme1n1         259:4    0 476.9G  0 disk├─nvme1n1p1     259:5    0   256M  0 part│ └─md0           9:0    0   255M  0 raid1├─nvme1n1p2     259:6    0     1G  0 part│ └─md1           9:1    0  1022M  0 raid1└─nvme1n1p3     259:7    0 475.7G  0 part  └─md2           9:2    0 475.6G  0 raid1    ├─vg0-root  253:0    0    50G  0 lvm    ├─vg0-swap  253:1    0     8G  0 lvm    └─vg0-var   253:2    0 417.6G  0 lvm

How to read it: two physical disks (nvme0n1, nvme1n1) each have three slices. Matching slices join as md0 / md1 / md2 (software RAID). The big array md2 is split by LVM into vg0-root, vg0-swap, vg0-var. Your root filesystem is almost always vg0-root here.

NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTsda       8:0    0 476.9G  0 disk├─sda1    8:1    0   256M  0 part├─sda2    8:2    0     1G  0 part└─sda3    8:3    0 475.7G  0 part

Here there is no md and no vg0. Root is usually the last big partition: /dev/sda3.

cat /proc/mdstat — RAID status

/proc/mdstat is a live kernel file: multiple-device RAID status. Rescue often auto-starts arrays so you can mount them.

root@rescue ~ # cat /proc/mdstat

Personalities : [raid1]md2 : active raid1 nvme0n1p3[0] nvme1n1p3[1]      498860032 blocks super 1.2 [2/2] [UU]md1 : active raid1 nvme0n1p2[0] nvme1n1p2[1]      1046528 blocks super 1.2 [2/2] [UU]md0 : active raid1 nvme0n1p1[0] nvme1n1p1[1]      262080 blocks super 1.2 [2/2] [UU]unused devices: <none>

Look at [2/2] [UU]: two devices expected, two present, both Up. [_U] or [U_] means one disk is missing. A progress bar under the array is a rebuild — wait, do not wipe yet.

Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] [raid10]unused devices: <none>

Only “Personalities” and “unused devices” means Linux knows how to do RAID, but no array is running. Match this with the plain sda / nvme0n1 tree from lsblk.

ls /dev/mapper/ — LVM names

LVM volumes show up as files under /dev/mapper/. control is always there (the LVM device mapper). Extra names like vg0-root are your volumes.

root@rescue ~ # ls /dev/mapper/

control  vg0-root  vg0-swap  vg0-var

control

If you only see control, mount a partition such as /dev/sda3, not a mapper path.

Where is / really?

The three cases below cover most Hetzner installimage boxes. They are a starting set, not every layout Linux can boot. Root is often a stack: partition, then RAID, then maybe encryption, then LVM, then the filesystem.

Layers stack. You will often see more than one of the rows above at once:

• Plain partition
• LVM on a partition
• RAID → LVM (typical Hetzner RAID1 + PART lvm)
• LUKS → LVM (you encrypted the PV yourself)
• RAID → LUKS → LVM
• Btrfs on LUKS

root@rescue ~ # lsblk -froot@rescue ~ # blkidroot@rescue ~ # cat /proc/mdstatroot@rescue ~ # pvs; vgs; lvsroot@rescue ~ # ls /dev/mapper/root@rescue ~ # findmnt /# only if the disk looks encrypted (crypto_LUKS in lsblk -f):root@rescue ~ # cryptsetup luksDump /dev/nvme0n1p3

NAME            FSTYPE            LABEL UUID  MOUNTPOINTnvme0n1p3       linux_raid_member└─md2           LVM2_member  └─vg0-root    ext4                    8f3a…  /dev/mapper/vg0-root: UUID="8f3a…" TYPE="ext4"md2 : active raid1 nvme0n1p3[0] nvme1n1p3[1]  [2/2] [UU]  LV   VG  Attr       LSize  root vg0 -wi-a----- 50.00gcontrol  vg0-root  vg0-swap  vg0-varTARGET SOURCE          FSTYPE/      none            rootfs   # Rescue itself — disks not mounted yet

findmnt / in Rescue shows Rescue’s own RAM root, not your OS, until you mount … /mnt. After the mount, findmnt /mnt should show ext4/xfs on vg0-root (or whichever device you picked).

Mount the installed system

Mounting is “plug this filesystem into a folder”. Rescue uses /mnt as that folder. Success prints nothing — you just get the prompt back.

root@rescue ~ # mount /dev/md2 /mnt

root@rescue ~ # mount /dev/mapper/vg0-root /mnt

root@rescue ~ # mount /dev/sda3 /mnt

root@rescue ~ # cryptsetup luksOpen /dev/nvme0n1p3 cryptrootroot@rescue ~ # mount /dev/mapper/cryptroot /mnt# Btrfs with Ubuntu-style @ subvolume:root@rescue ~ # mount -o subvol=@ /dev/sda2 /mnt

Enter passphrase for /dev/nvme0n1p3:root@rescue ~ #

root@rescue ~ # mount /dev/mapper/vg0-root /mntroot@rescue ~ #

Check it worked with ls /mnt. You should see the installed OS folders: bin, etc, home, usr.

root@rescue ~ # ls /mnt

bin   dev  home  lib64       media  opt   root  sbin  sys  usrboot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var

mount: /mnt: special device /dev/md2 does not exist.mount: /mnt: fsconfig() failed: /dev/sda3: Can't lookup blockdev.

If mount errors, go back to lsblk and pick the device that actually exists. Do not guess.

For a full chroot you also need boot, EFI, and pseudo filesystems. Hetzner Rescue includes a helper:

root@rescue ~ # chroot-prepare /mntroot@rescue ~ # chroot /mntroot@rescue ~ # passwdroot@rescue ~ # exit

root@rescue ~ # chroot-prepare /mntroot@rescue ~ # chroot /mntroot@Ubuntu-2404-noble-amd64-base / # passwdNew password:Retype new password:passwd: password updated successfullyroot@Ubuntu-2404-noble-amd64-base / # exitexitroot@rescue ~ #

After chroot /mnt the hostname in the prompt is the installed OS (here Ubuntu). passwd hides what you type — that is normal. exit drops you back to root@rescue.

Install or reinstall an OS

From Rescue, run Hetzner’s installer. It wipes the drives you select:

root@rescue ~ # installimage

Hetzner Online GmbH - installimage  Please choose the operating system to install    Ubuntu-2404-noble-amd64-base.tar.gz    Ubuntu-2204-jammy-amd64-base.tar.gz    Debian-12-bookworm-amd64-base.tar.gz    Debian-13-trixie-amd64-base.tar.gz    Archlinux-latest-amd64-base.tar.gz    ...  <OK>          <Cancel>

Arrow keys, Enter to pick an image. Next, Midnight Commander opens the config file. Edit drives, RAID, and PART lines, then save with F10 (or Esc then 0 if F10 prints a ~). When the installer finishes it prints something like:

=>  Done.=>=>  The system is now ready to boot.=>  Please reboot the server.=>root@rescue ~ #

Then reboot. Details and example configs: Ubuntu & Debian.

root@rescue ~ # mdadm --stop /dev/md*root@rescue ~ # wipefs -fa /dev/sd*root@rescue ~ # wipefs -fa /dev/nvme*n1

mdadm: stopped /dev/md2mdadm: stopped /dev/md1mdadm: stopped /dev/md0

mdadm: error opening /dev/md*: No such file or directory

“No such file” here only means no /dev/md… devices existed. Safe to continue if cat /proc/mdstat already showed unused devices.

/dev/nvme0n1: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54/dev/nvme0n1: 8 bytes were erased at offset 0x70c7ffe00 (gpt): 45 46 49 20 50 41 52 54/dev/nvme0n1: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa/dev/nvme0n1p1: 8 bytes were erased at offset 0x00000000 (vfat): .../dev/nvme0n1p3: 4 bytes were erased at offset 0x00000000 (linux_raid_member): fc 4e 2b a9

Each line is one old label being erased (GPT, FAT, RAID member). After this, lsblk still shows the disks, but they look empty — ready for installimage.

Hardware RAID first

If Robot or lspci shows a hardware RAID card, build the virtual disk in the controller BIOS / Hetzner tools before installimage. Software RAID (mdadm) is more common on auction servers with two plain NVMe or SATA drives.

root@rescue ~ # lspci | grep -i -E 'raid|megaraid|smartarray'

root@rescue ~ # lspci | grep -i -E 'raid|megaraid|smartarray'root@rescue ~ #

Empty output means: no RAID card. Use software RAID in installimage if you have two disks.

02:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)

If you see a line like that, configure the card first so Rescue and installimage see one virtual disk (often /dev/sda), not the raw members.