Introduction
Hetzner's Rescue System includes the installimage tool, which automates installing popular Linux distributions on dedicated servers.
This tutorial shows how to use installimage to install Debian with
full-disk encryption on two NVMe drives using a software RAID with LVM.
MD RAID ➔ LUKS ➔ LVM ➔ filesystemIn addition remote unlocking via SSH (dropbear) embedded in the initramfs,
with the initramfs stored on a separate unencrypted /boot partition is
explained. The steps were tested on an AX41-NVMe dedicated server with Debian
11.1 Bullseye and Debian 13.0 Trixie. It should work with Debian 12.12
Bookworm.
Prerequisites
- Hetzner account
- Server with IPv4 address booted into the rescue system
- RSA or ECDSA/ED25519 SSH public key (can be created on the fly)
- No private networks attached on Hetzner Cloud
Step 1 - Create or copy SSH public key
In order to remotely unlock the encrypted disk at least one SSH key is
required. This key will also be used to later log in to the booted system. The
dropbear SSH daemon (version 2020.81-3) included in Debian 11.1 supports
Rivest–Shamir–Adleman (RSA) and elliptic curve keys, so does 2025.88-2 in
Debian 13.0. If you do not have such a key, you need to generate one. This can
be the case if a password is used to access the rescue system. In case you
already have such a key in ~/.ssh/{authorized,robot_user}_keys, continue with
step 2. (The installimage script can use an additional robot_user_keys file
for installation keys.)
For example, to generate ed25519 (a non National Institute of Standards and
Technology (NIST) elliptic-curve) SSH key run:
ssh-keygen -t ed25519Copy the public key to the rescue system and add it to existing keys.
cat id_ed25519.pub|ssh root@<your-host> -T 'cat>>/root/.ssh/authorized_keys'
cat id_ed25519.pub|ssh root@<your-host> -T 'cat>>/root/.ssh/robot_user_keys'Step 2 - Create or copy installimage configuration file
When installimage is called without any options, it will search for the file
/autosetup. If it does not find it, it starts in interactive mode and will
open an editor after a distribution image has been selected. On exiting the
editor, the installation will proceed and the corresponding configuration is
saved as /installimage.conf on the filesystem of the installed system. In
this tutorial we will use such a configuration to install directly via the
/autosetup file.
Create the file /autosetup with the following content or copy it to the
server running the rescue system.
Note 1: Replace <secret> with a secure password and adjust drive names and
partitioning as needed. Usually Linux Unified Key Setup (LUKS) via cryptsetup
is configured for a maximum password length of 512 characters (the maximum
length was not verified for this tutorial).
Note 2: If installing in UEFI mode, you must add PART /boot/efi esp 256M.
This tutorial installed Debian 13 without UEFI. However this might be
considered for other OSes, different hardware or future version of Debian.
Note 3: To understand what image file names are available, see
https://download.hetzner.com/bootimages/ and use credentials provided via
e-mail or use the installimage script until you see the image name you would
like to use (press ESC and then choose 'exit' to quit). The usage and creation
of custom images is not covered in this guide. Be aware that the naming
convention for Debian versions is to omit the first zero. It is Debian 13.0
for example, however the corresponding Hetzner image is Debian-1300...
CRYPTPASSWORD <secret>
SWRAID 1
SWRAIDLEVEL 1
DRIVE1 /dev/nvme0n1
DRIVE2 /dev/nvme1n1
#PART /boot/efi esp 256M # Only needed in UEFI mode, see note 2 above
PART /boot ext4 2G
PART lvm vg0 all crypt
LV vg0 swap swap swap 32G
LV vg0 root / xfs all
BOOTLOADER grub
HOSTNAME <host.example.com>
# For custom images:
# IMAGEPATH
# IMAGE
# For Hetzner images:
# IMAGE /root/images/Debian-1107-bullseye-amd64-base.tar.gz
# IMAGE /root/images/Debian-1208-bookworm-amd64-base.tar.gz
IMAGE /root/images/Debian-1300-trixie-amd64-base.tar.gz
SSHKEYS_URL /root/.ssh/authorized_keysThis example configures 32GB of swap, 2GB for /boot and uses the xfs
filesystem (better for large files). Other values or filesystems, like ext4
are possible.
Step 3 - Create or copy post-install script
In order to remotely unlock the encrypted partition, we need to install and add
the dropbear SSH server to the initramfs which is stored on the unencrypted
/boot partition. This will also trigger the inclusion of dhclient to
configure networking, but without any extras. To enable support for Hetzner
Cloud, we need to add a hook which includes support for RFC3442 routes.
In order to run these additional steps we need a post-install script for
installimage.
Create a file /tmp/post-install.sh on the rescue system with the following
content:
#!/bin/bash
add_rfc3442_hook() {
cat << EOF > /etc/initramfs-tools/hooks/add-rfc3442-dhclient-hook
#!/bin/sh
PREREQ=""
prereqs()
{
echo "\$PREREQ"
}
case \$1 in
prereqs)
prereqs
exit 0
;;
esac
if [ ! -x /sbin/dhclient ]; then
exit 0
fi
. /usr/share/initramfs-tools/scripts/functions
. /usr/share/initramfs-tools/hook-functions
mkdir -p \$DESTDIR/etc/dhcp/dhclient-exit-hooks.d/
cp -a /etc/dhcp/dhclient-exit-hooks.d/rfc3442-classless-routes \$DESTDIR/etc/dhcp/dhclient-exit-hooks.d/
EOF
chmod +x /etc/initramfs-tools/hooks/add-rfc3442-dhclient-hook
}
# (1) Install hook
add_rfc3442_hook
# (2) Copy SSH keys for dropbear
# Detect Debian version
VERSION_ID=$(grep VERSION_ID /etc/os-release | sed -e 's/.*"\(.*\)"/\1/')
echo "Detected Debian $VERSION_ID"
case "$VERSION_ID" in
11|12)
echo "Copy authorized_keys to /etc/dropbear-initramfs/authorized_keys"
mkdir -p /etc/dropbear-initramfs
cp -a /root/.ssh/authorized_keys /etc/dropbear-initramfs/authorized_keys
;;
13)
echo "Copy authorized_keys to /etc/dropbear/initramfs/authorized_keys"
mkdir -p /etc/dropbear/initramfs
cp -a /root/.ssh/authorized_keys /etc/dropbear/initramfs/authorized_keys
;;
*)
echo "WARNING: Unknown Debian version ($VERSION_ID)!"
echo "(You will not be able to log in)"
exit 2
;;
esac
# (3) Update system
apt-get update >/dev/null
apt-get -y install cryptsetup-initramfs dropbear-initramfsImportant note: make the post-install script executable:
chmod +x /tmp/post-install.shStep 4 - Start installation
Before starting the installation, check again the content of the following files:
/root/.ssh/authorized_keys- our public SSH key(s) (RSA or ECDSA/ed25519)/autosetup-installimageconfiguration/tmp/post-install.sh- is executable and contains the post-install script
Now we are ready to start the installation with the following command:
installimage -x /tmp/post-install.shIf you want to capture parts of the visible output or measure the time, you can use this command instead:
time installimage -x /tmp/post-install.sh |tee -a /root/my-install.logWait until the installation completes and check the /root/debug.txt and the
screen output (or my-install.log) for any errors. For Debian 13 the
installation takes less than 2 minutes.
In the screen output the following 2 lines show a successful configuration, in this case Debian 13:
Detected Debian 13
Copy authorized_keys to /etc/dropbear/initramfs/authorized_keysStep 5 - Boot installed system
After the installation has finished and any errors are resolved (see section
debugging below in case), we can run reboot to restart the server and
boot the newly installed initramfs. We can watch the boot process if we
have a KVM attached or via remote console on a cloud instance.
After some time the server should respond to ping. Now log in via SSH into
BusyBox initramfs shell provided by dropbear and run the command
cryptroot-unlock to unlock the encrypted partition(s) from the BusyBox
command line.
ssh -i /root/.ssh/id_ed25519 root@<your-host>
The authenticity of host '<your-host> (<your-host>)' can't be established.
ECDSA key fingerprint is SHA256:uY/i2eu8jee8RB9aWwapQquSO+G6E/VMorWQ1fA9clM.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '<your-host>' (ECDSA) to the list of known hosts.
To unlock root partition, and maybe others like swap, run `cryptroot-unlock`.
BusyBox v1.37.0 (Debian 1:1.37.0-6+b3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
~ # cryptroot-unlock
Please unlock disk luks-53152dbf-856e-4e13-b09f-8d05f01a43bc:
cryptsetup: luks-53152dbf-856e-4e13-b09f-8d05f01a43bc set up successfully
~ # Connection to <your-host> closed by remote host.
Connection to <your-host> closed.If the password is correct the boot from the initramfs will continue to the
new installed system. Before the new system is up and running, we will
automatically be disconnected from the temporary SSH session of the initramfs.
After a few seconds we can log in to our new system via SSH into. Be aware that
if we use the same name to connect to the initramfs of BusyBox via SSH
(dropbear) and to the booted system, SSH will give warnings about a changed
host name. It is advised to use different names. Either by using IP and name or
by setting up two different DNS names for the same IP. By doing this we keep
the SSH function of warning if the booted server host (Debian) fingerprint
changes.
Step 6 - Verification (Optional)
The correct installation can be verified by lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT
either from the rescue system or from the booted
system. The hierarchy of NVMe ➔ MD (RAID) ➔ LUKS ➔ LVM ➔ XFS is
clearly visible:
lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT
NAME FSTYPE SIZE MOUNTPOINT
nvme0n1 476.9G
├─nvme0n1p1 linux_raid_member 2G
│ └─md0 ext4 2G /boot
└─nvme0n1p2 linux_raid_member 474.9G
└─md1 crypto_LUKS 474.8G
└─luks-53152dbf-856e-4e13-b09f-8d05f01a43bc LVM2_member 474.8G
├─vg0-swap swap 32G [SWAP]
└─vg0-root xfs 442.8G /
nvme1n1 476.9G
├─nvme1n1p1 linux_raid_member 2G
│ └─md0 ext4 2G /boot
└─nvme1n1p2 linux_raid_member 474.9G
└─md1 crypto_LUKS 474.8G
└─luks-53152dbf-856e-4e13-b09f-8d05f01a43bc LVM2_member 474.8G
├─vg0-swap swap 32G [SWAP]
└─vg0-root xfs 442.8G /
Step 7 - Harden the system (Optional)
Writing the cryptsetup password inside the /autosetup file is very
convenient. A copy of this file is placed on the installed root filesystem as
/installimage.conf. And not in /root/debug.txt but in /installimage.debug
we can find a line like Sent install.conf to statsserver: HTTP/2 201.
It is possible to update the cryptsetup password with a different one
manually after the initial setup via /autosetup. The update procedure do
not require to store the password in a text file.
First test which slot the cryptsetup password is stored. Usually it is slot 0.
This can be done with cryptsetup --verbose open --test-passphrase <DEVICE>.
cryptsetup --verbose open --test-passphrase /dev/md1
Enter passphrase for /dev/md1: <secret>
Key slot 0 unlocked.
Command successful.Usually this should give the same result, but will display other slots in the case more than one slot is used:
cryptsetup luksDump /dev/md1|grep luks2
0: luks2To change the cryptsetup password, set the password for slot 0 with:
cryptsetup luksChangeKey /dev/md1 -S 0
Enter LUKS passphrase to be changed:
Enter new LUKS passphrase:Step 8 - Debugging (Optional)
The installimage is not idempotent in regard to software RAID. The devices
/dev/md0 and /dev/md1 will exists after the first run. If we would run the
installimage twice to reinstall the machine we will get errors like Device /dev/md/1 is in use. In this case remove and stop the software RAID
before running installimage again. Also it might be advisable to delete
mdadm meta data. If the meta data is not deleted, the old metadata will
remain on the devices, but will appear older than the new metadata and so will
usually be ignored. The old metadata can be removed by giving the option
--zero-superblock. This was not tested for this tutorial. Note: Be
careful to call --zero-superblock with clustered RAID, make sure the array
isn't used or assembled in an other cluster node before execute it.
mdadm --remove /dev/md0
mdadm --remove /dev/md1
mdadm --stop /dev/md0
mdadm --stop /dev/md1Conclusion
After configuration and executing installimage, booting into the
initramfs, entering the cryptsetup password, the installed system is
booted with a two-tier process and can be used.
While the boot or reboot of the system needs one additional step (the manual
SSH session to the BusyBox initramfs) it adds a layer of security towards
the storage subsystem. However an unattended reboot is not possible with this
dropbear setup.
Credits
The tutorial for Debian 11
(install-debian-11-with-lvm-encrypted-nvme-software-raid) was inspired by
How to install Ubuntu 20.04 with full disk
encryption
and parts (for example the cloud hook) had been copied under the MIT license at
2022-04-07.
This tutorial for Debian in general, tested with Debian 13, and hopefully for
other future versions of Debian improves on the former Debian 11 tutorial with
various changes to wording, typos, some changes to installation (for example
adding robot_user_keys, changes of package names due to Debian 13) and minor
other configuration changes (for example using 2GB for /boot, to add more space
for kernels).