Introduction
The Hetzner script installimage located on the Hetzner Rescue System provides an easy way to install various Linux distributions.
This tutorial shows how to use installimage to install an encrypted Debian 11
(Bullseye) operating system on a NVMe software RAID with LVM and add remote
unlocking via SSH (dropbear) in initramfs stored on a separate /boot
partition. The order of setup for the root partition is: RAID ➔ LUKS ➔ LVM ➔ File System. This tutorial was tested on an AX41-NVMe system with Debian
11.01 Bullseye.
Prerequisites
- Hetzner account
- Server with IPv4 address booted into the rescue system
- RSA or ECDSA/ED25519 SSH public key (can be created on the fly)
- No private networks attached on Hetzner Cloud
Step 1 - Create or copy SSH public key
In order to remotely unlock the encrypted disk at least one SSH key is
required. This key will also be used to later login to the booted system. The
dropbear SSH daemon (version 2020.81-3) included in Debian 11.01 supports
Rivest–Shamir–Adleman (RSA) and elliptic curve keys. If you do not have such a
key, you need to generate one. This can be the case if a password is used to
access the rescue system. In case you already have such a key, continue with
step 2.
For example, to generate ed25519 (a non National Institute of Standards and Technology (NIST) elliptic-curve) SSH key run:
ssh-keygen -t ed25519Copy the public key to the rescue system and add it to existing keys.
cat id_ed25519.pub|ssh root@<your-host> -T 'cat>>/root/.ssh/authorized_keys'Step 2 - Create or copy installimage configuration file
When installimage is called without any options, it will search for the file
/autosetup. If it does not find it, it starts in interactive mode and will
open an editor after a distribution image has been selected. On exiting the
editor, the installation will proceed and the corresponding configuration is
saved as /installimage.conf on the file system of the installed system. In
this tutorial we will use such a configuration to install directly via the
/autosetup file.
Create the file /autosetup with the following content or copy it to the
server running the rescue system.
Note: Replace <secret> with a secure password and adjust drive names and
partitioning as needed. Usually Linux Unified Key Setup (LUKS) via cryptsetup
is configured for a maximum password length of 512 characters (the maximum
length was not verified for this tutorial).
When installing in UEFI mode, you need to additionally add PART /boot/efi esp 256M.
CRYPTPASSWORD <secret>
SWRAID 1
SWRAIDLEVEL 1
DRIVE1 /dev/nvme0n1
DRIVE2 /dev/nvme1n1
PART /boot ext4 1G
PART lvm vg0 all crypt
LV vg0 swap swap swap 32G
LV vg0 root / xfs all
BOOTLOADER grub
HOSTNAME <host.example.com>
IMAGE /root/images/Debian-1107-bullseye-amd64-base.tar.gz
SSHKEYS_URL /root/.ssh/authorized_keysThis example configures 32G of swap and uses the xfs file system.
Other values or file systems, like ext4 are possible.
Step 3 - Create or copy post-install script
In order to remotely unlock the encrypted partition, we need to install and add
the dropbear SSH server to the initramfs which is stored on the unencrypted
/boot partition. This will also trigger the inclusion of dhclient to
configure networking, but without any extras. To enable support for Hetzner
Cloud, we need to add a hook which includes support for RFC3442 routes.
In order to run these additional steps we need a post-install script for
installimage.
Create a file /tmp/post-install.sh on the rescue system with the following
content:
#!/bin/bash
add_rfc3442_hook() {
cat << EOF > /etc/initramfs-tools/hooks/add-rfc3442-dhclient-hook
#!/bin/sh
PREREQ=""
prereqs()
{
echo "\$PREREQ"
}
case \$1 in
prereqs)
prereqs
exit 0
;;
esac
if [ ! -x /sbin/dhclient ]; then
exit 0
fi
. /usr/share/initramfs-tools/scripts/functions
. /usr/share/initramfs-tools/hook-functions
mkdir -p \$DESTDIR/etc/dhcp/dhclient-exit-hooks.d/
cp -a /etc/dhcp/dhclient-exit-hooks.d/rfc3442-classless-routes \$DESTDIR/etc/dhcp/dhclient-exit-hooks.d/
EOF
chmod +x /etc/initramfs-tools/hooks/add-rfc3442-dhclient-hook
}
# Install hook
add_rfc3442_hook
# Copy SSH keys for dropbear
mkdir -p /etc/dropbear-initramfs
cp -a /root/.ssh/authorized_keys /etc/dropbear-initramfs/authorized_keys
# Update system
apt-get update >/dev/null
apt-get -y install cryptsetup-initramfs dropbear-initramfsImportant note: make the post-install script executable:
chmod +x /tmp/post-install.shStep 4 - Start installation
Before starting the installation, check again the content of the following files:
/root/.ssh/authorized_keys- our public SSH key(s) (RSA or ECDSA/ed25519)/autosetup-installimageconfiguration/tmp/post-install.sh- is executable and contains the post-install script
Now we are ready to start the installation with the following command:
installimage -x /tmp/post-install.shWait until the installation completes and check the debug.txt for any errors.
Step 5 - Boot installed system
After the installation has finished and any errors are resolved (see section
debugging below in case), we can run reboot to restart the server and
boot the newly installed init RAM disk. We can watch the boot process if we
have a KVM attached or via remote console on a cloud instance.
After some time the server should respond to ping. Now login via SSH into busy
box init RAM disk using SSH dropbear and run the command cryptroot-unlock
to unlock the encrypted partition(s) from the busybox command line.
ssh -i /root/.ssh/id_ed25519 root@<your-host>
The authenticity of host '<your-host> (<your-host>)' can't be established.
ECDSA key fingerprint is SHA256:uY/i2eu8jee8RB9aWwapQquSO+G6E/VMorWQ1fA9clM.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '<your-host>' (ECDSA) to the list of known hosts.
To unlock root partition, and maybe others like swap, run `cryptroot-unlock`.
BusyBox v1.30.1 (Debian 1:1.30.1-6+b3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
~ # cryptroot-unlock
Please unlock disk luks-53152dbf-856e-4e13-b09f-8d05f01a43bc:
cryptsetup: luks-53152dbf-856e-4e13-b09f-8d05f01a43bc set up successfully
~ # Connection to <your-host> closed by remote host.
Connection to <your-host> closed.If the password is correct the boot from the init RAM disk will continue to the new installed system. Before the new system is up and running, we will automatically be disconnected from the temporary SSH session of the init RAM disk.
After a few seconds we can login to our new system via SSH. Be aware that if we
use the same name to connect to the init RAM disk of busybox via SSH
(dropbear) and to the booted system, SSH will give warnings about a changed
host name. It is advised to use different name. Either by using IP and name or
by setting up two different DNS names for the same IP. Buy doing this we keep
the SSH function of warning if the booted server host (Debian 11) finger print
changes.
Step 6 - Verification (Optional)
The correct installation can be verified by lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT
either from the rescue system or from the booted
system. The hierarchy of NVMe ➔ MD (RAID) ➔ LUKS ➔ VLM ➔ XFS is
clearly visible:
lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT
NAME FSTYPE SIZE MOUNTPOINT
nvme0n1 476.9G
├─nvme0n1p1 linux_raid_member 1G
│ └─md0 ext4 1022M /boot
└─nvme0n1p2 linux_raid_member 475.9G
└─md1 crypto_LUKS 475.8G
└─luks-53152dbf-856e-4e13-b09f-8d05f01a43bc LVM2_member 475.8G
├─vg0-swap swap 32G [SWAP]
└─vg0-root xfs 443.8G /
nvme1n1 476.9G
├─nvme1n1p1 linux_raid_member 1G
│ └─md0 ext4 1022M /boot
└─nvme1n1p2 linux_raid_member 475.9G
└─md1 crypto_LUKS 475.8G
└─luks-53152dbf-856e-4e13-b09f-8d05f01a43bc LVM2_member 475.8G
├─vg0-swap swap 32G [SWAP]
└─vg0-root xfs 443.8G /
Step 7 - Harden the system (Optional)
Writing the cryptsetup password inside the /autosetup file
is very convenient. A copy of this file is placed on the installed
root file system as /installimage.conf. And in /installimage.debug
we can find a line like Sent install.conf to statsserver: HTTP/2 201.
It is possible to update the crypsetup password with a different one manually
after the initial setup via /autosetup. The update procedure do not
require to store the password in a text file.
First test which slot the crypsetup password is stored. Usually it is slot 0.
This can be done with cryptsetup --verbose open --test-passphrase <DEVICE>.
cryptsetup --verbose open --test-passphrase /dev/md1
Enter passphrase for /dev/md1: <secret>
Key slot 0 unlocked.
Command successful.Usually this should give the same result, but will display other slots in the case more than one slot is used:
cryptsetup luksDump /dev/md1|grep luks2
0: luks2To change the crypsetup password, set the password for slot 0 with:
cryptsetup luksChangeKey /dev/md1 -S 0
Enter LUKS passphrase to be changed:
Enter new LUKS passphrase:Step 8 - Debugging (Optional)
The installimage is not idempotent in regard to software RAID. The devices
/dev/md0 and /dev/md1 are existent and if we would run the installimage
twice to reinstall the machine we will get errors like Device /dev/md/1 is in use. In this case remove and stop the software RAID before running
installimage again.
mdadm --remove /dev/md0
mdadm --remove /dev/md1
mdadm --stop /dev/md0
mdadm --stop /dev/md1Conclusion
After configuration and executing installimage, booting into the
initramfs, entering the cryptsetup password, the installed system is
booted with a two tier process and can be used.
While the boot or reboot of the system needs one additional step (the manual
SSH session to the busybox init RAM disk) it adds a layer of security towards
the storage subsystem. However an unattended reboot is not possible with this
dropbear setup.
Credits
This tutorial was inspired by How to install Ubuntu 20.04 with full disk encryption and parts (for example the cloud hook) had been copied under the MIT license at 2022-04-07.