How to install Debian 11 encrypted with LVM on a NVMe software RAID

Introduction

The Hetzner script installimage located on the Hetzner Rescue System provides an easy way to install various Linux distributions.

This tutorial shows how to use installimage to install an encrypted Debian 11 (Bullseye) operating system on a NVMe software RAID with LVM and add remote unlocking via SSH (dropbear) in initramfs stored on a separate /boot partition. The order of setup for the root partition is: RAID ➔ LUKS ➔ LVM ➔ File System. This tutorial was tested on an AX41-NVMe system with Debian 11.01 Bullseye.

Prerequisites

• Hetzner account
• Server with IPv4 address booted into the rescue system
• RSA or ECDSA/ED25519 SSH public key (can be created on the fly)
• No private networks attached on Hetzner Cloud

Step 1 - Create or copy SSH public key

In order to remotely unlock the encrypted disk at least one SSH key is required. This key will also be used to later login to the booted system. The dropbear SSH daemon (version 2020.81-3) included in Debian 11.01 supports Rivest–Shamir–Adleman (RSA) and elliptic curve keys. If you do not have such a key, you need to generate one. This can be the case if a password is used to access the rescue system. In case you already have such a key, continue with step 2.

For example, to generate ed25519 (a non National Institute of Standards and Technology (NIST) elliptic-curve) SSH key run:

ssh-keygen -t ed25519

Copy the public key to the rescue system and add it to existing keys.

cat id_ed25519.pub|ssh root@<your-host> -T 'cat>>/root/.ssh/authorized_keys'

Step 2 - Create or copy installimage configuration file

When installimage is called without any options, it will search for the file /autosetup. If it does not find it, it starts in interactive mode and will open an editor after a distribution image has been selected. On exiting the editor, the installation will proceed and the corresponding configuration is saved as /installimage.conf on the file system of the installed system. In this tutorial we will use such a configuration to install directly via the /autosetup file.

Create the file /autosetup with the following content or copy it to the server running the rescue system.

Note: Replace <secret> with a secure password and adjust drive names and partitioning as needed. Usually Linux Unified Key Setup (LUKS) via cryptsetup is configured for a maximum password length of 512 characters (the maximum length was not verified for this tutorial). When installing in UEFI mode, you need to additionally add PART /boot/efi esp 256M.

CRYPTPASSWORD <secret>

SWRAID 1
SWRAIDLEVEL 1
DRIVE1 /dev/nvme0n1
DRIVE2 /dev/nvme1n1

#PART  /boot/efi  esp    256M       #Only needed in UEFI mode
PART  /boot  ext4    1G
PART  lvm    vg0    all crypt

LV  vg0  swap  swap  swap 32G
LV  vg0  root  /     xfs  all

BOOTLOADER grub

HOSTNAME <host.example.com>

IMAGE /root/images/Debian-1107-bullseye-amd64-base.tar.gz

SSHKEYS_URL /root/.ssh/authorized_keys

This example configures 32G of swap and uses the xfs file system. Other values or file systems, like ext4 are possible.

Step 3 - Create or copy post-install script

In order to remotely unlock the encrypted partition, we need to install and add the dropbear SSH server to the initramfs which is stored on the unencrypted /boot partition. This will also trigger the inclusion of dhclient to configure networking, but without any extras. To enable support for Hetzner Cloud, we need to add a hook which includes support for RFC3442 routes.

In order to run these additional steps we need a post-install script for installimage.

Create a file /tmp/post-install.sh on the rescue system with the following content:

#!/bin/bash

add_rfc3442_hook() {
  cat << EOF > /etc/initramfs-tools/hooks/add-rfc3442-dhclient-hook
#!/bin/sh

PREREQ=""

prereqs()
{
        echo "\$PREREQ"
}

case \$1 in
prereqs)
        prereqs
        exit 0
        ;;
esac

if [ ! -x /sbin/dhclient ]; then
        exit 0
fi

. /usr/share/initramfs-tools/scripts/functions
. /usr/share/initramfs-tools/hook-functions

mkdir -p \$DESTDIR/etc/dhcp/dhclient-exit-hooks.d/
cp -a /etc/dhcp/dhclient-exit-hooks.d/rfc3442-classless-routes \$DESTDIR/etc/dhcp/dhclient-exit-hooks.d/
EOF

  chmod +x /etc/initramfs-tools/hooks/add-rfc3442-dhclient-hook
}

# Install hook
add_rfc3442_hook

# Copy SSH keys for dropbear
mkdir -p /etc/dropbear-initramfs
cp -a /root/.ssh/authorized_keys /etc/dropbear-initramfs/authorized_keys

# Update system
apt-get update >/dev/null
apt-get -y install cryptsetup-initramfs dropbear-initramfs

Important note: make the post-install script executable:

chmod +x /tmp/post-install.sh

Step 4 - Start installation

Before starting the installation, check again the content of the following files:

• /root/.ssh/authorized_keys - our public SSH key(s) (RSA or ECDSA/ed25519)
• /autosetup - installimage configuration
• /tmp/post-install.sh - is executable and contains the post-install script

Now we are ready to start the installation with the following command:

installimage -x /tmp/post-install.sh

Wait until the installation completes and check the debug.txt for any errors.

Step 5 - Boot installed system

After the installation has finished and any errors are resolved (see section debugging below in case), we can run reboot to restart the server and boot the newly installed init RAM disk. We can watch the boot process if we have a KVM attached or via remote console on a cloud instance.

After some time the server should respond to ping. Now login via SSH into busy box init RAM disk using SSH dropbear and run the command cryptroot-unlock to unlock the encrypted partition(s) from the busybox command line.

ssh -i /root/.ssh/id_ed25519 root@<your-host>

The authenticity of host '<your-host> (<your-host>)' can't be established.
ECDSA key fingerprint is SHA256:uY/i2eu8jee8RB9aWwapQquSO+G6E/VMorWQ1fA9clM.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '<your-host>' (ECDSA) to the list of known hosts.
To unlock root partition, and maybe others like swap, run `cryptroot-unlock`.


BusyBox v1.30.1 (Debian 1:1.30.1-6+b3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

~ # cryptroot-unlock
Please unlock disk luks-53152dbf-856e-4e13-b09f-8d05f01a43bc:
cryptsetup: luks-53152dbf-856e-4e13-b09f-8d05f01a43bc set up successfully
~ # Connection to <your-host> closed by remote host.
Connection to <your-host> closed.

If the password is correct the boot from the init RAM disk will continue to the new installed system. Before the new system is up and running, we will automatically be disconnected from the temporary SSH session of the init RAM disk.

After a few seconds we can login to our new system via SSH. Be aware that if we use the same name to connect to the init RAM disk of busybox via SSH (dropbear) and to the booted system, SSH will give warnings about a changed host name. It is advised to use different name. Either by using IP and name or by setting up two different DNS names for the same IP. Buy doing this we keep the SSH function of warning if the booted server host (Debian 11) finger print changes.

Step 6 - Verification (Optional)

The correct installation can be verified by lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT either from the rescue system or from the booted system. The hierarchy of NVMe ➔ MD (RAID) ➔ LUKS ➔ VLM ➔ XFS is clearly visible:

lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT
NAME                                            FSTYPE          SIZE MOUNTPOINT
nvme0n1                                                           476.9G
├─nvme0n1p1                                     linux_raid_member     1G
│ └─md0                                         ext4               1022M /boot
└─nvme0n1p2                                     linux_raid_member 475.9G
  └─md1                                         crypto_LUKS       475.8G
    └─luks-53152dbf-856e-4e13-b09f-8d05f01a43bc LVM2_member       475.8G
      ├─vg0-swap                                swap                 32G [SWAP]
      └─vg0-root                                xfs               443.8G /
nvme1n1                                                           476.9G
├─nvme1n1p1                                     linux_raid_member     1G
│ └─md0                                         ext4               1022M /boot
└─nvme1n1p2                                     linux_raid_member 475.9G
  └─md1                                         crypto_LUKS       475.8G
    └─luks-53152dbf-856e-4e13-b09f-8d05f01a43bc LVM2_member       475.8G
      ├─vg0-swap                                swap                 32G [SWAP]
      └─vg0-root                                xfs               443.8G /

Step 7 - Harden the system (Optional)

Writing the cryptsetup password inside the /autosetup file is very convenient. A copy of this file is placed on the installed root file system as /installimage.conf. And in /installimage.debug we can find a line like Sent install.conf to statsserver: HTTP/2 201.

It is possible to update the crypsetup password with a different one manually after the initial setup via /autosetup. The update procedure do not require to store the password in a text file.

First test which slot the crypsetup password is stored. Usually it is slot 0. This can be done with cryptsetup --verbose open --test-passphrase <DEVICE>.

cryptsetup --verbose open --test-passphrase /dev/md1
Enter passphrase for /dev/md1: <secret>
Key slot 0 unlocked.
Command successful.

Usually this should give the same result, but will display other slots in the case more than one slot is used:

cryptsetup luksDump /dev/md1|grep luks2
  0: luks2

To change the crypsetup password, set the password for slot 0 with:

cryptsetup luksChangeKey /dev/md1 -S 0
Enter LUKS passphrase to be changed:
Enter new LUKS passphrase:

Step 8 - Debugging (Optional)

The installimage is not idempotent in regard to software RAID. The devices /dev/md0 and /dev/md1 are existent and if we would run the installimage twice to reinstall the machine we will get errors like Device /dev/md/1 is in use. In this case remove and stop the software RAID before running installimage again.

mdadm --remove /dev/md0
mdadm --remove /dev/md1
mdadm --stop /dev/md0
mdadm --stop /dev/md1

Conclusion

After configuration and executing installimage, booting into the initramfs, entering the cryptsetup password, the installed system is booted with a two tier process and can be used.

While the boot or reboot of the system needs one additional step (the manual SSH session to the busybox init RAM disk) it adds a layer of security towards the storage subsystem. However an unattended reboot is not possible with this dropbear setup.

Credits

This tutorial was inspired by How to install Ubuntu 20.04 with full disk encryption and parts (for example the cloud hook) had been copied under the MIT license at 2022-04-07.

License: MIT