Introduction
This method lets you easily clone the exact file systems of each guest to the new server while they are running. There is no need to make a proxmox cluster.
The transfer is as fast as your network card can manage, no downtime is wasted creating dump files.
The initial transfer will be made while all guests are running, new files will be synced until it is time to do the switchover. At that time the guests are shutdown and the latest changes are synced over. Once everything is synced the guests can be started right away on the new server.
If you did everything correctly total downtime will be about 5-10 minutes.
Prerequisites
For Hetzner dedicated servers, if you want to transfer your IP subnet to the new server make sure to order it in the same data center and write in the comments that you want option 2: "Upgrade in parallel operation with transferred IPv4 subnet(s)". See also upgrades for dedicated servers.
Make sure that your server uses ZFS for guest storage. If the old server does not have ZFS you cannot use this tutorial unless there is an empty partition where you could make a ZFS pool.
Install Proxmox on the new server according to Proxmox documentation or this tutorial: Install and Configure Proxmox VE
When installing Debian only make a small 64GB software raid on each drive for the operating system and swap (you could of course also have swap outside raid but don't put it on ZFS). Later when the installation is finished use cfdisk to make a SOLARIS partition on the remaining space on each drive that you can create a ZFS mirror on.
The partition table would look something like this:
/dev/nvme0n1p1 512M fd Linux raid (MD raid for boot)
/dev/nvme0n1p2 64G fd Linux raid (MD raid for LVM with root+swap)
/dev/nvme0n1p3 889.4G bf Solaris (empty partition for ZFS mirror)On an EX51-NVME dedicated server you would use a command like:
zpool create -f -o ashift=12 tank mirror nvme0n1p3 nvme1n1p3Proxmox ZFS documentation: https://pve.proxmox.com/wiki/ZFS_on_Linux
PVE-zsync documentation: https://pve.proxmox.com/wiki/PVE-zsync
Step 1 - Preparing guests
Check that all of your guests only have ZFS hard drives and root disks.
If a guest has non-ZFS disks you can stop it and move the disk under the Resources or Hardware tab. It should start right up after the move is complete.
Step 2 - Preparing the servers
The ZFS storage of the new server needs to have the same ID in Proxmox web interface as on the old server. The Pool and filesystem name can be different if you like. Create a new ZFS storage on the new server for each one that exists on the old server.
Make sure the new server has a root password so you can connect remotely.
Install PVE-zsync on both servers:
apt-get install pve-zsync
If you want to use disk compression enable it on both servers before transfer:
zfs set compress=on tank
Step 3 - Starting transfers
- For each guest create a sync job on the old server, specify ip of new server not dns.
pve-zsync create --source 101 --dest <10.0.0.1>:tank/zfs-containers --verbose --maxsnap 7
pve-zsync create --source 102 --dest <10.0.0.1>:tank/zfs-vmstore --verbose --maxsnap 7
...- Edit /etc/cron.d/pve-zsync and change time from */15 to */3
This initial transfer will take some time, remove --verbose if you want to run it in background. New files will be synced automatically every 3 minutes.
Step 4 - Activate config and test guests on new server
- On the new server, add the IP subnet in proxmox for vmbr0 identically to the old server
- For each CT copy the config file:
cp /var/lib/pve-zsync/zfs-containers/101.conf.lxc.rep_default_... /etc/pve/lxc/101.conf - For each VM copy the config file:
cp /var/lib/pve-zsync/zfs-vmstore/102.conf.qemu.rep_default_... /etc/pve/qemu-server/102.confIf you like you can start each guest on the new server and login through web console to inspect everything is ok. Shut the guests down afterwards before going to the next step.
Step 5 - Switching over
- Send in a support request for the new server and ask to have the subnet transferred from your old server. Specify the subnet, server id and requested time (between 7:30 am and 5:30 pm CET weekdays).
- Log in to the old server with ssh and both servers with the web interface
- Start a continuous ping to one of the guests from your workstation
- As soon as ping timeouts the subnet has transferred, shutdown all the guests
- Write down the time the last guest finished shutting down
- Check that all guests have finished syncing:
root@oldserver# pve-zsync list
SOURCE NAME STATE LAST SYNC TYPE CON
100 default ok 2021-08-24_18:27:11 lxc ssh
101 default ok 2021-08-24_18:27:21 lxc ssh
102 default ok 2021-08-24_18:27:14 qemu sshLast Sync must be LATER than when you shutdown the last guest
- Edit /etc/cron.d/pve-zsync and comment out all lines with #
- Start guests on new server
- Disable autostart on old server
Conclusion
You have now managed to migrate all your guests to the new Proxmox server with just a few minutes of downtime. Since the IP subnet was transferred all services continue running without DNS changes.
As you have learned PVE-zsync is a very useful tool. If you have extra space on another production server, or an old decommissioned server you can have critical guests syncing continuously which makes it possible to quickly start up the spare server in case of a failure in the main server.