Introduction
Are you looking to amplify your infrastructure's capabilities with the power of Hetzner's dedicated servers? This tutorial will guide you through the process of creating a bare metal Kubernetes cluster with Cluster API Provider Hetzner (CAPH).
This article is the second in a series about Managing Kubernetes on Hetzner with Cluster API (CAPI), and will focus on using dedicated servers also known as bare metal servers, which can be up to 40% cheaper than Hetzner cloud instances.
The Cluster API Provider Hetzner (CAPH) is an open-source project (maintained by Syself GmbH and the community) that allows you to leverage the capabilities of Cluster API to manage highly-available Kubernetes clusters on both Hetzner baremetal servers (Robot) and Hetzner cloud instances.
The Cluster API approach allows you to manage multiple Kubernetes clusters in a scalable and declarative way. Unlike traditional infrastructure-as-code tools like Terraform and Ansible, CAPI brings powerful features such as automatic recovery from failures and smart automation of updates, making it easier to keep your systems running smoothly and reliably. The software will look after your infrastructure resources continuously, and ensure it stays healthy all the time, not only when a pipeline is triggered.
The CAPH project has reached general availability with its v1.0 release in October 2024, after four years of development and production use by hundreds of organizations.
This tutorial will guide you through the process of creating a bare metal Kubernetes cluster with CAPH. If you are coming from the first article and already have a running setup, you can skip this part.
Why Bare Metal?
Bare metal servers offer several benefits when compared to virtual machines, including increased computing performance, up to 40% lower costs than similar VMs on Hetzner, faster NVMe disks, and improved cost-efficiency for hosting databases and other disk-intensive workloads.
You are skipping the virtualization layer, and thus have direct access to the hardware. This not only allows you to make a more efficient use of resources, but also ensures you are the sole user of that machine, eliminating the need for confidential computing in some cases.
However, Hetzner bare metal servers lack the extensive API integrations that cloud VMs offer and can be considered somewhat clunky, with more complex provisioning processes and a less refined interface, making them more challenging to use.
But CAPH solves these issues, and the automation built around the bare metal servers turns them into first-class citizens in your Kubernetes clusters. This allows you to benefit from the advantages of these servers, while retaining the ease of use.
Prerequisites
- Docker, for running containers
- Kind, to create a local Kubernetes cluster
- kubectl and clusterctl, to access and manage your clusters
- A Hetzner account, with bare metal servers
- An SSH key
Step 1 - Prepare your Hetzner Account
Create a new project in the Hetzner Cloud Console, go to the "Security" tab and create an API token with read and write access. Note it down.
Next, add your public SSH key to the project. Note down the name you used for it, this will be used later as an environment variable.
For CAPH to manage the bare metal servers, you'll need to create a new Hetzner Robot web service user. For that, you can click on the "Create User" button in the Hetzner Robot console. The user ID will be provided to you via email.
Step 2 - Create a management cluster
A Kubernetes cluster is needed to run the Cluster API and CAPH stack. It will act as a management cluster, where you can manage your Kubernetes infrastructure using Kubernetes API objects. In this way, the controllers will handle the entire lifecycle of the machines and infrastructure.
We will start with a local Kind cluster to serve as a temporary bootstrap cluster. Later, we will be able to run the controllers on the new workload cluster in Hetzner Cloud, and move our resources there. If you already have a running cluster, feel free to use it instead.
Create a local Kind (Kubernetes in Docker) cluster:
# Create a cluster with Kind
kind create cluster --name caph-mgt-cluster
# Initialize it
clusterctl init --core cluster-api --bootstrap kubeadm --control-plane kubeadm --infrastructure hetznerWe need to create a secret with the access data for the provider integration to communicate with the Hetzner APIs (HCloud API and Robot API), and the SSH key that will be added to the provisioned servers and used by CAPH to access them. There are also some cluster definitions that we'll store as variables for now:
export HCLOUD_TOKEN="<YOUR-TOKEN>" # API token with access to the Hetzner project
export HETZNER_ROBOT_USER="<YOUR-ROBOT-USER>" # Username of the newly created user in Hetzner Robot
export HETZNER_ROBOT_PASSWORD="<YOUR-ROBOT-PASSWORD>" # Password set for the user above
export SSH_KEY_NAME="<YOUR-SSH-KEY-NAME>" # Name of the SSH key you added to the project in step 1
export HETZNER_SSH_PUB_PATH="<YOUR-SSH-PUBLIC-PATH>" # Path to the public SSH key file in your machine
export HETZNER_SSH_PRIV_PATH="<YOUR-SSH-PRIVATE-PATH>" # Path the the private SSH key file in your machine
export HCLOUD_CONTROL_PLANE_MACHINE_TYPE="" # Hetzner machine type for your controlplanes (VMs)
export HCLOUD_REGION="" # Hetzner cloud region where your control plane VMs will be created
export HCLOUD_WORKER_MACHINE_TYPE="" # Hetzner machine type for cloud workers (these are in addition to the bare metal machines)
export KUBERNETES_VERSION="1.30.5" # Kubernetes version used for your clusterSave secrets:
kubectl create secret generic hetzner --from-literal=hcloud=$HCLOUD_TOKEN --from-literal=robot-user=$HETZNER_ROBOT_USER --from-literal=robot-password=$HETZNER_ROBOT_PASSWORD
kubectl create secret generic robot-ssh --from-literal=sshkey-name=$SSH_KEY_NAME \
--from-file=ssh-privatekey=$HETZNER_SSH_PRIV_PATH \
--from-file=ssh-publickey=$HETZNER_SSH_PUB_PATHIf you already have the secret, you can edit it and add the new variables.
Step 2.1 Understanding Cluster API and CAPH Custom Resources
Now that you have a working management cluster, it's a good idea to understand how the operators you just deployed manage Kubernetes clusters and their components.
Cluster API manages infrastructure just like Kubernetes manages containers. In Kubernetes, you have Deployments, that creates ReplicaSets for Pods. In Cluster API, you have a MachineDeployment, which creates MachineSets, that in turn are responsible for a Machine.
However, since CAPI is provider-agnostic, you also need a component responsible for managing the provider-specific resources. In the case of Hetzner, this is the Cluster API Provider Hetzner project. And CAPH also has its Custom Resources.
Each CAPI Machine is tied to a HCloudMachine, in the case of a cloud VM; or a HetznerBareMetalMachine in the case of dedicated servers.
Cloud VMs can be provisioned via the API, but this is not the case for dedicated servers. To manage dedicated servers, you need a HetznerBareMetalHost resource, which represents the inventory of bare metal servers available in your account.
These are not all resources CAPI and CAPH have, but they are the most important ones. If you want to learn more, please refer to the Cluster API and CAPH official documentations.
Step 3 - Register your servers
As briefly explained in the previous step, the Cluster API Provider Hetzner uses an inventory approach, where you register your bare metal servers as hosts, making them available for clusters.
When a cluster requests bare metal machines, they are selected from this inventory.
Apply the HetznerBareMetalHost resource, replacing the spec.serverID with the ID of your bare metal machine, and spec.rootDeviceHints.wwn with your machine disk WWN.
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HetznerBareMetalHost
metadata:
name: baremetal-1
annotations:
capi.syself.com/wipe-disk: all
spec:
description: My first bare metal machine
serverID: 1234567
rootDeviceHints:
wwn: eui.5367492f3d453310
The annotation is used to wipe the disks of your bare metal server. This is recommended since most servers bought from Hetzner come with RAID enabled, and we won't use it in this guide.
If you don't know the WWN of the disk, you can omit the entire rootDeviceHints block, and retrieve it from the resource status once the host is picked for provisioning.
kubectl get hbmh baremetal-1 -o yaml | yq .spec.status.hardwareDetails.storageThe WWN can come in various formats, so if you see something different from the example, don’t worry.
Step 4 - Create the workload cluster
Now that our bare metal servers are registered, we can create a cluster using them.
Run the command below in your management cluster:
# Generate the manifests defining a workload cluster, and apply them to the management cluster
clusterctl generate cluster my-cluster --flavor hetzner-hcloud-control-planes | kubectl apply -f -This will generate and apply all the necessary resources to create a cluster using the hetzner-hcloud-control-planes flavor, that uses virtual machines for the control planes and bare metal servers for the workers.
A flavor is a bundle of Custom Resource manifests with preset configuration. It contains the necessary resources for creating a cluster like the MachineDeployments, plus additional manifests depending on the flavor like the ones needed for bare metal nodes. They serve as starting points for cluster configuration in CAPH.
For a full list of flavors, you can refer to the CAPH documentation.
Note: These flavors are meant only for demonstration purposes. For production use, it is recommended that you craft your own cluster configuration, optimized for your needs. For example, hosting the control planes in the same data center as your dedicated instances and set up load balancing.
After applying your resources, you can retrieve the workload cluster's kubeconfig:
# Get the kubeconfig for the new cluster
clusterctl get kubeconfig my-cluster > hetzner-cluster-kubeconfig.yamlStep 5 - Install components in your cluster
Your newly created cluster needs a few key components before it gets usable. These are a Container Network Interface (CNI), responsible for networking capabilities, and a Cloud Controller Manager (CCM), which allows you to properly use Hetzner resources such as Load Balancers and initializing your nodes.
export KUBECONFIG=hetzner-cluster-kubeconfig.yaml
# Install Hetzner CCM
helm repo add syself https://charts.syself.com/
helm repo update syself
helm install ccm syself/ccm-hetzner -n kube-system
# Install Flannel CNI - You can use your preferred CNI instead, e.g. Cilium
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.ymlIn this example we used Flannel, but you can use any CNI you want, like Cilium or Calico.
Now edit the deployment hcloud-cloud-controller-manager:
kubectl edit deployment ccm-ccm-hetzner -n kube-systemYour text editor will open. On Linux, this is vim by default. Hit i to enter vim "insert mode". Then, move down to the container templates env section, and add the items below:
- name: HCLOUD_TOKEN
valueFrom:
secretKeyRef:
key: hcloud
name: hetzner
- name: ROBOT_USER
valueFrom:
secretKeyRef:
key: robot-user
name: hetzner
- name: ROBOT_PASSWORD
valueFrom:
secretKeyRef:
key: robot-password
name: hetzner
When you're done, hit esc to exit "insert mode" and enter :wq to save your changes and exit.
Now, edit the machine deployment for the bare metal machines:
kubectl edit machinedeployment my-cluster-md-1Change the spec.replicas field to 1. Wait a few minutes for the machine to join the cluster.
And that's it! You now have a working cluster with Hetzner bare metal machines as worker nodes.
If you want to delete the cluster, you can run the command below:
kubectl delete cluster my-clusterThis will delete the cluster and the cloud instances used as the control planes. The bare metal machines will be disassociated from the cluster, but they will still be in your account. If you want to stop paying for them, you'll need to delete the servers in the Hetzner Robot console.
Next steps
A common way to make the most out of bare metal is to combine them with virtual machines in a mixed setup with autoscaling. The bare metal servers can handle the base load, while VMs are used to scale during peaks. For that you'll need Metrics Server.
This cluster was created with the default kubeadm bootstrap and controlplane providers, but for production use you may want to add additional layers to this configuration and create your own node images, as the default configuration provides only the basics to have a running cluster.
Other parts of the infrastructure should also be configured, such as load balancing, fault-tolerancy and network security (firewall, policies, etc), and operating system hardening. If you plan to use storage, you will also need a CSI (Container Storage Interface).
For more information on which aspects are handled by CAPH, you can check the project's GitHub readme and documentation.
Conclusion
With Cluster API Provider Hetzner, you can leverage Hetzner dedicated servers as nodes in your Kubernetes clusters with automated provisioning. This enables you to have a more efficient setup, while retaining benefits commonly associated with cloud VMs.
In this tutorial, you created your own Kubernetes cluster on Hetzner powered by bare metal, with high availability. A sample cluster configuration was used to simplify the process.