Installing Kubernetes using Flannel, Ingress Nginx controller and the Hetzner Cloud Load Balancer

Introduction

The tutorial contains instructions for installing a Kubernetes cluster version 1.30 using kubeadm and Helm. All cluster nodes will be located within a single Hetzner Network. Requests from the Internet to the cluster will be made through the Hetzner Load Balancer. This configuration allows you to close cluster nodes from external connections with a firewall. It makes your cluster more secure. By following this tutorial, you will install Flannel as CNI (Container Network Interface) and CRI-O as CRI (Container Runtime Interface).

CRI-O was chosen due to the following advantages:

• CRI-O is more secure than containerd due to its limited set of features and internal components, which reduces the attack surface
• CRI-O has a clear component structure that is published
• CRI-O has an active community

By installing the components step by step, you will assemble the cluster as a constructor and learn how to manage the Hetzner Load Balancer using the annotations specified in the values ​​file for the Ingress Controller.

Prerequisites

• 3 virtual machines in the Hetzner Cloud (1 for Control Plane, 2 for Workers)
  • With the Ubuntu 24.04 operating system
  • Connected to the same Hetzner Network
  • With root access
• Helm installed on one of the Control Plane virtual machines
• Hetzner Cloud API token in the Hetzner Cloud Console
• TLS/SSL certificate imported into the Hetzner Cloud Console:

  https://console.hetzner.cloud/projects/<project_id>/security/certificates

Step 1 - Configure repositories

In this step we will add the Kubernetes and CRI-O repositories on all cluster nodes.

Setting the Kubernetes version as an environment variable

KUBERNETES_VERSION=v1.30

Adding Kubernetes key and repository

curl -fsSL https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list

Adding CRI-O key and repository

curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/prerelease:/main/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/prerelease:/main/deb/ /" | tee /etc/apt/sources.list.d/cri-o.list

Step 2 - Install packages

The next step is to install the CNI packages and Kubernetes components on all nodes in the cluster.

Updating list of available packages

apt update

Installing packages

apt install -y cri-o kubelet kubeadm kubectl
crio --version && kubelet --version && kubeadm version && kubectl version --client

Disabling automatic updates for installed packages

apt-mark hold kubelet kubeadm kubectl

Step 3 - Operation System configuration

Kubernetes requires some preliminary operating system settings. If you are using the Ubuntu 24.04 image provided by Hetzner, then you do not need to disable SWAP, just run the commands below on all nodes in the cluster.

Enabling br_netfilter kernel module

echo "br_netfilter" >> /etc/modules-load.d/modules.conf

Enabling IP forward. It allows packets to be routed between different networks, enabling communication between subnets or acting as a gateway

sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g' /etc/sysctl.conf

Rebooting node to apply changes

reboot

Step 4 - Initializate cluster

Cluster initialization is performed on the future Control Plane node. In our case, only 1 Control Plane node is assumed. To ensure fault tolerance in a production environment, you must use at least 3.

Getting the default init configuration file

kubeadm config print init-defaults > InitConfiguration.yaml

Change or add the following values inside the file:

• bootstrapTokens.token: abcdef.0123456789abcdef - you can get a bootstrap token by running the command kubeadm token generate
• localAPIEndpoint.advertiseAddress: 10.0.0.2 - this configuration object lets you customize what IP/DNS name and port the local API server advertises it's accessible on. By default, kubeadm tries to auto-detect the IP of the default interface and use that, but in case that process fails you may set the desired value here. Set the IP address of your Control Plane node in the Hetzner Network here
• nodeRegistration.criSocket: unix:///var/run/crio/crio.sock - path to CRI-O socket file
• nodeRegistration.kubeletExtraArgs.cloud-provider: external - set to external for running with an external cloud provider
• nodeRegistration.kubeletExtraArgs.node-ip: 10.0.0.2 - IP address (or comma-separated dual-stack IP addresses) of the node. Set the IP address of your Control Plane node in the Hetzner Network here
• nodeRegistration.name: your_host - the name of the Control Plane node
• apiServer.certSANs: ['your_host.example.com', '10.0.0.2'] - Subject Alternative Names for API certificate. It can be both, FQDN and local IP address of your Control Pane node
• networking.podSubnet: 10.244.0.0/16 - the subnet used by Pods

Initializing the cluster

kubeadm init --config InitConfiguration.yaml

Step 5 - Join nodes into the cluster

Now we can join nodes into the cluster. Run the commands below on each worker node.

Getting default join configuration file

kubeadm config print join-defaults > JoinConfiguration.yaml

Change or add the following values inside the file:

• discovery.bootstrapToken.apiServerEndpoint: your_host.example.com:6443 - set your Control Plane node FQDN as the API endpoint
• discovery.bootstrapToken.token: abcdef.0123456789abcdef - the same like in InitConfiguration.yaml file
• discovery.tlsBootstrapToken: abcdef.0123456789abcdef - the same like in InitConfiguration.yaml file
• nodeRegistration.criSocket: unix:///var/run/crio/crio.sock - path to CRI-O socket file
• nodeRegistration.kubeletExtraArgs.cloud-provider: external - Set to external for running with an external cloud provider
• nodeRegistration.kubeletExtraArgs.node-ip: 10.0.0.3 - IP address (or comma-separated dual-stack IP addresses) of the node. Set the IP address of your Worker node in the Hetzner Network
• nodeRegistration.name: your_host - the name of the Worker node

Joining the node to the cluster

kubeadm join --config JoinConfiguration.yaml

Assign the worker role to worker nodes by adding the corresponding label. Run these commands on the Control Plane node

In the commands below, replace k8s-test-worker-1 and k8s-test-worker-2 with the names of your worker nodes.

mkdir -p $HOME/.kube
cp /etc/kubernetes/admin.conf $HOME/.kube/config
kubectl get nodes
kubectl label node <first worker node name>  node-role.kubernetes.io/worker=worker
kubectl label node <second worker node name>  node-role.kubernetes.io/worker=worker

Step 6 - Install CNI plugin

This guide uses Flannel as the CNI. If you want to use network policies to provide a higher level of security, you should consider other plugins.

Creating a namespace for Flannel

kubectl create ns kube-flannel

And adding privileges there. The privileged policy has no restrictions. All Pods in the kube-flannel namespace will be able to bypass typical container isolation mechanisms. For example, they will be able to access the node's network

kubectl label --overwrite ns kube-flannel pod-security.kubernetes.io/enforce=privileged

Adding helm repository

helm repo add flannel https://flannel-io.github.io/flannel/

You can get the values.yaml file from the kube-flannel repository and set the value of podCidr to the same value as specified for networking.podSubnet in the InitConfiguration.yaml file.

Installing Flannel into the previously created namespace

helm install flannel flannel/flannel --values values.yaml -n kube-flannel

Components that specify cloud-provider to external will add a taint node.cloudprovider.kubernetes.io/uninitialized with an effect NoSchedule during initialization. This marks the node as needing a second initialization from an external controller before it can be scheduled work. Note that in the event that a cloud controller manager is not available, new nodes in the cluster will be left unschedulable. That why we have to path CoreDNS after flannel installation to evade problems with CoreDNS Pods initialization

kubectl -n kube-system patch deployment coredns --type json -p '[{"op":"add","path":"/spec/template/spec/tolerations/-","value":{"key":"node.cloudprovider.kubernetes.io/uninitialized","value":"true","effect":"NoSchedule"}}]'

Step 7 - Install Hetzner Cloud Controller

We need to install the Hetzner Cloud Controller manager to integrate our Kubernetes cluster with the Hetzner Cloud API. This will allow us to use the functions provided by Hetzner. We will be mainly interested in Managed Load Balancer.

Creating a Kubernetes secret resource that contains your Hetzner Cloud API token (see the Prerequisites in the Introduction) and Hetzner Network ID. Hetzner Network ID can be extracted from the last digits of your network's URL in the Hetzner Cloud Console. For example, for the URL https://console.hetzner.cloud/projects/98071/networks/2024666/resources it will be 2024666

kubectl -n kube-system create secret generic hcloud --from-literal=token=<HETZNER_API_TOKEN> --from-literal=network=<HETZNER_NETWORK_ID>

Adding and updating helm repository

helm repo add hcloud https://charts.hetzner.cloud
helm repo update hcloud

Install Hetzner Cloud Controller

helm install hccm hcloud/hcloud-cloud-controller-manager -n kube-system

Step 8 - Install Ingress Controller

In this tutorial, we will use Nginx Ingress Controller. TLS encryption will be done on the Hetzner Load Balancer, which will be automatically added and configured during the installation of the Ingress Controller.

Before starting the installation, we need to make the cluster worker nodes available for scheduling

kubectl taint nodes <first worker node name> node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule-
kubectl taint nodes <second worker node name> node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule-

Adding and updating helm repository

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

The values.yaml file can be downloaded from the official Ingress Nginx repository. Change the following values inside the file:

• controller.dnsPolicy: ClusterFirstWithHostNet

• controller.hostNetwork: true- by default, while using hostNetwork, name resolution uses the host's DNS. If you wish Ingress Controller to keep resolving names inside the Kubernetes network, use ClusterFirstWithHostNet

• controller.kind: DaemonSet - install Ingress Controller as a DaemonSet

• controller.service.annotations - these annotations define the settings for the Hetzner Load Balancer, which will be created by the Hetzner Сloud Сontroller automatically after the Ingress Controller is installed. The list and description of available annotations can be found in the official Hetzner Cloud Controller repository

  Click here to view an example

  controller:
    [...]
    service:
      [...]
      annotations:
        load-balancer.hetzner.cloud/name: "k8s-test-lb"
        load-balancer.hetzner.cloud/location: "fsn1"
        load-balancer.hetzner.cloud/type: "lb11"
        load-balancer.hetzner.cloud/ipv6-disabled: "true"
        load-balancer.hetzner.cloud/use-private-ip: "true"
        load-balancer.hetzner.cloud/protocol: "https"
        load-balancer.hetzner.cloud/http-certificates: "certificatename"
        load-balancer.hetzner.cloud/http-redirect-http: "true"

  • load-balancer.hetzner.cloud/name: "k8s-test-lb" - this is the name of the Load Balancer. The name will be visible in the Hetzner Cloud Console
  • load-balancer.hetzner.cloud/location: "fsn1" - specifies the location where the Load Balancer will be created in
  • load-balancer.hetzner.cloud/type: "lb11" - specifies the type of the Load Balancer
  • load-balancer.hetzner.cloud/ipv6-disabled: "true" - disables the use of IPv6 for the Load Balancer
  • load-balancer.hetzner.cloud/use-private-ip: "true" - configures the Load Balancer to use the private IP for Load Balancer server targets. This is necessary so that traffic from the Hetzner Load Balancer to the cluster nodes goes inside the Hetzner Network
  • load-balancer.hetzner.cloud/protocol: "https" - specifies the protocol of the service
  • load-balancer.hetzner.cloud/http-certificates: "certificatename" - a comma separated list of IDs or Names of Certificates (see the Prerequisites in the Introduction)
  • load-balancer.hetzner.cloud/http-redirect-http: "true" - create a redirect from HTTP to HTTPS

• controller.service.enableHttp: false - enable the HTTP listener on both controller services or not

• controller.service.targetPorts.https: http - port of the Ingress Controller the external HTTP listener is mapped to. Since we have configured HTTP to HTTPS redirect on the Hetzner Load Balancer and assigned it as responsible for TLS encryption, there is no need to use encryption between the Hetzner Load Balancer and the cluster nodes. Accordingly, the port should be specified as HTTP (80)

Installing Ingress Controller

helm install ingress-nginx-controller ingress-nginx/ingress-nginx -f values.yaml -n kube-system

Step 9 - Install CSI driver (Optional)

At the time of writing, Hetzner Cloud only has ReadWriteOnce volumes available via Container Storage Interface, but it can still be useful, so I recommend installing the CSI driver.

Adding and updating helm repository

helm repo add hcloud https://charts.hetzner.cloud
helm repo update hcloud

Installing Hetzner Cloud CSI driver

helm install hcloud-csi hcloud/hcloud-csi -n kube-system

Conclusion

As a result, you have a cluster consisting of 1 Control Plane node and 2 Worker nodes. You also have the Load Balancer operating over HTTPs.

Now you can block all Internet traffic with firewall rules on all cluster nodes and create the remaining resources necessary to run your own application on the cluster.

License: MIT