CockroachDB cluster on Hetzner Cloud

Introduction

A basic production setup consisting of 3 availability zones and 1 node per AZ (see also https://www.cockroachlabs.com/docs/v21.1/topology-basic-production).

Step 1 - Create/Order components

You will need:

1x Hetzner Cloud Loadbalancer LB11
3x CX21 cloud servers
1x Cloud network
Docker Compose (https://docs.docker.com/compose/)

You can use https://github.com/hetznercloud/cli to order servers, load balancers and cloud networks

Step 1.1 - Placement groups

After creating a hcloud CLI context using an API token:

$ hcloud context create my-api-token
Token: 
Context my-api-token created and activated

you first need to create placement groups for each availability zone:

$ hcloud placement-group create --name pg-az1 --type spread
Placement group 1096 created

Repeat placement-group create for AZ2 and AZ3

Step 1.2 - Database network

Run the following commands to create a database network:

$ hcloud network create --ip-range 10.0.0.0/8 --name db-network 
Network 1206536 create

$ hcloud network add-subnet --ip-range 10.0.0.0/8 \
  --network-zone eu-central \
  --type cloud \
  db-network
528ms [==================================] 100.00%
Subnet added to network 1206536

Step 1.3 - Load balancer

You can order the db-lb using the following command:

$ hcloud load-balancer create --algorithm-type least_connections \
  --location nbg1 \
  --name db-lb \
  --type lb11
529ms [==================================] 100.00%
LoadBalancer 420294 created
IPv4: [REDACTED]
IPv6: [REDACTED]

Please adjust --location to where you want to place the majority of your database nodes.

Then run

$ hcloud load-balancer attach-to-network --network db-network db-lb
528ms [==================================] 100.00%
Load Balancer 420294 attached to network 1206536

to attach the load balancer to the database network

Step 1.4 - Cloud servers

Finally you can order the database node servers:

$ hcloud server create --image debian-11 \
  --location nbg1 \
  --name region1-az1-dbnode1 \
  --network db-network \
  --placement-group pg-az1 \
  --ssh-key my-ssh-key \
  --type cx21 
4.029s [=================================] 100.00%
Waiting for server 14071317 to have started
 ... done

Server 14071317 created
IPv4: [REDACTED]

Omit --ssh-key if you do not want to use SSH keys for authentication.

You then can attach each server to the load balancer:

$ hcloud load-balancer add-target --server region1-az1-dbnode1 \
  --use-private-ip \
  db-lb
515ms [==================================] 100.00%
Target added to Load Balancer 420294

Repeat server create and load-balancer add-target for the two other nodes az2-node1 and az3-node1 (adjust --name and --placement-group accordingly).

Step 2 - Install database nodes

The following steps need to be performed on all database node servers.

First you need to install docker-compose and create a cockroach directory:

# apt-get update
…
# apt-get install docker.io docker-compose apparmor
…
# DEBIAN_FRONTEND=noninteractive dpkg-reconfigure apparmor
# mkdir /srv/cockroach

You then need to create a docker-compose.yml file.

--advertise-addr and binding addresses must be set to a nodes db-network IP. You can query the current nodes IP using a link local instance metadata query:

# curl http://169.254.169.254/latest/meta-data/private-networks
- ip: 10.0.0.3
  …

--join must be set to the load balancers db-network IP.

Run

$ hcloud load-balancer describe db-lb -o 'format={{(index .PrivateNet 0).IP}}'
10.0.0.2

on your local computer to query the load balancers private net IP.

Sample /srv/cockroach/docker-compose.yml:

---
# host:             region1-az1-dbnode1
# db-network ip:    10.0.0.3
# db-lb private ip: 10.0.0.2
version: "3"
services:
  cockroach:
    command: start --advertise-addr=10.0.0.3
      --certs-dir=certs
      --join=10.0.0.2
      --locality=zone=az1
    image: cockroachdb/cockroach
    ports:
      - 10.0.0.3:8080:8080
      - 10.0.0.3:26257:26257
    restart: unless-stopped
    volumes:
      - ./data:/cockroach/cockroach-data
      - ./certs:/cockroach/certs

Finally run

# cd /srv/cockroach
# docker-compose up -d

to start cockroach

Step 3 - Setup Cockroach

Step 3.1 - Create certificates

CA

After you have setup all database nodes you need to create a CA on a trusted computer:

$ mkdir cockroach-ca
$ cd cockroach-ca
$ mkdir certs
$ mkdir private
$ docker run --rm \
  -v "$PWD/certs:/certs" \
  -v "$PWD/private:/private" \
  cockroachdb/cockroach cert create-ca \
  --certs-dir=/certs \
  --ca-key=/private/ca.key

Node certificates

Then create and upload certificates for each node:

$ docker run --rm \
  -v "$PWD/certs:/certs" \
  -v "$PWD/private:/private" \
  cockroachdb/cockroach cert create-node \
  <db-network IP of database node server> \
  <private IP of load balancer> \
  <public IP of load balancer> \
  --certs-dir=/certs \
  --ca-key=/private/ca.key
$ sudo chown "$USER" certs/node.key
$ scp certs/ca.crt \
  certs/node.crt \
  certs/node.key \
  root@<public IP of database node server>:/srv/cockroach/certs
$ rm certs/node.crt certs/node.key

Add additional names as agruments to cockroach cert create-node if you plan to use DNS names.

Public IP addresses can be listed using the hcloud CLI.

Client certificate for root

Finally you need to create a client certificate for the root user:

$ docker run --rm \
  -v "$PWD/certs:/certs" \
  -v "$PWD/private:/private" \
  cockroachdb/cockroach cert create-client root \
  --certs-dir=/certs \
  --ca-key=/private/ca.key

Step 3.2 - Add services to load balancer

The database nodes should have picked up their certificates by now and we finally can add services to the DB loadbalancer:

$ hcloud load-balancer add-service \
  --destination-port 26257 \
  --listen-port 26257 \
  --protocol tcp \
  db-lb
518ms [==================================] 100.00%
Service was added to Load Balancer 420294

Cockroach will handle SSL/TLS, we therefore need to pass 443 HTTPS traffic via TCP:

$ hcloud load-balancer add-service \
  --destination-port 8080 \
  --listen-port 443 \
  --protocol tcp \
  db-lb
525ms [==================================] 100.00%
Service was added to Load Balancer 420294

Step 3.3 - Initialize the cluster

The only thing left to do is initializing the cluster via the load balancers public IP:

$ docker run --rm \
  -v "$PWD/certs:/certs" \
  cockroachdb/cockroach init \
  --certs-dir=/certs \
  --host=<public IP of load balancer>

Step 4 - SQL and Cockroach Console access

Run

$ docker run -i --rm -t \
  -v "$PWD/certs:/certs" \
  cockroachdb/cockroach sql \
  --certs-dir=/certs \
  --host=<public IP of load balancer>

to open a SQL console.

After setting a root password:

ALTER USER root WITH PASSWORD '[REDACTED]';

and adding the CA cert you previously created to your browsers trust store, you can log in to the Cockroach Console via

https://<public IP of load balancer>

CockroachDB cluster on Hetzner Cloud

Author

Published

Time to read

Table of Contents