Prometheus Service Discovery

Introduction

Prometheus is a modern open-source metrics and alerting/monitoring solution. Starting from version 2.21.0, Prometheus has a Service Discovery for Hetzner Dedicated and Cloud servers. This tutorial will help you to configure the Prometheus Service Discovery for the servers you have in your Hetzner Account.

The service discovery will talk to the Hetzner Robot Webservice and Hetzner Cloud API to get a list of your servers and automatically add them as scrape targets to collect metrics.

Prerequisites

You need at least Prometheus version 2.21.0, which you can download on prometheus.io. This tutorial does not cover the installation of Prometheus itself. For general help with installing and configuring steps please see the excellent Prometheus documentation.

We assume you have installed Prometheus and all your servers are running the node_exporter on Port 9100.

Step 1 - Generate credentials

Depending on which servers you would like to "discover", you will need credentials for the appropriate access.

Generating credentials for Dedicated servers

For Robot servers, you will need Robot Webservice credentials.

To get these, log in to Robot, go to "Settings" by clicking the user menu icon in the upper right corner, and select "Webservice and app settings".

Here, specify a password for the webservice and select "Create user". A username will be sent to your email address.

Generating an API token for Cloud servers

For Cloud servers, you will need a Hetzner Cloud API token, which acts like a combined username and password.

To create one, log in to the Cloud Console, select your project, go to the "Security" section in the left menu, and open the "API TOKENS" tab. Here, click the "GENERATE API TOKEN" button.

You will be asked for a description and permissions.

Description is just a text which will be displayed next to the token. Fill out this field, and keep in mind that the description will help you to easily find a token which you need to delete.

"Read" permissions are enough for discovery purposes.

Once you clicked the "GENERATE API TOKEN" button, the newly created token will be displayed to you. Please copy it immediately. It won't be shown again for security reasons.

Step 2 - Add Service discovery to Prometheus config

Extend your scrape_config section in the prometheus.yml file like in the example below.

For Hetzner Cloud servers use the "hcloud" role.
The "robot" role is used for discovering Hetzner Dedicated Root servers, managed via the Robot web interface.

scrape_configs:
  - job_name: hetzner_service_discovery
    hetzner_sd_configs:
      - role: "hcloud"
        bearer_token: "<your Hetzner Cloud API token>"
        port: 9100
      - role: "robot"
        basic_auth:
          username: "<your Hetzner Robot Webservice username>"
          password: "<your Hetzner Robot Webservice password>"
        port: 9100

For more details on the configuration of the service discovery you can check the official Prometheus documentation.

Step 3 - Restart Prometheus and check discovered servers

Restart Prometheus and check the Prometheus Dashboard at http://<your-prometheus-ip>/service-discovery. After a few seconds you should see all your discovered Hetzner Robot and Hetzner Cloud servers there.

The following labels are automatically added to each discovered target:

__meta_hetzner_server_id: the ID of the server
__meta_hetzner_server_name: the name of the server
__meta_hetzner_server_status: the status of the server
__meta_hetzner_public_ipv4: the public ipv4 address of the server
__meta_hetzner_public_ipv6_network: the public ipv6 network (/64) of the server
__meta_hetzner_datacenter: the datacenter of the server

The labels below are only available for targets with the role set to "hcloud":

__meta_hetzner_hcloud_image_name: the image name of the server
__meta_hetzner_hcloud_image_description: the description of the server image
__meta_hetzner_hcloud_image_os_flavor: the OS flavor of the server image
__meta_hetzner_hcloud_image_os_version: the OS version of the server image
__meta_hetzner_hcloud_datacenter_location: the location of the server
__meta_hetzner_hcloud_datacenter_location_network_zone: the network zone of the server
__meta_hetzner_hcloud_server_type: the type of the server
__meta_hetzner_hcloud_cpu_cores: the CPU cores count of the server
__meta_hetzner_hcloud_cpu_type: the CPU type of the server (shared or dedicated)
__meta_hetzner_hcloud_memory_size_gb: the amount of memory of the server (in GB)
__meta_hetzner_hcloud_disk_size_gb: the disk size of the server (in GB)
__meta_hetzner_hcloud_private_ipv4_<networkname>: the private ipv4 address of the server within a given network
__meta_hetzner_hcloud_label_<labelname>: each label of the server

The labels below are only available for targets with the role set to "robot":

__meta_hetzner_robot_product: the product of the server
__meta_hetzner_robot_cancelled: the server cancellation status

Conclusion

You can find more information about running and configuring Prometheus in the official documentation.

Prometheus Service Discovery

Author

Published

Time to read

Table of Contents