Setting Up a Secure Fedora Webserver in the Hetzner Cloud

Introduction

This tutorial will walk you through the steps to be taken to secure and publish a newly created Fedora webserver in the Hetzner Cloud infrastructure.

Prerequisites

Hetzner account with access to Cloud and DNS Console
Registered domain name to be used

Step 1 - Cloud Instance Creation

Creating a cloud server instance is really simple in the Hetzner Cloud Console, one simply needs to select the desired capacity, the operating system, upload a public SSH key, and is ready to go in less than a minute:

In the console open the Project you wish to add the new instance to, and click Add Server
Select your desired data center location
For OS Image select Fedora
Select your instance type
- Be sure to read the details carefully, since this will affect your monthly costs
- For this scenario we won't add any other resources, than a cloud instance
Add an SSH key to be used when connecting
- If you do not have a key, the console allows you to upload one at this step, be sure to upload your public key
- To create a new one, you can use ssh-keygen - link
  - Example: $ ssh-keygen -t ed25519
  - If you use multiple keys with your daily workflow (eg. dev-ops, sys-admin), it is advised to organize your keys with custom names, and/or subdirectories in your local user's ~/.ssh directory - not to interfere with keys used for other tools, or privately
Name your server based on your workflow's naming convention, eg. example-host
Create the instance

Once your server is created, you can obtain its IP address, be sure to save it for later use!

Step 2 - DNS Zone Setup

Since our aim is to create a secure public webserver, this is the time to assign our registered domain name to it. To do so, first we need to create a new DNS zone in the Hetzner DNS Console:

In the console click Add New Zone
Insert your domain name as the Zone Name
- Be sure to enter it correctly, without subdomain (or the www. prefix), eg. example.com
Since we assume a fresh domain, you can opt out auto scanning for records
Create the zone

Once your zone is ready, you will see some default records created for you, including the Hetzner nameservers, A records for @ (representing your domain without any prefix), www, and mail (and also a mail MX record).

We will not cover mail servers now, so you can delete the mail regarding A, and MX records
Add the previously acquired server IP address as value to the @, and www record
Save your DNS zone settings

Now that your DNS zone is ready, copy the values of the Hetzner name servers, and set those up with your domain's registar (propagation might take up to 48 hours, but usually you’ll be good to go in a few hours).

NOTE: Setting up nameservers for registered domains may vary based on your registar, you might have access to a web based console, or might need to ask them to make those changes for you in eg. email.

Step 3 - First SSH Connection

Once your server is up and running, you can connect through SSH to it with the newly assigned IP address. Open your local SSH config file (located at ~/.ssh/config), and add a new host configuration:

Host <example-host>
  HostName <hetzner-cloud-ip>
  User root
  IdentityFile ~/.ssh/<private-key-path>
  Port 22

Where:

<example-host> will be a local identifier for your server, but it is a good practice to use the same name used when creating your cloud instance
<hetzner-cloud-ip> is the previously acquired server IP address (once your domain is set up with your DNS Zone, you could also use the domain name)
<private-key-path> is the path where your private key is located in your local system (for which we previously uploaded its public pair)

Now to connect to your newly created instance, simply run in your shell:
$ ssh <example-host>

Step 4 - System Update

Once connected, update the system with the following commands:

$ dnf -y update && dnf -y upgrade
$ yum clean all && yum -y update && yum -y upgrade
$ yum -y install yum-utils
$ dnf remove --oldinstallonly --setopt installonly_limit=2

To keep only the latest two kernels installed on your system, you can edit the file /etc/dnf/dnf.conf modifying the following line to look like this:

installonly_limit=2

Step 4.1 - Install Midnight Commander (Optional)

To install, run the following command:

$ yum -y install mc

Once installation is complete, you can start Midnight Commander with the command:

$ mc

To check out available skins in the running application, press F9 to access the menu, navigate to Options > Appearance, and try loading different skins.

To disable the "hint bar", press F9 to access the menu, navigate to Options > Layout, and uncheck the Hintbar visible option.

Step 5 - Secure SSH

First, change the root password generated by Hetzner with the command:

$ passwd root

Be sure to remember this password, since this will be needed every time you need root privileges for an action.

To create a new user which will be used for your new connections, and also add it to the wheel group to have sudo rights run:

$ adduser <new-username>
$ usermod -aG wheel <new-username>

Next, to keep using your previous key pair, move the default /root/.ssh directory to your newly created user's home, and set according privileges:

$ mv /root/.ssh /home/<new-username>
$ chown -R <new-username>:<new-username> /home/<new-username>/.ssh

To change the SSH port, disable password authentication, and restrict the root user from logging in remotely only allowing our newly created user to connect open the server's SSH configuration located at /etc/ssh/sshd_config, and ensure that these settings are met (either by modifying/un-commenting default values, or adding them as extra lines if not present):

Port <new-ssh-port>
AddressFamily inet
PasswordAuthentication no
PermitRootLogin no
AllowUsers <new-username>

Step 5.1 - Enable and Setup Selinux

To install and setup the required packages, run:

$ yum -y install selinux-policy-targeted selinux-policy libselinux libselinux-utils policycoreutils setroubleshoot setroubleshoot-server setroubleshoot-plugins
$ semanage port -a -t ssh_port_t -p tcp <new-ssh-port>

Now edit the configuration file /etc/selinux/config to set selinux policy to enforcing:

SELINUX=enforcing

Step 5.2 - Enable and Setup Firewalld

To install the required packages and create the default setup, run:

$ yum -y install firewalld
$ cp /usr/lib/firewalld/services/ssh.xml /etc/firewalld/services/

Now edit the newly created service /etc/firewalld/services/ssh.xml, and change the default SSH port in the following line:

<port protocol="tcp" port="<new-ssh-port>"/>

Change <new-ssh-port> in this XML entry too, eg "3322"!

To enable the new service instantly, run:

$ systemctl enable --now firewalld

Step 5.3 - Update Local SSH Config

In your local SSH config file (located at ~/.ssh/config) change the new user and port configuration as set on the remote:

User <new-username>
Port <new-ssh-port>

If your DNS Zone is already working with your domain, you may change the HostName entry too to your domain, eg. example.com

Now you can safely reboot, and after connect to the updated SSH service running on your cloud server.

If you wish to grant access to others than your newly created user on your instance, it is a good practice to notify all users of the consequences of their actions, that way if things get nasty, you might have a legal point for logging -, and in worst-case-scenario, using those logs of their actions. Although the roots of such warning labels originate from way before, it is a zero-hassle, and a simple to set-up protection. Since we already disallowed password based login, what we can do is remind all users of their legal responsibility upon login.

To set up such banner, open the file /etc/motd, and paste your preferred legal warning to be shown upon successful connection. The following sample is what I use, made up from ASCII art, and a good old CISCO warning:

#################################################################
#   ___        _       _    _          _     _                  #
#  | _ \___ __| |_ _ _(_)__| |_ ___ __| |   /_\  _ _ ___ __ _   #
#  |   / -_|_-<  _| '_| / _|  _/ -_) _` |  / _ \| '_/ -_) _` |  #
#  |_|_\___/__/\__|_| |_\__|\__\___\__,_| /_/ \_\_| \___\__,_|  #
#                                                               #
#       UNAUTHORIZED ACCESS TO THIS DEVICE IS PROHIBITED        #
#                                                               #
# You must have explicit, authorized permission to access or    #
# configure this device. Unauthorized attempts and actions to   #
# access or use this system may result in civil and/or criminal #
# penalties. All activities performed on this device are logged #
# and monitored.                                                #
#                                                               #
#################################################################

An online Figlet ASCII font generator
CISCO's Legal Banner Notification for reference

NOTE I am in no place to give legal advise. If you think your -, or your company's needs reach further, do consult with a lawyer (who is well prepared in online statements)! I take no legal responsibility for the above advice, it is merely just a friendly reminder.

NOTE If you ssh from Windows PowerShell, you might see the contents of /etc/motd and last login twice (or even thrice), this is not a problem with sshd or pam settings, but probably a bug in PowerShell itself. If in doubt, you might want to ensure that things work properly from eg. Git Bash.

NOTE for Further Commands on Remote

From now on all below commands assume you have root privileges, so with the newly created user you will have to either run the commands with the sudo prefix, or act in the name of root user with the command:

$ su root -

Step 6 - Install Nginx

To install the required packages and create an empty setup run:

$ yum -y install nginx

To enable public http and https traffic, first configure your firewall, then start the Nginx service:

$ firewall-cmd --permanent --zone=public --add-service=http
$ firewall-cmd --permanent --zone=public --add-service=https
$ firewall-cmd  --reload
$ systemctl enable --now nginx

Step 6.1 - Setup Support for Server Blocks

Create a new directory for the static contents of your server block:

$ mkdir -p /var/www/<your-domain>/html
$ chmod -R 755 /var/www

Now create the necessary directories inside /etc/nginx as per convention:

$ mkdir /etc/nginx/sites-available
$ mkdir /etc/nginx/sites-enabled

Now modify the Nginx configuration file located at /etc/nginx/nginx.conf, add the following lines under the existing include /etc/nginx/conf.d/*.conf; directive inside the http block:

# Load modular configuration files from the /etc/nginx/sites-enabled directory.
include /etc/nginx/sites-enabled/*.conf;

# General customizations.
server_names_hash_bucket_size 64;
server_tokens                 off;
proxy_hide_header             X-Powered-By;
ssl_session_cache             shared:SSL:10m;
ssl_session_timeout           10m;

Then comment out / delete the whole default server block in this file under http - we will create our own one in a later step.

Step 6.1.1 - NOTE - Hardening Your Webserver with Additional Headers

It is out of the scope of the current tutorial, but for additional security measurements, consider hardening the webserver with additional HTTP headers based on your requirements. Some usual ones include (you can find great resources online explaining them):

add_header X-Frame-Options "SAMEORIGIN";
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";
add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
add_header X-XSS-Protection "1; mode=block";
add_header X-Content-Type-Options nosniff;

Step 6.2 - Create Virtual Host Configuration

Now we need to create a configuration file for our virtual host, run:

$ touch /etc/nginx/sites-available/<your-domain>.conf

Where the configuration file's name could look like example.com.conf.

Edit this newly created configuration adding the following lines:

server {
    listen       80;
    server_name  <your-domain> www.<your-domain>;
    root         /var/www/<your-domain>/html;
    index        index.html;
    try_files    $uri $uri/ =404;

    error_page 404 /40x.html;
    location = /40x.html {
    }

    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
    }
}

Create an index.html for your static content:

$ touch /var/www/<your-domain>/html/index.html

Now edit the newly created index.html, add some example content to see later if your site is loading.
You may want to create the custom 40x.html and 50x.html accordingly in /var/www/<your-domain>/html, otherwise Nginx will fall back to the default ones.

To enable the site create a symlink in the sites-enabled directory:

$ ln -s /etc/nginx/sites-available/<your-domain>.conf /etc/nginx/sites-enabled/<your-domain>.conf

Test your configuration, and if everything is ok restart the Nginx service run:

$ nginx -t
$ service nginx restart

The only thing left is to tell selinux, that we want to serve static content from the /var/www/<your-domain>/html directory:

$ semanage fcontext -a -t httpd_sys_content_t "/var/www/<your-domain>/html(/.*)?"
$ restorecon -Rv /var/www/<your-domain>/html

To ensure the updated policy, you can run:
$ semanage fcontext -l | grep --color <your-domain>

Now your domain should show the contents of the previously created index.html.

Step 7 - Acquire a Wildcard Certificate and Setup SSL

To install prerequisites, the required packages, and issue the certificate run:

$ openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048
$ yum -y install certbot python3-certbot-nginx
$ certbot certonly --agree-tos --email <your-email> --manual --preferred-challenges dns -d <your-domain> -d *.<your-domain> --server https://acme-v02.api.letsencrypt.org/directory

Be sure, to list <your-domain> and *.<your-domain> too, eg. -d example.com -d *.example.com!

When prompted to add a TXT record to your DNS for verification, do so in the previously created DNS Zone in the Hetzner DNS Console before continuing:

Open your zone settings
Under create record:
- Set type to TXT
- Set name as required (usually _acme-challenge)
- Set the value to your secret verification code
Click add record

Now you can continue running the certbot script, it will generate your certificate files at the location shown in the output.

After acquired the certificate files, be sure to delete the temporary verification TXT record from your DNS setup (renewal will require different values, we will automate that later).

Step 7.1 - Update Nginx Configuration

Since we choose to do a manual install, we still need to notify Nginx about the certificates. To do so edit your virtual host configuration located at /etc/nginx/sites-available/<your-domain>.conf, and add the following new server block:

server {
    listen                     443 ssl http2;
    server_name                <your-domain> www.<your-domain>;
    keepalive_timeout          70;
    root                       /var/www/<your-domain>/html;
    index                      index.html;
    try_files                  $uri $uri/ =404;
    ssl_certificate            /etc/letsencrypt/live/<your-domain>/fullchain.pem;
    ssl_certificate_key        /etc/letsencrypt/live/<your-domain>/privkey.pem;
    ssl_dhparam                /etc/ssl/certs/dhparam.pem;
    ssl_protocols              TLSv1.2;
    ssl_prefer_server_ciphers  on;
    ssl_ciphers                "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";

    error_page 404 /40x.html;
    location = /40x.html {
    }

    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
    }
}

Currently we do not want to redirect all http traffic to https, but to do so, you can replace the complete port 80 server block configuration below the server_name directive with:

return 301 https://$host$request_uri;

Test your configuration, and if everything is ok restart the Nginx service:

$ nginx -t
$ service nginx restart

Step 8 - Setup Certificate Auto-renewal

To automatically check if renewal is necessary, and if so, do it we will need some prerequisites, first install the following packages:

$ yum -y install crontabs jq

We will need some extra scripts to communicate with Hetzner DNS API during the renewal process, we will use the ones referenced in this article:

$ curl https://raw.githubusercontent.com/dschoeffm/hetzner-dns-certbot/master/certbot-hetzner-auth.sh > /usr/local/bin/certbot-hetzner-auth.sh
$ curl https://raw.githubusercontent.com/dschoeffm/hetzner-dns-certbot/master/certbot-hetzner-cleanup.sh > /usr/local/bin/certbot-hetzner-cleanup.sh
$ chmod +x /usr/local/bin/certbot-hetzner-auth.sh
$ chmod +x /usr/local/bin/certbot-hetzner-cleanup.sh

NOTE: Always double check scripts downloaded from the internet before putting them to use!

To create necessary Hetzner DNS API Token, navigate to the access management page of the DNS Console, create a token, then save it on your server:

$ echo <dns-api-token> > /etc/hetzner-dns-token

Now all that left is to create a cron job to run the certbot renew script:

$ crontab -e

This will open the vi editor, so first press i to enter INSERT mode, than paste the following line:

15 3 * * * /usr/bin/certbot renew --manual-auth-hook /usr/local/bin/certbot-hetzner-auth.sh --manual-cleanup-hook /usr/local/bin/certbot-hetzner-cleanup.sh --deploy-hook "service nginx restart" --quiet

After this, press Esc to return to the command input, type :wq and press Enter to save and exit.

From now on cron will run this command daily, and if necessary, will update your certificate in the background at 3:15am.

Step 9 - Setup SFTP Based Deployment (Optional)

First we will create a new user group for our SFTP deploy users:

$ groupadd sftp-deploy

Next we create a new user for deployment purposes:

$ useradd sftp-<deploy-username>
$ mkdir -p /home/sftp-<deploy-username>/.ssh/
$ touch /home/sftp-<deploy-username>/.ssh/authorized_keys
$ chown -R sftp-<deploy-username>:sftp-<deploy-username> /home/sftp-<deploy-username>/.ssh
$ usermod -aG sftp-deploy sftp-<deploy-username>

After this step we need to add contents to the /home/sftp-<deploy-username>/.ssh/authorized_keys file, so acquire an appropriate public key from the user you plan to grant permissions, and paste it inside the above file.

Next, we need to calibrate the SSH daemon to support this group (and also restrict their actions), so head to the /etc/ssh/sshd_config file, and uncomment/change/add the following lines:

# override default of no subsystems
Subsystem sftp internal-sftp

# SFTP Group Settings
Match Group sftp-deploy
    X11Forwarding       no
    AllowTcpForwarding  no
    ChrootDirectory     /var/www
    ForceCommand        internal-sftp

AllowUsers <new-username> sftp-<deploy-username>

Here we set up the following things:

Users in the previously created sftp-deploy group are only allowed to use the SFTP service
They are not allowed to use terminal commands
They are not allowed to leave the /var/www directory
We extend users allowed to connect to our instance with the newly created sftp-<deploy-username>

Now the only thing left is to grant appropriate permissions for the new group inside the /var/www/<your-domain>/html directory, so run the following commands:

$ chgrp -R sftp-deploy /var/www/<your-domain>/html
$ chmod -R u=rw,g=rw,o=r /var/www/<your-domain>/html
$ setsebool -P ssh_chroot_rw_homedirs on

NOTE: Depending on your requirements, you may want to add different permissions to your SFTP deployment users, this is merely an example scenario.

Conclusion

Now you have a secure Nginx webserver running on latest Fedora distribution able to serve static content on both http, and https.