How to use Hetzner Object Storage with Django Storages

Introduction

This tutorial shows how to use Django and the django-storages library with Hetzner Object Storage.

It explains how to set up Django models for uploading files with either public or private access permissions. When you upload a file with private access permissions, Django automatically generates a pre-signed URL for you.

Although Hetzner Object Storage is S3-compatible, there are some differences you need to be aware of to make it work with Django and django-storages.

This is how the admin interface looks like, when you are done:

Prerequisites

• Basic Linux command knowledge
• A local, Debian-based machine with Python installed, and a browser to access localhost:8000
• Account with Hetzner

Terminology

• Bucket

  A Bucket is a container for storing objects in Hetzner Object Storage. In contrast to S3, Hetzner Object Storage does not add the Bucket name as a prefix to the domain name. All Buckets in a region are served on the same domain. This is important to know for security and for constructing the URLs and endpoints.

• S3 signature version

  Hetzner Object Storage is not compatible with the latest S3 signature version, which is the default in the underlying boto library. This is a major difference to S3 and needs to be taken care of when using Hetzner Object Storage with django-storages.

• Pre-built Access Control Lists (ACLs)

  Hetzner Object Storage supports the standard private and public-read pre-built Access Control Lists (ACLs). This makes it possible to serve both public and private media files.

• boto3

  boto3 is the underlying library used by django-storages to interact with S3-compatible storage services. Even if you don't need to interact with it directly, you can use it for troubleshooting. If you see errors, you can activate boto3 debug logging by placing the following code in your settings.py file:

  import boto3
  boto3.set_stream_logger(name='botocore')

  Now you should see detailed logs in your console. This is useful for debugging and understanding what is happening behind the scenes. If you have issues, it's also a good idea to check the boto3 documentation for S3 as it's much more detailed than the django-storages documentation.

Let's get started!

Step 1 - Create a Bucket and S3 Credentials

On the Hetzner Cloud Console, open an existing project or create a new project.

Step 1.1 - Create a Bucket

Open the Object Storage tab in the left menu of your project, and create a new Bucket using the following values:

Hit Create and Buy now to create the Bucket.

See Creating a Bucket documentation for more information on how to create an Object Storage Bucket.

Step 1.2 - Generate S3 Credentials

Open the Security tab in the left menu of your project, then open the S3 credentials tab, and click Generate credentials.

On the popup: Copy the ACCESS KEY and the SECRET KEY and store it in a safe place. You will need these values later in your Django project.

See Generating S3 keys for more information on how to generate S3 credentials.

Step 2 - Create a Django Project and App

On your lokal machine, check if you have Python installed with python --version or python3 --version. I recommend Python 3.12 at this point, but the tutorial should work with Python 3.10 or later.

If your version is older than 3.10, or if you don't have Python installed, please download and install Python first.

Step 2.1 - Create Project Directory and Python Virtual Environment

Create a folder for your Django project, and initialize a virtual Python environment in this directory. You can use virtualenv or venv for this.

If you don't have virtualenv installed, you can install it with pip install virtualenv.

mkdir example-project
cd example-project
python3 -m venv venv

Activate the virtual environment:

source venv/bin/activate

If you are using Windows, use venv\Scripts\activate instead.

Step 2.2 - Install Django, and required packages, and Create a New Django Project

Install Django and the required packages. Create a requirements.txt file with the following content:

# requirements.txt
Django
django-environ
django-storages[boto3]
whitenoise[brotli]

django-environ is used to load configuration values from the environment following best practices from The Twelve-Factor App manifesto. This also supports using a .env file to store configuration variables and credentials. Make sure to add the .env file to your .gitignore file, so that it won't be pushed to your git repository.

WhiteNoise with Brotli compression is used to serve static files in production.

Now, install the packages and create a new Django project:

pip install -r requirements.txt
django-admin startproject example_project

Step 2.3 - Create a Django App

Change into the Django project directory and start a Django app:

cd example_project
python manage.py startapp media_app

Test if that works by running the Django development server:

python manage.py runserver

Open your browser and go to http://localhost:8000/. You should see the Django welcome page:

Step 3 - Change Django Settings

Step 3.1 - Configure Imports and Load Environment Variables

example_project/
├── example_project/
│   └── settings.py
└── manage.py

Open the settings.py file in your Django project and add the following imports at the top of the file:

# settings.py (top of the file)
import os
from pathlib import Path
import environ

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
PROJECT_NAME = os.path.basename(BASE_DIR)

env = environ.Env()
environ.Env.read_env(env_file=BASE_DIR.parent / '.env')

Notes:

• environ is imported and initialized to load the environment variables from the .env file.
• The PROJECT_NAME is set to the name of the project folder. This is used to construct default values.

Step 3.2 - Configure Object Storage Settings

Now, add the Object Storage settings at the bottom of the settings.py file:

Keep the variables for the S3 information. Those are set in the next step.

# settings.py (bottom of the file)
# default storage backends when not using Hetzner Object Storage
# whitenoise is used to serve static files in production
STORAGES = {
    "default": {
        "BACKEND": "django.core.files.storage.FileSystemStorage",
    },
    "staticfiles": {
        "BACKEND": "whitenoise.storage.CompressedStaticFilesStorage",
    },
}

LOCATION_PREFIX = ""
PUBLIC_MEDIA_LOCATION = f"{LOCATION_PREFIX}media"
PRIVATE_MEDIA_LOCATION = f"{LOCATION_PREFIX}private"

USE_S3_MEDIA = env.bool("USE_S3_MEDIA", default=False)

if USE_S3_MEDIA:
    AWS_ACCESS_KEY_ID = env("AWS_ACCESS_KEY_ID", default="")
    AWS_SECRET_ACCESS_KEY = env("AWS_SECRET_ACCESS_KEY")
    AWS_STORAGE_BUCKET_NAME = env("AWS_STORAGE_BUCKET_NAME", default=PROJECT_NAME)
    # Set "virtual" for https://<bucket_name>.<endpoint>.com
    # Set "path" for https://<endpoint>.com/<bucket_name>
    # Hetzner Object Storage uses "virtual".
    AWS_S3_ADDRESSING_STYLE = "virtual"
    AWS_S3_ENDPOINT_URL = env(
        "AWS_S3_ENDPOINT_URL",
        default="https://fsn1.your-objectstorage.com"
    )
    AWS_DEFAULT_ACL = None # private by default
    # prefix for this project's files on the Bucket, "" to disable
    LOCATION_PREFIX = env("LOCATION_PREFIX", default=f"{PROJECT_NAME}/")
    PUBLIC_MEDIA_LOCATION = f"{LOCATION_PREFIX}media"
    PRIVATE_MEDIA_LOCATION = f"{LOCATION_PREFIX}private"
    MEDIA_URL = f"{AWS_S3_ENDPOINT_URL}/{AWS_STORAGE_BUCKET_NAME}/{LOCATION_PREFIX}media/"
    STORAGES["default"] = {
        "BACKEND": "media_app.storage_backends.PublicMediaStorage"
    }
    PRESIGNED_URL_EXPIRATION = env.int("PRESIGNED_URL_EXPIRATION", default=600)

Notes:

When you leave the Bucket name and the location prefix at their default values, the project will use the project name both as the Bucket name and the location prefix. This might be a bit confusing, as you see the same name repeated. Thus, if you share a Bucket across projects, you might want to set the Bucket name to a different name, and if you use a Bucket exclusively for a single project, you might want to set the LOCATION_PREFIX to an empty string.

As Hetzner Object Storage Buckets have a fixed base cost per month, you can save some money if you share the Bucket across projects.

Step 3.3 - Add Installed Apps and Middleware

Finally, add the required apps and middleware to the already existing sections INSTALLED_APPS and MIDDLEWARE in the settings.py file:

# settings.py (add to the existing INSTALLED_APPS and MIDDLEWARE)
INSTALLED_APPS = [
    #...
    'whitenoise.runserver_nostatic',  # whitenoise runserver
    'media_app',
]
MIDDLEWARE = [
    # after the security middleware
    'whitenoise.middleware.WhiteNoiseMiddleware',  # whitenoise middleware
]

Step 4 - Create a .env file

Create a .env file in the root directory of your Django project.

example_project/
├── example_project/
├── manage.py
└── .env

This file will contain the sensitive configuration values for your project. In production environments, the platform (eg. Docker, Kubernetes) can inject the environment variables.

# .env
#DJANGO_SECRET_KEY=your_secret_key
USE_S3_MEDIA=True
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_S3_ENDPOINT_URL=https://fsn1.your-objectstorage.com
AWS_STORAGE_BUCKET_NAME=your_bucket_name
#LOCATION_PREFIX=your_location_prefix (must end with a /)
AWS_REQUEST_CHECKSUM_CALCULATION="WHEN_REQUIRED"
AWS_RESPONSE_CHECKSUM_CALCULATION="WHEN_REQUIRED"

Notes:

By the way: You can generate a new secret key using the django-admin command:

django-admin shell -c """
from django.core.management.utils import get_random_secret_key; 
print(get_random_secret_key())
"""

Step 5 - Implement Storage Backends

Now, implement Django file storage backends for public, and private media files. This is how the media_app/storage_backends.py file should look like:

#media_app/storage_backends.py
from django.conf import settings
from django.core.files.storage import FileSystemStorage
from storages.backends.s3boto3 import S3Boto3Storage

settings.LOCATION_PREFIX

class BaseMediaStorage(S3Boto3Storage):
    signature_version = "s3"
    file_overwrite = False
    custom_domain = False


class PublicMediaStorage(BaseMediaStorage):
    location = settings.PUBLIC_MEDIA_LOCATION
    default_acl = "public-read"


class PrivateMediaStorage(S3Boto3Storage):
    location = settings.PRIVATE_MEDIA_LOCATION
    default_acl = "private"


def get_private_file_storage():
    if not settings.USE_S3_MEDIA:
        return FileSystemStorage()
    else:
        return PrivateMediaStorage()

Notes:

• The implementation can be based on S3BotoStorage from the django-storages package.
• Common settings are defined in a base class.
• The signature_version to s3 to make it work with Hetzner Object Storage. The default is s3v4, which is not compatible with Hetzner Object Storage.
• The default_acl must be set to private for the private storage backend and public-read for the public storage backend.
• A helper function returns a storage backend dependent on the USE_S3_MEDIA setting. This way, you can develop locally without using the Cloud Storage.

Step 6 - Create Django Models with File Fields

Now, open the media_app/models.py file, and implement Django models to hold references to the public and private files.

#media_app/models.py
import os
import uuid

from django.conf import settings
from django.core.exceptions import ValidationError
from django.db import models

from .storage_backends import get_private_file_storage, PrivateMediaStorage


def _get_random_filename(instance, filename):
    model_name = instance.__class__.__name__.lower()
    ext = filename.split('.')[-1]
    new_filename = f"{uuid.uuid4()}.{ext}"
    return os.path.join(model_name, new_filename)


class PublicDocument(models.Model):
    title = models.CharField(max_length=255)
    file = models.FileField(upload_to=_get_random_filename)


class PrivateDocument(models.Model):
    title = models.CharField(max_length=255)
    file = models.FileField(
        upload_to=_get_random_filename,
        storage=get_private_file_storage
    )

    def get_presigned_url(self):
        if settings.USE_S3_MEDIA:
            storage = PrivateMediaStorage()
            return storage.url(
                self.file.name,
                expire=settings.PRESIGNED_URL_EXPIRATION
            )
        return None

Notes:

Create migrations and migrate the database to create the tables needed for the models:

python manage.py makemigrations
python manage.py migrate

If everything works as expected, you should see no errors in the console.

Step 7 - Create a Django Admin Interface

Open the media_app/admin.py file and implement a django admin class for the PublicDocument and PrivateDocument models:

#media_app/admin.py
from django.contrib import admin
from django.utils.html import format_html
from .models import PublicDocument, PrivateDocument


@admin.register(PublicDocument)
class PublicDocumentAdmin(admin.ModelAdmin):
    list_display = ('title', 'file_url')

    def file_url(self, obj):
        return format_html(
            '<a href="{}" target="_blank">View File</a>',
            obj.file.url
        )
    file_url.short_description = 'File Link'


@admin.register(PrivateDocument)
class PrivateDocumentAdmin(admin.ModelAdmin):
    list_display = ('title', 'get_file_url')
    readonly_fields = ('get_file_url',)

    def get_file_url(self, obj):
        url = obj.get_presigned_url()
        return format_html(
            '<a href="{}" target="_blank"> Download File</a>',
            url
        )
    get_file_url.short_description = 'Download Link'

Notes:

• The PublicDocumentAdmin just adds a computed field to display a link to the file in the list view.
• The PrivateDocumentAdmin adds a computed field to display a link to the file in the list view.
• The get_file_url method generates a pre-signed URL for the private file.

Step 8 - Test the Implementation

To test the implementation, create a superuser, run the Django development server again, and open the Django admin interface to test the file uploads and downloads:

python manage.py createsuperuser
python manage.py runserver

Click here if you get an error message

If you get the error message Error: That port is already in use., the port is probably still in use from the test in step 2.3. Check if the port is in fact used by python from step 2.3. If it is, stop the process and try running the Django development server again.

lsof -i :8000
kill -9 <PID>
python manage.py runserver

Open your browser and go to:

http://localhost:8000/admin/

Log in with the superuser credentials you just created. You should see the Django admin interface with the PublicDocument and PrivateDocument models listed.

Step 8.1 - Testing Public Files

Create a new PublicDocument object. Upload a file and save the object. You should see a link to the file in the list view. Click on the link to open the file in a new tab. The file should be publicly accessible.

Copy the URL and open it in a new private browser window. You should still be able to access the file without logging in.

Step 8.2 - Testing Private Files

Create a new PrivateDocument object. Upload a file and save the object. You should see a link to the file in the list view. Click on the link to open the file in a new tab. The file should be accessible.

Copy the URL, wait 15 minutes, and open it in a new private browser window. You should see an error message saying that the file is not accessible. This is because the file is private and requires authentication to access it.

Step 8.3 - Check Hetzner Object Storage Contents

Open the Object Storage Bucket in the Hetzner Cloud Console. You should see the files uploaded to the media and private folders.

Conclusion

In this tutorial, you've learned

• how to use Hetzner Object Storage with Django Storages to serve public and private media files, and
• how to use pre-signed URLs to securely serve private files.

Checklist

In summary, you need to get the following seven aspects right when using Hetzner Object Storage with Django:

1. Use a private Bucket if you want to store private files.
2. Use the correct S3 signature version (s3).
3. Use the correct endpoint URL and Bucket path for Hetzner Object Storage.
4. Use ACLs to control access to the files.
5. Use pre-signed URLs to securely serve private files.
6. Set the AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_CALCULATION to WHEN_REQUIRED to avoid errors when uploading files to Hetzner Object Storage.
7. Optionally, use location prefixes to share a Bucket across multiple projects.

As you may have noticed, the tutorial does not cover the use of Object Storage for static media files. This is because Hetzner Object Storage serves files for all Buckets in a region on the same domain. This renders CORS security settings unusable. You need to allow the whole domain with all Buckets even if you cannot trust the files coming from this domain. This means your site may become vulnerable to cross site scripting attacks. You can serve static media using a CDN or use whitenoise to serve static files in production from your app itself.

Further Reading

For more information, please refer to the following resources:

• Hetzner Object Storage documentation
• django-storages documentation
• boto3 documentation for S3

Security Notice

Please note that this is just an example implementation. You may need to adapt it to your specific use case, and you should add security measures to your implementation, such as:

• file name validation
• limit valid file types
• limit maximum file size

Also, make sure to check permissions on views that return pre-signed URLs. In this example, this is implicitly done by using the Django admin interface.

License: MIT