Setup custom domain for S3-compatible object storage via reverse proxy

Introduction

This tutorial will guide you to setup a custom domain for S3-compatible object storage using reverse proxy. The advantages of a custom domain are to enable seamless integration with existing infrastructure or services under a unified domain.

There are different ways to configure a custom domain, such as using a CNAME record or a reverse proxy. This tutorial focuses on configuring a custom domain using a reverse proxy.

Note: When using a reverse proxy, outgoing traffic is billed twice — once from the object storage to the proxy and once from the proxy to the client. However, this only has an impact for high download volumes with dozens of TB per month.

Prerequisites

• A server (e.g. with Hetzner Cloud)
• An S3-compatible bucket (e.g. with Hetzner)
• A domain you want to use (e.g. storage.example.com).

Step 1 - Create Object Storage Bucket

Create an S3-compatible bucket. With Hetzner, see the getting started "Creating a Bucket". Make sure it is set to public access permissions. Not much benefit to using a custom domain for private buckets.

Create S3 credentials to access your bucket. With Hetzner, see the getting started "Generating S3 keys".

Step 2 - Create Server

Create a new server. With Hetzner, see the getting started "Creating a Server". To install Docker and Docker Compose, follow the official Docker documentation.

Step 3 - Deploy Caddy

SSH to your server ssh root@<server-ip>.

Create a directory for your Docker Compose files and folders for the persistent storage of the Caddy container:

sudo mkdir -p /opt/caddy/data

Step 3.1 - Create Docker deployment and configuration files

• Add a Docker compose file

  sudo vim /opt/caddy/compose.yaml

  Add the following content:

  services:
    caddy:
      container_name: caddy
      image: caddy:latest
      restart: unless-stopped
      ports:
        - 80:80
        - 443:443
      volumes:
         - ./data/Caddyfile:/etc/caddy/Caddyfile
         - ./data/certs:/certs
         - ./data/config:/config
         - ./data/data:/data
         - ./data/sites:/srv

• Add a Caddyfile

  sudo vim /opt/caddy/data/Caddyfile

  Add the following content:

  Replace storage.example.com with your own domain.
  Replace fsn1.your-objectstorage.com with the endpoint of your object storage bucket. If the bucket name comes after the endpoint (e.g. https://s3-endpoint.example.org/<bucket_name>) add your endpoint without the bucket name.

  storage.example.com {
  
      tls {
          issuer acme {
              dir https://acme-v02.api.letsencrypt.org/directory
          }
      }
  
      reverse_proxy https://<bucket_name>.fsn1.your-objectstorage.com {
      #reverse_proxy https://s3-endpoint.example.org {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Forwarded-Host {host}
      }
  }

Step 3.2 - Start Caddy

cd /opt/caddy
docker compose up -d
docker ps

After the Docker container started, you can access your files via storage.example.com.

If your bucket name comes after the endpoint, note:

The request URL would be https://storage.example.com/<bucket_name>/object.txt.
It is equivalent to https://s3-endpoint.example.org/<bucket_name>/object.txt.

Step 3.3 - Create Kubernetes deployment and configuration files (Optional)

Assuming you already have configured Kubernetes, gateway API.

Replace storage.example.com with your own domain.
Replace fsn1.your-objectstorage.com with the endpoint of your object storage bucket. If the bucket name comes after the endpoint (e.g. https://s3-endpoint.example.org/<bucket_name>) add your endpoint without the bucket name.

apiVersion: v1
kind: Service
metadata:
  name: caddy-storage
  namespace: caddy
spec:
  type: ClusterIP
  selector:
    service: caddy-storage
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: caddy-storage
  namespace: caddy
spec:
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      service: caddy-storage
  template:
    metadata:
      labels:
        service: caddy-storage
    spec:
      containers:
        - image: "caddy:latest"
          name: caddy
          ports:
            - name: http
              protocol: TCP
              containerPort: 80
          volumeMounts:
            - name: config-volume
              mountPath: /etc/caddy
      volumes:
        - name: config-volume
          configMap:
            name: caddy-storage-config

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: caddy-storage-config
  namespace: caddy
data:
  Caddyfile: |
    storage.example.com:80 {
      reverse_proxy https://<bucket_name>.fsn1.your-objectstorage.com {
      #reverse_proxy https://s3-endpoint.example.org {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Forwarded-Host {host}
      }
    }

---

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: caddy-storage-route
  namespace: caddy
spec:
  parentRefs:
    - name: kubernetes-gateway
      namespace: istio-gateway
  hostnames:
    - "storage.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: caddy-storage
          port: 80
          weight: 100

Conclusion

You should now be able to access the contents of your S3-compatible object storage via a custom domain.

License: MIT