Install Dify on Linux with automated backups

Introduction

In this tutorial, I will guide you through the process of installing Dify, an open-source LLMOps platform, on a Linux server (Ubuntu, Debian, or CentOS). I will also set up automated backups using Restic and S3-compatible object storage (e.g. Cloudflare R2). By the end of this tutorial, you'll have a fully functional Dify instance running on your server with regular backups.

Dify is a powerful tool that allows you to create, deploy, and manage AI applications. It provides a user-friendly interface for building AI applications without extensive coding knowledge, integrating various AI models and services.

This setup will give you a robust environment for developing and running AI applications.

Prerequisites

Before we begin, you'll need:

A Linux server (Ubuntu, Debian, or CentOS) and access to a user with sudo permissions.
A domain name pointed to your server's IP address.
An SSL certificate for your domain name (e.g. with Cloudflare)

To generate an SSL certificate with Cloudflare, sign up and:
1. Buy a domain or point your existing domain to Cloudflare nameservers.
2. Generate an SSL certificate by selecting your domain and navigating to SSL/TLS » Origin Server » Create Certificate.

We'll use Ubuntu as the example operating system, but the steps should be similar for Debian and CentOS.

Step 1 - Preparing the Server

First, we'll update the system, install Docker and Docker Compose, and set up Git. This single command will do all of that for us:

sudo apt update -y && \
sudo apt upgrade -y && \
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common gnupg lsb-release && \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg && \
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \
sudo apt-get update -y && \
sudo apt-get install -y docker-ce && \
sudo usermod -aG docker $USER && \
sudo curl -L https://github.com/docker/compose/releases/download/v2.26.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose && \
sudo chmod +x /usr/local/bin/docker-compose && \
sudo apt install git -y

This command does the following:

Updates the system package list and upgrades installed packages
Installs necessary dependencies
Adds Docker's official GPG key and repository
Installs Docker and Docker Compose
Adds your user to the Docker group
Installs Git

To check, you can run:

docker --version && docker compose version && git version

You might need to log out and back in to update your groups.

Step 2 - Cloning the Dify Repository

Now, let's clone the Dify repository:

cd ~ && git clone https://github.com/langgenius/dify.git

This command downloads the Dify source code to your server.

Step 3 - Configuring Dify

Next, we'll set up the configuration file:

cd ~/dify/docker
cp .env.example .env
nano .env

These commands:

Change to the Dify docker directory
Create a copy of the example environment file
Open the new .env file in the nano text editor

In the .env file, customize the following variables:

Update domain variables

APP_WEB_URL=your_domain_name
NGINX_HTTPS_ENABLED=true

Update certificate variables

If you want to use your own certificate, you can keep the default values and add the respective key files in the next step.

If you want to use the Cloudflare certificate, update the values of the certificate variables like this:
```
NGINX_SSL_CERT_FILENAME=cert.pem
NGINX_SSL_CERT_KEY_FILENAME=key.pem
```

Update additional variables

Review other variables in the .env file as well, and adjust them according to your needs.

After making changes, press Ctrl+S to save and Ctrl+X to exit the nano editor.

Step 4 - Setting Up SSL Certificates

Let's add the SSL certificate.

If you use your own certificate, you can add it in dify.crt and dify.key.

If you use the Cloudflare certificate, you should add it in cert.pem and key.pem.

Own certificate

Create two new files:
```
nano ~/dify/docker/nginx/ssl/dify.crt
nano ~/dify/docker/nginx/ssl/dify.key
```
Copy and paste your certificate and key into those files. Save and exit (Ctrl+S, Ctrl+X).

Cloudflare certificate
```
nano ~/dify/docker/nginx/ssl/cert.pem
```
Copy and paste the Cloudflare certificate content into this file. Save and exit (Ctrl+S, Ctrl+X).
```
nano ~/dify/docker/nginx/ssl/key.pem
```
Copy and paste the Cloudflare SSL key content into this file. Save and exit (Ctrl+S, Ctrl+X).

Step 5 - Deploying Dify

We're now ready to deploy Dify:

cd ~/dify/docker
docker compose up -d

This command starts all the necessary Docker containers for Dify. The -d flag runs the containers in detached mode (in the background).

After a minute or so, you should be able to access your Dify instance by visiting your domain in a web browser.

You can run docker ps -a to check the status of the containers.

Step 6 - Setting Up Automated Backups

After deploying Dify, let's set up automated backups using Restic and S3-compatible object storage (e.g. Cloudflare R2). First, we need to switch to the root user:

sudo su
cd ~

Step 6.1 - Install Restic

The commands above bring us to the root directory. Now, let's install Restic:

sudo apt update
sudo apt install restic

Step 6.2 - Prepare Your s3-compatible Bucket

If you haven't already, create a new bucket with your preferred provider or an R2 bucket in your Cloudflare account. Note down the following information:

Bucket Name
Access Key ID
Secret Access Key
Endpoint URL

With Cloudflare, you can find this information in your Cloudflare dashboard under R2 » Overview and R2 » Manage R2 API Tokens.

Step 6.3 - Create the Backup Script

Create a new directory and backup script:

mkdir -p /root/restic
nano /root/restic/backup.sh

Copy and paste the following content into the file:

#!/bin/bash

# Set environment variables for Restic
export RESTIC_REPOSITORY="s3:YOUR_S3_ENDPOINT_URL/YOUR_BUCKET_NAME"
export AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY"
export RESTIC_PASSWORD_FILE="/etc/restic-password"

# Set the backup source
# Replace "YOUR_USER" with the user that installed Dify
BACKUP_SOURCE="/home/YOUR_USER/dify/docker/volumes"

# Perform the backup
restic backup $BACKUP_SOURCE

# Prune old snapshots (keep last 7 daily, last 4 weekly, and last 12 monthly backups)
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune

# Check the repository for errors
restic check

Replace the first three placeholders with your actual S3 bucket information, and "YOUR_USER" with the user that installed Dify. Save and exit the editor (Ctrl+S, Ctrl+X).

Make the script executable and secure:

chmod +x /root/restic/backup.sh
chmod 700 /root/restic/backup.sh

Step 6.4 - Initialize the Restic Repository

Set the environment variables and initialize the repository:

export RESTIC_REPOSITORY="s3:YOUR_S3_ENDPOINT_URL/YOUR_BUCKET_NAME"
export AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY"
restic init

Save the password in a new file for the cron job:

echo "YOUR_PASSWORD" > /etc/restic-password
chmod 600 /etc/restic-password

These commands:

Save the restic password in a new file
Restrict the file's access to only your user

Step 6.5 - Set Up Daily Backups

Open the crontab file:

crontab -e

Add the following line at the end of the file:

0 3 * * * /root/restic/backup.sh > /root/restic/backup.log 2>&1

This sets up a daily backup at 3 AM. Save and exit the editor.

Step 7 - Verifying the Setup

To ensure everything is set up correctly:

Run the backup script manually:
```
/root/restic/backup.sh
```
Check the Restic repository for snapshots:
```
restic snapshots
```

Step 8 - Upgrading Dify in the Future

To upgrade Dify in the future, run these commands:

cd ~/dify/docker
docker compose down
git pull origin main
docker compose pull
docker compose up -d

This will stop the current containers, pull the latest changes, download updated images, and start the new containers.

Conclusion

Congratulations! You've successfully installed Dify on your Linux server, set up SSL certificates, and configured automated backups. Your Dify instance is now ready for use, with daily backups ensuring your data is safe.

To list available snapshots:

restic snapshots

To restore from a backup if needed:

restic restore latest --target /path/to/restore

Replace /path/to/restore with the directory where you want to restore the data.

Make sure to securely store your Restic repository password, as it's crucial for accessing your backups.

Remember to regularly check your backups and keep your system updated for optimal performance and security.