Using Restic to backup your self-hosted apps

If you have started self hosting applications, you soon realize “what happens to all of my data if my house (or datacenter) disappeared?”. Having a good backup strategy is an important step in any self-hosted infrastructure.

I’m using this process for backing up my Immich and Syncthing apps (see what I use for more details)
but it can be broadly applied to anything you backup as long as it has one or more file directories containing the data to backup.

3-2-1 Backup Strategy

There are a lot of good articles online about the 3-2-1 backup strategy (such as this one), so I won’t repeat them. I’d say the TLDR is:

have 3 copies of your data - the original data and two backup copies
have one backup copy onsite for quick recovery of a drive or other hardware failure
have one backup copy offsite for disaster recovery if primary location disappears (fire, theft, etc)

Why Restic

There are a lot of good backup tools out there and Restic met the criteria I was looking for.

free and open-source
supports different backend storage types like SFTP and S3
easy to implement encryption
it has been around long enough to trust that issues have been worked out and it will likely continue to be supported

Backup Wrapper Script

I have a few different apps to backup, so I created a central re-usable backup script. This script expects certain environment variables to be set before it is called so I can keep those separate for each app.

Contents of /opt/restic/backup.sh - you should be able to use this without any changes:

#!/bin/bash

# Full path to the restic executable
RESTIC_BIN="/usr/local/bin/restic"

# Function to log messages
log() {
  echo "$(date '+%Y-%m-%d %H:%M:%S'): $1"
}

# Do this inside a block that requires the lock to prevent multiple backups from running at once.
(
  # Confirm we have a lock.
  flock -n 99 || { echo "ERROR - Another instance of the script is already running. Exiting."; exit 1; }
 
  # Backup to both our local and remore repos (if defined). 
  for REPO in $LOCAL_REPO $REMOTE_REPO; do
    # You need to run restic init the first time to create the repository. Use the cat command
    # to check if repo exists as recomended by the scripting guide: https://restic.readthedocs.io/en/latest/075_scripting.html
    "$RESTIC_BIN" -r $REPO cat config > /dev/null
    CAT_EXIT_CODE=$?

    # Exit code 10 is returned if everything works but repo doesn't exist.
    # NOTE: This requires restic version 0.17.0 or above.
    if [ $CAT_EXIT_CODE -eq 10 ]; then
      log "Restic repo ${REPO} appears to be empty - initializing it."
      "$RESTIC_BIN" -r $REPO init
      INIT_EXIT_CODE=$?

      if [ $INIT_EXIT_CODE -ne 0 ]; then
        log "ERROR - Restic init command ${REPO} failed with exit code ${INIT_EXIT_CODE}."
        exit 1
      fi
    elif [ $CAT_EXIT_CODE -ne 0 ]; then
      log "ERROR - Restic cat command for ${REPO} failed with exit code ${CAT_EXIT_CODE}."
      exit 1
    fi

    # Run the backup
    log "**********  Starting Restic backup to ${REPO} **********"
    "$RESTIC_BIN" -r $REPO backup $BACKUP_PATHS --limit-upload 5000 -v --tag "cron-daily"
    BACKUP_EXIT_CODE=$?
    
    if [ $BACKUP_EXIT_CODE -ne 0 ]; then
        log "ERROR - Restic backup command to ${REPO} failed with exit code ${BACKUP_EXIT_CODE}."
        exit 1
    fi
    
    log "Backup to ${REPO} completed successfully."
    
    # Forget old snapshots and prune the repository
    log "**** Starting forget and prune to ${REPO} ****"
    "$RESTIC_BIN" -r $REPO forget $FORGET_POLICY --prune
    FORGET_EXIT_CODE=$?
    
    if [ $FORGET_EXIT_CODE -ne 0 ]; then
        log "ERROR - Restic forget/prune command failed with exit code $FORGET_EXIT_CODE."
        exit 1
    fi
    log "Prune of ${REPO} completed successfully."
  done 
  log "********** Restic backup finished **********"
) 99>"$LOCK_FILE"
exit 0

Environment Files

Then I setup two environment vars files for each app - one which is stored as plain text in source control and one is encrypted to protect the keys. Having most of the settings in plain-text helps to easily view them in Git and track diffs over time.

Contents of /opt/immich/backup-settings.sh (replace any token wrapped with < and > for your environment):

#!/bin/bash

# Backup URLs and Access Key 
export AWS_ACCESS_KEY_ID="<YOUR_KEY_ID>"
export LOCAL_REPO="sftp:<LOCAL_SERVER>:<BACKUP_DIR>"
export REMOTE_REPO="s3:s3.<REGION>.backblazeb2.com/<YOUR_BUCKET>/immich"

# Directories to back up
export BACKUP_PATHS="/<IMMICH_DIRECTORY>/library/backups /<IMMICH_DIRECTORY>/library/library /<IMMICH_DIRECTORY>/library/profile /<IMMICH_DIRECTORY>/library/upload"

# Lock file to prevent multiple instances
export LOCK_FILE="/var/lock/backup-immich.lock"

# Retention policy for 'forget' command
export FORGET_POLICY="--keep-monthly 12 --keep-yearly 5"

Contents of /opt/immich/backup-keys.sh (replace any token wrapped with < and > for your environment):

#!/bin/bash
export AWS_SECRET_ACCESS_KEY="<YOUR_SECRET_KEY>"
export RESTIC_PASSWORD="<YOUR_ENCRYPTION_PASSWORD>"

Running with CRON

This is run daily by dropping a file in /etc/cron.daily.

Contents of /etc/cron.daily/backup-immich:

#!/bin/bash
APP=immich
exec >> "/var/log/backup-${APP}.log" 2>&1

# NOTE: This next line is Immich specific but provides a good idea of how to do something special for a single app 
# before the regular backup runs. This uses "podmad exec" but you can easily replace with "docker exec" if 
# you use Docker.
#
# Backup database right before we run the backup script so our DB and filesystem are as close as possible. Don't 
# compress DB backup so restic can handle deduplicating as much of content as possible.
podman exec -t immich_postgres pg_dumpall --clean --if-exists --username=postgres > /<IMMICH_DIRECTORY>/library/backups/immich-database.sql

# Perform a local and offsite backup via restic backup script.
source /opt/${APP}/backup-settings.sh && source /opt/${APP}/backup-keys.sh && /opt/restic/backup.sh

Note that doing a manual DB backup means you can disable the scheduled Immich backups. This is similar to how Immich recommends running backups in their Borg backup docs.

Troubleshooting

For some reasons it always takes me some time to get cron jobs to run smoothly. It is important to monitor the output of the logs for at least a few days and make sure everything is working. Ideally you should also do a restore test to a test
environment a regular basis to confirm the backup process is working.

Common problems I encounter are:

the path and environment vars in cron jobs are different than your regular shell
cron is running as root instead of your user so SSH keys and other config are different
file permissions on your script files can keep them from executing
typos are not easy to notice

Adding Other Apps

You can basically follow the same approach for any other apps. Create the two environment files and a cron file and you easily expand to backup other items.

3-2-1 Backup Strategy#

Why Restic#

Backup Wrapper Script#

Environment Files#

Running with CRON#

Troubleshooting#

Adding Other Apps#