Backups

Identifying backup targets

First off, what needs backing up?

UDM configuration extraction can be handled via a cronjob that SCPs the relevant backup files from the UDM directly into FreeNAS. Something like scp root@unifi:/usr/lib/unifi/data/backup/autobackup/* /mnt/Pool/Hosts/unifi/Backups/ will work. So long as we’re monitoring the success of cronjobs (monitoring will come shortly), that should be fine.

Proxmox hypervisor will need file-level backups, as will the VMs. Scheduled VM snapshots can be handled by Proxmox (Datacenter -> Backup -> Add), which puts its dumps into /var/lib/vz/dump/, which can be symlinked to a FreeNAS mount.

FreeNAS DB backups can be handled by a cronjob which copies the databases out of the host and over to the hypervisor: scp /data/*.db root@system-apps:/root/freenas-db-backup/. At least for the time being that’ll allow for fast recovery. Offsite backups are still going to be necessary, but that can come later.

Lastly, laptop backups in our household can be handled by Apple Time Machine (Peter Hanneman’s article on FreeNAS + Time Machine covers how one sets this up).

File-level backup software

There are a lot of file-level backup systems out there. Wikipedia documents them in two pages (first, second), /r/Backup has strong opinions, Restic has made a decent effort in cataloguing its competition.

The key attributes that are going to be important for us are

After spending far too much time reading forum posts and comparison pages, I’ve decided to give Borg a try, augmented by Borgmatic to help simplify a lot of the interface.

Restoring a single file

The one feature that Borgmatic doesn’t have (at least not in the version I’m using) is the ability to diff a current path against a backed up path and restore it on demand. What follows is a quick proof-of-concept that does just that.

#!/usr/bin/env python3
import time
import argparse
import json
import logging
import os
import subprocess
import tempfile


def run(*args, **kwargs):
    return subprocess.run(*args, **kwargs, check=True, capture_output=True).stdout.decode()


def get_preferred_archive():
    logger.debug("Retrieving archives through borgmatic")
    archives = json.loads(run(["borgmatic", "--list", "--json"]))[0]["archives"]
    archives = sorted(archives, key=lambda x: x["start"], reverse=True)

    logger.debug("Presenting archive choices")
    for index, archive in enumerate(archives):
        print(f"""({index}) {archive["start"]}""")
    index = int(input("\nWhich archive would you like to consider? "))
    return archives[index]


def get_info():
    logger.debug("Retrieving repository info from borgmatic")
    info = json.loads(run(["borgmatic", "--info", "--json"]))

    # Why is info returning potentially multiple results? Might the call
    # to `borgmatic --info --json` ever _actually_ return more than one
    # result?
    if len(info) != 1:
        raise NotImplementedError("This script doesn't know how to handle multiple results from `borgmatic --info --json`")

    return info[0]


def compare_to_archive(info, archive, live_path):
    with tempfile.TemporaryDirectory() as tmpdir:

        logger.debug(f"Extracting requested path to {tmpdir}")
        composite_name = f"""{info["repository"]["location"]}::{archive["name"]}"""
        run([
            "borg", "extract",
            composite_name,
            live_path[1:],
        ], cwd=tmpdir)

        logger.debug(f"Diffing {live_path} with extracted files")
        extracted_path = os.path.join(tmpdir, live_path[1:])
        diff_command = ["diff", "--color", "-ur", live_path, extracted_path]
        response = subprocess.run(diff_command)

        if response.returncode == 0:
            logger.warning("No difference")
        elif response.returncode == 1:
            offer_replacement(live_path, extracted_path, composite_name)
        else:
            raise Exception("""Unexpected exit code from diff command {diff_command}""")


def offer_replacement(live_path, extracted_path, composite_name):
    response = input(f"\nWould you like to restore `{live_path}` from `{composite_name}`? (y/N)")
    if response.lower() in ("y", "yes"):

        logger.debug("Moving old path out of the way")
        run(["mv", live_path, f"""{live_path}.{time.time()}"""])

        logger.info("Moving restored archive into position")
        run(["mv", extracted_path, live_path])


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("live_path")
    parser.add_argument("--verbose", "-v", action="store_true")
    args = parser.parse_args()

    logging.basicConfig(level=logging.DEBUG if args.verbose else logging.INFO)
    logger = logging.getLogger()

    archive = get_preferred_archive()
    info = get_info()
    compare_to_archive(info, archive, os.path.abspath(args.live_path))

I should probably test and lint this file properly, but for the time being it serves as a demonstration of the kind of functionality I’d like. I’ll revisit this later. Too many things to tackle and not enough time!