Sensible backup strategy (using a Synology NAS)

[micropatate]

Hello,

On the one hand, I have a freshly installed peertube instance, on the other hand a Synology NAS. I want to use the latter to backup the former (past experiences have shown me that not having a backup is a bad idea).

So far, my plan, based on this page of the peertube documentation is the following:

1. Create a script that executes the following operations

• use pg_dump to generate a dump of the database
• use rsync on the server to copy to the NAS:
  • the database dump just created
  • the ‹ /var/www/peertube/storage › directory
  • the ‹ /var/www/peertube/config › file
  • the ‹ /etc/letsencrypt/live/<my_peertube_url>/ › directory
  • the ‹ /etc/nginx/sites-available/peertube › directory
  • the few scripts (cron) I have

2. Create a cron that executes this script monthly.

In the peertube documentation pg_dump is called after the database is shut down, but from here I think I can run it also while the database is running (I don’t want to shut down the instance for each backup).

My questions are:

1. Is my plan sensible or is there an obviously better alternative? (For example, I considered setting up a database replica on the NAS, but since I only want to make a backup, and since if the server crashes I am not in a hurry to fix it, this seems like an overkill compared to pg_dump.)
2. Assuming something is changed on the database between the dump and the copy of the storage, I imagine this could lead to an inconsistent database, in particular if a video or a comment is removed between the two. How resilient is the database to this kind of things? Is there a simple way to limit the damages? [Note that I don’t expect a lot of activities on the server, so if I avoid the peak viewing times (that I assume to be the evenings and week-ends in my timezone), this is unlikely to happen at all]
3. Some files are unlikely to change much, like the nginx setup. Would it make more sense to back these up only once? If yes, are there other such files?
4. This solution doesn’t use anything specific to the Synology NAS. Do you know of something specific to it that would be better suited?
5. Have I forgotten anything? (A priori, I don’t plan on making a backup of the systemd config files and startup scripts since I just followed the peertube doc.)

Thanks in advance for your advice.
micropatates

[JohnLivingston]

Hello,

Here are a few remarks in bulk:

• it is not necessary to backup /etc/letsencrypt, as you can recreate everything on the new server
• no need to shutdown posgresql during the dump. Where did you see that it is required? (so we can fix the documentation)
• you can use sudo -Hiu postgres 'pg_dumpall' > your_dump_file.psql to backup all posgresql databases, including internal posgresql database
• you should backup on a daily base, not monthly
• you should use an incremental backup system (see below)
• you should encrypt your backups
• there is also a redis database. But as far as I know, it is not critical for peertube to backup it. The only thing you can loose, is your unfinished jobs (and the job history)

Why an incremental backup system?

Incremental backup are really nice because:

• you can backup daily, it won’t grow too much
• you can easily restore data that were accidently lost. For example:
  • a user that deleted a video by accident => you can get the video back from the backup that was done the day before
  • you made a wrong command and destroyed some data
• it runs really fast, because it only backup changes
• it don’t consume too much bandwidth (except for the first backup)

Why encrypted backups?

If someone steals your NAS, or access to it because of a missing security policy, it could compromise your users private data.

Which incremental backup system?

There are many backup system that both provide incremental and encrypted backups. For example Duplicity.

I personnaly use Borg. Borg is really easy to setup, and is very efficient.

I know there is a Borg client available for Synology. I’m not sure if the Borg Server is available.
If not, you can mount a Synology folder on your server using SSH-FS, and backup in this mounted folder. I am doing this for one of my Peertube instance. It is not optimal, but it works.

To do this sshfs mount, you just have to add this in your /etc/fstab:

synology_user_name@your_synology_domain.tld:/TheSynologyVolumeName       /mnt/backup_synology       fuse.sshfs      port=22,idmap=user,user,noauto,noatime     0 0

You have to create a ssh key on your server, and setup your Synology to accept this ssh key to connect to this user and volume.

Then, you can configure Borg on the Peertube server to backup in /mnt/backup_synology.

How to easily configure it?

There is a nice tool to configure backups, that is able to use Borg, or Duplicity, Rdiff, …
It is called Backupninja.

You just have to configure 1 file per task:

• a file to dump your postgresql database
• a file to do the backup

And it come with a very nice tool: ninjahelper

With ninjahelper, you just have to answer a few questions to configure your backups. And you can test each step!

Important Note: Don’t forget to save your backup passphrase in a safe place! In case your server crashes!

More consistency?

Using incremental backups, the daily cron task will only last a few seconds/minutes.
But you can have very little inconsistency. For example if a new video is added during the backup.
It should not be a problem.
But if you really want your backup to be consistent, and if your server use LVM for its partition, you can use LVM Snapshots to minimize the risk.

Please note that this is probably overkill for your case. I’m just mentionning it «as an advanced tip».

Using ninjabackup, here is how you can achieve this.

First, add a «mount» task, that happens after your psql_dumpall. For exemple, if you have a /etc/backup.d/20.psql file, add a `/etc/backup.d/30.lvm.mount.sh file, containing:

echo "Creating lv snapshot"
lvcreate -s -n snap_backup -L5G /dev/your_lvm_volume

echo "Mounting snapshot"
if [ ! -d /srv/snapshot/ ]; then
    mkdir -p /srv/snapshot/
fi

mount -o ro /dev/your_lvm_volume/snap_backup /srv/snapshot/

Note: I’m not 100% sure that the above file is correct, your have to test it (I copied and modified a file from a slightly different use case, there might be typo)

This file will mount the backup as Read Only.

Then, change your Borg (or Duplicity, or Rdiff, or whatever) configuration to backup directories in /srv/snapshot instead of /.

At last, add a new task after your backup (for example as 95.lvm.umount.sh):

echo "Unmounting snapshot"
umount /srv/snapshot/

echo "Removing snapshot"
lvremove -f /dev/panel_ssd/snap_backup

By doing so, all your files will be «frozen» to a state that is very close to your psql_dumpall (only a few seconds apart), reducing the risk of inconsistency.

[micropatate]

Hi!

@JohnLivingston

it is not necessary to backup /etc/letsencrypt, as you can recreate everything on the new server

Noted. The migration page of the doc (linked in my original post) suggests to do it as an optional step.

@JohnLivingston

no need to shutdown posgresql during the dump. Where did you see that it is required? (so we can fix the documentation)

I didn’t see that it is required, but the migration page of the doc starts with the shut down of the old server so I wasn’t sure if I could do it with the server up. Thanks for confirming my assumption that this is possible!

Thanks a lot also for the rest. I’ll certainly check Borg out and post here if this worked or not.

[JohnLivingston]

@micropatate

I didn’t see that it is required, but the migration page of the doc starts with the shut down of the old server so I wasn’t sure if I could do it with the server up. Thanks for confirming my assumption that this is possible!

Oh ok, I see. This documentation is to migrate from a running server to another. This documentation says to shutdown the old server, so it does not respond anymore to ActivityPub requests (if it still respond, you will loose some updates).
So I can confirm that for backups you don’t have to shutdown anything.

[micropatate]

By the way, it could be helpful to have a dedicated page for backups in the documentation. I understand that there are many possibilities, but a few examples would help, or even a one liner: « Backup: the list of files and folders to backup is the same as that for migration ». It took me a while to stumble upon the migration page.

I’d be happy to contribute to such a page if you think it makes sense.

[JohnLivingston]

@micropatate

I’d be happy to contribute to such a page if you think it makes sense.

Good idea!

Feel free to copy parts of my response if you think it is usefull.

[micropatate]

I’ll certainly do that (but don’t expect it anytime soon, I must first deal with the backup issue and it may take some time).

[Vikarti]

how I do this:
Peertube is just VM on Proxmox instance.
versioned backup is handled by regular backup to Proxmox Backup Server instance on separate old physical machine near server.