Backup Automation With Paperless-ngx and Rclone

Regularly creating backups in your paperless office, especially when using Paperless-ngx, is indispensable for several reasons:

• Data recovery: All your documents, including important contracts, invoices, and personal records, are stored digitally. Regular backups protect against data loss due to hardware failures, software issues, and accidental deletion. In the event of natural disasters, theft, or cyberattacks, you can quickly restore your data and maintain your business operations.
• Legal requirements: You may be legally required to retain certain documents for a specific duration. Through backups, you ensure that you have the documents available for legal processes or audits.
• Version control: With regular backups, you can create a chronological archive of your documents. This allows you to access older versions of your documents, which is useful for tracking changes.

The 3-2-1 Backup Rule

You may have already heard of the 3-2-1 backup rule. It’s a recommended strategy for creating backups and ensuring high data security. The rule is simple but effective.

The three criteria:

• 3 copies of your data: Create at least three separate copies of your data. This includes your original data in Paperless and two additional backups. If one backup fails, you have another option to restore your data.
• 2 different devices: Store these backups on at least two distinct devices. These can be hard drives or cloud storage. Different storage technologies have different vulnerabilities, and diversifying across multiple types reduces device-specific risks.
• 1 off-site location: At least one of the backups should be stored at a different location. This protects your data in the event of natural disasters or theft at your main office. Cloud storage is often used for off-site backups because you can access it from anywhere.

Implementing the 3-2-1 backup rule helps you effectively protect your data by diversifying across multiple storage types and locations.

If you want to follow the 3-2-1 backup rule, you can now consider where you want to secure your data.

I will show you an example of setting up a Paperless backup automation from Linux to Google Drive using the open-source tool rclone.

Most NAS or VPS run a Linux distribution. Therefore, the steps should be quite similar regardless of the device.

What is rclone?

Rclone is a program for creating backups on cloud storage, running on Linux, macOS, and Windows, and supports over 70 cloud providers (Google Drive, OneDrive, Dropbox, Amazon S3, etc.)

Rclone mirrors your data from your server to the cloud storage. Thus, not everything needs to be re-uploaded with each change. To prevent the cloud provider from reading your data, we will also encrypt the backup with rclone.

Requirements

The prerequisite for this guide is that you have already successfully installed Paperless-ngx and fed it with some documents. You also need an account with a cloud provider. Here I use Google Drive. The following steps are very similar and sometimes simpler for other cloud providers.

Paperless Document Exporter

There are various ways to make Paperless-ngx backups. I recommend using the native “Paperless Document Exporter” & “Importer” instead of simply copying your entire Paperless folders. Since Paperless is probably running in Docker for you, some of the data might be in Docker volumes. You can run the Document Exporter with just one command, and it exports not only all files but also any configurations, user accounts, tags, and everything the algorithm has learned.

Log in to your server via SSH and execute the command to install rclone via snap:

The snap store daemon must be installed for this (it is usually pre-installed on Ubuntu). If the command above fails, install the snap store daemon first:

For this guide, it’s important that rclone is installed on a device with web browser access (in addition) to retrieve a token for connecting with the cloud provider (see Step 3).

Since I am using a VPS without a web browser, I install rclone additionally on my Mac. For this, I use the package manager homebrew:

You can find downloads for other operating systems here.

Log into the Google API Console. It must be the same account that you want to use for Google Drive.

Select a project or create a new project.

Under APIs & Services, click on + ENABLE APIS AND SERVICES and search for drive, then activate the Google Drive API.

Click on Credentials in the left sidebar, then click on CONFIGURE CONSENT SCREEN.

Select External and click CREATE. Then enter an app name, the user support email (your own email is okay), and the developer contact information (your own email is okay). Click SAVE AND CONTINUE (all other fields are optional).

Click on ADD OR REMOVE SCOPES. Add the scopes with the scope .../auth/docs, .../auth/drive, and .../auth/drive.metadata.readonly so that you can edit, create, and delete files with rclone. After that, click UPDATE and then SAVE AND CONTINUE.

Add your own account as a test user. Then click SAVE AND CONTINUE.

Then click on Credentials in the left sidebar again. Click on + CREATE CREDENTIALS and select OAuth client ID.

Select Desktop app as the application type and click CREATE.

Now you will see the Client ID and Client secret. Copy both! We will need these in Step 3 for rclone.

Click on OAuth consent screen in the left sidebar, then click on PUBLISH APP and confirm.

Go back to your terminal and create a new remote connection with rclone. The command for this is:

Type n to create a new remote. Then enter a name for the remote (e.g., gdrive).

After that, select the cloud provider. For this, enter the correct number (17 for Google Drive).

Next, enter the Client ID from Step 2:

Next, enter the Client Secret from Step 2:

Choose the access permissions for rclone by entering a number (for the backup to work, number 3 is sufficient).

For the next step, you can leave it blank by pressing ENTER.

After that, select the default by pressing ENTER.

If your server does not have a web browser, enter n.

Copy the entire line that starts with rclone authorize “drive” and execute it on your computer. To do this, open a new tab in your terminal. Your web browser should launch, and you can log into your Google account. Then copy the token and paste it back into your server terminal.

Select the default by pressing ENTER:

You will see a summary of your configuration. Press ENTER.

Now the connection to your Google Drive account is established.

We will now create another remote. This remote will form an encryption layer over the cloud storage remote that we just created. If you do not wish to encrypt the backup, you can skip this step and proceed to Step 5. The advantage of an unencrypted backup is that you can use Google Drive OCR search to quickly find your documents in Google Drive.

Start the configuration again with rclone config and create a new remote (n). You could name it, for example, gdrive_encrypted.

Next, select crypt from the list of providers by entering number 13.

Then you specify the name of the Google Drive remote along with the desired path where the backup should be stored. My remote is called gdrive and I want to store the backup in a folder named paperless_backup_encrypted. So, I enter gdrive:paperless_backup_encrypted:

Then you select the encryption of the file names. I choose the default by pressing ENTER:

Then you select the encryption of the folder names. I choose the default again:

Then you set two passwords for the encryption. I let the passwords be generated with the maximum length.

Copy the passwords and save them in a secure place. This completes the configuration of the crypt remote. Confirm with ENTER and terminate the process with CTRL+C.

I’ll now show you how to create a script that will create a backup for you when executed. Go to the folder where you want to save the script. In my case, I’m going to the paperless-ngx folder:

Then create a new file:

Then paste the script. My script is deliberately kept very simple. You can copy it, but you need to adjust the absolute path to your docker-compose.yml file and your export folder:

Save and close the file afterwards with CTRL+C, Y, and ENTER.

To make the script executable, enter this command:

Test now if your script works by executing it via the command line:

Depending on the amount of data, it may take several minutes. The process should terminate automatically.

If you now open Google Drive in your web browser, you should find a new folder in your storage. In this folder, you will find the encrypted data from Paperless.

Now we will create a new cron job that automatically executes the script at a defined time:

If the script, for example, is to be executed daily at 3:00 a.m., add the following line at the end of the file:

Here you need to specify the absolute path to your backup script and the path where the log file should be created. Save and close the file. When you execute crontab -l, you should see a list of your cron jobs including the added line.

First and foremost, make sure to securely store the passwords for the backup encryption. If your server becomes unreachable, the backup will be useless if you cannot decrypt it.

If your server has failed and you don’t have a snapshot, you’ll need to reinstall rclone and Paperless on your server. Follow the same steps and then don’t choose a new password for the crypt remote, but instead, enter the previous passwords. This way, rclone can decrypt your backup again.

Afterwards, execute the rclone sync command to import the backup:

Once your Paperless export folder is filled again, you can restore your Paperless instance with the Document Importer:

The presented Paperless backup automation can be similarly modeled with other cloud providers. Typically, an API key or client ID with a client secret is sufficient for authentication.

When you run the Document Exporter, the export folder in your Paperless directory contains everything to restore your instance.

If you want to make backups to local hard drives, you can simply copy the contents of the export folder. You can use tools like rsync paired with a cron job for this purpose. Some NAS operating systems already have integrated local backup functions.