I have used a number of different backup and high-availability (HA) strategies over the years. These have gotten rather complex especially on the HA side. Therefore, for 2024, I decided to move away from HA and simplify the network. I also reviewed what I was backing up in order to further streamline the backup process and minimize the number of things that were being backed up. The backup strategy for 2024 had the following main goals:
- All irreplaceable files such as photos and home movies can be recoverable from physical or digital destruction such as a fire at the house or a ransomware attack.
- The ability to quickly restore services after a mistake or other issue such as disk corruption causes the service to be unavailable. At most, no more than a day’s worth of data should be lost.
- Be able to support core network features such as DNS and DHCP even if the things usually providing those services are unavailable for an extended length of time. If all my servers are down, I still want to be able to stream music or videos over the Internet, etc.
- Only back up what is needed to support the goals of the overall backup strategy. Having good documentation on how to rebuild things can be a quick way to restore a service without needing a full backup available.
- Optimize power consumption of the network.
The Home Server implementation consists of two nearly-identical computers. In the past, both of these were running 24/7 and actively synchronizing data and services between them. Using software that monitored the various capabilities allowed for things to automatically and nearly seamlessly move between the two servers. For example, if one server was brought down for maintenance, the other server would automatically detect this within a minute, and take over hosting the services that had been running on the other server. While this was interesting to research and implement, it was also complex and at times the systems could get out of sync requiring manual work to fix the issue and rebuild the synchronization. Since I am not running a business, the complexity was overkill as it really did not matter if my web site was not reachable for a period of time. Also, having two computers running full-time used a non-insignificant amount of power; power that could be saved if the secondary server was only turned on to receive backups, or if the primary server would be unavailable for an extended length of time. The backup server could be in a deep sleep most of the time using very little power while using just enough to enable waking it up automatically without needing to physically flip a power switch.
Both servers contain a “cache” area which consists of fast solid state drives (SSDs), and a pool of slower, mechanical hard disk drives. The cache drives hold all the services like Docker containers or virtual machines (VMs) while the slower drives hold all of the media and other files. The cache drives are formatted to use the ZFS file system while the pool drives utilize the XFS file system. The reason ZFS was chosen for the cache drives is that it supports taking snapshots of things stored on those drives without having to shut down the service. The thought was that I could take daily snapshots of my Docker containers and VMs which then could be reloaded as needed to almost instantly return the service to where it was at the time of the snapshot. However, I decided to not use the snapshot approach as it did not meet the goal of only backing up what was strictly needed to restore a service.
Finally, a backup is not really a backup if one does not actually perform a restore to a different platform on a somewhat regular basis. I do this about once every year or two for various things in order to make sure that I could completely lose a server, a phone, or other device and get back to a usable state fairly quickly using my backup strategy and documentation. This also allows me to adjust the hardware, software, and design decisions I use in my life on a somewhat regular basis. While this is the first stab at documenting my current backup strategy, it is something that I have been tweaking for decades, and hopefully will continue to do so for the foreseeable future.
Data Pool
The “data pool” is what I call the local, network-accessible storage where I place all my media files such as pictures, videos, music, TV recordings, and other things that I do not want to store as part of my Windows profile on OneDrive. The technologies and providers used in this solution are as follows:
- unRAID for providing the platform for the data pool.
- unRAID plugin “Dynamix S3 Sleep” for putting the backup server to sleep when it is not being used.
- unRAID plugin “CA User Scripts” for performing the backup using a custom Linux shell script.
- A custom Linux shell script to perform the backup using rsync.
- Power management settings in the server BIOS to “wake up” the backup server at a certain time (the so-called “wake-on-lan” packets would not work with multiple Ethernet ports in a “bonded” configuration that I use on my servers to make four 1G ports as a single 4G network interface).
- CloudBerryBackup Docker image for unRAID to replicate the files in the data pool to an external storage provider.
- Backblaze as an external storage provider.
Every afternoon, when solar power generation is usually at a peak and, on “nice” days at least, more power is being generated than is used by the house and to charge the home’s batteries, the backup server turns on. This is done through a BIOS setting that uses the real-time clock (RTC) to trigger a “wake” command at a specific time each day. 15 minutes after that–to allow for the server to be fully active–various maintenance tasks are fired off in a staggered fashion to update software on the server. Then, a custom script based on rsync is run that reaches out to the primary server and downloads any changes to the data pool. Rsync only downloads (or deletes) things that have changed since the last backup, so this process usually does not take a long time even though there are several terabytes of data in the data pool.
Every night, (early morning), the primary server uses CloudBerryBacup to back up the data pool to Backblaze, a company which provides “cloud”-based storage. This is an offsite backup solution that protects the data in case both the primary and backup servers are destroyed at the same time such as in a fire. Backblaze is only sent new files, modified files, and a list of deletions, so it does not usually consume a lot of time or network use. It also stores several previous versions of the files in case something like a ransomware attack occurs and you need to retrieve the picture, etc. as it was e.g. 10 days ago before the attack. The one big part of the data pool that is not sent to Backblaze are recorded TV shows. These are large files that change frequently, but could viewed in other ways such as a subscription to a streaming service if lost.
Docker Containers
Docker containers are basically stripped-down virtual machines (VMs) that tend to provide one, specific service. These containers usually consist of the application and supporting pieces, along with an area where data and configurations are stored. This allows for the core application to be updated in place when new features, or security or bug fixes, are released without impacting the configuration or data files. This means that the core components do not need to be backed up as they can just be redownloaded and installed as needed. Depending on the service, there may not be data files that need to be backed up. For example, if the service uses a database to store its configuration and data, then reinstalling the container from scratch may only require installing the container from the Internet and pointing it to the database.
Some services distributed as Docker containers may have their own internal backup solution. For example, the software used to provide the web service which you are probably using to read this article has several backup and restore options that one may use. Therefore, to fully restore the web site, the Docker container is installed along with the backup and restore component used. The backup file can then be imported and the web server is back to the state of the last backup.
ChannelsDVR
I use ChannelsDVR to record “over-the-air” television shows. Since these may be watched by signing up with various subscription services, I do not care too much if the recorded videos are lost. However, the ChannelsDVR database, which contains things like the shows I want to record and what I’ve watched, is useful to keep around. Thankfully, ChannelsDVR automatically backs up its database daily, which is stored on the data pool. Unfortunately, it does not allow control over how many backups (or log files) it stores. Therefore, a script was created to run daily and remove database backups and log files that were created more than a certain number of days ago.
CloudBerryBackup
This application is used to synchronize select things stored on the data pool to an offsite storage provider. It is a simple and documented process to restore this manually, and thus no backups are taken of it.
ddclient
This service, which handles dynamic DNS for my domains, is a simple and documented configuration process that only takes a few minutes to restore manually. Therefore no backups of this are taken.
Emby
The Emby software is used for distributing media such as pictures and video over a network. It has an internal process that automatically backs up its configuration data to the data pool of the server.
FreshRSS
FreshRSS is a news aggregation service which uses a database for most of its configuration and data storage. Therefore, there is no need to back this up as a reinstall requires only to install the Docker container and point it to the database.
MariaDB
This is a database that is used by other services on the network such as the web site and RSS news aggregator. There are a couple main ways to back up the database: an image backup or a logical one. The image backup approach is basically a snapshot of the data store used by the database while the logical backup creates a set of plain text files that can be used to rebuild the database. The benefits of using a logical backup are: 1) the database could be rebuild on a different database such as MySQL if I ever chose to change what database software I used, and 2) the files could be used if only certain things within the database need to be restored without doing a full restore of everything.
Since MariaDB is in a Docker container, the easiest way I saw to implement backups was to have a script that wrote the database backup files to the data pool every night. To implement this I used the following technologies:
- BorgBackup (formerly automysqlbackup) – Software that can be configured to take and manage backups of the database.
- Borgmatic – Software that adds features to BorgBackup such as managing backups of MariaDB.
- mysqldump – An application that is used by Borgmatic (and automysqlbackup) to communicate with the database.
- cron – A tool used to automatically run Borgmatic on a certain schedule.
The first step was to create a “Backups” folder (share) on the main drive pools holding my data. Once this was done, a virtual machine (VM) was created to connect to both this “Backups” folder and MariaDB. It was then set up to run Borkmatic every night to back up all the data in the database and store it on the data pool in an encrypted format. This process was timed so that it wrote the data prior to the data pool being sent to Backblaze. This may mean that a couple hours more than a day of data may be lost, which slightly violates one of the original goals of “no more than a day’s worth of data should be lost”.
PlexMediaServer
This software, used for distributing media such as pictures and video over a network, has an internal process that automatically backs up its configuration data to the data pool of the server. (This may be replaced by Emby or another service that replaces both.)
WordPress
WordPress is (currently) used as a platform for web sites such as https://karylstein.com/. It uses both a database and local files. The database has its own backup process (see the section MariaDB above), but files such as images and customizations also need to be backed up. The following technologies are used to implement this:
- Appdata Backup – A plug-in for unRAID that allows automated backing up of Docker containers.
While there are plug-ins for WordPress that provide backup capabilities, it is not easy to perform automated backups without some sort of subscription. Therefore the Appdata Backup plug-in was chosen only to back up the WordPress Docker container. The downside of using Appdata Backup is that it stops the container before taking a backup, meaning that the WordPress web sites I host are unavailable briefly while the backup is taking place. This is not enough of a problem for me to pursue a different backup strategy. In the future, I may automatically redirect clients using the web sites to something that says maintenance is occurring and to reload the page in a couple minutes.
Linux
I have a couple of Linux-based virtual machines on the home server that manage things like firewalls, network services, and home automation. While the creation and configuration of these servers is well-documented, sometimes configurations change without the documentation being updated–my bad. Therefore, backups of certain directories that contain configuration files, (such as the /etc directory), are desired. To implement this the following technologies were used:
- BorgBackup – Software that can be configured to take and manage backups of files.
- Borgmatic – Software that adds features to BorgBackup.
- cron – A tool used to automatically run Borgmatic on a certain schedule.
BorgBackup implements deduplication and compression, which can save space. It also handles encrypting the files as they may contain sensitive data. Cron is used to trigger a backup on a daily basis. If a file is modified or deleted, the original file remains in the backup for a certain period of time, which is also the case on Backblaze.
unRAID
While most of the main portions of the unRAID server such as the data pool, Docker containers, and Linux Virtual Machines, are already backed up using the methods described in this document, there is still one piece that should be backed up: the boot drive. The Appdata Backup plug-in, (also used for backing up WordPress), provides this functionality.
Windows
In the past, I have not wanted to use cloud storage services such as OneDrive for storing personal data. However, over time, my aversion to putting my data on cloud storage has reduced significantly. I had looked into some cloud services that I could host on my home network, but the ease of using something like OneDrive was much more appealing. Some things such as electronic copies of my tax returns or things that contain sensitive data I do not make available on cloud storage, but a vast majority of files that I have are things that are either already public such as on my website, or things that I would not mind if they were public.
As I’ve said before, I tend to reinstall my devices every year or two just to clean out any cruft or things I no longer use. Therefore, I have a set of documentation that I can follow to fairly easily rebuild and reconfigure my computer, phone, etc. from scratch. So, my backup strategy for Windows PCs these days is to just store most of my files on OneDrive and use documentation if I need to rebuild anything else. By using OneDrive and linking it to the account I use to log into the PC, any Windows PC I use will automatically have all my personal files available, assuming it has a connection to the Internet. I set my personal files to be downloaded and stored locally, so they are still available when I do not have an Internet connection, and any modifications I make are then synchronized when the computer is back online.
I can also access my files from other devices such as my Android tablet or phone. While I do not store those files locally on those devices due to space restrictions, I can for example, download some files before a plane ride, edit them while on the plane, then they will sync back to OneDrive when my table has Internet connectivity once again.
Andriod / i* Device
While I am currently mostly an Android user for phones and other hardware such as watches or earbuds, (and a specific brand at that), I have a collection of devices that I chose for one reason or another. There are pros and cons to using the same or different ecosystems for your hardware and software, but, as of today, I have settled on the following:
- Personal Contacts / Personal Calendar: Google
- Work Contacts / Work Calendar: (restricted) iCloud
- Personal Phone / Personal Tablet: Samsung
- Work Phone: (restricted) iCloud
- Personal Phone / Personal Tablet Photos: Microsoft OneDrive
- Work Phone Photos: (restricted) iCloud
TODO
- Set up nginx to redirect to “maintenance” screen if the web server is not responding