5209R/5210R/5211R: Easy-Backup

We're happy to announce the release of Easy-Backup, our complete and detailed backup and restore solution for BlueOnyx.

Easy-Backup is a commercial product available for BlueOnyx in the BlueOnyx-Shop. It allows the server administrator to easily back up and restore all relevant data, including Vsites, Users, DNS, SQL, Emails, settings and more. Easy-Backup consists of a CLI command (/usr/sausalito/sbin/easy-backup) and a GUI that tightly integrates with the existing BlueOnyx GUI interface.

Please note: Easy-Backup replaces the product "Automated Backup". Clients who in the past bought the "Automated Backup" are eligible to link and install "Easy-Backup", provided their support did not expire prior to the release date of Easy-Backup (2023-08-30). If your support expired before that date, you will only see the older "Automated Backup" as available product instead.

How does it work?

Easy-Backup uses similar methods as "Easy-Migrate" to retrieve and transfer data and settings from users. But instead of direct transfer of these settings via CCE set transactions and transfer of the data via RSYNC, it generates an export data set. This consists of JSON files for the settings and tarballs for the data of the Vsites, users, DNS and SQL. Additionally, there is a checksum file which is used to check the data integrity of the dataset.

This export dataset is backed up to a remote machine using the Linux tool "Duplicity". "Duplicity" can handle various protocols (SSH, SCP, RSYNC, FTP, FTPS, SFTP). Storage via SSHfs, Samba, NFS and some "clouds" (AWS, Azure, Dropbox, MegaNZ, Google Drive) is also possible in principle, but are not implemented to avoid various complications.

With "Duplicity", you can also store data sets in many generations quite easily and in a relatively space-saving way, because it can do incremental backups in addition to full backups and then also stores them smartly and in a space-saving way. It also takes away the work of managing the aging and recombination of full backups and increments yourself when the number of desired increments is reached.

If you choose incremental storage of backups, you can either restore from the latest available backup, or from any of the previous increments. So if you have seven days of backups, you could restore the backup from any of these previous days, going as far back as seven days.

GUI-Integration:

Under "Server Management" / "Maintenance" / "Easy-Backup" you can schedule when the backup runs and where it stores the backups.

At the bottom of the above image you see four options that define if and how Easy-Backup is available to siteAdmin's of Vsite and regular Users of Vsites.

Show Backup-List

This defines if siteAdmin's and Users can see which backups are available. Or you can choose to not show them that list.

Allow Remote Restore

This defines if siteAdmins and/or Users can trigger a restore using the data on the remote backup server. This will connect to the remote storage server and will synchronize the local backup storage with the remote backup storage for the given age of the restored backup set. This may require the transfer of a lot of data between the backup server and your BlueOnyx, so it may not be desirable to have this enabled. If you disable this feature, but still give siteAdmins and Users the ability to restore their data? In that case only the local backup storage will be used for restoring from whatever data is store locally. This means they will only be able to restore from the backup set that was last exported.

Allow siteAdmins to restore

If enabled, all siteAdmins have the ability to restore their own Vsite and/or Users from the backup, given the constrains imposed by "Allow Remote Restore" mentioned above.

Allow User Restore

This defines if individual Users who are not siteAdmin of a Vsite have the right to restore their user account and settings from the local or remote backup. Again limited by the constrains imposed by "Allow Remote Restore" mentioned above.

Storage of Backups

Exports first go to a local directory and are then transferred to the specified remote destination. Optionally local backups can be deleted after this transfer has completed successfully. You can choose to do incremental backups (default), or to do full backup every time. Incremental is recommended for saving space.You can also specify how many days worth of backups you want to keep.

In the tab "Backup-Status" you can see the status of past backup runs and this tells you not only when backups were performed, but also if those backups were full backups or incrementals.

In the tab "Restore Vsites" you will find the option to restore all or selected Vsites from a local or remote backup. So even if you had to wipe the server and reinstalled BlueOnyx, you can simply reinstall Easy-Backup and configure it to connect to the remote storage where you have your backups. Hit the button "Poll Backup Vsite List" and it will connect to the remote storage server and will present you the list of Vsites that are contained in the externally stored backup. You can then also choose to either restore the latest backup, an older version and choose which Vsites you want to restore.

Simply mark the Vsites and hit save and the restore from the remote backup will commence. Or if no remote backup is configured, it will use a local backup (if present!) instead.

It should be mentioned that restoring many or very large Vsites is perhaps best performed from the command line. The command for this is:

/usr/sausalito/sbin/easy-backup --restore --vsites all

That would restore all Vsites (and their users and associated data and settings!) from the remote backup (if configured) or the local backup (if present). Of course you can also choose to only restore selected Vsites or could give them a different IP address on import:

/usr/sausalito/sbin/easy-backup --restore --vsites www.blueonyx.it,support.blueonyx.it --ip 208.77.151.215

The above would only restore the specified two Vsites and would assign them the IP address given on the command line.

After a GUI initiated restore process is complete, you will be shown the relevant results that happened on the command line when the restore was performed. That may look like below.

 Or if an error happened, you will also see this report and the errors that happened during the restore attempt.

Further Easy-backup GUI pages for Admin-Users

Even if you do not allow siteAdmins (or users) the ability to perform their own restores from the backup: As server-admin you still have the right and the ability to perform these restores yourself. For example under "Site Management" / <Vsite> / "Maintenance" / "Easy-Backup" of each Vsite you can click on "Restore Vsite" to restore the entire Vsite and its Users (and also the configured SQL databases and DNS!) from the latest backup or an older backup set:

Likewise: If you need to restore an individual Vsite user to a specific backup state? Go to the User List of the Vsite, choose the User in question and tick the checkbox "Restore User". If you like, also adjust the "Restore Age" to restore this particular user from and older backup.

Of course this only works if the User is still present. If he was (intentionally or accidentally) deleted, the restore of that users is only possible from the command line:

/usr/sausalito/sbin/easy-backup --restore --user <username> --age 5

That particular command would only restore the user <username> from the backup set that was generated five days ago. If you want to restore him from the latest backup, then of course you would omit the --age switch. Easy-backup knows which Vsite the user belongs to, so he will automatically be restored to the correct Vsite.

siteAdmin / regular User views

If granted that privilege, siteAdmins and regular Users have the ability to restore their own data (and settings) from the backup as well. Below are two screenshots of how that looks, but in principle it is the same format as for admin-users. Possibly with the exception of seeing the list of available backups and/or the ability to restore from the remote backups.

siteAdmin view for restoring the entire Vsite:

siteAdmin just restored his own account:

Command Line Usage

You (as server administrator) of course have the ability to use the full power of Easy-Migrate directly form the command line as "root":

~# /usr/sausalito/sbin/easy-backup  
┌─────────────────────────────────────────────────────────┐ 
│                   Easy-Backup v1.0.7                    │ 
│       Easily Backup and Restore BlueOnyx Servers        │ 
│                                                         │ 
│ Copyright (c) 2008-2023 Michael Stauber, SOLARSPEED.NET │ 
│       Proprietary Software - All Rights Reserved        │ 
└─────────────────────────────────────────────────────────┘ 
 
[ERROR] Please specify some command line options. 
 
usage:     easy-backup.pl [OPTIONS] 
 
Example backup:  ./easy-backup.pl --backup  --vsite all --admins 
Example restore: ./easy-backup.pl --restore --vsite all 
 
   --backup        Use this to perform a backup. 
   --restore       Use this to restore a backup. Will restore the latest backup, unless 
                   you want to restore an older backup. In that case add the --age switch 
                   and specify the number of days to go back in the backup-set. 
                   want to go back in the backup-set in case you want to restore a backup 
                   older than the latest one.  
   --age           Only use this in conjunction with the --restore switch. Add a number to 
                   specify how many days you want to go back in the backup-set to restore 
                   an older backup. 
   --dir           Overrides local storage directory, which is specified in the config file. 
   --config        Exports or restores service and system settings such as Email, DNS, 
                   Apache, PHP, Nginx, Vsite- and User-defaults, SNMPd, Sitestats and 
                   Support settings. It will NOT touch the SSH or AdmServ settings  
                   for security reasons. 
   --list          Human readable list of Vsites present in a backup dump stored in  
                   the backup directory specified via the --dir switch. 
   --remotelist    Human readable list of Vsites present in the remotely store backup. 
   --ip            IPv4 IP Address that newly restored Vsites should get on import. 
   --ipv6          IPv6 IP Address that newly restored Vsites should get on import. 
   --nodns         If Vsites are restored, the we will not import their DNS records. 
   --dnsonly       Restore only the DNS records, but nothing else. 
   --mysql         Only restore all MySQL databases and users, but nothing else. 
   --vsite         To only backup or restore import a single Vsite specify   
                   --vsite www.domain.com. To backup or restore multiple Vsites 
                   specify --vsite www.domain1.com,www.domain2.com. To backup or 
                   restore all Vsites and Users use --vsite all 
   --admins        Will export/import all server administrator users as well.  
                   User 'admin' and 'alter-admin' will be exported, but for security 
                   reasons the import of their accounts is not supported. 
   --user          Allows to restore a single user account from the backup set.  
                   The Vsite he belongs to must already exist.  
                   Example: --restore --user john_doe.  
   --check         If remote backup is configured, you can use this option to check 
                   the status and availability of your remote backups. 
   --unlock        Allows to unlock 'easy-migrate' if a restore lock was left as set. 
   --license       Show the software license of this script. 
   --version       Show Version of this script. 
   -h|--help       This help text

The Easy-Backup command reads the GUI created configuration file /etc/easy-backup.conf, which has the information about which local directory you export to (default: /home/easy-backup), which remote server to connect to for remote storage, which user-name to use for that and which transfer protocol should be used.

Available protocols:

• ftp (requires password)
• ftps (requires password)
• rsync (over SSH - requires SSH key exchange!)
• scp (over SSH, no password needed if SSH key is exchanged)
• sftp (over SSH, requires password!)

The recommended option is SCP or RSYNC and be sure to exchange the key of user "root" on your BlueOnyx with the account of the backup user on the remote storage server. In that case you don't need to enter a password for that in the GUI on the page where you configure the remote storage. Any other storage protocol requires that you save the password of the backup account from the remote storage server in your GUI.

Full backup of the server:

~# /usr/sausalito/sbin/easy-backup --backup --admins

That will backup the servers configuration, will export all Vsites, Users, DNS (and DNS include files), user cronjobs, MySQL/MariaDB databases and user privileges. If "--admins" is specified as above, the export will also include the server admin accounts.

Special note on SQL exports:

If a Vsite has SQL databases configured via the Vsite GUI, then Easy-Backup triggers the GUI mechanism that causes an SQL dump to be written to the Vsite directories where the MySQL/MariaDB GUI usually stores those backups. But as you may not have all SQL databases and SQL users associated with Vsites, Easy-Backup also does a full SQL dump and exports the SQL privilege table that contains the SQL users, their passwords and SQL privileges.

The data is first written to the local storage directory and if remote storage is enabled and configured, it will be transferred to the remote backup server via "duplicity".

Restore from the backup:

This will restore all server-admins (except "admin" and "alter-admin") from the backup:

~# /usr/sausalito/sbin/easy-backup --restore --admins --vsite all

This will restore all Vsites and their settings, data and users, including Vsite DNS and SQL (if configured and associated to the Vsite):

~# /usr/sausalito/sbin/easy-backup --restore --vsite all

To restore the whole MySQL/MariaDB databases and all SQL user privileges you can use this:

~# /usr/sausalito/sbin/easy-backup --restore --mysql

To restore all DNS records, configuration and DNS include files, you can use this:

~# /usr/sausalito/sbin/easy-backup --restore --dnsonly

Additional options:

--age

This specifies the age of the backup-set (in days) from which you want to restore.

--config

Just exports or imports the configuration of the server and its services, but nothing else.

--list

Human readable list of Vsites present in the local backup dump.

--remotelist

Human readable list of Vsites present in the remote backup dump.

--ip and --ipv6

 On Vsite restore the Vsites will not inherit the IP addresses they had before, but will use the IP addresses specified via these two switches.

--nodns

On import this will NOT import any DNS related data.

--check

If remote backup is enabled and configured, this option allows you to check the status, availability and integrity of your remotely stored backups.

--unlock

The Easy-Backup CLI command can only perform certain tasks (like exports, imports or the synchronizing of remote storage) one after the other. So if a restore runs, you cannot run another backup (or restore!) at the same time. And vice versa. So these privileged functions are performed on a first come, first served basis and a lock will be set in CODB to prevent other Easy-Backup instances to cause interference. If one instance of Easy-Backup is run and someone tries to do something using Easy-Backup via the GUI? They get the info that Easy-Backup is currently busy and they should try later. If two command line processes try to launch the Easy-Backup CLI command, the first one gets the lock and the second one enters a wait state (maximum: 90 minutes) and periodically checks if the lock is still in effect. It will only continue if the first process removes the lock on termination of its tasks.

However: If a task is aborted or killed, the lock may remain around. The Easy-Backup CLI has a check to eventually expire old locks by itself, but you can also use the --unlock command to force the removal of the lock. But that will only be allowed if there is no other Easy-Backup process currently running.

Tech-Talk:

How does this compare to CMU?

Simply put and just counting the Easy-Backup CLI command: It is CMU on steroids and without +20 years of accumulated crud and cruft.

The GUI then brings it to levels we've never had available before.

The usage of "duplicity" to store remote backups incrementally? A dream come true: Incremental backups, restore-able via the GUI (and command-line, of course), even by siteAdmins and Users.

Compatibility of backups between BlueOnyx versions?

Easy-Backup supports BlueOnyx 5209R, 5210R and 5211R. Any Easy-Backup export generated on one of these supported platforms can be imported on the same platform AND any other supported BlueOnyx platform.

Can this be used for migrations between BlueOnyx versions?

Yes! You can easily backup from an older BlueOnyx (5209R or 5210R) and import that backup set on a BlueOnyx 5211R. Easy-Backup will make all the necessary adjustments, no matter in which direction (old BlueOnyx to a newer one or back!) you "migrate". Just point the target server to the remote storage server and then import the latest backup.

How does it compare to the free Easy-Migrate?

When we created Easy-Migrate (which is included for free in BlueOnyx) the task and purpose was set clearly: Allow to online-migrate all settings, Vsites, Users, DNS, SQL and data from the source server to the target server. Easy-Migrate does this exceptionally well. But this usually means that after that migration you have to either change the DNS records to the IP of the target server, or need to flip the IPs of the target server and the migrated Vsites back to the original IPs they had before. This can be a bit of a hassle if you have many Vsites.

Easy-Backup uses many mechanisms and code functions of Easy-Migrate, but as it creates full and/or incremental backup dumps, you can export, turn the old server off and import the dump on a new server. This then involves a hell of a lot fewer hassles and cleanup tasks.

Beyond that: Easy-Backup is more robust and does some things slightly better than Easy-Migrate. In fact: While working on Easy-Backup and re-using some of the Easy-Migrate code for it, we discovered various glitches in Easy-Migrate. Most of those are fixed by now, but some are on the back-burner until we've had a chance to take a breather. Easy-Backup has about twice the amount of code compared to Easy-Migrate (roughly 5000 lines of Perl code) and has lots of checks and balances built in. Like the file and process locking, proper UTF-8 encoding of all exports, SHA256 checksums for all export files and checking of file integrity before imports, a MySQL/MariaDB export/import method that even allows exports from vastly newer SQL database servers to be imported on really old SQL servers like the one found on CentOS 7. So you can even migrate from 5211R back to 5209R if you were so inclined for whatever reason. Easy-Backup will then STILL be able to restore your SQL databases and Users with all matching privileges.

The code complexity and the required level of perfection that went into Easy-Backup was quite insane and (as I must admit) some setbacks and delays in the development were for a time demoralizing. We had gotten the CLI command to export/import to about 98% completion. The only thing that was missing was some fixes in the DNS import routine. Then it would have been in a state that one could consider "good enough". But I didn't allow us stop there. Another 300 hours of coding went into it to go the extra mile and make it rock solid and as close to perfection as possible. Creating the GUI for it (actually three GUIs, considering that the 5211R, 5210R and 5209R GUIs are internally quite different!) took another 230 hours.

Likewise: In the process to build the GUI for Easy-Backup we found and fixed a whole heap of minor issues and methodologies in the existing BlueOnyx 5209R, 5210R and 5211R GUI code. Some of these were outright bug fixes, or porting features and internal GUI functions back from 5211R to 5210R and 5209R. Like some of the newer ACLs, usage of PHP sessions for error message handling and other odds and sods.

So yeah: Easy-Backup is Easy-Migrate on steroids and it's a complete new ball game. Having Easy-Backup will not only allow you to easily migrate your BlueOnyx servers to newer BlueOnyx platforms (as a side-effect), but it will allow you to have several generations of backups in a safe place to easily recover from any disaster that might strike. And if you have unruly users or siteAdmins which constantly wreck their own sites and need data recovery? They now can do it themselves via the GUI.