Preview: Easy Backup

Her is a small preview of something we're currently working on: "Easy Backup".

Easy Backup - how it came to be

As you might now, we retired the "Cobalt Migration Utility" (CMU) a while ago and replaced it with "Easy Migrate". Which allows you to migrate sites, users, settings, data, DNS and databases from one server to another. Easy Migrate was a radical departure from CMU as it directly exports and applies CODB database settings and performs the necessary adaptions if you migrate between different supported platforms that don't have identical database schemas. Such as going fro 5209R to 5210R or back.

The plan had always been to create a similar tool called "Easy Backup" to perform exports of all data on a BlueOnyx and to restore said data at a later time. On the same BlueOnyx or a different one.

By now we have a working demonstrator of "Easy Backup". It's still a rough draft with several odds and sods that need to be taken care of before a release. But it already does a fine job at exporting / importing Vsites as you can see in the video below: 

In this case I had already done a full export on another BlueOnyx 5210R and had moved the exported files to /home/import on another BlueOnyx 5210R. The video shows the help text of the easy-backup.pl command, how to check a backup set to see which Vsites it contains and which possible conflicts an import would create (if any). Next a single Vsite is imported from the backup set, which creates the Vsite, its users, restores all settings and imports the Vsite- as well as the User-data. Including emails, cronjobs (if any) and MySQL/MariaDB databases associated with the Vsite in question.

General Technical Description

How this works? In a way "Easy Backup" re-uses a lot of means and methods of "Easy Migrate". For each Vsite that is being exported we trigger a MySQL/MariaDB database backup, which writes individual SQL dumps into the 'sql' directory of the Vsites. This also takes care of WebApp SQL exports - in case you have the WebApp installer from the store installed. During such an export we then poll CODB and build Perl Hashes with all relevant data that we need to export.

These hashes are then JSON encoded and written off into three config files:

• SYSTEM.json (Has all server related data)
• VSITES.json (Has all data related to Vsites)
• USERS.json (Has all data related to Users)

The actual payload (Vsite- and User-data) is packed up into individual Tarballs and (of course!) a MySQL-dump containing *all* MySQL databases is also generated. And finally the data is written to the directory you specified on the command line. Or if none is given, it defaults to /home/easy-backup/ instead.

For backup purposes this directory can be moved via various means to another server, network share or whatever.

A restore simply works the other way around: You point "Easy Backup" to a directory with a prior export and can use it to examine and validate the export before you import it. During an import you can either do a full restore, which would restore all Vsites, Users, settings, DNS, MySQL and pretty much everything to what it used to be during the initial backup. Or you can selectively restore individual Vsites and all their data. Or you could just restore the DNS or just the MySQL databases.

"Easy Backup" is fully command line oriented and all relevant functions of it can be performed from the command line. However: It will also come with a GUI that allows you to schedule automated backups and will allow you to configure backup storage locations and retention of backups. Furthermore the GUI will allow restores of individual Vsites from a locally stored backup.

Availability

The BlueOnyx shop currently has a product called "Automated Backup" available for BlueOnyx 5209R. This GUI driven backup still uses CMU for exports/imports and we will remove the ability to purchase further copies of the "Automated Backup" once "Easy Backup" is ready. It will replace the product "Automated Backup" in the BlueOnyx Shop and customers who have an ongoing subscription for "Automated Backup" will also get access to "Easy Backup" at no additional costs.

We expect that "Easy Backup" might be ready for release within the next two weeks.

How does it compare with CMU?

To be brutally honest: It doesn't. It just blows it out of the water.

Don't get me wrong: CMU was great back in the day and the concept behind it was as brilliant as the execution. Making server settings and data 'portable' between incompatible server appliances is and was no small feat. The technical challenges there are quite steep and required deepest insights into all supported server platforms. And then some. But the more platforms CMU needed to support, the clunkier it became. If you still happen to have a Cobalt RaQ3 and do a CMU-Export, you can still import said backup on a BlueOnyx 5210R, which was released around 20 years later. Still: This comes at a price which is paid in performance and there is a lot of friction where a 100 percent picture perfect restore cannot be guaranteed.

CMU had several performance hogs. One was the massive amount of CODB data that needed to be polled from CCEd, needed to be adjusted and converted into XML and then had to be stored. When you jumped several platforms by exporting on a (say) 5106R and imported on a 5210R, then CMU needed to process all exported settings step by step and transform them to the desired platform. In our example it meant that the 5106R data was converted to 5107R, 5108R, 5207R, 5208R, then to 5209R and finally it was converted into the 5210R data format that we needed to use during the import on a 5210R. That's a lot of loops to jump through.

For "Easy Migrate" we already had developed a different method. Say we need to create a Vsite. Its settings are stored in a "Vsite" database object that has different individual "NameSpaces". Both the Vsite object itself as well as the NameSpaces might have different key/value pairs between BlueOnyx platforms. And even if the keys were the same, there could be different (supported or unsupported) values that could be set between platforms.

For example on 5208R the key that stored which PHP type a Vsite used had fewer options for values than it has on 5209R or 5210R. Same applies for "Shell Enabled" on a 5210R, which now has more than a simple '0' or '1' value due to the addition of Chrooted Jails.

"Easy Migrate" as well as "Easy Backup" create a 'baseline' of possible settings on the import platform. During an import they compare each CODB Object and NameSpace from the backup set with with the respective counterparts on the target server. If an import has keys that are missing on the target server, they will be adjusted or discarded. If an import is missing keys, then defaults will be added as appropriate to the import platform. If older values need to be adjusted to conform with the newer platform, it will be done on the fly and on an as needed basis. This eliminates repetitive iterations if more than one platform is "jumped" during a migration and is generally a lot faster than the methods that CMU used to use.

The second massive performance hog during export was CMU's usage of the Perl 'Archive' module to create tarballs. This module requires that you feed it an array containing a list with every single file you want to export. It then creates an empty archive and sequentially adds each file to it. One by one.

That meant: CMU had to get a directory tree list of (say) each Vsite first. Then it md5-summed each file. Then the Perl module Archive accessed, compressed and added each file to the export archive.

During restore it went the other way around: CMU opened the archive and based on a set of instructions for each individual supported platform it unpacked and stored each file from the Archive(s) in the right place. And did an md5sum check on each file it restored and compared that with the md5sum recorded in the exports XML file(s).

Check-summing a backup set isn't a bad idea. But md5 is no longer a sensible method and check-summing each file with (say) sha256 both during backup and restore just takes way too long. So "Easy Backup" just check-sums whole tarballs, but not individual files within. Additionally: We use straight up 'tar' and 'gzip' (and not the Perl module Archive::Tar) to back up and restore. Which is just plain up faster by several magnitudes and consumes fewer resources.

Due to this "Easy Backup" is considerably faster than CMU could ever be. Still: The ultimate caveat emptor is of course the amount of data that needs to be lugged around. If you have hundreds of Vsites and Users that consume hundreds of gigabytes of data, then creation (or restore) of their tarballs will take time. That simply cannot be avoided. But now we at least do it without having the handbrake pulled all the way up.

Anyway: Watch this space for the announcement of the release of "Easy Backup". It'll be out soon ...