Zero Downtime Upgrades Made Easy with ClusterControl

“Keep your database upgraded to the latest version – it’s for your safety” is something you may frequently hear as sound advice and best practice when it comes to database management. On the other hand, upgrading your database can be a time-consuming task. Even a minor version upgrade requires that you thoroughly test the upgrade in a staging environment before upgrading your production setup. So what’s the big deal? If you’re only lagging behind one minor version, it shouldn’t matter, right? Well, it might not…until it does. And are you really prepared to take that kind of risk?

Earlier this year, a new potentially dangerous vulnerability was identified in Galera Cluster (CVE-2021-27928). At first glance, we see that the severity was marked as high, and when we start digging into the issue further, it does indeed look severe. It appears that a SUPER user may execute any arbitrary code by changing wsrep_provider and wsrep_notify_cmd variables at the runtime. It allows the user to load the .so library and point towards a script that the server will execute. As you can imagine, this is not a good situation. Sure, you need to have access to the SUPER user, and you would need to have something available to execute on the database node, but the fact that Galera can be configured to execute arbitrary code as a ‘mysql’ user is bad enough on its own.

As usual, in cases such as these, the fixes have been created, and new versions of the software, unaffected by the vulnerability, have been pushed. This particular issue has been fixed in MariaDB 10.5.9, 10.4.18, 10.3.28, and 10.2.37, as well as Percona XtraDB Cluster 5.6.51-28.46, Percona XtraDB Cluster 5.7.33-31.49, and Percona XtraDB Cluster 8.0.22-13.1. All seems to be back to normal. Right?

Wrong. There are countless systems running on production that have not yet been upgraded to the new, unaffected version. Severalnines support team is in touch with many database environments in the wild, and we are constantly working with prospects to help them migrate to an environment managed by ClusterControl. We see all kinds of MySQL (and not only MySQL) running in outdated versions, sometimes even versions that have reached their End Of Life and are no longer getting security updates. That should not be the case, especially if you are a ClusterControl user.

ClusterControl comes with a set of features that will help you to stay up to date with all security fixes. Let’s take a look:

First of all, ClusterControl comes with Operational Reports, one of them being the Package Upgrade Report:

Like all of ClusterControl’s operational reports, the Package Upgrade Report can be scheduled to be executed regularly and then delivered via email. It will contain information about the package versions installed on the nodes and if there are any kind of upgrades that should be performed:

The Package Upgrade Report presents a list of packages that should be updated for all databases, loadbalancers, security fixes, and any other packages installed on the node. For all of the system packages, the solution is to upgrade them using standard methods (apt, yum). When it comes to the databases and loadbalancers, ClusterControl comes with functionality that allows you to perform the minor version upgrade directly from the UI.

Before we head there, let’s assume that the database has to be updated. You do not want to just proceed and run the upgrade blindly – it might potentially cause problems for your application. It shouldn’t – minor versions do not break backwards compatibility (except when you use MySQL 8.0 – then yes, you may expect anything when going from 8.0.x to 8.0.x+1); however, there is always some risk involved. What you should do first is test the upgrade in a separate environment.

We have a simple MariaDB Galera cluster with ProxySQL and Keepalived:

We would like to build a test cluster so that we can test the upgrade process. With ClusterControl, it is as easy as using Create Replica Cluster job:

We can get the fresh data from the existing cluster, or we can use the data from a backup.

We also have to pick a source node in the production cluster:

Then we have to go through a regular deployment wizard, picking the version and vendor of the database, defining root password, and so on. We conclude by passing the nodes on which the cluster will be installed.

As a result, you will see a new cluster on the list with a clear mark that it is replicating off the production cluster. One thing worth mentioning, in the default setup, ClusterControl will use the latest versions of the packages to create the replica cluster. If you want to double-check just the queries, this is enough. If you want to go through the whole upgrade process, you would need to pin down older versions of the MySQL packages in order to install an old version (and then unpin them and test the upgrade).

One way or the other, after successful tests, you will eventually want to perform the upgrade. ClusterControl can help you to accomplish this:

In Manage -> Upgrades, you will find a UI to perform the upgrade.

You can use “Check For New Packages” to refresh the database of available packages. We can also pick which nodes we want to upgrade and which services: 

Simply confirm and that’s it – ClusterControl will perform the upgrade and get you the latest version of the packages.

As you can see, ClusterControl makes keeping your databases up to date easy and straightforward. The only step that you must handle manually is the proper testing. Otherwise – everything else can be performed for you by ClusterControl. Interested in learning more about how ClusterControl can help you effectively manage your database? Try it free for 30 days.