blog
How to configure AppArmor for MySQL-based systems (MySQL/MariaDB Replication + Galera)
Last week, we discussed how to configure AppArmor for MongoDB Replica Sets which basically has the same concepts applicable when configuring this for your MySQL-based systems. Indeed, security is very important because you have to make sure that your data is very well protected and encapsulated against unwanted data gathering of information from your business domain.
A little bit of history about AppArmor
AppArmor was first used in Immunix Linux 1998–2003. At the time, AppArmor was known as SubDomain, a reference to the ability for a security profile for a specific program to be segmented into different domains, which the program can switch between dynamically. AppArmor was first made available in SLES and openSUSE, and was first enabled by default in SLES 10 and in openSUSE 10.1.
In May 2005 Novell acquired Immunix and rebranded SubDomain as AppArmor and began code cleaning and rewriting for the inclusion in the Linux kernel. From 2005 to September 2007, AppArmor was maintained by Novell. Novell was taken over by SUSE who are now the legal owners of the trademarked name AppArmor.
AppArmor was first successfully ported/packaged for Ubuntu in April 2007. AppArmor became a default package starting in Ubuntu 7.10, and came as a part of the release of Ubuntu 8.04, protecting only CUPS by default. As of Ubuntu 9.04 more items such as MySQL have installed profiles. AppArmor hardening continued to improve in Ubuntu 9.10 as it ships with profiles for its guest session, libvirt virtual machines, the Evince document viewer, and an optional Firefox profile.
Why do we need AppArmor?
In our previous blog, we have tackled a bit of what is the use of AppArmor. It is a Mandatory Access Control (MAC) system, implemented upon the Linux Security Modules (LSM). It is used and mostly enabled by default in systems such as Ubuntu, Debian (since Buster), SUSE, and other distributions. It is comparable to RHEL/CentOS SELinux, which requires good userspace integration to work properly. SELinux attaches labels to all files, processes, and objects and is therefore very flexible. However, configuring SELinux is considered to be very complicated and requires a supported filesystem. AppArmor, on the other hand, works using file paths and its configuration can be easily adapted.
AppArmor, like most other LSMs, supplements rather than replaces the default Discretionary Access Control (DAC). As such it is impossible to grant a process more privileges than it had in the first place.
AppArmor proactively protects the operating system and applications from external or internal threats and even zero-day attacks by enforcing a specific rule set on a per-application basis. Security policies completely define what system resources individual applications can access, and with what privileges. Access is denied by default if no profile says otherwise. A few default policies are included with AppArmor and using a combination of advanced static analysis and learning-based tools, AppArmor policies for even very complex applications can be deployed successfully in a matter of hours.
Every breach of policy triggers a message in the system log, and AppArmor can be configured to notify users with real-time violation warnings.
AppArmor for MySQL
I have setup a MySQL-replication-based cluster using ClusterControl to my target database nodes running in Ubuntu Bionic. You can further follow this blog on how to deploy it or follow this video tutorial. Take note that ClusterControl checks or disables the AppArmor during deployment. You might have to uncheck this according to your setup just like below:
ClusterControl will just issue a warning that it is not touching your current AppArmor configuration. See below:
Managing your AppArmor profiles
Standard installation of your AppArmor in Ubuntu would not include utilities that would help manage the profiles efficiently. So let’s install these packages like so:
$ apt install apparmor-profiles apparmor-utilsOnce installed, check the status of your AppArmor in the system by running aa-status command. In the node I am using, I have the following output without MySQL 8 installed.
$ aa-status
apparmor module is loaded.
15 profiles are loaded.
15 profiles are in enforce mode.
/sbin/dhclient
/usr/bin/lxc-start
/usr/bin/man
/usr/lib/NetworkManager/nm-dhcp-client.action
/usr/lib/NetworkManager/nm-dhcp-helper
/usr/lib/connman/scripts/dhclient-script
/usr/lib/snapd/snap-confine
/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
/usr/sbin/tcpdump
lxc-container-default
lxc-container-default-cgns
lxc-container-default-with-mounting
lxc-container-default-with-nesting
man_filter
man_groff
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.Since I am using ClusterControl to deploy my MySQL-replication based cluster with AppArmor (i.e. ClusterControl won’t touch my current AppArmor config), the deployment shall have the MySQL profile in place and that shows up in the list running aa-status.
$ aa-status
apparmor module is loaded.
56 profiles are loaded.
19 profiles are in enforce mode.
...
/usr/sbin/mysqld
...
37 profiles are in complain mode.
...
1 processes have profiles defined.
1 processes are in enforce mode.
/usr/sbin/mysqld (31501)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
It is worth noting that a profile is in one of the following modes:
-
Enforce – Default setting. Applications are prevented from taking actions restricted by the profile rules.
-
Complain – Applications are allowed to take restricted actions, and the actions are logged.
-
Disabled – Applications are allowed to take restricted actions, and the actions are not logged.
You can also mix enforce and complain profiles in your server.
Based on the output above, let’s elaborate more about the profile complain. AppArmor will allow it to perform (almost as complain mode status will still enforce any explicit deny rules in a profile) all tasks without restriction but it will log them in the audit log as events. This is useful when you are attempting to create a profile for an application but are not sure what things it needs access to. Whereas the unconfined status, on the other hand, will allow the program to perform any task and will not log it. This usually occurs if a profile was loaded after an application is started, meaning it runs without restrictions from AppArmor. It is also important to note that only processes that have profiles are listed under this unconfined status. Any other processes that run on your system but do not have a profile created for them will not be listed under aa-status.
If you have disabled AppArmor but then realize you wanted to enhance your security or comply with security regulations, you can use this MySQL 8.0 profile that is provided by MySQL itself. To apply that, just run the following command:
$ cat /etc/apparmor.d/usr.sbin.mysqld | sudo apparmor_parser -aIt is worth noting that AppArmor profiles are stored by default in /etc/apparmor.d/. It is a good practice to add your profiles in that directory.
Diagnosing your AppArmor profiles
AppArmor logs can be found in the systemd journal, in /var/log/syslog and /var/log/kern.log (and /var/log/audit.log when auditd is installed). What you need to look for is the following:
-
ALLOWED (logged when a profile in complain mode violates the policy)
-
DENIED (logged when a profile in enforce mode actually blocks an operation)
The full log message should provide more information on what exact access has been denied. You can use this to edit profiles before turning them on in enforce mode.
For example,
$ grep -i -rn -E 'apparmor=.*denied|apparmor=.*allowed' /var/log/
/var/log/kern.log:503:Jun 18 18:54:09 ubuntu-bionic kernel: [ 664.680141] audit: type=1400 audit(1624042449.006:19): apparmor="DENIED" operation="capable" profile="/usr/sbin/mysqld" pid=30349 comm="mysqld" capability=2 capname="dac_read_search"
Binary file /var/log/journal/877861ee473c4c03ac1512ed369dead1/system.journal matches
/var/log/syslog:1012:Jun 18 18:54:09 ubuntu-bionic kernel: [ 664.680141] audit: type=1400 audit(1624042449.006:19): apparmor="DENIED" operation="capable" profile="/usr/sbin/mysqld" pid=30349 comm="mysqld" capability=2 capname="dac_read_search"Customizing your AppArmor profile
Profiles prepared by Oracle MySQL shall not be a one-size-fits-all pattern. In that case, you might decide to change, for example, the data directory where your MySQL instance data is located. After you apply the changes to your configuration file and restart your MySQL instances, AppArmor will deny this action. For example,
$ egrep -i -rn 'apparmor=.*denied|apparmor=.*allowed' /var/log/
/var/log/kern.log:503:Jun 18 18:54:09 ubuntu-bionic kernel: [ 664.680141] audit: type=1400 audit(1624042449.006:19): apparmor="DENIED" operation="capable" profile="/usr/sbin/mysqld" pid=30349 comm="mysqld" capability=2 capname="dac_read_search"
/var/log/kern.log:522:Jun 18 19:46:26 ubuntu-bionic kernel: [ 3801.151770] audit: type=1400 audit(1624045586.822:67): apparmor="DENIED" operation="mknod" profile="/usr/sbin/mysqld" name="/mysql-data/mysql.sock.lock" pid=5262 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=1002 ouid=1002
Binary file /var/log/journal/877861ee473c4c03ac1512ed369dead1/system.journal matches
/var/log/syslog:1012:Jun 18 18:54:09 ubuntu-bionic kernel: [ 664.680141] audit: type=1400 audit(1624042449.006:19): apparmor="DENIED" operation="capable" profile="/usr/sbin/mysqld" pid=30349 comm="mysqld" capability=2 capname="dac_read_search"
/var/log/syslog:1313:Jun 18 19:46:26 ubuntu-bionic kernel: [ 3801.151770] audit: type=1400 audit(1624045586.822:67): apparmor="DENIED" operation="mknod" profile="/usr/sbin/mysqld" name="/mysql-data/mysql.sock.lock" pid=5262 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=1002 ouid=1002Together with the error I had earlier, now it adds that I had just decided to use the /mysql-data directory but that is denied.
To apply the changes, let’s do the following. Edit the file /etc/apparmor.d/usr.sbin.mysqld. You might find these lines:
# Allow data dir access
/var/lib/mysql/ r,
/var/lib/mysql/** rwk,
Those flags such as r, rwk are the so-called access modes. These mean the following:
r - read
w - write -- conflicts with append
k - lock
The man page explains those flags in more detail.
Now, I have changed it to the following:
# Allow data dir access
/mysql-data/ r,
/mysql-data/** rwk,Then I reload the profiles as follows:
$ apparmor_parser -r -T /etc/apparmor.d/usr.sbin.mysqldRestart the MySQL server:
$ systemctl restart mysql.serviceWhat if I set my mysqld to a complain mode?
As mentioned earlier, complain mode status will still enforce any explicit deny rules in a profile. Though this works for example:
$ aa-complain /usr/sbin/mysqldSetting /usr/sbin/mysqld to complain mode.
Then,
$ aa-status
apparmor module is loaded.
56 profiles are loaded.
18 profiles are in enforce mode.
...
38 profiles are in complain mode.
...
1 processes have profiles defined.
0 processes are in enforce mode.
1 processes are in complain mode.
/usr/sbin/mysqld (23477)
0 processes are unconfined but have a profile defined.After you restarted MySQL, it will run and will show log files such as:
/var/log/syslog:1356:Jun 18 19:58:51 ubuntu-bionic kernel: [ 4545.427074] audit: type=1400 audit(1624046331.098:83): apparmor="ALLOWED" operation="open" profile="/usr/sbin/mysqld" name="/mysql-data/mysql.sock.lock" pid=5760 comm="mysqld" requested_mask="wrc" denied_mask="wrc" fsuid=1002 ouid=1002
/var/log/syslog:1357:Jun 18 19:58:51 ubuntu-bionic kernel: [ 4545.432077] audit: type=1400 audit(1624046331.102:84): apparmor="ALLOWED" operation="mknod" profile="/usr/sbin/mysqld" name="/mysql-data/mysql.sock" pid=5760 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=1002 ouid=1002
/var/log/syslog:1358:Jun 18 19:58:51 ubuntu-bionic kernel: [ 4545.432101] audit: type=1400 audit(1624046331.102:85): apparmor="ALLOWED" operation="mknod" profile="/usr/sbin/mysqld" name="/mysql-data/mysql.pid" pid=5760 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=1002 ouid=1002
And it will work. However, it will probably have issues with networking as it still denies entries as what we have in /etc/apparmor.d/usr.sbin.mysqld. For example, my replica is not able to connect to the primary:
Last_IO_Error: error connecting to master '[email protected]:3306' - retry-time: 10 retries: 1 message: Host '192.168.40.247' is not allowed to connect to this MySQL server
Last_SQL_Errno: 0In that case, using enforce and reloading your profile shall be an efficient and easy approach to manage your MySQL profiles with AppArmor.
