blog
Creating an Advisor to Check for SELinux and Meltdown/Spectre: Part Two
In part 1 of this blog, we showed you how to integrate a basic check with SELinux modes. In this part 2 blog, we’ll discuss and go over with Meltdown/Spectre integration to our Advisors, set an alarm, and then see how to debug the script.
Let’s have a quick review of the Meltdown/Spectre vulnerability first. Currently, there are 8 variants that this CVE is covering. These are,
- CVE-2017-5753 aka ‘Spectre Variant 1, bounds check bypass’
- CVE-2017-5715 aka ‘Spectre Variant 2, branch target injection’
- CVE-2017-5754 aka ‘Variant 3, Meltdown, rogue data cache load’
- CVE-2018-3640 aka ‘Variant 3a, rogue system register read’
- CVE-2018-3639 aka ‘Variant 4, speculative store bypass’
- CVE-2018-3615 aka ‘Foreshadow (SGX), L1 terminal fault’
- CVE-2018-3620 aka ‘Foreshadow-NG (OS), L1 terminal fault’
- CVE-2018-3646 aka ‘Foreshadow-NG (VMM), L1 terminal fault’
Some of these vulnerabilities or the so-called L1 Terminal Fault (L1TF) which is a speculative execution side-channel attack were just recently discovered, in contrast to the other variants Meltdown/Spectre. So let’s go ahead and integrate this in our Advisor, and scan our servers to identify if any is affected or not, and trigger an Alarm if required.
ClusterControl Advisor Implementation and Shell Script Invocation
First, we’ll setup the shell script. I’m using this implementation from Stéphane Lesimple (a.k.a. speed47). In my MySQL nodes, I run the following bash commands below:
$ sudo wget https://meltdown.ovh -O /usr/bin/spectre-meltdown-checker.sh
$ sudo chmod +x /usr/bin/spectre-meltdown-checker.shNow, let’s create the new file as follows:
Then, let’s copy the code that I have in my own Github repository and paste it into Cluster > Manage > Developer Studio. Then open the file we have just created under the myadvisors/host/spectre-meltdown-checker.js as seen below, and paste the code there.
Before going through the code, let’s discuss first the logical flow that we want to achieve.
Now, let’s dig into the code and explain what we are going to achieve. In lines 16 – 25, we have just declared the myAlarm(<title>, <message>, <recommendation>) function. This function will just return an integer which does the alarmId(<category>, <isHost>, <title>, <message>, <recommendation>) function returns. The returned value can be used when creating an alarm.
function myAlarm(title, message, recommendation)
{
return Alarm::alarmId(
Node,
true,
title,
message,
recommendation
);
}Now let’s skip some lines and go to the important ones. In line 48 – 50, we’re calling
var shell = "test -e /usr/bin/spectre-meltdown-checker.sh";
var retval = host.system(shell);
var reply = retval["result"];We’re checking if the spectre-meltdown-checker.sh has been setup and installed at /usr/bin path. Lines 56 – 72, this is a failback in case we don’t find the script, notify the user through Advisors page that this has to be installed and setup. If found that it has been setup properly, lines 75 – 85 will check if a report has been generated and it looks as follows:
var dateTime = CmonDateTime::currentDateTime();
var dateToday = dateTime.toString(ShortDateFormat);
dateToday = dateToday.replace("/", "-");
var todayLogFile = "/tmp/spectre-meltdown" + "_" + dateToday + ".log";
shell = "test -e " + todayLogFile; // if /tmp/spectre-metldown_m.yy.dd.log exist.
retval = host.system(shell);
shell = "wc -l " + todayLogFile + "|awk -F ' ' '{print $1}'|tr -d '\n'"; // check if the file has contents
var retvalLogContents = host.system(shell);
replyLogContents = retvalLogContents["result"];If it has no generated result yet, then the lines 90 – 99 handles the generation of report by such calling the spectre-meltdown-check.sh. Let’s see below,
// Let's generate result if command fails and log do not exist.
shell = "sudo nohup /usr/bin/spectre-meltdown-checker.sh --batch json > " + todayLogFile;
retval = host.system(shell);
reply = retval["result"];
var jsonMetldownCheckerReply = JSON::parse(reply.toString());
var errTimeoutMsg = "ssh timeout: Execution timeout";
print(shell);
print ("retval" + retval);
print("jsonMetldownCheckerReply " + retval);Now line 90 shows that we are calling a nohup (no hang up) for /usr/bin/spectre-meltdown-checker.sh and passing --batch json to format the output as a JSON formatted output. Why using nohup? Well, CCDSL has its own limitation and we’re setting a 5-second call for such host.system() command calls, thus the return message would be like
ssh timeout: Execution timeout 5 secs has reached (channel_closing).We’re planning to make this a configurable value, but we have to live with it for now.
Since we’re going to generate a report, the lines 115 – 128 will handle a notification in case the generated log file hasn’t yet been created and notify the user to come back after a minute. You can also manually run the advisor by clicking the “Compile and run” button for the spectre-meltdown-checker.js script we have in the Developer Studio.
We’re almost done now with the script, let’s go further down. Lines 130 – 144,
shell = "cat " + todayLogFile + " | tr -d '\n'";
retval = host.system(shell);
reply = retval["result"];
jsonStr = '{"list":' + reply.toString() + '}';
mapSpectreMeltdownResult = JSON::parse(jsonStr);
for (ndex=0; ndex < mapSpectreMeltdownResult["list"].size(); ndex++) {
if (mapSpectreMeltdownResult["list"][ndex]["VULNERABLE"]) {
msg += "<br />Host is affected by "
+ mapSpectreMeltdownResult["list"][ndex]["CVE"] + "/"
+ mapSpectreMeltdownResult["list"][ndex]["NAME"] + ". "
+ "Suggested action: "" + mapSpectreMeltdownResult["list"][ndex]["INFOS"] + """;
}
}Let’s have a look at that. Line 130 – 131 does a shell invocation in which we just invoke the Linux command using cat command. After fetching the result with cat command, we parse the result again as a JSON array format with keyname == “list”. One of CCDSL’s limitation as of this time is that it is not able to handle JSON formatted array when it starts and ends with squared brackets “[“ and “]”. However, there’s a work around, as enclosing with curly braces resolves the case here. Then lines 136 – 144 is doing a for loop in order to handle the multi-arrayed values of variable mapSpectreMeltdownResult. This is where we are storing the contents taken from the generated report while we parse it.
Setting up the Alarm
Now, since we’re almost in the end of the code. Go to lines 146 – 156,
if (msg.length()) {
var recommendation = "We advise to update your kernel to the latest version "
"or check your Linux Distro and see the recent updates about this CVE.";
advice.setSeverity(Warning);
advice.setJustification(msg);
advice.setAdvice(recommendation);
myAlarmId = myAlarm("Metldown/Spectre Affected!", msg, recommendation);
// Let's raise an alarm.
host.raiseAlarm(myAlarmId, Warning);
}we check the message length if any (which means there’s a vulnerability detected on the host), then set the Advisor advice now which is found in 149 – 151. Now on 153 – 155, you’ll see that we have setup the Alarm. It’s pretty easy eh? The line 153 calls the function that we have defined above the code which was discussed earlier at lines 16 – 25. By just calling host.raiseAlarm(<type>, <severity>, <[message]>) method, we are able to raise an Alarm which would look like below:
Once you click on the warning, it’ll show as follows:
and clicking the “Full Alarm Details”, it’ll reveal all of the messages as follows:
Since we have setup the alarm, once you have this script compiled and run under Developer Studio, schedule this Advisor. To quickly check the result and see it in action, you can try to schedule it, let say, every 5 minutes or 10 minutes since the script will run only once per-day as long as the generated report exists. This is how it looks like:
Isn’t it nice? This is cool stuff, we’re able to integrate security checks like these Spectre/Meltdown vulnerabilities. It’s pretty easy though specially if you have Javascript background but regardless of the familiarity, it would be easy to handle this one as there are plenty of examples we have in our Github repository.
Debugging the Advisor Script
The CCDSL is not that rich and sophisticated kind of language but it does provide the ways to debug your script or there’s a tool to achieve this.
Let’s go back to lines 97 – 99,
print(shell);
print ("retval" + retval);
print("jsonMetldownCheckerReply " + retval);These allows us to print the value of the variables on what it holds and what it contains. It’s not pretty but it works and does what we need. For that example, it’ll show the following result in the Message tab of the Developer Studio:
Debugging with ClusterControl CLI tools
Aside from using “print()” function, we can actually use the CLI tools to run the .js file. For example, let’s try and debug the JavaScript file.
[root@ccnode vagrant]# s9s script --execute --cluster-id=2 -u admin --password=b8be2b56-80f9-45c7-a248-65ee6744a12f --print-json spectre-meltdown-checker.js
{
"controller_id": "clustercontrol",
"reply_received": "2018-09-28T23:39:40.084Z",
"request_created": "2018-09-28T23:39:40.082Z",
"request_id": 2,
"request_processed": "2018-09-28T23:39:40.087Z",
"request_status": "Ok",
"request_user_id": 3,
"results":
{
"exitStatus": "null",
"fileName": "spectre-meltdown-checker.js",
"messages": [
{
"lineNumber": 16,
"message": "spectre-meltdown-checker.js:16: syntax error.",
"severity": "error"
} ],
"status": "ParseError"
}
}Let me explain what we’re doing here.
1. I uploaded the file named spectre-meltdown-checker.js to the ClusterControl monitor host
2. Locate your username and password which you can find in /etc/s9s.conf. Take note that, you have to open or read the file with sudo privilege since it’s owned by root, e.g. content of /etc/s9s.conf
#
# Configuration file created by the Cmon Controller for the
# s9s command line tool. Please feel free to edit or
# remove this file
#
[global]
controller = https://localhost:9501
cmon_user = "admin"
cmon_password = "b8be2b56-80f9-45c7-a248-65ee6744a12f"3. Run the command as follows,
$ s9s script --execute --cluster-id=<your-cluster-id> -u <your-admin-username> --password=<your-admin-password> --print-json spectre-meltdown-checker.jsBased on the example I have, I intentionally make an error on the script to show you how it works. We can even play around and get with the JSON values and pipe with python, e.g.,
// Let’s get the advice value
[root@ccnode vagrant]# s9s script --execute --cluster-id=2 -u admin --password=b8be2b56-80f9-45c7-a248-65ee6744a12f --print-json spectre-meltdown-checker.js | python -c 'import json,sys;obj=json.load(sys.stdin);print obj["results"]["exitStatus"]["0"]["advice"]';
We advise to update your kernel to the latest version or check your Linux Distro and see the recent updates about this CVE.To get the contents of the message, you can even do like this, e.g.,
[root@ccnode vagrant]# s9s script --execute --cluster-id=2 -u admin --password=b8be2b56-80f9-45c7-a248-65ee6744a12f --print-json spectre-meltdown-checker.js | python -c 'import json,sys;obj=json.load(sys.stdin);size=len(obj["results"]["messages"]); map(lambda i: sys.stdout.write(obj["results"]["messages"][i]["message"] + "\n"), range(size))';
192.168.70.10
==========================
Meltdown/Spectre Check
We advise to update your kernel to the latest version or check your Linux Distro and see the recent updates about this CVE.
<br />Host is affected by CVE-2017-5753/SPECTRE VARIANT 1. Suggested action: "Kernel source needs to be patched to mitigate the vulnerability"<br />Host is affected by CVE-2017-5715/SPECTRE VARIANT 2. Suggested action: "IBRS+IBPB or retpoline+IBPB is needed to mitigate the vulnerability"<br />Host is affected by CVE-2017-5754/MELTDOWN. Suggested action: "PTI is needed to mitigate the vulnerability"<br />Host is affected by CVE-2018-3640/VARIANT 3A. Suggested action: "an up-to-date CPU microcode is needed to mitigate this vulnerability"<br />Host is affected by CVE-2018-3639/VARIANT 4. Suggested action: "Neither your CPU nor your kernel support SSBD"<br />Host is affected by CVE-2018-3620/VARIANT 4. Suggested action: "Your kernel doesn't support PTE inversion, update it"
192.168.70.20
==========================
Meltdown/Spectre Check
We advise to update your kernel to the latest version or check your Linux Distro and see the recent updates about this CVE.
<br />Host is affected by CVE-2017-5753/SPECTRE VARIANT 1. Suggested action: "Kernel source needs to be patched to mitigate the vulnerability"<br />Host is affected by CVE-2017-5715/SPECTRE VARIANT 2. Suggested action: "IBRS+IBPB or retpoline+IBPB is needed to mitigate the vulnerability"<br />Host is affected by CVE-2017-5754/MELTDOWN. Suggested action: "PTI is needed to mitigate the vulnerability"<br />Host is affected by CVE-2018-3640/VARIANT 3A. Suggested action: "an up-to-date CPU microcode is needed to mitigate this vulnerability"<br />Host is affected by CVE-2018-3639/VARIANT 4. Suggested action: "Neither your CPU nor your kernel support SSBD"<br />Host is affected by CVE-2018-3620/VARIANT 4. Suggested action: "Your kernel doesn't support PTE inversion, update it"
192.168.70.30
==========================
Meltdown/Spectre Check
Script Setup/Installation required!
<br />Empty reply from the command. It looks like you have not setup the shell script yet. You can get the shell script from https://meltdown.ovh<br /> or run "sudo wget https://meltdown.ovh -O /usr/bin/spectre-meltdown-checker.sh; sudo chmod +x /usr/bin/spectre-meltdown-checker.sh"Conclusion
The scripts we have created for selinux-checker.js and spectre-meltdown-checker.js are simple implementations, it shows you how you can integrate these checks. For Spectre/Meltdown checks, we can even improve the implementation like to create an Advisor as well that will just remove the /tmp/spectre-meltdow*.log files in case the number of files goes high.
The ClusterControl Domain Specific Language is simple but powerful. There are plenty of ways to incorporate your tools or scripts into the Advisors. We’d like to hear more of your feedback and experiences
