Over the last couple of years, we’ve seen an unprecedented rise in the number of reported security vulnerabilities affecting infrastructure software. According to common vulnerabilities and exposure (CVE) statistics issued by the National Vulnerability Database, 2017 saw more than twice as many vulnerabilities than reported in 2016, and 2018 has already seen roughly the same number of vulnerabilities revealed in 2016. Those numbers also don’t even include the nearly 8,000 vulnerabilities that weren’t assigned a CVE in 2017.
While 2017 saw its fair share of especially lethal vulnerabilities—with the WannaCry and the Equifax hack among the most well-known—2018 started strong right out of the gate with the Spectre and Meltdown vulnerabilities.
In January, we shared our insight about how SaltStack can help remediate Spectre vulnerabilities. We still believe, as we did then, that intelligent, event-driven security automation is the key to assessing risk and mitigating threats for any given organization.
Here are a few ways to help secure your data against Spectre as well as against numerous other vulnerabilities, many of which have yet to be revealed:
Typical Vulnerability Response and Patch Cycle
Organizations typically employ a two-tiered approach when patching vulnerabilities within their infrastructure. The first tier usually involves vulnerabilities that are patched as part of a routine patch cycle; this process could take place weekly, monthly, quarterly or even yearly (yes, it's true: some organizations only patch yearly).
The second tier is reserved for extremely urgent, high-severity vulnerabilities such as WannaCRY, Apache Struts2/Equifax (CVE-2017-5638) or HeartBleed. When these kind of vulnerabilities hit the news, the usual process to patch vulnerabilities takes a backseat. The mindset of “do whatever it takes to patch the vulnerability now” takes hold. This is especially true when C-Level executives ask for daily—if not hourly—updates on vulnerability-remediation status.
SaltStack can be invaluable to SecOps teams in both situations, but it’s really in the second scenario that its true value is revealed.
In such a scenario, three urgent questions need to be answered:
- Are we vulnerable to [insert vulnerability here]?
- If so, how many assets are vulnerable?
- How soon can we patch, remediate, or implement a workaround?
To explain how SaltStack can help, I’ll use two very well-known vulnerabilities from recent history: HeartBleed and MS17-010, a.k.a WannaCRY.
Identifying and Patching Vulnerable Assets
The first step to patching vulnerabilities is identifying which assets are vulnerable. Vulnerabilities are typically tied to a specific version of the software or hardware; for example, v1.0 is vulnerable and v1.1 is patched. Step one of the process is to enumerate software versions of a vulnerable application installed in the infrastructure.
SaltStack ships with an extremely powerful remote execution and automation framework that normalizes routine system operations (such as listing software versions and upgrading and removing software with a single command), which are referred to as execution modules.
Let’s take the example of HeartBleed and see how SaltStack would have helped.
In the case of Heartbleed, the first question to be answered was: “How many assets have OpenSSL on them and which versions are installed?”
With Salt, that question can be answered with single command:
mehul@saltmaster:~$ sudo salt \* pkg.version openssl web-prod1: 1.0.1-4ubuntu5.11 web-centos: 1.0.1e-15 web-debian: 1.0.1e-2+deb7u4
Notice the system standardized the execution module and function “pkg.version,” which works across a wide variety of operating systems. SaltStack asks for the package version intelligently by calling rpm on Red Hat systems, apt-get on Ubuntu, and related utilities for other operating systems.
This makes identifying vulnerable assets in tense situations easy. Output of the command can be formatted into YAML or json to perform additional processing on the output.
Here’s an example :
mehul@saltmaster:~$ sudo salt \* pkg.version openssl --output=json
Just like listing OpenSSL versions, upgrading OpenSSL to the latest version can also be done with a single Salt command. Here’s an example :
mehul@saltmaster:~$ sudo salt '*' pkg.upgrade name=openssl
[Note: on apt-based systems, this will perform a full system upgrade.]
That returns the following output after completion:
openssl: ---------- new: 1.0.2g-1ubuntu4.12 old: 1.0.2g-1ubuntu4.10
These commands execute and complete in a matter of seconds across thousands of systems. So, having access to a powerful automation platform such as SaltStack helps decrease the risk from a vulnerability that is actively being exploited in the wild from days to minutes.
In some scenarios, such as zero-day vulnerabilities when no patches are available, the only option is to implement a workaround or stop the service. SaltStack can be helpful in such scenarios as well. Here’s an example:
mehul@saltmaster:~$ sudo salt 'fs-dev*' service.stop ftpd
SaltStack ships with 475+ execution modules such as pkg & service, and it’s easy to add new modules or extend existing modules to satisfy a specific use case.
When WannaCry ransomware broke out, it quickly infected a large number of systems by exploiting a vulnerability in the SMBv1.0 protocol. It leveraged the EternalBlue exploit, which was disclosed as part of the ShadowBrokers dump.
Just as in the previous case, SaltStack users could accomplish this by running a single Salt command against their Windows infrastructure using the reg and system Windows execution modules. Here’s an example:
mehul@saltmaster:~$ sudo salt 'win*' reg.set_value HKEY_LOCAL_MACHINE'SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters' 'SMB1' ‘0’
If the registry fix was implemented through some other method, such as GPO or PowerShell, SaltStack could help verify the fix with reg read function.
mehul@saltmaster:~$ sudo salt 'win*' reg.read_value HKEY_LOCAL_MACHINE 'SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters' 'SMB1'
Setting the registry key was not sufficient; for it to take effect, the systems needed to be rebooted as well. SaltStack orchestration can be applied to task automation to reboot systems as needed:
mehul@saltmaster:~$ sudo salt 'win*' system.reboot
And finally, when it's time to apply the patch, Salt can download and install the patch:
mehul@saltmaster:~$ salt 'win7*' win_wua.install KB4022168
System Hardening with SaltStack
Gartner predicts that, "...through 2020, 95 percent of cloud security failures will be the customer’s fault.” While reactive patching of vulnerabilities is good, preventing the exploitation of vulnerabilities through system hardening is a prudent approach as well. In fact, Gartner also suggests “workloads that leverage the programmability of cloud infrastructure to improve security protection will suffer at least 60 percent fewer security incidents than those in traditional data centers.”
SaltStack can be used to make any system programmable across hybrid cloud environments. A best practice is to configure systems properly the first time and every time, and then automate self-healing routines if or when a system falls out of a hardened and secure state.
System hardening prevents exploitation of vulnerabilities even when a vulnerable version of the software is installed. Disabling or removing Apache from a system if it’s not used as a web server is one example. SaltStack can enforce proper configuration on a remote system to meet or exceed system hardening requirements through a concept called Salt State files.
State files allow users to define the desired “state” of the remote system in a YAML file. Here are some examples:
remove_telnet: telnet: pkg.removed Ensure_telnet_off: service.dead: - name: telnet.socket sshd_root_restrict_login: file.replace: - name: /etc/ssh/sshd_config - pattern: '^\s*PermitRootLogin.*[yY]es.*' - repl: 'PermitRootLogin no'
Being able to run CLI commands quickly over a wide range of systems is a great capability to have at disposal, but we realize it also opens up the possibility of misuse. Also, limiting the usability of Salt to the command line limits its adoption to a small subset of users. That’s why we offer SaltStack Enterprise, which provides enterprise-grade support for Salt with an intuitive GUI, enterprise-grade reporting, and robust role-based access controls (RBAC) to restrict user's commands they can run with Salt. It also ships with enterprise API for easier integrations with other third-party products.
New vulnerabilities are announced almost every day, while the processes to patch them or implement workarounds are roughly the same. Firms must patch better, but it is also in their best interest to harden their systems, as described above.
While no system is ever completely locked down, the double-pronged approach of correctly installing patches and updates while also using hardening techniques will go a long way to achieving your security goals.