One of the great features of Salt is that it allows you to store global data using any of a number of databases and other stores, all by way of a subsystem called Pillar. At SaltConf19, SaltStack senior software developer Gareth Greenaway walked through using Pillar and, beyond that, using external Pillar data to store encrypted secrets.

Greenaway began by describing a scenario in which variables were retrieved into a Salt state file using a connection to a database, with parameters for using the database included in the state file in plain text. To share across multiple state files, a separate state file could contain the variables, but they would still be in plain text.

“On occasion,” Greenaway noted, “the values that we’re storing in our state files and that we’re using within Salt are going to be sensitive in nature. This would be information like username and password, or it could be API keys. We won’t necessarily want to keep that those values in our state files. So that’s where we turn to Pillar.”

The default Pillar subsystem can then be used to store the variable data. “So now we can update our variable definitions within our state file to pull these credentials from Pillar, rather than storing it in the state file. And we can now reuse that same state file for both our staging and production servers, being confident that they’ll receive the right credentials for their environments.”

Grains versus Pillars

Said Greenaway: “The Pillar subsystem is an interface for Salt designed to offer global values that can be distributed to all minions or targeted at specific minions.” He noted that those new to Salt are sometimes confused by the difference between Salt ‘grains’ and pillars. “It’s important to remember that grains are information that is stored about a particular minion. This data is stored on and generated by the minion and the information in generally considered static, though it is sometimes updated if the source data is updated. An example would be network information. Other examples include OS and CPU information.”

On the other hand, “Pillar data is data about a minion or many minions and the data is stored on and generated by the master. It’s usually highly sensitive data such as API keys and passwords.”

Even in Pillar though, the default scenario is that data is stored in plain text. “We may need to store them encrypted. And we may need to avoid storing them on the Salt master completely, opting to store them in an external system. For this we turn to Salt’s external Pillars. As the name implies, it’s a mechanism for generating Pillar data by calling external Pillar interfaces. And we have a number of them to choose from.”

Indeed, external Pillars in the current Fluorine release of Salt can be SQL databases, NOSQL databases, git repositories, LDAP servers, and a whole host of other options, right on down to comma-separated-value files.

Greenaway showed demonstrations of creating a GPG key pair and then using the key to encrypt the username and password from the earlier examples.

Finally, Greenaway pointed out that, “like any subsystem within Salt, such as modules, grains and states, we can extend Pillar with our own custom modules to utilize systems that are not currently supported by Salt’s Pillar modules—and then include the custom module in our external Pillar configuration to allow our minions to pull data using this new module.”