SaltStack tends to be used to configure and maintain existing network nodes, but at SaltConf19, Kuali’s senior site reliability engineering lead Ben Gridley showed how Kuali also uses files stored in an external Salt Pillar to control the creation and maintenance of AWS infrastructure components.
Gridley noted that Kuali has a lot of self-steering product teams, each with its own AWS accounts. In looking at this situation, Gridley said, “we found that there were a lot of different ways that people were managing their AWS accounts—but what really was happening was that they would go into the AWS console and just click around until it did what they wanted it to do. Oftentimes they would make changes, things would break, and then nobody would notice the broken thing till the next day, when they’d forget that the changes were made or that it even related to the broken thing at hand.
“So we wanted all our changes to be in version control,” Gridley said. They needed uniform AWS automation and they needed their changes to leave a trail.
SaltStack Proved Most Capable
Kuali went on a search for the best tool so that “any of the things that we rely on in our AWS infrastructure” could only be maintained in a controlled environment. Grisley said Kuali wanted infrastructure “to be changed through the same process that code gets changed through.” As part of their AWS automation, they wanted peer reviews, tests performed ahead of committing the changes, and then to make the changes live by putting the code back into the master branch of a repository.
There was another win that Kuali was after as well: “We also wanted to create some consistency across teams to be able to use similar tool sets.” Grisley said they wanted a cohesive solution, “but also to allow our teams to remain self steering.
“We came up with a solution to use SaltStack to manage all the infrastructure.” The approach was somewhat unusual, though: “We wanted to run SaltStack in a Dockerized container. Doing that allows each team to use this tool portably. Today we can run it locally on our laptops, we run it in AWS codebuild, and we actually run it in our ECS clusters as well. But the idea is that it will provide portability for people to implement their changes wherever they need to run their code from.” SaltStack Enterprise contains a number of features specifically targeting cloud infrastructures.
Another advantage to running SaltStack in a Docker container: “Our engineers don’t have to maintain a Salt master. They don’t have to set up and configure a Salt master for every AWS account that exists.”
Once committed to the Dockerized SaltStack, the first thing Gridley said was needed was an external Pillar. In the Kuali case, Git is used for this because of its inherent version control capabilities. Beyond that, Gridley explained, they needed some environment variables to feed into the container, which runs the Salt minion in a masterless mode. In the Git repository, files that are formulated very similarly to Salt state files contain descriptions of the AWS services that need to exist. “This is the only thing the end users need to maintain to use this tool,” Gridley said. “So it’s really easy for our engineers to manage all the different AWS accounts we have at Kuali.”
The state files allow for AWS automation that enables the creation of fundamental infrastructure elements like route tables, subnets, and the like. When the containerized minion runs, it evaluates the AWS account to see whether all the elements exist and whether they are in the right state, then creates and reconfigures as needed, just like any other Salt deployment.
Gridley said many of the elements used to run AWS configuration were already built in to Salt, but that Kuali had also written custom code for some of the things they wanted to be able to control.
One additional nice feature: two of the environment variables that are passed into the minion container are optional Slack user and channel information, so that notifications can be sent via Slack as components are spun up or reconfigured.