Perfect System Administration

Puppet is a powerful tool for centralized configuration management and can be thought of as a programming language of system configuration. It follows that an admin using Puppet is well advised to consider software development's best practices to organize his setup. Software development is a vast field with plenty of ideas for admins to explore. In this blog post I want to concentrate on the idea of continuous integration as a means to easing development and deployment of a puppet manifest.

Continuous integration (CI) is a software development practice, in which developers integrate their local work early and often into the main baseline. Each commit triggers an automatic build and test run of the software, giving early feedback about problems. It is the basis for the next step in evolution, continuous deployment.

A CI setup consists of several components:

1. Central Code Repository
2. Development Workflow
3. Automated Build
4. Automated Test Suite
5. CI Server

Central Code Repository

The more complex work gets, the more important a central code repository becomes. It gives you a history of development steps and the ability to easily revert to an older state if necessary. In our case it is the foundation of the CI workflow.

I chose git to maintain my Puppet repository, mostly because I had some prior exposure to it and its cheap branching permits for nice workflows. The tool you use doesn't really matter, as long as your CI server supports it.

Development Workflow

At the moment I am the only admin working on Puppet, so my workflow is quite simple and I'll have to see how to adapt it, once others join me.

I use a master branch, where I push all new developments and a production branch for deployment and as a stable base for development. All development is done on a dedicated development virtual machine (VM) with a local puppetmaster, pointing to my development repository. My workflow looks like this:

1. login to dedicated Puppet development VM
2. update local development tree from central repository

   
   $ git pull
   

3. checkout a feature branch based on the production branch

   
   $ git checkout -b feature-X production
   

4. develop in feature branch, committing often
5. apply feature to local machine, fixing bugs

   
   $ sudo puppetd --server localhost --confdir ~/puppet --test
   

6. merge feature into master branch

   
   $ git checkout master
   $ git merge feature-X
   

7. push master branch to central repository

   
   $ git push
   

Automated Build

The automated build is the first line of quality assurance. Code that doesn't compile is obviously broken.

With Puppet I consider manifest compilation as the build, which leads to a big difference between software and Puppet development: the amount of products to build. With Puppet we easily have tens or even hundreds of different configurations, if, as in my case, most machines differ from each other. A successful build thus includes compilation of the manifest for each kind of node. This way basic syntax errors and missing classes can be detected.

Puppet has an automated builder built-in, as it compiles manifests on demand. I use Ohad Levy's manitest.rb to alleviate building the manifest for each kind of machine. This script takes a machine's node file (a YAML file containing that machine's facts) as input and compiles the manifest for this machine. I adapted the script a little to better suit my needs and wrote a Rakefile to fetch and loop over all (relevant) nodes.

Automated Test Suite

The automated test suite is the second line of defense against errors. In this stage, specific tests are run, verifying puppets behaviour.

This step is still missing from my setup, as I haven't figured out an easy way to test all different configurations. My idea goes in the direction of having cucumber features for each module, VMs with basic operating system (OS) installations for all architecture-OS-network-whatever combinations and execute the stories on each relevant VM. This should catch errors like bad path names in file sources, missing dependencies (like creating a file in a non-existent directory), missing file attributes, and unset variables (in templates or elsewhere). These are just the errors I have experienced so far, there might be more kinds, of course.

Such a test suite requires lots of time to set up, so it's questionable whether it delivers enough business value to justify the investment. In my environment I consider it nice to have for now and resort to manual testing.

CI Server

The CI server brings above steps together. It regularly checks the Puppet repository for commits, compiles the manifest, (runs the test suite,) and merges commits back into the production branch.

There are many CI servers out in the wild, so I did some reading and quickly settled on Hudson. It has a nice GUI, is easy to setup and extremely powerful.

After installing hudson, I activated user authentication, otherwise anyone could mess with the CI server. Then I added the git plugin and started a new job to deal with my Puppet manifest.

Some hints on my setup:

• The hudson user needs access to two machines, the puppetmaster for fetching current node files and the git server for repository access. I use password-less ssh public key authentication for both cases.
• In the extended git options (screenshot) I activated Merge before build to merge commits into the production branch and in the Post-build Action I activated Push Merges back to origin.
• The repository is checked for new commits every 5 minutes.

Conclusion

The setup described above is far from finished, as it still lacks the automated test suite. Nevertheless, it has already proven itself useful in detecting compile time errors and facilitates automatic deployment. My production puppetmaster updates its checkout of the production branch every few minutes. While not exactly good practice, it simplifys my life dramatically.