Puppet is a powerful tool for
centralized configuration management and can be thought of as a programming
language of system configuration. It follows that an admin using Puppet is well
advised to consider software development's best practices to organize his
setup. Software development is a vast field with plenty of ideas for admins to
explore. In this blog post I want to concentrate on the idea of continuous
integration as a means to easing development and deployment of a puppet
manifest.
Continuous integration
(CI) is a software development practice, in which developers integrate their
local work early and often into the main baseline. Each commit triggers an
automatic build and test run of the software, giving early feedback about
problems. It is the basis for the next step in evolution, continuous
deployment.
A CI setup consists of several components:
- Central Code Repository
- Development Workflow
- Automated Build
- Automated Test Suite
- CI Server
The more complex work gets, the more important a central code repository
becomes. It gives you a history of development steps and the ability to easily
revert to an older state if necessary. In our case it is the foundation of the
CI workflow.
I chose git
to maintain my Puppet repository, mostly because I had some prior exposure to
it and its cheap branching permits for nice workflows. The tool you use doesn't
really matter, as long as your CI server supports it.
At the moment I am the only admin working on Puppet, so my workflow is quite
simple and I'll have to see how to adapt it, once others join me.
I use a master branch, where I push all new developments and a production
branch for deployment and as a stable base for development. All development is
done on a dedicated development virtual machine (VM) with a local puppetmaster,
pointing to my development repository. My workflow looks like this:
- login to dedicated Puppet development VM
- update local development tree from central repository
$ git pull
- checkout a feature branch based on the production branch
$ git checkout -b feature-X production
- develop in feature branch, committing often
- apply feature to local machine, fixing bugs
$ sudo puppetd --server localhost --confdir ~/puppet --test
- merge feature into master branch
$ git checkout master
$ git merge feature-X
- push master branch to central repository
$ git push
The automated build is the first line of quality assurance. Code that doesn't
compile is obviously broken.
With Puppet I consider manifest compilation as the build, which leads to a big
difference between software and Puppet development: the amount of products to
build. With Puppet we easily have tens or even hundreds of different
configurations, if, as in my case, most machines differ from each other. A
successful build thus includes compilation of the manifest for each kind of
node. This way basic syntax errors and missing classes can be detected.
Puppet has an automated builder built-in, as it compiles manifests on demand.
I use Ohad Levy's manitest.rb to
alleviate building the manifest for each kind of machine. This script takes a
machine's node file (a YAML file containing that machine's facts) as input and
compiles the manifest for this machine. I adapted the
script a little to better suit my needs and wrote a
Rakefile to fetch and loop over all (relevant) nodes.
The automated test suite is the second line of defense against errors. In this
stage, specific tests are run, verifying puppets behaviour.
This step is still missing from my setup, as I haven't figured out an easy way
to test all different configurations. My idea goes in the direction of having
cucumber features for each module,
VMs with basic operating system (OS) installations for all
architecture-OS-network-whatever combinations and execute the stories on each
relevant VM. This should catch errors like bad path names in file sources,
missing dependencies (like creating a file in a non-existent directory),
missing file attributes, and unset variables (in templates or elsewhere). These
are just the errors I have experienced so far, there might be more kinds, of
course.
Such a test suite requires lots of time to set up, so it's questionable whether
it delivers enough business value to justify the investment. In my environment
I consider it nice to have for now and resort to manual testing.
The CI server brings above steps together. It regularly checks the Puppet
repository for commits, compiles the manifest, (runs the test suite,) and
merges commits back into the production branch.
There are many CI servers out in the wild, so I did some reading and quickly
settled on Hudson. It has a nice GUI, is easy to
setup and extremely powerful.
After installing hudson, I activated user authentication, otherwise anyone
could mess with the CI server. Then I added the git plugin and started a new
job to deal with my Puppet manifest.
Some hints on my setup:
- The hudson user needs access to two machines, the puppetmaster for fetching
current node files and the git server for repository access. I use
password-less ssh public key authentication for both cases.
- In the extended git options (screenshot) I
activated Merge before build to merge commits into the production branch
and in the Post-build Action I activated Push Merges back to origin.
- The repository is checked for new commits every 5 minutes.
Conclusion
The setup described above is far from finished, as it still lacks the automated
test suite. Nevertheless, it has already proven itself useful in detecting
compile time errors and facilitates automatic deployment. My production
puppetmaster updates its checkout of the production branch every few minutes.
While not exactly good practice, it simplifys my life dramatically.
Subscribe