Perfect System Administration

Puppet is a powerful tool for centralized configuration management and can be thought of as a programming language of system configuration. It follows that an admin using Puppet is well advised to consider software development's best practices to organize his setup. Software development is a vast field with plenty of ideas for admins to explore. In this blog post I want to concentrate on the idea of continuous integration as a means to easing development and deployment of a puppet manifest.

Continuous integration (CI) is a software development practice, in which developers integrate their local work early and often into the main baseline. Each commit triggers an automatic build and test run of the software, giving early feedback about problems. It is the basis for the next step in evolution, continuous deployment.

A CI setup consists of several components:

1. Central Code Repository
2. Development Workflow
3. Automated Build
4. Automated Test Suite
5. CI Server

Central Code Repository

The more complex work gets, the more important a central code repository becomes. It gives you a history of development steps and the ability to easily revert to an older state if necessary. In our case it is the foundation of the CI workflow.

I chose git to maintain my Puppet repository, mostly because I had some prior exposure to it and its cheap branching permits for nice workflows. The tool you use doesn't really matter, as long as your CI server supports it.

Development Workflow

At the moment I am the only admin working on Puppet, so my workflow is quite simple and I'll have to see how to adapt it, once others join me.

I use a master branch, where I push all new developments and a production branch for deployment and as a stable base for development. All development is done on a dedicated development virtual machine (VM) with a local puppetmaster, pointing to my development repository. My workflow looks like this:

1. login to dedicated Puppet development VM
2. update local development tree from central repository

   
   $ git pull
   

3. checkout a feature branch based on the production branch

   
   $ git checkout -b feature-X production
   

4. develop in feature branch, committing often
5. apply feature to local machine, fixing bugs

   
   $ sudo puppetd --server localhost --confdir ~/puppet --test
   

6. merge feature into master branch

   
   $ git checkout master
   $ git merge feature-X
   

7. push master branch to central repository

   
   $ git push
   

Automated Build

The automated build is the first line of quality assurance. Code that doesn't compile is obviously broken.

With Puppet I consider manifest compilation as the build, which leads to a big difference between software and Puppet development: the amount of products to build. With Puppet we easily have tens or even hundreds of different configurations, if, as in my case, most machines differ from each other. A successful build thus includes compilation of the manifest for each kind of node. This way basic syntax errors and missing classes can be detected.

Puppet has an automated builder built-in, as it compiles manifests on demand. I use Ohad Levy's manitest.rb to alleviate building the manifest for each kind of machine. This script takes a machine's node file (a YAML file containing that machine's facts) as input and compiles the manifest for this machine. I adapted the script a little to better suit my needs and wrote a Rakefile to fetch and loop over all (relevant) nodes.

Automated Test Suite

The automated test suite is the second line of defense against errors. In this stage, specific tests are run, verifying puppets behaviour.

This step is still missing from my setup, as I haven't figured out an easy way to test all different configurations. My idea goes in the direction of having cucumber features for each module, VMs with basic operating system (OS) installations for all architecture-OS-network-whatever combinations and execute the stories on each relevant VM. This should catch errors like bad path names in file sources, missing dependencies (like creating a file in a non-existent directory), missing file attributes, and unset variables (in templates or elsewhere). These are just the errors I have experienced so far, there might be more kinds, of course.

Such a test suite requires lots of time to set up, so it's questionable whether it delivers enough business value to justify the investment. In my environment I consider it nice to have for now and resort to manual testing.

CI Server

The CI server brings above steps together. It regularly checks the Puppet repository for commits, compiles the manifest, (runs the test suite,) and merges commits back into the production branch.

There are many CI servers out in the wild, so I did some reading and quickly settled on Hudson. It has a nice GUI, is easy to setup and extremely powerful.

After installing hudson, I activated user authentication, otherwise anyone could mess with the CI server. Then I added the git plugin and started a new job to deal with my Puppet manifest.

Some hints on my setup:

• The hudson user needs access to two machines, the puppetmaster for fetching current node files and the git server for repository access. I use password-less ssh public key authentication for both cases.
• In the extended git options (screenshot) I activated Merge before build to merge commits into the production branch and in the Post-build Action I activated Push Merges back to origin.
• The repository is checked for new commits every 5 minutes.

Conclusion

The setup described above is far from finished, as it still lacks the automated test suite. Nevertheless, it has already proven itself useful in detecting compile time errors and facilitates automatic deployment. My production puppetmaster updates its checkout of the production branch every few minutes. While not exactly good practice, it simplifys my life dramatically.

We are currently in the process of migrating to a different Linux distribution at work and with that introduce Puppet to manage these machines (about 130 desktop PCs and 20 Servers).

Recently we started noticing failed puppetd runs from time to time. Cause of the problem was, that our puppet clients turned out to be running in batches, so every 30 minutes our puppetmaster suffered a huge load spike.

Having read R.I.Pienaar's Scheduling Puppet with MCollective blog post, I figured I needed a similar solution. Installing MCollective crossed my mind for a moment, but it seemed overkill. Instead I came up with the following simple, pragmatic solution.

I restart each puppetd process at a host specific time, thus spreading them reasonably well across puppet's run interval.

The crucial component is my external node classifier, which I use to calculate the host specific time. It's a ruby script that knows about all our machines and is easily able to compute a nodes rank, in our case based on the hostname, modulo 30, puppet's run interval. This rank is used to terminate a cronjob restarting puppetd daily.

Lets look at an example. Say you have hosts a.foo.com, b.foo.com and c.foo.com, then their rank would be 0, 1 and 2 respectively. On host a.foo.com, the cronjob would run at 11:00, on b.foo.com at 11:01 and on c.foo.com at 11:02. (I chose the window between 11:00 and 11:30, as I figured most people would be at work at that time and thus their desktop PCs switched on.)

As you can see, the result is far from perfect, but still quite impressive:

I know, load graphs are mostly useless, but it's all the data I have ATM. ;)

Where 4 cores couldn't handle load spikes, 2 cores are evenly used and I might even get by with just 1.

I would very much like to see some kind of scheduling support in the puppetmaster, organizing clients for maximal spread.

I am happy to announce the release of cucumber-puppet 0.0.3. cucumber-puppet is the glue between cucumber and Puppet, allowing you to write behavioural tests, or features as cucumber calls it, for your Puppet manifest. You can find introductiary material to cucumber here.

cucumber-puppet is currently in alpha-testing.

Installation

cucumber-puppet comes packaged as a gem

$ gem install cucumber-puppet

or can be cloned from github

$ git clone http://github.com/nistude/cucumber-puppet.git

Usage

Initial setup

Before writing your first feature, you have to setup the infrastructure in your Puppet directory.

$ cd puppet
$ cucumber-puppet-gen world

This installs some example step definitions for cucumber to ./features/steps/ and ensures the cucumber-puppet glue code is available. You can adapt cucumber-puppet to your needs in ./features/support/hooks.rb.

Writing features

cucumber-puppet assumes you have your Puppet manifest organized in modules and does the same with your feature files.

$ cucumber-puppet-gen feature foo bar

generates ./features/modules/foo/bar.feature from the standard template. Use this file to write your feature and add missing step definitions to files in ./features/steps/.

A feature might look like this:

Feature: cucumber-puppet
  In order to run my puppet manifest's test suite
  As an admin
  I want the cucumber-puppet gem installed

  Scenario: Install cucumber-puppet
    Given a node of class "cucumber-puppet"
    When I compile the catalog
    Then gem "cucumber-puppet" should be "installed"

Running features

To run above feature, execute

$ cucumber-puppet features/modules/foo/bar.feature

and see it complain about missing step definitions. Add these to .rb files in ./features/steps/.

If you have any questions or feedback, feel free to leave a comment or email me at cucumber-puppet at erisiandiscord.de.

When I started writing Puppet manifests, I soon figured out, that there are many possibilities for mistakes, which I usually found when test-applying (--noop) changes to a machine. This manual process was quite cumbersome, only to find a missing comma. So I wondered how I could test manifests more easily and on my development machine.

After playing with existing tools like Ohad Levy's Manitest, I stumbled upon Cucumber and wrote some glue code, so that I could write cucumber features for my manifests.

In my first round of testing, I wrote features like this one:

Feature: Setup Amanda client
  In order to backup my servers
  As an admin
  I want my servers to be setup as amanda clients

  Scenario: Setup remote access
    Given a node of class "amanda::client"
    When I compile the catalog
    Then there should be a resource "File[/operator/.amandahosts]"
    And the file should contain "amanda.no.domain operator"
    And the file should have a "group" of "operator"
    And the file should have a "mode" of "0400"
    And the file should have an "owner" of "operator"
    And the file should require "Package[amanda-client]"
    And the state should be "present"

  ...

If you compare this to the puppet code being tested

file { "/operator/.amandahosts":
  content => "amanda.no.domain operator",
  ensure => present,
  group => "operator",
  mode => 0400,
  owner => "operator",
  require => Package["amanda-client"],
}

you will notice, that the scenario is basically a reimplementation of the puppet code. This doesn't make much sense.

To start over, I collected my actual problems when developing manifests in order to think of a more sensible approach to writing Cucumber features.

1. catalog does not compile

   • syntax errors
   • missing template files (locally or in repository)

2. catalog compiles, but cannot be applied

   • unreachable or non-existent resources are referenced (before, notify, ...)
   • missing file sources in repository

3. catalog applies but is faulty

   • faulty files due to empty manifest variables or wrong values
   • missing dependencies between resources (wrong order, missing service restarts after config file changes, ...)
   • files are installed without ensuring directory creation beforehand

In order to deal with these issues, I first expanded cucumber-puppet to permit defining a host by its YAML node file. This way local (on my development machine) catalog compilation can be verified easily. For remote catalog compilation, I added a step ensuring files and templates are checked into git. Finally I added a step to ensure resolvability of all referenced resources. Looks like issues 1 and 2 are solved. :-)

Catalogs can now be verified like this:

Feature: General catalog policy
  In order to ensure applicability of a host's catalog
  As a manifest developer
  I want all catalogs to obey some general rules

  Scenario: Compile and verify catalog for host test
    Given a node specified by "features/yaml/test.no.domain.yaml"
    When I compile its catalog
    Then compilation should succeed
    And all resource dependencies should resolve
    And all file sources should exist in git repository
    And all file templates should exist in git repository

My new way of writing cucumber features thus is to move away from single resources and instead concentrate on the BDD principle of Outside-In specification. Above feature specifies a generic policy regarding catalog compilation. This specification could be augmented with rules like each file should have group, mode and owner set explicitly or each file written to directory "/etc/apache2" should require "Package[apache2]".

This new approach does not solve all issues mentioned above, there is still more than enough room for manually verifying catalogs on real target hosts, but it is in my opinion much more useful than my first attempt.

Following up on Thoughts on testing Puppet manifests, I released cucumber-puppet 0.0.6. This version implements the steps mentioned in that blog post and adds the ability to compile catalogs for nodes specified by yaml node files.

Further documentation is available as man pages

$ gem man cucumber-puppet

or can be found at the cucumber-puppet github page.