Perfect System Administration

Puppet is a powerful tool for centralized configuration management and can be thought of as a programming language of system configuration. It follows that an admin using Puppet is well advised to consider software development's best practices to organize his setup. Software development is a vast field with plenty of ideas for admins to explore. In this blog post I want to concentrate on the idea of continuous integration as a means to easing development and deployment of a puppet manifest.

Continuous integration (CI) is a software development practice, in which developers integrate their local work early and often into the main baseline. Each commit triggers an automatic build and test run of the software, giving early feedback about problems. It is the basis for the next step in evolution, continuous deployment.

A CI setup consists of several components:

1. Central Code Repository
2. Development Workflow
3. Automated Build
4. Automated Test Suite
5. CI Server

Central Code Repository

The more complex work gets, the more important a central code repository becomes. It gives you a history of development steps and the ability to easily revert to an older state if necessary. In our case it is the foundation of the CI workflow.

I chose git to maintain my Puppet repository, mostly because I had some prior exposure to it and its cheap branching permits for nice workflows. The tool you use doesn't really matter, as long as your CI server supports it.

Development Workflow

At the moment I am the only admin working on Puppet, so my workflow is quite simple and I'll have to see how to adapt it, once others join me.

I use a master branch, where I push all new developments and a production branch for deployment and as a stable base for development. All development is done on a dedicated development virtual machine (VM) with a local puppetmaster, pointing to my development repository. My workflow looks like this:

1. login to dedicated Puppet development VM
2. update local development tree from central repository

   
   $ git pull
   

3. checkout a feature branch based on the production branch

   
   $ git checkout -b feature-X production
   

4. develop in feature branch, committing often
5. apply feature to local machine, fixing bugs

   
   $ sudo puppetd --server localhost --confdir ~/puppet --test
   

6. merge feature into master branch

   
   $ git checkout master
   $ git merge feature-X
   

7. push master branch to central repository

   
   $ git push
   

Automated Build

The automated build is the first line of quality assurance. Code that doesn't compile is obviously broken.

With Puppet I consider manifest compilation as the build, which leads to a big difference between software and Puppet development: the amount of products to build. With Puppet we easily have tens or even hundreds of different configurations, if, as in my case, most machines differ from each other. A successful build thus includes compilation of the manifest for each kind of node. This way basic syntax errors and missing classes can be detected.

Puppet has an automated builder built-in, as it compiles manifests on demand. I use Ohad Levy's manitest.rb to alleviate building the manifest for each kind of machine. This script takes a machine's node file (a YAML file containing that machine's facts) as input and compiles the manifest for this machine. I adapted the script a little to better suit my needs and wrote a Rakefile to fetch and loop over all (relevant) nodes.

Automated Test Suite

The automated test suite is the second line of defense against errors. In this stage, specific tests are run, verifying puppets behaviour.

This step is still missing from my setup, as I haven't figured out an easy way to test all different configurations. My idea goes in the direction of having cucumber features for each module, VMs with basic operating system (OS) installations for all architecture-OS-network-whatever combinations and execute the stories on each relevant VM. This should catch errors like bad path names in file sources, missing dependencies (like creating a file in a non-existent directory), missing file attributes, and unset variables (in templates or elsewhere). These are just the errors I have experienced so far, there might be more kinds, of course.

Such a test suite requires lots of time to set up, so it's questionable whether it delivers enough business value to justify the investment. In my environment I consider it nice to have for now and resort to manual testing.

CI Server

The CI server brings above steps together. It regularly checks the Puppet repository for commits, compiles the manifest, (runs the test suite,) and merges commits back into the production branch.

There are many CI servers out in the wild, so I did some reading and quickly settled on Hudson. It has a nice GUI, is easy to setup and extremely powerful.

After installing hudson, I activated user authentication, otherwise anyone could mess with the CI server. Then I added the git plugin and started a new job to deal with my Puppet manifest.

Some hints on my setup:

• The hudson user needs access to two machines, the puppetmaster for fetching current node files and the git server for repository access. I use password-less ssh public key authentication for both cases.
• In the extended git options (screenshot) I activated Merge before build to merge commits into the production branch and in the Post-build Action I activated Push Merges back to origin.
• The repository is checked for new commits every 5 minutes.

Conclusion

The setup described above is far from finished, as it still lacks the automated test suite. Nevertheless, it has already proven itself useful in detecting compile time errors and facilitates automatic deployment. My production puppetmaster updates its checkout of the production branch every few minutes. While not exactly good practice, it simplifys my life dramatically.

NoMachine's NX client is a great tool for running desktop applications on a remote machine. You can either attach to the remote machine's desktop session (session shadowing) or start an independent NX session solely for remote usage.

As session shadowing unlocks a potentially running screensaver, it is only suited for accessing PCs in locked offices or at home. In an open office environment, only an independen NX session makes sense. This, however, has a drawback as well. Many desktop applications like Firefox or OpenOffice do not easily permit running multiple instances concurrently. Thus you have to close these programs before starting them in the other session (local desktop session or remote NX session). This gets quite annoying.

For people working in an open office environment, a combined solution is needed, that uses the same session locally and remotely, while not opening it to passersby. This means all programs are running inside an NX session and the user starts nxclient even when locally logging in to the machine. The rest of this blog post will deal with the peculiarities of this local setup.

Nxclient has two technical drawbacks preventing it from running stand-alone:

1. At least in my setup (Ubuntu 9.10 with GDM), it doesn't grab keyboard focus.
2. Nxclient forks nxssh for the actual session handling and ends itself, thus closing the local desktop session.

To work around the first issue, a window manager is needed. This should be as minimal and lightweight as possible so as not to interfere with whatever is running inside the NX session (think keyboard shortcuts or mouse gestures). I chose fvwm2 with a minimal configuration, stripping it of virtually all its functionality. I even chose to not lock the screen from fvwm2 but instead lock the session from within NX. This way NX users and regular users, with only a local desktop session, share the same screensaver configuration.

A natural workaround for the second issue would be to start fvwm2 after nxclient and keeping it in the foreground. This way, a user would have to explicitly exit fvwm2 to logout. To make the window manager as transparent as possible, it would be much nicer, to terminate the session once nxssh is terminated. This can be accomplished with fvwm2 running in the background and a loop, waiting for nxssh to finish.

Putting it all together, this is how to setup an nxclient-only session with GDM on Ubuntu 9.10:

1. install fvwm2
2. install nxclient
3. install support files

   
   $ sudo install -o root -g root -m 644 nxclient.desktop /usr/share/xsessions
   $ sudo install -o root -g root -m 755 nxsession /usr/local/bin
   $ sudo install -o root -g root -m 644 nx.fvwm2 /etc
   

4. login with GDM, choosing NX Client as session (you might have to restart gdm first)
5. configure nxclient for fullscreen mode
6. use CTRL-ALT-T to disconnect from the NX session, terminating nxclient