When I started writing Puppet manifests, I soon figured out, that there are many possibilities for mistakes, which I usually found when test-applying (--noop) changes to a machine. This manual process was quite cumbersome, only to find a missing comma. So I wondered how I could test manifests more easily and on my development machine.

After playing with existing tools like Ohad Levy’s Manitest, I stumbled upon Cucumber and wrote some glue code, so that I could write cucumber features for my manifests.

In my first round of testing, I wrote features like this one:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

Feature: Setup Amanda client
  In order to backup my servers
  As an admin
  I want my servers to be setup as amanda clients

  Scenario: Setup remote access
    Given a node of class "amanda::client"
    When I compile the catalog
    Then there should be a resource "File[/operator/.amandahosts]"
    And the file should contain "amanda.no.domain operator"
    And the file should have a "group" of "operator"
    And the file should have a "mode" of "0400"
    And the file should have an "owner" of "operator"
    And the file should require "Package[amanda-client]"
    And the state should be "present"

  ...

If you compare this to the puppet code being tested

1
2
3
4
5
6
7
8

file { "/operator/.amandahosts":
  content => "amanda.no.domain operator",
  ensure => present,
  group => "operator",
  mode => 0400,
  owner => "operator",
  require => Package["amanda-client"],
}

you will notice, that the scenario is basically a reimplementation of the puppet code. This doesn’t make much sense.

To start over, I collected my actual problems when developing manifests in order to think of a more sensible approach to writing Cucumber features.

1. catalog does not compile

   • syntax errors
   • missing template files (locally or in repository)

2. catalog compiles, but cannot be applied

   • unreachable or non-existent resources are referenced (before, notify, …)
   • missing file sources in repository

3. catalog applies but is faulty

   • faulty files due to empty manifest variables or wrong values
   • missing dependencies between resources (wrong order, missing service restarts after config file changes, …)
   • files are installed without ensuring directory creation beforehand

In order to deal with these issues, I first expanded cucumber-puppet to permit defining a host by its YAML node file. This way local (on my development machine) catalog compilation can be verified easily. For remote catalog compilation, I added a step ensuring files and templates are checked into git. Finally I added a step to ensure resolvability of all referenced resources. Looks like issues 1 and 2 are solved. :-)

Catalogs can now be verified like this:

1
2
3
4
5
6
7
8
9
10
11
12

Feature: General catalog policy
  In order to ensure applicability of a host's catalog
  As a manifest developer
  I want all catalogs to obey some general rules

  Scenario: Compile and verify catalog for host test
    Given a node specified by "features/yaml/test.no.domain.yaml"
    When I compile its catalog
    Then compilation should succeed
    And all resource dependencies should resolve
    And all file sources should exist in git repository
    And all file templates should exist in git repository

My new way of writing cucumber features thus is to move away from single resources and instead concentrate on the BDD principle of Outside-In specification. Above feature specifies a generic policy regarding catalog compilation. This specification could be augmented with rules like each file should have group, mode and owner set explicitly or each file written to directory “/etc/apache2” should require “Package[apache2]”.

This new approach does not solve all issues mentioned above, there is still more than enough room for manually verifying catalogs on real target hosts, but it is in my opinion much more useful than my first attempt.