Nikolay Sturm's Blog

Musings about Development and Operations

Reusability in Configuration Management Systems

| Comments

Since puppet and later chef took off, there was discussion in the community about code reusability (chef cookbooks or puppet modules). As far as I can see, there is hardly any progress to see and it is still a big PITA to use other people’s cookbooks or modules.

As I am mostly working with chef these days, I will present my ideas with respect to chef cookbooks. From my experience with puppet, I expect them to apply similarly.

The status quo is unusable

In recent months I have been incorporating some community chef cookbooks into our company setup and felt the pain first hand. Besides varying levels of quality, my main problem was with cookbooks being too tightly coupled to the author’s infrastructure. I could hardly find a cookbook that was usable as is. Instead I usually removed all the stuff I didn’t need, rewrote all the stuff I didn’t like and ended up with a more or less simple cookbook that did the job for our infrastructure. I believe there has to be a better way to do this.

When I was not working on chef, I was improving my Ruby skills and learning about principles of object-oriented software development. I wonder, if we consider Infrastructure as Code, wouldn’t it make sense to apply some good software development practices to cookbook development as well? So what actually makes a cookbook dependent on the author’s infrastructure?

Lack of Separation of Concerns

The main reason, in my opinion, is missing separation of concerns. Many cookbooks are basically just monolithic blocks of code without much abstraction. Recipes do everything, creating users, directories, installing software and configuring it the way the author liked.

Granted, people split their client setup from their server setup, but this isn’t enough to actually making cookbooks reusable. Cookbooks have to adapt easily to different contexts, different policies.

Ideas from Object-Oriented Software Development

Two principles of object-oriented software development I found particularly applicable to this problem are the Single Responsibility Principle and the Open/Closed Principle.

The Single Responsibility Principle

Applied to recipes, the SRP states:

A recipe should have a single responsibility.

In other words, it should just install an application user, or just create configuration files, or just delegate work to other recipes. The corollary to this principle, as with software development, are cookbooks with many short recipes, some of which will do some work (I consider these low-level recipes) and some of which just delegate tasks in order to compose a variant of the application(I consider these high-level recipes).

The Open/Closed Principle

Applied to cookbooks, the OCD states:

Cookbooks should be open for extension, but closed for modification.

It should not be necessary for users to change distributed recipes, files or templates. If a user wants to change some aspect of a cookbook, that should be possible via configuration data (cookbook attributes), or extension recipes (adding alternative recipes and using application composition).

An example of recipe composition

Let’s have a look at a sample cookbook’s recipes:

recipes
├── _group.rb
├── _server_config.rb
├── _server_runit.rb
├── _server_install_from_source.rb
├── _user.rb
├── _webui_config.rb
├── _webui_cronjobs.rb
├── _webui_source.rb
├── server.rb
└── webui.rb

In this example I prefixed all low-level recipes with an underscore to differentiate them from high-level, service composing recipes.

Our sample application consists of a server component and a webui that are both composed from a couple of low-level recipes.

Let’s have a look at the server composition:

This recipe composes a server by just including all relevant low-level recipes. In this convenience recipe, the cookbook author decided on installing the application from source and using runit to manage the server process.

Let’s assume a user prefers to install the application from a pre-built package. It is easy for him to extend the cookbook with a recipe of his own, like _server_install_from_package.rb that would just install the server package. He could then compose his custom server from a role or a custom_server.rb recipe like this one:

As you can see, the user addeded their internal company repository and installed their pre-built package. No changes to existing recipes were necessary and the user can still use most parts of the cookbook. If a package was generally available, the _server_install_from_package.rb recipe could even be added upstream.

As I said above, the SRP leads to many small low-level recipes, like group.rb, which are concerned with just a single responsibility:

If the recipe turns out to contain just a single statement, so be it. If anyone ever has to implement group addition differently, for instance, because they use specific company wide IDs for their groups, they could easily replace this recipe or provide upstream with a better version, that would optionally include a group id setting. Other recipes, like _server_install_from_source.rb probably contain more steps, like downloading, compiling and installing the application.

Conclusion

Applying the Single Responsibility and the Open/Closed Principles to the space of configuration management could be a road to furthering reusability of chef cookbooks and puppet modules. Having many small low-level recipes makes it easier for users to adapt cookbooks to their needs and extend them as local policy prescribes. Deviation from the upstream version is minimized.

What do you think of this approach? Do you have other ideas about furthering code reusability? Let us know in the comments!

Comments