Six months with puppet

When I retired the server in my closet in favour of Amazon EC2 instances, it quickly became clear that I needed to operate as if the EC2 instances could be lost at any moment: when all was said and done, these systems were not under my control. Putting effort into setting up individual services by hand no longer made sense. Thus, I needed a better configuration management system and I needed true automation that could quickly rebuild machines if they were lost.

My traditional method for managing, automating and documenting the installation of a service was using a Mercurial repository with a makefile and whatever configuration files, cron jobs, firewall setup commands, et cetera, needed to get that service up and working. This approach no longer made sense, as it required a lot of installation and copying to the target instance in order to set it up.

The other main failing with my traditional method is that it has no mechanism to handle dependencies between services. There is no way individual hosts can declare host names to make them show up in DNS. There is no way for a service that spans multiple hosts or services to collect all its logically dependent components in one place.

Wanting to address this failing is why I chose puppet. After six months of wrestling with it, I felt like sharing my experience.

First, let me say that in some ways, puppet combined with a service like EC2 is fantastic. Never again do you need to feel nervous about upgrades. When I want to upgrade my drupal service, I simply create a new VM and instruct puppet to build a new drupal service on it. If I am happy with how it looks and works, I reassign the public IP and kill the old VM. Setting up and building a VM with a drupal service typically takes less than 3 minutes. Compared to the risk and complexity of doing such operations in-place, this is a huge gain.

However, puppet has a large downside. The effort needed to create and maintain new services is simply too large. This article is my attempt to explain what I think is problematic with puppet.

The assembly language problem

Puppet in its current form can be compared to assembly language. It gives you a set of low-level primitives such as tracking files, installed packages, et cetera. In order to set up a service, I must first construct the infrastructure and semantics. This becomes a rather steep slope, since it has to be done for each service I want to deploy; a dozen to date. There are numerous contributed modules for various services, but most are limited to a narrow usage of the service, and many are no longer maintained.

Even on this low level, puppet lack supporting functions. A good example of this is the file fragment pattern. Most cases where a service is configured through puppet, we  need to put together several fragments into a full configuration file. Contributed patterns for solving this problem has been around for years but are still not incorporated into the main puppet distribution.

At this point, puppet is reasonably mature. I think the team should focus most of its effort on curating a set of good contributed modules that cover the common use cases of infrastructure level components, such as rsyslog, bind, openntpd, isc-dhcpd, et cetera. As it it, each user of puppet starts with two nearly empty hands.

Exported resources should be the norm

Puppet's strength and unique feature is something called exported resources. Basically, when puppetmaster inspect its configuration files, some resources (e.g. cron job descriptions) declared by a certain node are marked for "export." The exported resources are not supplied to the puppet agent on the declaring node, but instead can be collected by some other host. For example, a node may declare a DNS A-record for export, which is then collected (together with NS records for slave DNS server, MX records for mail servers, et cetera) by another node that runs the DNS master for the zone containing the A-record.

As soon as we get to a cluster of VMs, even a small one, this feature makes a lot of sense and makes it much easier to create new VMs with new services and reinstall old ones.

The fundamental problem with the logical grouping of configuration is that it creates a new configuration language for all managed services. An administrator skilled in maintaining traditional bind configuration files need to relearn how to configure bind the puppet way. Never the less, for standard and well-defined services (such as bind, syslog, et c) it is worth the effort to break apart configuration into more managable blocks. The standard modules that puppet lack, should use exported resources as a norm.

Conclusion

It is worth noting that none of these shortcomings are designed into puppet, or even require drastic technical changes that would break installed base. They are rather a misdirected focus.

However, as long as they remain, puppet will not really be a good alternative for small- and mid-sized installations. This is a pity because the benefits of virtual machines, good automation and configuration management becomes noticeable at around 5 instances.