Here at Puppet Labs, we're proud to make software that helps free people from mindless, repetitive fire-fighting work so that they can focus on the important and interesting aspects of their job. But the same attributes of Puppet Enterprise that make it such a powerful tool—its ability to handle diverse environments, the number of components and the complexity of their interactions—can make it incredibly difficult to develop for. Add to this the fact that Puppet Enterprise's power as a system tool can make even minor bugs potentially very destructive, and it's easy to see why our team needs a robust environment for developing and testing.
For all the progress that has been made in software development, the unpleasant truth is that a great deal of software development and testing is done in fragile, unrealistic environments, often directly on developer machines. For many reasons, this approach (or lack of one) would never have worked for developing Puppet Enterprise. Apart from the fact that this would have cluttered (and potentially clobbered) developer machines, Puppet Enterprise is a complex and system-critical application, and lack of a sufficient development environment could allow bugs to lurk unnoticed.
What, then, were the specific requirements for a Puppet Enterprise development environment?
- It had to be realistic. Puppet Enterprise consists of several different sub-applications that interact in complex ways. Developers needed to be able to test individual code changes live by watching these services interact with each other as they would in the wild. This meant, at a minimum, two hosts representing the master and agent roles.
- It had to be consistent. Every developer had to have confidence that they were seeing the same thing as every other developer when running the same code. Inconsistencies in development environments can cause lots of wasted time as devs try to replicate bugs.
- It had to be rebuildable. As developers, our #1 specialty is screwing things up irreparably, so setup and teardown had to be simple and fast. "Oops, I borked my master" should mean a 5-minute rebuild, not a whole day lost re-crafting a snowflake.
- It had to be live. In order for the development process to be efficient, any local code changes had to be reflected instantly in the environment.
An environment meeting these requirements would need tooling to help automate the setup, interaction, and provisioning of VMs. Of course, because we are developing for a system provisioning and management tool, we already had some of our work done for us: Puppet Enterprise uses Puppet internally to provision and manage many aspects of itself. We were also able to use pre-existing Puppet modules from the Puppet Forge to manage other aspects of the environment, including auxiliary services used by Puppet Enterprise such as LDAP and MCollective. We wrote a Puppet module to manage other aspects of our environment, including linking in source files from NFS shares and support for toggling the environment between production and development modes.
However, we still needed a tool to automate the setup and teardown of VMs for the environment. Vagrant provided us with everything we needed and more: it handled our NFS mounts, IP forwarding, and its built-in Puppet integration capabilities worked seamlessly to apply our own custom modules and manifests. Vagrant's configuration-centric approach allowed us to define our desired setup in a Vagrantfile and then easily destroy and rebuild our environment without worrying about the details. It also enabled easy versioning of the environment using source control.
By combining the power of Puppet and Vagrant, we ended up with a development environment that not only met all of our requirements, but that could be set up and managed with an almost absurdly simple workflow. All we have to type is `vagrant up`, and Vagrant builds VMs from our Vagrantfile specs, mounts the Puppet Enterprise source code in an NFS share, installs Puppet Enterprise on the VMs, then applies our custom Puppet manifests, allowing us to develop live against installed software. At any time, we can toggle the environment to production with `vagrant environment production` and revert the software to its installed state for testing.
All of this is not to claim that our development environment is never the source of problems for us any longer—our ability to break things still outpaces our ability to automate and fix them, and we still spend a healthy amount of our time ranting and cursing at our setup. But in the process of running up against the limits of our tooling, we have pushed the boundaries of the types of features we are capable of developing. Embracing automation has made collaboration a seamless experience: all you need to do is check out a coworker's branch, and you're immediately on the same page. The same story that gets told about embracing DevOps tooling in all kinds of environments—energy that was wasted on frustrating and repetitive maintenance can now be directed towards fruitful work—has proven true for our development team, allowing us to focus on improving Puppet Enterprise.