Did you know that you can use workflow and testing to ensure you get the configurations you asked for in a secure manner? In this post, we discuss how.

Puppet compiles a catalog using Puppet DSL code and data that’s usually at rest on the local filesystem of the puppet master. An agent requests a catalog from a master, and subsequently enforces the catalog locally. It’s common for multiple teams to collaborate on the configuration code and data, with the different teams being responsible for different modules. Often times, this means that the Puppet module path is actually made up of two or more module paths, where each modulepath entry is a different version control system repository managed by a different team.

  modulepath = /data/puppet/core/modules:/data/puppet/webapps/modules

I will have a manifest (maybe an ENC too, but let’s just talk about manifests here):

  manifest = /data/puppet/manifests/site.pp

I am more likely to have my core security modules in the first part of my module path. Potentially I *could* have a class security {..} in both halves of my module path; however, the module found first by the autoloader will take precedence, and the module in the web apps part of the path won’t be evaluated. It’s important to get the configurations that you want in first, especially because there’s no hard and fast rule that will restrict resource definitions for security configurations to that security module. Any resource declaration might be in any module, subject to uniqueness constraints, and so forth.

Thus, my security class manages a resource:

   user { 'root': 
    password => 'myspecialpasswordhash',
  } 

If another class attempts to manage the same resource, Puppet will return a catalog compilation error, and no configurations will be managed. This is good. My root user won’t get hosed by a rogue configuration. However, there’s a twist to this story. What if my manifest (site.pp) declares another class ‘configurethewebserver‘? That class is defined in the the second half of my module path (which is where the modules that deploy my applications live, as opposed to the core standard operating environment), and gets declared, quite legitimately, in a node definition in the site manifest.

Read the rest of this entry »

Today we released the following versions with security updates:

• Puppet Enterprise 1.2.7
• Puppet Enterprise 2.7.2
• Puppet 2.6.18
• Puppet 2.7.21
• Puppet 3.1.1

These patches contain the fixes for multiple CVEs that have just been published on our website. You can read more information here.

All fixes in these patches were either identified by us or resulted from cooperation with upstream technologies, so there are no known exploits that occurred in any customer environments. Regardless, we strongly urge you to download and install the remediation patches immediately.

If you have any questions or concerns, you are welcome to reach out to us on the #puppet IRC channel, the Puppet mailing list or Puppet Enterprise mailing list, or contact customer support if you are a Puppet Enterprise customer.

One of the major interfaces to extend functionality with Puppet is the use of Facter and custom facts. A fact is a key/value data pair that represents some aspect of node state, such as its IP address, uptime, operating system, or whether it’s a virtual machine. Custom facts are able to provide catalog compile time data to the puppet master to customise the configurations for a specific node (using PuppetDB, this data becomes available even when you’re doing things elsewhere). Facts are provided by Facter, an independent, cross-platform Ruby library designed to gather information on all the nodes you will be managing with Puppet.

For an example of using a custom fact within Puppet, you can use this data in the context of a catalog compile to make a decision about the catalog that you sent back to the node. A fact that tells us the node’s operating system could cause some conditional logic in a class that tells the node the right set of packages to use to configure ntp on that particular system. Because the package names differ between Linuxes (let alone between Linux and Windows), this fact simplifies ntp configuration. Alternatively, you could use the $::ipaddress fact to customise the appropriate listen directive in Apache or SSH.

Read the rest of this entry »

• What my toddler taught me about information security (Racker Hacker). Evolving from the ‘entirely oblivious’ to the ‘passionately proactive.’
• Comparison of VMware, Microsoft, and Citrix Virtualization (VirtualizationMatrix.com). Relatively objective comparison of hypervisors, management, extensions, pricing, and more.
• Amazon’s cloud is bigger than we think (GigaOm). @gigabarb walks through a recent report that suggests Amazon’s IaaS market success may be bigger than commonly believed.
• Rails DevOps Cheat Sheet (Rubytune). Collection of command line snippets that the Rubytune lads have found helpful over the years.

Learn More

• Previous digest
• Sign-up for our monthly newsletter.

A few weeks ago, the Puppet Labs Security team received emails from two community members informing us the our website was vulnerable. Two of the attacks mentioned were CRIME and BEAST attacks. Up until now, most of the mails that have been addressed to this team have been to report security issues with one of our products, but rarely about our infrastructure.

When we received the message, it was an immediate reminder of how little attention had been paid to the parts of infrastructure that didn’t immediately require our attention. Sheepish facepalms abounded.

Puppet Labs strives for transparency as much as we can, and we like to keep our users and community in the loop when a bug is reported. For this particular case, it was left to the Operations team to validate and respond to the issue. The Operations team began immediately trying to first understand what the heck CRIME and BEAST were and then validate that there was indeed a problem.

Within a few minutes the Internet had told us that these attacks could quickly be mitigated on our Debian infrastructure by a config change to turn off compression on TLS sessions, and to upgrade Apache to the most recent version available in Debian. Now, a configuration change seemed simple enough. We only had 30 divergent Apache vHost ERB templates lying around for Puppet to compile and deploy. It was going to be a minute to fix, but we needed to get back to the user. The claims were quickly validated by a free SSL scanner put out by Quallys. Yes, Portland, we have a problem.

Now for the mitigation. We began to dive into some of the oldest Puppet code in our infrastructure — Apache deployments — the oldest of which was written nearly four years ago and by some guy named ‘root’, whoever that is.

As our users know, module development has been something of a craft that has evolved in recent years, since our Apache module has carried some of the legacy methods for module authorship. (Yes, even at Puppet Labs, we have to update how we write modules. We haven’t always known the best way.) Our Apache module used a template for each vhost. Even though we had the ‘default’ vhost, almost no site we had deployed even used it. This meant that in order to correct the configuration issue and disable compression for TLS in Apache, we were going to have to hunt down every place in our templates that we had configured SSL and modify the behavior. Luckily, we were able to get the list of Apache servers that were not up to date with one simple MCollective query. And this exercise gave us the opportunity to dig into the way we deployed our Apache configurations! Instead of fixing every instance of our SSL configuration, we fixed the central SSL configuration and simply included the template fragment in all of the other templates.

<%= scope.function_template(['apache/vhost/_ssl.conf.erb']) %>

This turned out to be awesome, since now we can tune all of our Apache SSL configurations at the same time. We also found a reason to finally get rid of that last remaining Ubuntu box since it did not support the version of Apache without doing some backporting. We chose to move the application to a Debian system to be in line with the rest of our infrastructure. We then combined this with an upgrade to our affected Apache instances and merged the new code for our templates. We’re more than a bit chuffed that we not only resolved the issues but resolved them using Puppet. It’s the best thing in the world when “eating your own dog food” actually means something tangible about managing our business.

But more importantly we’re thrilled that when a security issue arose two members of the community chose to let us know about it. We’d like to thank: Adam Ziaja and Chiragh Dewan for their responsible disclosure of the vulnerabilities. We’d also would like to thank our community for its history, and the continued practice, of disclosing security issues responsibly.

How You Can Help
We take these issue seriously and are committed to the security of our products and its surrounding infrastructure. If you find a security issue with any of our products or services, please let our security team know by sending an email to security@puppetlabs.com.