You’ve bought Pro Puppet, downloaded a couple of modules from the Puppet Forge (and have written some of your own too), and you’re on your way to implementing your Puppet environment when it hits you: something feels bulky with the way you’ve designed your Puppet code. Your modules may not be portable between environments (development, testing, production) without significant tweaks, each of your node declarations may require a number of variables in order for the code to work, or you’re constantly needing to open up your modules to account for changes in your environment.

There’s GOT to be an easier way to do this, right?

We hear stories from many customers about problems in their Puppet environments, and many of them can be traced back to the way their configuration data is integrated with their Puppet code. Configuration data is the term we use for the environment-specific data that needs to be plugged in to your Puppet code (i.e. variables, class parameters). Take the following bit of Puppet code for example:

1
2
3
4
5
6
7

$dnsserver    = '8.8.8.8'
$searchdomain = 'puppetlabs.vm'
 
file { '/etc/resolv.conf':
  ensure  => present
  content => "search ${searchdomain}\n nameserver ${dnsserver}\n",
}

The configuration data in this example would be the hard-coded variables $dnsserver and $searchdomain and the Puppet code would be the file resource block declaring /etc/resolv.conf. This example is intentionally kept simple in order to highlight the methods by which you will separate your configuration data from your Puppet code, but imagine code that needs to set different variables in different environments (MySQL servers, databases, usernames, and passwords, for example) and you can see how the above example can quickly become unwieldy. How else can this be done?

Legacy Method – Node Inheritance

The first method that people usually tried was node inheritance. By defining variables in separate node definition blocks, and inheriting from a nested list of definitions, you could SIMULATE data separation with this method. This was the go-to method before Puppet 2.6 was released, and as such we consider it to be a legacy solution that we don’t recommend using with versions of Puppet newer than 0.25 (note that if you’re still using node inheritance, please read this advisory on dynamic scoping).

1
2
3
4
5
6
7
8
9
10
11
12
13
14

node common {
  $dnsserver    = '8.8.8.8'
  $searchdomain = 'puppetlabs.vm'
}
 
node production inherits common {
  $dnsserver = '10.13.1.3'
}
 
node 'agent.puppetlabs.vm' inherits production {
  file { '/etc/resolv.conf':
    content => "search ${searchdomain}\n nameserver ${dnsserver}\n",
  }
}

PROS

• It was the easiest method to employ.
• Your data was in one location and, technically, separate from your modules.

CONS

• There was no easy way to find the value of a variable for a specific node.
• FINDING the value of a variable required “human parsing,” or reading through each and every node declaration to trace variable values.
• The data still resided in your Puppet code repository.
• There are better ways to implement this strategy, and this should be considered a legacy solution provided solely for information purposes.

Parameterized Classes

Puppet version 2.6 gave us the ability to pass parameters with class declarations. This allows you to completely remove configuration data from your classes and provide ‘sane’ default values should a class declaration not pass a parameter. While this is an entry-level step in beginning to separate your configuration data from your Puppet code (the data is now in its own class—in this case dns::params), the configuration data is STILL in your Puppet code repository (and thus isn’t a full separation). See below for an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

class dns::params {
  $dnsserver    = '8.8.8.8'
  $searchdomain = 'puppetlabs.vm'
}
 
class dns(
  $dnsserver    = $dns::params::dnsserver,
  $searchdomain = $dns::params::searchdomain
) inherits dns::params {
 
  file { '/etc/resolv.conf':
    content => "search ${searchdomain}\n nameserver ${dnsserver}\n",
  }
}

PROS

• Class parameters can be defaulted back to a ‘sane’ value as outlined in our Smart Parameter Defaults document.
• Modules that utilize this methodology are more portable—parameters need only be changed in a single ‘params’ class.

CONS

• All logic must be embedded in each module’s ‘params’ class.
• If you use this methodology to keep your configuration data separate, every module must have a ‘params’ class and any logic you introduce (picking different values based on operating system, for example) must be repeated in every module.
• The data isn’t truly separate from your Puppet code as it still resides INSIDE the module (and, technically, your Puppet code repository).

External Node Classifier

Many large sites decide to use an External Node Classifier script to solve the problem of looking up configuration data. External Node Classifiers (also known as ENCs) allow you to provide class declarations, parameters, and variables to Puppet in the form of YAML. The previous example would look like this in YAML:

1
2
3
4
5

classes:
  - dns
parameters:
  searchdomain :  ‘puppetlabs.vm’
  dnsserver    :  ‘8.8.8.8’

PROS

• Flexible – you design how the information lookup is done (query a database, parse a hostname or other Facter fact, etc).
• Can be written in any language: shell, perl, ruby, python, etc…
• Plugs into your existing CMDB (Configuration Management Database) to retrieve information that already exists in another source of truth

CONS

• You are responsible for writing and maintaining the External Node Classifier Script
• If the script breaks, your Puppet runs are endangered

Extlookup

Extlookup was introduced in Puppet version 2.6.0 as a hierarchical way to lookup values of parameters or variables based on a Facter fact value. To use Extlookup, you would first define a data directory that Extlookup would search based on a specific fact value (location, environment, operatingsystem, etc), and then you would specify a lookup precedence (look for a parameter/variable in a file named after the node’s certname FIRST, and then search in a file named after the node’s environment SECOND, and so on). Finally, you would assign a parameter/variable’s value by invoking Extlookup with the built-in (as of Puppet version 2.6.0) ‘extlookup()’ function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

$extlookup_datadir    = "/etc/puppetlabs/puppet/data"
$extlookup_precedence = [$environment, 'common']
 
node 'agent.puppetlabs.vm' {
  include dns
}
 
class dns {
  $dnsserver    = extlookup('dnsserver')
  $searchdomain = extlookup('searchdomain')
 
  file { '/etc/resolv.conf':
    content => "search ${searchdomain}\n nameserver ${dnsserver}\n",
  }
}

Sample common.csv file used with Extlookup

1
2

dnsserver, '8.8.8.8'
searchdomain, 'puppetlabs.vm'

PROS

• Extlookup supports a dynamic and hierarchical lookup based on a node’s Facter fact values.
• There could be a single node declaration that would use Extlookup to look up the value of every variable/parameter used in Puppet.
• The extlookup() function is built into Puppet as of version 2.6.0.

CONS

• You must use comma-separated value files (CSV) ONLY for your lookups (i.e. variable, value), so structured data (like arrays and hashes) is not supported.
• Data lookups only return the first-matched value.
• It doesn’t have the ability to concatenate a list of matches returned throughout the full hierarchy.

Introducing: Hiera

Hiera, short for “hierarchy” and written by R.I. Pienaar, is a pluggable, hierarchical database that can query YAML and JSON files (and any other data serialization for which you write a custom backend), as well as Puppet manifests, for configuration data. Hiera builds upon the model that Extlookup created and also adds support for structured data. With Hiera, you can dynamically lookup parameters based on a node’s Facter facts. Let’s look configuring Hiera for use with the previous example:

The hiera.yaml configuration file:

1
2
3
4
5
6

---
:backends: - yaml
:hierarchy: - %{environment}
            - common
:yaml:
    :datadir: /etc/puppetlabs/puppet/hieradata

The common.yaml file that Hiera uses for parameter lookup:

1
2
3

---
dnsserver    : '8.8.8.8'
searchdomain : 'puppetlabs.vm'

Puppet code using Hiera:

1
2
3
4
5
6
7
8

class dns {
  $dnsserver    = hiera('dnsserver')
  $searchdomain = hiera('searchdomain')
 
  file { '/etc/resolv.conf':
    content => "search ${searchdomain}\n nameserver ${dnsserver}\n",
  }
}

PROS

• Data is truly separated from your Puppet code—it exists in an entirely separate directory structure.
• Parameter lookup is hierarchical and dynamic based on Facter facts that describe your node.
• Hiera supports structured data—like arrays and hashes—that can be fed back to Puppet.
• Using Hiera, your Puppet modules contain zero proprietary data (which makes the module much more portable).
• Hiera will be integrated with the next version of Puppet (codenamed Telly).

CONS

• As of this writing Hiera is not YET built into Puppet , so utilizing it requires an initial installation step.

Conclusion

While there are a myriad of options to solve the problem of configuration data and Puppet code separation, we recommend using Hiera for its ability to adapt to every situation. This post only gives a brief glimpse of its awesome functionality. Stay tuned for a post dedicated to Hiera, where we will be looking in-depth at its usage, flexibility, and advanced features that can simplify the management of your environment whether you’re a sysadmin of 10, 100, or 10,000 nodes!

Additional Resources

• Come to Puppet Camp Atlanta on February 3, and check out Gary’s talk on this topic.
• Read the docs: Smart Parameter Defaults
• Read the docs: External Nodes
• Read the docs: Extlookup
• Check out the Hiera project, and look for a more in-depth introduction next week.

One of the issues that crops up when working with Puppet is ensuring that your manifests do what you expect. Errors are bound to happen. A missed brace can make a manifest not compile, or forgetting to include a module or set a variable may mean that running Puppet on the host fails to enforce the expected state. All in all, it would help to have some tools to make sure we’re writing valid code, that it does what it expects, and that if it doesn’t we catch it as soon as possible.

Syntax Checking

At the lowest level of checking, you can use the Puppet parser to do syntax validation. Typos and errors are bound to creep into code, so syntax checking at the end of a long day can go far to improve the quality of your life.

There are a couple of places where you can insert syntax validation. One method is by manually running `puppet parser validate selinux.pp` to make sure that the manifest can be parsed before you commit your changes or deploy them to a live environment. If I left out a curly brace in a manifest and then used ‘puppet parser validate selinux.pp’, then:

    % puppet parser validate selinux.pp
    err: Could not parse for environment production: Syntax error at ‘{‘; expected '}' at /Users/adrien/puppetlabs-mrepo/manifests/repo.pp:252
    err: Try 'puppet help parser validate' for usage

Puppet parser tells me what went wrong, and which line contains the error.

In addition, you can integrate syntax checking into your editor. Vim has built in code compilation functionality that can be used to run error checking, so you can quickly validate your code and jump to sections of code with syntax errors. There are plugins like Syntastic that will do continuous checking so you’re immediately alerted when syntax errors are made. For example, if you used Syntastic with the same syntax error, you would see this:

Syntastic output identifies your syntax errors.

Lastly, there’s the puppet-lint tool developed by GitHub’s Tim Sharpe that will analyze your manifests and look for deviations from the Puppet style guide. It’s a quick and easy way to ensure that everybody is following a common set of conventions, so as your module collection grows, you’ll have a consistent set of modules instead of sections with cobwebs. Running puppet-lint against a manifest could produce something like the following:

% puppet-lint init.pp 
    WARNING: top-scope variable being used without an explicit namespace on line 79
    WARNING: top-scope variable being used without an explicit namespace on line 81
    WARNING: define defined inside a class on line 59
    ERROR: single quoted string containing a variable found on line 124
    WARNING: string containing only a variable on line 81
    WARNING: => on line isn't properly aligned for resource on line 71
    ERROR: two-space soft tabs not used on line 50
    WARNING: line has more than 80 characters on line 83
    WARNING: line has more than 80 characters on line 84
    ERROR: trailing whitespace found on line 163
    WARNING: mode should be represented as a 4 digit octal value on line 55

Be warned that these steps only validate syntax. If you have variables that are incorrectly spelled or have a bad value, your code will still be completely valid—and will not do what you want. That’s why we have additional tools available to make sure you code actually works as intended.

Writing Automated Tests

Automated testing is one of the key ways to ensure that your libraries and manifests are meeting your expectations. Out of all the ways that you could test your manifests, I’ll highlight two: testing modules and their catalogs, and testing entire systems.

Testing Modules

As mentioned in a previous post by our Release team, you can add rspec and cucumber tests to ensure that your modules are creating resources as you expect.

For example, you can write tests that ensure that when including a module to install Apache, the package Apache is installed and the service is started. If you were to further develop on that Apache module, you could move forward knowing that the tests would always ensure those basic behaviors would still exist, and if something changed by accident you would definitively know it had changed.

The puppet-apt module has been a focus of a lot of testing, and is a great example of how you can test your modules. Given rspec tests looking like this:

   it { should create_exec("apt_update")\
      .with_subscribe(['File[sources.list]','File[sources.list.d]'])\
      .with_refreshonly(true)}

would produce output like this:

  apt
    Apt class with no parameters, basic test
      should create Class["apt"]
      should create Exec["apt_update"]
 
Finished in 0.36027 seconds
2 example, 0 failures

Writing tests is a good way to verify your modules are functional and reusable. Testing also serves as a indicator of quality, demonstrating you have taken the time to ensure the module does what you want it to do. Additionally, you can verify modules in the Puppet Forge if they have tests, and more easily check them for correctness.

Testing Systems

Unit testing individual modules is a great step to take, but at the end of the day you want to know running Puppet on a host will build the host the way you want, and will have the behavior you expect. Being able to pragmatically verify that services like SSH, Postgres, and nginx are running and serving resources is powerful stuff.

You can use Cucumber in a standalone manner to ensure that if you run Puppet on a host, you get the host you asked for when all is said and done. You can couple unit tests with system tests, and run everything in a testing environment before your changes go to production systems.

Martin Englund has blogged about his experiences with “Behavior Driven Infrastructure” (a play on Behavior Driven Development or BDD) with Cucumber, and did an excellent PuppetConf presentation about his experiences with Puppet and Cucumber.

Benefits of Testing Puppet Code

Everybody wants to have as smooth and seamless a work flow as possible. Deploying changes to your Puppet manifests only to discover that you forgot a comma or brace, or writing a manifest that can’t actually run successfully, can require debugging time that would be better spent elsewhere. Adding a few proactive tools will prevent errors from propagating out, and being able to automatically verify your systems means that you can deploy changes fearlessly and become more agile in your day to day operations.

Additional Resources

• Puppet Labs Style Guide
• Testing Modules in the Puppet Forge
• Syntastic
• puppet-lint

With the release of Puppet Enterprise 2.0, Puppet Labs has added cloud provisioning capabilities with support for some of the most popular public and private cloud offerings including Amazon EC2 and VMware. PE 2.0 allows you to quickly provision virtual machines and start managing them with Puppet right away.

Getting Started

The following set of “micro” tutorials will show you how to leverage Puppet Enterprise 2.0 cloud provisioning capabilities on a public cloud (Amazon EC2). In order to get up and running you will need access to an Amazon’s EC2 account. Please note that while this tutorials will focus on Amazon EC2, the Puppet Enterprise 2.0 provisioning capabilities are designed to support a wide range of cloud providers. Initially there is support for Amazon EC2 which is covered in this tutorial and VMware, which is covered here; with support for additional providers being added over time.

Setup EC2 for use with Cloud Provisioner

1. Download and install PE 2.0
2. Gather Amazon AWS details into the fog configuration file

Before launching instances with the Cloud Provisioner module, you’ll need to register with Amazon AWS and obtain your access credentials.

Once you have completed the registration process, follow the AWS configuration sections here.

Now that you are all set up, we are ready to create our first virtual server in the cloud.

Provisioning in the Cloud

Provisioning machines on public or private clouds offers a huge benefit over traditional methods used on bare metal machines. Gone are the delays caused by hardware procurement and network setup; new machines are just a few clicks away. But to take full advantage of these new possibilities, you’ll need the right tools for the job.

Create virtual machines on EC2

1. Gather Amazon Instance details
   • image — AMI to use when creating the instance
   • keyname — The AWS SSH key name as shown in the AWS console
   • type — Type of instance
2. Run the puppet node_aws create command

Create a Debian 6.0 micro instance on EC2:

# puppet node_aws create --image='ami-0ce41865' --keyname='kelsey_rsa' --type='t1.micro'
notice: Creating new instance ...
notice: Creating new instance ... Done
notice: Creating tags for instance ...
notice: Creating tags for instance ... Done
notice: Launching server i-2a38eb48 ...
#############
notice: Server i-2a38eb48 is now launched
notice: Server i-2a38eb48 public dns name: ec2-174-129-158-192.compute-1.amazonaws.com
ec2-174-129-158-192.compute-1.amazonaws.com

At this point you should be able to ssh into your EC2 instance:

# ssh -i .ssh/puppetlabs_id_rsa root@ec2-174-129-158-192.compute-1.amazonaws.com
Enter passphrase for key '.ssh/puppetlabs_id_rsa':
Linux ip-10-212-169-9 2.6.32-5-xen-686 #1 SMP Wed Mar 9 01:27:54 UTC 2011 i686

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@ip-10-212-169-9:~#

That’s all there is too it, and doing the same for VMware is equally as easy:
http://docs.puppetlabs.com/pe/2.0/cloudprovisioner_vmware.html

Cloud Maintenance

The ability to provision an unlimited number of virtual machines has its downsides, like virtual machine sprawl, it’s easy to lose track of how many machines you have and how they are configured. In short, virtual machines require real maintenance.

PE 2.0 greatly reduces this overhead by providing tools such as the Puppet Enterprise Console—built on top of the Puppet we all know and love—which enables you to visualize and manage your entire infrastructure, virtual or physical, right out of the box. When you provision your virtual machines using the new cloud provisioning capabilities, you get access to these features immediately.

Classify, and managing machines via the PE 2.0 Console (GUI)

# export PUPPET_ENC_AUTH_PASSWD=’XXXXXXXX’
# puppet node classify ec2-174-129-158-192.compute-1.amazonaws.com \ --node-group='default' --enc-ssl --enc-server dashboard.puppetlabs.com \
--enc-port 443 --enc-auth-user console
notice: Contacting https://master.hightower.puppetlabs.com:443/ to classify ec2-174-129-158-192.compute-1.amazonaws.com
complete

At this point you should be able to access your node via the Puppet Enterprise Console:



Conclusion

Whether you have an existing cloud infrastructure or just getting started, PE 2.0 can streamline the complete life cycle of running machines in the cloud while incorporating “cloud” best practices and world class configuration management.

Look for support for additional Cloud vendors in the near future.

Related Content:

• Read the docs on cloud provisioning in Puppet Enterprise 2.0. (videos embedded)
• Learn what you can do with those classified nodes in the Puppet Enterprise 2.0 recorded webinar
• Test it out — download ten free nodes of Puppet Enterprise 2.0.

Why do we want configuration management? There are plenty of reasons, but at the core of them is that we want to streamline the configuration and deployment of systems. We want this process to be repeatable, well-understood, and predictable. We want to make it deterministic.

Determinism — the idea that a process should result in the same outcome every time it is applied — is not a new idea, but it’s also not one that system administrators apply to their daily work. Can you prove that the bash script you just wrote to deploy your new NoSQL database will work across all of your system configurations? Will it work the same way if you execute it again next week—or next year? Automating the deployment and configuration of systems is meaningless if we can’t guarantee that these systems will complete the configuration process with predictable results.

For years, organizations have tried to handle this problem using scripts to deploy and manage their environments. These scripts evolve over years as they’re passed from admin to admin, growing to contain the personal quirks of their developers and long-irrelevant legacy workarounds. It’s not until a script is needed—usually when the server fails—that sysadmins discover it’s tightly pegged to a particular version of bash or sed, or requires a specific return code from an init script that no longer exists. The script’s lack of predictability impacts organizations at the worst possible time in the development cycle.

Periodically, organizations recognize that these scripts have become unmanageable and will attempt to refactor or rebuild them from scratch, resulting in a whole new cycle of trial-and-error. This tug of war with scripts takes time from other, more valuable tasks. How many times have you fixed or updated the same process when you could have been learning or applying new information to your environment?

Your credibility as an admin is dependent on your ability to construct reliable, stable, consistent systems. It’s critical that the process you employ works in a predictable way every time.

Puppet’s View of the Server

What makes Puppet different from a library of shell scripts, and from other configuration management tools, is that it forms a model of your system configuration prior to performing any activity on the system itself.

Before a single command is executed on a host, Puppet has constructed what we refer to as a “resource graph” of that node’s existing configuration, and determined how your changes will impact that graph. Below is a very simple example of such a resource graph:

It’s a difficult leap for new users, particularly as we as sysadmins are trained to think procedurally and not deterministically. Admins tend to think of processes: managing the server as a group of activities that leave the system in a desired state when properly applied. We spend most of our careers mastering the processes (and trying to pass certification exams that prove we have).

Puppet asks you to think differently about what administration means, and to focus on what the server is supposed to look like after all processes are finished. It doesn’t ignore process, but considers it a means to an end—you tell Puppet what you want the server to look like, and Puppet works out how to get the server there.

Noop and Dry Runs

One of the big advantages of the resource graph model is Puppet’s unique and unrivaled “noop” functionality, which allows you to simulate a change before you deploy it. Other tools can run in some semblance of a dry run or do-nothing mode, but they often repeat back what your script says, without offering the impact on your system as a whole. Puppet understands what your environment currently looks like as of the last Puppet run (represented by the orange rectangle in the below example) and what the end state is supposed to look like (the green parallelogram). Puppet then tells you both what it will do when executed without noop, and how that action would change the overall configuration of your system. These steps are identical in noop mode and in regular execution mode. The only difference is whether or not Puppet performs the action in the red shape at the bottom, or merely reports to you what will happen if you run Puppet.

Where Determinism Poses a Challenge

This resource graph model can be problematic for you if you don’t know what a server is supposed to look like at the end of your administration process. Puppet insists that you understand an application’s impact will be before you begin the process of installing it. This is how it should be. Nothing should be placed on a production system without you, the administrator, being able to fully document and understand the changes that application will impose; yet we are all guilty of starting procedures on systems without fully understanding the impact.

While this deterministic core is at the heart of Puppet, there are still elements that can cause confusion. Until recently, there was an issue with resources being applied in a non-deterministic order on nodes, so that a manifest might compile and execute in one order on one node, and another on a second. This was based on the assumption that all resources are atomic, and is remedied in the Puppet 2.7 and Puppet Enterprise 2.0.

Determinism is not a panacea—you still have to validate that your Puppet manifests do what they’re supposed to, and you should put them through the same rigorous testing you expect of dev environments. But you are able to increase your confidence that something working in your test environment will work in production, and that a proposed change will not interfere with a production environment because of a surprising interaction.

Further Advantages to a Deterministic Approach with Puppet

Aside from noop mode, what do you as an administrator get out of Puppet’s deterministic nature? A few notable benefits:

• Your Puppet manifests describe what your systems look like and how to install them. While there is never a replacement for good long-form documentation, in the real world people rarely have the time to write it. Fortunately, Puppet manifests can serve as accurate documentation of a server’s entire configured environment, from unusual post-install procedures to open ports—and in fact this can be better than written documentation, since it is by definition always up-to-date.
• When you do deploy, that process is automated. Your best and most senior engineer does not have to be the guy who pushes the button—and it’s very unlikely that on deploy night that you will be writing or heavily modifying legacy bash or Perl scripts.

You can ultimately think of the choice between a deterministic and non-deterministic tool as the choice between predictability and randomness, or better, between configuration management and ad-hoc system administration. With Puppet, you think your problem through, saving you time and effort later on. Non-deterministic tools may seem to solve a short-term problem, but you’re merely accruing technical debt.

Related Content:

• How to look at your resource graph in Puppet

Last week we released Puppet Enterprise 2.0, and one of the new orchestration capabilities we’re most excited about is Live Management. Puppet has always given you incredible power for managing your infrastructure, and Live Management harnesses that power into a clean, simple-to-use graphical interface, the Puppet Enterprise Console. Now with just a few clicks of the mouse, you can survey the state of all of your nodes, and clone configurations to other machines. By combining the power of Puppet’s advanced messaging technology with our native resource abstraction layer, we have created a simple entry point to get a detailed overview of your network and manage it on the fly.

Specifically, Live Management will allow you to browse and manage users, groups, software packages, and host entries across your network. These resources can be cloned from, or to, any subset of your infrastructure, which can be narrowed down by a simple hostname search, or more narrowly by the values of different facts about the machines. This new capability will also give you a preview of the impact of this change, which can then later be integrated into a fully Puppetized infrastructure. To demonstrate the capabilities of this new application, this blog post will walk through two example workflows.

Example Workflow 1: Setting up new Webservers

Our first example workflow will demonstrate how easy Live Management makes it to copy aspects of your configuration from existing hosts to new machines. In our example scenario, we have just added two CentOS machines to our network, and want to configure both of them as webservers. Specifically, we want to set them up in the exact same manner as an existing web server with the hostname “MAIN_WWW”. To do this, we’ll need to clone the version of Apache we have installed on MAIN_WWW to our new machines.

First off, we’ll need to find the correct version of the ‘httpd’ package. To do so, we can load the ‘package’ resource tab, and click ‘Find Resources’.

Scrolling down and selecting the ‘httpd’ package will show us information about which versions of this package we have installed on our various hosts.

All package resources with the title ‘httpd’ will be collected on this page, separated into variations according to their specific characteristics. To begin the process of cloning this package to the new hosts, we will click “Clone this variation” under the version installed on MAIN_WWW and continue by clicking the “Preview” button that appears at the top of the screen.

This takes us to a page previewing the results of cloning this resource to the currently selected set of nodes.

To proceed with cloning, we need to restrict this set of nodes to the desired target: all nodes running the CentOS operating system. To do this, we can search on the ‘operatingsystem’ fact with the value ‘CentOS’ under the Advanced Search tab.

Now that we’ve selected the resource we want and the nodes we want to clone it to, we can proceed by clicking “clone.” This will take us to a page showing the results of the operation. Success!

We’ve just set up these nodes to be web servers.

Example Workflow 2: Unifying Accounts

In this workflow scenario, we’ll look at a situation where ad-hoc changes have caused difficult-to-manage configuration drift; specifically, the case where developers have created similar users for logging in to various hosts that all behave a bit differently. Our task is to unify this developer account so that it’s identical across the entire network.

To start off, we’ll find all of the user resources on our network, and select the resource titled ‘developer’.

This resource currently exists in 3 different forms across our networks, with different shells and home directories. To proceed, we will select ‘Clone this variation’ for the version we want on all of our nodes: the one with the shell ‘/bin/zsh’. This takes us to a preview screen, showing which nodes will have their current version of the user ‘developer’ overwritten by the change.

By clicking ‘Clone’, we will unify this user resource to have the same characteristics across all of our currently selected nodes. Again, this takes us to a results page showing success.

These example workflows only begin to show the power available through Live Management, and the wider orchestration capabilities in Puppet Enterprise 2.0. In the next post in this series, we will explore more advanced capabilities available via the new Puppet Enterprise Console, including controlling your Puppet runs and manually controlling our framework that supports distributed, parallel job execution (aka the Marionette Collective) through the Live Management GUI.

Related Content:

• Download ten free nodes of Puppet Enterprise 2.0 and try out this new capability.
• Sign up for a webinar that demos Live Management.
• Read about the new capabilities in Puppet Enterprise 2.0.
• Learn about the advanced uses in Live Management Part 2: Controlling Puppet and Invoking Agents.

One of the features offered by Puppet is the ability to break up infrastructure configuration into environments. With environments, you can use a single Puppet master to serve multiple isolated configurations. For instance, you can adopt the development, testing and production series of environments embraced by a number of software development life cycles and by application frameworks such as Ruby on Rails, so that new functionality can be added incrementally without interfering with production systems. Environments can also be used to isolate different sets of machines. A good example of this functionality would be using one environment for web servers and another for databases, so that changes made to the web server environment don’t get applied to machines that don’t need that configuration.

Mapping the Puppet code base against the environments shows the power of this method. People often use a version control system to manage the code, and create a set of branches that each map to an environment. Adopting the development, testing and production workflow, we can have a puppet.conf that looks something like this:

[main]
  server = puppet.example.com
  environment = production
  confdir = /etc/puppet
[agent]
  report = true
  show_diff = true
[production]
  manifest = /etc/puppet/environments/production/manifests/site.pp
  modulepath = /etc/puppet/environments/production/modules
[testing]
  manifest = /etc/puppet/environments/testing/manifests/site.pp
  modulepath = /etc/puppet/environments/testing/modules
[development]
  manifest = /etc/puppet/environments/development/manifests/site.pp
  modulepath = /etc/puppet/environments/development/modules

With this configuration, we could map three Git branches for these environments and set up a central Git repository with post receive hooks. When changes were pushed to this repository, they would be automatically deployed to the puppet master. The example post-receive hook later in this post will work with this kind of environment setup.

Dynamic Environments

While there are benefits to having a set of branches and environments like the previously outlined configuration, there are also drawbacks with a static set of environments. This model is somewhat constrained by enforcing a single workflow. For instance, if multiple people are working on experimental features in the development branch, they’ll have to constantly merge their code through the development process. If multiple people are developing different features in the same branch, they’ll all have to work with unrelated changes in the code. In addition, migrating a single feature to testing or production gets more complicated if multiple features are in the same branch.

Modern distributed version control systems like Git handle these constraints by making branch creation and merging lightweight operations, allowing us to generate Puppet environments on the fly. Puppet can set up environments with an explicitly defined section in the configuration file, but we can exploit the fact Puppet will set the $environment variable to the name of environment it is currently running under. With this in mind, we can write a puppet.conf to resemble this:

[main]
  server = puppet.example.com
  environment = production
  confdir = /etc/puppet
[master]
  environment = production
  manifest    = $confdir/environments/$environment/manifests/site.pp
  modulepath  = $confdir/environments/$environment/modules
[agent]
  report = true
  show_diff = true
  environment = production

This handles the dynamic environment aspect; all we have to do is create a directory with manifests in $confdir/environments and we have created that environment. Generating new environments using Git is similarly easy. We can create a central Git repository somewhere, with a post-receive hook that looks something like this:

#!/usr/bin/env ruby
# Puppet Labs is a ruby shop, so why not do the post-receive hook in ruby?
require 'fileutils'
 
# Set this to where you want to keep your environments
ENVIRONMENT_BASEDIR = "/etc/puppet/environments"
 
# post-receive hooks set GIT_DIR to the current repository. If you want to
# clone from a non-local repository, set this to the URL of the repository,
# such as git@git.host:puppet.git
SOURCE_REPOSITORY = File.expand_path(ENV['GIT_DIR'])
 
# The git_dir environment variable will override the --git-dir, so we remove it
# to allow us to create new repositories cleanly.
ENV.delete('GIT_DIR')
 
# Ensure that we have the underlying directories, otherwise the later commands
# may fail in somewhat cryptic manners.
unless File.directory? ENVIRONMENT_BASEDIR
  puts %Q{#{ENVIRONMENT_BASEDIR} does not exist, cannot create environment directories.}
  exit 1
end
 
# You can push multiple refspecs at once, like 'git push origin branch1 branch2',
# so we need to handle each one.
$stdin.each_line do |line|
  oldrev, newrev, refname = line.split(" ")
 
  # Determine the branch name from the refspec we're received, which is in the
  # format refs/heads/, and make sure that it doesn't have any possibly
  # dangerous characters
  branchname = refname.sub(%r{^refs/heads/(.*$)}) { $1 }
  if branchname =~ /[\W-]/
    puts %Q{Branch "#{branch}" contains non-word characters, ignoring it.}
    next
  end
 
  environment_path = "#{ENVIRONMENT_BASEDIR}/#{branchname}"
 
  if newrev =~ /^0+$/
    # We've received a push with a null revision, something like 000000000000,
    # which means that we should delete the given branch.
    puts "Deleting existing environment #{branchname}"
    if File.directory? environment_path
      FileUtils.rm_rf environment_path, :secure => true
    end
  else
    # We have been given a branch that needs to be created or updated. If the
    # environment exists, update it. Else, create it.
 
    if File.directory? environment_path
      # Update an existing environment. We do a fetch and then reset in the
      # case that someone did a force push to a branch.
 
      puts "Updating existing environment #{branchname}"
      Dir.chdir environment_path
      %x{git fetch --all}
      %x{git reset --hard "origin/#{branchname}"}
    else
      # Instantiate a new environment from the current repository.
 
      puts "Creating new environment #{branchname}"
      %x{git clone #{SOURCE_REPOSITORY} #{environment_path} --branch #{branchname}}
    end
  end
end

(Make sure that the post receive hook is executable!)

Creating and using a new environment

% git branch
* production
% git checkout -b new_feature
Switched to a new branch 'new_feature'
% mvim site.pp
% git add site.pp
% git commit -m 'Implemented my feature'
[new_feature 25a9e1b] Implemented my feature
 1 files changed, 1 insertions(+), 0 deletions(-)
% git push origin new_feature
Counting objects: 5, done.
Writing objects: 100% (3/3), 255 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
remote: Creating new environment new_feature
To git@git.host:deploy.git
 * [new branch]      new_feature -> new_feature

And from here on out, you can use the new_feature environment on your hosts, and use git like you would with any code base.

This development model gives us some simple access control. Utilizing access control with a tool like gitolite, we can allow people to generate new environments to test their own code, but deny them access to change the production environment. This allows us to institute some sort of change control, by requiring all code to be reviewed by a merge master before inclusion into production, and allows code to be tested and verified before the request for submission is made.

This is ideal for junior sysadmins, as it allows them to experiment with Puppet while preventing accidental pushes of incorrect code. However, it is important to keep in mind that while this can prevent accidents, it cannot prevent malice. Unless otherwise configured, the Puppet master will run all manifests as one user, so a malicious user could attempt to manipulate other branches than the one they created or used.

Additional resources/related content:

• The model of dynamic Puppet environments with Git was pioneered at Portland State University, and one of our Professional Service Engineers, Hunter Haugen, originally wrote up the basic concept on his blog.
• Gitolite has been a fundamental underpinning for managing and deploying puppet manifests, and it’s a very powerful tool. You can check out the documentation here.
• If you want to learn more about how environments can be used and configured, you can find the official documentation here.
• Pro Git is an excellent book on Git, and they do a great job of outlining the different git hooks and how you can use them. You can read the relevant chapter online.

Today we’re excited to announce Puppet Enterprise 1.2 with two great new features that give you the intelligence you need to prove you’re in compliance with your change management processes.

With Puppet Enterprise Compliance you set a desired-state for each of your systems and monitor them for any changes, right from our web-based Dashboard, creating a baseline. You’ll be alerted to changes on monitored nodes and you choose to accept or reject each change. Accepted changes will become part of the baseline, and rejected changes will still show up in Dashboard until you manually update the node to your desired state. This helps you create a maintenance to do list, ensuring nothing slips through the cracks.

There are lots of different times this type of monitoring and insight is important. If you have a tremendous amount of change in your environment Puppet Enterprise Compliance monitors priority resources, giving you the agility to act immediately on unapproved changes. Compliance also allows you to track spent time and resources on unmanaged resources that go through periodic, high-volume change, indicating when these troublesome resources are ready for Puppet’s continuous automated management.

With Puppet Enterprise Compliance you can:

• Confirm changes are in compliance with change management policies
• Identify unauthorized or unexpected change
• Improve visibility and enforce accountability across the enterprise
• Reduce unplanned downtime and improve mean time to repair
• Gather data to track IT resources and costs
• Monitor systems under consideration for Puppet automated management

And with Puppet you’re not limited to auditing just content and metadata of files, like other monitoring software. Using built-in Puppet resource types you can also audit user accounts, packages, services, cron jobs, or anything else that Puppet manages. You can even write plug-ins to monitor your custom resources.

Also with this Puppet Enterprise 1.2 release, we’ve solved another compliance headache when it comes to managing user accounts. Puppet Enterprise now has the built-in capability to support best practices for user account management and ensure compliance with internal policies. With Puppet you can assess and make changes on all of your machines with one command, without the availability risks you find in central directory software. Puppet even manages SSH keys for password-less access, using public and private keys instead of insecure passwords. And Puppet records all account changes, creating an auditable trail and ensuring internal change management policies are followed.

Puppet Enterprise user account management now makes it easier to:

• Perform required periodic password changes
• Provision a new user
• Grant user access
• Revoke user access
• Remove a user from the database
• Grant limited access to a user, as in giving someone permission to reboot a web server but not permission shut down the machine

Find out more about components included in this release in the Puppet Enterprise FAQ. 

To see PE 1.2 in action register for Introduction to Puppet Enterprise 1.2 Live Webinar this Wednesday, August 31 at 11 am PDT.

Want to try Puppet Enterprise 1.2 for yourself?

Download Puppet Enterprise 1.2 now and start managing 10 nodes for free.

The actions of IT Operations teams influence the efficiency, availability, cost of delivery, and customer satisfaction of the services your organization is delivering to your customers. This means getting IT right, particularly IT Operations, has become a crucial part of doing business in the modern market. IT Operations managers face three major challenges—finding the right people, building the right culture, and creating the right processes. This won’t be a surprise to most managers, as they are the building blocks of any good team or organization. But IT has often lagged behind in tackling these challenges.

Of all of these challenges, finding the right IT Operations people has perhaps become the most serious and potentially has the most impact on improving your operations. The methods and incentives for recruitment have changed. Premium candidates expect challenging workplaces, cutting edge technology, excellent benefits, and a competitive salary.

The best candidates also expect companies to provide environments where they can share their skills and ideas with the wider Operations community. This includes allowing them to contribute to open source projects, publishing tools and code, and engaging with the community through social media and technical channels.

By opening up the way you work both internally and externally, companies can begin attracting the right candidates. Internally, look at the ideas fomenting around the DevOps community about communication and collaboration. Externally, publish your successes. Publish your Puppet modules. Look at the tools you develop internally and identify those that you think others might be interested in and open source them. Publish them on places like GitHub.

Follow this up with getting your staff to talk, blog, and present about the tools and techniques they use and most importantly about the difference it makes to how you run your infrastructure. Once you start to treat and view your IT Operations team as ambassadors for your organization in the broader Operations community, you’ll find that their success is reflected in a greater ability to hire.

There is a place where you can meet Operations rock stars and parade your organization’s virtues. That place is the upcoming PuppetConf event being held over September 22nd and 23rd in Portland, Oregon. Send your IT Operations staff, let them share ideas and meet a thriving community and demonstrate that you’re working towards creating a business that employs IT Operations rock stars. It’s more than just Puppet of course, it’s an Operations conference with DevOps, Cloud and Puppet tracks and will be attended by some of the smartest people in Operations and technology. Register today and save 10% using discount code cultureshift.

Cross-posted from the PuppetConf blog.

A wise television character once advised to “Live every week like it’s Shark Week,” and while we appreciate the programming on the Discovery Channel, we’d rather apply this adage to System Administrator Appreciation Day. Your sysadmin works day in and day out (or you, as a sysadmin, work day in and day out), and there are little ways to show gratitude that don’t require waiting for a ThinkGeek shipment. Rounding out our contest winners, Ramin Khatibi explains the small things that can make every day a little more like SysAdmin Day.

Perfect Day

Technology can fail spectacularly, but it hardly ever surprises me. Requirements, projects, schedules, and coworkers are always surprising me.

“We promised what?”

“DNS doesn’t work like that.”

“The new environment will be up in five hours, no sooner. So… when did we decide this project was going forward?”

“Fry’s doesn’t stock SANs. At least not yet.”

The start of my perfect day is being invited to all the pertinent planning meetings. Managers that understand Ops can use just as much lead time as Dev. Dev comes out of planning and says we need to look at MySQL capacity due to the new features going into this sprint. Sales mentions uptake is flat at the moment, but the end of the month deals are going to be strong.

The middle is planning, coding, fixing, tweaking, herding, and knocking off tasks with time to test, write docs, design review, and do all the extra things that ensure quality work.

The end is knowing that no department is waiting for me to complete a task for them.

Related posts:

SysAdmin Day Approacheth: Are you Ready?

The Big Red Button: The Dream of a SysAdmin

System Administrator Appreciation Day: Thank You, Thank You, Thank You

SysAdmin Day Contest Winner: Saving the Day

Reason #7959 to thank you sysadmin: Finding and implementing solutions at 1 am on New Year’s Day. Bejoy Mathews tells the tale:

SysAdmin to the Rescue

I’ll tell you a story which saved one of my old organizations a great deal of data loss. This was on a New Year’s Eve. I handled support for PeopleSoft on RedHat Linux—OS/Web/App. Though DB was on Oracle, I supported the OS part. The main database administrator did not backup for a week, and the data was crucial for the whole organization’s daily basic requirements—especially HR. The storage device cracked one of the hard disk drives some time earlier, and since that was not monitored properly we did not put the proper HDD in in time. One more HDD of the three went down and the database crashed. This was a huge question mark for the team of sysadmins and database admins. Many did not turn up due to fear of losing their jobs. The storage company told us the only way to correct this was to format the whole data on the storage. We all were really worried—we had already crossed the Dec 31st 11:59pm line, and it was January 1st, sitting in the Data Centre at 1-2 am in the morning. Even though the storage company advised us to format, we thought of using one of their own web-based utilities to see if we could recover. That worked really well—we were able to do something that even the storage owners couldn’t guide us to. I did not have to format it. And once the storage was back on line, I was able to retrieve data through Dec 30th at least, with only one day of lost data. That brought smiles to everyone and brought many out of their hiding places. Sysadmin skilled experience was a real help, when all seemed lost! Servers/Apps/DBs all had their respective halos on. It was truly a Happy New Year for me at least!

Check back later this week for the last of our sysadmin stories.

Related posts:

SysAdmin Day Approacheth: Are you Ready?

The Big Red Button: The Dream of a SysAdmin

System Administrator Appreciation Day: Thank You, Thank You, Thank You