Module of the Week: puppetlabs/stdlib - Puppet Labs Standard Library Part 4

Purpose Standard library for creating Puppet modules
Module puppetlabs/stdlib
Puppet Version 2.6+
Platforms Redhat, Debian, Solaris, Mac OS X, Windows

Welcome back to the module of the week! Previously we covered facter-dot-d. This week we're switching our focus to the puppetlabs/stdlib data functions: getvar, loadyaml, and merge.

The data functions provide the ability to dynamically lookup data from Puppet namespaces, load data from external YAML files, and merge Hashes. These powerful features make it easy to separate Puppet code from data without the complexity of using an External Node Classifier or Custom Facts; leaving your manifests more readable and maintainable as a result.

Example Usage

In this brief tutorial I'm going to cover the loadyaml and merge functions to show you just how easy it is to work with external data:

The loadyaml function

As its name implies the loadyaml function allows us to load data from an external YAML file.

/etc/puppet/data/webservers/data.yaml

ports:
  http: 80
  https: 443
  tomcat: 8080

webdir: /var/www/html

/etc/puppet/modules/webservers/manifests/init.pp

class webservers {
  include 'stdlib'

  $data = loadyaml('/etc/puppet/data/webservers/data.yaml')

  $http_port  = $data[ports][http]   # => 80
  $https_port = $data[ports][https]  # => 443
  $webdir     = $data[webdir]        # => '/var/www/html'

  ...
}

Lets step through this:

  • First we create a data file /etc/puppet/data/webservers/data.yaml
  • Then we make a call to loadyaml with the full path to our data file
  • Next loadyaml processes data.yaml and returns a Hash, which is then assigned to $data
  • Finally we set the $http, $https, and $webdir variables by referencing the $data Hash

The merge function

The merge function allows us to merge two or more Hashes:

/etc/puppet/data/webservers/data.yaml

ports:
  http: 80
  https: 443
  tomcat: 8080

webdir: /var/www/html

/etc/puppet/data/database/data.yaml

ports:
  db: 5432

/etc/puppet/modules/webservers/manifests/init.pp

class webservers {
  include 'stdlib'

  $myhash         = { database_user => 'webdata', webdir => '/var/html' }
  $webserver_data = loadyaml('/etc/puppet/data/webservers/data.yaml')
  $database_data  = loadyaml('/etc/puppet/data/database/data.yaml')

  $data = merge($myhash, $webserver_data, $database_data)

  $http_port  = $data[ports][http]   # => 80
  $https_port = $data[ports][https]  # => 443
  $webdir     = $data[webdir]        # => '/var/www/html'
  $db_port    = $data[ports][db]     # => 5432
  ...
}

Here we are calling the merge function with three Hashes ($myhash, $webserver_data, and $database_data) which yields a single $data Hash.

Note, the merge function overrides duplicate keys with the argument farthest to the right "winning". In the above example the webdir value provided by $myhash is overridden by the value provided by $webserver_data.

Conclusion

The data functions provided by puppetlabs/stdlib makes it easy to work with internal and external data. While great for simple use-cases, this mode of operation doesn't scale. Once the number of external data files or Puppet modules grows, you'll find your modules littered with lots of getvar, loadyaml, and merge function calls.

A better solution for more complex requirements is Hiera. While providing all the features of getvar, loadyaml, and merge, Hiera includes the ability to load data from virtually anywhere. Via Hiera's pluggable backend interface, you'll be able to pull data from external databases such as Redis, or JSON files; the possibilities are endless.

That wraps up our coverage of the puppetlabs/stdlib module. Hopefully you can now leverage some of its features in your own modules and have a sense of how robust Puppet modules can be.

Additional Resources

Comments

jblaine

jblaine

What happened to getvar?

Kelsey Hightower

Kelsey Hightower

jblaine,

I intentionally left out coverage of getvar mainly to keep the post focused. IMO the details of getvar require a little more explanation around Puppet scope, namespacing, and the dangers of coupling between modules.

For those interested in getvar there is a small example on how to use it on the puppetlabs-stdlib home-page. I may also consider doing a "special" post if there is demand for deeper coverage.

Barry

Barry

Hiera doesn't really include the functionality of merge. In fact, Hiera's lack of merging for defaults is exactly the reason for requiring stdlib's merge; functions like hiera_hash take three arguments, the root key, the defaults, and the overrides. But the defaults don't end up as the left hand side of a merge with the hiera value, they are only used if the hiera key is completely missing.

So if you want a deep set of defaults, then a selective set of configuration in hiera (possibly merged across multiple yaml files), you can't do it without stdlib's merge.

Leave a comment

Tradeshow
Sep 3
Puppet Practitioner
Sep 9
Speaking Engagement
Sep 11
Tradeshow
Sep 12