Entries filed under: Open Source

Back to Index

Module of the Week: jmcdonagh/clamav – Manage ClamAV and the freshclam service

Posted on
By
Joe McDonagh
in
Blog, Community, Extending Puppet, guest post, How to, Module of the Week, Modules, Open Source
Responses
0 Comments
Purpose Comprehensive clamav and freshclam classes
Module jmcdonagh/clamav
Puppet Version 2.6+
Platforms Ubuntu 10.04 LTS+

ClamAV is an open-source scanning engine for malware, virus, and trojan detection. It is often used in conjunction with an MTA such as Postfix. It comes with a built-in service for AV signature updates called freshclam.

This module is intended to offer a comprehensive interface for Puppet to configure both freshclam and clamav services. Each class takes many parameters, all of which are down-cased versions of clamav or freshclam configuration options. Both the clamav and clamav::freshclam classes are also intended to be easily removed from a system, by setting their $ensure parameters to ‘absent’.

Installing the module

Complexity Easy
Installation Time 5 minutes

This module is on Puppet Forge and can be installed with puppet’s module sub-command:

puppet module install jmcdonagh/clamav

Note that if you run this as a regular user, the module will be installed in your home directory. If you are root, it will probably go into /etc/puppet/modules.

Module and Resource Overview

First, you should either check out the code from GitHub (https://github.com/thesilentpenguin/puppet-clamav), or install the module with the module tool as outlined above and browse the code with your favorite editor. I recommend having the manifest open while I walk you through the code.

Let’s have a look at the clamav base class in manifests/init.pp. Scroll down to the beginning of the class and you will immediately notice that there are many parameters. These parameters were taken from the clamav man pages on Ubuntu 10.04 LTS systems, and down-cased to maintain status quo code style.

After all the parameters, we’ve got some sanity checks. We’ll fail the catalog compilation (using the fail() function) if we get passed any bizarre values or we’re trying to compile a catalog for an incompatible system. One of my stock checks is a minimum OS version. This is a pretty good practice for numerous reasons. For one, it helps consumers know where your code is intended to be used. That is absolutely crucial if you will be supporting this code. Before someone breaks a Solaris system using a Debian-only manifest, the compilation simply fails and outputs a clear error.

After the sanity checks, various variables are set based on the value of $ensure. This will alleviate the need to use selectors inside resource definitions, which can lead to hard-to-read and thus hard-to-understand code. Here is the code that I am referring to which sets all of these variables based on $ensure:

   if ($ensure == "present") {
  	$file_notify    = Service["clamav-daemon"]
  	$file_require   = Package["clamav-daemon"]
  	$svc_before	= undef
  	$svc_enable	= "true"
  	$svc_ensure	= "running"
  	$svc_require    = Package["clamav-daemon"]
   } else {
  	$file_notify    = undef
  	$file_require   = undef
  	$svc_before	= Package["clamav-daemon"]
  	$svc_enable	= "false"
  	$svc_ensure	= "stopped"
  	$svc_require    = undef
   }

You may have noticed that some of these variables appear to be relationship targets. This is because I always attempt to include the ability to remove all of a class’ resources properly. You’ll notice that, if removing the clamav class by setting $ensure to ‘absent’, the resource requirements go the opposite direction of the way they go with $ensure set to ‘present’. You might get away with using $ensure in this manner and not reversing relationships, but for me it’s not worth taking that chance, and I just do it right the first time.

Now onto the actual resources. The clamav and freshclam classes are what I call FPS classes. That is, File-Package-Service classes. What I mean by this is that there is a Package to install, configuration File(s) to set up, and a Service to manage. The Service and File(s) Require(s) the Package, and the File(s) Notify the Service. This paradigm is likely common to you if you have been using Puppet for some time now.

So first you’ll notice clam’s configuration file. This is a template that references all the class parameters to configure clamav. It uses some of the variables defined above to set up relationships and notify the service.

Next is the Package resources, which include a couple of extras. I include the dev packages because a lot of my work involves the rubygem ecosystem. This means that when I have a requirement for software on a system, I often need the -dev packages to compile extensions. Remember, now that you have parameterized classes at your disposal, it is trivial to add boolean features to your modules such as ‘$dev_packages = “true”’. That could serve as an option to include or not include the management of the -dev packages.

Finally we have the service. Not much here, mostly all configured via the variables defined based on $ensure. All our resources look nice and compact thanks to that variable setting stanza.

The clamav::freshclam class is also an FPS class. It is similar in the sense that it has every freshclam configuration option down-cased as a class parameter. The only noticeable difference between the two classes is the actual clamav and freshclam configuration options.

Testing the module

This module is most easily tested by using ‘include clamav’, which will by default set up clamav on the current system.

puppet apply -e ‘include clamav’ clamav/manifests/init.pp

If you do this on an incompatible system, you should see something like this:

Your OS (Darwin) is not supported by this code! at /etc/puppet/modules/clamav/manifests/init.pp:104 on node goldmember.microcosm.thesilentpenguin.com

After this successfully finishes on a compatible system, you should be able to see the clamav daemon running:

[~] > sudo service clamav-daemon status
 * clamd is running

You can test the config by scanning a test file by using the clamdscan program:

[~] > clamdscan website.erb
/home/jmcdonagh/website.erb: OK
 
----------- SCAN SUMMARY -----------
Infected files: 0
Time: 0.017 sec (0 m 0 s)

The freshclam class can be tested in the same way.

Configuring the module

Complexity Easy
Installation Time 5 minutes

The module configuration should be sane by default. Every parameter available in the Ubuntu 10.04 LTS repo version of clam should be available. The default values for the various parameters should match the defaults in Ubuntu 10.04 LTS. If you need to tweak any setting whatsoever, simply look up the configuration option in the clam man page, down-case all upper-case letters, and pass that as a parameter to the class. For example:

class {
   “clamav”:
      ensure    => “present”,
      logsyslog => “true”;
}

Would set up clamav with the LogSyslog clamav configuration option set to true.

Example usage

The same usage outlined above for testing is a typical use case of clamav. Simply enable those classes in the node definition and you will have a working clam setup, ready to check for malicious content coming in through e-mail. You could even potentially integrate clam with other services, like scanning user uploads to your custom webapp. Freshclam will update from the canonical repo every day with the default settings, ensuring you have the latest virus definitions.

Conclusion

To me, this module is an ideal example of module design. It is small in scope, yet as complete as possible of an interface to the clam configuration. This allows for you to easily set up clamav and freshclam on a node with nearly no tweaking or external dependencies necessary. If you end up needing some tweaking, as you have seen it is trivial to make small changes to clam with this module. It also has some basic compatibility checks to ensure your consumers know where this module will work.

This module would probably work on Debian 6, but I haven’t tested. To add Red Hat support some variables for the package and service name(s) would probably have to be added, and all of my gigs that use clam are on Ubuntu 10.04 LTS.

Additional Resources

Join us for PuppetConf 2012

Posted on
By
jose
in
Blog, Community, Conferences and Workshops, General News, Open Source, Puppet Enterprise, PuppetConf, Tips, Training
Responses
0 Comments

Registration for PuppetConf ’12 is now open. We’re in a new city and a new facility, with new tracks and new programs. Look forward to 5 concurrent tracks over 2 days focusing on all things operations. Our new venue offers one large theater for keynotes, while our second auditorium will be dedicated to Puppet Community presentations and hacking space. We’re introducing a hands-on lab component to the conference, and we’re happy to announce that we’ll be offering the first ever Puppet Admin and Puppet Developer Certification exams at PuppetConf.

With over 70 speakers, 600+ community members, and the Puppet Labs team, PuppetConf is a must-attend event. We’re looking forward to seeing you in San Francisco!

Here’s a quick recap of last year:

All 2011 talks can be viewed on the Puppet Labs YouTube Channel.

Module of the Week: rtyler/jenkins – Continuous Integration Server

Posted on
By
Jeff McCune
in
Blog, Community, Extending Puppet, How to, Module of the Week, Modules, Open Source, Tips
Responses
3 Comments »
Purpose Jenkins is an extensible, open source continuous integration server.
Module rtyler/jenkins
Puppet Version 2.7+
Platforms Enterprise Linux 5 and 6, Debian 6, Ubuntu

This week’s module helps implement and manage Jenkins CI with Puppet. We all need to test our code when we make changes. Saying so is easy, doing so is a lot more difficult. Jenkins makes testing easier. Every time you make a change, Jenkins will test everything still works. It’s easy to get started with Jenkins by using a Puppet module written by R. Tyler Croy. This module automatically installs the Jenkins software, configures the service, and provides an simple way to add functionality by automatically installing any of the 400+ Jenkins plugins. You should give this module a spin if you’re looking for an easy way to try out Jenkins or puppetize an existing Jenkins server. Install it, classify a node with the jenkins class in the Puppet Enterprise Console and Jenkins is running and ready to do your bidding.

Installing the module

Complexity Easy
Installation Time 5 minutes

Installing the Jenkins module is straightforward. The module is published on the Puppet Forge so we’re able to install it directly onto a Puppet Master system with one command. In this example we’re going to use Puppet Enterprise 2.5 to install the module on our Puppet Master. Then we’ll classify a Ubuntu Lucid node with the jenkins class and see how Puppet automatically brings up the Jenkins console web interface.

In this example we’ve installed Puppet Enterprise 2.5 on a CentOS 6.2 machine. (Note: if you don’t have Puppet Enterprise, you can easily install this module using the puppet-module gem or upgrade to Puppet 2.7.13 or later to get the puppet module subcommand.)

root@pe-centos6:~# puppet module install rtyler-jenkins
Preparing to install into /etc/puppetlabs/puppet/modules ...
Downloading from http://forge.puppetlabs.com ...
Installing -- do not interrupt ...
/etc/puppetlabs/puppet/modules
└─┬ rtyler-jenkins (v0.2.2)
  └── puppetlabs-apt (v0.0.2)
root@pe-centos6:~#

Now that we have the Jenkins module installed, we just need to add the jenkins class to the configuration catalog of an agent system. In this example I’m going to use the Puppet Enterprise Console to do this, but you could just as easily accomplish this using a node declaration in your site.pp manifest in Puppet.

First, we’ll add the jenkins class to the Puppet Enterprise Console. To do this we click the “Add Class” button in the bottom left corner of the console, and name it “jenkins”:

Once the jenkins class has been added to the Puppet Enterprise Console we can create a group to associate our Ubuntu Lucid node with the jenkins class. This image shows the “Add group” screen:

Once we click “Create” the jenkins class will be included in the configuration catalog of the Ubuntu Lucid node. Let’s see what this looks like on the Puppet Agent now.

root@pe-ubuntu-lucid:~# puppet agent --test
info: Retrieving plugin
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/puppet_vardir.rb
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/facter_dot_d.rb
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/root_home.rb
info: Caching catalog for pe-ubuntu-lucid.2012-04-09.15837
info: Applying configuration version '1334091722'
notice: /Stage[main]/Jenkins/Package[jre]/ensure: \
  current_value purged, should be 1.7.0 (noop)
notice: Class[Jenkins]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Jenkins::Repo::Debian/Apt::Source[jenkins]/\
  Apt::Key[Add key: D50582E6 from Apt::Source jenkins]/\
    Exec[3ae8ac921fb6f9e7a9be3adca21e45c2226e43b2]/returns: \
      executed successfully
notice: /Stage[main]/Jenkins::Repo::Debian/Apt::Source[jenkins]/\
  File[jenkins.list]/ensure: \
    defined content as '{md5}941681fa47a0a1b579f98bce22bbcaf6'
info: /Stage[main]/Jenkins::Repo::Debian/Apt::Source[jenkins]/\
  File[jenkins.list]: Scheduling refresh of Exec[jenkins apt update]
notice: /Stage[main]/Jenkins::Repo::Debian/Apt::Source[jenkins]/\
  Exec[jenkins apt update]: Triggered 'refresh' from 1 events
notice: /Stage[main]/Jenkins::Package/\
  Package[jenkins]/ensure: ensure changed 'purged' to 'present'
notice: Finished catalog run in 150.88 seconds
root@pe-ubuntu-lucid:~#

Once puppet agent finished, the Jenkins service will be available at http://HOSTNAME:8080/ where HOSTNAME is the hostname of the node Puppet configured Jenkins on. You should expect to see something like this:

In the next section we’ll cover some additional resources that make it easier to manage and automate Jenkins plugins.

Resource Overview

The main jenkins class is be the only class you need to add to your node classification. The module contains other classes, but these are automatically managed for you by the jenkins class.

Jenkins is a very extensible CI system. Jenkins-ci.org provides over 400 plugins to customize it to your specific needs. The Jenkins module makes it easy to manage these plugins with Puppet by defining the jenkins::plugin resource type.

Jenkins cannot monitor a Git repository for changes by default, but developers and system administrators often need to have their jobs run when they publish changes to a Git repository. Jenkins helps solve this problem by providing a Git plugin allowing Jenkins to automatically clone Git repositories and poll them for changes.

To have Puppet automatically install and manage the Git plugin, declare a jenkins::plugin resource in the nodes configuration catalog. In this example, I’m going to create a new class named site::jenkins_plugins and declare the necessary resource. We’ll then add this new class and see how Puppet automatically manages the service.

Here’s the copy of /etc/puppetlabs/puppet/modules/site/manifests/jenkins_plugins.pp that defines the class:

# = Class: site::jenkins_plugins
#
#   This class manages the Jenkins plugins we want to use at our site.
#
# = Requires
#
#   puppet module install rtyler-jenkins
#
# = Sample Usage
#
#   include site::jenkins_plugins
#
# (MARKUP: http://links.puppetlabs.com/puppet_manifest_documentation)
class site::jenkins_plugins {
  # This resource default will ensure the Jenkins service
  # restarts automatically after Puppet configures all plugins.
  Jenkins::Plugin { notify => Class[jenkins::service] }
  # Tell Puppet to configure and manage the Jenkins Git Plugin.
  jenkins::plugin { 'git': }
}

With this class in place in a manifest we simply need to classify the Lucid system with this site::jenkins_plugin class like we did before with the jenkins class.

First, add the class to the Puppet Enterprise Console:

Then, update the jenkins group to also contain the site::jenkins_plugins class. This will associate nodes in this group with both the jenkins and the site::jenkins_plugins class.

Finally, run puppet agent on the Lucid node to apply the new configuration with the Git plugin resource.

root@pe-ubuntu-lucid:~# puppet agent --test
info: Retrieving plugin
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/puppet_vardir.rb
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/facter_dot_d.rb
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/root_home.rb
info: Caching catalog for pe-ubuntu-lucid.2012-04-09.15837
info: Applying configuration version '1334094999'
notice: /Stage[main]/Jenkins/Package[jre]/ensure: \
  current_value purged, should be 1.7.0 (noop)
notice: /Stage[main]/Site::Jenkins_plugins/Jenkins::Plugin[git]/\
  Group[jenkins]/ensure: created
notice: /Stage[main]/Site::Jenkins_plugins/Jenkins::Plugin[git]/\
  User[jenkins]/gid: gid changed '65534' to 'jenkins'
notice: /Stage[main]/Site::Jenkins_plugins/Jenkins::Plugin[git]/\
  File[/var/lib/jenkins]/group: group changed 'adm' to 'jenkins'
notice: /Stage[main]/Site::Jenkins_plugins/Jenkins::Plugin[git]/\
  File[/var/lib/jenkins/plugins]/group: group changed 'nogroup' to 'jenkins'
notice: /Stage[main]/Site::Jenkins_plugins/Jenkins::Plugin[git]/\
  Exec[download-git]/returns: executed successfully
info: /Stage[main]/Site::Jenkins_plugins/Jenkins::Plugin[git]/\
  Exec[download-git]: Scheduling refresh of Service[jenkins]
info: Jenkins::Plugin[git]: Scheduling refresh of Class[Jenkins::Service]
notice: Class[Jenkins]: Would have triggered 'refresh' from 1 events
info: Class[Jenkins::Service]: Scheduling refresh of Service[jenkins]
notice: /Stage[main]/Jenkins::Service/Service[jenkins]: \
  Triggered 'refresh' from 2 events
notice: Finished catalog run in 11.37 seconds
root@pe-ubuntu-lucid:~#

After running Puppet, we’re able to see the newly installed plugin in the Jenkins console interface.

The Jenkins module would be pretty darn good if all it did was install the package and manage the Jenkins service automatically. In this section we’ve seen that the module puts forth some extra effort and makes it easy to install and activate additional functionality in Jenkins through the use of a Puppet defined resource type. This combination allows you, as a Puppet user, to easily get started with Jenkins and customize it to your specific needs. All with only a few commands and a couple of lines of Puppet.

Conclusion

In this post we walked through installation and use of R. Tyler’s jenkins module, which allows us to automatically run the Jenkins service and manage the Git plugin. With this configuration, we’re ready to start automatically watching our Git repositories for changes and running automated tests after each change. Not only does this module provide an easy and fast way to configure the Jenkins service on RHEL and Ubuntu systems, it provides an easy way to install any of the 400+ Plugins available on jenkins-ci.org.

If you’re interested in contributing to the Jenkins module some good features would be the ability to manage jobs using Puppet resources and adding additional platform support. Pull requests are always welcome!

Additional Resources

Writing Great Modules: An Introduction

Posted on
By
Ryan Coleman
in
Blog, Community, DevOps, Extending Puppet, How to, Module of the Week, Modules, Open Source, Tips
Responses
1 Comment »

We’re taking a brief break from your regularly scheduled module of the week goodness with a mini series to discuss some of the elements that go into writing great Puppet modules. I’ll try to sum it up in a few sentences:

Your module should be immediately useful to me after I install it. Documentation should be complete and the code itself should adhere to the community style guide. If I need to modify the behavior of your module, you should provide me an interface to do it so I don’t have to modify your modules manifests. If I contribute features or bug fixes, I expect you to explain the process and collaborate with me on improving the module.

I’ll be honest, writing a module isn’t always an easy task, and all the things discussed here will add time and complexity to that module. This multi-part series on writing great modules is all about giving you the knowledge and tools to make your module more reusable, immediately useful to others and easier for your team and the community at large to maintain. My theory is that time spent initially building a more robust module will return you ten-fold the value over the life of that module in your environment.

With that in mind, I hope you find these concepts useful and return to the series next time as I dig deeper, leaving you to build your next great Puppet Forge module!

Before We Begin

If you’re new to Puppet Modules or simply want a refresher, our incredible documentation team has recently created a page on Module Fundamentals.

De-Duplication and Collaboration

To borrow a phrase, “An hour in the Forge can save you days in the editor.” There are over 300 modules in the Puppet Forge now, covering any number of things. When you get that itch for your next Puppet module, try searching the Puppet Forge first. If you find something you like that suits your need, awesome! If you find something that is mostly what you want but need some features, consider contributing to the author’s module instead of creating your own. Here are a couple of great docs to help you in consuming or publishing a Forge module:

Documentation

So you’ve written an awesome module that conjures unicorns from the cloud and you’ve published it to the Puppet Forge. But when I download it, I shouldn’t have to dig into your manifest’s directory and read the code just to figure out how it works. Documentation is key to quality modules that people will use.

There’s really no secret sauce here, just write some text about your module, addressing the following questions:

  • What does your module do?
  • Does your module require anything else to function?
  • How should I declare your class(es)? Include example usage for each piece of functionality.
  • How can the user customize the module’s behavior? (parameters, etc)
  • Where can the user turn for help?
  • Would you like contributors? What’s the process?

At the very least, these questions should be addressed in the module’s README file but it’s also a good idea to place some appropriate documentation as comments above each class definition. Puppet ships with a subcommand called doc which will help you generate those commands into plain text (perhaps providing your README’s content) or can even build a html-filled folder of rdoc goodness.

Take a look at Carl Caum’s Bacula module for example. He clearly spells out what the module claims to do, what its requirements are and how to use it. If you take a look at this modules manifests, each has fairly robust comments in-line explaining usage and implementation decisions. Between the two, both users and contributors have most of the information they’ll need.

Testing

It may sound obvious that you should test your module, but you really should. To start, this means making sure your module does what you claimed it can do in your documentation. This is commonly and pretty simply solved with smoke tests. As an overview, we have a great docs article on writing, running, and reading smoke tests.

On a basic level, this means you test your module’s functionality against the operating systems you claim to support and include the puppet code you used to conduct your tests (usually class declarations) in the tests folder of your module. This convention allows consumers of your module to easily rerun the smoke tests you ran but also serve as great ways for someone to quickly try out its functionality. If a contributor to your module adds new functionality, make sure they include a smoke test for you to use.

Extra Credit: Smoke tests are great, but some members of the Puppet community are building automated tests for their modules. The big two I’m aware of are rspec-puppet and cucumber-puppet. These are advanced techniques that come with a steep learning curve but if you’re curious, there’s a recent blog post on test driven development. Once achieved, these tests make it easy to protect against regressions when merging new code. If you’re investing heavily into Puppet, it may be worth your time to learn.

If you’re particularly interested in rspec-puppet, the project’s author has an awesome tutorial on rspec-puppet.com.

Interfaces

When I download your module, I shouldn’t have to modify your code to do something awesome with it. It’s really tempting to code just for your environment and call it a day but a little extra effort will get you a module others can use and contribute to!

When you come across something that may need tuning depending on the user’s situation, build a way for that user to modify your module’s behavior without touching your manifests directory.

Let’s say for example you have a motd module with a motd class for setting the contents of /etc/motd. You wrote the class to build /etc/motd from a template that includes your company’s name in it. If I want to reuse your module, I’ll have to modify your template or your manifest. This presents a challenge to reusability. Instead, a little extra Puppet code can provide me with a way to modify your module’s behavior.

For example, provide two parameters (source & template) in a parameterized class. With a little bit of extra conditional logic and variables, users can specify their own template in the class declaration like this:

class { 'motd':
  template => template('motd/my_template.erb'),
}

All they’ll need to do is drop their template into the templates folder or they could even specify a template from an entirely different location. As they’re writing the template function, they can even concatenate multiple templates together!

Let’s look an example of how this can be done.

# Static class with hard-coded template
class motd {
 
  file { '/etc/motd':
    ensure  => file,
    mode    => '0644',
    owner   => 'root',
    group   => 'root',
    content => template('motd/my_template.erb'),
  } 
 
}
## Example declaration with the built-in template
include motd
 
## Or specify your own template
class { 'motd':
  template => template('mymodule/mytemplate.erb'),
}
 
## Even specify your own static file
class { 'motd':
  source => 'puppet:///modules/mymodule/myfile',
}
 
# Class with an interface for serving files vs templates
class motd (
  $template = '',
  $source = '',
) {
 
  # Fail gracefully if user tries to supply both source and template
  if $template and $source {
    fail('You cannot supply both template and source to the motd class')
  }
 
  # If nothing is specified, default to our template
  # If something was specfied, set the motd_* variable
  if $template == '' and $source == '' {
    $motd_template = template('motd/default_template.erb')
  } elsif $source != '' {
    $motd_source = $source
  } elsif $template != '' {
    $motd_template = $template
  } 
 
  file { '/etc/motd':
    ensure  => file,
    mode    => '0644',
    owner   => 'root',
    group   => 'root',
    content => $motd_template,
    source  => $motd_source,
  } 
 
}

This is a fairly simple pattern but hopefully it makes sense. Again, we’re just leveraging the tools in Puppet’s language to make it easy for users of our module to use it in their environment without creating code they have to maintain themselves. Please invest some time thinking about this problem when you write your modules.

Versioning

A version number is required when you submit a module to the Forge. Version numbers can be tricky. In a place like the Puppet Forge, the versioning scheme you came up with for your modules may rock and make perfect sense to you and your team but may be completely different than the system I came up with for mine. How are you, as a module consumer, supposed to translate between the two and know at a glance whether functionality will fundamentally change between 1.0.1 and 1.1.0?

My recommendation is to follow the Semantic Versioning specification when you’re publishing that next release. In the author’s words: “Under this scheme, version numbers and the way they change convey meaning about the underlying code and what has been modified from one version to the next.” We strive to follow this approach at Puppet Labs for modules we write and my hope is that each Forge user will as well.

Style

In the same way consistent versioning is important; consistent, clean and usable code is important. When you hear someone ask, “Does this module adhere to The Style Guide?”, they are asking about the latter. We host a document on our website that we call the Style Guide. It’s a collection of stylistic preferences and code structure best practices that was originally written to help our organization develop code that we could all easily read and develop in a uniform manner. It’s even more appropriate for the large, distributed audience consuming and contributing modules to the Puppet Forge.

So read and adhere to the Style Guide. Do it for your teammates, do it for the Forge. To make that task easier, Tim Sharpe (@rodjek) maintains an awesome tool called puppet-lint. Like a dryer dutifully collects dead sock fibers, puppet-lint will scan your manifest code and alert you to certain violations against the Style Guide.

gem install puppet-lint
[root@centos6 modules]# cat helloworld/manifests/init.pp 
class helloworld {
  notify { 'example':
    message => "Hello World!",
  }
}
[root@centos6 modules]# puppet-lint helloworld/manifests/init.pp 
WARNING: double quoted string containing no variables on line 3

Puppet-lint doesn’t replace the Style Guide, so I encourage you to read it and keep it handy. You will find that it covers more than just the code styling that puppet-lint checks for. These are great tools to keep your code consistent, clean and consumable.

Publishing

Once you’re finished writing your module, you’ll want to publish it to the Puppet Forge. I’ll cover that, more on these techniques and more advanced concepts next time. I’ll also discuss updates to the Puppet Module tool which also now ships with Puppet Enterprise. If you’d like to whet your appetite, check out this quick Puppet Module Tool screencast created by one of the developers responsible for the tool updates.

Now What?

The Puppet Forge now has over three hundred modules helping sysadmins solve problems all over the map. You’ve got our Module of the Week blog series continuing to highlight great work and with Puppet Enterprise 2.5, you’ve got a greatly improved command-line Forge experience. Let’s carry this momentum on to over six hundred awesome modules by PuppetConf 2012, and see you next time!

Additional Resources

Module of the Week: puppetlabs/apt – Pull APT Strings with Puppet

Posted on
By
Ryan Coleman
in
Blog, Community, Extending Puppet, How to, Module of the Week, Modules, Open Source, Tips
Responses
2 Comments »
Purpose Provides helpful definitions for dealing with APT
Module puppetlabs/apt
Puppet Version 2.7+
Platforms Debian, Ubuntu

I know we’ve been covering Puppet Labs modules for the last few weeks, but I’d like to specifically call out all the great community contributions we’ve had on this module. In fact, this module started out as a community members project, was forked by someone at Puppet Labs, modified and later enjoyed significant contributions from other community members.

Thank you (in no particular order) for your contributions!

So, what does the module do? Its description on the Puppet Forge is modest, but accurate: “Provides helpful definitions for dealing with APT.” Did you know that Puppet Labs has an APT repository? I’ll show you how to use this module to configure it on your Debian/Ubuntu system and mention some other features as well.

Let’s dig in.

Installing the module

Complexity Easy
Installation Time 2 minutes

This should look familiar to MOTW devotees. I used Puppet 2.7.12 & Ubuntu 11.10 while writing this blog post.

Installing the puppetlabs/apt module is as simple as using the puppet module tool (available as a rubygem, on github; and coming soon in Puppet and Puppet Enterprise):

$ puppet config print modulepath
/etc/puppet/modules:/usr/share/puppet/modules
 
$ cd /etc/puppet/modules
$ gem install puppet-module
$ puppet-module install puppetlabs/apt
You should now have an apt directory in your module path.
/etc/puppet/modules/apt

Resource Overview

This module can do a number of things for you, mostly through defined resource types. I’ll briefly describe the functionality modeled in each class/define in the modules manifests folder.

manifests/
├── builddep.pp # Define: Interface to apt-get build-dep.
├── force.pp      # Define: Forcibly install a package from a specific release.
├── init.pp          # Class: Sets options/logic, manages sources.list & sources.list.d.
├── key.pp        # Define: Manage APT gpg keys.
├── params.pp  # Class: Sets up a few parameters.
├── pin.pp         # Define: Pin a release in APT.
├── ppa.pp        # Define: Manage ppa repositories. 
├── release.pp  # Class: Set the default APT release. 
└── source.pp   # Define: Manage APT sources.

This blog post will cover installing an APT source and key.

Testing the module

This module is mostly a collection of defined resource types, which provide reusable logic that consumers of the module can leverage for managing APT.

It provides smoke tests for testing its functionality on a target system and spec tests for checking a compiled catalog against an expected set of resources as you’re developing on the module. Both sets of tests are pretty comprehensive for this module but if you find something missing or broken, feel free to open an issue against it.

If you’d like to quickly try out some of the modules functionality, take a look at and apply the smoke tests located in the modules tests folder.

Configuring the module

Complexity Easy
Installation Time 5 minutes

There isn’t much to configure in this module as there aren’t many classes. Some parameters are declared in params.pp while others are configured through the apt class itself. That said, the default set of options should be perfectly appropriate for your use. The modules documentation covers the available parameters pretty well, though it could definitely be improved.

Let’s look at a couple of important options in the apt class, both set to false by default:

$purge_sources_list 
$purge_sources_list_d

When you declare the apt class, Puppet will manage your systems sources.list file and sources.list.d directory but will do its best to respect existing content. If you declare your apt class with the above parameters set to true, Puppet will unapologetically purge any existing content it finds that you haven’t declared with Puppet. So by default, Puppet assumes you’ll be managing APT with Puppet along with system defaults and/or locally managed content but you have the option to authoritatively manage your APT sources.

With this in mind, let’s move on to actually managing an APT source.

Example usage

We’re going to use this module to setup the Puppet Labs apt repository where we can get the latest stable releases of Puppet, right from the horse’s mouth.

Start by creating a new smoke test in your newly installed apt modules test folder, called puppetlabs-apt.pp. Inside we’re going to declare a single resource representing our desired APT source and key.

$ cd module_path_from_above/apt
$ vim tests/puppetlabs-apt.pp
# The Puppet Labs APT source and gpg key.
apt::source { 'puppetlabs':
  location   => 'http://apt.puppetlabs.com',
  repos      => 'main',
  key        => '4BD6EC30',
  key_server => 'pgp.mit.edu',
}

With this resource, we’re creating an APT source named puppetlabs and gave Puppet information about the repositories location and the key used to sign its packages. Puppet will leverage Facter to determine the appropriate release (like oneiric) but you can set it directly by adding the release attribute.
Check your new smoke test for syntax errors using the Puppet Parser subcommand.

$ puppet parser validate tests/puppetlabs-apt.pp

Receive no output from that command? Great, nothing is wrong! Let’s apply this code now, as an example usage of the apt module.

$ puppet apply --verbose tests/puppetlabs-apt.pp
notice: /Stage[main]//Apt::Source[puppetlabs]/File[puppetlabs.list]/ensure: defined content as '{md5}3be1da4923fb910f1102a233b77e982e'
info: /Stage[main]//Apt::Source[puppetlabs]/File[puppetlabs.list]: Scheduling refresh of Exec[puppetlabs apt update]
notice: /Stage[main]//Apt::Source[puppetlabs]/Exec[puppetlabs apt update]: Triggered 'refresh' from 1 events>

Great, you’re ready to update Puppet directly from Puppet Labs!

We used a new smoke test to easily lay down our resource declaration and apply it on our system. Realistically, you may want to declare your APT sources inside the classes where they’re needed, say in a class that installs packages requiring a certain APT repository. Remember that dependency meta-parameters like require and before work just fine on the defined resource types provided by the APT module.

Conclusion

Hopefully I’ve shown that this module provides extremely simple interfaces for managing an APT source and key and lets you leverage Puppet to do all the heavy lifting, a thousand times over if you want. That’s the power of this module and that’s the power of Puppet.

Before I leave you, I have a confession to make. My systems administration experience is almost exclusively RedHat based and until recently, my knowledge of APT went about as far as apt-get install foo. Traditionally, I understand how something works and go about implementing that thing in Puppet but in reviewing community pull requests against the module, I stumbled onto something awesome — I learned how APT worked by reading the Puppet code in the puppet-apt module! It had never occurred to me before that we as sysadmins can leverage the shared language Puppet provides us to discover and understand how systems work in an entirely different way.

If you follow the Forge link at the top of this post, you’ll see that this module does much more than what I’ve shown you here and it’s all made possible by the contributors credited at the top of the post. I can’t thank our community enough for its contributions to Puppet and over 300(!) Forge modules. You make system administration better each and every day. Thank you!

Additional Resources

Module of the Week: puppetlabs/stdlib – Puppet Labs Standard Library Part 4

Posted on
By
Kelsey Hightower
in
Blog, Community, DevOps, Extending Puppet, How to, Module of the Week, Modules, Open Source, Systems Management, Tips
Responses
2 Comments »
Purpose Standard library for creating Puppet modules
Module puppetlabs/stdlib
Puppet Version 2.6+
Platforms Redhat, Debian, Solaris, Mac OS X, Windows

Welcome back to the module of the week! Previously we covered facter-dot-d. This week we’re switching our focus to the puppetlabs/stdlib data functions: getvar, loadyaml, and merge.

The data functions provide the ability to dynamically lookup data from Puppet namespaces, load data from external YAML files, and merge Hashes. These powerful features make it easy to separate Puppet code from data without the complexity of using an External Node Classifier or Custom Facts; leaving your manifests more readable and maintainable as a result.

Example Usage

In this brief tutorial I’m going to cover the loadyaml and merge functions to show you just how easy it is to work with external data:

The loadyaml function

As its name implies the loadyaml function allows us to load data from an external YAML file.

/etc/puppet/data/webservers/data.yaml

ports:
  http: 80
  https: 443
  tomcat: 8080
 
webdir: /var/www/html

/etc/puppet/modules/webservers/manifests/init.pp

class webservers {
  include 'stdlib'
 
  $data = loadyaml('/etc/puppet/data/webservers/data.yaml')
 
  $http_port  = $data[ports][http]   # => 80
  $https_port = $data[ports][https]  # => 443
  $webdir     = $data[webdir]        # => '/var/www/html'
 
  ...
}

Lets step through this:

  • First we create a data file /etc/puppet/data/webservers/data.yaml
  • Then we make a call to loadyaml with the full path to our data file
  • Next loadyaml processes data.yaml and returns a Hash, which is then assigned to $data
  • Finally we set the $http, $https, and $webdir variables by referencing the $data Hash

The merge function

The merge function allows us to merge two or more Hashes:

/etc/puppet/data/webservers/data.yaml

ports:
  http: 80
  https: 443
  tomcat: 8080
 
webdir: /var/www/html

/etc/puppet/data/database/data.yaml

ports:
  db: 5432

/etc/puppet/modules/webservers/manifests/init.pp

class webservers {
  include 'stdlib'
 
  $myhash         = { database_user => 'webdata', webdir => '/var/html' }
  $webserver_data = loadyaml('/etc/puppet/data/webservers/data.yaml')
  $database_data  = loadyaml('/etc/puppet/data/database/data.yaml')
 
  $data = merge($myhash, $webserver_data, $database_data)
 
  $http_port  = $data[ports][http]   # => 80
  $https_port = $data[ports][https]  # => 443
  $webdir     = $data[webdir]        # => '/var/www/html'
  $db_port    = $data[ports][db]     # => 5432
  ...
}

Here we are calling the merge function with three Hashes ($myhash, $webserver_data, and $database_data) which yields a single $data Hash.

Note, the merge function overrides duplicate keys with the argument farthest to the right “winning”. In the above example the webdir value provided by $myhash is overridden by the value provided by $webserver_data.

Conclusion

The data functions provided by puppetlabs/stdlib makes it easy to work with internal and external data. While great for simple use-cases, this mode of operation doesn’t scale. Once the number of external data files or Puppet modules grows, you’ll find your modules littered with lots of getvar, loadyaml, and merge function calls.

A better solution for more complex requirements is Hiera. While providing all the features of getvar, loadyaml, and merge, Hiera includes the ability to load data from virtually anywhere. Via Hiera’s pluggable backend interface, you’ll be able to pull data from external databases such as Redis, or JSON files; the possibilities are endless.

That wraps up our coverage of the puppetlabs/stdlib module. Hopefully you can now leverage some of its features in your own modules and have a sense of how robust Puppet modules can be.

Additional Resources

Module of the Week: puppetlabs/stdlib – Puppetlabs standard library – Part 3

Posted on
By
Kelsey Hightower
in
Blog, Community, DevOps, Extending Puppet, Module of the Week, Modules, Open Source, Systems Management, Tips
Responses
4 Comments »
Purpose Standard library for creating Puppet modules
Module puppetlabs/stdlib
Puppet Version 2.6+
Platforms Redhat, Debian, Solaris, Mac OS X, Windows

Welcome back to the Puppet Module of the Week! Previously we covered the validation functions provided by puppetlabs/stdlib. This week we’re switching our focus to facter-dot-d; a simple way of pulling facts from external sources.

From the docs:

A simple little framework to get facts from external sources.
You can create files in /etc/facter/facts.d which can be text, yaml, json or executables. In the case of executables it supports caching so you only need to run your expensive scripts every now and then.

facter-dot-d is great for people who already have information about their nodes and would like to reuse that data without writing custom facts. facter-dot-d makes this possible by scanning the /etc/facter/facts.d directory for data files(YAML, JSON, or plain text) and scripts and then processes them to create new facts.

This may seem like magic, but facter-dot-d is really just a custom fact. By using the Facter API, facter-dot-d converts external data into custom facts instead of returning a single fact.

Neat huh? We thought so too, so it’ll be built into the next major release of Facter 1.7.0.

Example usage

Before we get started with the examples we need to create the /etc/facter/facts.d directory:

$ mkdir /etc/facter/facts.d

Next we need to ensure the node uses the stdlib class:

node 'pmotw.puppetlabs.com' {
  include stdlib
}

Then we use pluginsync to sync the facter-dot-d fact to the agent:

$ puppet agent -t --pluginsync
…
info: Loading downloaded plugin /var/lib/puppet/lib/facter/facter_dot_d.rb
…
info: Loading facts in facter_dot_d
info: Loading facts in root_home
info: Loading facts in facter_dot_d
info: Loading facts in root_home
info: Caching catalog for pmotw.puppetlabs.com
info: Applying configuration version '1331932291'
notice: Finished catalog run in 0.07 seconds

Adding a custom fact using JSON

In order to use JSON files as a source for facter-dot-d you need to ensure the json gem is installed on your system. On my test system, Debian 6, the json gem is not installed by default:

$ gem list
 
*** LOCAL GEMS ***
 
puppet-lint (0.1.12)
puppet-module (0.3.4)

We can use the gem command to install it:

$ gem install json
Building native extensions.  This could take a while...
Successfully installed json-1.6.5
1 gem installed
Installing ri documentation for json-1.6.5...
Installing RDoc documentation for json-1.6.5...
 
$ gem list
 
*** LOCAL GEMS ***
 
json (1.6.5)
puppet-lint (0.1.12)
puppet-module (0.3.4)

Now that we have the json gem installed we can simply drop our JSON file into the /etc/facter/facts.d directory:

$ cat /etc/facter/facts.d/datacenter.json
{
  "location": "Portland",
  "cluster": "Web"
}

On the next use of Facter, the datacenter.json file will be processed by facter-dot-d adding custom Facts for the location and cluster keys as a result. It should also be noted that since facter-dot-d was distributed via pluginsync, we must use the --puppet flag. The following will not work:

$ facter location

Using the --puppet flag produces the desired results:

$ facter --puppet location cluster
cluster => web
location => portland

We can achieve the same results using a YAML or TXT file:

$ cat /etc/facter/facts.d/datacenter.yaml
---
location: portland
cluster: web
 
$ facter -p location cluster
cluster => web
location => portland
$ cat /etc/facter/facts.d/datacenter.txt
location=portland
cluster=web
 
$ facter -p location cluster
cluster => web
location => portland

Staying out of trouble

Be sure to use either JSON, YAML, or TXT files consistently. While the following will work:

$ cat /etc/facter/facts.d/datacenter.yaml
---
location: portland
cluster: web
 
$ cat /etc/facter/facts.d/web.json
{
  "api_version": 2.1,
  "web_port": 8080
}
 
$ facter -p location cluster api_version web_port
api_version => 2.1
cluster => web
location => portland
web_port => 8080

The follow will lead to unexpected behavior:

$ cat /etc/facter/facts.d/datacenter.json
{
  "location": "atlanta",
  "cluster": "database"
}
 
$ cat /etc/facter/facts.d/datacenter.yaml
---
location: portland
cluster: web

With both datacenter.json and datacenter.yaml specifying the same data, there is no reliable way to tell which values cluster and location will be set to.

$ facter -p location cluster
cluster => database
location => atlanta

Silent failures

Invalid JSON, YAML, and TXT files will silently fail:

$ cat /etc/facter/facts.d/web.json
{
  "api_version": 2.1,
  "web_port": 8080,
}
 
$ facter -p web_port

Notice we do not get the expected output of 8080 here. We can get more insight into what’s going on by using the --debug flag:

$ facter -p --debug
...
Parsing /etc/facter/facts.d/datacenter.json using json_parser
Failed to handle /etc/facter/facts.d/datacenter.json as json facts: JSON::ParserError: 618: unexpected token at '{
 
  "api_version": 2.1,
 
  "web_port": 8080,
 
}
 
'

Though not clear from this error message, datacenter.json is not a valid JSON file. Notice the extra ‘,’ after 8080.

Fixing this yields the expected result:

$ facter -p web_port
web_port => 8080

Conclusion

The Facter Ruby API is the standard way of creating custom facts, which sets the bar pretty high for creating new facts. facter-dot-d lowers the bar significantly by adding the ability to use flat files as well as scripts in any language to produce custom facts. This is a huge win for people just getting started and for people who have existing data they would like to use as Facts.

Join us next week as we wrap up our coverage of the puppetlabs/stdlib module. We’ll be taking a look at the remaining functionality by exploring some data functions that allow you to do things like load external data from YAML files and merge parameter values inside of Puppet manifests.

Additional Resources

Module of the Week: puppetlabs/stdlib – Puppet Labs Standard Library Part 2

Posted on
By
Kelsey Hightower
in
Blog, Community, DevOps, Extending Puppet, How to, Module of the Week, Modules, Open Source, Systems Management, Tips
Responses
1 Comment »
Purpose Standard library for creating Puppet modules
Module puppetlabs/stdlib
Puppet Version 2.6+
Platforms Redhat, Debian, Solaris, Mac OS X, Windows

Welcome back to the module of the week! Last week we began our coverage of the puppetlabs/stdlib module by examining the file_line resource, which allows you to manage individual lines in a file.

We’ll pick up where we left off by covering the data validation functions provided by the puppetlabs/stdlib module and how to use them to make our Puppet manifests more resilient to runtime failures.

There are five validation functions in total. The following four functions: validate_array, validate_bool, validate_hash and validate_string allow you to validate the type of data used in your manifests. These are useful when you want to be sure the input value is an Array, Hash, Boolean, or a String.

Sometimes you need to go beyond type validation and ensure something more specific like an ipaddress or hostname. The validate_re function helps here by validating input using regular expressions. With access to the full power of Ruby’s regular expressions, you can validate just about anything.

Resource Overview – validation functions


Function validate_array
validate_bool
validate_hash
validate_re
validate_string

Example usage

In the following example we use the validate_array and validate_bool functions to validate the input types used by the ntp module.

$ cat /etc/puppet/modules/ntp/manifests/init.pp
 
# Example Usage
# 
# class { "ntp":
#   servers    => ['0.debian.pool.ntp.org'],
#   autoupdate => false,
# }
#
class ntp ($servers="UNSET",
           $ensure="running",
           $autoupdate=false,
) {
  validate_array($servers)
  validate_bool($autoupdate)
}

In the above example we are using the validate_array function to ensure the $severs class parameter is an array: ['0.debian.pool.ntp.org'], and not a String: '0.debian.pool.ntp.org'.

We’re also using the validate_bool function to ensure $autoupdate is a Boolean value: true or false, and not: 'true', 'false', or 'string'.

Lets see what happens if I use a String value for the $server class parameter:

$ cat /etc/puppet/manifests/site.pp
 
node 'pmotw.puppetlabs.com' {
  class { "ntp":
    servers => '0.debian.pool.ntp.org'
  }
}
 
$ puppet agent -t
…
err: Could not retrieve catalog from remote server: Error 400 on SERVER: "0.debian.pool.ntp.org" is not an Array.  It looks to be a String at /etc/puppet/modules/ntp/manifests/init.pp:5 on node pmotw.puppetlabs.com
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

Yikes! Looks like we’ve managed to piss off our Puppetmaster. It’s basically yelling at us for using a String and not an Array. It’s Puppet’s way of saying, “You’re doing it wrong!”, and indeed we are. The fix is pretty simple:

node 'pmotw.puppetlabs.com' {
  class { "ntp":
    servers => ['0.debian.pool.ntp.org']
  }
}

When the concern is not the type, but the actual value, then the validate_re function is the right tool for the job. From the docs:

Perform simple validation of a string against one or more regular expressions. The first argument of this function should be a string to test, and the second argument should be a stringified regular expression (without the // delimiters) or an array of regular expressions. If none of the regular expressions match the string passed in, compilation will abort with a parse error.

A “real” world example

You just got a job at the Federation, an organization committed to space exploration. You’re a rising star! You’ve been put in charge of the Intergalactic Compute Bay (ICB), aka the datacenter. Your job is to take things to the next level, so you start by creating a Puppet module to manage space probes. Yeah, space probes.

Your manifest looks like this:

$ cat /etc/puppet/modules/voyager/manifests/init.pp
 
# Class: voyager
#
#   This module manages billion dollar space probes, use with caution.
#   Please do not send probes to non-planets, avoid things like black holes!
#
#   Tested probes:
#    - Voyager 1
#    - Voyager 2
#    - Voyager 3
#
# Parameters:
#  
#   $destination
#
# Actions:
#  
#  Configures and manages space probes.
#
# Example Usage:
# 
# class { "voyager":
#    destination => 'Mars'
# }
#
class voyager ($destination) {
  define space_mission ($destination) {
    # Top secret stuff we cannot show
    $message = "Sending probe to ${destination}"
    notify { $message: }
  }
 
  space_mission { 'voyager3':
    destination => $destination
  }
}

Wow, that’s awesome! You have a module that makes it easy to configure a space probe to go anywhere. Your well documented module even warns people that probes should not be sent into Black Holes. After a few rounds of QA, the voyager module is ready for production.

Egor Homakov, the new security lead, has reviewed your module and pointed out some possible bugs. You ignore him—your module is badass and passed QA. Plus, you give clear instructions on how to use it, right there in the module.

It’s show time. It’s time to launch 1000 new Voyager 3 space probes, and instead of using the old manual system of strategic button pushing, the Voyager Puppet module has been called to action.

Egor has volunteered to help out, his part of the configuration looks like this:

# lots of node definitions omitted
node 'voyager3.federation.org' {
  class { "voyager":
    destination => 'Sun'
  }
}
…

With little time to review each entry you spot check things and give the green light.

During the launch you notice this scroll by the screen:

notice: Sending probe to Sun
notice: /Stage[main]/Voyager/Voyager::Space_mission[voyager3]/Notify[Sending probe to Sun]/message: defined 'message' as 'Sending probe to Sun'
notice: Finished catalog run in 0.03 seconds

Yes my friend, you are indeed sending one of those billion dollar probes to the Sun. In case it’s not entirely obvious, the Sun is not a planet. Your attempt to play it off is short-lived; you just witnessed a space probe burst into flames!

You made two mistakes here. First you trusted Egor with your Puppet configurations, and second you did not validate the input for the $destination class parameter. At this point your only option is to resign. Yep, not even Puppet can help you recover from this.

Even though it’s to late for you, lets see how you could have saved your job.

class voyager ($destination) {
  $approved_planets = [
    'Mercury',
    'Venus',
    'Earth',
    'Mars',
    'Jupiter',
    'Saturn',
    'Uranus',
    'Neptune'
  ]
 
  validate_re($destination, $approved_planets)
 
  define space_mission ($destination) {
    # Top secret stuff we cannot show
    $message = "Sending probe to ${destination}"
    notify { $message: }
  }
 
  space_mission { 'voyager3':
    destination => $destination
  }
}

This time we catch the error before destroying a billion-dollar space probe:

$ puppet agent -t
…
Error 400 on SERVER: validate_re(): "Sun" does not match ["Mercury", "Venus", "Earth", "Mars", "Jupiter", "Saturn", "Uranus", "Neptune"] at /etc/puppet/modules/voyager/manifests/init.pp:37 on node voyager3.federation.org

More validate_re examples

The regular expressions can be more complex than we have shown so far. For example, if you wanted to validate a mac address you could do the following:

validate_re($macaddress, '^([0-9a-f]{2}[:]){5}([0-9a-f]{2})$')

You can also use the validate_re function in combination with other validation functions such as the validate_hash function like this:

$network_device = { macaddress => '00:0c:29:0d:1d:970' }
validate_hash($network_device)
validate_re($network_device['macaddress'], '^([0-9a-f]{2}[:]){5}([0-9a-f]{2})$')

The above validates that the macaddress key of the $network_device hash contains a valid mac address string value.

Reading and writing regular expressions can be a challenge, one tool that I use to help with the process is Rubular, an online Ruby regular expression editor:

Rubular allows me to play around with an expression until I get it right.

Conclusion

Using the validation functions provided by puppetlabs/stdlib can help catch issues quickly and prevent disaster from striking. Try not to get carried away by validating everything under the Sun; start with things that are critical and be sure to document what valid values should look like.

Join us next week as we continue our coverage of the puppetlabs/stdlib module with a quick look at facts-dot-d and discover how we can get Facts from external sources and cache them in a central location so Facter can find them.

Additional Resources:

Module of the Week: puppetlabs/stdlib – Puppet Labs standard library

Posted on
By
Kelsey Hightower
in
Blog, Company, Module of the Week, Modules, Open Source, Tips
Responses
9 Comments »
Purpose Standard library for creating Puppet modules
Module puppetlabs/stdlib
Puppet Version 2.6+
Platforms Redhat, Debian, Solaris, Mac OS X, Windows

This week we’re going to cover the puppetlabs/stdlib module. This module is packed with lots of Puppet goodness, including custom Puppet functions for validating manifest data, a resource type for managing individual lines in files, extensions to facter, and more. Clearly, this is one bad-ass Puppet module!

Since the puppetlabs/stdlib module has so much utility, it’s required by many of the modules built by the crew here at Puppet Labs. My goal this week is to show you how to harness these features in your own modules. To ensure we cover this module properly we’re going to break the coverage into four posts.

We’ll kick things off with the file_line resource which allows you to manage individual lines in a file. This is really great when you’re not in position to manage the entire file. You can use file_line as many times as you like, but be sure to give each resource a unique name. Don’t get carried away—this is not a replacement for managing an entire file with the proper file resource and the optional template.

In Part 2 we’ll cover the validation functions provided by the puppetlabs/stdlib module and how to validate input in your manifests.

In Part 3 we’ll cover the use of facts-dot-d and discover how we can get Facts from external sources and cache them in a central location so Facter can find them.

We’ll round out the mini-series in Part 4 with a look at the remaining functionality by exploring some data functions that allow you to do things like load external data from YAML files and merge parameter values.

Resource Overview – file_line

Resources file_line

Installing the module

Complexity Easy
Installation Time 2 minutes

Installing the puppetlabs/stdlib module is as simple as using the puppet module tool (available as a rubygem on github; and coming soon in Puppet and Puppet Enterprise):

$ puppet config print modulepath
/etc/puppet/modules:/usr/share/puppet/modules
 
$ cd /etc/puppet/modules
$ puppet-module install puppetlabs/stdlib

You should now have a stdlib directory in your module path.

/etc/puppet/modules/stdlib

Testing the module

The puppetlabs/stdlib module comes with tests—let’s run them.

$ puppet apply tests/init.pp --noop
notice: Finished catalog run in 0.05 seconds
 
$ puppet apply tests/file_line.pp  --noop
notice: /Stage[main]//File[/tmp/dansfile]/ensure: current_value absent, should be present (noop)
err: /Stage[main]//File_line[dans_line]: Could not evaluate: No such file or directory - /tmp/dansfile
notice: Class[Main]: Would have triggered 'refresh' from 1 events
notice: Stage[main]: Would have triggered 'refresh' from 1 events
notice: Finished catalog run in 0.02 seconds

Hmm, that’s not telling me much. I need to run this test in normal mode to get the full effect. While running tests without --noop is normally a dangerous thing to do, this test only needs to create a file named dansfile under the /tmp directory—a change that won’t affect the state of my system. I’ll confirm this by looking at the code:

cat tests/file_line.pp
# This is a simple smoke test
# of the file_line resource type.
file { '/tmp/dansfile':
  ensure => present
}->
file_line { 'dans_line':
  line => 'dan is awesome',
  path => '/tmp/dansfile',
}

Looks good to me. Lets rerun the test, this time without the --noop flag:

$ puppet apply tests/file_line.pp
notice: /Stage[main]//File[/tmp/dansfile]/ensure: created
notice: /Stage[main]//File_line[dans_line]/ensure: created
notice: Finished catalog run in 0.02 seconds
 
$ cat /tmp/dansfile
dan is awesome

The test works as expected, and I can confirm that Dan is in fact awesome.

Complexity Easy
Installation Time 0 Minutes

Since file_line is a resource type, there is really nothing to configure here. Once the stdlib module is available in your modulepath, file_line will be automatically distributed and usable as a resource on all of your nodes.

Example usage

When prepping for blog posts, I normally start with a fresh VM and, without fail, forget to add a host entry for my test Puppetmaster to /etc/hosts. This leads to my Puppet agent complaining that it cannot locate its master, reminding me of the obvious mistake. It’s an easy fix. I just add the following line to /etc/hosts:

172.16.240.200  master.dev.puppetlabs.com master

Problem solved! Until next week, that is.

Some of you may be thinking, “Hard code the line above and take a snapshot of the VM.” I will then pretend that I did not just hear you say that. I’ll then remind you that this is a blog about Puppet and there’s obviously a better way to solve this problem. Besides, that approach is less than optimal. I would have to repeat that process for each of my VMs and remember to keep them all updated if things were to change.

Lets put the file_line resource to use.

The current state of my /etc/hosts file looks like this:

cat /etc/hosts
127.0.0.1	localhost
127.0.1.1	pmotw.puppetlabs.com	pmotw
 
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Notice the missing entry for master.dev.puppetlabs.com?

We’ll skip creating a whole manifests and declare our file_line resource on the node:

$ cat /etc/puppet/manifests/site.pp
node 'pmotw.puppetlabs.com' {
  file_line { 'puppet master host entry':
    ensure => present,
    line   => '172.16.240.200  master.dev.puppetlabs.com	 master',
    path   => '/etc/hosts',
  }
}

Apply the changes.

$ puppet agent -t
...
notice: /Stage[main]//Node[pmotw.puppetlabs.com]/File_line[puppet master host entry]/ensure: created
notice: Finished catalog run in 0.02 seconds
 
$ cat /etc/hosts
127.0.0.1	localhost
127.0.1.1	pmotw.puppetlabs.com	pmotw
 
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.16.240.200 master.dev.puppetlabs.com master

It works, but what happens if I remove that line and rerun Puppet?

$ vim /etc/hosts
remove 172.16.240.200  master.dev.puppetlabs.com    master
 
$ puppet agent -t
...
notice: /Stage[main]//Node[pmotw.puppetlabs.com]/File_line[puppet master host entry]/ensure: created
notice: Finished catalog run in 0.02 seconds

Puppet does the right thing, and puts it back. The other nice thing about file_line, it continues to work even if I manually add new lines above or below the one being managed:

$ cat /etc/hosts
..
172.16.240.200  master.dev.puppetlabs.com	     master
172.16.240.100  someotherhost.puppetlabs.com  someotherhost
 
$ puppet agent -t
notice: Finished catalog run in 0.02 seconds
 
Notice: no changes are being reported.

Conclusion

We’ve only scratched the surface of the puppetlabs/stdlib module. Be sure to join us next week as we cover the validation functions provided by puppetlabs/stdlib which can help make your manifests more resilient to run-time failures by utilizing the ability to validate input; nothing like catching errors before hosing your systems!

Additional Resources:

Module of the Week: saz/sudo – Manage sudo configuration

Posted on
By
Kelsey Hightower
in
Blog, Community, Extending Puppet, How to, Module of the Week, Modules, Open Source, Puppet Enterprise, Tips
Responses
0 Comments
Purpose Manage sudo configuration
Module saz/sudo
Puppet Version 2.6+
Platforms Debian, Ubuntu

This week we’ll be reviewing a sudo module contributed by Steffen Zieger (saz), a contributor with over 20 well maintained modules on the Forge. The saz/sudo module provides support for managing sudo configuration using Puppet.

Resource Overview

The saz/sudo module manages the following resources:

Files /etc/sudoers
Directories /etc/sudoers.d
Packages sudo
Defined types sudo::conf

Installing the module

Complexity Easy
Installation Time 2 minutes

Installing the saz/sudo module is as simple as using the puppet module tool (available as a rubygem on github; and coming soon in Puppet and Puppet Enterprise):

$ puppet config print modulepath
/etc/puppet/modules:/usr/share/puppet/modules
 
$ cd /etc/puppet/modules
$ puppet-module install saz/sudo

You should now have a sudo directory in your module path.

/etc/puppet/modules/sudo

Testing the module

The saz/sudo module comes with tests—let’s run them.

$ puppet apply tests/init.pp --noop
notice: /Stage[main]/Sudo/Package[sudo]/ensure: current_value absent, should be present (noop)
notice: /Stage[main]/Sudo/File[/etc/sudoers.d/]/ensure: current_value absent, should be directory (noop)
notice: /Stage[main]/Sudo/File[/etc/sudoers]/ensure: current_value absent, should be file (noop)
notice: Class[Sudo]: Would have triggered 'refresh' from 3 events
notice: Stage[main]: Would have triggered 'refresh' from 1 events
notice: Finished catalog run in 0.06 seconds

Notice the use of the --noop flag here. Puppet’s noop, or simulation mode, allows us to see the changes that would occur without actually applying them. Without the tag we would have made actual changes to our system—in this case, our Puppet Master!

Configuring the module

Complexity Easy
Installation Time 5 minutes

This module exposes seven (7) configuration parameters using parameterized classes with defaults defined in the sudo::params class:

class sudo(
  $ensure = 'present',
  $autoupgrade = false,
  $package = $sudo::params::package,
  $config_file = $sudo::params::config_file,
  $config_file_replace = true,
  $config_dir = $sudo::params::config_dir,
  $source = $sudo::params::source
) inherits sudo::params {
…

This a common pattern used to create reusable modules. Let’s take a look at the manifest where the sudo::params class is defined:

# cat /etc/puppet/modules/sudo/manifests/params.pp 
class sudo::params {
  case $::operatingsystem {
    ubuntu, debian: {
      $package = 'sudo'
      $config_file = '/etc/sudoers'
      $config_dir = '/etc/sudoers.d/'
      $source = 'puppet:///modules/sudo/sudoers'
    }
    default: {
      fail("Unsupported platform: ${::operatingsystem}")
    }
  }
}

If we step through the sudo::params class, we see on line #2 a Puppet case statement that checks the operatingsystem fact. If the operatingsystem is Ubuntu or Debian, then the $package, $config_file, $config_dir, and $source variables are set. Otherwise we get an error that the operating system is not supported.

This is a classic use of “Smart Parameter Defaults” as described in the docs on using parameterized classes. Basically the saz/sudo module manages all default values in one place, shielding the consumer of the primary sudo class from the gory details.

Example usage

Using the saz/sudo module with default values is pretty straight forward. Using a node declaration we declare the parameterized class like this:

$ cat /etc/puppet/manifest/site.pp
…
node 'pmotw.puppetlabs.com' {
  class { 'sudo': }
}
…

Then on the agent we issue a Puppet run:

$ puppet agent -t
info: Caching catalog for pmotw.puppetlabs.com
info: Applying configuration version '1329932829'
notice: /Stage[main]/Sudo/Package[sudo]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]/Sudo/File[/etc/sudoers.d/]/mode: mode changed '0755' to '0550'
notice: /Stage[main]/Sudo/File[/etc/sudoers]/content:
--- /etc/sudoers	2012-02-22 12:47:50.000000000 -0500
+++ /tmp/puppet-file20120222-19626-yy7j0r-0	2012-02-22 12:47:50.000000000 -0500
…

At this point we have the sudo package installed and a sudo configuration file that doesn’t do very much.

If I try to use the sudo command as a non-root user, I run in to the following problem:

kelseyhightower@pmotw:~$ sudo su -
 
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
 
    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.
 
[sudo] password for kelseyhightower:
kelseyhightower is not in the sudoers file.  This incident will be reported.

I’m not in the sudoers file! This is an easy fix.

First we update our node definition with a sudo::conf declaration:

# cat /etc/puppet/manifests/site.pp 
node 'pmotw.puppetlabs.com' {
  class { 'sudo': }
  sudo::conf { 'kelseyhightower':
    priority => 10,
    content  => 'kelseyhightower ALL=(ALL) NOPASSWD: ALL',
  }
}

After a Puppet run on the agent, we end up with the following sudo configuration file:

# cat /etc/sudoers.d/10_kelseyhightower
kelseyhightower ALL=(ALL) NOPASSWD: ALL

Lets try using the sudo command to switch to the root user:

kelseyhightower@pmotw:~$ sudo su -
root@pmotw:~#

That works! If I wanted to use a more complex configuration for the kelseyhightower user, I could have used a source attribute in-place of content:

# cat /etc/puppet/manifests/site.pp 
node 'pmotw.puppetlabs.com' {
  class { 'sudo': }
  sudo::conf { 'kelseyhightower':
    priority => 10,
    source   => 'puppet:///files/etc/sudoers.d/users/kelseyhightower',
  }
}

Conclusion

The saz/sudo module sports all the characteristics of a great Puppet module:

  • Uses “Smart Parameter Defaults”
  • Exposes the right amount of class parameters and leaves more complex configuration to templates and files.
  • Serves a single purpose, managing sudo.

If you’re looking to manage sudo and related configuration, the saz/sudo module fits the bill nicely. Until next time, keep adding to the Puppet Forge!