Puppet Labs

OlinData sponsored Hyderabad India Puppet Master Training

Heidi — Wed, 23 May 2012 21:10:17 +0000

Namaste India!

Puppet Master Curriculum (3 Days)

This training is ideal for those who want a Puppet jumpstart. Newer members at an organization already using Puppet, or experienced sysadmins wanting to bring Puppet into their team will get everything they need to deploy solutions.

Prerequisites:

Attendees should have at least the equivalent experience of a junior Unix/Linux administrator.

Topics covered include:

Configuring Puppet and Puppetmaster
Resource Types and the Resource Abstration Layer
Virtual Resources, Exported Resources and Stored Configs
Meta-parameters, Dependencies and Events
Classes, Modules and Definitions
Tags and Environments
Puppet Language Patterns and Best Practices

The topics are covered over 3 days. Sessions will mix theory and practice, balancing lectures with hands-on exercises. (Each student should bring a WiFi enabled laptop with VMWare installed to participate in the labs.)

Pricing

INR 38,000 ($699.00) + eventzilla fee by June 30, 2012; INR 45,000 ($819.00) + eventzilla fee on or after July 1, 2012.
To pay directly, please contact training@olindata.com. Note about the venue: OlinData can help you arrange a hotel stay at a reduced rate. Contact us at training@olindata.com for more details.

Module of the Week: puppetlabs/razor – Razor Provisioning Application

Nan Liu — Wed, 23 May 2012 16:01:23 +0000

Purpose	Razor Provisioning Software
Module	puppetlabs/razor
Puppet Version	2.7+
Platforms	Ubuntu Precise

Razor is next generation provisioning software that handles bare metal hardware and virtual server provisioning with inventory discovery and tagging, rule-based policy management, and extensible broker plugin integration. The usage of Razor for provisioning is discussed briefly in this blog, and additional information is available on a separate post by Nick Weaver, one of authors of the Razor project. Broker handoffs will be discussed in a follow up article.

Razor is currently released as a beta for preview, so there are no installation packages yet. The Razor module is intended to simplify the process for installing Razor, since it handles all application dependencies, and clones the Razor repo from GitHub.

The module is limited to Ubuntu Precise due to the availability of nodejs packages. In Debian, these packages are available in sid, so install this module only if you are comfortable adding the unstable repo. Razor puppet module support for other platforms will be evaluated as we continue to develop Razor (this is not the same as node provisioning platforms which includes Debian, Ubuntu, RedHat, OpenSuSE, and VMware ESX).

Installing the module

Complexity	Easy
Installation Time	5 minutes

On puppet 2.7.14 an PE 2.5, puppet module tool will automatically download and install all dependencies from forge.

$ puppet module install puppetlabs-razor
Preparing to install into /etc/puppet/modules ...
Downloading from http://forge.puppetlabs.com ...
Installing -- do not interrupt ...
/etc/puppet/modules
└─┬ puppetlabs-razor (v0.1.0)
├─┬ puppetlabs-mongodb (v0.0.1)
│ └── puppetlabs-apt (v0.0.3)
├── puppetlabs-nodejs (v0.2.0)
├── puppetlabs-stdlib (v2.3.2)
├── puppetlabs-tftp (v0.1.0)
├── puppetlabs-vcsrepo (v0.0.4)
└── saz-sudo (v2.0.0)

Once all the modules have been installed, the appropriate permission should be applied to the files under the modules folder:

$ chown -R puppet:puppet /etc/puppet/modules

Configuring the module

Complexity	Easy
Installation Time	5 minutes

The puppetlabs/razor module does not deploy DHCP service. The DHCP server needs to specify the initial boot file as pxelinux.0 and next-server option should direct traffic to the tftp server.If you currently don’t have DHCP service in your environment, and have control over your network environment, the puppetlabs/dhcp module will deploy dhcp services.

class { 'dhcp':
  dnsdomain   => [
                   'puppetlabs.lan,
                   '1.0.10.in-addr.arpa',
                 ],
  nameservers => ['10.0.1.20'],
  ntpservers  => ['us.pool.ntp.org'],
  interfaces  => ['eth0'],
  pxeserver   => '10.0.1.50',
  pxefilename => 'pxelinux.0',
}

If you want to test Razor on your local system in a virtual environment such as VMware Fusion 4, simply add the following lines into /Library/Preferences/VMware Fusion/vmnet8/dhcpd.conf the appropriate DHCP subnet and substitute $tftp_server_address with the Razor VM node ipaddress:

filename "pxelinux.0";
next-server ${tftp_server_ipaddress};

At the moment, the tftp service and razor service will be deployed to the same system. On the puppet master assign the razor class to the appropriate node, and trigger a puppet agent run afterwards:

node razor_host {
  class { 'razor':
    username  => 'razor',
    directory => '/opt/razor',
  }
}

Razor can also be deployed directly to the system with the modules installed via:

puppet apply /etc/puppet/modules/razor/tests/init.pp --verbose

At this point, login to the Razor installation directory and execute Razor to confirm the application is installed successfully:

$ razor
ProjectRazor - v0.1.6.0
 
	Usage:
	project_razor [slice name] [command argument] [command argument]...
	 Switches:
		 --debug        : Enables printing proper Ruby stacktrace
		 --verbose      : Enables verbose object printing
		 --no-color-out : Disables console color. Useful for script wrapping.
 
Loaded slices:
	[bmc] [broker] [image] [log] [model] [node]
	[policy] [tag]

Example usage

The example below will show the minimal steps from the end of Razor installation to provisioning an Ubuntu Precise system. We will focus on the process of getting a new system provisioned with Ubuntu.

Razor needs a micro kernel (MK) image upon initial pxeboot to detect hardware information and register systems to the inventory of nodes. The MK image is available on GitHub, and it should be loaded into razor image service first:

$ wget https://github.com/downloads/puppetlabs/Razor/rz_mk_dev-image.0.8.8.0.iso
$ razor image add mk ./rz_mk_dev-image.0.8.8.0.iso
Attempting to add, please wait...
New image added successfully
Images:
	UUID: 1nnkuB5BiH1C93HOO0PTFi
	Type: MicroKernel Image
	ISO Filename: rz_mk_dev-image.0.8.8.0.iso
	Path: /mnt/nfs/Razor/image/mk/1nnkuB5BiH1C93HOO0PTFi
	Status: Valid
	Version: 0.8.9.0
	Build Time: 2012-05-09 13:11:01 -0700

Once the razor MK images is loaded into Razor, new system connected to the network performing network boot will obtain a dhcp address, boot the microkernel and register with Razor. We can get a list of systems discovered via this process via the Razor node:

$ razor node
Discovered Nodes
         UUID           Last Checkin                  Tags
5PfILygjPv4WTayOnTOYMk  32 seconds     [cpus_1,memsize_4GiB,nics_1,vmware_vm]
26k3If4FI7HpsZIn7N7BXu  45 seconds     [cpus_2,memsize_16GiB,nics_2,vmware_vm]
66XZwZUJOpRXosg5SHHdbW  7 seconds      [cpus_4,memsize_96GiB,nics_4,physical]

Attributes regarding specific nodes can be drilled down into further details by specifying its UUID:

$ razor node attrib 5PfILygjPv4WTayOnTOYMk
Node Attributes:
          	Name                        Value
	architecture              i386
	domain                    localdomain
	fqdn                      mk000C294F881F.localdomain
	hardwareisa               unknown
	hardwaremodel             i686
	hostname                  mk000C294F881F
...

To provision a node, an Ubuntu Precise ISO should also be loaded along with a model which associates an instance of model template configuration with a specific ISO image:

$ razor image add os ../ubuntu-12.04-server-amd64.iso ubuntu_precise 12.04
Attempting to add, please wait...
New image added successfully
Images:
	UUID: 274HnNlQF5jvbo0y6U0aok
	Type: OS Install
	ISO Filename: ubuntu-12.04-server-amd64.iso
	Path: /mnt/nfs/Razor/image/os/274HnNlQF5jvbo0y6U0aok
	Status: Valid
	OS Name: ubuntu_precise
	OS Version: 12.04

Once the MK and operating system images are loaded into razor, we can create a deployment model using that image. First, list the available model templates:

$ razor model get template
Model Templates:
Template Name         Description         
centos_6        CentOS 6 Model            
debian_wheezy   Debian Wheezy Model       
opensuse_12     OpenSuSE Suse 12 Model    
ubuntu_oneiric  Ubuntu Oneiric Model      
ubuntu_precise  Ubuntu Precise Model      
vmware_esxi_5   VMware ESXi 5 Deployment

Next, select the ubuntu_precise template along with the iso_uuid generated when the ISO was loaded into Razor. This will trigger a series of configuration questions applicable for the Precise model template:

$ razor model add template=ubuntu_precise label=install_precise image_uuid=274HnNlQF5jvbo0y6U0aok
--- Building Model (ubuntu_precise):
Please enter node hostname prefix (will append node number) (example: node)
default: node
(QUIT to cancel)
> ubuntu
Please enter root password (> 8 characters) (example: P@ssword!)
default: test1234
(QUIT to cancel)
> testEnvPass!
Model created
Label =>  install_precise
Template =>  linux_deploy
Description =>  Ubuntu Precise Model
UUID =>  3LCN86Cpx0Te3Of5WbORkQ
Image UUID =>  274HnNlQF5jvbo0y6U0aok

The newly created model contains the hostname (which will be appended with node number), root password, and the ISO for deployment. We can certainly perform more customization within the preseed file, but in most cases this process is deferred to Puppet via broker plugin which will configure the target puppet master to handoff the node for management. At this point we can deploy this operating system, and based on our initial hardware inventory we will target nodes with 4GB of memory that are VMware virtual systems.

$ razor policy add template=linux_deploy label=precise model_uuid=3LCN86Cpx0Te3Of5WbORkQ broker_uuid=none tags=memsize_4GiB,vmware_vm enabled=true
Policy created
UUID =>  41o1z77j2R4ZsgjD9KTpPe
Line Number =>  2
Label =>  precise
Enabled =>  true
Template =>  linux_deploy
Description =>  Policy for deploying a Linux-based operating system.
Tags =>  [vmware_vm]
Model Label =>  install_precise
Broker Target =>  none
Bound Count =>  0

At this point we review our policy rules and based on razor node output we expect one system that matches the tag requirements to deploy Precise.

$ razor policy
Policies
#  Enabled     Label         Tags              Label          Count           UUID
0  true     precise  [memsize_4GiB,vmware_vm]  install_precise  0      41o1z77j2R4ZsgjD9KTpPe

In the future, other systems that match this policy rule will also deploy Precise. Once the appropriate nodes install Precise, we can either remove this policy completely via ‘policy remove ’ or temporarily disable it via ‘policy disable ’.

Conclusion

The Razor puppet module provides a quick and easy way to install Razor for testing. The module could benefit from additional puppet class parameters for razor::nodejs and razor::tftp that separates those components especially when tftp can be installed on a separate system. There are also manual processes in Razor such as loading ISO’s that could benefit from a Puppet type/provider. We certainly welcome recommendations for additional Razor service deployment platform, and contributions from our community for enhancements to this module and its dependencies.

Learn More:

Introducing Razor, a Next-Generation Provisioning Solution
Lex Parsimoniae : Cloud Provisioning with a Razor
puppetlabs/razor
Join the Razor webinar
Participate in the Razor Twitter chat at #puppetize – 2012-05-28 18:00 UTC

Introducing Razor, A Next-Generation Provisioning Solution

Nan Liu — Wed, 23 May 2012 13:16:02 +0000

Puppet Labs is really excited to introduce Razor, a next-generation hardware provisioning solution developed collaboratively with our friends at EMC. I’ll be diving into details about specific Razor features in a series of blog posts, which includes installation via a Puppet module, but in this post I’ll focus on a high level overview of Razor’s features and the problem it’s solving.

Problems With Current Solutions

EMC actually initiated Razor in response to challenges they were having internally with existing hardware provisioning technologies. In building-up test and burn-in environments for their own software solutions, EMC found themselves bottlenecked by provisioning tools that were vendor-specific, monolithic, and closed. Moreover, public and private clouds gave their Development & QA teams infrastructure that responded nimbly and on-demand; why couldn’t their hardware infrastructure give them the same responsiveness and efficiencies?

To remedy, EMC first surveyed the landscape of provisioning technologies. Given the importance of provisioning as the first, critical step in building compute infrastructure, surely someone had innovated a solution to these problems? But nothing seemed to really address the needs of the system administrators responsible for infrastructure, so EMC set out to solve the problem themselves.

Enter Puppet

EMC was already very familiar with Puppet Labs and our products, and these served as an inspiration and foundation for Razor. Specifically, the EMC team sought to bring Puppet’s declarative, model-based approach of configuring software to the provisioning of server hardware with an operating system. In addition, Puppet Labs’ metadata-driven enterprise message bus and node inventory technologies, MCollective and Facter respectively, were critical enabling components for Razor.

Further considerations in selecting to partner with Puppet Labs were the pervasiveness of its open source technologies as well as its active and engaged community. EMC’s Razor team recognized that existing provisioning technologies were handicapped by their tight association with a single vendor, and open sourcing Razor to an active community was the surest way to avoid a similar fate.

Razor: Cloud Agility and Efficiency for Hardware Infrastructure

OK, enough already … what’s different about Razor?

First, during boot the Razor client auto-discovers the inventory of the server hardware – CPUs, disk, memory, everything – and feeds this to the Razor server in real-time. This means you always have the latest, most up-to-date information about every server in your data center, eliminating the need for manual inventory processes and tools.

Second, Razor knows the latest state of every server, and this state is updated with the auto-discovered inventory data. Razor also maintains a set of rules to match the appropriate operating system images with server capabilities as expressed in metadata. This match is made dynamically, so any change in the underlying hardware is automatically detected, reflected in the model, and can potentially result in the selection of a different operating system image – all automatically.

Third, Razor is integrated with Puppet for a seamless hand-off. After Razor selects the operating system image it then automatically installs the Puppet agent and classifies the node. Puppet picks up the ball and begins configuring the operating system, and so on up the stack.

Fourth, in contrast to other provisioning technologies Razor is open, pluggable, and programmable. Hardware has a unique boot sequence? No problem, create a new model. Want to change how operating system images are selected? Just update the rules via the Razor CLI. Need to support a new OS? Easy. Razor is all about giving choice, agility, and automation to the system administrator so they can eliminate the repetitive and menial from their day.

What this all means is that now Razor and Puppet enable the system administrator to automate the entire infrastructure stack, from bare metal to fully deployed applications on the cloud.

Check It Out

Nigel Kersten, Puppet Labs’ CTO, will be on stage during Chad’s World at EMC World today at 5:30pm PDT to demo Razor, and we’re also in the EMC Innovation Center on the show floor. Also, our friends from the EMC CTO’s Office – Dan Hushon, Chuck Hollis, Nick Weaver – have blog posts out today (here, here, and here, respectively) sharing the details of EMC’s work and perspective on Razor.

Razor is available for download now from the Puppet Forge. Check it out – we’re looking forward to your feedback. And stay tuned for more Razor blog posts soon!

- Nan

Learn More

Participate in the Razor Twitter chat at #puppetize – Mon May 28 11am PDT
Join the webinar with EMC – Tue May 29 11am PDT
Download Razor from the Puppet Forge
Read the docs on Github
Check-out the official press release

The Design Behind Puppet Sites

Daniel Sauble — Tue, 22 May 2012 20:29:45 +0000

Design is an integral part of the way we build software at Puppet Labs. More specifically, we strive to answer a very simple question: what does the user need? This isn’t always an easy question to answer, but we’ve been happy with our success in doing so. User empathy is our conduit for user satisfaction.

In real-life terms, what does design at Puppet look like? Let me show you a project that just finished its initial design phase: Puppet Sites.

What do users need?

Puppet admins want to spend their time managing nodes that are under Puppet control. They don’t want to spend their time adding nodes to their deployments. On the contrary, their goal is to start managing new nodes as quickly as possible. We formalize this need in the following way:

As a Puppet admin I want to easily add new nodes to my deployment so that I can start managing them as quickly as possible.

How do we we satisfy user needs?

The current workflow for adding nodes to a deployment is as follows:

Configure puppet.conf with the location of the master/CA
Perform an agent run on the node
Login to the CA
Sign the node’s certificate
Perform another agent run on the node

The problem with the existing workflow is that it appears to have nothing to do with adding new nodes to a deployment and everything to do with signing certificates. There’s a disconnect between the workflow and the user need.

To fix this, we did three things:

We introduced the concept of a site. A site is a service that owns the list of nodes in your deployment, the authentication mechanism for adding new nodes to your deployment, and the configuration of Puppet services in your deployment.
We changed the semantics of the workflow, so that it directly addresses the user need.
We eliminated the overhead of signing into the CA and manually signing node certificates. There are secure ways to do this that don’t involve user interaction over SSH.

The workflow now resembles the following:

Login to the site host
Generate a pre-shared key
Join a node to the site using the pre-shared key
Repeat step 3 for every node you want to add to the site

We now have a workflow to fulfill the user’s goal. But notice that we’re still not talking about sites in technical terms. This workflow could apply equally well to a Puppet Face, a REST API, or a Dashboard plugin.

How do users interact with our workflows?

It’s all well and good to produce a general workflow for addressing a user need, but eventually you have to tie it to a real user interface. For the initial release of Puppet Sites, we decided to focus on two user interfaces: a Puppet Face and a REST API.

A designer is nothing without regular interaction with customers. While designing user interaction, we stayed engaged with our internal operations and professional services teams. After several rounds of feedback, the design of the Puppet Face for the workflow given above resembles the following:

node02$ ssh admin@site02.domain.com
Last login: Mon May  7 18:15:43 2012
site02$ mount /media/usbdisk
site02$ puppet site generate key > /media/usbdisk/site.key
site02$ umount /media/usbdisk
site02$ exit
node02$ mount /media/usbdisk
node02$ puppet node join site02.domain.com < /media/usbdisk/site.key
Trying to add node02.domain.com to the site at site02.domain.com...
 Use `puppet site status node02.domain.com` to confirm success
 To stop waiting for the command to complete, press Ctrl-C.
   The command will still complete in the background.
Added node02.domain.com to the site at site02.domain.com

What do users think?

We strive to make our design process as agile as our engineering process. We specifically avoid a waterfall model, where the design is completed in an ivory tower, then handed over the wall for implementation. Instead, our designers sanity check their design with internal and external users, talk to architects and engineers to make sure they aren’t violating the laws of physics, and stay involved in the project until it ships, making course corrections as necessary.

The Puppet Sites project is still underway, but we’ve already discovered mistakes made early in the process, and corrected them. Each group of users has been vital to this iterative process.

Our internal users made us aware that omitting a pre-shared key system was a poor idea. By themselves, the two other fixes—changing semantics and introducing sites—weren’t sufficient to solve the user goal of “quickly” adding nodes to their deployments.

Our architects made us aware of a “law of physics” deal-breaker in the design, where users could have passed their pre-shared key as a flag to puppet node join. This would have exposed the key to sniffer processes on the host machine, and represented an unacceptable security risk.

Our external users helped us realize that the concept of a “site” is foggy at best and they didn’t really understand the problem it was designed to solve. Also, they were concerned that we were deprecating existing functionality without providing a drop-in replacement. We’ve been working to ameliorate these concerns by formulating clearer stories around the problems that sites solve.

Design at Puppet Labs

Of course, Puppet Sites does far more than just provide admins with an easy way to add new nodes to their deployments. For the sake of brevity, I’ve only highlighted this one user story. However, having a site also helps Puppet admins who want to…

…get a list of all the nodes in their deployment with a single command, so they don’t have to trawl multiple services to get this information.
…centrally manage the configuration of all nodes in their deployments, so they don’t have to manually manage puppet.conf on each node.
…access information about Puppet services from their manifests, so they don’t have to hardcode the location of their services into their manifests.

You can expect Puppet Sites to be available soon, as a point release for Puppet 3.0.

In summary, the UX team at Puppet produces three main artifacts: user stories, workflows, and wireframes.

User stories allow us to talk about the needs of our users in a non-technical, goal-oriented way.
Workflows describe the tasks users go through to achieve their goals.
Wireframes are mockups of the actual systems that users use to complete these tasks.

Not all of our projects follow this process exactly, but, in general, the above holds true. User story to workflow to wireframe. Rinse and repeat as necessary. That’s how we do design at Puppet Labs.

Learn More

Phoenix Puppet Fundamentals Training

Joe Henderson — Tue, 22 May 2012 16:30:34 +0000

Puppet Fundamentals Curriculum (3 Days)

This training course teaches students a best practice approach to managing their infrastructure using Puppet IT automation software. (Replaces the former Puppet Master course.)

Prerequisites:

This course has no prerequisites, but to complete the course successfully attendees need to have at least the equivalent experience of a junior Unix/Linux administrator

Topics covered include:

Developing modules/classes on a system that represent target systems
Using Puppet Apply to test and iterate modules
Placing modules on the Puppet Master
Declaring appropriate classes in node definitions
Triggering a Puppet run using the Live Management GUI
Collecting and analyzing results in the Enterprise Console
Puppet Language Patterns and Best Practices

The topics are covered over 3 days. Sessions will mix theory and practice, balancing lectures with hands-on exercises. (In order to perform the hands-on exercises, each student needs to bring a WiFi enabled laptop with VMware installed to participate in the labs.)

Pricing

$2,195 by Aug 27, 2012; $2,395 on or after Aug 28, 2012.

Open Source Twitter Chat on May 28th: #Puppetize

michelle — Mon, 21 May 2012 22:21:56 +0000

We’ve had a lot of open source goodness this month, and we want to talk about it and get your feedback. We’re hosting an hour-long Twitter chat on May 28 at 11 am PDT to answer your questions and converse about new releases like PuppetDB, MCollective 2.0, Puppet 3.0 (currently in RC), and more.

We’ll have Michael Stahnke, our community manager; Deepak Giridharagopal, major developer of PuppetDB; R.I. Pienaar, author of MCollective; and Nan Liu, integration specialist, ready to respond and converse on Twitter.

Just tag your tweets and questions with #puppetize to join the conversation, and we’ll tag our responses. If you want to follow what’s going on, just track #puppetize for the full stream. To get an idea of what this is like, check out a snippet of the last Twitter chat we had:

We look forward to talking with you on 2012-05-28 18:00 UTC!

Learn More:

Introducing PuppetDB

Announcing the Marionette Collective 2.0

Puppet Labs on Twitter

Module of the Week: pdxcat/amanda – Advanced Network Backup

Reid Vandewiele — Sat, 19 May 2012 17:36:36 +0000

The following is a guest post by Reid Vandewiele, a system administrator at the Portland State University Computer Action Team (PDX CAT). Reid, William Van Hevelingen, Spencer Krum and other CATs are big contributors to various modules on the Puppet Forge and also host a few of their own. They are active members of the Puppet community and can usually be found on IRC under the monikers marut, blkperl and nibalizer, respectively. Thanks guys for the awesome guest post!

Purpose	Provides amanda server and client configuration
Module	pdxcat/amanda
Puppet Version	2.7+
Platforms	Debian, Solaris, FreeBSD, SuSE

The Advanced Maryland Automatic Network Disk Archiver, or Amanda for short, is a network backup solution in the same class as Bacula. Proponents tout its smart automatic planner, use of native tools to perform data dumps, ability to recover data from tape in the absence of the tool itself, and the available commercial support through Zmanda. A venerable bastion of free and open source software, Amanda has been around since 1991 and is still actively maintained with the most recent stable version having been released on February 12, 2012.

Let’s Puppetize that!

The pdxcat/amanda module was developed to handle the installation and basic configuration of the client and server components of the Amanda system in a heterogeneous computing environment. Besides the core classes, the module ships a couple of defined types to assist in setting up multi-server authentication. In the most basic configuration, however, it is only necessary to include amanda::server and/or amanda::client on a given node.

Installing the module

Complexity	Easy
Installation Time	5 minutes

The newest version of pdxcat/amanda can always be obtained from GitHub. That said, it’s a heck of a lot easier and faster to install it from the Puppet Forge. In this example I’ll be using three machines all running Ubuntu 12.04 “Precise” (please see the addendum for an important disclosure regarding the use of this version). This means that I won’t be able to demonstrate pdxcat/amanda on Solaris or FreeBSD, but that’s one of the beautiful aspects of Puppet for well-maintained modules: The steps I take to configure my Ubuntu machines using Puppet should translate directly to performing the same configuration on any supported platform. Now on to the Puppet Forge:

The Amanda software follows the traditional Unix philosophy of “do one thing and do it well,” letting appropriate pre-existing utilities dump data at the system level, and relying on either inetd or ssh for network transport. Amanda does not attempt to reinvent the wheel. The pdxcat/amanda Puppet module correspondingly utilizes and has a dependency on pdxcat/xinetd (a fork of ghoneycutt/xinetd) in addition to its dependency on the utility module ripienaar/concat.

The Puppet Module Tool (PMT) handled that minor bit of complexity for me. After issuing just one command I’m all ready to go.

Well, almost. I actually ran into snag with this setup that needs to be addressed but which due to the off-topic nature I’m disinclined to cover in detail.

Every time Ubuntu comes out with a new release I do this and every time I forget. It’s a Bad Idea (TM) to assume that less than three weeks after launch the newest bleeding-edge version of Ubuntu will “just work”. As it turns out, at the time of writing the stock Amanda packages that ship with Ubuntu 12.04 were broken. See Bug 932064. I’m sure it will be fixed soon, but in the meantime – and In order to finish this blog post – I was forced to backport Amanda packages from Quantal into my repo to test the module on 12.04 VMs. C’est la vie.

Resource Overview

The README provides a decent overview of the resources intended for the end user, but it doesn’t go into a lot of detail beyond the most basic usage and parameters. It introduces two classes and three defined types. To get at the gory details though, it’s necessary to open up the manifests themselves. I personally like to use GitHub for this when inspecting unfamiliar modules but as the PMT has just installed the files in /etc/puppet/modules, they’re only a few keystrokes away.

Amanda::Server

The server class exposes nine parameters to the user, none of which are required. This means that if I wanted to, I could literally issue `include amanda::server` to handle the software installation and take a pass on any additional functionality.

What additional functionality is available mostly has to do with installing one or more manually written Amanda configs – a term which in-context refers to a set of files and directories that together constitute all the information Amanda needs to perform backups. Most of the parameters are tied into this and will be addressed later in the configuring section.

The xinetd parameter is standalone, and boolean. When true, the pdxcat/xinetd module will be used to install the Amanda server-side services (amindexd, amidxtaped) in xinetd. False, and no xinetd services will be installed.

Amanda::Client

The client class is much simpler in comparison. This makes sense, since all native Amanda logic is located on the server and the client software usually serves only as a dumb data dumper. The interesting configuration in the client class has to do with setting up authentication to the remote Amanda server.

The remote_user parameter is used to tell the client what unix user the server will be running as (important because Amanda cares, part of the authentication determination), and the server parameter tells the client what backup server will be used. Like in the server class, the xinetd parameter tells the client whether or not to install xinetd services (amdump, in this case).

If more than one backup server/user is being used at a site, the amanda::amandahosts defined type can be used to specify that kind of additional information.

Defined Types

When the client and server classes aren’t precise enough, more manual configuration is possible using the defined types provided: amanda::amandahosts, amanda::config, and amanda::ssh_authorized_key. We won’t get into detail about them here. Suffice it to say that they provide a handy platform-agnostic way of tweaking Amanda nuts and bolts in a way that takes advantage of the module’s knowledge of platform quirks.

For those interested the defined types, the README file for the module contains examples and additional information. It’s certainly worth the extra foray into the code because as it turns out, outside of the (surprisingly gnarly) package installation, most pdxcat/amanda class logic is a wrapper for using these defines. The additional exploration will also expose why pdxcat/amanda declares a dependency on ripienaar/concat (Dun dun dun!).

(Spoiler: the .amandahosts file is managed with the concat pattern.)

Testing the module

The top-level view of the module is somewhat misleading when it comes to tests.

At first it appears that the module may contain both smoke tests and rspec tests, but as it turns out the existence of tests/ and spec/ directories is a false positive. The directories are empty in the 0.1.2 release.

Not to be daunted by so ephemeral a problem as missing tests, let’s forge bravely ahead to perform what sanity checking we can. Syntax validation and style evaluation are two easy checks to run, as previous articles in this series have demonstrated. Note that while the Puppet syntax checker is included in core, the puppet-lint tool is not and to use it you’ll need to install the `puppet-lint` gem.

Pretty good! No syntax errors and no heinous violations of style. This is by no means to say that the code is perfect (or even necessarily good), but adherence to the recommendations set forth in the Puppet Labs Style Guide says something meaningful about maintainability and potential community involvement. It suggests that the maintainers are up to date and in sync with the greater Puppet community …or alternatively that they are aware of puppet-lint and are suckers for perfect output. Whatever works. :-)

The absence of smoke tests is something that could be easily rectified. In fact, the module’s README file already has all the necessary code in it. Filling in the smoke tests for this module will almost literally be a cut-n-paste operation.

Configuring the module

Complexity	Easy
Installation Time	5 minutes

The client class seems pretty straight-forward and even intuitive, but there’s definitely something strange going on with the server. Luckily the README does a pretty good job of explaining what is going on.

The short version is that the configs_source parameter works kind of like a Puppet File source parameter “puppet:///” URI, but without the “puppet:///” part. That is, “modules/data/amanda” means the “files/amanda” directory out of the “data” module. Okay… Weird, but okay. Let’s just roll with it for a minute and see where this leads.

The “configs” parameter is an array of values that is intended to specify which Amanda configs are to be installed on the server. It is combined with the configs_source parameter to produce the name of a directory to effectively rsync from the puppet master to the agent. Here we are syncing all files and directories from the master’s $modulepath/data/files/amanda/daily and $modulepath/data/files/amanda/archive to the node’s /etc/amanda/daily and /etc/amanda/archive.

Besides configs and configs_source, there are a few more parameters that can be used to more finely define the behavior of that sync operation. Those parameters are:

configs_directory – The basedir on the node to which the config directories will be synced
manage_configs_directory – Whether or not to declare a file resource to manage the configs_directory
owner – The owner to apply to all files synced as part of a config
group – The group to apply to all files synced as part of a config
file_mode – The mode to apply to all regular files synced as part of a config
directory_mode – The mode to apply to all directories synced as part of a config

Under the hood config syncing is accomplished by using a custom function in the Amanda module to ferret out the list of files while the catalog is being compiled on the master. It was implemented this way largely to eliminate the need to enumerate all the files that make up an Amanda config, and from a purely functional perspective it works great. It seems a little out of sync with the feel of other puppet resources but I can’t think of a better way to do it yet and so I’m inclined to give it my blessing and move on to the test run.

Example usage

We’ve walked through the config. Now let’s see how it runs on our virtual Amanda server node.

…Wait, what? The error message reads: “Unable to find referenced module ‘data’ at [...]”. Oh. Right, I just finished talking all about how the configs parameter and configs_source parameter works, but I didn’t actually set it all up. The Amanda module is letting me know in no uncertain terms that it’s not gonna happen until I do.

Setting aside for the moment the details of what an Amanda config looks like, I’ll just sketch out the “data” module that I told Amanda I would be using.

Cool. Let’s roll the server again.

A wall of blue and green text (most of which is omitted here) scrolls by. As far as the Amanda module is concerned, the server is ready to go. Let’s do the same thing on the client.

Board is green. Stepping back for a moment, we should now have the following systems in place:

puppet.local, on which we installed the pdxcat/amanda module and set up node definitions for server.local and client.local.
server.local, with an Amanda server set up and the mock “daily” and “archive” configs installed.
client.local, for which server.local should be able to perform a backup.

It all looks pretty good, but I have to break the fourth wall and talk about Amanda again for a minute. In order to test and verify that this all works we need to actually have an Amanda config, and not just the empty directory sketch-up I did to get the catalog(s) to compile. At its simplest, an Amanda config needs to have an amanda.conf file and a disklist file. I’m going to install now the following contrived example.

# File: amanda.conf
define tapetype "vtape" {
    length 64 mbytes
}
define changer "vtapes" {
  tpchanger "chg-disk:/tmp"
}
define dumptype "default" {
  program "GNUTAR"
  auth "bsdtcp"
}
dumpcycle 7 days
dumpuser "backup"
indexdir "/var/tmp/amanda/index"
infofile "/etc/amanda/daily/curinfo"
labelstr "^V[0-9][0-9][0-9]$"
logdir "/etc/amanda/daily"
mailto "root@server.local"
org "pdxcat/amanda"
runspercycle 1
runtapes 1
tapecycle 9
tapetype "vtape"
tpchanger "vtapes"
 
# File: disklist
client.local /etc default

What this config will do is backup the /etc directory of the client machine to “virtual tapes” in /tmp on the server. Brilliant, I know. Now to install the config, we just need to include the files in the “data” module, as specified in the Puppet configuration for server.local, which if you will recall was given the parameters

configs        => [ 'daily', 'archive' ],
configs_source => 'modules/data/amanda’,

First, install the files in the “data” module on the puppetmaster.

So, pausing for a moment: Why is it that we are specifying that the configs come from the “data” module? Why aren’t we installing them in the Amanda module instead?

The simple answer is we’re splitting things up like this because we can. The pdxcat/amanda module follows best practice in providing an interface that allows full utilization of the module without the need to modify any of the files and directories provided. Ideally, a reusable module can be thought of and used as a library. When writing a text munger in C, one calls library functions like read() and write(), but doesn’t typically patch their libc.so. Seperation of data and code is a common maintainability practice, and just as applicable to configuration management – especially once your Puppet installation starts getting big.

The “data” module used here represents a site-local module containing non-reusable data or code that is specific to our operation – data such as the amanda.conf and disklist files we just created. Using the configs_source parameter in the amanda::server class lets us cleanly tie together reusable function from the Amanda module and custom information from our own data module.

Enough talking. Let’s kick off a Puppet to run on the server to make this all more real.

Note that the two config files from the “data” module have been installed in /etc/amanda/daily by amanda::server.

For this demo, there’s a manual config-specific step necessary to make the virtual Amanda server pretend that it’s a real boy. Normally a backup server would have some hardware attached to it on which to store the backup data it receives from clients. Maybe a giant disk array, or a standalone tape robot… Something corporeal that is referenced in the config.

Recall from the config given previously that we have opted to use /tmp. Here follows an appropriate hack to initialize our “backup tapes”.

And that’s it. We have an Amanda server on which Amanda was set up and the config installed by Puppet, and we have a client machine on which Amanda was installed and configured by Puppet to allow the server to perform backups. There was some manual setup necessary on the server to install “tapes” on which to save backup data, but we’re finally able to answer the important question “Does it work?” with a resounding “Yes!”

The check command ran and returned without error. The amdump command ran and returned silently, in typical Unix fashion. Amanda sends the full report to the user specified in the config (in this case root) but I’ll spare you the gory details of the long and verbose output. Suffice it to say that the backup test was successful.

Conclusion

The pdxcat/amanda Puppet module provides a good head-start towards managing the Amanda backup solution. Writing the configs is still left up to the administrator, but installing and ensuring them once written is easy and fast. Much of the legwork on the server side can be handled by Puppet, and all of the client-side configuration is effectively automated. The notable client configuration that is missing (configuration which was not used in this demonstration) is the amanda-client.conf file, which could have been used to set up client-side restore operation defaults.

Looking through the module code it feels like there is a lot of complexity around installing the client/server packages, but at its core the functionality provided by the module is simple and focused.

What the module does:

Install Amanda client and/or server packages
Configure the client(s) to allow the server to perform backups
Ensure config files written by an administrator are installed on the server
Support a variety of platforms

What the module doesn’t do:
Parameterize or build Amanda config files (server-side)
Manage the Amanda disklist or otherwise provide for puppet-integrated configuration of backup targets
Schedule the backup runs
Provide tests

Of particular note is that testing the module at present requires some upfront knowledge of Amanda and the writing of a custom config. Well wait a second, we just did all that work, right? How about turning it into an all-in-one stand-alone smoke test for the Puppet module?

Writing this post has brought to light a couple of awesome ideas that would make the Amanda module even better, some of which wouldn’t even take all that much time to code. In no particular order, here’s the brainstorm list from the review:

Smoke tests for the classes and defines
>Integrated smoke test for a server that includes setting up a contrived Amanda config
Rspec tests for the classes, defines, and functions
Allow management of amanda-client.conf through the amanda::client class
Provide a defined type to schedule backup runs for a specified config

Since the first draft of this blog post was written, brainstorm item #1 has been added to the module on Github, a couple of bugs have been fixed, and the “demo” config that was written for this blog post has been turned into a working smoke test, completing brainstorm item #2. Fixing that item #2 is especially exciting as it significantly reduces the overhead for installing and testing the Amanda module (albeit with just a single VM), whittling it down to these five commands on Ubuntu:

# puppet module install pdxcat-amanda
# puppet apply /etc/puppet/modules/amanda/tests/demo.pp
# su - backup
$ amcheck demo
$ amdump demo

The detailed results will be mailed to root@localhost. These changes are available in the at-the-time-of-writing newest pdxcat/amanda release from the Puppet Forge, pdxcat/amanda 0.1.3.

A solid set of core classes and a good collection of defined type tools already makes pdxcat/amanda well worth using for deploying Amanda with Puppet. There are certainly areas that could be improved, but all in all it’s a decent module for a potentially complicated application.

Learn More

Washington, DC Puppet Fundamentals Training

Heidi — Fri, 18 May 2012 20:20:48 +0000

Puppet Fundamentals Curriculum (3 Days)

This training course teaches students a best practice approach to managing their infrastructure using Puppet IT automation software. (Replaces the former Puppet Master course.)

Prerequisites:

This course has no prerequisites, but to complete the course successfully attendees need to have at least the equivalent experience of a junior Unix/Linux administrator

Topics covered include:

Developing modules/classes on a system that represent target systems
Using Puppet Apply to test and iterate modules
Placing modules on the Puppet Master
Declaring appropriate classes in node definitions
Triggering a Puppet run using the Live Management GUI
Collecting and analyzing results in the Enterprise Console
Puppet Language Patterns and Best Practices

Pricing

$2,195 by Sept 3, 2012; $2,395 on or after Sept 4, 2012.

Chicago Puppet Fundamentals Training

Heidi — Fri, 18 May 2012 17:22:18 +0000

Puppet Fundamentals Curriculum (3 Days)

This training course teaches students a best practice approach to managing their infrastructure using Puppet IT automation software. (Replaces the former Puppet Master course.)

Prerequisites:

This course has no prerequisites, but to complete the course successfully attendees need to have at least the equivalent experience of a junior Unix/Linux administrator

Topics covered include:

Developing modules/classes on a system that represent target systems
Using Puppet Apply to test and iterate modules
Placing modules on the Puppet Master
Declaring appropriate classes in node definitions
Triggering a Puppet run using the Live Management GUI
Collecting and analyzing results in the Enterprise Console
Puppet Language Patterns and Best Practices

Pricing

$2,195 by Aug 27, 2012; $2,395 on or after Aug 28, 2012.

Austin Puppet Fundamentals Training

Heidi — Fri, 18 May 2012 17:14:21 +0000

Puppet Fundamentals Curriculum (3 Days)

This training course teaches students a best practice approach to managing their infrastructure using Puppet IT automation software. (Replaces the former Puppet Master course.)

Prerequisites:

This course has no prerequisites, but to complete the course successfully attendees need to have at least the equivalent experience of a junior Unix/Linux administrator

Topics covered include:

Developing modules/classes on a system that represent target systems
Using Puppet Apply to test and iterate modules
Placing modules on the Puppet Master
Declaring appropriate classes in node definitions
Triggering a Puppet run using the Live Management GUI
Collecting and analyzing results in the Enterprise Console
Puppet Language Patterns and Best Practices

Pricing

$2,195 by Sept 3, 2012; $2,395 on or after Sept 4, 2012.