Puppet Labs » General News

Module of the Week: puppetlabs/razor – Razor Provisioning Application

Nan Liu — Wed, 23 May 2012 16:01:23 +0000

Purpose	Razor Provisioning Software
Module	puppetlabs/razor
Puppet Version	2.7+
Platforms	Ubuntu Precise

Razor is next generation provisioning software that handles bare metal hardware and virtual server provisioning with inventory discovery and tagging, rule-based policy management, and extensible broker plugin integration. The usage of Razor for provisioning is discussed briefly in this blog, and additional information is available on a separate post by Nick Weaver, one of authors of the Razor project. Broker handoffs will be discussed in a follow up article.

Razor is currently released as a beta for preview, so there are no installation packages yet. The Razor module is intended to simplify the process for installing Razor, since it handles all application dependencies, and clones the Razor repo from GitHub.

The module is limited to Ubuntu Precise due to the availability of nodejs packages. In Debian, these packages are available in sid, so install this module only if you are comfortable adding the unstable repo. Razor puppet module support for other platforms will be evaluated as we continue to develop Razor (this is not the same as node provisioning platforms which includes Debian, Ubuntu, RedHat, OpenSuSE, and VMware ESX).

Installing the module

Complexity	Easy
Installation Time	5 minutes

On puppet 2.7.14 an PE 2.5, puppet module tool will automatically download and install all dependencies from forge.

$ puppet module install puppetlabs-razor
Preparing to install into /etc/puppet/modules ...
Downloading from http://forge.puppetlabs.com ...
Installing -- do not interrupt ...
/etc/puppet/modules
└─┬ puppetlabs-razor (v0.1.0)
├─┬ puppetlabs-mongodb (v0.0.1)
│ └── puppetlabs-apt (v0.0.3)
├── puppetlabs-nodejs (v0.2.0)
├── puppetlabs-stdlib (v2.3.2)
├── puppetlabs-tftp (v0.1.0)
├── puppetlabs-vcsrepo (v0.0.4)
└── saz-sudo (v2.0.0)

Once all the modules have been installed, the appropriate permission should be applied to the files under the modules folder:

$ chown -R puppet:puppet /etc/puppet/modules

Configuring the module

Complexity	Easy
Installation Time	5 minutes

The puppetlabs/razor module does not deploy DHCP service. The DHCP server needs to specify the initial boot file as pxelinux.0 and next-server option should direct traffic to the tftp server.If you currently don’t have DHCP service in your environment, and have control over your network environment, the puppetlabs/dhcp module will deploy dhcp services.

class { 'dhcp':
  dnsdomain   => [
                   'puppetlabs.lan,
                   '1.0.10.in-addr.arpa',
                 ],
  nameservers => ['10.0.1.20'],
  ntpservers  => ['us.pool.ntp.org'],
  interfaces  => ['eth0'],
  pxeserver   => '10.0.1.50',
  pxefilename => 'pxelinux.0',
}

If you want to test Razor on your local system in a virtual environment such as VMware Fusion 4, simply add the following lines into /Library/Preferences/VMware Fusion/vmnet8/dhcpd.conf the appropriate DHCP subnet and substitute $tftp_server_address with the Razor VM node ipaddress:

filename "pxelinux.0";
next-server ${tftp_server_ipaddress};

At the moment, the tftp service and razor service will be deployed to the same system. On the puppet master assign the razor class to the appropriate node, and trigger a puppet agent run afterwards:

node razor_host {
  class { 'razor':
    username  => 'razor',
    directory => '/opt/razor',
  }
}

Razor can also be deployed directly to the system with the modules installed via:

puppet apply /etc/puppet/modules/razor/tests/init.pp --verbose

At this point, login to the Razor installation directory and execute Razor to confirm the application is installed successfully:

$ razor
ProjectRazor - v0.1.6.0
 
	Usage:
	project_razor [slice name] [command argument] [command argument]...
	 Switches:
		 --debug        : Enables printing proper Ruby stacktrace
		 --verbose      : Enables verbose object printing
		 --no-color-out : Disables console color. Useful for script wrapping.
 
Loaded slices:
	[bmc] [broker] [image] [log] [model] [node]
	[policy] [tag]

Example usage

The example below will show the minimal steps from the end of Razor installation to provisioning an Ubuntu Precise system. We will focus on the process of getting a new system provisioned with Ubuntu.

Razor needs a micro kernel (MK) image upon initial pxeboot to detect hardware information and register systems to the inventory of nodes. The MK image is available on GitHub, and it should be loaded into razor image service first:

$ wget https://github.com/downloads/puppetlabs/Razor/rz_mk_dev-image.0.8.8.0.iso
$ razor image add mk ./rz_mk_dev-image.0.8.8.0.iso
Attempting to add, please wait...
New image added successfully
Images:
	UUID: 1nnkuB5BiH1C93HOO0PTFi
	Type: MicroKernel Image
	ISO Filename: rz_mk_dev-image.0.8.8.0.iso
	Path: /mnt/nfs/Razor/image/mk/1nnkuB5BiH1C93HOO0PTFi
	Status: Valid
	Version: 0.8.9.0
	Build Time: 2012-05-09 13:11:01 -0700

Once the razor MK images is loaded into Razor, new system connected to the network performing network boot will obtain a dhcp address, boot the microkernel and register with Razor. We can get a list of systems discovered via this process via the Razor node:

$ razor node
Discovered Nodes
         UUID           Last Checkin                  Tags
5PfILygjPv4WTayOnTOYMk  32 seconds     [cpus_1,memsize_4GiB,nics_1,vmware_vm]
26k3If4FI7HpsZIn7N7BXu  45 seconds     [cpus_2,memsize_16GiB,nics_2,vmware_vm]
66XZwZUJOpRXosg5SHHdbW  7 seconds      [cpus_4,memsize_96GiB,nics_4,physical]

Attributes regarding specific nodes can be drilled down into further details by specifying its UUID:

$ razor node attrib 5PfILygjPv4WTayOnTOYMk
Node Attributes:
          	Name                        Value
	architecture              i386
	domain                    localdomain
	fqdn                      mk000C294F881F.localdomain
	hardwareisa               unknown
	hardwaremodel             i686
	hostname                  mk000C294F881F
...

To provision a node, an Ubuntu Precise ISO should also be loaded along with a model which associates an instance of model template configuration with a specific ISO image:

$ razor image add os ../ubuntu-12.04-server-amd64.iso ubuntu_precise 12.04
Attempting to add, please wait...
New image added successfully
Images:
	UUID: 274HnNlQF5jvbo0y6U0aok
	Type: OS Install
	ISO Filename: ubuntu-12.04-server-amd64.iso
	Path: /mnt/nfs/Razor/image/os/274HnNlQF5jvbo0y6U0aok
	Status: Valid
	OS Name: ubuntu_precise
	OS Version: 12.04

Once the MK and operating system images are loaded into razor, we can create a deployment model using that image. First, list the available model templates:

$ razor model get template
Model Templates:
Template Name         Description         
centos_6        CentOS 6 Model            
debian_wheezy   Debian Wheezy Model       
opensuse_12     OpenSuSE Suse 12 Model    
ubuntu_oneiric  Ubuntu Oneiric Model      
ubuntu_precise  Ubuntu Precise Model      
vmware_esxi_5   VMware ESXi 5 Deployment

Next, select the ubuntu_precise template along with the iso_uuid generated when the ISO was loaded into Razor. This will trigger a series of configuration questions applicable for the Precise model template:

$ razor model add template=ubuntu_precise label=install_precise image_uuid=274HnNlQF5jvbo0y6U0aok
--- Building Model (ubuntu_precise):
Please enter node hostname prefix (will append node number) (example: node)
default: node
(QUIT to cancel)
> ubuntu
Please enter root password (> 8 characters) (example: P@ssword!)
default: test1234
(QUIT to cancel)
> testEnvPass!
Model created
Label =>  install_precise
Template =>  linux_deploy
Description =>  Ubuntu Precise Model
UUID =>  3LCN86Cpx0Te3Of5WbORkQ
Image UUID =>  274HnNlQF5jvbo0y6U0aok

The newly created model contains the hostname (which will be appended with node number), root password, and the ISO for deployment. We can certainly perform more customization within the preseed file, but in most cases this process is deferred to Puppet via broker plugin which will configure the target puppet master to handoff the node for management. At this point we can deploy this operating system, and based on our initial hardware inventory we will target nodes with 4GB of memory that are VMware virtual systems.

$ razor policy add template=linux_deploy label=precise model_uuid=3LCN86Cpx0Te3Of5WbORkQ broker_uuid=none tags=memsize_4GiB,vmware_vm enabled=true
Policy created
UUID =>  41o1z77j2R4ZsgjD9KTpPe
Line Number =>  2
Label =>  precise
Enabled =>  true
Template =>  linux_deploy
Description =>  Policy for deploying a Linux-based operating system.
Tags =>  [vmware_vm]
Model Label =>  install_precise
Broker Target =>  none
Bound Count =>  0

At this point we review our policy rules and based on razor node output we expect one system that matches the tag requirements to deploy Precise.

$ razor policy
Policies
#  Enabled     Label         Tags              Label          Count           UUID
0  true     precise  [memsize_4GiB,vmware_vm]  install_precise  0      41o1z77j2R4ZsgjD9KTpPe

In the future, other systems that match this policy rule will also deploy Precise. Once the appropriate nodes install Precise, we can either remove this policy completely via ‘policy remove ’ or temporarily disable it via ‘policy disable ’.

Conclusion

The Razor puppet module provides a quick and easy way to install Razor for testing. The module could benefit from additional puppet class parameters for razor::nodejs and razor::tftp that separates those components especially when tftp can be installed on a separate system. There are also manual processes in Razor such as loading ISO’s that could benefit from a Puppet type/provider. We certainly welcome recommendations for additional Razor service deployment platform, and contributions from our community for enhancements to this module and its dependencies.

Learn More:

Introducing Razor, a Next-Generation Provisioning Solution
Lex Parsimoniae : Cloud Provisioning with a Razor
puppetlabs/razor
Join the Razor webinar
Participate in the Razor Twitter chat at #puppetize – 2012-05-28 18:00 UTC

The Design Behind Puppet Sites

Daniel Sauble — Tue, 22 May 2012 20:29:45 +0000

Design is an integral part of the way we build software at Puppet Labs. More specifically, we strive to answer a very simple question: what does the user need? This isn’t always an easy question to answer, but we’ve been happy with our success in doing so. User empathy is our conduit for user satisfaction.

In real-life terms, what does design at Puppet look like? Let me show you a project that just finished its initial design phase: Puppet Sites.

What do users need?

Puppet admins want to spend their time managing nodes that are under Puppet control. They don’t want to spend their time adding nodes to their deployments. On the contrary, their goal is to start managing new nodes as quickly as possible. We formalize this need in the following way:

As a Puppet admin I want to easily add new nodes to my deployment so that I can start managing them as quickly as possible.

How do we we satisfy user needs?

The current workflow for adding nodes to a deployment is as follows:

Configure puppet.conf with the location of the master/CA
Perform an agent run on the node
Login to the CA
Sign the node’s certificate
Perform another agent run on the node

The problem with the existing workflow is that it appears to have nothing to do with adding new nodes to a deployment and everything to do with signing certificates. There’s a disconnect between the workflow and the user need.

To fix this, we did three things:

We introduced the concept of a site. A site is a service that owns the list of nodes in your deployment, the authentication mechanism for adding new nodes to your deployment, and the configuration of Puppet services in your deployment.
We changed the semantics of the workflow, so that it directly addresses the user need.
We eliminated the overhead of signing into the CA and manually signing node certificates. There are secure ways to do this that don’t involve user interaction over SSH.

The workflow now resembles the following:

Login to the site host
Generate a pre-shared key
Join a node to the site using the pre-shared key
Repeat step 3 for every node you want to add to the site

We now have a workflow to fulfill the user’s goal. But notice that we’re still not talking about sites in technical terms. This workflow could apply equally well to a Puppet Face, a REST API, or a Dashboard plugin.

How do users interact with our workflows?

It’s all well and good to produce a general workflow for addressing a user need, but eventually you have to tie it to a real user interface. For the initial release of Puppet Sites, we decided to focus on two user interfaces: a Puppet Face and a REST API.

A designer is nothing without regular interaction with customers. While designing user interaction, we stayed engaged with our internal operations and professional services teams. After several rounds of feedback, the design of the Puppet Face for the workflow given above resembles the following:

node02$ ssh admin@site02.domain.com
Last login: Mon May  7 18:15:43 2012
site02$ mount /media/usbdisk
site02$ puppet site generate key > /media/usbdisk/site.key
site02$ umount /media/usbdisk
site02$ exit
node02$ mount /media/usbdisk
node02$ puppet node join site02.domain.com < /media/usbdisk/site.key
Trying to add node02.domain.com to the site at site02.domain.com...
 Use `puppet site status node02.domain.com` to confirm success
 To stop waiting for the command to complete, press Ctrl-C.
   The command will still complete in the background.
Added node02.domain.com to the site at site02.domain.com

What do users think?

We strive to make our design process as agile as our engineering process. We specifically avoid a waterfall model, where the design is completed in an ivory tower, then handed over the wall for implementation. Instead, our designers sanity check their design with internal and external users, talk to architects and engineers to make sure they aren’t violating the laws of physics, and stay involved in the project until it ships, making course corrections as necessary.

The Puppet Sites project is still underway, but we’ve already discovered mistakes made early in the process, and corrected them. Each group of users has been vital to this iterative process.

Our internal users made us aware that omitting a pre-shared key system was a poor idea. By themselves, the two other fixes—changing semantics and introducing sites—weren’t sufficient to solve the user goal of “quickly” adding nodes to their deployments.

Our architects made us aware of a “law of physics” deal-breaker in the design, where users could have passed their pre-shared key as a flag to puppet node join. This would have exposed the key to sniffer processes on the host machine, and represented an unacceptable security risk.

Our external users helped us realize that the concept of a “site” is foggy at best and they didn’t really understand the problem it was designed to solve. Also, they were concerned that we were deprecating existing functionality without providing a drop-in replacement. We’ve been working to ameliorate these concerns by formulating clearer stories around the problems that sites solve.

Design at Puppet Labs

Of course, Puppet Sites does far more than just provide admins with an easy way to add new nodes to their deployments. For the sake of brevity, I’ve only highlighted this one user story. However, having a site also helps Puppet admins who want to…

…get a list of all the nodes in their deployment with a single command, so they don’t have to trawl multiple services to get this information.
…centrally manage the configuration of all nodes in their deployments, so they don’t have to manually manage puppet.conf on each node.
…access information about Puppet services from their manifests, so they don’t have to hardcode the location of their services into their manifests.

You can expect Puppet Sites to be available soon, as a point release for Puppet 3.0.

In summary, the UX team at Puppet produces three main artifacts: user stories, workflows, and wireframes.

User stories allow us to talk about the needs of our users in a non-technical, goal-oriented way.
Workflows describe the tasks users go through to achieve their goals.
Wireframes are mockups of the actual systems that users use to complete these tasks.

Not all of our projects follow this process exactly, but, in general, the above holds true. User story to workflow to wireframe. Rinse and repeat as necessary. That’s how we do design at Puppet Labs.

Learn More

Open Source Twitter Chat on May 28th: #Puppetize

michelle — Mon, 21 May 2012 22:21:56 +0000

We’ve had a lot of open source goodness this month, and we want to talk about it and get your feedback. We’re hosting an hour-long Twitter chat on May 28 at 11 am PDT to answer your questions and converse about new releases like PuppetDB, MCollective 2.0, Puppet 3.0 (currently in RC), and more.

We’ll have Michael Stahnke, our community manager; Deepak Giridharagopal, major developer of PuppetDB; R.I. Pienaar, author of MCollective; and Nan Liu, integration specialist, ready to respond and converse on Twitter.

Just tag your tweets and questions with #puppetize to join the conversation, and we’ll tag our responses. If you want to follow what’s going on, just track #puppetize for the full stream. To get an idea of what this is like, check out a snippet of the last Twitter chat we had:

We look forward to talking with you on 2012-05-28 18:00 UTC!

Learn More:

Introducing PuppetDB

Announcing the Marionette Collective 2.0

Puppet Labs on Twitter

Introducing PuppetDB: Put Your Data to Work

Nick Lewis — Fri, 18 May 2012 15:06:12 +0000

PuppetDB is the next-generation open source storage service for Puppet-produced data. Today, this includes catalogs and facts, and will be extended in the near future. The initial release provides a drop-in replacement for both storeconfigs and inventory service.

We’ve designed PuppetDB to empower Puppet deployments, and built it from the ground up with performance in mind. It’s built on technologies known for their performance, and is highly parallel, making full use of available resources. It also stores all of its data asynchronously, freeing up the master to go compile more catalogs. Beyond that, we’ve devoted copious time to benchmarking and optimizing the performance.

Why PuppetDB?

The most immediate benefit of PuppetDB is improved performance for storeconfigs users, but even for others, it has a lot to offer. As a centralized store, PuppetDB knows about every node, resource, relationship, and fact across your entire infrastructure. All this information is easily queryable, so you can integrate it into your tools and workflow, or just satisfy your curiosity. It also provides a platform on which powerful new tooling will be built.

And if you’re not using storeconfigs, you should be. At its heart, storeconfigs can be thought of as “higher-order Puppet.” It’s a way for multiple nodes to interact with each other through Puppet, which is an immensely powerful feature. In any case where one node knows what another node is doing, storeconfigs may help.

For instance, storeconfigs can be used to configure a monitoring service, without knowing upfront any of the nodes or services being monitored. Each node to be monitored can simply define what ought to be checked, and those checks can be collected on the node doing the monitoring. Or it can be used to share SSH authorized keys, by having each node export its key, and collect everyone else’s.

Built for performance

Let’s talk about performance. I told you it was a key design goal, but just how much faster is PuppetDB than the existing solution? To find out, I ran an experiment against the old, ActiveRecord storeconfigs implementation.

I compiled and saved a catalog of 650 resources, using an initially empty PostgreSQL database. Compilation took 5.6 seconds. With nothing in the database, it took 53 seconds to store the catalog. That’s brushing right up on the agent’s timeout, risking an outright failure. With the database now primed, I submitted the same catalog a second time, unmodified, which took 4 seconds.

To find out how PuppetDB performs, we have much more information available to us. The service is highly instrumented to keep metrics on every aspect of its performance, all of which is made available over HTTP and JMX.

This is the PuppetDB dashboard, which uses the HTTP metrics API to give an overview of the current state of the system. The dashboard comes built-in, and updates live, even on your mobile device! Taking a look at this screenshot (taken from our internal PuppetDB instance), we can see the backlog of work, how long command processing is taking, how much work has been done, how large the database is, and much more. And yet this is still only a small subset of the metrics we track and make available.

In particular, we see that the queue is empty, meaning PuppetDB is keeping up with demand. Looking at the number of nodes and resources in the population, we can easily calculate that the average size of a catalog is ~670 resources. The average time to process a command is 394ms. This is around 130x faster than the worst case time of old-school storeconfigs, and 10x better than the case where catalogs are already present. We also see that PuppetDB is responding to storeconfigs queries in only 65ms.

Admittedly, these numbers are somewhat incomparable; for instance, the very first catalog stored in PuppetDB may take some extra time, but catalogs which are unchanged will be negligible. But this gives some indication of the improvement we’re talking about. It’s also important to note that all of this storage is asynchronous, freeing up the master to continue serving catalogs. Previously, the master would have been occupied waiting for storeconfigs.

Reliable data store

So we can see that PuppetDB stores your data more quickly, but what about the data itself? After all, that’s what you really care about. PuppetDB makes a few promises about its data: it will be complete, it will be accurate, and it will be current.

Every aspect of the catalog is stored, including edges and unexported resources, which are omitted in old storeconfigs and the popular thin_storeconfigs mode respectively. Nuances of the catalog like resource aliases are also respected, ensuring that every resource and edge is present and accurately represented.

It’s downright difficult to lose your data with PuppetDB. It takes great care not to let that happen, by accepting it into a persistent queue, and trying up to sixteen times (even across service restarts) to handle the command, ensuring that if the data is good, it will make it into the database. And if it somehow still doesn’t make it in, the command will be saved away with plenty of forensic data for later investigation and reprocessing.

In that vein, when configured to use PuppetDB, Puppet will refuse to serve catalogs if PuppetDB is down and the catalog can’t be persisted. This means the data PuppetDB has will always be current; an agent will never use a catalog that PuppetDB doesn’t know about.

And it’s secure. All communication between the puppet master and PuppetDB happens over SSL, authenticated with the same certificates as used for communication between puppet master and agents. Similarly, if PuppetDB and its database are separate, it’s a simple matter to secure their connection.

Plays well with others

PuppetDB is a key component of the Puppet Data Library, and brings that to bear in its query API. Resources, facts, nodes, and metrics can all be queried over HTTP. For resources and nodes, there is a simple query language which can be used to form arbitrarily complex requests. The public API is the same one that Puppet uses to make storeconfigs queries (using the <<||>> operator) of PuppetDB, but provides a superset of the functionality provided by storeconfigs. The API is fully documented and versioned, for use in scripts, Faces, or custom Puppet functions.

PuppetDB is faster, smarter, and has more complete data than ever before. If you’re a current storeconfigs user, there’s no reason not to try it out immediately. If you don’t use storeconfigs (and especially if performance was the reason), now is the time to start. We know that storeconfigs, while being a powerful and important feature, has historically been a pain point for users. One of the goals of PuppetDB is to alleviate that and personally, I want a world in which everyone uses storeconfigs and loves it. PuppetDB offers great power over and insight into your infrastructure, and it’s only going to get bigger and better.

Learn More

Download the open beta of PuppetDB from our APT repository, YUM repository, or GitHub.
Read the PuppetDB documentation.
Open tickets on the PuppetDB project.
Come see Deepak Giridharagopal talk in-depth about the development and design decisions, and ask all your usage questions this Saturday at Puppet Camp Los Angeles.

Removing lint from the Puppet’s belly button of fluffy automation chaos, or: Using puppet-lint to save yourself from style faux pas

Ben Hughes — Mon, 14 May 2012 20:42:22 +0000

Wouldn’t it be nice if you never made a mistake or a typo again? Okay, that’s a slightly misleading offer. How about just never committing such gaffes in with your code? “How?” I hear you cry. With the illustrious Tim Sharpe’s puppet-lint and some version control derring-do!

Following on from Adrien’s wonderful post on syntax checking and writing tests, you have a way of checking the syntax and style guide from the command line. Using this approach, even on a mature (like a fine wine) Puppet code base, you can move towards having beautiful, aligned, ordered manifests—the type of manifests you’d be happy taking home to meet the family.

Puppet-lint takes the Puppet Labs Style Guide and places it nicely in a command line tool. Think of it as a programmatic PEP-8 for those from a Pythonic world. Now, where this comes in handy is when you’re using revision control (which is all the time, obviously!), because you can tie the two together. Yes, every time you try and commit your code to your RCS, it checks the files you’re committing, and makes sure you haven’t used a tab where there should be a space.

In Puppet Labs’ kickass Operations department, we’re ever so slightly keen on Git, not least due to the wonderful GitHub. RCSHub just doesn’t cut it for us these days.

With Git, as with many an RCS, you’re free to define hooks to do weird and wondrous things upon certain actions. This chapter on git hooks from Customizing Git explains them in detail. If you’re using Subversion, then this pre-commit documentation suggests what you can do on the server side to accomplish the same thing.

For this, the pre-commit hook is the one we want to utilise. I first make a bash script to get a list of the files that change in the commit. Then, the hook script goes through them one at a time, seeing if they’re Puppet manifests by name, and running puppet-lint on them as it goes. If any of the manifests fail linting, it exits there and then I may go fix them at my leisure!

#!/bin/bash
# Requires bash, as it uses the [[ ]] syntax.
#
# If it's puppet code, lint it up.
 
# I we don't have puppet-lint, so just exit and leave them be.
which puppet-lint >/dev/null 2>&1 || exit
 
# Variables goes hither
declare -a FILES
IFS="
"
FILES=$(git diff --cached --name-only --diff-filter=ACM )
 
for file in ${FILES[@]}
do
  if [[ $file =~ \.*.pp\$ ]]
  then
    puppet-lint --with-filename "$file"
    RC=$?
    if [ $RC -ne 0 ]
    then
      exit $RC
    fi
  fi
done
 
exit 0

I save that file to .git/hooks/pre-commit in my repository, give it a light sprinkling of ‘chmod +x’, a dash of ‘gem install puppet-lint’ and I am ready to roll.

Exhibit A, the Larch

So let’s see this bad-boy in use, I hear you cry! Let’s take a manifest that breaks all of the rules, and see what happens…

node 'hubert.humphrey.edu' {
 
	user{ "mhunter":
	   managehome => true,
	   home => "/home/mhunter",
	   comment => 'Mark Hunter',
	   ensure => 'present'
   }
 
}

[enlil:puppetlabs-modules]% git commit hubert.pp
hubert.pp - WARNING: double quoted string containing no variables on line 3
hubert.pp - WARNING: double quoted string containing no variables on line 5
hubert.pp - WARNING: => on line isn't properly aligned for resource on line 5
hubert.pp - WARNING: => on line isn't properly aligned for resource on line 6
hubert.pp - WARNING: => on line isn't properly aligned for resource on line 7
hubert.pp - ERROR: two-space soft tabs not used on line 4
hubert.pp - ERROR: two-space soft tabs not used on line 5
hubert.pp - ERROR: two-space soft tabs not used on line 6
hubert.pp - ERROR: two-space soft tabs not used on line 7
hubert.pp - ERROR: two-space soft tabs not used on line 8
hubert.pp - ERROR: trailing whitespace found on line 2
hubert.pp - WARNING: ensure found on line but it's not the first attribute on line 7

Woah, that’s a lot of errors. By default, puppet-lint will let warnings through, but stop dead on errors, and we got them all! I’ll tidy this manifest up…

node 'hubert.humphrey.edu' {
 
  user{ 'mhunter':
    ensure     => present,
    managehome => true,
    home       => '/home/mhunter',
    comment    => 'Mark Hunter',
  }
 
}

Much more readable, and now when I try and commit it, I get:

[enlil:puppetlabs-modules]% git commit hubert.pp
[master 6020331] Add in a new pupil to the school!
 1 file changed, 10 insertions(+)
 create mode 100644 hubert.pp

Clean, and the best part is I don’t have to alter my workflow, I can just carry on editing with Vim, commit my work as normal, and just be kindly reminded when a manifest has things that need tidying up.

Learn More

Announcing The Marionette Collective 2.0

R.I. Pienaar — Wed, 09 May 2012 16:52:58 +0000

I am proud to announce the release of the next major production version of The Marionette Collective (MCollective). This release brings together almost a year’s worth of work and introduces great improvements in security, stability, platform support and new features in the core messaging layer.

The major areas of advance are below:

Complete messaging protocol rewrite to enable direct style connectivity that would allow programs to bypass normal discovery instead using their own data sources
An additional more robust messaging paradigm supporting a more assured addressing and delivery scheme
Batched mode allowing users to address machines in small groups thus avoiding thundering herd and enabling more granular changes
A more complete language for expressing discovery that includes and/or/not style queries across the infrastructure
Improved Stomp connection security using normal industry standard Certificate Authority validated TLS
New connector that uses ActiveMQ-specific features for better performance and scalability
Security of the SSL and AES security plugins have been improved for tamper protection by middle men
A message validity period has been introduced to lower the window of message replay attacks
Better error handling and better logging for Stomp connections
JSON output from the ‘rpc’ application
Ability to pipe RPC requests into each other creating a chain of related RPC calls
Better validations, better error handling and better documentation creation from the DDL
Performance improvements in the CLI, more consistently formatted output of received data
A Ruby GEM of the client is now made available on rubygems.org
The rc script for Debian based systems have been improved to prevent duplicate daemons from running
Built in packager for plugins into native OS packages—RedHat and Debian supported
MS Windows Support

The full release notes for this release expands on key areas of the above list and you can download this release from our download area, Yum repository or our Apt repository.

Upcoming Puppet Events: New Camps, PuppetConf, and Velocity

jose — Tue, 08 May 2012 15:46:24 +0000

Puppet Labs has been a busy place in the last month: we announced PuppetConf and released the very first seats for the Puppet Professional Certification Program (more details coming soon), we also announced our support for OpenStack and we announced PuppetDB, affectionately known around the office as project Greyskull. As we move ahead into May and June we hope to see you at some of the following events:

May 19 – Puppet Camp Los Angeles: Senior Developer Deepak Giridharagopal will introduce PuppetDB, and the Puppet projects team lead Daniel Pittman will share the Puppet roadmap for 2012.

May 21 – Find us at EMC world, we’ll be around and looking to find you. Follow us @puppetlabs to find out where we are.

We’ve announced new Puppet Camps for June and July:

Southeast Asia in Kuala Lumpur on June 5th
Sydney on June 8th (tickets are almost gone!)
District of Columbia on June 19th
Puppet Camp Boston on June 22
Chicago on July 23rd

And don’t forget about our previously announced our European camps in Dublin, Ireland on July 6th and Geneva, Switzerland on July 11th.

After the end of July we’ll be working to produce an epic PuppetConf in San Francisco on September 27th and 28th. Register now to guarantee a seat. Like many of our Puppet Camps, PuppetConf is sure to sell out.

We’re also happy to offer you a 10% discount to Velocity on June 25-28. Just use code PUPPETLABS when you register and then come find us at booth #517 and check out presentations from James Turnbull and Luke Kanies at Velocity.

We’re looking forward to connecting with you this summer.

Announcing Puppet Labs CTO Nigel Kersten

Nigel Kersten — Thu, 26 Apr 2012 16:37:39 +0000

This week I took on the role of CTO at Puppet Labs, and started reflecting on the awesome journey that led me here.

It was 2006, and I was scrambling to make it onto the last bus back to San Francisco from the Apple WWDC Beer Bash down in Cupertino. I’d been to quite a few Beer Bashes and knew the drill: forget the lineup for the campus store, just concentrate on finding beer and the few Apple employees who could fix the OpenDirectory bugs that were making my life hell. Both objectives were completed, leaving me only minutes to avoid having to spend way too much money getting a taxi back up to the city.

Fatefully, I found a seat next to this intensely opinionated sysadmin, Jeff McCune (later to become one of the first pro services guys at Puppet Labs, and now one of our core developers). He recognized me from my WWDC presentation that year and started grilling me about how I ran my university campus, particularly the file-based configuration management system I used, Radmind, and the hacked up framework I’d put in place to try to manage higher level objects than mere files.

He’d been to a talk by Luke Kanies (now the CEO of Puppet Labs) at LISA and was very excited about this guy who had built a tool that worked the way sysadmins actually needed to work with a pragmatic, model-based approach. Even more importantly though, Luke was serious about fostering adoption, and had helped Jeff write some useful extensions for Puppet. Jeff had already gotten religion about idempotent, declarative approaches for sysadmins, and spent pretty much all of the bus ride bending my ear about this new project called “Puppet” and how it was going to change the world of operations.

After WWDC, I flew back to Australia, fully intending to try out this magical Puppet project, but got distracted by the day to day life of running campus IT operations on a shoestring budget for users who were academics and artists.

I was even more distracted a few months later when one of the MacEnterprise community members came out of lurking and told me I should apply for a role at Google in Mountain View. Several of the toughest interviews of my life followed, and within a couple of months, I was moving my young family to the other side of the world to run Mac Operations at Google HQ.

It was clear that tools like Radmind simply weren’t going to work at Google for the many thousands of corporate Macs. Opinionated engineers who demanded a high degree of customization, immense growth, globally distributed offices and a very small team meant that it was completely insane to even think about trying the old methods of file-based config management of the entire system.

We needed a better and more sustainable way, a solution that gave us higher levels of abstraction with meaningful entities such as users, groups, services and packages, and that didn’t require you manage the entire machine.

Jeff and I had kept in contact, and he was presenting on Puppet at WWDC that year. I popped up to San Francisco with some of my coworkers, and made sure we turned up to his talk.

10 minutes into his presentation we were getting pretty excited, and we started experimenting over VPN. By the time Jeff finished his talk, we had a working Puppet master back at Google managing the contents and permissions of a few critical files in /etc, and knew we had a great match.

As it turned out, the Mac deployment was such a rapid success that one of the Linux Ops team started a skunkworks project to manage the internal Linux distro with Puppet, as there had been a few failed CFEngine attempts. This worked so well that Puppet eventually managed all the Google corporate Mac and Linux desktops, laptops and servers.

Puppet was a much younger project in those days. We were building a lot of custom Puppet extensions for Mac OS X that went back into the core, and were having to scale Puppet to manage tens of thousands of nodes, so I spent a lot of time on the mailing lists and IRC channels brainstorming with Luke and the community. I quickly fell in love with the community. It was full of thoughtful sysadmins, people who were frustrated with the unreliable state of operations tools, and knew there was a better way out there than continually reinventing arcane bash/ssh frameworks.

We have some great technology with Puppet, but one of our greatest strengths is our outstanding community.

I ended up at the first ever Puppet Camp, San Francisco, 2009. It was small, but was one of the most exhilarating conferences I’ve ever been to. I love looking back at those photos and seeing how many of that group are now part of the Puppet Labs team. Dan Bode, James Turnbull, Ben Hughes, Gary Larizza, Michael Stahnke, Carl Caum, Deepak Giridharagopal (Little known fact: his last name is actually Tamil for “Grid Computing”).

That’s an awesome group of people to end up working with, let alone all the other great people we have here at Puppet Labs.

I had an amazing couple of years at Google, surrounded by super sharp minds and working on truly interesting operations problems at a scale greater than anything I’d ever touched before, but I was starting to look enviously at friends who left for early stage startups and the breadth of knowledge they were acquiring. Luke had poked me a couple of times about coming to work for him, but I didn’t seriously consider it until late 2010 when he, Teyo and James made a much more concerted effort.

“You’re opinionated about Puppet. Want to put your money where your mouth is?”

One visit to Portland and I knew I wanted to live in this awesome food, beer, and cycling-obsessed city full of people following obscure passions. I jumped ship from Google and we moved north, where I dived headfirst into being responsible for Product at a very quickly growing startup.

It’s been an immense 18 months. We started with our first commercial release, Puppet Enterprise 1.0, and followed that up with several great releases, all solving real problems for real users. We’ve brought on the open source MCollective and Hiera projects from the incomparable RI Pienaar, released Puppet 2.7.0, and grown at an amazing pace. We’ve grown from 2 events a year to 15, including the incredibly successful PuppetConf `11 and are building up to an even bigger PuppetConf this year. Nothing like startup speed to quicken the blood.

From the original 20 odd folks I started with in the tiny office in the seedy and urine-drenched Old Town to our shiny digs in the Pearl, with over 80 employees. From a distinct lack of in-house beverages to decent espresso and delicious local beer. From a company that knew the user experience was critical, to one with a growing UX/Design department headed up by Randall of the impenetrable Gandalf gaze.

I love this company. I love what we’ve done already to change the face of operations, I love the ambition we have to change it even more, and I especially love the people I get to do it with.

I’m thrilled to take on the role of CTO, and to concentrate on fostering our culture of technical innovation so that we continue to build applications and platforms that truly advance the state of IT infrastructure. The world of operations is undergoing radical change right now. The cloud, pervasive virtualization, corporate adoption of FOSS, BYOD, IaaS, PaaS and SaaS are all forcing sysadmins to be truly agile and adaptive. Some of the brightest people in our industry work in operations, and it’s going to be incredible to see what they come up with when IT automation gives them space to concentrate on genuinely important matters.

Join us for PuppetConf 2012

jose — Thu, 19 Apr 2012 15:19:23 +0000

Registration for PuppetConf ’12 is now open. We’re in a new city and a new facility, with new tracks and new programs. Look forward to 5 concurrent tracks over 2 days focusing on all things operations. Our new venue offers one large theater for keynotes, while our second auditorium will be dedicated to Puppet Community presentations and hacking space. We’re introducing a hands-on lab component to the conference, and we’re happy to announce that we’ll be offering the first ever Puppet Admin and Puppet Developer Certification exams at PuppetConf.

With over 70 speakers, 600+ community members, and the Puppet Labs team, PuppetConf is a must-attend event. We’re looking forward to seeing you in San Francisco!

Here’s a quick recap of last year:

All 2011 talks can be viewed on the Puppet Labs YouTube Channel.

Puppet Labs at AWS Cloud Summit, EucaDay NYC, and Puppet Camp NYC

jose — Mon, 16 Apr 2012 18:46:41 +0000

Spring is the time to be in New York. We know it, and apparently so do our partners. Whether you live in or around New York or just happen to be in town visiting over the next two weeks come find us at these awesome events:

Wed 4/18 – Come meet us at the AWS Cloud for Start-Ups & Developers social hosted by Amazon Web Services!
Thur 4/19 – Stop by booth “S2″ at the AWS Cloud Summit to talk to our Puppet experts, or get some free schwag.
Fri 4/20 – Meet Ohad Levy at the PuppetNYC meetup.
Wed 4/25 – Check out our presentation at the inaugural EucaDay conference.

And, last but not least:

Fri 4/27 – Join us for an awesome Puppet Camp NYC featuring Puppet Labs’ own Luke Kanies, James Turnbull, Kelsey Hightower and Eric Shamow!