Wouldn’t it be nice if you never made a mistake or a typo again? Okay, that’s a slightly misleading offer. How about just never committing such gaffes in with your code? “How?” I hear you cry. With the illustrious Tim Sharpe’s puppet-lint and some version control derring-do!

Following on from Adrien’s wonderful post on syntax checking and writing tests, you have a way of checking the syntax and style guide from the command line. Using this approach, even on a mature (like a fine wine) Puppet code base, you can move towards having beautiful, aligned, ordered manifests—the type of manifests you’d be happy taking home to meet the family.

Puppet-lint takes the Puppet Labs Style Guide and places it nicely in a command line tool. Think of it as a programmatic PEP-8 for those from a Pythonic world. Now, where this comes in handy is when you’re using revision control (which is all the time, obviously!), because you can tie the two together. Yes, every time you try and commit your code to your RCS, it checks the files you’re committing, and makes sure you haven’t used a tab where there should be a space.

In Puppet Labs’ kickass Operations department, we’re ever so slightly keen on Git, not least due to the wonderful GitHub. RCSHub just doesn’t cut it for us these days.

With Git, as with many an RCS, you’re free to define hooks to do weird and wondrous things upon certain actions. This chapter on git hooks from Customizing Git explains them in detail. If you’re using Subversion, then this pre-commit documentation suggests what you can do on the server side to accomplish the same thing.

For this, the pre-commit hook is the one we want to utilise. I first make a bash script to get a list of the files that change in the commit. Then, the hook script goes through them one at a time, seeing if they’re Puppet manifests by name, and running puppet-lint on them as it goes. If any of the manifests fail linting, it exits there and then I may go fix them at my leisure!

#!/bin/bash
# Requires bash, as it uses the [[ ]] syntax.
#
# If it's puppet code, lint it up.
 
# I we don't have puppet-lint, so just exit and leave them be.
which puppet-lint >/dev/null 2>&1 || exit
 
# Variables goes hither
declare -a FILES
IFS="
"
FILES=$(git diff --cached --name-only --diff-filter=ACM )
 
for file in ${FILES[@]}
do
  if [[ $file =~ \.*.pp\$ ]]
  then
    puppet-lint --with-filename "$file"
    RC=$?
    if [ $RC -ne 0 ]
    then
      exit $RC
    fi
  fi
done
 
exit 0

I save that file to .git/hooks/pre-commit in my repository, give it a light sprinkling of ‘chmod +x’, a dash of ‘gem install puppet-lint’ and I am ready to roll.

Exhibit A, the Larch

So let’s see this bad-boy in use, I hear you cry! Let’s take a manifest that breaks all of the rules, and see what happens…

node 'hubert.humphrey.edu' {
 
	user{ "mhunter":
	   managehome => true,
	   home => "/home/mhunter",
	   comment => 'Mark Hunter',
	   ensure => 'present'
   }
 
}

[enlil:puppetlabs-modules]% git commit hubert.pp
hubert.pp - WARNING: double quoted string containing no variables on line 3
hubert.pp - WARNING: double quoted string containing no variables on line 5
hubert.pp - WARNING: => on line isn't properly aligned for resource on line 5
hubert.pp - WARNING: => on line isn't properly aligned for resource on line 6
hubert.pp - WARNING: => on line isn't properly aligned for resource on line 7
hubert.pp - ERROR: two-space soft tabs not used on line 4
hubert.pp - ERROR: two-space soft tabs not used on line 5
hubert.pp - ERROR: two-space soft tabs not used on line 6
hubert.pp - ERROR: two-space soft tabs not used on line 7
hubert.pp - ERROR: two-space soft tabs not used on line 8
hubert.pp - ERROR: trailing whitespace found on line 2
hubert.pp - WARNING: ensure found on line but it's not the first attribute on line 7

Woah, that’s a lot of errors. By default, puppet-lint will let warnings through, but stop dead on errors, and we got them all! I’ll tidy this manifest up…

node 'hubert.humphrey.edu' {
 
  user{ 'mhunter':
    ensure     => present,
    managehome => true,
    home       => '/home/mhunter',
    comment    => 'Mark Hunter',
  }
 
}

Much more readable, and now when I try and commit it, I get:

[enlil:puppetlabs-modules]% git commit hubert.pp
[master 6020331] Add in a new pupil to the school!
 1 file changed, 10 insertions(+)
 create mode 100644 hubert.pp

Clean, and the best part is I don’t have to alter my workflow, I can just carry on editing with Vim, commit my work as normal, and just be kindly reminded when a manifest has things that need tidying up.

Learn More

• Puppet Labs style guide
• puppet-lint
• Verifying Puppet: Checking Syntax and Writing Automated Tests
• Puppet-lint SubVersion commit hook by Jason Hancock

I am proud to announce the release of the next major production version of The Marionette Collective (MCollective). This release brings together almost a year’s worth of work and introduces great improvements in security, stability, platform support and new features in the core messaging layer.

The major areas of advance are below:

• Complete messaging protocol rewrite to enable direct style connectivity that would allow programs to bypass normal discovery instead using their own data sources
• An additional more robust messaging paradigm supporting a more assured addressing and delivery scheme
• Batched mode allowing users to address machines in small groups thus avoiding thundering herd and enabling more granular changes
• A more complete language for expressing discovery that includes and/or/not style queries across the infrastructure
• Improved Stomp connection security using normal industry standard Certificate Authority validated TLS
• New connector that uses ActiveMQ-specific features for better performance and scalability
• Security of the SSL and AES security plugins have been improved for tamper protection by middle men
• A message validity period has been introduced to lower the window of message replay attacks
• Better error handling and better logging for Stomp connections
• JSON output from the ‘rpc’ application
• Ability to pipe RPC requests into each other creating a chain of related RPC calls
• Better validations, better error handling and better documentation creation from the DDL
• Performance improvements in the CLI, more consistently formatted output of received data
• A Ruby GEM of the client is now made available on rubygems.org
• The rc script for Debian based systems have been improved to prevent duplicate daemons from running
• Built in packager for plugins into native OS packages—RedHat and Debian supported
• MS Windows Support

The full release notes for this release expands on key areas of the above list and you can download this release from our download area, Yum repository or our Apt repository.

Puppet Labs has been a busy place in the last month: we announced PuppetConf and released the very first seats for the Puppet Professional Certification Program (more details coming soon), we also announced our support for OpenStack and we announced PuppetDB, affectionately known around the office as project Greyskull. As we move ahead into May and June we hope to see you at some of the following events:

May 19 – Puppet Camp Los Angeles: Senior Developer Deepak Giridharagopal will introduce PuppetDB, and the Puppet projects team lead Daniel Pittman will share the Puppet roadmap for 2012.

May 21 – Find us at EMC world, we’ll be around and looking to find you. Follow us @puppetlabs to find out where we are.

We’ve announced new Puppet Camps for June and July:

• Southeast Asia in Kuala Lumpur on June 5th
• Sydney on June 8th (tickets are almost gone!)
• District of Columbia on June 19th
• Puppet Camp Boston on June 22
• Chicago on July 23rd

And don’t forget about our previously announced our European camps in Dublin, Ireland on July 6th and Geneva, Switzerland on July 11th.

After the end of July we’ll be working to produce an epic PuppetConf in San Francisco on September 27th and 28th. Register now to guarantee a seat. Like many of our Puppet Camps, PuppetConf is sure to sell out.

We’re also happy to offer you a 10% discount to Velocity on June 25-28. Just use code PUPPETLABS when you register and then come find us at booth #517 and check out presentations from James Turnbull and Luke Kanies at Velocity.



We’re looking forward to connecting with you this summer.

This week I took on the role of CTO at Puppet Labs, and started reflecting on the awesome journey that led me here.

It was 2006, and I was scrambling to make it onto the last bus back to San Francisco from the Apple WWDC Beer Bash down in Cupertino. I’d been to quite a few Beer Bashes and knew the drill: forget the lineup for the campus store, just concentrate on finding beer and the few Apple employees who could fix the OpenDirectory bugs that were making my life hell. Both objectives were completed, leaving me only minutes to avoid having to spend way too much money getting a taxi back up to the city.

Fatefully, I found a seat next to this intensely opinionated sysadmin, Jeff McCune (later to become one of the first pro services guys at Puppet Labs, and now one of our core developers). He recognized me from my WWDC presentation that year and started grilling me about how I ran my university campus, particularly the file-based configuration management system I used, Radmind, and the hacked up framework I’d put in place to try to manage higher level objects than mere files.

He’d been to a talk by Luke Kanies (now the CEO of Puppet Labs) at LISA and was very excited about this guy who had built a tool that worked the way sysadmins actually needed to work with a pragmatic, model-based approach. Even more importantly though, Luke was serious about fostering adoption, and had helped Jeff write some useful extensions for Puppet. Jeff had already gotten religion about idempotent, declarative approaches for sysadmins, and spent pretty much all of the bus ride bending my ear about this new project called “Puppet” and how it was going to change the world of operations.

After WWDC, I flew back to Australia, fully intending to try out this magical Puppet project, but got distracted by the day to day life of running campus IT operations on a shoestring budget for users who were academics and artists.

I was even more distracted a few months later when one of the MacEnterprise community members came out of lurking and told me I should apply for a role at Google in Mountain View. Several of the toughest interviews of my life followed, and within a couple of months, I was moving my young family to the other side of the world to run Mac Operations at Google HQ.

It was clear that tools like Radmind simply weren’t going to work at Google for the many thousands of corporate Macs. Opinionated engineers who demanded a high degree of customization, immense growth, globally distributed offices and a very small team meant that it was completely insane to even think about trying the old methods of file-based config management of the entire system.

We needed a better and more sustainable way, a solution that gave us higher levels of abstraction with meaningful entities such as users, groups, services and packages, and that didn’t require you manage the entire machine.

Jeff and I had kept in contact, and he was presenting on Puppet at WWDC that year. I popped up to San Francisco with some of my coworkers, and made sure we turned up to his talk.

10 minutes into his presentation we were getting pretty excited, and we started experimenting over VPN. By the time Jeff finished his talk, we had a working Puppet master back at Google managing the contents and permissions of a few critical files in /etc, and knew we had a great match.

As it turned out, the Mac deployment was such a rapid success that one of the Linux Ops team started a skunkworks project to manage the internal Linux distro with Puppet, as there had been a few failed CFEngine attempts. This worked so well that Puppet eventually managed all the Google corporate Mac and Linux desktops, laptops and servers.

Puppet was a much younger project in those days. We were building a lot of custom Puppet extensions for Mac OS X that went back into the core, and were having to scale Puppet to manage tens of thousands of nodes, so I spent a lot of time on the mailing lists and IRC channels brainstorming with Luke and the community. I quickly fell in love with the community. It was full of thoughtful sysadmins, people who were frustrated with the unreliable state of operations tools, and knew there was a better way out there than continually reinventing arcane bash/ssh frameworks.

We have some great technology with Puppet, but one of our greatest strengths is our outstanding community.

I ended up at the first ever Puppet Camp, San Francisco, 2009. It was small, but was one of the most exhilarating conferences I’ve ever been to. I love looking back at those photos and seeing how many of that group are now part of the Puppet Labs team. Dan Bode, James Turnbull, Ben Hughes, Gary Larizza, Michael Stahnke, Carl Caum, Deepak Giridharagopal (Little known fact: his last name is actually Tamil for “Grid Computing”).

That’s an awesome group of people to end up working with, let alone all the other great people we have here at Puppet Labs.

I had an amazing couple of years at Google, surrounded by super sharp minds and working on truly interesting operations problems at a scale greater than anything I’d ever touched before, but I was starting to look enviously at friends who left for early stage startups and the breadth of knowledge they were acquiring. Luke had poked me a couple of times about coming to work for him, but I didn’t seriously consider it until late 2010 when he, Teyo and James made a much more concerted effort.

“You’re opinionated about Puppet. Want to put your money where your mouth is?”

One visit to Portland and I knew I wanted to live in this awesome food, beer, and cycling-obsessed city full of people following obscure passions. I jumped ship from Google and we moved north, where I dived headfirst into being responsible for Product at a very quickly growing startup.

It’s been an immense 18 months. We started with our first commercial release, Puppet Enterprise 1.0, and followed that up with several great releases, all solving real problems for real users. We’ve brought on the open source MCollective and Hiera projects from the incomparable RI Pienaar, released Puppet 2.7.0, and grown at an amazing pace. We’ve grown from 2 events a year to 15, including the incredibly successful PuppetConf `11 and are building up to an even bigger PuppetConf this year. Nothing like startup speed to quicken the blood.

From the original 20 odd folks I started with in the tiny office in the seedy and urine-drenched Old Town to our shiny digs in the Pearl, with over 80 employees. From a distinct lack of in-house beverages to decent espresso and delicious local beer. From a company that knew the user experience was critical, to one with a growing UX/Design department headed up by Randall of the impenetrable Gandalf gaze.

I love this company. I love what we’ve done already to change the face of operations, I love the ambition we have to change it even more, and I especially love the people I get to do it with.

I’m thrilled to take on the role of CTO, and to concentrate on fostering our culture of technical innovation so that we continue to build applications and platforms that truly advance the state of IT infrastructure. The world of operations is undergoing radical change right now. The cloud, pervasive virtualization, corporate adoption of FOSS, BYOD, IaaS, PaaS and SaaS are all forcing sysadmins to be truly agile and adaptive. Some of the brightest people in our industry work in operations, and it’s going to be incredible to see what they come up with when IT automation gives them space to concentrate on genuinely important matters.

Registration for PuppetConf ’12 is now open. We’re in a new city and a new facility, with new tracks and new programs. Look forward to 5 concurrent tracks over 2 days focusing on all things operations. Our new venue offers one large theater for keynotes, while our second auditorium will be dedicated to Puppet Community presentations and hacking space. We’re introducing a hands-on lab component to the conference, and we’re happy to announce that we’ll be offering the first ever Puppet Admin and Puppet Developer Certification exams at PuppetConf.

With over 70 speakers, 600+ community members, and the Puppet Labs team, PuppetConf is a must-attend event. We’re looking forward to seeing you in San Francisco!

Here’s a quick recap of last year:

All 2011 talks can be viewed on the Puppet Labs YouTube Channel.

Spring is the time to be in New York. We know it, and apparently so do our partners. Whether you live in or around New York or just happen to be in town visiting over the next two weeks come find us at these awesome events:

• Wed 4/18 – Come meet us at the AWS Cloud for Start-Ups & Developers social hosted by Amazon Web Services!
• Thur 4/19 – Stop by booth “S2″ at the AWS Cloud Summit to talk to our Puppet experts, or get some free schwag.
• Fri 4/20 – Meet Ohad Levy at the PuppetNYC meetup.
• Wed 4/25 – Check out our presentation at the inaugural EucaDay conference.

And, last but not least:

• Fri 4/27 – Join us for an awesome Puppet Camp NYC featuring Puppet Labs’ own Luke Kanies, James Turnbull, Kelsey Hightower and Eric Shamow!

Puppet Labs has been on a roll…and this time it is not our user community, but the physical one we live in that is giving rave reviews! The City of Portland’s Sustainability at Work Program recently announced Puppet Labs as a Gold Certification winner!

It all began with the culture here…they care about their community and take immense pride in doing their bit for the environment. Most of the Puppet Labs employees (80% to be precise) come to work on bikes, public transport, or on foot. They even have an in-house bike rack in their dining/meeting area!

The company is housed in a LEED Gold building, and the employees adopt energy efficient measures in their daily lives. “Daily life” includes bringing dogs to work, eating cupcakes and drinking from the kegorator at weekly meetings, and recycling most of what they throw into the bins. Even the copious coffee habit comes with a green benefit, with buckets of grounds being saved for employees’ gardens. In keeping with the relaxed yet results-driven work environment, I saw people coming up with ways to make the company greener—not because they were looking for an award, but because caring about the environment comes naturally to them.

I should introduce myself: I’m an MBA student from the University of Oregon, and I helped document all the environmentally friendly activities going on at Puppet Labs. I reported the activities to Sustainability at Work in the areas of Energy (e.g. the T8 lighting, the energy star equipment), Water (e.g. we have tap water not bottled water, we have water-saving faucets), Materials and Waste (we do cardboard, glass recycling, we avoid printing out documents) and Transportation. Beyond activities in these four areas, there’s also an internal portal with postings about various volunteering opportunities, sustainable transportation options, green living etc. Once I gathered all the information, we used the Sustainability at Work calculator to report our work to City of Portland. There was another round of reporting where we filled in another checklist of actions. The final round had a City of Portland expert come over for an on-site verification process. Finally, Puppet Labs received the highest possible certification, GOLD!

Puppet Labs is one of the 7 businesses in Portland to be gold certified, and the only software company. This certification holds good for the next three years. The City of Portland’s website has already featured Puppet Labs as a Gold Winner. Mayor Sam Adams (those of you who visited Portland for PuppetConf last year may remember him speaking) had the following statement:

“Congratulations to Puppet Labs, Inc. on becoming Sustainability at Work Gold Certified. We appreciate Puppet Labs, Inc.’s leadership in taking concrete actions to make Portland a better place to live and work. We hope that Puppet Labs, Inc.’s achievement inspires other businesses to make innovative changes that improve profitability and sustainability.”

This is a great achievement for Puppet Labs, and we feel proud to be recognized as a responsible environmental and community steward.

Join us in our first Puppet Labs-hosted Twitter chat this Wednesday at 11 am PT. There are three simple steps to participating:

1. Be on Twitter between 11 am and 12 pm PT on April 4th
2. Track the hashtag #puppetize
3. Jump in the conversation and tag your tweets with #puppetize

Our product manager Nigel Kersten, the team lead for our Windows development team Josh Cooper, and community manager Mike Stahnke will be on the @puppetlabs handle, ready to answer your questions, take your suggestions, and host the conversation. They’ll start with questions about Puppet Enterprise 2.5, but are open to talking about a number of Puppet-y topics (configuration management in general, best music for module-writing, etc).

That’s it! If you’re following #puppetize you can catch all sides of the conversation. Looking forward to chatting with you on April 4th.

Additional Resources

• Follow @puppetlabs on Twitter
• Read up on what’s new in Puppet Enterprise 2.5
• Catch our introductory PE 2.5 webinar at 11 am PT on Tuesday, April 3rd or 4 pm PT on April 5.

One of the criticisms of Puppet we see from time to time is that it is not that great at editing and managing files. While our goal for Puppet is for it to be a management platform you can use to build a solution to any problem, it’s true that it’s not as good at just editing and generating files as it could be. What’s not necessarily obvious as that we intentionally made a trade-off between being good at managing everything, or just being good at managing files. This article will hopefully explain why we made that trade-off, and how it helps you get more done with less pain.

Puppet was built originally as an automation tool for primarily UNIX-like machines, such as Solaris, Red Hat Linux, and Mac OS X, and one of the great things about UNIX is that nearly everything can be treated as a file. Thus, it would have made sense in a way for most of Puppet’s functionality to focus on managing files. In fact, that was the major focus of most of the tools around when Puppet was started—files, templates, and managing large sets of files across many machines.

However, while Puppet is reasonably good at file management, including a decent templating system, it does everything it can to force you to think less about files and more about abstract concepts like users, packages, and services. This can often feel awkward if you are approaching Puppet thinking that you really just need to manage files, and it doesn’t want you to. Why doesn’t it just manage files?

It all goes back to frustration I had with these file-centric tools when I was a sysadmin and then a consultant. Yes, nearly everything that the UNIX kernel cares about can be treated like a file—files, pipes, sockets, etc.—but there are a lot of things above the layer of the kernel that aren’t treated as a file. For instance, where can I remove the file corresponding to a user? Or edit the file corresponding to a running service? Yes, this information is in files somewhere, but there isn’t a file-per-user or file-per-filesystem mapping, which makes life very complicated.

Just look at users for example. This starts out easy and gets very painful very quickly. You might start by having the same file on all of your Solaris machines, but what happens when your database servers need accounts for your DBAs but your web servers don’t? Even worse, what happens when you need the same DBA accounts on both your Solaris and Red Hat machines? “Ah hah!”, you think, “that’s why I use templates!”

Yes, templates can be used to make a lot of this pain go away. Or rather, using templates to make your file management system not as painful absolutely does work. But why not pick a management methodology actually built for your problems, as opposed to one built because it was easy for its developer to build? This becomes especially apparent once you have to support Mac OS X. It’s POSIX-compliant (at least, as far as I know), but all of its user tables are stored in some kind of directory service (with the specifics varying by version). You can absolutely build a template that can generate NetInfo tables for OS X and passwd files for Debian, but you can’t do it and stay sane.

It’s not just the file contents that are the problem. File locations are also a big issue. Everyone runs SSH, and basically everyone is running the standard OpenSSH tool. However, nearly everyone names the service differently (sshd? opensshd? ssh?), and puts the files in different locations (/etc/ssh, /etc/openssh, /opt/local/etc/sshd). Again, you can absolutely build a file templating system that provides the flexibility to pick the right location for a given platform (although shockingly few templating systems actually do this—most stick to real filesystem paths, and thus only work on one platform), but you can’t do it and stay sane.

But these are still the easy problems, the things you can reasonably manage with files. The hard stuff—or really, the fun stuff—all gets done with shell scripts. Oh, you might be wrapping those shell scripts in perl or ruby scripts, but in the end, you’re exec-ing a bunch of commands to do your work. You could even go so far as to say what Puppet is really best at is generating and running tiny shell scripts, since the shell is where the rubber really meets the road on UNIX. (It’s obviously different on Windows, but it’s not as different as you’d like it to be.) These little shell scripts are where the file management tools really fall down.

What do you need scripts for? Well, nearly everything, actually. In my experience, the vast majority of work follows some portion of the pattern ‘install package, configure service, run service’. You might only need the configuration part, or the installation part, but you often need all three. At best, files are okay at configuration, but they’re horrible at running services or installing packages. The only real way to do this is using the system commands for, well, managing packages and services.

See, that’s the other thing that the file management tools get wrong—they think we’re still in 1991, before apt, SMF, Yum, RPM, launchd, NetInfo, NFS, and all of the other great tools that make our operating systems modern and awesome, and coincidentally less and less like files. You can either manage your system like a bunch of files, or you can use these modern tools to make your life easier and stay sane; you can’t do both.

So, that’s why Puppet’s not a file management tool. We wanted to leverage all of these great tools your operating system vendors are building, and we wanted you to be able to focus on the problems—running services, deploying software, providing access to users—rather than the implementation, which may or may not be files. This can be confusing to people who really just see their OS as a bunch of files, but if you just want the problem to go away, and you want the solution to actually work with your operating system instead of against it, we think you’ll find Puppet a much simpler and easier solution than file management.

Additional Resources

• Inside Puppet: About Determinism
• Looking Forward to 2012: Design, Big Data in the Infrastructure, and DevOps
• Video: You Should be Iron Man, Not a Robot

We’re happy to announce three European Puppet Camps:

Edinburgh, Scotland March 23 (only 20 tickets left!)

Register

Stockholm, Sweden March 28 (20 tickets left!)

Register

Amsterdam, Netherlands April 2

Register

Puppet Camps are 1-day community events for local audiences. Typically we’ll feature 3-5 local speakers along with a Puppet Labs Engineer. Schedules for all three Puppet Camps are available, but may change as more talks are submitted.

Teyo Tyree, the VP of Business Development and co-founder of Puppet Labs, will be at all three events. RI Pienaar, the creator of MCollective (the backbone of Puppet Enterprises Live Management feature), will also be at Edinburgh. Core contributors to the Puppet project and interesting users are participating in all of the camps.

Puppet Camp Edinburgh is hosted in conjunction with FLOSS UK’s spring conference. Puppet Camp Amsterdam is being co-hosted by Cloudera and (Puppet Labs’ local partner) AmazicSource. We’re in the process of expanding Puppet Camps from a bi-annual format to a roadshow appearing frequently throughout the year.

This year we are planning to hit New York, DC, Chicago, and either Melbourne or Sydney, but we’re also having conversations with organizers in Berlin and Geneva. If you think your city has enough interest in Puppet to support a 100+ person event, shoot us an email.

Our end user conference PuppetConf will be held at the Mission Bay Conference Center in San Francisco, CA on September 27th and 28th. Look for more details on PuppetConf in the next few weeks, along with a call for participation.