Puppet Labs » General News

Removing lint from the Puppet’s belly button of fluffy automation chaos, or: Using puppet-lint to save yourself from style faux pas

Ben Hughes — Mon, 14 May 2012 20:42:22 +0000

Wouldn’t it be nice if you never made a mistake or a typo again? Okay, that’s a slightly misleading offer. How about just never committing such gaffes in with your code? “How?” I hear you cry. With the illustrious Tim Sharpe’s puppet-lint and some version control derring-do!

Following on from Adrien’s wonderful post on syntax checking and writing tests, you have a way of checking the syntax and style guide from the command line. Using this approach, even on a mature (like a fine wine) Puppet code base, you can move towards having beautiful, aligned, ordered manifests—the type of manifests you’d be happy taking home to meet the family.

Puppet-lint takes the Puppet Labs Style Guide and places it nicely in a command line tool. Think of it as a programmatic PEP-8 for those from a Pythonic world. Now, where this comes in handy is when you’re using revision control (which is all the time, obviously!), because you can tie the two together. Yes, every time you try and commit your code to your RCS, it checks the files you’re committing, and makes sure you haven’t used a tab where there should be a space.

In Puppet Labs’ kickass Operations department, we’re ever so slightly keen on Git, not least due to the wonderful GitHub. RCSHub just doesn’t cut it for us these days.

With Git, as with many an RCS, you’re free to define hooks to do weird and wondrous things upon certain actions. This chapter on git hooks from Customizing Git explains them in detail. If you’re using Subversion, then this pre-commit documentation suggests what you can do on the server side to accomplish the same thing.

For this, the pre-commit hook is the one we want to utilise. I first make a bash script to get a list of the files that change in the commit. Then, the hook script goes through them one at a time, seeing if they’re Puppet manifests by name, and running puppet-lint on them as it goes. If any of the manifests fail linting, it exits there and then I may go fix them at my leisure!

#!/bin/bash
# Requires bash, as it uses the [[ ]] syntax.
#
# If it's puppet code, lint it up.
 
# I we don't have puppet-lint, so just exit and leave them be.
which puppet-lint >/dev/null 2>&1 || exit
 
# Variables goes hither
declare -a FILES
IFS="
"
FILES=$(git diff --cached --name-only --diff-filter=ACM )
 
for file in ${FILES[@]}
do
  if [[ $file =~ \.*.pp\$ ]]
  then
    puppet-lint --with-filename "$file"
    RC=$?
    if [ $RC -ne 0 ]
    then
      exit $RC
    fi
  fi
done
 
exit 0

I save that file to .git/hooks/pre-commit in my repository, give it a light sprinkling of ‘chmod +x’, a dash of ‘gem install puppet-lint’ and I am ready to roll.

Exhibit A, the Larch

So let’s see this bad-boy in use, I hear you cry! Let’s take a manifest that breaks all of the rules, and see what happens…

node 'hubert.humphrey.edu' {
 
	user{ "mhunter":
	   managehome => true,
	   home => "/home/mhunter",
	   comment => 'Mark Hunter',
	   ensure => 'present'
   }
 
}

[enlil:puppetlabs-modules]% git commit hubert.pp
hubert.pp - WARNING: double quoted string containing no variables on line 3
hubert.pp - WARNING: double quoted string containing no variables on line 5
hubert.pp - WARNING: => on line isn't properly aligned for resource on line 5
hubert.pp - WARNING: => on line isn't properly aligned for resource on line 6
hubert.pp - WARNING: => on line isn't properly aligned for resource on line 7
hubert.pp - ERROR: two-space soft tabs not used on line 4
hubert.pp - ERROR: two-space soft tabs not used on line 5
hubert.pp - ERROR: two-space soft tabs not used on line 6
hubert.pp - ERROR: two-space soft tabs not used on line 7
hubert.pp - ERROR: two-space soft tabs not used on line 8
hubert.pp - ERROR: trailing whitespace found on line 2
hubert.pp - WARNING: ensure found on line but it's not the first attribute on line 7

Woah, that’s a lot of errors. By default, puppet-lint will let warnings through, but stop dead on errors, and we got them all! I’ll tidy this manifest up…

node 'hubert.humphrey.edu' {
 
  user{ 'mhunter':
    ensure     => present,
    managehome => true,
    home       => '/home/mhunter',
    comment    => 'Mark Hunter',
  }
 
}

Much more readable, and now when I try and commit it, I get:

[enlil:puppetlabs-modules]% git commit hubert.pp
[master 6020331] Add in a new pupil to the school!
 1 file changed, 10 insertions(+)
 create mode 100644 hubert.pp

Clean, and the best part is I don’t have to alter my workflow, I can just carry on editing with Vim, commit my work as normal, and just be kindly reminded when a manifest has things that need tidying up.

Learn More

Announcing The Marionette Collective 2.0

R.I. Pienaar — Wed, 09 May 2012 16:52:58 +0000

I am proud to announce the release of the next major production version of The Marionette Collective (MCollective). This release brings together almost a year’s worth of work and introduces great improvements in security, stability, platform support and new features in the core messaging layer.

The major areas of advance are below:

Complete messaging protocol rewrite to enable direct style connectivity that would allow programs to bypass normal discovery instead using their own data sources
An additional more robust messaging paradigm supporting a more assured addressing and delivery scheme
Batched mode allowing users to address machines in small groups thus avoiding thundering herd and enabling more granular changes
A more complete language for expressing discovery that includes and/or/not style queries across the infrastructure
Improved Stomp connection security using normal industry standard Certificate Authority validated TLS
New connector that uses ActiveMQ-specific features for better performance and scalability
Security of the SSL and AES security plugins have been improved for tamper protection by middle men
A message validity period has been introduced to lower the window of message replay attacks
Better error handling and better logging for Stomp connections
JSON output from the ‘rpc’ application
Ability to pipe RPC requests into each other creating a chain of related RPC calls
Better validations, better error handling and better documentation creation from the DDL
Performance improvements in the CLI, more consistently formatted output of received data
A Ruby GEM of the client is now made available on rubygems.org
The rc script for Debian based systems have been improved to prevent duplicate daemons from running
Built in packager for plugins into native OS packages—RedHat and Debian supported
MS Windows Support

The full release notes for this release expands on key areas of the above list and you can download this release from our download area, Yum repository or our Apt repository.

Upcoming Puppet Events: New Camps, PuppetConf, and Velocity

jose — Tue, 08 May 2012 15:46:24 +0000

Puppet Labs has been a busy place in the last month: we announced PuppetConf and released the very first seats for the Puppet Professional Certification Program (more details coming soon), we also announced our support for OpenStack and we announced PuppetDB, affectionately known around the office as project Greyskull. As we move ahead into May and June we hope to see you at some of the following events:

May 19 – Puppet Camp Los Angeles: Senior Developer Deepak Giridharagopal will introduce PuppetDB, and the Puppet projects team lead Daniel Pittman will share the Puppet roadmap for 2012.

May 21 – Find us at EMC world, we’ll be around and looking to find you. Follow us @puppetlabs to find out where we are.

We’ve announced new Puppet Camps for June and July:

Southeast Asia in Kuala Lumpur on June 5th
Sydney on June 8th (tickets are almost gone!)
District of Columbia on June 19th
Puppet Camp Boston on June 22
Chicago on July 23rd

And don’t forget about our previously announced our European camps in Dublin, Ireland on July 6th and Geneva, Switzerland on July 11th.

After the end of July we’ll be working to produce an epic PuppetConf in San Francisco on September 27th and 28th. Register now to guarantee a seat. Like many of our Puppet Camps, PuppetConf is sure to sell out.

We’re also happy to offer you a 10% discount to Velocity on June 25-28. Just use code PUPPETLABS when you register and then come find us at booth #517 and check out presentations from James Turnbull and Luke Kanies at Velocity.

We’re looking forward to connecting with you this summer.

Announcing Puppet Labs CTO Nigel Kersten

Nigel Kersten — Thu, 26 Apr 2012 16:37:39 +0000

This week I took on the role of CTO at Puppet Labs, and started reflecting on the awesome journey that led me here.

It was 2006, and I was scrambling to make it onto the last bus back to San Francisco from the Apple WWDC Beer Bash down in Cupertino. I’d been to quite a few Beer Bashes and knew the drill: forget the lineup for the campus store, just concentrate on finding beer and the few Apple employees who could fix the OpenDirectory bugs that were making my life hell. Both objectives were completed, leaving me only minutes to avoid having to spend way too much money getting a taxi back up to the city.

Fatefully, I found a seat next to this intensely opinionated sysadmin, Jeff McCune (later to become one of the first pro services guys at Puppet Labs, and now one of our core developers). He recognized me from my WWDC presentation that year and started grilling me about how I ran my university campus, particularly the file-based configuration management system I used, Radmind, and the hacked up framework I’d put in place to try to manage higher level objects than mere files.

He’d been to a talk by Luke Kanies (now the CEO of Puppet Labs) at LISA and was very excited about this guy who had built a tool that worked the way sysadmins actually needed to work with a pragmatic, model-based approach. Even more importantly though, Luke was serious about fostering adoption, and had helped Jeff write some useful extensions for Puppet. Jeff had already gotten religion about idempotent, declarative approaches for sysadmins, and spent pretty much all of the bus ride bending my ear about this new project called “Puppet” and how it was going to change the world of operations.

After WWDC, I flew back to Australia, fully intending to try out this magical Puppet project, but got distracted by the day to day life of running campus IT operations on a shoestring budget for users who were academics and artists.

I was even more distracted a few months later when one of the MacEnterprise community members came out of lurking and told me I should apply for a role at Google in Mountain View. Several of the toughest interviews of my life followed, and within a couple of months, I was moving my young family to the other side of the world to run Mac Operations at Google HQ.

It was clear that tools like Radmind simply weren’t going to work at Google for the many thousands of corporate Macs. Opinionated engineers who demanded a high degree of customization, immense growth, globally distributed offices and a very small team meant that it was completely insane to even think about trying the old methods of file-based config management of the entire system.

We needed a better and more sustainable way, a solution that gave us higher levels of abstraction with meaningful entities such as users, groups, services and packages, and that didn’t require you manage the entire machine.

Jeff and I had kept in contact, and he was presenting on Puppet at WWDC that year. I popped up to San Francisco with some of my coworkers, and made sure we turned up to his talk.

10 minutes into his presentation we were getting pretty excited, and we started experimenting over VPN. By the time Jeff finished his talk, we had a working Puppet master back at Google managing the contents and permissions of a few critical files in /etc, and knew we had a great match.

As it turned out, the Mac deployment was such a rapid success that one of the Linux Ops team started a skunkworks project to manage the internal Linux distro with Puppet, as there had been a few failed CFEngine attempts. This worked so well that Puppet eventually managed all the Google corporate Mac and Linux desktops, laptops and servers.

Puppet was a much younger project in those days. We were building a lot of custom Puppet extensions for Mac OS X that went back into the core, and were having to scale Puppet to manage tens of thousands of nodes, so I spent a lot of time on the mailing lists and IRC channels brainstorming with Luke and the community. I quickly fell in love with the community. It was full of thoughtful sysadmins, people who were frustrated with the unreliable state of operations tools, and knew there was a better way out there than continually reinventing arcane bash/ssh frameworks.

We have some great technology with Puppet, but one of our greatest strengths is our outstanding community.

I ended up at the first ever Puppet Camp, San Francisco, 2009. It was small, but was one of the most exhilarating conferences I’ve ever been to. I love looking back at those photos and seeing how many of that group are now part of the Puppet Labs team. Dan Bode, James Turnbull, Ben Hughes, Gary Larizza, Michael Stahnke, Carl Caum, Deepak Giridharagopal (Little known fact: his last name is actually Tamil for “Grid Computing”).

That’s an awesome group of people to end up working with, let alone all the other great people we have here at Puppet Labs.

I had an amazing couple of years at Google, surrounded by super sharp minds and working on truly interesting operations problems at a scale greater than anything I’d ever touched before, but I was starting to look enviously at friends who left for early stage startups and the breadth of knowledge they were acquiring. Luke had poked me a couple of times about coming to work for him, but I didn’t seriously consider it until late 2010 when he, Teyo and James made a much more concerted effort.

“You’re opinionated about Puppet. Want to put your money where your mouth is?”

One visit to Portland and I knew I wanted to live in this awesome food, beer, and cycling-obsessed city full of people following obscure passions. I jumped ship from Google and we moved north, where I dived headfirst into being responsible for Product at a very quickly growing startup.

It’s been an immense 18 months. We started with our first commercial release, Puppet Enterprise 1.0, and followed that up with several great releases, all solving real problems for real users. We’ve brought on the open source MCollective and Hiera projects from the incomparable RI Pienaar, released Puppet 2.7.0, and grown at an amazing pace. We’ve grown from 2 events a year to 15, including the incredibly successful PuppetConf `11 and are building up to an even bigger PuppetConf this year. Nothing like startup speed to quicken the blood.

From the original 20 odd folks I started with in the tiny office in the seedy and urine-drenched Old Town to our shiny digs in the Pearl, with over 80 employees. From a distinct lack of in-house beverages to decent espresso and delicious local beer. From a company that knew the user experience was critical, to one with a growing UX/Design department headed up by Randall of the impenetrable Gandalf gaze.

I love this company. I love what we’ve done already to change the face of operations, I love the ambition we have to change it even more, and I especially love the people I get to do it with.

I’m thrilled to take on the role of CTO, and to concentrate on fostering our culture of technical innovation so that we continue to build applications and platforms that truly advance the state of IT infrastructure. The world of operations is undergoing radical change right now. The cloud, pervasive virtualization, corporate adoption of FOSS, BYOD, IaaS, PaaS and SaaS are all forcing sysadmins to be truly agile and adaptive. Some of the brightest people in our industry work in operations, and it’s going to be incredible to see what they come up with when IT automation gives them space to concentrate on genuinely important matters.

Join us for PuppetConf 2012

jose — Thu, 19 Apr 2012 15:19:23 +0000

Registration for PuppetConf ’12 is now open. We’re in a new city and a new facility, with new tracks and new programs. Look forward to 5 concurrent tracks over 2 days focusing on all things operations. Our new venue offers one large theater for keynotes, while our second auditorium will be dedicated to Puppet Community presentations and hacking space. We’re introducing a hands-on lab component to the conference, and we’re happy to announce that we’ll be offering the first ever Puppet Admin and Puppet Developer Certification exams at PuppetConf.

With over 70 speakers, 600+ community members, and the Puppet Labs team, PuppetConf is a must-attend event. We’re looking forward to seeing you in San Francisco!

Here’s a quick recap of last year:

All 2011 talks can be viewed on the Puppet Labs YouTube Channel.

Puppet Labs at AWS Cloud Summit, EucaDay NYC, and Puppet Camp NYC

jose — Mon, 16 Apr 2012 18:46:41 +0000

Spring is the time to be in New York. We know it, and apparently so do our partners. Whether you live in or around New York or just happen to be in town visiting over the next two weeks come find us at these awesome events:

Wed 4/18 – Come meet us at the AWS Cloud for Start-Ups & Developers social hosted by Amazon Web Services!
Thur 4/19 – Stop by booth “S2″ at the AWS Cloud Summit to talk to our Puppet experts, or get some free schwag.
Fri 4/20 – Meet Ohad Levy at the PuppetNYC meetup.
Wed 4/25 – Check out our presentation at the inaugural EucaDay conference.

And, last but not least:

Fri 4/27 – Join us for an awesome Puppet Camp NYC featuring Puppet Labs’ own Luke Kanies, James Turnbull, Kelsey Hightower and Eric Shamow!

Puppet Labs is Sustainability at Work Gold Certified

Nandini Mitra — Fri, 06 Apr 2012 15:56:56 +0000

Puppet Labs has been on a roll…and this time it is not our user community, but the physical one we live in that is giving rave reviews! The City of Portland’s Sustainability at Work Program recently announced Puppet Labs as a Gold Certification winner!

It all began with the culture here…they care about their community and take immense pride in doing their bit for the environment. Most of the Puppet Labs employees (80% to be precise) come to work on bikes, public transport, or on foot. They even have an in-house bike rack in their dining/meeting area!

The company is housed in a LEED Gold building, and the employees adopt energy efficient measures in their daily lives. “Daily life” includes bringing dogs to work, eating cupcakes and drinking from the kegorator at weekly meetings, and recycling most of what they throw into the bins. Even the copious coffee habit comes with a green benefit, with buckets of grounds being saved for employees’ gardens. In keeping with the relaxed yet results-driven work environment, I saw people coming up with ways to make the company greener—not because they were looking for an award, but because caring about the environment comes naturally to them.

I should introduce myself: I’m an MBA student from the University of Oregon, and I helped document all the environmentally friendly activities going on at Puppet Labs. I reported the activities to Sustainability at Work in the areas of Energy (e.g. the T8 lighting, the energy star equipment), Water (e.g. we have tap water not bottled water, we have water-saving faucets), Materials and Waste (we do cardboard, glass recycling, we avoid printing out documents) and Transportation. Beyond activities in these four areas, there’s also an internal portal with postings about various volunteering opportunities, sustainable transportation options, green living etc. Once I gathered all the information, we used the Sustainability at Work calculator to report our work to City of Portland. There was another round of reporting where we filled in another checklist of actions. The final round had a City of Portland expert come over for an on-site verification process. Finally, Puppet Labs received the highest possible certification, GOLD!

Puppet Labs is one of the 7 businesses in Portland to be gold certified, and the only software company. This certification holds good for the next three years. The City of Portland’s website has already featured Puppet Labs as a Gold Winner. Mayor Sam Adams (those of you who visited Portland for PuppetConf last year may remember him speaking) had the following statement:

“Congratulations to Puppet Labs, Inc. on becoming Sustainability at Work Gold Certified. We appreciate Puppet Labs, Inc.’s leadership in taking concrete actions to make Portland a better place to live and work. We hope that Puppet Labs, Inc.’s achievement inspires other businesses to make innovative changes that improve profitability and sustainability.”

This is a great achievement for Puppet Labs, and we feel proud to be recognized as a responsible environmental and community steward.

Join the Conversation: #Puppetize on Twitter

michelle — Tue, 03 Apr 2012 01:29:57 +0000

Join us in our first Puppet Labs-hosted Twitter chat this Wednesday at 11 am PT. There are three simple steps to participating:

Be on Twitter between 11 am and 12 pm PT on April 4th
Track the hashtag #puppetize
Jump in the conversation and tag your tweets with #puppetize

Our product manager Nigel Kersten, the team lead for our Windows development team Josh Cooper, and community manager Mike Stahnke will be on the @puppetlabs handle, ready to answer your questions, take your suggestions, and host the conversation. They’ll start with questions about Puppet Enterprise 2.5, but are open to talking about a number of Puppet-y topics (configuration management in general, best music for module-writing, etc).

That’s it! If you’re following #puppetize you can catch all sides of the conversation. Looking forward to chatting with you on April 4th.

Additional Resources

Follow @puppetlabs on Twitter
Read up on what’s new in Puppet Enterprise 2.5
Catch our introductory PE 2.5 webinar at 11 am PT on Tuesday, April 3rd or 4 pm PT on April 5.

Three Puppet Camps in Europe on the way

jose — Wed, 29 Feb 2012 09:21:54 +0000

We’re happy to announce three European Puppet Camps:

Edinburgh, Scotland March 23 (only 20 tickets left!)

Stockholm, Sweden March 28 (20 tickets left!)

Amsterdam, Netherlands April 2

Puppet Camps are 1-day community events for local audiences. Typically we’ll feature 3-5 local speakers along with a Puppet Labs Engineer. Schedules for all three Puppet Camps are available, but may change as more talks are submitted.

Teyo Tyree, the VP of Business Development and co-founder of Puppet Labs, will be at all three events. RI Pienaar, the creator of MCollective (the backbone of Puppet Enterprises Live Management feature), will also be at Edinburgh. Core contributors to the Puppet project and interesting users are participating in all of the camps.

Puppet Camp Edinburgh is hosted in conjunction with FLOSS UK’s spring conference. Puppet Camp Amsterdam is being co-hosted by Cloudera and (Puppet Labs’ local partner) AmazicSource. We’re in the process of expanding Puppet Camps from a bi-annual format to a roadshow appearing frequently throughout the year.

This year we are planning to hit New York, DC, Chicago, and either Melbourne or Sydney, but we’re also having conversations with organizers in Berlin and Geneva. If you think your city has enough interest in Puppet to support a 100+ person event, shoot us an email.

Our end user conference PuppetConf will be held at the Mission Bay Conference Center in San Francisco, CA on September 27th and 28th. Look for more details on PuppetConf in the next few weeks, along with a call for participation.

Introducing the Puppet Labs Community Manager

Mike Stahnke — Fri, 24 Feb 2012 19:22:25 +0000

Hello, I’m Michael Stahnke and I’m the new community manager at Puppet Labs. I’ll provide a little bit of background about myself, but then I want to jump into the important stuff around the Puppet community.

I joined Puppet Labs in April of 2011 as the first Release Manager. I was a member of the community and using Puppet since late 2006. I learned about Puppet from being a member of the Fedora Project’s Infrastructure group. I also was one of the people helping get EPEL launched (Extra Packages for Enterprise Linux) very early on. I still contribute regularly to Fedora/EPEL, and of course to Puppet, Facter, Dashboard, Hiera and Mcollective. I also love Infrastructure Operations.

There’s obviously a ton I could write about in regards to community. The community around Puppet is awesome. It’s certainly a key reason I wanted to use Puppet, learn about it, learn Ruby, and join Puppet Labs. Today I’d like to focus on what my goals for the community are, and address some areas where we’ve fallen short in the past.

My main goal for the Puppet Community shouldn’t be any different than the goals of the Puppet project:

I want infrastructure problems to go away.

I want quality system administrators to have more time to explore their domain, and make their infrastructure services an investment rather than a cost center. I want infrastructure engineers to spend time on differentiating activities rather than having every shop reinvent time synchronization, DNS management, and authentication setups.

One of the real reasons I liked Puppet when I got started with it was the community. There were smart people here. I don’t just mean people good at Puppet, but people who solved difficult systems management problems. The people in the community discussed solutions for provisioning, patching, packaging, deployment, compliance, auditing, disaster recovery and everything else that was consuming my day as an infrastructure admin. So the Puppet community centered (and centers) around people who were (and are) awesome at infrastructure. I’m very interested in continuing to foster those types of discussions.

Beyond that, I have some more concrete goals around community and Puppet.

I’d like to enable the community to host Puppet User Groups or meet-ups. This means developing a formula for what works, a budget process, and hopefully the output is something like a “user-group-in-a-box” kit.

I want to define and recognize contributors. We spend a lot of time focused on contributors providing us with code. We love you, please keep doing it. We also want contributors to be recognized for things like documentation, filing excellent bug reports, leading/attending user groups, uploading modules to the forge, integrating Puppet with other tools, etc.

In the current event bucket, we have the process around Google Summer of Code 2012. We participated in GSOC in 2010, and liked it a lot. This year we’d like to participate again. This means we need awesome ideas for projects. We’ve started an idea page at http://projects.puppetlabs.com/projects/puppet/wiki/GSOC12, please contribute if you have any ideas for GSOC 2012.

In sitting down to identify gaps in the community, or friction points, a few items bubbled up quickly. So, we’re going to address them as soon as we can.

It is difficult to find material about the Open Source projects on our main website.
We don’t do a super job recognizing members of the community.
We don’t capture a lot of information about what you, the community, are doing. This type of input can make our product and documentation much better.

In the near future, look for some new content on our website, around community and the open source ecosystem.

I have several ideas on what we’ll do to fix these gaps. Some of this will require some patience and experimentation to get right as well. If you have comments and suggestions around the Puppet community, I’m available via Twitter (@stahnma), Freenode (stahnma), and email (stahnma@puppetlabs.com).

Look for a lot more to come in the future around community goals and involvement.

Additional Resources

Contribute a Google Summer of Code project idea
Join a user list: Puppet users and Puppet Enterprise users
The Puppet Labs projects page
The Puppet Module Forge
The Puppet Labs GitHub