At PuppetConf 2012 this past September, Puppet Labs launched its Puppet Certification Program. The first round of certification exams at PuppetConf was highly successful, with more than a hundred participants taking the challenging exams.

Today, we want to recognize the very first person to become Puppet certified: Steve Nielsen of comScore. We were able to get on the phone with Steve and ask him a few questions about his experience in taking — and passing — the Puppet Professional Certification exam.

Q: Steve, how did you get started with Puppet?
A: I manage the open source engineering group at comScore, and my team has been using Puppet since September 2010. We were originally using CFEngine for about seven-to-eight years. Over the course of time, it became increasingly difficult to make changes. I would change one thing and break something else. People on my team became afraid of making CFEngine changes. When we decided to upgrade our Linux environment, we started using Puppet. We now manage most of our Linux systems with Puppet.

Read the rest of this entry »

For our final PuppetConf Preview, we’ve got Jesse Newland from GitHub talking about his upcoming presentation. If you can’t make it to PuppetConf, be sure to sign up for live video streaming of two of the large rooms. If you are out in San Francisco for the conference, GitHub is sponsoring a drink-up on Friday evening.

Jesse Newland from GitHub

Puppet Labs: Tell me a little about yourself and your technical background.

Jesse: I started doing sysadmin work on the student newspaper at the University of Georgia, where I learned all the good and bad things about system administration. I learned how to take down production sites on accident, and how to break and fix people’s desktops. I figured out a lot of anti-patterns for how to manage large infrastructure, and that’s where I started the search for utilities and tools to manage large infrastructure. I started more as an administrator than a developer, and as I came across more complex problems I started trying to work them out in Perl and C.

After that I started working at LexBlog, a company that does blogs for lawyers. So at that point I started really owning my skills in terms of web infrastructure. I worked at a company called Rails Machine, which is a Ruby on Rails web host. And at that job I honed my Ruby skills, and at that job I wrote some code and started actually using Puppet in production at the job. We had a very specific need for configuration management, and it pushed me into writing a Ruby DSL for Puppet before there was an official Ruby DSL for Puppet. It was a project called Moonshine, and it’s largely infamous in the Puppet community. It did a lot of of things very differently from how Puppet’s Ruby DSL did things, and a lot of things differently from how it might make sense for a lot of people who use Puppet as their primary tool. Moonshine was written for people that didn’t understand Puppet, for people that understood Ruby on Rails, so they could make subtle changes from a standardized set of configuration that would be easy for them to understand and easy for them to apply and work with.

And from there, I moved to GitHub, and continue to use Puppet as my primary tool.

Puppet Labs: Could you elaborate on how you use Puppet as your primary tool?

Read the rest of this entry »

John Willis from Enstratus

Puppet Labs: Do you mind telling me a little bit about yourself and your technical background?

John Willis: I’ve been in IT infrastructure as an Ops guy for almost 35 years now. I started back in the big old mainframes and for many years I worked around the IBM ecosystem of systems management operational infrastructure. I had a consulting company for about 10 years that focused on Tivoli, one of IBM’s brands, doing configuration management and monitoring event management. I sold my business about six or seven years ago and started working around open source alternatives, blogging about open source alternatives to the big fat Tivoli portfolio products and other proprietary ones. It led me into getting involved with the cloud community, and I eventually became a cloud evangelist at Canonical and worked as a VP of Services for Opscode. Now I’m working with a cloud management vendor called Enstratus.

Read the rest of this entry »

I was O’Reilly’s Velocity conference back in June, giving a talk on hacking Puppet, and Puppet’s configuration language came up a lot. Most people love the language and find it the simplest way of expressing their configurations, but some are frustrated by how simple it is and wish they had a full Turing-complete language like Ruby for specification. I thought it would be worthwhile to discuss why Puppet has a custom language, and dive into some of the benefits and costs.

Read the rest of this entry »

Design is an integral part of the way we build software at Puppet Labs. More specifically, we strive to answer a very simple question: what does the user need? This isn’t always an easy question to answer, but we’ve been happy with our success in doing so. User empathy is our conduit for user satisfaction.

In real-life terms, what does design at Puppet look like? Let me show you a project that just finished its initial design phase: Puppet Sites.

What do users need?

Puppet admins want to spend their time managing nodes that are under Puppet control. They don’t want to spend their time adding nodes to their deployments. On the contrary, their goal is to start managing new nodes as quickly as possible. We formalize this need in the following way:

As a Puppet admin I want to easily add new nodes to my deployment so that I can start managing them as quickly as possible.

How do we we satisfy user needs?

The current workflow for adding nodes to a deployment is as follows:

1. Configure puppet.conf with the location of the master/CA
2. Perform an agent run on the node
3. Login to the CA
4. Sign the node’s certificate
5. Perform another agent run on the node

The problem with the existing workflow is that it appears to have nothing to do with adding new nodes to a deployment and everything to do with signing certificates. There’s a disconnect between the workflow and the user need.

To fix this, we did three things:

1. We introduced the concept of a site. A site is a service that owns the list of nodes in your deployment, the authentication mechanism for adding new nodes to your deployment, and the configuration of Puppet services in your deployment.
2. We changed the semantics of the workflow, so that it directly addresses the user need.
3. We eliminated the overhead of signing into the CA and manually signing node certificates. There are secure ways to do this that don’t involve user interaction over SSH.

The workflow now resembles the following:

1. Login to the site host
2. Generate a pre-shared key
3. Join a node to the site using the pre-shared key
4. Repeat step 3 for every node you want to add to the site

We now have a workflow to fulfill the user’s goal. But notice that we’re still not talking about sites in technical terms. This workflow could apply equally well to a Puppet Face, a REST API, or a Dashboard plugin.

How do users interact with our workflows?

It’s all well and good to produce a general workflow for addressing a user need, but eventually you have to tie it to a real user interface. For the initial release of Puppet Sites, we decided to focus on two user interfaces: a Puppet Face and a REST API.

A designer is nothing without regular interaction with customers. While designing user interaction, we stayed engaged with our internal operations and professional services teams. After several rounds of feedback, the design of the Puppet Face for the workflow given above resembles the following:

node02$ ssh admin@site02.domain.com
Last login: Mon May  7 18:15:43 2012
site02$ mount /media/usbdisk
site02$ puppet site generate key > /media/usbdisk/site.key
site02$ umount /media/usbdisk
site02$ exit
node02$ mount /media/usbdisk
node02$ puppet node join site02.domain.com < /media/usbdisk/site.key
Trying to add node02.domain.com to the site at site02.domain.com...
 Use `puppet site status node02.domain.com` to confirm success
 To stop waiting for the command to complete, press Ctrl-C.
   The command will still complete in the background.
Added node02.domain.com to the site at site02.domain.com

What do users think?

We strive to make our design process as agile as our engineering process. We specifically avoid a waterfall model, where the design is completed in an ivory tower, then handed over the wall for implementation. Instead, our designers sanity check their design with internal and external users, talk to architects and engineers to make sure they aren’t violating the laws of physics, and stay involved in the project until it ships, making course corrections as necessary.

The Puppet Sites project is still underway, but we’ve already discovered mistakes made early in the process, and corrected them. Each group of users has been vital to this iterative process.

Our internal users made us aware that omitting a pre-shared key system was a poor idea. By themselves, the two other fixes—changing semantics and introducing sites—weren’t sufficient to solve the user goal of “quickly” adding nodes to their deployments.

Our architects made us aware of a “law of physics” deal-breaker in the design, where users could have passed their pre-shared key as a flag to puppet node join. This would have exposed the key to sniffer processes on the host machine, and represented an unacceptable security risk.

Our external users helped us realize that the concept of a “site” is foggy at best and they didn’t really understand the problem it was designed to solve. Also, they were concerned that we were deprecating existing functionality without providing a drop-in replacement. We’ve been working to ameliorate these concerns by formulating clearer stories around the problems that sites solve.

Design at Puppet Labs

Of course, Puppet Sites does far more than just provide admins with an easy way to add new nodes to their deployments. For the sake of brevity, I’ve only highlighted this one user story. However, having a site also helps Puppet admins who want to…

• …get a list of all the nodes in their deployment with a single command, so they don’t have to trawl multiple services to get this information.
• …centrally manage the configuration of all nodes in their deployments, so they don’t have to manually manage puppet.conf on each node.
• …access information about Puppet services from their manifests, so they don’t have to hardcode the location of their services into their manifests.

You can expect Puppet Sites to be available soon, as a point release for Puppet 3.0.

In summary, the UX team at Puppet produces three main artifacts: user stories, workflows, and wireframes.

• User stories allow us to talk about the needs of our users in a non-technical, goal-oriented way.
• Workflows describe the tasks users go through to achieve their goals.
• Wireframes are mockups of the actual systems that users use to complete these tasks.

Not all of our projects follow this process exactly, but, in general, the above holds true. User story to workflow to wireframe. Rinse and repeat as necessary. That’s how we do design at Puppet Labs.

Learn More

• What is User Experience in Puppet?
• Looking Forward to 2012: Design, Big Data in the Infrastructure, and DevOps

The actions of IT Operations teams influence the efficiency, availability, cost of delivery, and customer satisfaction of the services your organization is delivering to your customers. This means getting IT right, particularly IT Operations, has become a crucial part of doing business in the modern market. IT Operations managers face three major challenges—finding the right people, building the right culture, and creating the right processes. This won’t be a surprise to most managers, as they are the building blocks of any good team or organization. But IT has often lagged behind in tackling these challenges.

Of all of these challenges, finding the right IT Operations people has perhaps become the most serious and potentially has the most impact on improving your operations. The methods and incentives for recruitment have changed. Premium candidates expect challenging workplaces, cutting edge technology, excellent benefits, and a competitive salary.

The best candidates also expect companies to provide environments where they can share their skills and ideas with the wider Operations community. This includes allowing them to contribute to open source projects, publishing tools and code, and engaging with the community through social media and technical channels.

By opening up the way you work both internally and externally, companies can begin attracting the right candidates. Internally, look at the ideas fomenting around the DevOps community about communication and collaboration. Externally, publish your successes. Publish your Puppet modules. Look at the tools you develop internally and identify those that you think others might be interested in and open source them. Publish them on places like GitHub.

Follow this up with getting your staff to talk, blog, and present about the tools and techniques they use and most importantly about the difference it makes to how you run your infrastructure. Once you start to treat and view your IT Operations team as ambassadors for your organization in the broader Operations community, you’ll find that their success is reflected in a greater ability to hire.

There is a place where you can meet Operations rock stars and parade your organization’s virtues. That place is the upcoming PuppetConf event being held over September 22nd and 23rd in Portland, Oregon. Send your IT Operations staff, let them share ideas and meet a thriving community and demonstrate that you’re working towards creating a business that employs IT Operations rock stars. It’s more than just Puppet of course, it’s an Operations conference with DevOps, Cloud and Puppet tracks and will be attended by some of the smartest people in Operations and technology. Register today and save 10% using discount code cultureshift.

Cross-posted from the PuppetConf blog.

Indeed uses puppet in testing, integration, staging and production environments.
Puppet has enabled Indeed Operations to scale to hundreds of servers across multiple datacenters with differing infrastructure while maintaining configuration consistency in testing, integration, staging and production environments.

Our new Vice President of Marketing, Scott Johnston, was recently poking around on Indeed.com and discovered the trend feature. He did a quick search:

In an email to all, our CEO proclaimed: “Pretty sweet.”

What can only be described as a the virtual equivalent of the Ninja Turtles signature jumping group high five quickly followed. One thing we’ve all learned is that unless the x-axis is “time after doomsday” and the y-axis is “pain and suffering incurred,” graphs going up and to the right are worth celebrating.

Fortunately, That Guy came along before someone pressed the red button that pops all the bottles of champaign hidden in the walls and drops the confetti from the ceiling:

“I’m having a hell of a time reconciling the Y axis of that graph. What does 35,000 “Percentage Growth” mean? I think the absolute mode is even more telling. If I’m reading it right, the percentage of all job postings matching “puppet” has tripled since early 2009.”

We’re all fortunate for this email as it saved us from doing shots of Patron while hanging off the balconies (probably in violation of a few laws), but why does That Guy always have to come along and ruin the party? Honestly.

Predicament: I have no idea what these graphs mean either, and I’ve already agreed to write this blog post about how awesome they are. Insert expletive!

I went on a long hunt to learn more about these graphs in order to draw conclusions about the growth of jobs for people with Puppet know how, and other information like the total number of jobs on Indeed in 2010, and the total number of jobs in Indeed.com. However, I learned that the relative growth lines are showing percentage growth from a baseline of zero and that the process that goes into generating them is rather complex, using algorithms and moving averages to smooth out cyclical factors and numerous variables unique to online job postings. At the end of the day I am still a bit confused by what they indicate, but hell, they go up and to the right.

What I can say with confidence is that there are 1,160 available jobs for people with Puppet knowhow on Indeed.com. That’s a pretty good sign that learning Puppet and participating in a Puppet Training can help you find your next gig, or become more valuable to your organization. Don’t you think it’s time you checked out our training opportunities?

I also learned the following:

“We’ve used Puppet for about two years.  We use Puppet to maintain configuration consistency after provisioning servers.  Puppet also manages httpd, java, tomcat, log4j, MongoDB, and operating system configuration.

Indeed uses Puppet in testing, integration, staging and production environments.
Puppet has enabled Indeed Operations to scale to hundreds of servers across multiple datacenters with differing infrastructure while maintaining configuration consistency in testing, integration, staging and production environments.

We adopted Puppet to assist in standardizing and enforcing configuration consistency.  Since we began harnessing Puppet, the number of servers we manage has grown from a handful to hundreds.  Provisioning new servers and application instances with Puppet has enabled us to grow rapidly with peace of mind.

Jason Koppe, Systems Administrator at Indeed

Some other interesting searches on Indeed.com:

An argument that I regularly hear from people regarding the adoption of
configuration management tools is that their systems are unique and
comprised of many one-offs. In this article I will address the one-off myth
and discuss why your systems are not beautiful snowflakes.

If you deploy a one-off system, you lack the testing systems to support it.
Most organizations have at least two levels of pre-production environments,
such as Dev and QA. Is it really wise to deploy a system without another
environment in which to test?

Systems fail. It is not “if” but “when”. If your one-off was important enough
to build the first time, is it not just as important to have around after it
fails? Disaster recovery is important and when the time comes that you need it
is not the time to have to figure out how you are going to rebuild that
one-off. This is where you can save your organization and yourself instead of
being stressed out and under pressure that your system failed.

Another argument I hear in defense of the one-off is that “it is only temporary.”
This always makes me laugh. Ask anyone who has been in the field for awhile and
I’m sure they can point you towards “temporary” systems that have been deployed
for ages. We deploy systems so that people can use them and once they do, it is
extremely hard to take them back.

One-off’s are notorious for not having good
documentation or supporting the ability to reproduce them. As the person building those
systems you are probably responsible for keeping them up. The lack of
documentation, the inability to reproduce the system, and no environment for
testing all add up to those late night “something is on fire” calls and
extended downtime. As the creator you get blamed for the shoddy work to boot.

Even systems that do completely different things still share many
common attributes. They generally need host name resolution, authentication services,
kernel tuning, text editors, shells, basic utilities, etc. Once you start
modeling these services and breaking down what it means to be a system on your
network, you will start to see that they are not so unique after all.

Next time someone prompts you to build a one-off, you can explain the drawbacks
and build it right. You will be doing your organization and yourself a big
favor.

“DevOps” is an oft-discussed term—new discipline or just hype; movement or toolset? Whether you’re a DevOps devotee or detractor, we want to know what the state of DevOps is in your organization. Become a part of the spirited debate by sharing your thoughts in the first Puppet Labs DevOps survey!

The survey takes no longer than ten minutes to complete, and participation enters you into a drawing for some fabulous prizes. Three participants will receive $100 Amazon gift cards, and 25 lucky folks will win snazzy new Puppet t-shirts. And of course, all participants will get to reap the benefits of the survey results.

We’re hoping to hear from as many people as possible, so please encourage other interested folks to take the survey. Share it on facebook, tweet about it, or just bring it up around the water cooler.

Take the Survey

As most of you realize by now, Puppet 2.7 was released under the Apache 2.0 license instead of under the GPL, and Facter has already been released under the Apache license. My goal in this post is to explain why, and what effects you might expect to see as a result.

We’ve been talking about the possibility of this change for about two years, but it was only in the last six months that it’s been solidified as the right plan. For the vast majority of people, this change won’t affect you at all—Puppet is still open source, and under one of the most open licenses available. For a few of you, however, this license change will make it easier to embed Puppet into your software, ship it as part of a solution you’re building, or contribute code to it.

Our goal at Puppet Labs, and my goal since I started the company and the project, has been to have Puppet everywhere. I want every OS to ship with Puppet, I want every appliance to use Puppet, and I want every device that can’t run Puppet to speak its language and still integrate with it. Heck, I want to replace the package manifest formats with Puppet’s language. I’m a big believer in open source, but for strictly practical reasons. Puppet always had to be open source, because I couldn’t get the kind of ubiquity that I wanted with a purely commercial product—too much of our infrastructure is already open, and too many sysadmins understand the risks of locking yourself into software you can’t control and have no visibility into. Also, when I started the project I had no reputation and, um, no money; open source was a great way to bootstrap the company and project with very little expense.

With the GPL, realizing this goal of ubiquity was very difficult—some companies are comfortable with the GPL and have no problems including software released under it, but plenty of other companies are dreadfully afraid of the potential for it to force releasing of other code, whether or not that fear is rational. With Apache, these companies can focus on whether Puppet will make their solution better, and not worry about whether they’ll have to make significant business changes in order to use it. You won’t see me making arguments about freedom here (and you probably won’t have much luck engaging in such a conversation with me separately), but you have already seen that my practical perspective on this drives toward open source as much as anything else.

To me, the choice between GPL and Apache was never about which one is more free. It has always been about which one can best accomplish the goals of the project, and possibly which can do the best job of helping me fund those goals. In the end, choosing GPL means more companies that want to partner with us have to pay us, else their software must also be open, while choosing Apache means anyone can use, embed, and extend Puppet without having to worry about it affecting any other software. In other words, the choice between GPL and Apache for us as a company largely comes down to the GPL enabling fewer partnerships but some number of which that pay us, while Apache enables far more partnerships but few (if any) that pay us.

Puppet is primarily meant to be directly used by sysadmins. Switching to Apache, and hopefully seeing far more integrations and embedding, seems like a good trade-off. We give up business we might never have in exchange for relationships right now. Thus, the goal of ubiquity feels a bit closer.

I know this argument doesn’t persuade all of you, and I’ve already exchanged emails with one person who is convinced that this change means Puppet is no longer free software, but it is my sincere hope that we can quickly get back to writing and releasing software rather than talking about licenses. And, of course, it’s also my sincere hope that we see far more people embedding and integrating with Puppet.

As always, please contact me directly at luke@puppetlabs.com if you have any questions or concerns.