The actions of IT Operations teams influence the efficiency, availability, cost of delivery, and customer satisfaction of the services your organization is delivering to your customers. This means getting IT right, particularly IT Operations, has become a crucial part of doing business in the modern market. IT Operations managers face three major challenges—finding the right people, building the right culture, and creating the right processes. This won’t be a surprise to most managers, as they are the building blocks of any good team or organization. But IT has often lagged behind in tackling these challenges.

Of all of these challenges, finding the right IT Operations people has perhaps become the most serious and potentially has the most impact on improving your operations. The methods and incentives for recruitment have changed. Premium candidates expect challenging workplaces, cutting edge technology, excellent benefits, and a competitive salary.

The best candidates also expect companies to provide environments where they can share their skills and ideas with the wider Operations community. This includes allowing them to contribute to open source projects, publishing tools and code, and engaging with the community through social media and technical channels.

By opening up the way you work both internally and externally, companies can begin attracting the right candidates. Internally, look at the ideas fomenting around the DevOps community about communication and collaboration. Externally, publish your successes. Publish your Puppet modules. Look at the tools you develop internally and identify those that you think others might be interested in and open source them. Publish them on places like GitHub.

Follow this up with getting your staff to talk, blog, and present about the tools and techniques they use and most importantly about the difference it makes to how you run your infrastructure. Once you start to treat and view your IT Operations team as ambassadors for your organization in the broader Operations community, you’ll find that their success is reflected in a greater ability to hire.

There is a place where you can meet Operations rock stars and parade your organization’s virtues. That place is the upcoming PuppetConf event being held over September 22nd and 23rd in Portland, Oregon. Send your IT Operations staff, let them share ideas and meet a thriving community and demonstrate that you’re working towards creating a business that employs IT Operations rock stars. It’s more than just Puppet of course, it’s an Operations conference with DevOps, Cloud and Puppet tracks and will be attended by some of the smartest people in Operations and technology. Register today and save 10% using discount code cultureshift.

Cross-posted from the PuppetConf blog.

Indeed uses puppet in testing, integration, staging and production environments.
Puppet has enabled Indeed Operations to scale to hundreds of servers across multiple datacenters with differing infrastructure while maintaining configuration consistency in testing, integration, staging and production environments.

Our new Vice President of Marketing, Scott Johnston, was recently poking around on Indeed.com and discovered the trend feature. He did a quick search:

In an email to all, our CEO proclaimed: “Pretty sweet.”

What can only be described as a the virtual equivalent of the Ninja Turtles signature jumping group high five quickly followed. One thing we’ve all learned is that unless the x-axis is “time after doomsday” and the y-axis is “pain and suffering incurred,” graphs going up and to the right are worth celebrating.

Fortunately, That Guy came along before someone pressed the red button that pops all the bottles of champaign hidden in the walls and drops the confetti from the ceiling:

“I’m having a hell of a time reconciling the Y axis of that graph. What does 35,000 “Percentage Growth” mean? I think the absolute mode is even more telling. If I’m reading it right, the percentage of all job postings matching “puppet” has tripled since early 2009.”

We’re all fortunate for this email as it saved us from doing shots of Patron while hanging off the balconies (probably in violation of a few laws), but why does That Guy always have to come along and ruin the party? Honestly.

Predicament: I have no idea what these graphs mean either, and I’ve already agreed to write this blog post about how awesome they are. Insert expletive!

I went on a long hunt to learn more about these graphs in order to draw conclusions about the growth of jobs for people with Puppet know how, and other information like the total number of jobs on Indeed in 2010, and the total number of jobs in Indeed.com. However, I learned that the relative growth lines are showing percentage growth from a baseline of zero and that the process that goes into generating them is rather complex, using algorithms and moving averages to smooth out cyclical factors and numerous variables unique to online job postings. At the end of the day I am still a bit confused by what they indicate, but hell, they go up and to the right.

What I can say with confidence is that there are 1,160 available jobs for people with Puppet knowhow on Indeed.com. That’s a pretty good sign that learning Puppet and participating in a Puppet Training can help you find your next gig, or become more valuable to your organization. Don’t you think it’s time you checked out our training opportunities?

I also learned the following:

“We’ve used Puppet for about two years.  We use Puppet to maintain configuration consistency after provisioning servers.  Puppet also manages httpd, java, tomcat, log4j, MongoDB, and operating system configuration.

Indeed uses Puppet in testing, integration, staging and production environments.
Puppet has enabled Indeed Operations to scale to hundreds of servers across multiple datacenters with differing infrastructure while maintaining configuration consistency in testing, integration, staging and production environments.

We adopted Puppet to assist in standardizing and enforcing configuration consistency.  Since we began harnessing Puppet, the number of servers we manage has grown from a handful to hundreds.  Provisioning new servers and application instances with Puppet has enabled us to grow rapidly with peace of mind.

Jason Koppe, Systems Administrator at Indeed

Some other interesting searches on Indeed.com:

An argument that I regularly hear from people regarding the adoption of
configuration management tools is that their systems are unique and
comprised of many one-offs. In this article I will address the one-off myth
and discuss why your systems are not beautiful snowflakes.

If you deploy a one-off system, you lack the testing systems to support it.
Most organizations have at least two levels of pre-production environments,
such as Dev and QA. Is it really wise to deploy a system without another
environment in which to test?

Systems fail. It is not “if” but “when”. If your one-off was important enough
to build the first time, is it not just as important to have around after it
fails? Disaster recovery is important and when the time comes that you need it
is not the time to have to figure out how you are going to rebuild that
one-off. This is where you can save your organization and yourself instead of
being stressed out and under pressure that your system failed.

Another argument I hear in defense of the one-off is that “it is only temporary.”
This always makes me laugh. Ask anyone who has been in the field for awhile and
I’m sure they can point you towards “temporary” systems that have been deployed
for ages. We deploy systems so that people can use them and once they do, it is
extremely hard to take them back.

One-off’s are notorious for not having good
documentation or supporting the ability to reproduce them. As the person building those
systems you are probably responsible for keeping them up. The lack of
documentation, the inability to reproduce the system, and no environment for
testing all add up to those late night “something is on fire” calls and
extended downtime. As the creator you get blamed for the shoddy work to boot.

Even systems that do completely different things still share many
common attributes. They generally need host name resolution, authentication services,
kernel tuning, text editors, shells, basic utilities, etc. Once you start
modeling these services and breaking down what it means to be a system on your
network, you will start to see that they are not so unique after all.

Next time someone prompts you to build a one-off, you can explain the drawbacks
and build it right. You will be doing your organization and yourself a big
favor.

“DevOps” is an oft-discussed term—new discipline or just hype; movement or toolset? Whether you’re a DevOps devotee or detractor, we want to know what the state of DevOps is in your organization. Become a part of the spirited debate by sharing your thoughts in the first Puppet Labs DevOps survey!

The survey takes no longer than ten minutes to complete, and participation enters you into a drawing for some fabulous prizes. Three participants will receive $100 Amazon gift cards, and 25 lucky folks will win snazzy new Puppet t-shirts. And of course, all participants will get to reap the benefits of the survey results.

We’re hoping to hear from as many people as possible, so please encourage other interested folks to take the survey. Share it on facebook, tweet about it, or just bring it up around the water cooler.

Take the Survey

As most of you realize by now, Puppet 2.7 was released under the Apache 2.0 license instead of under the GPL, and Facter has already been released under the Apache license. My goal in this post is to explain why, and what effects you might expect to see as a result.

We’ve been talking about the possibility of this change for about two years, but it was only in the last six months that it’s been solidified as the right plan. For the vast majority of people, this change won’t affect you at all—Puppet is still open source, and under one of the most open licenses available. For a few of you, however, this license change will make it easier to embed Puppet into your software, ship it as part of a solution you’re building, or contribute code to it.

Our goal at Puppet Labs, and my goal since I started the company and the project, has been to have Puppet everywhere. I want every OS to ship with Puppet, I want every appliance to use Puppet, and I want every device that can’t run Puppet to speak its language and still integrate with it. Heck, I want to replace the package manifest formats with Puppet’s language. I’m a big believer in open source, but for strictly practical reasons. Puppet always had to be open source, because I couldn’t get the kind of ubiquity that I wanted with a purely commercial product—too much of our infrastructure is already open, and too many sysadmins understand the risks of locking yourself into software you can’t control and have no visibility into. Also, when I started the project I had no reputation and, um, no money; open source was a great way to bootstrap the company and project with very little expense.

With the GPL, realizing this goal of ubiquity was very difficult—some companies are comfortable with the GPL and have no problems including software released under it, but plenty of other companies are dreadfully afraid of the potential for it to force releasing of other code, whether or not that fear is rational. With Apache, these companies can focus on whether Puppet will make their solution better, and not worry about whether they’ll have to make significant business changes in order to use it. You won’t see me making arguments about freedom here (and you probably won’t have much luck engaging in such a conversation with me separately), but you have already seen that my practical perspective on this drives toward open source as much as anything else.

To me, the choice between GPL and Apache was never about which one is more free. It has always been about which one can best accomplish the goals of the project, and possibly which can do the best job of helping me fund those goals. In the end, choosing GPL means more companies that want to partner with us have to pay us, else their software must also be open, while choosing Apache means anyone can use, embed, and extend Puppet without having to worry about it affecting any other software. In other words, the choice between GPL and Apache for us as a company largely comes down to the GPL enabling fewer partnerships but some number of which that pay us, while Apache enables far more partnerships but few (if any) that pay us.

Puppet is primarily meant to be directly used by sysadmins. Switching to Apache, and hopefully seeing far more integrations and embedding, seems like a good trade-off. We give up business we might never have in exchange for relationships right now. Thus, the goal of ubiquity feels a bit closer.

I know this argument doesn’t persuade all of you, and I’ve already exchanged emails with one person who is convinced that this change means Puppet is no longer free software, but it is my sincere hope that we can quickly get back to writing and releasing software rather than talking about licenses. And, of course, it’s also my sincere hope that we see far more people embedding and integrating with Puppet.

As always, please contact me directly at luke@puppetlabs.com if you have any questions or concerns.

Swisstopo is Switzerland’s national cartography agency, providing customers with both physical and digital maps, many of which are also available to mobile phones with a GPS. Wanting to transition their infrastructure to the cloud, Swisstopo used Puppet to streamline their servers and increase uptime.

Hanspeter Christ, Deputy Head of the Federal Spatial Data Infrastructure (FSDI) within Swisstopo explains the benefits of Puppet:

“The biggest benefit from our investments is not from moving to the cloud, but more that we have ‘Puppetized’ our servers. We have become more efficient in provisioning servers so we can easily get 100 of them up and running in a very short time.”

To learn more, read the rest of the Swisstopo case study

The needs and benefits of configuration management software (CMS) to manage environments have long been recognized, even before the explosion of system growth due to virtualization and cloud computing. “Towards a High-Level Machine Configuration System” was not the first paper written about configuration management, but it underlines the need for configuration management software existed as far back as early 90s. Due to the lack of viable solutions, early adapters in this field had to create various homegrown solutions ranging from make, bash, and perl scripts tied with version control software such as CVS to manage complex environments. Rolling your own configuration management tool may have made sense back when Linux fit on floppy disks and open source was a new vocabulary where there were no viable options. But today, there are compelling reasons to adapt and convert to a tool like Puppet rather than writing and maintaining an in-house configuration management tool. Why should you make the effort to migrate to Puppet from an existing solution?

1. The cost to develop and maintain the entire tool chain. It’s cost-prohibitive for organizations to write and maintain a custom configuration management tool due to the time and cost to develop and maintain the software as well as training users on the custom solution. The initial developer/admin is often the only expert on the internally developed software, and this creates a vicious cycle of difficulty for hiring people who can understand and extend the home grown tool, which is often neglected and no longer maintainable. This often leads to new staff rewriting the same solution in a new language to replace the existing tool.

2. The complexity to support multiple platforms and large environments. Scripts developed internally were rarely flexible enough to handle multiple platforms. The internally developed solution is often targeted for one platform, so it needs to be ported and rewritten for another operating system, or even a newer version of the existing OS. And as the number of systems grows, the homegrown tool (which is usually several tools chained together) will stress and often break as the environment scales. Puppet, which is supported on most Unix platforms, has been tested and proven in several large-scale infrastructures. Now with Puppet Enterprise, Puppet is scalable out of the box with full stack including Ruby along with Apache and Passenger.

3. Lack of user community and open source ecosystem. Very few of the tools developed internally have adapted a clear licensing policy, so it’s rare for them to be published, documented, and reused/adapted outside the organization. This severely limits the scope of the users, and the number of minds that can work on any problems that may come up within the system. Puppet is open source, and users have published modules to deploy applications and extend functionality at forge.puppetlabs.com. There are thousands of users on the puppet user mailing list and hundreds of active participant on freenode #puppet—which means countless resources to help anyone writing Puppet manifest or extending Puppet on their own.

If these reasons convinced you to check out Puppet, see our intro to Puppet documentation, and download Puppet Enterprise.

James Turnbull, our very own Director of Operations, and the author of Pulling Strings with Puppet: Configuration Management Made Easy, recently shared his perspective on DevOps.

In a guest post on the Agile Web Development and Operations blog, James advocates some of the benefits of DevOps, despite admitting that there is still a lot of work to be done in making the concept a reality.

I’m not even all that attached to the concept of total Development and Operations cross-over. Although I think it does work and has the potential to make organizations more powerful and more effective. Make them better at delivering the services in the quantities and qualities that keep us in work and keep us all getting paid. If initially however this ends up being an Operations-centric growth then that’s good too.

We’re got a long way to go in that space before we can claim progress and there is always time to build those bridges later. Bridges that are far easier to build if Operations people understand and speak some of
the languages and use some of the techniques of developers. Bridges that are far easier to build if we understand each other better.

We are exploring these concepts with internal teams too. Let us know how DevOps has had an impact on your people and systems.