The Puppet CloudFormation Face integrates Puppet Enterprise (PE) with CloudFormation so that users can reliably create entire Puppet Enterprise “stacks” in Amazon’s EC2 from their workstation.

A “stack” refers to a collection of launched Amazon Web Services (AWS) “resources” that can be specified as a “template” in CloudFormation’s declarative json modeling language. Templates support a wide range of AWS resources, including: EC2 instances, security groups, credentials, as well as a host of other resources.

EC2 meta-data and user data can be associated with AWS resources in a CloudFormation template. This meta-data allows CloudFormation to not only provision EC2 instances, but also to bootstrap those instances into a functioning application stack.

The CloudFormation Face is a command line tool that can use this meta-data to specify all of the required information to bootstrap an entire Puppet Enterprise stack, including: modules to be downloaded from the Puppet Forge, groups to be created in the Enterprise Console, Puppet Agents to be provisioned, as well as security groups and classification information to be associated with those agents.

Pre-requisites

The following things need to be setup before the Puppet CloudFormation Face can be installed:

1. AWS credentials are required. These credentials must be associated with an account that has full administrative rights. Instructions for how to sign up for Amazon Web Services (AWS) can be found here.
2. Requires that a keypair exist within the region in which resources will be launched. This keypair is used to refer to user public keys that can be injected into created EC2 instances. We recommend us-west-1. Remember what region you made a keypair for, you’ll need it later.

Installation

These tools can be easily evaluated with the Learning Puppet VM, or you can follow the manual installation instructions in the README on GitHub.

1. Download the Learning Puppet VM, follow instructions to boot the virtual machine and log-in as root. Note: You may want to ssh into your running Learning Puppet VM for easy copying and pasting.
2. Once logged in, use the following command string to download the CloudFormation Face from the Puppet Forge, unpack it to Puppet’s module path and rename it to cloudformation.

   cd /etc/puppetlabs/puppet/modules && curl http://forge.puppetlabs.com/system/releases/p/puppetlabs/puppetlabs-cloudformation-v0.0.1.tar.gz | tar -xz && mv puppetlabs-cloudformation-0.0.1 cloudformation

3. Export the ec2 region you created your keypair for in the prerequisites section.

   example: export EC2_REGION=us-west-1

4. Make sure time on the virtual machine is up to date.

   ntpdate pool.ntp.org

5. CD into the cloudformation directory and edit examples/install.pp to include your AWS credentials and insert your java home. It will look something like this:

     class { 'cloudformation':
       aws_access_key => '< your key here >',
       aws_secret_key => '< your secret key here >',
       java_home => ‘/usr/lib/jvm/jre-1.6.0-openjdk’,
     }

6. Apply the modified manifest:

    	puppet apply examples/install.pp

   This will install the CloudFormation client tools and create the file: bashrc_cfn

7. Source the bashrc_cfn file to set up all of the necessary environment variables.

     	source bashrc_cfn

8. Verify that the CloudFormation client tools were successfully installed and configured by running the following:

   	cfn-describe-stacks

   A return of ‘No Stacks found’, validates that the client tools are correctly configured.

9. Verify that the CloudFormation Face is properly configured by running:

   	puppet help cloudformation deploy

   Basic usage information should be returned.

Creating a Puppet Enterprise environment from scratch

Since puppetlabs-cloudformation can reliably build out fully functional Puppet Enterprise environments, it is an ideal tool for evaluation or experimentation in EC2. Let’s build an example stack.

The ‘puppet cloudformation deploy’ action can be used to build out complete stacks of PE. The command below shows how to build out an prepared Puppet Enterprise evaluation environment. Supply your own ‘keyname’ and ‘stack-name’ to the above command.

	puppet cloudformation deploy \
		--stack-name DemoStack \
		--config config/pedemo.config \
		--disable-rollback \
		--master-type m1.small \
		--keyname <your-ec2-key-name>

Once the stack has successfully launched, use:

cfn-describe-stacks

and

	cfn-describe-stack-events <StackName>

to check the current state of this stack.

Once you see the state of CREATE_COMPLETE, you’ll also receive the public dns name of your new Puppet Master. You can open your web browser and browse to https:// and log in with the user name cfn_user and cfn_password to visit the Puppet Enterprise Console. Check out the video below to see this in action.

• install_modules – List of modules that should be downloaded from the Puppet Forge. The modules are downloaded to the modulepath of the master and all classes contained in these modules are populated into the Enterprise Console.
• dashboard_groups – Specifies a list of groups that should be added to the Enterprise Console. Groups are specified as hashes with the following keys:
  • classes – list of classes that should be contained in the group
  • parameters – hash of parameters to be contained in the group
  • parent_groups – list of parent groups
• puppet_agents – List of PE Agents that should be provisioned along with their classification information. Agents are specified as a hash with the following keys:
  • groups – Groups from the dashboard that should be applied to the agent.
  • classes – Classes that should be applied to the agent.
  • parameters – Node specific parameters to be added
  • ports – Creates a security group

The configuration file and its settings specify all required information to compose a stack. The information is sufficient for bootstrapping environments of PE from scratch. Although no information about a Puppet master has been specified in the configuration file, one is always created by default. This Puppet master installation includes the Puppet Enterprise Console and a security group to allow tcp access from agents to ports: 443, 8140, and 61613. It also allows tcp access on port 22 from any address.

Related Content:

• Learn more about CloudFormation
• Getting started with the Learning Puppet VM
• Download Puppet Enterprise 2.0 and test out ten nodes for free
• Learn more about Live Management with resource browsing and cloning and controlling Puppet and invoking agents

With the release of Puppet Enterprise 2.0, Puppet Labs has added cloud provisioning capabilities with support for some of the most popular public and private cloud offerings including Amazon EC2 and VMware. PE 2.0 allows you to quickly provision virtual machines and start managing them with Puppet right away.

Getting Started

The following set of “micro” tutorials will show you how to leverage Puppet Enterprise 2.0 cloud provisioning capabilities on a public cloud (Amazon EC2). In order to get up and running you will need access to an Amazon’s EC2 account. Please note that while this tutorials will focus on Amazon EC2, the Puppet Enterprise 2.0 provisioning capabilities are designed to support a wide range of cloud providers. Initially there is support for Amazon EC2 which is covered in this tutorial and VMware, which is covered here; with support for additional providers being added over time.

Setup EC2 for use with Cloud Provisioner

1. Download and install PE 2.0
2. Gather Amazon AWS details into the fog configuration file

Before launching instances with the Cloud Provisioner module, you’ll need to register with Amazon AWS and obtain your access credentials.

Once you have completed the registration process, follow the AWS configuration sections here.

Now that you are all set up, we are ready to create our first virtual server in the cloud.

Provisioning in the Cloud

Provisioning machines on public or private clouds offers a huge benefit over traditional methods used on bare metal machines. Gone are the delays caused by hardware procurement and network setup; new machines are just a few clicks away. But to take full advantage of these new possibilities, you’ll need the right tools for the job.

Create virtual machines on EC2

1. Gather Amazon Instance details
   • image — AMI to use when creating the instance
   • keyname — The AWS SSH key name as shown in the AWS console
   • type — Type of instance
2. Run the puppet node_aws create command

Create a Debian 6.0 micro instance on EC2:

# puppet node_aws create --image='ami-0ce41865' --keyname='kelsey_rsa' --type='t1.micro'
notice: Creating new instance ...
notice: Creating new instance ... Done
notice: Creating tags for instance ...
notice: Creating tags for instance ... Done
notice: Launching server i-2a38eb48 ...
#############
notice: Server i-2a38eb48 is now launched
notice: Server i-2a38eb48 public dns name: ec2-174-129-158-192.compute-1.amazonaws.com
ec2-174-129-158-192.compute-1.amazonaws.com

At this point you should be able to ssh into your EC2 instance:

# ssh -i .ssh/puppetlabs_id_rsa root@ec2-174-129-158-192.compute-1.amazonaws.com
Enter passphrase for key '.ssh/puppetlabs_id_rsa':
Linux ip-10-212-169-9 2.6.32-5-xen-686 #1 SMP Wed Mar 9 01:27:54 UTC 2011 i686

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@ip-10-212-169-9:~#

That’s all there is too it, and doing the same for VMware is equally as easy:
http://docs.puppetlabs.com/pe/2.0/cloudprovisioner_vmware.html

Cloud Maintenance

The ability to provision an unlimited number of virtual machines has its downsides, like virtual machine sprawl, it’s easy to lose track of how many machines you have and how they are configured. In short, virtual machines require real maintenance.

PE 2.0 greatly reduces this overhead by providing tools such as the Puppet Enterprise Console—built on top of the Puppet we all know and love—which enables you to visualize and manage your entire infrastructure, virtual or physical, right out of the box. When you provision your virtual machines using the new cloud provisioning capabilities, you get access to these features immediately.

Classify, and managing machines via the PE 2.0 Console (GUI)

# export PUPPET_ENC_AUTH_PASSWD=’XXXXXXXX’
# puppet node classify ec2-174-129-158-192.compute-1.amazonaws.com \ --node-group='default' --enc-ssl --enc-server dashboard.puppetlabs.com \
--enc-port 443 --enc-auth-user console
notice: Contacting https://master.hightower.puppetlabs.com:443/ to classify ec2-174-129-158-192.compute-1.amazonaws.com
complete

At this point you should be able to access your node via the Puppet Enterprise Console:



Conclusion

Whether you have an existing cloud infrastructure or just getting started, PE 2.0 can streamline the complete life cycle of running machines in the cloud while incorporating “cloud” best practices and world class configuration management.

Look for support for additional Cloud vendors in the near future.

Related Content:

• Read the docs on cloud provisioning in Puppet Enterprise 2.0. (videos embedded)
• Learn what you can do with those classified nodes in the Puppet Enterprise 2.0 recorded webinar
• Test it out — download ten free nodes of Puppet Enterprise 2.0.

Hadoop World, hosted by Cloudera for the Apache Hadoop project happened this past week in New York City. Puppet has a growing relationship with Cloudera, Hadoop, and their users. As savvy users begin storing endless petabytes of information, their infrastructures inevitably increase. Puppet allows admins to grow those deployments with ease, helped by the strong user community. A group of Puppet/Hadoop users have built modules for deploying CDH3 (see below for more).

The conference itself was great. A bizarre mix of executives and young quants made for an interesting array of apparel choices ranging from 3-piece suits to torn jean shorts. The team at Cloudera did a fantastic job keeping the talks on time and relevant, and all of the presentations were recorded and should be available from Cloudera in a few weeks.

I would suggest checking out the Etsy and Bit.ly talks. Both speakers showed an amazing passion for problem solving in the big data space and delivered talks that were relevant and entertaining. If you’re a Bit.ly user, you might benefit from this excellent Bit.ly hack: By shortening links to images and embedding them into a forum or other non-local content host like blog comments, you can track impressions of the image or through the bit.ly+ page. Every time the image loads Bit.ly will track a hit. While not robust feedback, it provides a basic level of analytics for pages outside of your domain.

All in all, Hadoop World offered an interesting insight into how to manage big data sets. Companies like Disney are collecting all kinds of information at unprecedented rates—last year, the organization generated more data points than the previous 80 years combined. These companies understand that their data is important, even as they develop strategies for how to handle it. IT storage, processing, and management tools and practices are contributing to the biggest benefits of big data.

Related posts:

• Deploying Hstack with Puppet
• Deploying Hadoop with Puppet
• Deploying Hbase with Puppet

Cloud computing has brought many benefits to enterprises and startups alike. Not only has it dramatically lowered the cost and time of getting from a prototype to a live product, it has lowered the time required to scale to increased demand. However, the benefits of cloud computing come with difficulties. One of these challenges is coordinating changes and automation throughout the running instances. Another is scaling up automatically when demand increases.

Puppet has helped solve the issues of coordination and automation of configuration management, while Puppet’s cloud provisioning capabilities aid IT groups in areas of automated scaling. For example, the most common approach for getting Puppet running in AWS is to have a base AMI that runs Puppet when launched. Then, Puppet configures the instance with its assigned role. This is not always a fast process. It may take upwards of several minutes for Puppet to get all the configurations in place for the instance to fulfill its assigned roles. If your auto scaler, such as an F5 or an apache balancer, fires up several hundred or several thousand instances, time is of the essence.

The time to scale can be lowered from several minutes to a few seconds if the AMI you use for a group of nodes is already up to date. This way, when Puppet runs on boot, it has to do very little, if anything, to configure the instance with the node’s assigned role.

You can ensure your AMIs are automatically updated whenever you update your production Puppet code by utilizing revision control system hooks, Puppet’s cloud provisioning, and a custom Puppet report processor. This will dramatically lower the time to scale, ensuring you’re ready for whatever may come.

This approach is based on the principle that your Puppet code is what’s authoritative in your network. Your build system may know the latest deployable version of your applications, but it can’t know what’s required for a system to actually push the code. However, if you update your Puppet code’s requirements to deploy your application, the code describes what the system should look like. It’s your Puppet code, or meta data, that should trigger AMI rebuilds.

Process

• Every time you update your production Puppet code, an instance of all the AMIs you want to manage should be automatically created.
• The instances run Puppet.
• A custom report processor on the Puppet master determines if changes occurred and if they were all successful.
• If yes, create a new version of the AMI through taking a snapshot of the instance.
• Register the new AMI with the auto scaler.

Requirements

• Puppet’s cloud provisioning fully configured. Take a look at the Quick Start guide.
• The puppet-autoami module, located https://github.com/ccaum/puppet-autoami
• Your Puppet code managed in a version control system that supports post commit/merge hooks such as git, subversion, mercurial, etc.
• All the AWS AMIs you want to auto update are EBS backed.

Further, for this example, Puppet cloud provisioning must be fully configured on the same node as Puppet master (for report processing) as well as the same node running the VCS post commit/merge hook.

Commit Hook

In this example, we’re using a git post merge hook. This hook at the link below triggers on merges, loads the cloudscale.conf file, processes the AMI groups, then loads instances of the AMIs.

https://github.com/ccaum/puppet-autoami/blob/master/ext/post-merge.git

Puppet autoami Configuration

The configuration file for the example module is /etc/cloudscale.conf. It’s INI format. Each section in the file represents a compute group that will use the configured AMI as its image. All AMIs you want to auto update should have their own section. For example, for a webservers group, you should have something resembling the following:

[webservers]
image   = 'ami-f19a6098'
type    = 'm1.small'
region  = 'us-east-1'
keypair = 'my_rsa'
keyfile = '/root/.ec2/my_rsa.pem'
master  = 'puppet.mydomain.com'
login   = 'root'
pe-install-tarball = /tmp/puppet-enterprise-1.2.1-all.tar.gz
pe-answer-file = /tmp/pe-answers.txt
pe-version = 1.2.1
install-script = puppet-enterprise

The Report Processor

The report processor determines if the node reporting in is one that was launched by the VCS hook. If so, it determines if there were changes to the instance. If there were changes, and they were all successful, then use the CloudScale face to create a new AMI based on a snapshot of the reporting instance.

https://github.com/ccaum/puppet-autoami/blob/master/lib/puppet/reports/cloudscale_build_image.rb

Puppet Node Declaration

In order for Puppet to know what roles the instance created by Puppet’s cloud provisioner should be, you must create a node declaration for each group. The node declaration must be a regular expression with the first part of the regex being the name of the group followed by an ‘_’ character. For the example above of the webserver group, the node declaration would look like this:

node /^webserver_.*$/ {
   include apache
   include webapp
}

When the instance launched by the VCS hook checks in, it will be assigned the apache and webapp classes. When Puppet is finished running on the instance, it will send its report to the Puppet master. The custom report processor will identify the instance as one created by the VCS hook, and check if there were any changes. If so, and they were all successful, the report processor will generate a new version of the AMI.

What Now

This part is up to you. The report processor doesn’t actually do anything with the AMI that was generated. However, with a few modifications to the report processor, you can have the new AMI version automatically registered with your auto scaler, for example.

https://github.com/ccaum/puppet-autoami/blob/master/lib/puppet/reports/cloudscale_build_image.rb

Conclusion

Together with Puppet’s configuration management, cloud provisioning capabilities, and the autoami module, you can have Puppet manage every aspect of your cloud environment completely autonomously. By using these tools together, you push your cloud infrastructure to fewer and fewer points of management allowing for more simplicity, more insight, and more efficiency.

Related content:

• Amazon Machine Images include Puppet and Facter
• Check out the cloud provisioning functionality coming up in Puppet Enterprise 2.0
• Watch Architecting for the Cloud: AWS Cloud Formation and Puppet, a PuppetConf talk by Jinesh Varia
• Read how Swisstopo uses Puppet and Amazon EC2 to scale

PuppetConf is a DevOps & Operations conference that provides practical information with immediate takeaway. The conference offers two days of talks, tutorials, demos, panels, and birds of a feather sessions. Our speakers hail from large enterprises, medium-sized companies, and start-ups and are the acknowledged experts and pundits in the field.

PuppetConf features four tracks. Our DevOps track headlines the conference with speakers hailing from Facebook, DTO Solutions, Dell, and more. Our Cloud Computing track brings together talks from Cloud Providers and managers of large Cloud infrastructures. Our Puppet 101 & 201 features users and our community showcasing a comprehensive series of step by step talks ranging from getting started with Puppet to using its most advanced features. Importantly, the Portland Center for Performing Arts was selected for its open spaces and quiet corners facilitating a strong hallway track.

PuppetConf also features a range of social and networking opportunities from parties, dinners, and tours of Portland landmarks, often involving beer. If you’re learning, hacking, recruiting, or just networking then PuppetConf is not to be missed.

Registration is now open and limited to 500 participants. The first 50 to register will receive a copy of Pro Puppet.

Puppet Labs and Eucalyptus Systems are partnering up to make data center infrastructure management in the cloud experience easier. Announced today, the Eucalyptus-Puppet Cloud integration kit vastly increases the speed and ease for users to configure, provision, and manage Eucalyptus on-premise IaaS clouds with Puppet. Learn how the best utilize these resources in our joint webinar with Eucalyptus tomorrow: Increasing the Speed and Agility of Cloud and Data Center Infrastructure Management with Puppet and Eucalyptus.

The webinar will feature co-founder Teyo Tyree on the Puppet side of things, and will cover:

• Overview of the new administration challenges and solutions for configuring, provisioning and managing IaaS clouds in the data center
• Description of how Data Center Administrators are configuring, provisioning and managing with Puppet and the advantages they are realizing
• Puppet-Eucalyptus use case demo: See how cloud and Data Center Administrators work together using Puppet and Eucalyptus

Register for the Webinar

Swisstopo is Switzerland’s national cartography agency, providing customers with both physical and digital maps, many of which are also available to mobile phones with a GPS. Wanting to transition their infrastructure to the cloud, Swisstopo used Puppet to streamline their servers and increase uptime.

Hanspeter Christ, Deputy Head of the Federal Spatial Data Infrastructure (FSDI) within Swisstopo explains the benefits of Puppet:

“The biggest benefit from our investments is not from moving to the cloud, but more that we have ‘Puppetized’ our servers. We have become more efficient in provisioning servers so we can easily get 100 of them up and running in a very short time.”

To learn more, read the rest of the Swisstopo case study

Puppet Labs is proud to introduce the newest product in our suite: Puppet Aura Power or PAP.  PAP is a 4th generation aura reading, auditing and holistic configuration management framework.  Nature gave us all the ability we need to see Auras, but since the emergence of Cloud and aura virtualisation most of our inherent abilities simply do not scale.  PAP provides the ability to deploy a fully scalable aura architecture that addresses your personal and organizational needs. Our PAP strongly draws from ancient traditions like Kabbalah, Magick and the Ancient Path of the Old Man of Cloud.

Puppet Aura Power is easy to use too!  All that is required is the knowledge of how to use your senses together with your conscious effort. You can see PAP at work in our exclusive screencast:

Red Monk analyst Michael Coté described the new product and its impact on the industry in a recent comment: “We know that proper Aura management is more of a cultural shift than a technology one. Still you need the tools to manage it, or else you wake up in the middle of the desert married to a three-legged coyote, asking yourself, “how did I get here?” and drinking martinis that turn out to be handfuls of sand. What you want is something that stops your Aura from controlling you, and lets you pull the strings. Little wonder that Puppet Aura Power is the industry leader for Aura management. From personal experiences, I can tell you it certainly knocks the socks off a handful of gravel you might have had some supposed truck-stop bruja blow on – and my three-legged, new wife loves it!”

Puppet Labs CEO, Luke Kanies adds, “There is nothing magic in the Universe, except our limited understanding of its nature. What we think we know about ourselves now is just a tiny drop in the Ocean of Knowledge in our personal CMDBs. Puppet Aura Power not only gives us ownership and stewardship of all that Knowledge, but more than that, it allows us to dynamically audit and re-configure our auras.”

Teyo was recently on a DevOps panel at Cloud Connect 2011 in Santa Clara. This hour-long panel is chock full of interesting tidbits from some DevOps and cloud computing superstars, as well as a surprisingly useful extended metaphor about fires and fire fighting.

DevOps Panel at Cloud Connect 2011 from dev2ops.org on Vimeo.

The video comes from the fine folks at dev2ops, and features (from left to right): Andrew Shafer of Cloud Scaling, our own Teyo Tyree, Alex Honor of DTO Solutions / Rundeck Project, James Urquhart of Cisco, Juan Paul Ramirez of Shopzilla, and Lloyd Taylor from ngmoco:).

My, what a busy week! We’re pleased as punch that Puppet Labs received recognition from two different organizations:

Puppet Labs was listed in the “Cloud—Infrastructure” category of the 2011 OnDemand Top 100 Private Companies. For more information on the award, feel free to read through the press release.

A collaboration between Dyn and CloudCamp, the CloudyAwards just finished up their third annual SXSW award show. Puppet won People’s Choice for Best Configuration Tools in the Cloud.

To find out how Puppet and the cloud work so well together, feel free to peruse the solutions section of our website, the recent post on AWS CloudFormation and Puppet, and read the OStatic post about Puppet in the cloud stack of Swisstopo (the federal office of topography in Switzerland).