Cloud computing has brought many benefits to enterprises and startups alike. Not only has it dramatically lowered the cost and time of getting from a prototype to a live product, it has lowered the time required to scale to increased demand. However, the benefits of cloud computing come with difficulties. One of these challenges is coordinating changes and automation throughout the running instances. Another is scaling up automatically when demand increases.
Puppet has helped solve the issues of coordination and automation of configuration management, while Puppet’s cloud provisioning capabilities aid IT groups in areas of automated scaling. For example, the most common approach for getting Puppet running in AWS is to have a base AMI that runs Puppet when launched. Then, Puppet configures the instance with its assigned role. This is not always a fast process. It may take upwards of several minutes for Puppet to get all the configurations in place for the instance to fulfill its assigned roles. If your auto scaler, such as an F5 or an apache balancer, fires up several hundred or several thousand instances, time is of the essence.
The time to scale can be lowered from several minutes to a few seconds if the AMI you use for a group of nodes is already up to date. This way, when Puppet runs on boot, it has to do very little, if anything, to configure the instance with the node’s assigned role.
You can ensure your AMIs are automatically updated whenever you update your production Puppet code by utilizing revision control system hooks, Puppet’s cloud provisioning, and a custom Puppet report processor. This will dramatically lower the time to scale, ensuring you’re ready for whatever may come.
This approach is based on the principle that your Puppet code is what’s authoritative in your network. Your build system may know the latest deployable version of your applications, but it can’t know what’s required for a system to actually push the code. However, if you update your Puppet code’s requirements to deploy your application, the code describes what the system should look like. It’s your Puppet code, or meta data, that should trigger AMI rebuilds.
Process
- Every time you update your production Puppet code, an instance of all the AMIs you want to manage should be automatically created.
- The instances run Puppet.
- A custom report processor on the Puppet master determines if changes occurred and if they were all successful.
- If yes, create a new version of the AMI through taking a snapshot of the instance.
- Register the new AMI with the auto scaler.
Requirements
- Puppet’s cloud provisioning fully configured. Take a look at the Quick Start guide.
- The puppet-autoami module, located https://github.com/ccaum/puppet-autoami
- Your Puppet code managed in a version control system that supports post commit/merge hooks such as git, subversion, mercurial, etc.
- All the AWS AMIs you want to auto update are EBS backed.
Further, for this example, Puppet cloud provisioning must be fully configured on the same node as Puppet master (for report processing) as well as the same node running the VCS post commit/merge hook.
Commit Hook
In this example, we’re using a git post merge hook. This hook at the link below triggers on merges, loads the cloudscale.conf file, processes the AMI groups, then loads instances of the AMIs.
https://github.com/ccaum/puppet-autoami/blob/master/ext/post-merge.git
Puppet autoami Configuration
The configuration file for the example module is /etc/cloudscale.conf. It’s INI format. Each section in the file represents a compute group that will use the configured AMI as its image. All AMIs you want to auto update should have their own section. For example, for a webservers group, you should have something resembling the following:
[webservers]
image = 'ami-f19a6098'
type = 'm1.small'
region = 'us-east-1'
keypair = 'my_rsa'
keyfile = '/root/.ec2/my_rsa.pem'
master = 'puppet.mydomain.com'
login = 'root'
pe-install-tarball = /tmp/puppet-enterprise-1.2.1-all.tar.gz
pe-answer-file = /tmp/pe-answers.txt
pe-version = 1.2.1
install-script = puppet-enterprise
The Report Processor
The report processor determines if the node reporting in is one that was launched by the VCS hook. If so, it determines if there were changes to the instance. If there were changes, and they were all successful, then use the CloudScale face to create a new AMI based on a snapshot of the reporting instance.
https://github.com/ccaum/puppet-autoami/blob/master/lib/puppet/reports/cloudscale_build_image.rb
Puppet Node Declaration
In order for Puppet to know what roles the instance created by Puppet’s cloud provisioner should be, you must create a node declaration for each group. The node declaration must be a regular expression with the first part of the regex being the name of the group followed by an ‘_’ character. For the example above of the webserver group, the node declaration would look like this:
node /^webserver_.*$/ {
include apache
include webapp
}When the instance launched by the VCS hook checks in, it will be assigned the apache and webapp classes. When Puppet is finished running on the instance, it will send its report to the Puppet master. The custom report processor will identify the instance as one created by the VCS hook, and check if there were any changes. If so, and they were all successful, the report processor will generate a new version of the AMI.
What Now
This part is up to you. The report processor doesn’t actually do anything with the AMI that was generated. However, with a few modifications to the report processor, you can have the new AMI version automatically registered with your auto scaler, for example.
https://github.com/ccaum/puppet-autoami/blob/master/lib/puppet/reports/cloudscale_build_image.rb
Conclusion
Together with Puppet’s configuration management, cloud provisioning capabilities, and the autoami module, you can have Puppet manage every aspect of your cloud environment completely autonomously. By using these tools together, you push your cloud infrastructure to fewer and fewer points of management allowing for more simplicity, more insight, and more efficiency.
Related content:
