Automate Google Compute Engine with Puppet

This blog post was published in a slightly different form on the Google Cloud Platform blog.

At Puppet Labs, we're all about enabling people to enact change quickly, predictably and consistently. It's at the core of everything we do, and one of the reasons we moved the Forge to Google Compute Engine.

Google Compute Engine immediately halved Forge response time on comparable instances, and offers a lot of flexibility for how we deploy and manage our instances. We know many Puppet users are also looking for flexibility and speed, coupled with predictability. So as of Puppet Enterprise 3.1, we have used GCE’s gcutil command-line utility and their REST API — tools that provide much of the service’s flexibility — to provide native support for Google Compute Engine.

If you're curious about Google Compute Engine and want to experiment with it, the gce_compute module, available on the Puppet Forge, provides everything you need to manage compute instances, disk storage, networks and load balancers as genuine Puppet resources in Google Compute Engine. The rest of this post offers a few examples of what you can do with the module.

Below is a really simple example of what an instance looks like in Puppet, and the local application of Puppet to manage the instance. Puppet is easy to install, so it's easy to follow along and create your own running Compute Engine instance. Simply save each example to a file, prepare gcutil by running gcloud auth login and then run puppet apply with your example file as an argument.

#gce.pp
gce_instance { 'ryan-compute':
   ensure       => present,
   machine_type => 'n1-standard-1',
   zone         => 'us-central1-a',
   network      => 'default',
   image        => 'projects/centos-cloud/global/images/centos-6-v20131120',
}    

ryan:gce ryan$ puppet apply gce.pp
Notice: /Stage[main]//gce_instance[ryan-compute]/ensure: created

With this simple example, I have described the instance I want in Compute Engine. I can share this with my co-workers to treat as documentation, or use in their own Google Cloud project to get an instance built that’s just like mine. This concept becomes more useful the more complex your infrastructure is.

Here's an example that’s much closer to the real world: It expresses two instances configured by Puppet to be proof-of-concept http servers, complete with a Compute Engine load balancer and health checks.

gce_instance { ['web1', 'web2']:
   ensure       => present,
   description  => 'web server',
   machine_type => 'n1-standard-1',
   zone         => 'us-central1-a',
   network      => 'default',
   image        => 'projects/centos-cloud/global/images/centos-6-v20131120',
   tags         => ['web'],
   modules      => ['puppetlabs-apache', 'puppetlabs-stdlib', 
 'puppetlabs-concat', 'puppetlabs-firewall'],
   manifest     => 'include apache
   firewall { "100 allow http access on host":
       port   => 80,
       proto  => tcp,
       action => accept,
   }',    
}

gce_firewall { 'allow-http':
   ensure      => present,
   network     => 'default',
   description => 'allows incoming HTTP connections',
   allowed     => 'tcp:80',
}
gce_httphealthcheck { 'basic-http':
   ensure       => present,
   require      => gce_instance['web1', 'web2'],
   description  => 'basic http health check',
}
gce_targetpool { 'web-pool':
   ensure       => present,
   require      => gce_httphealthcheck['basic-http'],
   health_checks => 'basic-http',
   instances    => 'us-central1-a/web1,us-central1-b/web2',
   region       => 'us-central1',
}
gce_forwardingrule { 'web-lb':
   ensure       => present,
   description  => 'Forward HTTP to web instances',
   port_range   => '80',
   region       => 'us-central1',
   target       => 'web-pool',
   require      => Gce_targetpool['web-pool'],
}

With Puppet Enterprise and Google Compute Engine, it becomes fairly simple to build and continuously manage complex services, from the storage/network/compute resources in Google Compute Engine, through operating system configuration and application management. Another cool feature is the relationship graph that Puppet automatically generates from the requirements you express. You can use this as a tool to communicate with your team on how your compute instances relate to each other, or to express the dependencies in your application.

These examples demonstrate how to apply Puppet configuration directly in the gce_instance resource, but it's more practical in production to manage the configuration of your entire infrastructure through a Puppet Enterprise master and its agents. If you want to run yours in Compute Engine, or just try it out, the gce_compute module makes it simple to bring up a fully functional Puppet Enterprise master and console.

gce_instance { 'puppet-enterprise-master':
   ensure       => present,
   description  => 'An evaluation Puppet Enterprise Master and Console',
   machine_type => 'n1-standard-1',
   zone         => 'us-central1-a',
   network      => 'default',
   image        => 'projects/centos-cloud/global/images/centos-6-v20131120',
   tags         => ['puppet', 'master'],
   startupscript        => 'puppet-enterprise.sh',
   metadata             => {
      'pe_role'          => 'master',
      'pe_version'       => '3.2.2',
      'pe_consoleadmin'  => 'admin@example.com',
      'pe_consolepwd'    => 'puppetize',
   },
   block_for_startup_script => true,
}

gce_instance { 'agent1':
   ensure => present,
   zone                 => 'us-central1-a',
   machine_type         => 'f1-micro',
   network              => 'default',
   image                => 'projects/centos-cloud/global/images/centos-6-v20131120',
   startupscript        => 'puppet-enterprise.sh',
   metadata             => {
      'pe_role'          => 'agent',
      'pe_master'        => 'puppet-enterprise-master',
      'pe_version'       => '3.2.2',
   },
   tags => ['puppet', 'agent'],
   require => Gce_instance["puppet-enterprise-master"],
}

This example will bring up a single master and agent, in sequence. The puppet master installation process may take a few minutes. When it's finished, you can browse over https to its external IP address, and log in to the Puppet Enterprise console. Once you have Puppet Enterprise installed, you also have access to our node_gce cloud provisioner, offering another way to manage Google Compute instances with Puppet.

From base compute, storage and networking all the way up to a consistently managed application serving your customers, Google Compute Engine and Puppet Enterprise offer a readable, reusable and shareable definition of how your cloud infrastructure is built and interrelated.

Ryan Coleman is product manager for the Puppet Forge.

Learn More

Leave a comment