Git Workflow and Puppet Environments

One of the features offered by Puppet is the ability to break up infrastructure configuration into environments. With environments, you can use a single Puppet master to serve multiple isolated configurations. For instance, you can adopt the development, testing and production series of environments embraced by a number of software development life cycles and by application frameworks such as Ruby on Rails, so that new functionality can be added incrementally without interfering with production systems. Environments can also be used to isolate different sets of machines. A good example of this functionality would be using one environment for web servers and another for databases, so that changes made to the web server environment don’t get applied to machines that don’t need that configuration.

Mapping the Puppet code base against the environments shows the power of this method. People often use a version control system to manage the code, and create a set of branches that each map to an environment. Adopting the development, testing and production workflow, we can have a puppet.conf that looks something like this:

[main]
  server = puppet.example.com
  environment = production
  confdir = /etc/puppet
[agent]
  report = true
  show_diff = true
[production]
  manifest = /etc/puppet/environments/production/manifests/site.pp
  modulepath = /etc/puppet/environments/production/modules
[testing]
  manifest = /etc/puppet/environments/testing/manifests/site.pp
  modulepath = /etc/puppet/environments/testing/modules
[development]
  manifest = /etc/puppet/environments/development/manifests/site.pp
  modulepath = /etc/puppet/environments/development/modules

With this configuration, we could map three Git branches for these environments and set up a central Git repository with post receive hooks. When changes were pushed to this repository, they would be automatically deployed to the puppet master. The example post-receive hook later in this post will work with this kind of environment setup.

Dynamic Environments

While there are benefits to having a set of branches and environments like the previously outlined configuration, there are also drawbacks with a static set of environments. This model is somewhat constrained by enforcing a single workflow. For instance, if multiple people are working on experimental features in the development branch, they'll have to constantly merge their code through the development process. If multiple people are developing different features in the same branch, they’ll all have to work with unrelated changes in the code. In addition, migrating a single feature to testing or production gets more complicated if multiple features are in the same branch.

Modern distributed version control systems like Git handle these constraints by making branch creation and merging lightweight operations, allowing us to generate Puppet environments on the fly. Puppet can set up environments with an explicitly defined section in the configuration file, but we can exploit the fact Puppet will set the $environment variable to the name of environment it is currently running under. With this in mind, we can write a puppet.conf to resemble this:

[main]
  server = puppet.example.com
  environment = production
  confdir = /etc/puppet
[master]
  environment = production
  manifest    = $confdir/environments/$environment/manifests/site.pp
  modulepath  = $confdir/environments/$environment/modules
[agent]
  report = true
  show_diff = true
  environment = production

This handles the dynamic environment aspect; all we have to do is create a directory with manifests in $confdir/environments and we have created that environment. Generating new environments using Git is similarly easy. We can create a central Git repository somewhere, with a post-receive hook that looks something like this:

#!/usr/bin/env ruby
# Puppet Labs is a ruby shop, so why not do the post-receive hook in ruby?
require 'fileutils'

# Set this to where you want to keep your environments
ENVIRONMENT_BASEDIR = "/etc/puppet/environments"

# post-receive hooks set GIT_DIR to the current repository. If you want to
# clone from a non-local repository, set this to the URL of the repository,
# such as git@git.host:puppet.git
SOURCE_REPOSITORY = File.expand_path(ENV['GIT_DIR'])

# The git_dir environment variable will override the --git-dir, so we remove it
# to allow us to create new repositories cleanly.
ENV.delete('GIT_DIR')

# Ensure that we have the underlying directories, otherwise the later commands
# may fail in somewhat cryptic manners.
unless File.directory? ENVIRONMENT_BASEDIR
  puts %Q{#{ENVIRONMENT_BASEDIR} does not exist, cannot create environment directories.}
  exit 1
end

# You can push multiple refspecs at once, like 'git push origin branch1 branch2',
# so we need to handle each one.
$stdin.each_line do |line|
  oldrev, newrev, refname = line.split(" ")

  # Determine the branch name from the refspec we're received, which is in the
  # format refs/heads/, and make sure that it doesn't have any possibly
  # dangerous characters
  branchname = refname.sub(%r{^refs/heads/(.*$)}) { $1 }
  if branchname =~ /[\W-]/
    puts %Q{Branch "#{branchname}" contains non-word characters, ignoring it.}
    next
  end

  environment_path = "#{ENVIRONMENT_BASEDIR}/#{branchname}"

  if newrev =~ /^0+$/
    # We've received a push with a null revision, something like 000000000000,
    # which means that we should delete the given branch.
    puts "Deleting existing environment #{branchname}"
    if File.directory? environment_path
      FileUtils.rm_rf environment_path, :secure => true
    end
  else
    # We have been given a branch that needs to be created or updated. If the
    # environment exists, update it. Else, create it.

    if File.directory? environment_path
      # Update an existing environment. We do a fetch and then reset in the
      # case that someone did a force push to a branch.

      puts "Updating existing environment #{branchname}"
      Dir.chdir environment_path
      %x{git fetch --all}
      %x{git reset --hard "origin/#{branchname}"}
    else
      # Instantiate a new environment from the current repository.

      puts "Creating new environment #{branchname}"
      %x{git clone #{SOURCE_REPOSITORY} #{environment_path} --branch #{branchname}}
    end
  end
end

(Make sure that the post receive hook is executable!)

Creating and using a new environment

% git branch
* production
% git checkout -b new_feature
Switched to a new branch 'new_feature'
% mvim site.pp
% git add site.pp
% git commit -m 'Implemented my feature'
[new_feature 25a9e1b] Implemented my feature
 1 files changed, 1 insertions(+), 0 deletions(-)
% git push origin new_feature
Counting objects: 5, done.
Writing objects: 100% (3/3), 255 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
remote: Creating new environment new_feature
To git@git.host:deploy.git
 * [new branch]      new_feature -> new_feature

And from here on out, you can use the new_feature environment on your hosts, and use git like you would with any code base.

This development model gives us some simple access control. Utilizing access control with a tool like gitolite, we can allow people to generate new environments to test their own code, but deny them access to change the production environment. This allows us to institute some sort of change control, by requiring all code to be reviewed by a merge master before inclusion into production, and allows code to be tested and verified before the request for submission is made.

This is ideal for junior sysadmins, as it allows them to experiment with Puppet while preventing accidental pushes of incorrect code. However, it is important to keep in mind that while this can prevent accidents, it cannot prevent malice. Unless otherwise configured, the Puppet master will run all manifests as one user, so a malicious user could attempt to manipulate other branches than the one they created or used.

Additional resources/related content:

  • The model of dynamic Puppet environments with Git was pioneered at Portland State University, and one of our Professional Service Engineers, Hunter Haugen, originally wrote up the basic concept on his blog.
  • Gitolite has been a fundamental underpinning for managing and deploying puppet manifests, and it’s a very powerful tool. You can check out the documentation here.
  • If you want to learn more about how environments can be used and configured, you can find the official documentation here.
  • Pro Git is an excellent book on Git, and they do a great job of outlining the different git hooks and how you can use them. You can read the relevant chapter online.

Comments

Adrien Thebo

Adrien Thebo

I wrote and tested the hook, and then went through and added comments and renamed variables, although apparently not all of them. Thanks for catching that!

Lars Preben Sørsdahl

Lars Preben Sørsdahl

There is a bug in the post hook script:

FileUtils.rm_rf environment, :secure => true
should be:
FileUtils.rm_rf environment_path, :secure => true

Adrien Thebo

Adrien Thebo

@damjanek, thanks for the catch! I swear that when I originally wrote the code it worked great, but then I did a little cleanup before I posted it and missed a number of things. Thanks again!

Joe McDonagh

Joe McDonagh

This is nice, though we take a different approach I wrote about here: http://blog.thesilentpenguin.com/blog/2012/02/21/puppet-with-git-submodu...

We have no juniors and are able to work very fast here continuously in the staging env. We then use my promotion tool to list changes and review before promoting to prod. I can see adding some restrictions with a pre-receive, but we're all GitHub and afaik GH does not currently support pre-receive.

Reid Vandewiele

Reid Vandewiele

Dynamic environments came up again today on IRC and of course, blkperl and I directed people to go and check out this blog. We've done that a few times and it has always impressed.

Back in December this writeup provided motivation to overhaul our old bourne shell setup and we've been using a more generalized hook & ruby combo to make it all happen since then (we opted for a more complicated hook in order to more sanely separate our git server from our puppetmaster). If anybody's interested you can check it out at https://github.com/pdxcat/puppetsync.

finch: this article rocks. Awesome job. B-)

Justin Ellison

Justin Ellison

I've added just a bit of functionality to the script to enable the use of Git submodules in this workflow. It's all detailed here: http://sysadminsjourney.com/content/using-git-submodules-dynamic-puppet-...

Justin Ellison

Justin Ellison

Do you have this hook on github anywhere? I've made some modifications that would likely prove useful to others.

Adrien Thebo

Adrien Thebo

The code is available at https://github.com/adrienthebo/puppet-git-hooks , and improvements you have would be great!

Logan Hendershot

Logan Hendershot

Thanks for this post Adrien, your insights are valued as always.

Vlastimil Holer

Vlastimil Holer

It's only pitty that the Puppet environments are on the halfway to be usable in serious deployments where custom types/providers are used. http://projects.puppetlabs.com/issues/12173

Marcos Ortiz

Marcos Ortiz

Excellent article, Adrien.

Marvin K Mooney

Marvin K Mooney

interesting post finch, interesting post.

Adrien Thebo

Adrien Thebo

Hi Lars,

You might want to checkout the git repo for this code at https://github.com/adrienthebo/puppet-git-hooks/blob/master/post-receive... ; someone was kind enough to add support for git submodules there. I want to keep the number of edits on this post down, but feel free to file pull requests against the git repo!

damjanek

damjanek

There's an error in line:
puts %Q{Branch "#{branch}" contains non-word characters, ignoring it.}

It should be replaced with
puts %Q{Branch "#{branchname}" contains non-word characters, ignoring it.}

Lars

Lars

Little diff for you, that will allow you to use submodules with above ruby post-receive hook

diff --git a/post-receive b/post-receive
index a568526..e596417 100644
--- a/post-receive
+++ b/post-receive
@@ -71,11 +71,12 @@ $stdin.each_line do |line|
Dir.chdir environment_path
%x{git fetch --all}
%x{git reset --hard "origin/#{branchname}"}
+ puts "Updating submodules in environment #{environment_name}"
+ %x{git submodule update --init --recursive}
else
# Instantiate a new environment from the current repository.
-
puts "Creating new environment #{environment_name}"
- %x{git clone #{SOURCE_REPOSITORY} #{environment_path} --branch #{branchname}}
+ %x{git clone --recursive #{SOURCE_REPOSITORY} #{environment_path} --branch #{branchname}}
end
end
end

Seamie

Seamie

Hi Adrien,
we are trying to move from traditional server management to puppet-based server management. I have found your post very interesting since we need several environments (no dynamic so far since I am puppet beginner ...). I am going to use this idea but how to bring it together with the concept that e.g. servers 1 to 10 are in development environment, server 11-30 in testing and the rest in production ? What I mean is that e.g. puppet class describing syslog configuration is applied first to development branch then to testing etc. But there are different hosts in each branch. I don't follow that ... Can you explain it ?

Brandon

Brandon

Just tried this on a fresh install of 3.1.1 and it seems that:
manifestdir = $confdir/environments/$environment/manifests
was also required in puppet.conf under [master] ? Wouldn't work for me otherwise.

Andy Shinn

Andy Shinn

I've been running into bug #12173 for a while and didn't even realize it. My environment is small enough so that I can merge the changes to master and then fix again if needed. Once in master, updated types seem to get picked up by Puppet master and work fine. Obviously, this isn't a good idea for big environments which rely on different branches for system separation.

Mike

Mike

This is now a year old and I would like to know if this is still the recommended approach for a new Puppet deployment.

James Turnbull

James Turnbull

Mike - it's still a recommended approach for managing multiple environments or modeling the dev->test->production lifecycle.

Adrien Thebo

Adrien Thebo

The core workflow is solid and I definitely recommend it. I'm working on a prototype deployment method for more complex cases involving modules from the Puppet Forge and Github; when it's ready I'll make sure word gets around.

Mike

Mike

Thanks, gentlemen. I've got this working and I have to say it's pretty slick. Well done.

Stefán Freyr Stefánsson

Stefán Freyr Stefánsson

Thanks for this great post Adrien!

I hope it's ok that I used your Ruby script as the base for a Python version I did.

The reason for this was that I wanted to host my configs on a Git hosting site (bitbucket) and so I needed a script that could act on the POST service hook that they offer. So I started off and in the end I had a script that has the following features:
1) Can create/update/delete environments (local directories) based on branches on the remote repo.
2) Can "rebuild" the local environment from scratch (figures out what remote branches there are and uses that to create/update/delete). This was needed because the POST service hook does not provide information about what branch was deleted.
3) Can run as a webserver listening for the POST service hook.

It's fairly easy to modify it to suit other POST service hooks.

I hope this is ok with you, I licensed it under the MIT license and the source code can be found here: https://bitbucket.org/ru_csit/puppet_branch2env/src/

Just say the word and I'll take it down.

Adrien Thebo

Adrien Thebo

Wow, I hadn't realized how many people were still reading and commenting on this blog!

Stefán: nice work on the Python post-receive hook! The webserver functionality is quite nice; I chose to leave that implementation up to the end user but your solution provides a good out of the box solution. Your implementation is fine by me; thanks for sharing it!

Matt and Andy:
Unfortunately #12173 is an ongoing problem. Ruby can only have one instance of a class loaded at a time, so to get this level of isolation you would need to have one process per environment, which could consume a lot of memory. There's some talk of using the JVM and taking advantage of JVM servlets to provide this sort of isolation but there's nothing usable right now.

In the short run, my experience has been that this isn't a huge issue and you can generally work around it. Types shouldn't have a lot of behavior; they're there to provide an interface to providers so they should be reasonably stable and thus shouldn't vary a lot. The best way to mitigate this is to develop custom types in an isolated environment using something like Vagrant, and try to avoid varying the types a lot in production. It's not an elegant solution but I've found that it's enough to get by.

Brandon: If you only have a single manifest like `site.pp` then you only need to specify the `manifest` parameter, otherwise you might need to specify `manifestdir` if you have a number of files in the equivalent of `/etc/puppet/manifests`. If that's not the case then I would need to see the errors produced by puppet to figure out what's happening.

Seamie:

You can use dynamic environments with a predetermined list of branches; there's no limitation on how long a branch is around so you can still use this workflow for your setup. With respect to your question, do you need some way to select what environment a machine is assigned to? If that's the case then you can use an external node classifier (http://docs.puppetlabs.com/guides/external_nodes.html) to select the environment. When a Puppetized machine checks in, the ENC can determine what environment a node should be in based on the name, so if you know any machine with `prod` in the hostname should be in production then you can assign it accordingly. Is this what you need?

Chris: If you have a separated Puppet master and git server then you'll need some way to 'forward' the post-receive hook. For instance, you could have a post-receive hook on your gitolite server that captures stdin and shells into your puppet master and forwards the original input to the post-receive hook on your puppet master. This allows you to have that desired separation but allows things to function as one unit.

In addition if anyone has further questions about git workflow, you can ping me on IRC on irc.freenode.net in #puppet, my nick is finch and I'm online from about 9:00 to 17:00 -7 GMT.

Matt

Matt

Given http://projects.puppetlabs.com/issues/12173, how is anyone dealing with the collisions in types when working in a collaborative environment?

Chris

Chris

A popular blog post! One thing I've been struggling to understand is how this post receive code copies the data down to the puppetmaster host. My git host is different from my puppet host. I can work on a branch/environment on my workstation. Push it back to the gitolite server. But the gitolite server complains that /etc/puppet/environment doesn't exist. Looking at the ruby it seems to be checking the git server, and not the puppet server for that directory. What am I missing?

quickshiftin

quickshiftin

While the dynamic environments enable easy development workflows, which is great by the way, it seems like you're still going to end up merging back through a set of standard environments once development code is ready for the wild. For example, production, obviously you're going to have those boxes statically set to the production environment, and most likely other machines through the stack that are important will have appropriate static environments set. Unless I'm missing something.., why would you want to allow a production or qa box to run my-crazy-dev-branch?!

PeterZ

PeterZ

I might be missing the point, but I can't get my head around this workflow. Aren't we mixing puppet code lifecycle with infrastructure lifecycle? Ie. Production and Development environments are different with different packages and accounts.

What I was doing in the past is having Unstable and Stable branches both containing code for Production and Development (etc) environments. Is that an overkill?

Leave a comment

Tradeshow
Sep 3
Puppet Practitioner
Sep 9
Speaking Engagement
Sep 11
Tradeshow
Sep 12