Puppet + MCollective make for Quick Inventory Queries (Part 2 of 2)
Check out Part 1 of this post here.
Using MCollective to Query Your Nodes
MCollective is an awesome solution for accessing information from all of your nodes very quickly. MCollective has a facter plugin that can pull fact information out of facter, but there’s a method that Jordan Sissel devised for exporting all of your facter facts (as well as all of the variables in the puppet scope) to a YAML file (specifically, /etc/mcollective/facts.yaml). Setting this up is pretty trivial (it’s a single-line inline template), and the benefits are exponential; not only is this MUCH faster than querying facter, but, as I mentioned, it pulls all of your variables out of the puppet scope in which you declared the /etc/mcollective/facts.yaml file. Follow the instructions in that link and let’s setup the /etc/mcollective/facts.yaml file.
Once you’ve set this up, let’s make sure MCollective is able to read from the file. This can be set in your /etc/mcollective/server.cfg config file with the following:
factsource = yaml plugin.yaml = /etc/mcollective/facts.yaml |
Next, we need to install a plugin so MCollective can handle YAML files as a factsource. The yaml_facts.rb file is included as a plugin with the MCollective source, but it can also be downloaded from the PuppetLabs Github site. Drop this file into your MCollective library’s facts folder (usually /usr/libexec/mcollective/mcollective/facts), ensure that the factsource is set in your server.cfg config file, and restart MCollective to test it out. You can execute “mc-inventory (hostname)” to test and make sure that facts are being reported properly.
With MCollective up and running and reading facts directly from a YAML file, let’s run a sample query with “mc-facts warranty_out” and see what we get:
Report for fact: warranty_out No found 38 times Yes found 75 times Finished processing 113 / 113 hosts in 1047.24 ms |
If you remember, this fact tells us if our machine’s warranty has expired. In this test with 113 nodes, you can see that 75 nodes have expired warranties and 38 nodes are currently under warranty. Next, let’s look at the expiration date for our warranties with “mc-facts warranty_end”:
Report for fact: warranty_end 2011.02.23 found 4 times 2011.04.30 found 1 times 2011.07.14 found 23 times 2011.07.15 found 3 times 2011.07.16 found 2 times 2011.07.17 found 2 times 2011.07.19 found 16 times 2011.09.16 found 2 times 2012.09.15 found 5 times Expired found 123 times Finished processing 183 / 183 hosts in 3098.34 ms |
Here’s a larger sampling of nodes with several warranty expiration dates. This is all great information, but wouldn’t it be nice to see exactly WHICH nodes are corresponding to these dates. You can do that by running “mc-facts warranty_end -v”. The -v argument will list nodes corresponding to each fact underneath that fact in your output:
2012.09.15 found 4 times berry-shawstaff-shel deville-staff-wes kluding-staff-shel solomon-staff-shel |
You can repeat this command with ANY fact. Since I am using the YAML solution as a factsource, it’s also exporting variables from my puppet scope. I have several variables associated with packages that are to be installed, and these package reside in the same scope as the /etc/mcollective/facts.yaml file declaration, so I can also query these variables with “mc-facts (variablename)”
Report for fact: firefox firefox3.6.11.dmg found 218 times Finished processing 247 / 247 hosts in 3027.37 ms |
NOTE: Pay attention to the scope where you declare the /etc/mcollective/facts.yaml file. If this file is declared within the scope of its own class, you won’t see variables OUTSIDE its scope. You can do creative things to change the scope of this file if you wish (possibly using puppet-concat), but I’ll leave that up to you.
What About Offline Nodes?
MCollective works great for nodes that are online and connecting to your middleware solution (ActiveMQ, RabbitMQ, etc.), but what about machines such as laptops (or machines that have hardware failures) that might not be online? Fortunately, your puppet master server keeps a YAML store of fact data for all nodes that have ever connected to it. Located in your puppet master’s $vardir/yaml/facts directory, there’s a YAML file (identified by $certname) for every node. Writing a script to traverse the directory and query the nodes is pretty trivial—you can even create an MCollective Agent to do the task for you. Let’s look at a ruby script I created to search the YAML store:
#!/usr/bin/env ruby # # File: yaml_store_search.rb # # Description: A script to search through your puppet master's YAML store # for a specific node's data. require 'getoptlong' require 'puppet' Puppet.settings.parse if not ARGV[0] puts "You must have an argument. Use --help for more information." exit(1) end # Parse the arguments here and print the help prompt if any incorrect arguments # are being used. opts = GetoptLong.new( [ '--hostname', '-n', GetoptLong::REQUIRED_ARGUMENT], [ '--certname', '-c', GetoptLong::REQUIRED_ARGUMENT], [ '--help', '-h', GetoptLong::NO_ARGUMENT] ) begin opts.each do |opt, arg| case opt when '--hostname' $hostname = arg when '--certname' $certname = arg when '--help' puts <<-EOF yaml_store_search.rb [OPTION] value -n, --hostname: Search for a specific hostname in the YAML store. Requires a hostname argument -c, --certname: Search for a specific certname in the YAML store. Requires a certname argument -h, --help: Display this help text EOF end end rescue GetoptLong::InvalidOption, GetoptLong::AmbigousOption puts <<-EOF -n, --hostname: Search for a specific hostname in the YAML store. Requires a hostname argument -c, --certname: Search for a specific certname in the YAML store. Requires a certname argument -h, --help: Display this help text EOF exit(1) end # Determine whether we're doing a lookup via hostname or certname searchfield = 'hostname' and arg = $hostname if $hostname searchfield = 'certname' and arg = $certname if $certname # Iterate through your puppet YAML store using the $vardir from # your puppet.conf settings. If we find the YAML file with a # matching certname or hostname, break out of the loop. Dir.glob("#{Puppet[:vardir]}/yaml/facts/*") {|file| $tempfile = YAML::load_file(file).values if $tempfile[searchfield] == arg $found_file = true break end } # Output the list of facts. if $tempfile $tempfile.each_pair {|key, value| puts "#{key} = #{value}" } end |
This script will gather the value of your $vardir and will search the $vardir/yaml/facts directory in search of a node by way of its hostname or certname. If the node was found, it will output every fact that it has kept about that node. This script will ONLY work on puppet masters as they’re the only machines who keep this YAML store.
Since we’re talking about MCollective, let’s look at an MCollective Agent that will execute this code on your puppet master. Here’s the sample agent I created:
module MCollective module Agent class Yaml_store "yaml_store.rb", :description => "A conduit to search your puppet master's YAML store.", :author => "Gary Larizza ", :license => "Apache License, Version 2.0", :version => "1.0", :url => "http://glarizza.posterous.com", :timeout => 3 # Search action: This action will check for the YAML file of a specified certname or hostname. # => If the file exists, it will output the contents of the YAML file. If it # => doesn't exist, you will receive an error message. # Variables: # => fact => The fact for which we're checking a value # => value => The value for which we're searching # Calling: # => Run the fact with 'mc-rpc --agent yaml_store --action search --arg hostname=lab01-hsimaclab-hhs -v' # => or with 'mc-rpc --agent yaml_store --action search --arg certname=lab01-hsimaclab-hhs -v' # => or with 'mc-rpc --agent yaml_store --action search --arg certname=lab01-hsimaclab-hhs --arg fact=warranty_end -v' # action "search" do if request.include?(:hostname) if request.include?(:fact) searchfield = 'hostname' arg = request[:hostname] reply[:facts] = get_specific_fact(searchfield, arg, request[:fact]) else searchfield = 'hostname' arg = request[:hostname] reply[:facts] = get_all_facts(searchfield, arg) end elsif request.include?(:certname) if request.include?(:fact) searchfield = 'certname' arg = request[:certname] reply[:facts] = get_specific_fact(searchfield, arg, request[:fact]) else searchfield = 'certname' arg = request[:certname] reply[:facts] = get_all_facts(searchfield, arg) end end end # Action end def get_all_facts(searchfield, arg) Dir.glob("#{Puppet[:vardir]}/yaml/facts/*") {|file| $tempfile = YAML::load_file(file).values if $tempfile[searchfield] == arg $found_file = true break end } if $found_file $tempfile.each_pair{|key, value| puts "#{key} = #{value}" } else puts "That #{searchfield} was not found." end end # Def end def get_specific_fact(searchfield, arg, thefact) Dir.glob("#{Puppet[:vardir]}/yaml/facts/*") {|file| $tempfile = YAML::load_file(file).values if $tempfile[searchfield] == arg $found_file = true break end } if $found_file if $tempfile[thefact] puts "#{thefact} = #{$tempfile[thefact]}" else puts "#{thefact} is not a valid fact, or wasn't found." end else puts "That #{searchfield} was not found." end end # Def end end # Class Etc_facts end end # Module Agent end end # Module MCollective end |
This agent will accept an argument of a node’s hostname or certname and also, optionally, a specific fact whose value we would want to inspect. I’ve also created a DDL file that will parse the output of this agent properly. Here’s that DDL file:
metadata :name => "Yaml Store", :description => "A conduit to search your puppet master's YAML store", :author => "Gary Larizza ", :license => "Apache License, Version 2.0", :version => "1.0", :url => "http://glarizza.posterous.com", :timeout => 60 action "search", :description => "Retrieves Facter facts for specified nodes" do display :always input :hostname, :prompt => "Hostname", :description => "The hostname of the node for which we're getting facts", :type => :string, :optional => true, :validation => '(.*?)', :maxlength => 230 input :certname, :prompt => "The Puppet Certname Variable", :description => "The certname of the node for which we're getting facts", :type => :string, :optional => true, :validation => '(.*?)', :maxlength => 230 input :fact, :prompt => "A Searchable Facter Fact", :description => "The facter fact for which we want a value", :type => :string, :optional => true, :validation => '(.*?)', :maxlength => 230 output 'facts', :description => "The Facter facts you want displayed", :display_as => "Fact Information" end |
In order for this agent to work, your puppet master must also be running MCollective and the agent and DDL file must be located in your MCollective library’s agent folder (typically /usr/libexec/mcollective/mcollective/agent). Once you’ve dropped these files into the agent folder, you’ll need to restart MCollective for it to work. Here’s the syntax you will use to run it:
mc-rpc --agent yaml_store --action search --arg hostname=machine_hostname mc-rpc --agent yaml_store --action search --arg certname=machine_certname mc-rpc --agent yaml_store --action search --arg hostname=machine_hostname --arg fact=facter_fact mc-rpc --agent yaml_store --action search --arg certname=machine_hostname --arg fact=facter_fact # Or Simplified: mc-rpc yaml_store search hostname=machine_hostname mc-rpc yaml_store search certname=machine_certname fact=facter_fact |
You can run this command from ANY MCollective client so long as the agent is installed on your puppet master server. The catch is that if you want the client to have the output parsed correctly, they will need to have ONLY the DDL file dropped into their MCollective library’s agent folder (the agent file itself is only needed on the puppet master server). If the client DOESN’T have the DDL file installed, they can simply append “-v” at the end of the above commands to see output from the reply hash that is used to communicate with the DDL file.
This agent will return either the entire list of facts for your specified hostname/certname or, optionally, the value of a specific fact. This is useful if your machine has malfunctioned (or is offline) and you quickly want to access whether the warranty has expired or not. Being able to access it from an MCollective client who has access to mc-rpc is handy if you’re not currently on your work computer. This access, however, is entirely up to you.
In Conclusion
If you’re familiar with MCollective and Puppet (and you have a functional understanding of Ruby), it’s pretty easy to write custom Facter facts and MCollective agents to automate simple tasks. The process of maintaining an inventory can be particularly troublesome, but these scripts should give you near-immediate access to some of the most important facts about your Macs. Like I mentioned in the opening, most vendors will provide you with a conduit to access similar information to facts I’ve presented here for my Macs. All it will take is a bit of tweaking to adapt my code for your use. Keep pulling those strings!
Gary Larizza is the Director of Technology for a small K-12 School district in Northern Ohio where he routinely breaks systems in the name of advancement. When he’s not trying to automate himself out of a job through the use of Puppet and Ruby, he enjoys seeking out awesome open source tools that will make Mac Management easier. He’s been using Puppet in the Education sector for the past two years on hardware that would make Linux admins cry (namely, G5 Xserves and an iMac) and loves to share the knowledge he’s gained with anyone who buys him a wheat beer (Weihenstephaner FTW). All of his notes are on his Posterous blog, and he will be speaking at the 2011 Penn State University Macadmins Conference if you’d prefer to chat with him in-person.
1 Comment
Guest Post: Puppet + MCollective make for Quick Inventory Queries | Puppet Labs
[...] Continue with Part 2 of this post here. [...]