Craig Watson had a touch of “green tick syndrome.” With 25 virtual machines running “every kind of system configuration you could squeeze into a LAMP stack, including DNS/DHCP servers, Collectd/Graphite system metrics, Memcached and Nagios monitoring,” he was looking for a green tick on the console that would tell him VMware’s tools were installed correctly. So he wrote a module.
| Purpose | Install and configure the Stingray Traffic Manager |
| Module | riverbed/stingray |
| Puppet Version | 2.7+ |
| Platforms | Linux x86, x86_64 ‐ Kernel 2.6 (2.6.8+, 2.6.22+ for IPv6), glibc 2.3.6+ |
Riverbed Stingray Traffic Manager is a full performance software and virtual Layer 7 application delivery controller (ADC) that enables enterprises and cloud operators to create, manage, and deliver key services more quickly, more flexibly, and at a lower cost. Stingray Traffic Manager offers much more than just load balancing. It controls and optimizes end-user services by inspecting, transforming, prioritizing, and routing application traffic.
The Stingray Traffic Manager Puppet module will automatically download and install the Stingray Traffic Manager software onto a node.
Four months ago, the Puppet Forge reached 600 modules, and our new team had started to build a bigger and better service. Today, we’re nearing our 1000th module. As one of you makes that contribution to the Puppet Forge, we want to celebrate with you through three little giveaways.
The Puppet Forge is your service for discovering Puppet modules [docs]. You’ll find modules for managing applications like Apache’s httpd web server, facts for automatically gathering warranty information, types and providers for writing better modules yourself, and so much more. Whether you’re looking for something to use as a starting point, or want to drive your entire OpenStack infrastructure with Puppet, the Forge has you covered.
To enter: Read below about our three giveaways and their rules. Then, fill out this form to make your submission.
This one’s simple. Simply tell us via our entry form about your favorite Puppet Forge module. I have a few personal favorites that I’ll tell you about once the contest ends. :-)
Having more modules is great, but let’s not forget that we’ve got nearly 1,000 existing modules to maintain and improve. Some modules may just need improved module documentation while others may be broken and need your CPR.
For this part of our celebration, we want to see you contribute to a community member’s module. Contribute something of substance to a non-Puppet Labs module between now and April 5th. Submit via our entry form, and five of you will receive a $50 gift card to Amazon.com.
Of course, we also want your new module contributions. We’ll give away three $35 gift cards to Amazon.com for new modules.
Here’s the rub: it needs to be a new module submitted between now and April 5th, and it must automate something we don’t already have. For example, don’t enter with a new Apache module but consider publishing a module for Zenoss. Before you submit your entry, do a search for your module and check the first release date. If it’s prior to March 25, your module may not be eligible.
Learn More
Want some tips for writing better modules?
| Purpose | Build Java keystores form existing keys and certificates. |
| Module | puppet/java_ks |
| Puppet Version | 2.7+ |
| Platforms | OpenJDK 6, OpenJDK 7 |
This module attempts to ease and shorten the workflow associated with Java applications.
The reason this module came to life was my frustration over the workflow needed to get a SSL protected ActiveMQ broker set up. When I wanted to integrate the Java keystore build workflow into the rest of ActiveMQ’s setup using a Puppet manifest… well, it got ugly. Converting a string of shell commands into Puppet exec resources eventually led me to a dark dark place. Personally I find that if you are running into a need for a lot of exec resources, especially when they are using the same command or operating on the same file, it is time to grab a copy of Puppet Types and Providers and get your hands dirty with some Ruby. You’ll usually notice a speed increase of your agent runs after a conversion to a type/provider to replace all the exec resources and always end up with easier to maintain manifests. Read the rest of this entry »
You, our community, have contributed over 800 modules to the Puppet Forge. We expect that by the end of March, one of you will have contributed the 1000th module. We’re so excited by what you’re building and can’t wait to see what’s next.
As you take us to our 1000th module, we’re striving to make it simpler for you to contribute content as well as display the important details from your module. Now, when you publish a new release of your module, your Readme, Changelog and License will be automatically displayed on the module page.
Previously, module documentation was provided during a separate step of the module publishing process. Today’s change removes that step and presents the README you shipped with your module front and center.
For the many of you that have provided a License and/or Changelog with your module, we’ve taken the additional step of presenting that information on your module page.
Nothing is required from you to make this happen. It’s already been done for existing content, and for new releases, simply include a README, License and/or Changelog file at the top level of your module and publish it. We’ll handle the rest, including Markdown rendering if applicable.
Writing better READMEs
On the subject of module documentation, we recently brought on a tech writer to improve documentation of modules that Puppet Labs contributes to the Puppet Forge, in addition to producing templates and guides to help you write more effective documentation for your modules. Check back soon for blog posts on that.
#puppetize
Check out our new module page with these random community contributions.
Learn More
A few days ago, I left you with Part 1: Top Questions on Puppet and Windows. As I mentioned in that blog post, I received a ton of questions during a Puppet Enterprise and Windows webinar that I recently hosted — so many that I couldn’t get to all of them. In part 1, I answered questions such as: how is Puppet Enterprise different from SCCM, future plans for Windows-capable modules on Puppet Forge, and whether or not the puppet master role can be installed on a Windows machine. Today, I complete the blog series with answers to more questions on Puppet and Windows. Enjoy!
Q: Can Puppet be used to copy contents recursively from one Windows shared folder to another Windows shared folder?
You can do this using a Puppet file type with a local source parameter. Puppet makes no particular distinction between any folder, regardless of whether it is shared (Windows), mounted over NFS (Linux), or on the same disk partition as the OS. The Puppet file type works at the filesystem layer so as long as it’s attached, it can be used.
To recursively ensure that one folder contains the same content as another using Puppet, you can specify a resource such as the following:
file { 'C:/packages':
ensure => directory,
source => 'Z:/depot',
recurse => true,
} |
Q: Is it possible to install Puppet with the MSI but without starting the service?
To achieve this currently, the MSI file needs to be edited (e.g. using Orca) to change the startup type. A feature request has been filed to expose this option on the command line so that it can be specified without adjustments to the MSI.
Q: How does Puppet for Windows handle file permissions (AD rights, etc.). When a file is copied from a share, what permissions are inherited?
Puppet can manage file owner and group per typical Puppet practice. For Windows ACLs, Puppet currently uses an interpretation of Posix-style modes to set permissions, but does not support enforcing more complex ACLs. Josh Cooper (Puppet Labs, Developer) describes the behavior in a little more detail:
If owner, group and mode are unspecified, then [a given] file/dir receive the default ACL for the user Puppet is running as, e.g. LocalSystem. If owner, group, or mode are specified, then Puppet maps the POSIX style permissions to Windows ACLs, similarly to how cygwin does this. Each file/dir is “protected” meaning it does not inherit from higher level containers, so that Puppet can explicitly manage the effective state of the resource.
In the Windows world, using Puppet to manage file ACLs more directly would be hugely useful. The correct approach for managing ACLs would be to create a separate type/provider to apply ACLs on top of managed or unmanaged files. Some discussion to that effect can be found here and a feature request can be found here.
Q: How does Puppet help with backup and disaster recovery for Windows?
Configuration management is all about defining and enforcing the important parts of your configuration in a central revision-controlled “source of truth.” If you use Puppet to define every piece of configuration implemented on a production system, you get two benefits:
Puppet is not a replacement for data-layer backups for variable data or database dumps, but it can be used to install and configure client backup software or ensure that scheduled backup jobs exist.
Q: Can I get the status of the installation programmatically from Puppet?
The data flow for a Puppet run is roughly as follows:
When the report is received by the master it is sent through a series of report processors, according to the master’s configuration. The report processor API can be used to write custom report processors that will react programmatically to any configuration management event, whether it is a particular resource failure, a resource change, a threshold number of changes, or any other trigger/condition based on the report data. For more information on report processors, click here.
If you would like real-time data about the progress of Puppet runs before the finished report is submitted to the master, open source projects exist to provide this even deeper level of introspection. One such tool is Hunter Haugen’s progress_mq logdest terminus plugin which submits a message to a message bus whenever an event is produced by the puppet application. Readers can then monitor the bus and report in real time the progress of a Puppet run. The progress_mq project can be found here.
Q: Can I do provisioning of an environment using Puppet?
The Puppet Cloud Provisioner tool allows integration with Puppet for the purposes of creating, bootstrapping, and terminating virtual machines using a command-line or Ruby API through Puppet. More information on the Cloud Provisioner can be found here. Also of interest is the early-alpha Razor application, which is designed to provide profile-based bare-metal provisioning. More information on Razor can be found here. The core Puppet application is designed to run on anything from Just Enough Operating System (JeOS) up and so does not address provisioning directly, leaving that up to the Cloud Provisioner, Razor, or other lower-layer applications.
Q: Is there a way to stop an .exe from being installed every 30 minutes? The ‘Package’ resource for a MSI can include the ‘ensure installed’ option. The ‘Exec’ resource will just fire off every time the agent checks in.
The ‘windows’ package provider which ships with Puppet 3.0 can handle arbitrary *.exe installers, can determine whether or not an application needs to be installed, and will not perform extraneous work or reinstallation without a reason. The most common cause of a package resource performing repeat-installation is a name error; the name of the package resource (in Puppet) needs to match exactly the name in Windows (the DisplayName property from Add/Remove Programs) in order for Puppet to detect that the “package” is already installed.
For exec resources check out the “refreshonly”, “creates”, and “unless” parameters. These exec-specific parameters allow for manual specification of condition checks that will determine whether or not to actually run the command specified. Click here for detailed technical information on each of the parameters and what they do.
Last week, I hosted a webinar on Puppet and Puppet Enterprise on Windows. During the half-hour-long demo, I covered a number of items, including how to use Puppet Enterprise to manage Windows machines, and how to manage fundamental resources on Windows (in contrast with the same resources on Linux). I also dove deeper into Windows-specific resources, such as the Windows registry module. Attendees asked a lot of questions that I couldn’t get to, so today I’d like to share the following questions and answers with you. Hopefully you find this helpful. If you have any additional questions on Puppet and Windows, please feel to drop them in the comments below.
Please note: This is part I of a two-part series on Puppet and Windows. There’s more to come in the next couple of days.
Q: As a developer, I see the potential of Puppet, but I struggle at times to communicate it to management. One of the questions I often get is: How is Puppet better than SCCM and/or GPO?
A: System Center Configuration Manager (SCCM) brings to the table a Windows-native tool that is well-integrated with its target software and OS, capable of managing configuration from the provisioning step on up. Puppet is limited to the configuration layer only and does not descend as low as provisioning, and it doesn’t come with a Windows-native GUI for setting up policies. What Puppet does differently than SCCM is offer true infrastructure-as-code configuration management.
Puppet doesn’t depend on a database for any part of the configuration management process (a database for storing and searching reports is optional), and everything it consumes and produces can be reduced to flat files. This makes Puppet easily portable, deployable, simple, and node-disposable. For organizations with a hybrid IT infrastructure, Puppet offers a cross-platform management tool that can capture a complete, lightweight, versioned, automation-executable description of all nodes in human-readable plaintext; not just Windows nodes, not just Linux nodes, all of them.
In terms of technical ability, Puppet core types and providers give a solid spread of out-of-the-box functionality that can be built on per typical Puppet practice to fashion larger abstractions either within the Puppet language or in Ruby. Puppet is explicitly designed to be a highly extensible framework, therefore additional resource types are easy to write, distribute, or find on the Puppet Forge.
All of this, combined with the significantly lower per-node price, make Puppet Enterprise a compelling choice for hybrid Windows/Unix/Linux IT environments, an agile alternative to SCCM, and a tool complementary to Group Policy.
Q: Can the puppet master role be installed on a Windows machine?
A: Puppet Enterprise officially supports Windows Server 2008 R2 and Windows 7, but for the agent role only. The puppet master role must reside on a Linux-based system such as Debian or RedHat.
It’s worth noting that while support is provided only for Server 2008 R2 and Windows 7, the Puppet agent has been demonstrated to run in an unsupported capacity on most Windows releases from Server 2003 up (including 2003, 2003R2, 2008, Vista, and Windows 8).
Q: Does MCollective extend to Puppet on Windows?
A: The MCollective product works on Windows, but MCollective is not included in the Puppet Enterprise 2.7.0 installer for Windows. This is one of the last major bridges in terms of Puppet Enterprise Windows feature-parity with the Linux platforms, so it’s definitely a release we’re working towards.
Q: What plans do you have to help increase availability of Windows-capable modules on Puppet Forge?
A: I pinged Ryan Coleman, Product Owner of the Puppet Forge, for an answer on this question. Below was his response:
“There are two primary drivers of Windows-focused content on the Puppet Forge right now: Puppet Labs development and community contributions.
Modules such as puppetlabs/registry and puppetlabs/dism are examples of the kinds of content we build to enable Windows users with Puppet. Because we have limited resources to devote to these kinds of projects, most of our contributions come from the community. Thankfully, the community has been steadily contributing Windows content to the Puppet Forge. The following search produces 15 results, each a useful module for working with Windows and Puppet: http://forge.puppetlabs.com/modules?q=windows
My responsibility as Product Owner of Puppet Forge is to continue to enable the community to succeed and contribute great content. To that end, I do a number of things to help. I speak at conferences, write blog posts on how to more easily write great modules, and directly mentor community members on their modules when asked. In addition, I’m always on the lookout for work community members have done, and help them get that content onto the Puppet Forge. We also have the module bounty contest which asks the community to provide content we feel is lacking and awards the best contributors. If you have specific needs that aren’t being met, please email ryan@puppetlabs.com with what you’re looking for and I’ll do my best to get a solution onto Puppet Forge.”
Q: Are there any limitations on deploying Puppet modules on Windows?
A: There are no architectural limitations with regards to deploying modules for Windows nodes. Platform support varies between individual published modules depending on the effort invested by the relevant author and contributors, but there are no big-picture concerns specific in any way to modules and Windows.
Q: What techniques do you recommend to avoid having to put paths to particular MSI/EXE installers for particular versions in Puppet modules for Windows?
A: Puppet is about managing the configuration state on a system and leveraging whatever utilities are available in the environment to perform the actual work. On the Linux side, Puppet leverages Yum or Apt repositories to perform the install media retrieval. On the Windows side, there currently isn’t a ubiquitous abstraction that we can leverage in a package provider so much of the logic of schlepping around media that Puppet can use to install software has to be specified as part of the configuration. There are a variety of approaches that can be used. I’ll mention two of them here.
In an environment with Windows file shares available a “depot” can be set up containing the installation media for all managed software. Puppet can be used to connect to this software depot and use it for package sources. The following pseudo-code demonstrates the pattern.
Because I’m not currently aware of a native type/provider for mounting windows shares, an exec resource is used as a stand-in directive to ensure that the depot location is mounted and available. The exec could relatively easily be replaced with a full native “net_use” type and provider written along the same lines as the simondean/net_share module from Puppet Forge.
# The depot share path and connection information. Could be stored
# in hiera instead
$username = 'domain\puppet'
$password = 'passw0rd'
$share = '\\depot\packages'
$safeshare = regsubst($share, '[\\$]', '.', 'G')
# The `net use` command for the depot_mount exec resource
$netuse = 'C:\Windows\system32\cmd.exe /c net use'
# An exec resource that makes sure the depot is connected. If it's
# already connected, do nothing. If it isn't connected, connect
# to it.
exec { 'depot_mount':
command => "$netuse $share /USER:$username $password",
unless => "$netuse | FindStr /R \"^OK.*$safeshare\"",
}
# A package resource. Note specifically that it lists the
# depot_mount exec as a requirement.
package { 'Orca':
ensure => installed,
source => "$share\\orca\\orca-3.0.3790.msi",
install_options => { 'INSTALLDIR' => 'C:\orca' },
require => Exec['depot_mount'],
} |
When the depot pattern doesn’t work or isn’t ideal for a particular environment, another alternative is to use Puppet to ensure that a local copy of the installation media is available before ensuring the package. One way is to ship the files directly from the puppet master node, using it as a fileserver. Another option is to use Puppet Forge modules for alternative file sources, e.g. branan/s3file.
Any of these approaches can be wrapped in a defined type that will use logic or active mechanisms to locate an appropriate package source.
Click here for more information on the type/provider abstraction and how it can be used to build alternative means of ensuring that packages are installed if there is a native utility or repository software that you would like to use with Puppet for retrieving and installing packages.
As I mentioned above, there were a number of questions asked at the webinar. Stay tuned for part II of the Puppet Enterprise and Windows blog series, or check out the resources below.
Good testing of Puppet code involves different approaches and techniques that can be complex and time consuming. You can test your manifests’ syntax with puppet parser validate, verify their code style with puppet-lint, test modules logic and behaviour at catalog level with rspec-puppet, and check the actual effect of a Puppet run on dispensable virtual machines managed by a tool like Vagrant.
You might be a beginner to Puppet looking to test and play with Puppet code, or you might be an experienced Puppet user and modules author who needs to test modules on different operating systems. Either way, you ought to try Example42′s Puppet Playground.
The Puppet Playground is essentially a Vagrant multi-VM environment for masterless Puppet provisioning, with some tools and defaults that ease quick and safe Puppet code playing and multi-OS testing of modules and toasters (appliances), based on librarian-puppet. To use it, you need Vagrant running on your (possibly smart) computer and follow the instructions below for a quick, easy and (hopefully) joyful way to play and work with Puppet. Let’s get started.
Clone the GitHub repository to a work directory of your choice (here, it’s puppet-playground):
git clone https://github.com/example42/puppet-playground.git puppet-playground |
Move into the newly created directory, from this point all commands are relative to this path:
cd puppet-playground |
What you have is a normal Vagrant multi VM environment:
vagrant status |
This is enough to play with Puppet in Masterless mode:
You can write your Puppet code in manifests/init.pp, and place your modules in the modules/ directory, or you can use the bundled ./play command, which eases common operations.
You have various alternatives on how to work with Puppet Forge modules in the Puppet Playground:
puppet module install <modulename> --modulepath modules/ |
So, for example, to install Puppet Labs’s apache module type:
puppet module install <modulename> --modulepath modules/ |
./play setup example42 |
This initializes the modules dir with the Example42 NextGen modules directly cloned from GitHub.
gem install librarian-puppet
./play status
./play list
./play install <toaster> |
Whatever method you use to populate the modules’ directory, you can test your Puppet code using Vagrant commands. First, review (if you want) the default Vagrantfile provided by puppet-playground. You will see a normal MultiVM setup with masterless Puppet integration.
cat Vagrantfile |
You can see the available VMs with:
vagrant status |
Expect an output like the one below:
Current VM states: Test_Centos6_64 not created Test_Ubuntu1204_64 not created Test_Ubuntu1004_64 not created Test_Ubuntu1004_32 not created Test_Debian6_64 not created Test_SuseLinux11_64 not created Test_OpenSuse12_64 not created ToFix_Solaris10_64 not created ToFix_FreeBSD9_64 not created ToFix_OpenBSD5_64 not created ToFix_Centos5_64 not created ToFix_Centos5_32 not created ToFix_Centos4_64 not created ToFix_Ubuntu1104_64 not created ToFix_RedHat6_64 not created ToFix_ScientificLinux6_64not created |
Boxes with the Test_ prefix have successfully been tested on an updated Vagrant/VirtualBox installation, the ones with ToFix_ have had some problem for a smooth automated Puppet run. This list is going to be updated and corrected with time, hopefully with the help of the community.
You can run any of the provided Vagrant boxes with:
vagrant up Test_Centos6_64 |
This may take some minutes, the first time you run it, to download the base box from the Internet.
Once created the VM, you can connect to it with:
vagrant ssh Test_Centos6_64 |
Note that at the moment headless mode is disabled, so you’ll see the VirtualBox console window pop up. If you encounter problems with ssh, you should be able to login with user ‘vagrant’ and password ‘vagrant’ and then sudo -s.
To exit from the shell on the VM
vm# exit |
To restart your VM:
vagrant reload Test_Centos6_64 |
To destroy and rebuild from scratch:
vagrant destroy Test_Centos6_64
vagrant up Test_Centos6_64 |
Once you see that Vagrant is doing its job, you can start to play with Puppet code: edit the default Puppet manifest applied on the boxes:
vi manifests/init.pp |
This is your test playground: add resources, use modules, declare classes…
You can place simple resources like:
file { 'motd':
path => '/etc/motd',
content => 'Hi there',
} |
Or use modules you’ve previously placed in the modules/ dir.
class { 'wordpress':
} |
Or whatever Puppet code you might want to apply.
If you need to provide custom files, the sanest approach is to place them in the templates directory of a custom “site” module (call it ‘site’ or however you want) and refer to it using the content parameter:
file { 'motd':
path => '/etc/motd',
content => template('site/motd'),
} |
This will populate /etc/motd with the template placed in
puppet-playground/modules/site/templates/motd.
To test your code’s changes on a single node, you have two alternatives:
vagrant provision Test_Centos6_64 |
vagrant ssh Test_Centos6_64 |
Once you’ve logged in the VM, get the superpowers and run Puppet:
vm# sudo -s
vm# puppet apply -v --modulepath '/tmp/vagrant-puppet/modules-0' --pluginsync /tmp/vagrant-puppet/manifests/init.pp |
You can also edit the manifests file both from your host or inside a VM:
vi manifests/init.pp |
vm# sudo -s
vm# cd /tmp/vagrant-puppet/
vm# vi manifests/init.pp |
If you have more VMs active you can test your changes on all of them with a simple:
vagrant provision |
The Puppet Playground provides also integration with librarian-puppet for a smarter management of Puppet code and the relevant modules. The intention is to provide a rich set of predefined toasters (or call them appliances or bundles of ready-to-cook recipes for a more or less complex setup), which can be quickly tested on different operating systems.
You can experiment with them with the included play command. It basically copies configurations from the selected toasters/ directory to manifests/init.pp and Puppetfile and runs librarian-puppet to automatically install the required modules in the modules/ directory.
To show the status of the playground
./play status |
To show the available toasters for ./play install:
./play list |
To install a specific toaster via librarian-puppet:
./play install garethr-riemann |
To run the current playground (same as vagrant provision)
./play run |
To cleanup the whole playground (Beware all the existing changes will be wiped off)
./play clean |
To run puppi commands on all the active boxes (note: Puppi must included in the playground)
./play puppi check
./play puppi info |
Puppet Playground has just been released, and it’s already useful. However, there is room for growth and improvement.
Everybody can help, with little effort, to make the Puppet Playground better:
Any relevant pull-request to https://github.com/example42/puppet-playground via GitHub is very welcomed, the Puppet Playground is more fun if there are more kids around :-)
Alessandro Franceschi
http://www.example42.com
At 5:53 PM on December 19th, the Puppet Forge hit a patch of turbulence. This resulted in intermittent service failures and extremely slow responses, and lasted until 2:42PM on December 20th – almost 21 hours, all told. We’d like to apologize for the performance issues and provide you with some explanation of what went wrong and what we’re doing about it.
When we launched the new Puppet Forge last month, one of our underlying concerns was that we wanted real data about what was happening, both about user behavior and about application and server behavior. (For a great talk about the importance of gathering good metrics, I will refer you to Coda Hale.) To that end, the new Puppet Forge was set up to report a large number of measurements to Graphite via StatsD. And, somewhat more recently, we started using New Relic to monitor both the application stack and the servers.
These tools have not only provided us with fascinating data but have already helped us diagnose and squash a number of bugs, and we plan to add additional instrumentation as we find gaps in our knowledge.
The New Relic instrumentation and the StatsD library take somewhat different approaches in how they send reports. We use New Relic with SSL enabled, so all communication takes place over an encrypted HTTP connection. To combat the performance cost inherent in this approach, the New Relic communication takes place independently of the main application thread, so response times are unaffected. The StatsD library we’ve employed takes advantage of the extremely low cost of UDP packet transmission and sends measurements from the application thread (in our case, over a secure tunnel to our StatsD server).
Sending StatsD measurements in the application thread tends to work out just fine, since UDP is basically a “fire-and-forget” operation – the sender puts its message in a properly addressed bottle, drops it in the ocean, and assumes destiny will cause the message to arrive; on the off-chance that this isn’t a quick process, the library simply aborts sending the metric after 0.1s. This timeout mechanism does potentially cause some data loss, but not necessarily more than you might expect from UDP itself.
At 4:08PM on the December 19th, the Puppet Forge began to show increases in the average request time, which continued to worsen as time went on. New Relic recorded the first serious failures about 30 minutes later, and the average response time soon passed 1.5s, which it stayed above for the remainder of the event.
Around the same time, our Ops team deployed some infrastructure changes. Of particular interest was a VPN tunnel from our cloud-based VMs back inside our local network, which our Linode-based Puppet Forge servers also received this update. While we’re still a little fuzzy on the details, this change caused some trouble for the StatsD library, which began timing out when trying to send its measurement reports, and a significant number of the packet transmissions began to timeout, meaning that each measurement we made was adding around 100ms to the response time. Individually or in small quantities, that is still a reasonably small delay, but our servers may sometimes send hundreds of such measurements per request, and those delays began to add up quickly.
Average Response Times
While we now know that the StatsD timeouts were the cause of our outage, our search for the failure was frustrated by a lack of data. New Relic let us see clearly what our application was doing, but all reports indicated that we were just spending a lot of time executing our Ruby logic. Except, we were barely utilizing any processor time. We considered that we might have been network bound, but there was almost no traffic there. Database queries hadn’t gotten any slower. Our memory usage was around 50% of physical RAM. We weren’t swapping, and we weren’t reading from or writing to disk. And we were not trying to serve an unusual request load.
It was rather as if the Puppet Forge was simply calling sleep() in the middle of requests.
When we began looking for causes outside the Puppet Forge as a system, we stumbled across the recent VPN changes. A quick test showed that disabling metrics gathering did relieve the problem, and an educated guess led us to disable the StatsD reporting which caused an almost immediate return to normalcy.
For the time being, we’re leaving our StatsD monitoring disabled. While this is a non-ideal situation for us to be in, we’re unwilling to flip that particular switch back on until we can demonstrate that doing so has a negligible effect.
The VPN, intended to be a low-risk win for everyone, ended up interacting poorly with the Puppet Forge’s particular configuration, in part because not everyone knows how the application works and nobody knows everything about it. We (the engineers) will be working with our Ops team to build a better shared understanding, an ongoing dialog, and clear documentation around how the Puppet Forge works, so that all of us will be more aware of how changes will affect the system, and we’ll all be better able to diagnose and solve the next set of problems we face.
Above all, we failed to communicate well that we were having difficulties. To solve that, we will be launching a site to act as a central source for information about service interruptions and outages on the Puppet Forge, including live monitoring data and diagnostic updates. This will enable us to communicate any problems we have – and their resolutions – much more quickly and easily. We’ll follow up on this post when we have a link for you.
We recognize that every failed request is an inconvenience to someone who is trying to get things done, and we’re working hard to ensure that our service provides the dependability you need. Events like this are never fun, but we are working deliberately to learn from them when they do happen, so that we can continue to provide a service that the entire community can rely on. If you have any questions or concerns about the recent Puppet Forge outage, please submit feedback to forge@puppetlabs.com.
| Purpose | Install and configure a Minecraft server on Linux |
| Module | branan/minecraft |
| Puppet Version | 2.7+ |
| Platforms | All platforms that provide cURL and Java |
Minecraft is a popular game in which you can explore and create a world made of blocks. One of its features is a multiplayer mode that allows playing the game with others over the Internet on public or private servers.
There are a number of public Minecraft servers, but they are not always the best solution. If you have young children or just don’t like playing online with strangers, you might want to have your own Minecraft server. With this Puppet module, you can start playing Minecraft with your friends and family in just a few minutes.