Introduction

The Marionette Collective AKA MCollective is a framework to build server orchestration or parallel job execution systems. Primarily we’ll use it as a means of programmatic execution of Systems Administration actions on clusters of servers. In this regard we operate in the same space as tools like Func, Fabric or Capistrano. We’ve attempted to think out of the box a bit designing this system by not relying on central inventories and tools like SSH, we’re not simply a fancy SSH “for loop.”

MCollective use modern tools like Publish Subscribe Middleware and modern philosophies like real time discovery of network resources using meta data and not hostnames. Delivering a very scalable and very fast parallel execution environment. To get an immediate feel for what I am on about you can look at some of the videos on the Screencasts page and then keep reading below for further info and links. We’ve also created an Amazon EC2 based demo where you can launch as many instances as you want to see how it behaves first hand.

What is MCollective and what does it allow you to do?

  • Interact with small to very large clusters of servers
  • Use a broadcast paradigm for request distribution]. All servers get all requests at the same time, requests have filters attached and only servers matching the filter will act on requests. There is no central asset database to go out of sync, the network is the only source of truth.
  • Break free from ever more complex naming conventions for hostnames as a means of identity. Use a very rich set of meta data provided by each machine to address them. Meta data comes from Puppet, Chef, Facter, Ohai or plugins you provide yourself.
  • Comes with simple to use command line tools to call remote agents.
  • Ability to write custom reports about your infrastructure.
  • A number of agents to manage packages, services and other common components are available from the community.
  • Allows you to write simple RPC style agents, clients and Web UIs in an easy to understand language - Ruby
  • Extremely pluggable and adaptable to local needs
  • Middleware systems already have rich authentication and authorization models, leverage these as a first line of control. Include fine grained Authentication, Authorization and Auditing of requests.
  • Re-use the ability of middleware to do clustering, routing and network isolation to realize secure and scalable setups.

Pluggable Core

We aim to provide a stable core framework that allows you to build it out into a system that meets your own needs, we are pluggable in the following areas:

MCollective is licensed under the Apache 2 license.

Next Steps and Further Reading

Introductory and Tutorial Pages:

Internal References and Developer Docs

  • Finding it hard to do something complex with Simple RPC? See WritingAgents for what lies underneath
  • Role based security, authentication and authorization using ActiveMQ
  • Structure of Request and Reply messages

Download MCollective