We have three major product teams at Puppet Labs: Platform, Applications, and Content. Content is the Puppet Forge and the toolchain around it, Applications are essentially Puppet Enterprise and the various capabilities we ship with it, but it’s Platform I want to talk about today. This is the team that manages the tools, services, and libraries that everything else is built on, like Puppet, MCollective, and Razor.

There’s a basic strategy to both the entire collection and each project within the collection, and one of the main points of that strategy is “system tools with comprehensive data.” We’ve spent a lot of time this year talking about data, but I want to focus on the ‘system tools’ bit. Most of my early computer usage was just breaking and fixing my computer, so I kind of cut my teeth on the system tools necessary to do both. As a result, I have a clear sense of what makes a system tool different from other software.

People started asking me what a system tool was and how they’d know when they had it. I was surprised to learn that system tools are a lot like pornography—which is to say, there isn’t a clear, agreed-upon definition, but I know it when I see it. To me, a system tool is a combination of fast, simple to use but hard to write, completely pervasive, with a clear and narrow purpose in life. In Unix, for instance, all of the tools are supposed to do one thing and do it well. Puppet isn’t really any of those characteristics right now, except maybe pervasive, but I want it and all of our other tools to be so.

As I talked about this more, I realized that the world of computing seems to have lost its appetite for system tools. I love git, and I think it absolutely qualifies as a system tool, but it’s one of very few such tools built recently. In fact, I’d say git‘s long-time refusal to ship with a web service makes it much more like a system tool than it otherwise would be; it has a more clearly defined use, and someone else has to provide all of the complicated networking and such. Who remembers the hell of setting up SVN in the early days? *shudder*

So what happened? Whither the new system tools? I think a lot of it comes down to services—the era of the system tool generally had local data with no networking needed to do its job, but now all key data is owned by some kind of centralized process, and any serious work requires access to that service. This means you’re writing a client, not a stand-alone tool, and that client is probably useless without the service. This is one of the great things about git: It works great even on an airplane. Coding on a plane, and having a clean commit history at the end, was git’s killer feature for me for a long time—I was flying all around the world while trying to maintain Puppet when we switched to git, and it was a huge productivity gain for me.

There is such a thing as a system service, and PuppetDB is one, but services are a whole different world. We need to build those too, but I want tools that don’t require services.

So what would it mean for Puppet in particular, and our tools in general, to be more like a system tool? Well, first and foremost they would be faster and use fewer resources. Some would argue they would also have maddening interfaces and be impossible for normal people to use, but with sufficient focus on user experience we can hopefully dodge that bullet. I would also love the key functionality in Puppet to be available as a library to other languages, which would require that functionality be built in a compiled language that supports relatively standard language bindings.

And I think that points to the real problem. We don’t have great system languages that match our current problems. C is still the standard, but the world has moved on and most programmers would prefer to write in a higher-level language—things like Java in the enterprise, and Ruby, Python, Scala, and Clojure in the early adopter world. When people are building new applications these days, they generally value developer productivity more than just about anything else—I know this is why I chose Ruby for Puppet. Given the crazy hardware stats these days, that’s not a bad bet. What it means, though, is that we’re not often writing in languages that are automatically suited for system tools.

Google is making a real effort at turning Go into a modern, popular systems language, and I hope they have success. I haven’t seen enough adoption yet to make me confident, and I’ve already gone through one cycle of adopting a language no one else is using (Ruby was chosen for Puppet before Rails came out).

I’m confident that we’ll achieve our main goals—that most of the tools in our platform will fit the “I know it when I see it” test of system tools—but I wish there were a more obvious path from where we are to where we want to be.

2 Comments

Leave a Response