Putting the Ops in DevOps

I was recently made aware of The Road To DevOps, a piece on the ActiveState blog, which reminded me that I’ve been meaning to write this post for a while, and finally got me to write it.

The author, Phil Whelan, posits a commonly-held false dichotomy: that you can come to DevOps via a developer-driven path, or an ops-driven path (and argues that the former is more satisfactory). There is (at least) one more way: a collaborative effort from both developers and operations engineers. What highlights the error is that he uses Etsy as an example of this developer-driven path to DevOps:

There are two roads to DevOps. The first is from the birth of hot new startups like Netflix and Etsy. This is generally a group of developers who have no desire to hire an Operations person. There are enough good open-source tools available and infrastructure services like Amazon Web Services are easy to consume. They have the know-how to build their infrastructure themselves, but they do not want the burden of operational maintenance, so they automate everything they can. They also build efficient test-driven pipelines that run all the way from development to production.

Etsy didn’t start out as a DevOps organisation. When I joined in 2009 we certainly had an operations person (we had a number of them, and they were great). We were (and still are) using our own hardware, not AWS.

Development was a sort of waterfallish agiley type situation. And it wasn’t working out all that well. Hiring John Allspaw (and later Kellan Elliott-McCrea) brought in a lot of hard-won knowledge about development and web operations from Flickr. Operations engineers and developers both embraced the idea that we could do better together, and we began, slowly, to transform how we did things. But at no point did we do away with Ops. (In fact, the post by Adrian Cockcroft that Whelan links to in his post is notably rebutted by John.)

Patrick Debois challenged me at DevOps Days Minneapolis to answer whether we can say “you’re doing it wrong” about DevOps, whether there is a True DevOps Path from which the benighted have fallen. I was, in the end, obliged to say no. I think there are core tenets of the DevOps philosophy (such as empathy, as described by Katherine Daniels) but multiple ways to solve the problems. But the nearest I can come to “you’re doing it wrong” is when people announce the death of Ops as a job function.

It’s true that in very small teams, a good operations engineer may be missing, and the developers may well do just fine with the operational aspects of the work. But when you come to scale your work, operations engineers are worth their weight in gold.

Heinlein may have said that specialization is for insects but in truth it’s very, very hard to excel in both the development and operational aspects of building and running a large computer system. These people do exist, but trying to find them and hire them is hard (and expensive). Moreover, beyond a certain (small) scale it’s actually woefully inefficient to have everyone doing everything. (I know the argument goes that AWS et al. are making it unnecessary to do traditional Ops but even without racking your own servers you’re still managing and tuning machines and traditional sysadmin skills are necessary.)

Professional developers and professional operations engineers complement each other, and in a DevOps environment as Etsy does it they’re working together side-by-side to achieve shared goals. I’ve tended to prefer the term “Dev/Ops cooperation” rather than “DevOps” (something Etsy’s then-CTO, now-CEO Chad Dickerson wrote about back in 2010 as we started on our own path to DevOps).

Devs do do some of the work that Ops would traditionally do, and Ops do ship code. Typically engineers exist on a spectrum between pure dev and pure ops—there are some who overlap both worlds, but they’re not the type we aim to hire exclusively. (And they’re also not their own special privileged “DevOps” team.)

But, fundamentally, “doing away with Ops” seems to me like madness. We gain so much from the professionalism, skills, and dedication of our Ops engineers. Working with them is one of the great privileges of working at Etsy.