Friday, October 23, 2009

Caveat Emptor

I work for AppNexus. It's an enterprise level cloud computing comany, and I'm the tech ops guy. When I try to explain what I do for a living, the answer changes depending on the audience. Here are some possible explanations:

  1. I'm in IT. I work with computers.
  2. I make lots of computers do "things".
  3. I'm the director of glue!
  4. My job is to automate myself out of my job.
  5. I use and write software to deploy and run other software, monitor applications, and collect metrics from those applications for analysis.

Oh, yes. It's true. I work with computers on a daily basis. In fact, not a single work day goes by that doesn't necessitate me using my computer. That day will come when we all go on that corporate retreat and do zip-lines in the woods, I suspect. I'm not holding my breath.

Let me go a bit deeper, for those of you reading the last explanation and bobbing their head ever so slightly. I'm in charge of technical operations from the application level on up. AppNexus has systems administrators who deal with the hardware layer and OS layer, although unsurprisingly there can be a fuzzy area at the OS layer in the handoff from the systems administrators and me. They deal with things like datacenter layout, racks, servers, networking, Xen kernels, et cetera. Bless 'em. It's not an easy job, and it's not my area of expertise. Once the hardware is available via AppNexus's sweet cloud APIs, I get to play.

Taking advantage of OSS, I'm building on the shoulders of giants to create a development and production environment that is as automated as possible. When you've got more than a small handful of machines to take care of, you simply don't want to do it by hand. When you've got applications that can never be down, you can't take them all down at once to upgrade them. The software I use (along with a plethora of code that I've written) allows our company to check their code out of subversion, package it up, and deploy it to our environments. The software handles versioned configuration, versioned applications, rollbacks, et ceterea. It makes sure that each machine is monitored properly.

The novice reader may wonder why one way I explain my job is to say that my job is to automate myself out of a job. Seems like a dumb way to work, considering the economy, perhaps? The reality is that the work will never be done. I just don't want to do the same thing over and over again. I want to turn the things that I do into a commodity that the rest of the company takes for granted because it's reliable and makes their life easier.

I'm leveraging OSS to do what I do. I'd like to mention a few projects in particular, without whom my job would be so much more difficult. To everyone contributing to the Open Source community, thank you.
  • Perl - My coding is almost exclusively in Perl, with a smattering of PHP.
  • Puppet - I use this to ensure state on all of my machines. Puppet handles our software rollouts, as well.
  • Nagios - I use Nagios to monitor our software stack and send out alerts when there's trouble.
  • Ganglia - I use this to collect metrics centrally
  • Graphite - While Ganglia is great for collecting metrics, I've found Graphite to have a wonderful interface for displaying those metrics
I hope to share insights about my world to help others that are doing similar things. Do note, however, that the opinions and suggestions expressed herein are my own and are not endorsed by AppNexus. In other words, blame me, not them, if there's anything here that causes your wonder-app to crash. However, I hope that everything you read here falls under the category of "best practices". I won't suggest you run a magnet over your hard drives to align the bits for higher performance. And for crying out loud, I won't suggest that you go and re-invent the wheel when you're doing similar tasks. Enjoy.

No comments: