Day 52: Happy 10th Birthday, OpenStack!

I just saw this announcement from the OpenStack Foundation about OpenStack’s 10th birthday! Yes, 10 years ago this week was the first OpenStack Summit, in Austin, TX, with the public announcement the following week at O’Rielly OSCON. Yet most people don’t know that I played a very critical role in the beginning!

OpenStack began as a joint venture between Rackspace (my employer at the time) and NASA. I was on the team at Rackspace that developed and supported its aging cloud compute services, and we were looking to develop something from scratch that could be much more scalable than our current system. Around that time Thierry Carrez saw an announcement from a group at NASA about their development of a compute virtualization system, and suggested to the powers that be at Rackspace that this might be a better way to go instead of developing the whole thing ourselves. From that followed a lot of discussion among the executives at Rackspace, as well as some conversations with NASA, and the conclusion was that we would team up. One of the first things to do was to get the developers for both groups together to discuss things from a more technical perspective. And this is where I believe that I made a critical decision that, had I chosen wrong, might have resulted in OpenStack never happening.

The NASA team was a consulting group, Anso Labs, and they were arriving in San Antonio, and we had plans to take them out to dinner, but no idea where. It was then that I suggested The Cove, a local place with lots of outdoor seating, a relaxed atmosphere, good beer, and delicious food. We had a great time that evening, and we all got to know each other. Had we been in a more conventional restaurant, people may have only gotten to know the people sitting next to them, but since The Cove is open seating, we moved around a lot, talking about both the technical stuff and personal stuff.

Over the next few days we began reviewing the code and exchanging idea on what needed to be developed next, and those discussions went very smoothly, getting a lot done in a short period of time. I still maintain to this day that without that first night having beer and food at The Cove, OpenStack might never have become the success that it did.

You’re welcome.

Day 35: Chopped Candidate

Have you ever watched the TV show Chopped? If you haven’t, it’s a competition among 4 chefs. There are 3 rounds, and after each round, one of them is chopped (eliminated), until one remains. The winner gets a cash prize. This would seem like a good way to determine who is the best of the group, right?

The problem is how the competition is run: each round the chefs are given a basket of “mystery” ingredients that they can’t see until the round begins. And more often than not, the basket contains, shall we say, “odd” combinations. One such basket contained blood orange syrup, the African spice blend ras el hanout, hot cross buns, and lamb testicles. The chefs can add other staple ingredients, but those four flavors have to be featured prominently in the result.

And if that isn’t difficult enough, there is a time limit that is always ridiculously short. The chefs had 20 minutes to create an appetizer from the basket I described above: 20 minutes to create a recipe, determine what other ingredients to add, prepare and cook the food, and then plate it for a beautiful presentation.

I must confess that I find the show very entertaining, and have watched countless episodes. And I’m not alone: the show has been running for 44 seasons over the past 11 years. But let me ask you: if you were opening a restaurant, would this be the way you would select your head chef? I would hope not! Any restaurant that would spring surprises on their chefs and expect them to deliver first-rate food in impossibly short time limits wouldn’t last very long.

Which brings me to the point of all this: if you are interviewing for a programmer, do your interviews actually determine how well they would be able to work in your team? How positive their contribution will be?

Making a candidate live code a solution to a problem they’ve never seen before in a short period of time with people watching their every keystroke is the software development equivalent to being on Chopped. I certainly hope that your work environment isn’t anything like that. So why would you think that a live coding session in an interview tells you anything about their potential?

What artificial scenarios like Chopped or live coding interviews do is test a candidate’s ability to handle stress. Personally, I’ve never had a problem with live coding, but then again I’ve never had test anxiety in school, either. I’ve seen many talented developers choke under those circumstances, but that doesn’t mean that I wouldn’t want to have them on my team.

What does it say about your company as a place to work if the bar they have to clear is how well they can handle high levels of stress?

When I first started interviewing candidates when I was at Rackspace, the standard was to have one interviewer do a live coding challenge, and another ask one of those bizarre, abstract brainteasers (“Walk us through your thoughts…”). Once again, these practices just show how nervous someone is in what is already an inherently stressful situation. That link includes a juicy quote:

These types of questions are likely to frustrate some interviewees so watch out for those who aren’t willing to play the game. It’s an interview after all and you make the rules.

Mark Wilkinson, head of recruitment, Coburg Banks

It’s all a game to him, and if asking questions with no right answers eliminates potentially good candidates, tough. It sounds like he is more interested in seeing who can tolerate being bullied than finding the best people for his company.

After sitting through some of these types of interviews at Rackspace, I campaigned internally to change these practices, because I saw some intelligent and capable candidates get flustered and end up looking dumb. I found that there are better ways to determine if someone is a good addition to your team. Perhaps I’ll elaborate more about these in a future post…

Day 2: What is an Artist?

I consider myself an artist, as do many others. But that title is thrown about quite a bit, and its meaning has been diluted. So let’s look at it.

In my mind, the essence of being an artist is creating something that not only is original, but captures or excites the interest of others. Often someone who paints or draws is automatically called an “artist”, and their product is called “art”. But that’s way too low a bar to set for that title. And for the record, I can’t paint or draw with any skill level whatsoever, and admire those who can.

I am a photographer. I recognized my attraction to photography as a child, and began taking it seriously in college. I attended a photography school for two years, and learned all about portraiture, lighting, studio arrangements, different films (yes, it was all film then!), color, tone, and print media. While I was able to master those techniques, I wouldn’t say I created art. Well, maybe with a few exceptions, such as:

Potato, 1981
Potato, ©1981

What I found I enjoyed the most was simply walking around and looking. Things would strike me as visually interesting, and I would use my photographic technique to record them in a way that made interesting images. For example, my photo school was about 6 miles from my home, and I used my bike to get there. Shortly into my first semester, though, someone cut the chain I had locked it with and stole my bike. I now had to walk a mile to a bus stop, take the bus to downtown, and then walk another mile to the school. As I was walking I would look around, and things would occasionally catch my eye. Since I was carrying my camera, I began to record them. At the end of the semester we had to produce a portfolio, and so I created one called Sidewalks – all of the images were taken of sidewalks I walked on my way to/from school.

Not only was the portfolio well-received, it was noticeably different than the others. Most of the others were what I would call “traditional” photographic subjects: sunsets, landscapes, weathered barns, pets, etc., but mine were anything but traditional. So not only did the portfolio receive a good grade, it was chosen to be displayed around the campus – my first exhibition!

This is when I began to understand my creative process: instead of creating a scene by arranging items, or posing people, or any other conscious construction of the subject in front of the camera, I would explore the world as it existed, and find beauty in what others don’t see. I take special pride in images that are unremarkable in themselves, but from which I can create an interesting image. As an example:

Castle Sidewalk, ©2011

Back in 2011 I worked at Rackspace, and the headquarters was in a refurbished shopping mall, nicknamed “The Castle”. Near the main entrance two different-colored sidewalks come together. You can see it in the center of this Google Maps view.

Over a thousand people walked past this point every day. I happened to walk past it on my morning break, looked down, and was struck by what I saw. I didn’t have a camera with me… or did I? In my pocket was my iPhone 4, so I took this photo with my phone. I’ll save my thoughts on photography gear for another day, though…

This is why I consider myself an artist: thousands walked past that spot that day, but only I saw this bit of transient beauty, and was able to capture it in a way that others could enjoy. Being able to take photographs, or paint pictures, or play piano, or sculpt clay – those are examples of crafts. But when you are able to use your craft to create something that moves other people – well, then I consider you an artist.

Moving On

It’s been a great run, but my days in the OpenStack world are coming to an end. As some of you know already, I have accepted an offer to work for DataRobot. I only know bits and pieces of what I will be working on there, but one thing’s for sure: it won’t be on OpenStack. And that’s OK with me, as I’ve been working on OpenStack in one form or another for 10 years now.

Wait a moment, you say – OpenStack is only 9 years old! Well, before the OpenStack project was started, I worked on Swift briefly when it was an internal, proprietary project at Rackspace. After that I switched to the Cloud Servers team, which was the team that started Nova with NASA. So yeah, it’s been a full decade. That’s a loooonnnnggg time to be on any development project!

So the feelings of burnout combined with the shift away from OpenStack within IBM made moving to DataRobot a very attractive option. And after having done several video interviews with the people there and getting their impression of life at DataRobot, I’m that much more excited to be joining that team. I’m sure that for the first few months it will be like drinking from the proverbial fire hose, and that’s perfectly fine by me. It’s been much too long since I’ve pushed the reset button for my career.

Over these past 10 years I have made many professional contacts, some of whom I consider true friends. I will miss the OpenStack community, and I hope to run into many of you at future tech events – PyCon, anyone?

Why OpenStack Failed, or How I Came to Love the Idea of a BDFL

OK, so the title of this is a bit clickbait-y, but let me explain. By some measures, OpenStack is a tremendous success, being used to power several public clouds and many well-known businesses. But it has failed to become a powerful player in the cloud space, and I believe the reason is not technical in nature, but a lack of leadership.

OpenStack began as a collaboration between Rackspace, a commercial, for-profit business, and a consulting group working for NASA. While there were several companies involved in the beginning, Rackspace dominated by sheer numbers. This dominance was a concern to many companies – why should they contribute their time and resources to a project that might only benefit Rackspace? This fear was not entirely unfounded, as the OpenStack API was initially created to match Rackspace’s legacy cloud API, and much of the early naming of things matched Rackspace’s terminology – I mean, who ever thought of referring to virtual machines as “servers”? But that matched the “Cloud Servers” branding that Rackspace used for its cloud offering, and that name, as well as the use of “flavor” for instance sizing, persist today. The early governance was democratic, but when one company has many more votes than the others…

The executives at Rackspace were aware of this concern, and quickly created the OpenStack Foundation, which would be an independent entity that would own the intellectual property, helping to guarantee that one commercial company would not control the destiny of OpenStack. More subtly, though, it also engendered a deep distrust of any sort of top-down control over the direction of the software development. Each project within OpenStack was free to pretty much do things however they wanted, as long as they remained within the bounds of the Four Opens: Open Source, Open Design, Open Development, and Open Community.

That sound pretty good, right? I mean, who needs someone imposing their opinions on you?

Well, it turns out that OpenStack needed that. For those who don’t know the term “BDFL“, it is an acronym for “Benevolent Dictator For Life”. It means that the software created under a BDFL is opinionated, but it is also consistently opinionated. A benevolent dictator listens to the various voices asking for features, or designing an API, and makes a decision based on the overall good of the project, and not on things like favoring corporate interests for big contributors, or strong personalities that otherwise dominate design discussions. Can you imagine what AWS would be like if each group within could just decide how they wanted to do things? The imposition of the design from above assures AWS that each of its projects can work easily with others.

The closest thing to that in OpenStack is the Technical Committee (TC), which “is an elected group that represents the contributors to the open source project, and has oversight on all technical matters”. Despite the typical meaning of “oversight”, the TC is essentially a suggestion body, and has no real enforcement power. They can spend months agonizing over the wording of mission statements and community goals, but shy away from anything that might appear to be a directive that others must do. I don’t think the word “must” is in their vocabulary.

They also bend over backwards to avoid potentially offending anyone. Here is one example from my interactions with them: one of the things the TC does is “tag” projects, so that newcomers to OpenStack can get a better idea how mature a particular project is, or how stable, etc. One of the proposed tags was to warn potential users that a project was primarily being developed by a single company; the concern is that all it would take is one manager at that company to decide to re-assign their employees, and the project would be dead. This is a very valid concern for open source projects, and it was proposed that a tag named “team:diverse-affiliation-danger” be created to flag such projects. What followed was much back-and-forth on the review of the proposal as well as in TC meetings about how the tag name was negative and would hurt people’s feelings, how it would be seen as an attack against a project, that it was more of a stick rather than a carrot, etc. All of this hand-wringing over an objective measurement of the content of a project’s current level of activity. (Epilogue: they ended up making it a positive-sounding tag: “team:single-vendor”, and no tears were shed)

Having ineffective leadership like the TC has ripple effects throughout all of OpenStack. Each project is an island, and calls its own shots. So when two projects need to interact, they both see it from the perspective of “how will this affect me?” instead of “how will this improve OpenStack?”. This results in protracted discussions about interfaces and who will do what thing in what order. And when I say “protracted”, I don’t just mean weeks or months; some, such as the CyborgNova integration discussions, have dragged on for two years! I cannot imaging that happening in a world with an OpenStack BDFL. This inter-project friction slows down development of OpenStack as a whole, and in my opinion, contributes to developer dissatisfaction.

So what would OpenStack have been like if it had had a BDFL? Of course, that would depend entirely on the individual, but I can say this: it would have flamed out very quickly with a poor BDFL, or it would be a much better product with a much higher adoption with a good one. Back in 2013 I had predicted that OpenStack would eventually rival the commercial clouds in much the same manner that Linux now dominates the internet over proprietary operating systems. In the early days of the internet, the ability for people to download and play with free software such as the LAMP stack enabled people with big ideas but small budgets to turn those ideas into reality. OpenStack began in the early days of cloud computing, and it seemed logical that having a freely-available alternative to the commercial clouds might likewise result in new cloud-native creations becoming reality. It was a believable prediction, but I missed the effect that a lack of coordination from above would have on OpenStack achieving the potential to fill that role.

By the way, many people point to Linux and its BDFL, Linus Torvalds, as the argument against having a BDFL, as Linus has repeatedly behaved as an offensive ass towards others when he didn’t like their ideas. But ass or not, Linux succeeded because of having that single opinion consistently shaping its development. Most BDFLs, though, are not insufferable asses, and their projects are better off as a result.