Category: Technical

The OpenStack Big Tent and Magnum

One of the most heavily-attended design summit events at last week’s OpenStack Summit in Paris was on Magnum, a proposed service for containers that would integrate into the Nova compute service. It seems that any session at any conference these days that involves Docker attracts a lot of interest, as Docker is an amazing new way of approaching how we think about virtualization and achieving efficiencies of scale.

Disclaimer: I know Adrian Otto, the leader of the Magnum project, from my days at Rackspace, and genuinely like him. I have no doubt that he would be able to put together a team that can accomplish all that he is setting out to do with this project. My thoughts and concerns about Magnum would be the same no matter who was leading the project.

The goal of the Magnum session was to present its concept and proposed architecture to the Nova ganttteam, with the hope of being designated as the official Docker project in OpenStack. However, there was a lot of push back from many members of the Nova team. Some of it had to do with procedural issues; I learned later that Magnum had been introduced at the Nova mid-cycle meetup, and the expectations set then had not been met. I wasn’t at that meetup, so I can’t personally attest to that. But the overall sentiment was that it was just too premature to settle on one specific approach to something as important and fast-moving as Docker. While I support the idea of Magnum and hope that it is a wild success, I also think that world of Docker/containers is moving so fast that what looks good today may look totally different 6 months from now. Moving such a project into OpenStack proper would only slow it down, and right now it needs to remain as nimble as possible.

I wrote a little while ago about my thoughts on the current discussions on the Big Tent vs. Layers vs. Small Core (Simplifying OpenStack), and I think that the Magnum effort is an excellent example of why we need to modify the approach to how we handle projects like this that add to OpenStack. The danger of the current Big Tent system of designating a single effort as the official OpenStack solution to a given problem is that by doing so we might be discouraging some group with a different and potentially better solution from pursuing development, and that would short-change the OpenStack ecosystem long-term. Besides, a little competition usually improves overall software quality, right?

OpenStack Paris Summit – Growing Up

I’ve just returned from the 5-day-long OpenStack Summit, and after a very long day of travel, my brain is still slightly crispy, but I wanted to record some impressions of the summit. Since it was held in Paris, there are a lot of non-technical experiences I may write about, but for now I’ll limit my thoughts to those concerning OpenStack.

For those who don’t know my history, I was one of the original OpenStack developers who began the project in 2010, and participated in all of the early summits. After two years I changed roles in my job, which meant that I was no longer actively contributing to OpenStack, so I no longer was able to attend the summits. But now that I’m back as a full-time contributor in my role at IBM, I eagerly anticipated re-acquainting myself with the community, which had evolved since I had last been an active member.

First, let me say how impressive it is to see this small project we started grow into the truly international phenomenon it has become. The sheer number of people and exhibitors who came to Paris to be involved in the world of OpenStack was amazing: the latest count I saw was over 4,600 attendees, which contrasts with around 70 at the initial summit in Austin.

Second, during my hiatus away from active development on OpenStack many of the active core contributors to Nova have moved on, and a whole new group has taken their place. In the months leading up to the summit I got to know many of them via IRC and the dev email list, but had never met them in person. One thing about OpenStack development that has always been true is that it’s very personal: you get to know the people involved, and have a good sense of what they know and how they work. It is this personal familiarity that forms the basis of how the core developers are selected: trust. There is no test or anything like that; once you’ve demonstrated that you contribute good code, that you understand the way the various parts fit together, that you can take constructive criticism of your code, and that you can offer constructive criticism on others’ code, eventually one of the existing core members nominates you to become core. The other cores affirm that choice if they agree. Rarely have I seen anyone nominated for core who was rejected by the group; instead, the reaction usually is along the lines of “oh, I thought they were core already!”. As one of my goals in the coming year is to once again become a core Nova developer, getting to meet much of the current core team was a great step in that direction.

And lastly, while the discussions about priorities for the Kilo cycle were lively, there was almost none of the polarizing disagreements that were part of Nova’s early days. I believe that Nova has reached a maturity level where everyone involved can see where the weak points are, and agree on the need to fix them, even if opinions on just how to do that differed. A great example was the discussion on what to do about Cells: do we fix the current approach, or do we shift to a different, simpler approach that will get us most, but not all, of what the current code can do, but with a cleaner, more maintainable design. After a few minutes of discussion the latter path was chosen, and we moved on to discussing how to start making that change. While I miss the fireworks of previous summit sessions, I much prefer the more cooperative atmosphere. We really must be growing up!

Default to Respect

If you know me, you know that I have a sense of humor that can be risqué at times (ok, perhaps crude would be a better description!). I’m also known to engage in the predominantly male form of communication that involves bonding by insulting each other: put downs, the dozens, whatever you want to call it. I also hold very opinionated positions on politics and religion, and enjoy engaging in lively discussions about them.

Yet when I am in a group of people I do not know very well, I do none of these things. Why? Because I am aware of their potential for offending people, or at the very least, making them feel uncomfortable. So I default to respect.

In programming, a default value is one that is used unless specifically overridden. Setting your default to respect means that unless you are certain that everyone within earshot (or who can otherwise observe you) knows you well enough to properly interpret your words or actions, limit yourself to those words or actions that do not require special interpretation; those that show respect for the people around you. Failing to do this is one of the biggest sources of the problems in the tech community when it comes to how women and other under-represented groups are treated. At conferences, or online, guys (yes, it’s a guy problem) act as they would normally do when they are within their tight-knit group of friends, and say/write/do something that is interpreted as offensive or even hostile. When their poor choices are pointed out, they get defensive, using the excuse that their intention was not to offend, so no one should take it badly. Or they attack, claiming that the person who pointed out their behavior is too “politically correct” (at best), or an over-sensitive bitch (if the reporter is a woman). These attacks all too frequently cross the line from name-calling to outright threats.

But refraining from sexual references or racial stereotypes is not being “politically correct”; it’s a sensible default value. Maybe later you might get to know these people better, and more importantly, they’ll get to know you better. Only then when you make a crude joke will they know that you mean no harm. But until then, the only sensible approach is to default to respect. The practice of a conference having (and enforcing!) a Code of Conduct is really a way of defining these sensible defaults for people who apparently never learned them growing up. It is encouraging to see them become more common than not, for that will help our communities “grow up” and become more inclusive. The days of the tech world being an old boys club are quickly drawing to a close, although it can’t happen fast enough for me.

So does that mean that you need to muzzle yourself? No, of course not. There are plenty of places where you can express yourself; hell, if you follow me on Google+, you’ll see that I’m not at all shy about stating my opinions. It’s totally appropriate there, because if you don’t like what I’m writing or the way I write it, you don’t have to follow me; there are plenty of other people you might like better. But a conference or an online forum is a community vehicle, and filling them with potentially hostile or offensive words or actions means that we will turn away many who would have otherwise helped the community grow better. We all suffer when otherwise talented and interesting people choose not to engage in our communities because they do not feel welcome. So do us all a favor and when you are in a community situation, set your default to respect.

Simplifying OpenStack

Recently I’ve returned to working on the OpenStack code base after a couple of years of working on related projects, and the sheer scope of the changes to the code, both in size and structure, is a bit overwhelming. I’ve also had to catch up with the current issues being discussed among the community, as a project that is growing the way OpenStack is will always have pain points. One such discussion that caught my attention concerns the very definition of what OpenStack is. The discussion not only addresses some of the experiences I’ve had returning to the world of OpenStack development, but it also feels like a continuation of the discussions we had when we were first shaping the project 4 years ago.

Sean Dague wrote an interesting take on this, envisioning OpenStack as a set of layers. The lowest layers provided the basic compute infrastructure, and the higher layers either built on this, or added additional capabilities. While I can see the basis for this division, it struck me as somewhat arbitrary; I mean, are Heat and Trove really that similar in either purpose or architecture?

Yesterday I read Monty Taylor’s blog post on this topic, and I think he has much more accurately captured the relationship between the various parts of OpenStack. His post is much more far-reaching than simply describing OpenStack; it’s really more concerned with how to help OpenStack move forward, and uses these relationships to make his case.

I think that working with the notion that there is a fundamental base for OpenStack (what Monty and Sean both refer to as ‘Layer #1‘), and that everything else is an add-on to this base, will have several positive results. First, it simplifies the internals of these projects. As Monty points out, “nova shouldn’t need a config file option pointing to where glance is, it should just be able to ask the keystone service catalog”. In other words, these Layer #1 projects can assume that they are all present, and don’t have to add needless complexity to handle these relationships. Glance and Keystone have to be there, or Nova can’t work – period.

Second, it streamlines the testing process considerably. The only tests for changes to any of these Layer #1 projects that need to be run are those for their internal workings, and those for their interfaces. This smaller, more manageable gate should relieve (but not eliminate) some of the testing bottlenecks OpenStack experiences today.

Speaking of interfaces, this is the third, and in my opinion, the most significant benefit of this concept: it forces the interfaces between the various projects to be defined clearly, and makes testing any potential change to these interfaces much cleaner. Let’s take the example of Trove: it started as a separate database-as-a-service built on top of Nova, but now is being consumed internally to handle Nova’s database needs. With the current “everything is OpenStack” approach, the lines between Nova and Trove might start to blur, and before long Trove and Nova would become too tightly bound to separate. By keeping the interface between the two clean and separate, it allows for the possibility that another DBaaS project may come along that performs better for Nova’s needs, and can be adopted without having to rip out any hard assumptions about Trove. Similarly, other DBaaS could be developed that are optimized for other use cases, and these can test their interfaces to assure that they can also be implemented as a drop-in replacement for Trove.

Monty identified yet another benefit: there will no longer be a “race” for teams to get officially designated as part of OpenStack, as this designation would go away. Instead, there would be a known set of interfaces that a project would have to support, and if there is more than one group with an idea for implementing a particular need, they can each develop their code without having to first get “blessed” as somehow the “official” OpenStack solution for this need. Developing to a known interface will allow the community to test these different solutions much more easily, and allow a superior solution arise, instead of having to settle for the solution that got there first.

Finally, I see a side benefit to this kind of simplicity of structure. Someone coming to OpenStack for the first time, the sheer number of projects is daunting. Never mind having to learn all the names and what each project does; it’s the feeling that OpenStack is too big to wrap your brain around that can potentially discourage new developers. A clear separation between the heart of OpenStack and the various peripheral projects would eliminate that confusion.

I’m greatly encouraged by these discussions, as it helps clarify the long-term direction of OpenStack. And I’m extremely happy to be part of it all once more.

Review: GlueCon 2014

GlueCon 2014 was held on May 21-22 at the Omni Interlocken Hotel in Broomfield, Colorado, and I was fortunate enough to be selected as a speaker there for the second year in a row. For those of you who aren’t familiar with GlueCon, it’s not quite an unconference, but it is a loosely-structured gathering of people working in lots of different technologies that drive the web today, with the focus on the “glue” that holds them all together: developers.

The morning of the 21st was a series of keynotes, and devops seemed to be the topic on everyone’s mind. There was one keynote in general that I strongly disagreed with: John Sheehan of Runscope claimed that “API SDKs Will Ruin Your Life” (the title of the keynote). The assumption, of course, was that the only reason that anyone would use an SDK is because the native API is terrible, and it is the reliance on SDKs that is causing API devs to get sloppy. While that might be true in some sense, the responsibility for creating a good API rests on the API developers, and if they are not professional enough to take pride in the quality of their APIs, I doubt that any SDK would change that. Creating good, consistent APIs is not easy, and typically shouldn’t be left to the the people creating the underlying product that the API is exposing.

The afternoon was broken into several tracks, with the idea that the talks in each track were related, and would happen one right after the other, with no break in between. This was when my talk was scheduled, and it was part of the Cloud track. The talk was titled “How to Leave Rackspace”, and no, I wasn’t encouraging people to leave Rackspace! (I like having a job!). It was about the realities of portability among clouds, and emphasized the importance of data gravity binding you to a provider. I also covered the importance of API compatibility, and how working with providers sharing a common API, such as OpenStack-based providers, won’t do anything about data gravity, but will at least allow your applications to run on different providers.

The script I showed used pyrax to transfer a running compute instance in Rackspace to the HP cloud, so that you could move an entire server with just a few lines of code. The overall lesson, however, was to take the time to choose your provider carefully up front, instead of relying on some promise of portability later. The audience seemed pretty engaged during the talk, but there weren’t many questions and what seemed like lukewarm applause afterwards. I was a little disappointed, but noticed the same pattern for the other speakers who followed me, so I thought it might be due to the format. Later on I did run into several people who attended my session, and their feedback was very positive, so that made me feel better about the talk. This was the first time I’ve given this particular talk, and so I was very appreciative of the feedback.

After my talk was an excellent talk by Cornelia Davis of Pivotal that gave us a peek under the hood of Cloud Foundry. I was familiar with what Cloud Foundry could do, but I hadn’t had any exposure to the underlying architecture, so this was very interesting. Following that were talks on OpenStack Swift and network virtualization, which were both full of useful information, and then there was a talk on “The New Development Experience” that had some good stuff on Bluemix, IBM’s implementation of Cloud Foundry, but was too filled with buzzwords to be interesting. For a sample of what I mean, here’s a sentence from the talk abstract:

“The cloud combined with the need-for-time to value is driving a movement toward a scalable and flexible cloud computing development model that provides application developers with the freedom to innovate rapidly, integrate with existing infrastructure, use or create APIs, and create cloud native applications”.

I spent the rest of the day working at the booth, and noticed a distinct difference in attitude from last year’s crowd. For some reason, the crowd last year was not only very pro-AWS, but actively hostile to OpenStack. This year I didn’t get any of that negativity; instead, I spoke with developers and business people who knew about OpenStack and Rackspace, and simply wanted more information. My favorite, though, was Atsushi Yamanaka, from IDC Frontier in Japan. He flew to the Denver area the previous day from Tokyo to attend GlueCon, and was flying back to Tokyo the day after GlueCon ended. Talk about dedication! He wanted to get a better idea of Rackspace’s presence in the Asia/Pacific area, both current and future, and unfortunately I had to explain that those decisions are made at much higher pay grades than mine!

The second day of the conference similarly opened with a series of keynotes. I had to work the booth and also work on some code, so I didn’t catch most of them. I did, however, catch the keynote by Joshua McKenty of Piston Cloud Computing, which I found to be equal parts entertaining and informative. I’ve known Joshua for several years now, since the birth of OpenStack, and know as well as anyone how he can tend to overstate things just to get a rise out of the audience. He also knocked SDKs, but made it clear that it was only when they were used as crutches for bad APIs, and I had to comment about it on Twitter. The example he gave was that if instead of a website, you had created a duck, when you create the duck API, it would be dumb to expose the gizzard and the inner workings of its circulatory system, yet developers do the equivalent of that regularly. When people want to use your duck, they are interested in its ability to fly or swim or eat or quack, and don’t really care about the incredibly complex implementation details that make those abilities possible. Developers, on the other hand, are so proud of their incredibly complex implementations that they tend to feature them front and center in their API design. This makes for an unwieldy API, which then almost forces users of it to rely on an SDK to be able to do anything useful.

There was another keynote by Julia Ferraioli from Google on developer outreach, and since that’s my job, I was interested to hear what she had to say. Unfortunately, the talk was really nothing more than the general knowledge stuff that people working in this field for any length of time already know; I certainly didn’t pick up anything new. I hope, though, that maybe some of the non-developers in the crowd learned something.

I wish I could comment on more of the talks, but again I spent a good part of the day either working the booth or writing code for work. I did see a couple of talks that revolved around novel uses of Docker to either speed up or reduce the cost of CI/CD enviroments; it certainly seems as though Docker was the new hotness among developers at GlueCon.

I can’t say enough good things about the event itself: the location, the organization, the food/refreshments, the attendees, and the quality of the talks were all excellent. If you get a chance to attend GlueCon 2015 next year, go for it!