Is Swift OpenStack?

There has been some discussion recently on the OpenStack Technical Committee about adding Golang as a “supported” language within OpenStack. This arose because the Swift project had recently run into some serious performance issues, which they solved by re-writing the bottleneck process in Golang with much success. I’m not writing here to debate the merits of making OpenStack more polyglot (it’s no secret that I oppose that), but instead, I want to address the issue of Swift not behaving like the rest of OpenStack.

Doug Hellman summarized this feeling well, originally writing it in a pastebin, but then copying it into a review comment on the TC proposal. Essentially, it says that while Swift makes some efforts to do things the “OpenStack Way”, it doesn’t hesitate to follow its own preferences when it chooses to.

I believe that there is good reason for this, and I think that people either don’t know or forget a lot of the history of OpenStack when they discuss Swift. Here’s some background to clarify:

Back in the late ’00s, Rackspace had a budding public cloud business (note: I worked for Rackspace from 2008-2014). It had bought Slicehost, a company with a closed-source VPS system that it used as the basis for its Cloud Servers product, and had developed a proprietary object storage system called NAST (Not Another S Three: S3, get it?). They began hitting limits with NAST fairly soon – it was simply too slow. So it was decided to write a new system with scalability in mind that would perform orders of magnitude better than NAST; this was named ‘Swift’ (for obvious reasons). Swift was developed in-house as a proprietary software project. The development team was a small, close-knit group of guys who had known each other for years. I joined the Swift development team briefly in 2009, but as I was the only team member working remotely, I was at a significant disadvantage, and found it really difficult to contribute much. When I learned that Rackspace was forming a distributed team to rewrite the Cloud Servers software, which was also beginning to hit scalability limits, I switched to that team. For a while we focused on keeping the Slicehost code running while starting to discuss the architecture of the new system. Meanwhile the Swift team continued to make strong progress, releasing Swift into production in the spring of 2010, several months before OpenStack was announced.

At roughly the same time, the other main part of OpenStack, Nova, was being started by some developers working for NASA. It worked, but it was, shall we say, a little rough in spots, and lacked some very important features. But since Nova had a lot of the things that Rackspace was looking for, we started talking with NASA about working together, which led to the creation of OpenStack. So while Rackspace was a major contributor to Nova development back then, from the beginning we had to work with people from a wide variety of companies, and it was this interaction that formed the basis of the open development process that is now the hallmark of OpenStack. Most of the projects in OpenStack today grew out of Nova (Glance, Neutron, Cinder), or are built on top of Nova (Trove, Heat, Watcher). So when we talk about the “OpenStack Way”, it really is more accurately thought of as the “Nova” way, since Nova was only half of OpenStack. These two original halves of OpenStack were built very differently, and that is reflected in their different cultures. So I don’t find it surprising that Swift behaves very differently. And while many more people work on it now than just the original team from Rackspace, many of that original team are still developing Swift today.

I do find it somewhat strange that Swift is being criticized for having “resisted following so many other existing community policies related to consistency”. They are and always have been distinct from Nova, and that goes for the community that sprang up around Nova. It feels really odd to ignore that history, and sweep Swift’s contributions away, or disparage their team’s intentions, because they work differently. So while I oppose the addition of languages other than Python for non-web and non-shell programming, I also feel that we should let Swift be Swift and let them continue to be a distinct part of OpenStack. Requiring Swift to behave like Nova and its offspring is as odd a thought as requiring Nova et. al. to run their projects like Swift.

A New (But Familiar) Adventure

OK, I admit it: I haven’t been posting here as regularly as I should be. It’s not like my life is so boring, or that I haven’t had any interesting thoughts to share. Rather, it’s because I have been so busy and my life changing so fast that I haven’t had the time to sit down, catch my breath, and write things down. I hope to be able to change that moving forward.

Today is the beginning of one of those changes: I start my new job as an OpenStack developer for IBM. In a way I feel like I’m coming back to a familiar world, even though I’ve never worked for IBM before. I was involved in the creation and initial design of OpenStack, and have always felt that it was partially “my baby” (ok, a very small part, but mine nonetheless). I left active involvement with OpenStack a few years ago to start the Developer Experience effort at Rackspace, and have only been tangentially working on OpenStack during that time. With the move to IBM, my focus will once again be on OpenStack, and I couldn’t be happier.

Review: GlueCon 2014

GlueCon 2014 was held on May 21-22 at the Omni Interlocken Hotel in Broomfield, Colorado, and I was fortunate enough to be selected as a speaker there for the second year in a row. For those of you who aren’t familiar with GlueCon, it’s not quite an unconference, but it is a loosely-structured gathering of people working in lots of different technologies that drive the web today, with the focus on the “glue” that holds them all together: developers.

The morning of the 21st was a series of keynotes, and devops seemed to be the topic on everyone’s mind. There was one keynote in general that I strongly disagreed with: John Sheehan of Runscope claimed that “API SDKs Will Ruin Your Life” (the title of the keynote). The assumption, of course, was that the only reason that anyone would use an SDK is because the native API is terrible, and it is the reliance on SDKs that is causing API devs to get sloppy. While that might be true in some sense, the responsibility for creating a good API rests on the API developers, and if they are not professional enough to take pride in the quality of their APIs, I doubt that any SDK would change that. Creating good, consistent APIs is not easy, and typically shouldn’t be left to the the people creating the underlying product that the API is exposing.

The afternoon was broken into several tracks, with the idea that the talks in each track were related, and would happen one right after the other, with no break in between. This was when my talk was scheduled, and it was part of the Cloud track. The talk was titled “How to Leave Rackspace”, and no, I wasn’t encouraging people to leave Rackspace! (I like having a job!). It was about the realities of portability among clouds, and emphasized the importance of data gravity binding you to a provider. I also covered the importance of API compatibility, and how working with providers sharing a common API, such as OpenStack-based providers, won’t do anything about data gravity, but will at least allow your applications to run on different providers.

The script I showed used pyrax to transfer a running compute instance in Rackspace to the HP cloud, so that you could move an entire server with just a few lines of code. The overall lesson, however, was to take the time to choose your provider carefully up front, instead of relying on some promise of portability later. The audience seemed pretty engaged during the talk, but there weren’t many questions and what seemed like lukewarm applause afterwards. I was a little disappointed, but noticed the same pattern for the other speakers who followed me, so I thought it might be due to the format. Later on I did run into several people who attended my session, and their feedback was very positive, so that made me feel better about the talk. This was the first time I’ve given this particular talk, and so I was very appreciative of the feedback.

After my talk was an excellent talk by Cornelia Davis of Pivotal that gave us a peek under the hood of Cloud Foundry. I was familiar with what Cloud Foundry could do, but I hadn’t had any exposure to the underlying architecture, so this was very interesting. Following that were talks on OpenStack Swift and network virtualization, which were both full of useful information, and then there was a talk on “The New Development Experience” that had some good stuff on Bluemix, IBM’s implementation of Cloud Foundry, but was too filled with buzzwords to be interesting. For a sample of what I mean, here’s a sentence from the talk abstract:

“The cloud combined with the need-for-time to value is driving a movement toward a scalable and flexible cloud computing development model that provides application developers with the freedom to innovate rapidly, integrate with existing infrastructure, use or create APIs, and create cloud native applications”.

I spent the rest of the day working at the booth, and noticed a distinct difference in attitude from last year’s crowd. For some reason, the crowd last year was not only very pro-AWS, but actively hostile to OpenStack. This year I didn’t get any of that negativity; instead, I spoke with developers and business people who knew about OpenStack and Rackspace, and simply wanted more information. My favorite, though, was Atsushi Yamanaka, from IDC Frontier in Japan. He flew to the Denver area the previous day from Tokyo to attend GlueCon, and was flying back to Tokyo the day after GlueCon ended. Talk about dedication! He wanted to get a better idea of Rackspace’s presence in the Asia/Pacific area, both current and future, and unfortunately I had to explain that those decisions are made at much higher pay grades than mine!

The second day of the conference similarly opened with a series of keynotes. I had to work the booth and also work on some code, so I didn’t catch most of them. I did, however, catch the keynote by Joshua McKenty of Piston Cloud Computing, which I found to be equal parts entertaining and informative. I’ve known Joshua for several years now, since the birth of OpenStack, and know as well as anyone how he can tend to overstate things just to get a rise out of the audience. He also knocked SDKs, but made it clear that it was only when they were used as crutches for bad APIs, and I had to comment about it on Twitter. The example he gave was that if instead of a website, you had created a duck, when you create the duck API, it would be dumb to expose the gizzard and the inner workings of its circulatory system, yet developers do the equivalent of that regularly. When people want to use your duck, they are interested in its ability to fly or swim or eat or quack, and don’t really care about the incredibly complex implementation details that make those abilities possible. Developers, on the other hand, are so proud of their incredibly complex implementations that they tend to feature them front and center in their API design. This makes for an unwieldy API, which then almost forces users of it to rely on an SDK to be able to do anything useful.

There was another keynote by Julia Ferraioli from Google on developer outreach, and since that’s my job, I was interested to hear what she had to say. Unfortunately, the talk was really nothing more than the general knowledge stuff that people working in this field for any length of time already know; I certainly didn’t pick up anything new. I hope, though, that maybe some of the non-developers in the crowd learned something.

I wish I could comment on more of the talks, but again I spent a good part of the day either working the booth or writing code for work. I did see a couple of talks that revolved around novel uses of Docker to either speed up or reduce the cost of CI/CD enviroments; it certainly seems as though Docker was the new hotness among developers at GlueCon.

I can’t say enough good things about the event itself: the location, the organization, the food/refreshments, the attendees, and the quality of the talks were all excellent. If you get a chance to attend GlueCon 2015 next year, go for it!

PyCon 2014 Review

mirror.jpgBetter late than never, right?

PyCon 2014 happened two weeks ago, and I’m just getting around to write about it now. Why the delay? Well, I took a few days of vacation to explore and enjoy Montréal with my woman after PyCon, and when I returned I found myself trapped under a mountain of work that had built up in my absence. I’ve finally dug myself out enough to take the time to write up my impressions.

This PyCon (my 11th) was different in many ways, not only for the fact that for the first time, the US PyCon was not in the US! It was held in Canada in beautiful (and cold!) Montréal. It was the first time in years that I did not arrive early enough to help with the bag stuffing event, which has always been a highlight of my PyCon experiences. I arrived late Thursday afternoon, and just made it to the Opening Reception, where I not only touched based with my fellow Rackers, but also ran into dozens of Python people I have gotten to know over the years. PyCon is special in that regard: while I have technical contacts at many of the conferences I attend, I consider many of the people at PyCon to be my friends.

I was pleasantly surprised by the bold choice for the opening keynote: John Perry Barlow, one of the founders of the Electronic Frontier Foundation (EFF).


He pulled no punches, and let the audience know exactly where he stood on matters of security, openness, government spying, and free information in a society. I enjoyed his talk immensely, but I knew that some with more conservative views might not respond positively to the anti-corporation, anti-BigBrother tone of his talk, and judging by the sight of several people leaving during the talk, I think I was right. Still, I appreciated that the primary conferences for one of the most important Open languages took that risk.

I attended several sessions, and bounced between them and my duties at the Rackspace booth in the vendor area. We were once again a major sponsor of PyCon, and this gives me great pleasure, as I had first convinced Rackspace to become a sponsor shortly after I joined the company 6 years ago. We benefit so much from Python and the work of the PSF, it’s only right that we give something back.


I won’t go into detail about the individual sessions, but I would encourage you to check out the videos of all the talks that are available on the pyvideo site. The quality of the presentations keeps getting better and better every year, which is a great reflection on the talk selection committee, who had to select just 95 talks from a total of 650 proposals. That is a thankless task, so let me say “thank you” to the folks who reviewed all those proposals and made the tough calls.

I would also like to note that this year 1/3 of the speakers were women, and by some estimates, the percentage of female attendees was nearly as high! (They don’t record the sex of a person when they register; hence the need for estimates). This is a phenomenal result in the traditionally male-dominated tech industry, and it isn’t by accident. The PSF has actively encouraged women to attend, both by creating (and standing behind) a Code of Conduct, as well as offering financial assistance to those who might not otherwise be able to attend. And this year was another first: onsite child care during the main conference days. I think this is an amazing addition to PyCon, even though my kids are grown. It allows many people to come who would otherwise not be able, and also encouraged more families to travel together, making PyCon the hub of their vacation plans.

That’s exactly what I did, too. Linda flew into Montréal on Saturday evening, hung around PyCon for the closing session so that she could get a glimpse into the strange geek world I inhabit regularly, and meet some of my friends. We spent the next few days exploring this city that neither of us had visited before, enjoying ourselves immensely. Coming from South Texas, we could have done without the near-freezing temperatures and snow, though! Here was the view from our hotel window:OLYMPUS DIGITAL CAMERA

It was a wonderful vacation, but much too short! We’re really looking forward to returning next year for PyCon 2015!OLYMPUS DIGITAL CAMERA



A Look Back at PyCon Australia 2013

In an earlier post I reviewed the OpenStack miniconf that preceded the main PyCon, which was held in Hobart, Tasmania on July 6–7. I had meant to write this shortly after PyCon ended, but the whirlwind of travel back to the US and getting back into the daily grind pushed it off my plate.

The conference was recorded, and all the videos are available on the website. I encourage you to watch as many of the sessions that interest you as you can – lots of good stuff in them!

The conference actually started for me earlier – the organizer, Chris Neugebauer, had asked for volunteers to help with the conference prep work: badges, swag, all that stuff. This was on the Wednesday before the conference, which happened to be the day I arrived, so it was as good an excuse as any to get out of my hotel and into Hobart. For those of you who have never gone to a PyCon, it is completely run by volunteers. No one gets paid; no one gets free admission; no one gets special perks. This was shocking to me when I moved from the Microsoft conference world a decade ago, where conferences were run as profit centers, and attendees paid for tickets that cost well over $1,000, but who could then relax and treat their time there as a vacation (which many did, at their employers’ expense). But PyCons are the exact opposite, and as a result everyone has a stake in the conference experience. I’ve found that volunteering not only makes you feel like you’re contributing, but it also means that you meet a lot of interesting people who might otherwise remain anonymous faces in the crowd.

PyCon AU volunteer night

The picture above is a quick snapshot I took as we carried out the task of wrapping a thousand coffee cups (yes, that’s correct: 1000) with vinyl cutouts of the coffee sponsor’s logo. Why did we do this? Because a run of pre-printed cups had a minimum of 50,000, and, well, PyCon just ain’t that big! And to make things even more fun, the logos were slightly bigger than the cup, which made it difficult to wrap cleanly. Tedious, huh? But rather than being a drag, it was a great time as the dozen or so people there had great conversations as we wrapped the cups one by one. As the only American there, and this being my first time in Australia, observations on cultural differences was a big topic, and I certainly learned a lot. We were all shocked when we were done to learn that we had spent 4 hours working on this stuff; it certainly went by much faster than that.

After the work was done, we went out to eat, where (among other things) I learned that Tasmania has a high-end distillery business that is producing spirits that rival other parts of the world, especially in the high-end whiskey area. Two of the best are the Nant and Lark distilleries. I’m not a big Scotch drinker, but I appreciated the quality that these two distilleries are producing. Unfortunately, they are relatively small, and their products are nearly impossible to find in the US.

The conference proper started off Saturday morning, with Alex Gaynor giving the keynote. It set the tone for the conference, and we spent much of the next two days exploring what it meant to be both a software developer in general, and a Pythonista in particular. I then went to Nick Coghlan’s talk about the state of Python packaging; it’s no secret that packaging has always been a pain point for Python, but it is great to see that the different efforts are now unified under Nick’s leadership.

Next up was Jacob Kaplan-Moss’s talk on security in web apps in the Python world. I was aware of most of the attack vectors thanks to my security training at Rackspace, but it was somewhat shocking to see how easy it is to expose a vulnerability if you don’t know it’s there. If you write web apps that are exposed to the public, make sure you watch the video of his talk. Now.

In the afternoon I drifted between various talks which, to be frank, varied in their appeal. That’s not to say that the content wasn’t valuable; it just wasn’t of particular interest to my work. I did end up skipping a few talks to work on my slides for my session; I don’t think I’ve ever given a talk without editing it right up to the last minute, and this was no different.

The day ended with a full hour of Lightning Talks, which are always one of my favorite parts of PyCon, and the lightning talks at PyCon AU did not disappoint. If you aren’t familiar with the concept, they are a series of short (5 minutes – enforced!) talks on whatever the speaker wants to talk about. There was a good mix of dense information, rants, product description and general wackiness. If you’ve never been to a PyCon, don’t miss the lightning talks! I didn’t take notes, but fortunately Thomas Sutton did.

One thing that was different at this PyCon was the Conference Dinner after the sessions on Saturday. All the attendees were gathered into a room with many large round tables that sat about 10 people each. There was wine and beer served, along with iced tea and soda. The food was served buffet-style, but this was no ordinary conference fare. Fresh oysters! Smoked salmon! Freshly-carved roasts! All this with an assortment of salads, vegetables, breads, and deserts. I followed my usual practice and sat at a table where I knew no one, in order to make new friends and learn about what they are doing with Python, but we ended up having much less technical discussion for some reason (big surprise!).

The evening wasn’t all food and drink, though. There was an evening keynote by Mark Pesce of MooresCloud that talked about their work integrating Raspberry Pi units to control LED displays. An example of this was used in the lightning talks to provide a visual indication of the time remaining for each speaker; as time wound down, the LEDs changed from green to yellow to red to flashing red to off. During the keynote, though, he demonstrated how the Pis could change the light display in response to a stream of Twitter hashtags, creating a virtual tug-of-war. He had half the room send tweets with one hashtag, while the other half tweeted a different hashtag, and the color distribution of the display changed in response. Apologies to anyone who follows me on Twitter for the hashtag spam that night! So while this was a somewhat silly example of what it could do, it certainly demonstrated that you could program the device to respond to any sort of dynamic data.

Sunday morning’s sessions were… ok, you got me. I skipped the sessions at the prompting of Alan Perkins, the director of our Sydney office, who grew up in Hobart and knows the area well. He wanted to take me along with a couple of other Rackers up to the peak of Mount Wellington, which is probably the most ubiquitous visual in the Hobart area. Unfortunately, being the middle of winter there, the roads were closed due to iciness. So instead Alan took us on a tour along the coastline of the Derwent estuary through some of the smaller towns and some truly gorgeous scenery.

We made it back in time for the last morning session, but I needed to prepare for my talk in the afternoon. After lunch Richard Jones gave an excellent talk entitled “Don’t Do This”, which was an exploration of some of the oddities that were possible in the Python language. Fortunately I can say that none of them could be found in my code! Next up was Dylan Lacey from Sauce Labs, with a talk on using Splinter to streamline your testing by providing an API that allows you to automate browser actions. Splinter looks pretty interesting, and if I were doing web acceptance testing, I would definitely check it out.

Next up was my session on using Python to build your application infrastructure. I won’t self-review, but I do want to thank the people who attended the talk for being understanding when the shaky hotel network caused my live demo to die halfway through. I did get a few good reviews from those who attended the talk.

After the afternoon tea break, I saw Luke Miller’s talk on developing an indie game with Python that featured a gay theme. I had never really thought about it before, but there weren’t any games that featured leading characters who were gay, and which might appeal to gay audiences more than the current selection available. The talk wasn’t just about the game’s theme; it was a very in-depth look into what it takes to develop a game and make it interesting. I must say that I learned more in that talk than in any other I attended, as the material was totally new to me, and it was presented very clearly.

The final session was by Adam Forsyth on Getting the Most Out of StackOverflow. Adam is seriously active on that site, and knows quite a bit about how things work, and how it can be overwhelming at times to a newcomer. He had many tips for those who wanted to both ask better questions that would have a good chance of receiving a great answer, as well as for those who wanted to be able to find questions to answer before others did.

Last but not least were the second day of lightning talks. Once again they were excellent, and again, Thomas Sutton took much better notes than I did.

That was the end of the conference, and Chris Neugebauer, as the conference chair, gave his closing remarks, thanking everyone from the sponsors to the attendees for making PyCon AU 2013 a truly remarkable conference. He received a standing ovation of thanks from the audience for all his hard work, and he deserved every clap.

Many people left after that, but for those of us who remained, things were not over. We had the upper level of Jack Greene’s reserved, and it was packed with Pythonistas enjoying some final food, drink, and conversation.

I’ve been to 10 PyCon US conferences, and while PyCon AU 2013 was much smaller in overall size, it matched its US counterpart in terms of spirit, content, and sense of community. Next year it moves to Brisbane, and I would certainly love to be able to return.

Heads up, Aussies!

In a couple of days I’ll be heading out to Australia for a couple of weeks. My proposal for a talk at PyCon Australia had been accepted, and coincidentally Rackspace just launched our cloud in our Sydney datacenter. So I’ll be flying to Sydney to meet all the Rackers in the Sydney office, and then flying the following week to Hobart, Tasmania for the conference.

Except for a trip to Italy on my honeymoon 27 years ago, I have never left the Americas. I have never been south of the Equator. This is going to be a brand-new experience for me, so I’m using it as my motivation to start writing again. I had deliberately avoided blogging for years because I tend to obsess on editing: just one more tweak to that phrasing, or maybe choose a synonym here, or… well, you get the idea. A great example of perfect being the enemy of the good. I’m resolving to change that by instituting a one edit rule: after the post is written, I’ll go over it once, and then publish it. I’m curious to see how difficult that will be to follow!