Hertz and the Great Tollway Ripoff

Last Thanksgiving we went on a wonderful holiday in the Florida Keys. We flew to Miami, picked up a rental car from Hertz, and drove away. There are a couple of toll roads along the way, but we had Hertz’s PlatePass, which would work on those toll sensors. I’ve used similar things with other rental companies, where a few weeks later I receive a bill for the accumulated tolls.

Not with PlatePass, however. Not only did I get a bill for the tolls (about $11), but a $25 service charge on top of that! Turns out that Hertz charges $5 a day (up to $25), even on days when you don’t run up any tolls! I’m stuck paying this, but you can be sure that I will avoid using and recommending Hertz in the future. Yeah, I found out that it’s documented on the website, but it wasn’t documented when I got in the car, nor was I given an option to disable it. So yeah, Hertz and/or PlatePass made an extra $25 off of me on that trip, but they will lose so much more than that in the future. This is what happens when businesses are short-sighted and go after the quick buck instead of developing long-term relationships with their customers.

OpenStack Focus

There has been a bit of concern expressed lately that OpenStack is somehow losing its focus, and is in danger of losing its momentum because of the effect of the Big Tent, and its  resulting growth in the number of projects that call themselves OpenStack. This growth has even been blamed for the recent layoffs of OpenStack developers. Some have called on the OpenStack leadership to dismantle the Big Tent approach, and only focus on a few core projects.

While it is true that the plethora of projects has diverted attention from the work needed in the heart of OpenStack (and I won’t go into how to draw the line separating the two here), I feel that the criticism is misplaced. It isn’t up to the governing bodies of OpenStack to enforce such a refocusing; rather, it is up to the contributors to make such decisions. That’s just the way that open source development works. It is silly to think that companies like HPE would take their marching orders from the OpenStack Foundation Board, or the OpenStack TC. The idea of the Big Tent, that all projects that are “one of us” shall have access to the same resources, is fine as it is.

The mistake that I believe many companies made is that they tried to focus on beefing up numbers that are irrelevant, such as lines of code, or the number of cores or PTLs they employed, as a way of demonstrating their commitment to OpenStack. They then would use those numbers for their sales teams as a selling point for their OpenStack-based offerings.

Open Source is a difficult sell for most companies; they certainly understand the benefit when they use it, but have a much harder time justifying the cost of paying their employees to work on something that is used by everyone, even their competitors. So they came up with ways of selling their particular spin on OpenStack, and used these contribution number to impress customers. So when that failed to generate the type of revenue that was expected, out came the axe.

I believe that many of these companies encouraged the development of these small peripheral projects because it would be easier for one of their employees to achieve core status, and possibly get elected PTL, which their marketing departments would use in an attempt to prove that company’s OpenStack-ness.

I don’t agree that there is anything that OpenStack itself needs to do. Rather, the companies who are contributing to OpenStack need to better understand the nature of open source development, and focus on those areas that will make OpenStack as a whole richer and more reliable, instead of gaming the system to make themselves look important. So please stop saying that this is the fault of the Big Tent.

PyCon 2015

PyCon 2015 ended over a week ago, so you might be wondering why I’m writing this so late. Well, once again (see my PyCon 2014 post) I blame the location: the city of Montreal. We like it so much that Linda and I planned on staying a few extra days on holiday afterwards. After returning, though, I again payed the price by digging out from the accumulated backlog. It was well worth it, though!

Old Montreal
Old Montreal at night

If you weren’t able to go to PyCon, or even if you were there and don’t possess the ability to be in multiple places at once, you missed a lot of excellent talks. But no need to worry: the A/V team did an amazing job this year, and not only recorded every session, but got them posted to YouTube in record time – many just a few hours after the talk was completed! Major kudos to them for an excellent job.

swagbags The swag table (top) and pile of stuffed bags (bottom)

PyCon is an amazing effort by many people, all of whom are volunteers. One of my favorite volunteer activity is the stuffing of the swag bags. Think about it: over 3,000 attendees each receive a bag filled with the promotional materials from the various sponsors. Those items – flyers, toys, pens, etc. – are shipped from the sponsors to PyCon, and somehow one of each must get put into each one of those bags. Over the years we’ve iterated on the approach, trying all sorts of concurrency models, and have finally found one that seems to work best: each box of swag has one person to dish it out, and then everyone else picks up an empty bag and walks down the table, and one item of each is deposited in their bag. Actually it took two very long tables, after which the filled bag is handed to another volunteer, who folds and stacks it. It’s both exhausting and exhilarating at the same time. We managed to finish in just under 3 hours, so that’s over 1,000 bags completed per hour!

In between talks, I spent much of my time staffing the OpenStack booth, and talked with many people who had various degrees of familiarity with OpenStack. Some had heard the name, but not much else. Others knew it was “cloud something”, but weren’t sure what that something was. Others had installed and played around with it, and had very specific configuration questions. Many people, even those familiar with what OpenStack was, were surprised to learn that it is written entirely in Python, and that it is by far the largest Python project today. It was great to be able to talk to so many different people and share what the OpenStack community is all about.

Last year PyCon introduced a new conference feature: onsite child care for people who wanted to attend, but who didn’t have anyone to watch their kids during the conference. Now, since my kids are no longer “kids”, I would not have a personal need for this service, but I still thought that it was an incredible idea. Anything that encourages more people to be able to be a part of the conference is a good thing, and one that helps a particularly under-represented group is even better. So in that tradition, there was another enabling feature added this year: live captioning of every single talk! Each room had one of the big screens in the front dedicated to a live captioned stream, so that those attendees who cannot hear can still participate. I took a short, wobbly video when they announced the feature during the opening keynote so you can how prominent the screens were. I have a bit of hearing loss, so I did need to refer to the screen several times to catch what I missed. Just another example of how welcoming the Python community is.

gabriellacolemanTrue to last year’s form, one of the keynotes was focused on the online community of the entire world, not just the limited world of Python development. Last year was a talk by John Perry Barlow, former Grateful Dead lyricist and co-founder of the Electronic Frontier Foundation, sharing his thoughts on government spying and security. This year’s talk was from Gabriella Coleman, a professor of anthropology at McGill University. Her talk was on her work studying Anonymous, the ever-morphing group of online activists, and how they have evolved and splintered in response to events in the world. It was a fascinating look into a little-understood movement, and I would urge you to watch her keynote if you are at all interested in either online security and activism, or just the group itself.

jkmmediocreThe highlight of the conference for me and many others, though, was the extremely thoughtful and passionate keynote by Jacob Kaplan-Moss that attempts to kill the notion of “rockstar” or “ninja” programmers (ugh!) once and for all. “Hi, I’m Jacob, and I’m a mediocre programmer”. You really do need to find 30 minutes of time to watch it all the way through.

This last point is a long-time peeve of mine: the notion that programming is engineering, and that there are objective measurements that can be applied to it. Perhaps that will be fodder for a future blog post…

One aspect of all PyCons that I’ve been to is the friendships that I have made and renewed over the years. It’s always great to catch up with people you only see once a year, and see how their lives are progressing. It was also fun to take advantage of the excellent restaurants that the host city has to offer, and we certainly did that! On Sunday night, just after the closing of PyCon, we went out to dinner at Barroco, a wonderful restaurant in Old Montreal, with my long-time friends Paul and Steve. Good food, wonderful wine, and excellent company made for a very memorable evening.

dinner picture
(L to R) Paul McNett, Steve Holden, Linda and me.

This was my 12th PyCon in a row, and I certainly don’t plan on breaking that streak next year, when PyCon US moves back to the US – to Portland, Oregon, to be specific. I hope to see many of you there!

The Core Deficiency

Core Reviewers

One of the key concepts in OpenStack development is that nothing can get merged into the codebase without being reviewed and approved by others. Not all approvals count the same, though: there are some developers who are designated core reviewers (also referred to as “core developers”) because of their extensive knowledge of the project. No matter how many other reviewers have approved a patch, it takes two core approvals to get any change merged, and any individual core can block a patch. Note: some projects change that requirement, so for this post I’m speaking entirely of Nova.

The Bottleneck

Imposing such a requirement means that there are some very tough hurdles a patch has to clear in order to be merged. This slows down the velocity tremendously. However, I tend to think of this as a very good thing, as it (usually) keeps development from going off in unwise directions. The one time that it is clearly a problem is just before the code freeze, like the one we had a little while ago in Nova for the Kilo release: any code not merged by March 19 would have to wait until the Liberty release, which would be 7 months later! That’s a long time to have to wait to have something that you may have had ready to go months ago. (Yes, there are exceptions to the freeze, but let’s not get into that. The aim is to have zero exceptions).

Core reviewers don’t just review code – they are some of the most active contributors to the project. They also have employers, who frequently require them to attend meetings and attend to other non-OpenStack tasks. And when these demands on their time happen during the rush before feature freeze, the backlog can get overwhelming.

Ideas for Improvement

Joe Gordon posted this message to the openstack-dev list last week, and it touched off a discussion about some issues related to this problem. There were several variations proposed, but they all seemed to revolve around the concept of adding another layer to the mix; in Joe’s case, it was to add a designation of maintainer, whose responsibilities aren’t just about code review, but is a more encompassing notion of investing one’s time to ensure the overall success of the project.

Other ideas were floated around, such as having some additional people designated as domain experts for some part of the Nova codebase, and require one core and one domain expert (junior core? core light?). I think that while that’s a great idea, since cores know the people who are working on the various parts of the Nova code base and tend to rely on their opinions, it would be a logistical nightmare, since a patch could span more than one such sub-domain. The main advantage of having cores approve is that they are familiar with (pretty much) the entire Nova code base, and also have a strong familiarity with the interactions between Nova and other projects, and it is this knowledge that makes their reviews so critically helpful.

The Big Problem

Where I see a problem is that there are only about 15 core reviewers for all of Nova, and there is no clear path to add new cores to Nova. In the early days of OpenStack I was a core reviewer for Nova, but job changes made me leave active development for 2 years. When I came back, it seemed that just about everything had changed! I spent a lot of time revisting code I was once familiar with, tracing the new paths that it took. I’ve caught up a bit, but I realize that it will take me a long time to learn enough to be qualified for core status.

So where are the new cores coming from? In general, it’s been a few very enthusiastic people who routinely spend 60-hour weeks on this stuff. That may be fine for them, but that doesn’t sound like a sustainable plan to me. If we are serious about growing the number of people qualified to approve changes to Nova, we need to have some kind of education and/or training for aspiring cores.

Idea: Core Training

It’s all done rather informally now; I’m just wondering if by making the process a little more explicit and deliberate, we might see better results. Hell, I would love to sit next to Dan Smith or Sean Dague for a couple of weeks and pick their brains, and have them show me their tricks for managing their workloads, and be able to get their insights on why they felt that a certain proposed change had issues that needed correction. Instead, though, I do as many code reviews as I feel qualified to do, and follow as much as I possibly can on IRC and the dev email list. But I know that at this rate it will be quite a while until I have learned enough. I can’t imagine how daunting of a challenge that would be to someone who wasn’t as familiar with OpenStack and its processes

So what would such training look like? It isn’t practical to co-locate a developer with a core, as OpenStack developers are scattered all across the globe (I certainly can’t imagine Dan or Sean wanting me sitting next to them all day, either! ;-). Maybe a regular IRC session? There could be different sessions each week, led by a core in Asia, one in Europe, and one in the Americas, so that developers in different time zones can learn without having to get up at 3am. Perhaps the core could select a review that they feel is illustrative, or the aspiring devs might pick reviews that they would like to understand better. I’m not sure on the details, but I’d like to get the discussion going. I’d love to hear other suggestions.

A New (But Familiar) Adventure

OK, I admit it: I haven’t been posting here as regularly as I should be. It’s not like my life is so boring, or that I haven’t had any interesting thoughts to share. Rather, it’s because I have been so busy and my life changing so fast that I haven’t had the time to sit down, catch my breath, and write things down. I hope to be able to change that moving forward.

Today is the beginning of one of those changes: I start my new job as an OpenStack developer for IBM. In a way I feel like I’m coming back to a familiar world, even though I’ve never worked for IBM before. I was involved in the creation and initial design of OpenStack, and have always felt that it was partially “my baby” (ok, a very small part, but mine nonetheless). I left active involvement with OpenStack a few years ago to start the Developer Experience effort at Rackspace, and have only been tangentially working on OpenStack during that time. With the move to IBM, my focus will once again be on OpenStack, and I couldn’t be happier.

Review: GlueCon 2014

GlueCon 2014 was held on May 21-22 at the Omni Interlocken Hotel in Broomfield, Colorado, and I was fortunate enough to be selected as a speaker there for the second year in a row. For those of you who aren’t familiar with GlueCon, it’s not quite an unconference, but it is a loosely-structured gathering of people working in lots of different technologies that drive the web today, with the focus on the “glue” that holds them all together: developers.

The morning of the 21st was a series of keynotes, and devops seemed to be the topic on everyone’s mind. There was one keynote in general that I strongly disagreed with: John Sheehan of Runscope claimed that “API SDKs Will Ruin Your Life” (the title of the keynote). The assumption, of course, was that the only reason that anyone would use an SDK is because the native API is terrible, and it is the reliance on SDKs that is causing API devs to get sloppy. While that might be true in some sense, the responsibility for creating a good API rests on the API developers, and if they are not professional enough to take pride in the quality of their APIs, I doubt that any SDK would change that. Creating good, consistent APIs is not easy, and typically shouldn’t be left to the the people creating the underlying product that the API is exposing.

The afternoon was broken into several tracks, with the idea that the talks in each track were related, and would happen one right after the other, with no break in between. This was when my talk was scheduled, and it was part of the Cloud track. The talk was titled “How to Leave Rackspace”, and no, I wasn’t encouraging people to leave Rackspace! (I like having a job!). It was about the realities of portability among clouds, and emphasized the importance of data gravity binding you to a provider. I also covered the importance of API compatibility, and how working with providers sharing a common API, such as OpenStack-based providers, won’t do anything about data gravity, but will at least allow your applications to run on different providers.

The script I showed used pyrax to transfer a running compute instance in Rackspace to the HP cloud, so that you could move an entire server with just a few lines of code. The overall lesson, however, was to take the time to choose your provider carefully up front, instead of relying on some promise of portability later. The audience seemed pretty engaged during the talk, but there weren’t many questions and what seemed like lukewarm applause afterwards. I was a little disappointed, but noticed the same pattern for the other speakers who followed me, so I thought it might be due to the format. Later on I did run into several people who attended my session, and their feedback was very positive, so that made me feel better about the talk. This was the first time I’ve given this particular talk, and so I was very appreciative of the feedback.

After my talk was an excellent talk by Cornelia Davis of Pivotal that gave us a peek under the hood of Cloud Foundry. I was familiar with what Cloud Foundry could do, but I hadn’t had any exposure to the underlying architecture, so this was very interesting. Following that were talks on OpenStack Swift and network virtualization, which were both full of useful information, and then there was a talk on “The New Development Experience” that had some good stuff on Bluemix, IBM’s implementation of Cloud Foundry, but was too filled with buzzwords to be interesting. For a sample of what I mean, here’s a sentence from the talk abstract:

“The cloud combined with the need-for-time to value is driving a movement toward a scalable and flexible cloud computing development model that provides application developers with the freedom to innovate rapidly, integrate with existing infrastructure, use or create APIs, and create cloud native applications”.

I spent the rest of the day working at the booth, and noticed a distinct difference in attitude from last year’s crowd. For some reason, the crowd last year was not only very pro-AWS, but actively hostile to OpenStack. This year I didn’t get any of that negativity; instead, I spoke with developers and business people who knew about OpenStack and Rackspace, and simply wanted more information. My favorite, though, was Atsushi Yamanaka, from IDC Frontier in Japan. He flew to the Denver area the previous day from Tokyo to attend GlueCon, and was flying back to Tokyo the day after GlueCon ended. Talk about dedication! He wanted to get a better idea of Rackspace’s presence in the Asia/Pacific area, both current and future, and unfortunately I had to explain that those decisions are made at much higher pay grades than mine!

The second day of the conference similarly opened with a series of keynotes. I had to work the booth and also work on some code, so I didn’t catch most of them. I did, however, catch the keynote by Joshua McKenty of Piston Cloud Computing, which I found to be equal parts entertaining and informative. I’ve known Joshua for several years now, since the birth of OpenStack, and know as well as anyone how he can tend to overstate things just to get a rise out of the audience. He also knocked SDKs, but made it clear that it was only when they were used as crutches for bad APIs, and I had to comment about it on Twitter. The example he gave was that if instead of a website, you had created a duck, when you create the duck API, it would be dumb to expose the gizzard and the inner workings of its circulatory system, yet developers do the equivalent of that regularly. When people want to use your duck, they are interested in its ability to fly or swim or eat or quack, and don’t really care about the incredibly complex implementation details that make those abilities possible. Developers, on the other hand, are so proud of their incredibly complex implementations that they tend to feature them front and center in their API design. This makes for an unwieldy API, which then almost forces users of it to rely on an SDK to be able to do anything useful.

There was another keynote by Julia Ferraioli from Google on developer outreach, and since that’s my job, I was interested to hear what she had to say. Unfortunately, the talk was really nothing more than the general knowledge stuff that people working in this field for any length of time already know; I certainly didn’t pick up anything new. I hope, though, that maybe some of the non-developers in the crowd learned something.

I wish I could comment on more of the talks, but again I spent a good part of the day either working the booth or writing code for work. I did see a couple of talks that revolved around novel uses of Docker to either speed up or reduce the cost of CI/CD enviroments; it certainly seems as though Docker was the new hotness among developers at GlueCon.

I can’t say enough good things about the event itself: the location, the organization, the food/refreshments, the attendees, and the quality of the talks were all excellent. If you get a chance to attend GlueCon 2015 next year, go for it!

PyCon 2014 Review

mirror.jpgBetter late than never, right?

PyCon 2014 happened two weeks ago, and I’m just getting around to write about it now. Why the delay? Well, I took a few days of vacation to explore and enjoy Montréal with my woman after PyCon, and when I returned I found myself trapped under a mountain of work that had built up in my absence. I’ve finally dug myself out enough to take the time to write up my impressions.

This PyCon (my 11th) was different in many ways, not only for the fact that for the first time, the US PyCon was not in the US! It was held in Canada in beautiful (and cold!) Montréal. It was the first time in years that I did not arrive early enough to help with the bag stuffing event, which has always been a highlight of my PyCon experiences. I arrived late Thursday afternoon, and just made it to the Opening Reception, where I not only touched based with my fellow Rackers, but also ran into dozens of Python people I have gotten to know over the years. PyCon is special in that regard: while I have technical contacts at many of the conferences I attend, I consider many of the people at PyCon to be my friends.

I was pleasantly surprised by the bold choice for the opening keynote: John Perry Barlow, one of the founders of the Electronic Frontier Foundation (EFF).


He pulled no punches, and let the audience know exactly where he stood on matters of security, openness, government spying, and free information in a society. I enjoyed his talk immensely, but I knew that some with more conservative views might not respond positively to the anti-corporation, anti-BigBrother tone of his talk, and judging by the sight of several people leaving during the talk, I think I was right. Still, I appreciated that the primary conferences for one of the most important Open languages took that risk.

I attended several sessions, and bounced between them and my duties at the Rackspace booth in the vendor area. We were once again a major sponsor of PyCon, and this gives me great pleasure, as I had first convinced Rackspace to become a sponsor shortly after I joined the company 6 years ago. We benefit so much from Python and the work of the PSF, it’s only right that we give something back.


I won’t go into detail about the individual sessions, but I would encourage you to check out the videos of all the talks that are available on the pyvideo site. The quality of the presentations keeps getting better and better every year, which is a great reflection on the talk selection committee, who had to select just 95 talks from a total of 650 proposals. That is a thankless task, so let me say “thank you” to the folks who reviewed all those proposals and made the tough calls.

I would also like to note that this year 1/3 of the speakers were women, and by some estimates, the percentage of female attendees was nearly as high! (They don’t record the sex of a person when they register; hence the need for estimates). This is a phenomenal result in the traditionally male-dominated tech industry, and it isn’t by accident. The PSF has actively encouraged women to attend, both by creating (and standing behind) a Code of Conduct, as well as offering financial assistance to those who might not otherwise be able to attend. And this year was another first: onsite child care during the main conference days. I think this is an amazing addition to PyCon, even though my kids are grown. It allows many people to come who would otherwise not be able, and also encouraged more families to travel together, making PyCon the hub of their vacation plans.

That’s exactly what I did, too. Linda flew into Montréal on Saturday evening, hung around PyCon for the closing session so that she could get a glimpse into the strange geek world I inhabit regularly, and meet some of my friends. We spent the next few days exploring this city that neither of us had visited before, enjoying ourselves immensely. Coming from South Texas, we could have done without the near-freezing temperatures and snow, though! Here was the view from our hotel window:OLYMPUS DIGITAL CAMERA

It was a wonderful vacation, but much too short! We’re really looking forward to returning next year for PyCon 2015!OLYMPUS DIGITAL CAMERA



Sales Should Not Equal Sleaze

I like to lease automobiles. I could go on at length about the financial pros and cons of leasing versus buying, but I’m not trying to convince anyone here. Just saying that it works for me.

I guess that I should mention that I worked in sales for many years, and have gone through several training programs that ranged from the high-pressure adversarial sales approach to the helpful partnering style that I prefer. So I know enough to recognize when someone is using these tactics against me. And it goes without saying that I strongly dislike when I feel that I’m being played.

My current lease ends this month, and so recently I started shopping around for a new car. Since I don’t care about cars beyond the ability to get me around reliably and comfortably, I talked to people I know who are much more knowledgeable about these things, did a little research, and then went to local dealers to test drive the cars. I liked a few models, including one from the dealer that is the focus of this post, so I asked for a quote on how much the lease would run.

Let me pause here to explain that I will not name the vehicle brand, dealers, or sales people that I am discussing in this post. My goal is not to publicly shame them, irrespective of whether they are deserving of such a shaming. If you know me and want details, I would be more than happy to share them with you privately. I will be telling everyone I know in the area who might be considering buying a car to avoid this particular dealership at all costs.

I received the quote for the lease: $179/month! The email said in nice bold letters: “I have great news we have the [vehicle type]  in stock and available and you have been selected to recive [sic] our Lease Special.

Wow! What a great price! It was around $100 less than what I was expecting! Aren’t I so lucky!

Yeah, and I was born yesterday. Still, even though I figured that the final price would be higher, I still liked the car, so I went in to iron out the details. Up until then, the sales person had been very pleasant and relaxed; not at all the stereotypically pushy type we all loathe.

All was going well until he brought a purchase agreement for me to sign. It had no numbers filled in, but stated that I was agreeing to purchase the vehicle from that dealer. Up until that point, the only number I had in writing was the $179/month quote from the email, and I was tempted to write that in and sign it, but instead pointed out that I wasn’t about to sign a blank commitment. He fumbled with an explanation, but I still refused. At that point he got up to “speak to his manager”, and about 10 minutes later returned with that sheet with a figure filled in: $354/month!

Now let me explain: if I had been quoted that price originally, I would still have been interested, even though it was a bit higher than what I would have been paying for one of the other car models I was considering. But the extreme nature of the bait-and-switch tactics employed just really turned me off. I don’t mind paying a fair price for things I want, but I absolutely refuse to do business with companies that use these sorts of dishonest tactics in order to trap you into doing something that may not be in your best interest.

After I explained that not only was the price higher than I was willing to pay, but that I felt like I was being duped, he went back to his manager for another 10 minutes or so. I probably should have walked out right then, but I guess I still had some glimmer of hope that they would at least realize that their BS tactics weren’t going to work with me, and would instead start dealing with me straight. Instead, he came back with a quote that was a little lower, but still way higher than anywhere else (yes, I checked other dealers before coming in!). The manager even came over to try to close me, but at that point I wasn’t having any of it. I got up and left.

I still liked the car, though, and looked online for a dealer over on the other side of town. Their website quoted me $326 for the exact same car with the same features. I contacted them online, and told them that I had just been through Salesman Hell, and was hoping that I could work with them. After a few emails and phone calls to iron out the details, I agreed to lease the car from them. Everyone I dealt with at the new dealership treated me with respect; they realized that I wanted to get a car, and that they wanted to sell their cars, so it was in both of our interests to work together to make that happen.

I feel sorry for businesses that are stuck in the mindset that the only way to get people to do business with them is to trick them. Unfortunately, there are many of these still around, and I just happened to run into one of them.

PyCon Canada Review

It’s been almost a week since PyCon Canada ended, and I had meant to write up a review, but I was busy, so better late than never.

Besides an opportunity to learn more about Python, this conference served as a “training ground” for next year’s PyCon Montréal, which is where the 2014 US PyCon will be held. Yeah, I know, it seems silly to still call it PyCon US instead of PyCon North America, but I suppose it has to do with domain names, trademarks, etc., so I’ll refrain from ranting about that. So to that end, I volunteered to be an MC for roughly half of the total conference. Normally for PyCon you have two roles: Session Chair, who introduces the speaker, times the talk, and manages the Q&A session; and Session Runner, who makes sure that the speaker gets from the green room to the correct session room, and who handles any problems such as missing video adapters, etc. But for PyCon CA, this was changed so that there was an MC and a Runner. The MC was someone with experience from PyCon US, and they served for half of the day in the room. There were fresh runners for each session, but rather than just escort the speaker, they did everything that session chairs and runners do, with the MC guiding them and making sure that they knew what was expected, and who could help out if there were any problems. So in many ways, though I spent half the conference running the talks, I felt like I hardly did anything. The runners did just about everything, and I only had to help out a few times. I have no worries that they’ll be ready for the big leagues next year!

As expected, there were many interesting sessions. I’m not going to give you a summary of everything I saw that I liked, but instead I’ll touch on a few highlights. Probably the one that made the biggest impression on me had nothing to do with Python per se, but instead was about Sketch, a programming language designed by MIT to get children thinking about programming without saddling them with the tedium of learning syntax. This was shown in the keynote on Sunday given by Karen Brennan, who is an Assistant Professor of Education at Harvard University. Besides making it easy for the kids to get started learning about programming concepts, it also makes it easy for them to share their projects, or to take someone else’s project and add to it. In other words, they are teaching kids about the benefits of open source and shared code! And while it’s designed for kids, I think it would also be a huge benefit for non-technical adults by helping familiarize themselves with programming without having to get totally geeked out.

Another talk I enjoyed was Git Happens, which was given by Jessica Kerr. This was one of the talks for which I was MC’ing, and hadn’t otherwise planned on attending, as I’m comfortable enough with Git. But rather than be bored, her way of explaining how Git works was much, much clearer than I have ever been able to manage when helping someone else get up to speed with Git.

Brandon Rhodes gave an excellent talk entitled Skyfield and 15 Years of Bad APIs , which covered his efforts to re-write his astronomical calculation library. It was fascinating to see how decisions which seemed reasonable at the time turned out to be less than optimal, and how he learned from them to make the new version much cleaner. As an aside, if you haven’t seen Brandon talk, you’re missing something special. He blends insight with an extremely dry sense of humor, making for a very enjoyable session.

I’ve already written about Dana Bauer‘s Red Balloon demo, so I won’t repeat it here. Suffice it to say that it captivated the attention of everyone in the room.

My talk was basically the same talk I had given the previous month at PyCon Australia, but in a much shorter time slot: 20 minutes instead of 45! I spent much of the time before the conference consolidating my slides, and eliminating anything I could in order to cut the length of the talk. I practiced giving the session over and over, speaking as fast as I could. When the time came, I apologized in advance for the speed at which I was about to give the talk, and then blazed through it. To my surprise, I managed to finish it in 18 minutes, and had time for a few questions! Afterwards I spoke with some who attended the talk to get an idea how it seemed to them, and they didn’t feel like I skipped over anything that made it hard to follow, which was my goal when I was paring it down.

After the sessions had ended, I went with a fairly large group of people to a local restaurant. I didn’t know very many people, and those with whom I sat were all from the team in Montréal who were going to be running PyCon 2014. They started out as strangers, but by the end of the evening (and more than a few pitchers of beer!), we became friends, and I look forward to seeing them again in Montréal next April!

Technical Career Tracks

I’ve been involved in discussions with technical people about the disconnect between companies that want/need to hire and retain more technical people, but whose policies seem to favor those in the managerial roles. About a year and a half ago, I had a long talk with a fellow Rackspace employee who was charged with developing our “Technical Career Track”, which was going to help us succeed in attracting and retaining the best technical talent we could. Being a technical person myself, this was of course of great interest to me. But though the plans were extremely well-intentioned, they fell short of their goals.

Since then I’ve had the opportunity to speak at a lot of tech conferences, and I’ve used them to talk with people I’ve met there about how they saw the situation where they work. Wow, talk about opening the floodgates! Almost without exception, every technical person in a company of more than, say, 100 people, felt the same way: that they would be better off switching to a manager role, at least as far as career prospects and salary are concerned.

Based on those discussions, I believe that I’ve identified a fundamental flaw in the way that most organizations are structured that prevents even the best-intentioned companies from being able to treat their technical talent equitably.

I should point out that although I work for Rackspace, and that we are struggling with the same issues, that this is not based solely on this one company. Rather, the discussions I’ve had inside of Rackspace have led me to talk with others when I’ve traveled to conferences, and who are in no way connected to Rackspace, in order to get a variety of insights and observations from employees of many different companies. I felt that the added viewpoints would help me to separate issues that were intrinsic to Rackspace’s culture with those that were common to most tech companies. The points I am making apply to nearly all companies of people I spoke with, including Rackspace, but this is in no way to be taken to be directed at Rackspace in particular.

After much discussion, I can say that the problem is inherent in the way that the Human Resources department organizes the employment hierarchy, and not the result of a company consciously discounting  the value of technical jobs. Let me explain: to HR, every employee needs to report to someone, and the person they report to needs to report to someone above them, on and on until eventually everyone reports to the CEO. This is an absolute requirement, and is modeled unquestioningly in every company with which I am familiar.

So what? Well, what that means is that if a new development project is started, a new team is formed, usually consisting of developers, business analysts, etc., and all of these people have to report to a manager. If no one of that group is already designated a manager, someone is placed in that role; a team simply cannot exist without a manager, at least in the view of almost all HR systems. This means that someone has to be promoted to that role, and usually it is someone who has displayed competency, initiative, and other positive qualities in their previous work. On the developer side, though, there is no such automatic role that must be filled. Usually technical levels are designated numerically, with the most junior devs being level 1, and the more experienced devs as level 3, 4, or even 5. In order to be equivalent to the managerial track, the technical track would have to require that every team have a Tech Lead as the leader of the developer team, but that isn’t the case in any company that I interviewed. And like the person selected to become a manager, the newly promoted developer would be the one on the team who has most consistently demonstrated solid work.

Typically technical promotions are always strictly review-based, and only happen on a pre-determined schedule (such as annual reviews). More importantly, they are completely at the discretion of the manager. There may be objective criteria for each level written down somewhere, but simply meeting those criteria is not a guarantee that you will be considered for a promotion to that level. And to make matters worse, in a dynamic, fast-moving company, developers rarely have continuity of management, so while a growing company needs more and more managers, each successive manager evaluates their devs based on their improvement from the day that the manager started, not since the dev started. This was far and away the biggest complaint that I heard, and it was the one constant across the various companies. Every time there is a re-org, or a new team created due to growth, or a re-assignment to another team, all the work you’ve done in the past is discounted, and you’re pushed back to the starting line.

What is the alternative? How can we change this? Imagine a company in which every development team had to have not only a manager, but a technical lead. That Tech Lead would be in most companies a Developer 4, 5 or higher level title, and if no one on the team was already at that level, someone would have to be promoted to it, just as someone would have to be promoted to a manager. It would be just as rigid an HR requirement. Then later, if a team is successful and grows to the point where it needs to be split, on the management side the former manager now becomes a Director or similar role, and a new manager is promoted for each of the new teams. To be equitable, on the technical side, the former Tech Lead should be promoted to whatever the next higher level is (the names they give these roles are laughably grandiose!), and two new Tech Lead promotions need to be made. This would ensure that the number of people at the equivalent level on both the managerial and technical tracks would remain equal, at least for technical teams. And it would remove the subjective, discretionary promotion practices that end up leaving developers feeling like they have no future with a company. The result is that on the technical side of the business, there is a requirement for parity in number between the managerial track roles and the technical track roles.

The objections I’ve heard when I’ve shared these insights is that not every developer is qualified to be a Tech Lead. That’s true, but let me ask you this: why is everyone promoted to a manager role any more qualified? Aren’t there just as many people who get into management only to find themselves either ill-fitted to the role, or in over their heads? The same would be true for developers: some would falter, while others would thrive. Many would not want the tech lead role; they would be more comfortable just writing code instead. That’s no different than someone who doesn’t want to be a manager, and have to deal with personality conflicts, and all the meetings and paperwork associated with the role.

I think that every company that is serious about attracting and retaining technical talent should look at the numbers: do they have just as many people in high technical roles as they do in high managerial roles? Are the numbers of people at the Director or VP level who are technical equal to the numbers of mangers? Are they even close? If not, then you need to question your HR structures. Change the rules so that technical people are required to be promoted at the same rate as managers, and you’ll probably be amazed at the results.