I had a few days between PyCon Canada and PyTexas, so I took my first vacation days in a long time to drive down to Rochester, NY, and visit my family and friends. For those who don’t know my history, I lived in the Rochester area for 15 years before moving to San Antonio a little more than 2 years ago, and since then I haven’t been back. That means that I hadn’t seen any of my friends since then, and, what was even stranger, I hadn’t seen my older son Mike in that time, either. Sure, we talked on the phone and video chatted from time to time, but it is really odd to think that it had been over two years since I had been in the same room as this person who dominated my life for so many years. I guess that’s the difference between having children and having adult offspring.
With such a short visit it was hard to consider it a proper “vacation”, but it was really refreshing to my soul to re-establish contact with some of the people who have played a significant role in my life. And it was fun to see Rocco, my puppy of many years!
Today was the opening of PyCon Canada 2013, with some volunteer prep work to get things ready in the afternoon, followed by a casual mixer in the evening. This is only the second PyCon here in Canada, and it’s already grown significantly from last year’s small venue. I didn’t go to that one, but I’ve been told it was small enough to be held in a Legion hall. This year it is being held in the Chestnut Conference Centre, which is part of the University of Toronto.
The mixer was a lot of fun, with food and beverages graciously supplied by Upverter. I got to meet several people as well as connect with many others whom I had already met. Based on my small sampling, there was a good mix of seasoned Python developers and relative newcomers to the language. One of the best parts about conferences like this is learning what others are doing with the language, and there was a wide range just in the discussions that I had: some doing web development, while others focused on internal tools for their companies, while still another was working with medical research teams to analyze data.
The conference proper begins tomorrow morning, with a keynote by Jacob Kaplan-Moss, followed by a full day of sessions. My talk isn’t until Sunday afternoon, which means that I’ll probably spend a lot of tomorrow revising my slides again and again between now and then.
In an earlier post I reviewed the OpenStack miniconf that preceded the main PyCon, which was held in Hobart, Tasmania on July 6–7. I had meant to write this shortly after PyCon ended, but the whirlwind of travel back to the US and getting back into the daily grind pushed it off my plate.
The conference was recorded, and all the videos are available on the pyvideo.org website. I encourage you to watch as many of the sessions that interest you as you can – lots of good stuff in them!
The conference actually started for me earlier – the organizer, Chris Neugebauer, had asked for volunteers to help with the conference prep work: badges, swag, all that stuff. This was on the Wednesday before the conference, which happened to be the day I arrived, so it was as good an excuse as any to get out of my hotel and into Hobart. For those of you who have never gone to a PyCon, it is completely run by volunteers. No one gets paid; no one gets free admission; no one gets special perks. This was shocking to me when I moved from the Microsoft conference world a decade ago, where conferences were run as profit centers, and attendees paid for tickets that cost well over $1,000, but who could then relax and treat their time there as a vacation (which many did, at their employers’ expense). But PyCons are the exact opposite, and as a result everyone has a stake in the conference experience. I’ve found that volunteering not only makes you feel like you’re contributing, but it also means that you meet a lot of interesting people who might otherwise remain anonymous faces in the crowd.
The picture above is a quick snapshot I took as we carried out the task of wrapping a thousand coffee cups (yes, that’s correct: 1000) with vinyl cutouts of the coffee sponsor’s logo. Why did we do this? Because a run of pre-printed cups had a minimum of 50,000, and, well, PyCon just ain’t that big! And to make things even more fun, the logos were slightly bigger than the cup, which made it difficult to wrap cleanly. Tedious, huh? But rather than being a drag, it was a great time as the dozen or so people there had great conversations as we wrapped the cups one by one. As the only American there, and this being my first time in Australia, observations on cultural differences was a big topic, and I certainly learned a lot. We were all shocked when we were done to learn that we had spent 4 hours working on this stuff; it certainly went by much faster than that.
After the work was done, we went out to eat, where (among other things) I learned that Tasmania has a high-end distillery business that is producing spirits that rival other parts of the world, especially in the high-end whiskey area. Two of the best are the Nant and Lark distilleries. I’m not a big Scotch drinker, but I appreciated the quality that these two distilleries are producing. Unfortunately, they are relatively small, and their products are nearly impossible to find in the US.
The conference proper started off Saturday morning, with Alex Gaynor giving the keynote. It set the tone for the conference, and we spent much of the next two days exploring what it meant to be both a software developer in general, and a Pythonista in particular. I then went to Nick Coghlan’s talk about the state of Python packaging; it’s no secret that packaging has always been a pain point for Python, but it is great to see that the different efforts are now unified under Nick’s leadership.
Next up was Jacob Kaplan-Moss’s talk on security in web apps in the Python world. I was aware of most of the attack vectors thanks to my security training at Rackspace, but it was somewhat shocking to see how easy it is to expose a vulnerability if you don’t know it’s there. If you write web apps that are exposed to the public, make sure you watch the video of his talk. Now.
In the afternoon I drifted between various talks which, to be frank, varied in their appeal. That’s not to say that the content wasn’t valuable; it just wasn’t of particular interest to my work. I did end up skipping a few talks to work on my slides for my session; I don’t think I’ve ever given a talk without editing it right up to the last minute, and this was no different.
The day ended with a full hour of Lightning Talks, which are always one of my favorite parts of PyCon, and the lightning talks at PyCon AU did not disappoint. If you aren’t familiar with the concept, they are a series of short (5 minutes – enforced!) talks on whatever the speaker wants to talk about. There was a good mix of dense information, rants, product description and general wackiness. If you’ve never been to a PyCon, don’t miss the lightning talks! I didn’t take notes, but fortunately Thomas Sutton did.
One thing that was different at this PyCon was the Conference Dinner after the sessions on Saturday. All the attendees were gathered into a room with many large round tables that sat about 10 people each. There was wine and beer served, along with iced tea and soda. The food was served buffet-style, but this was no ordinary conference fare. Fresh oysters! Smoked salmon! Freshly-carved roasts! All this with an assortment of salads, vegetables, breads, and deserts. I followed my usual practice and sat at a table where I knew no one, in order to make new friends and learn about what they are doing with Python, but we ended up having much less technical discussion for some reason (big surprise!).
The evening wasn’t all food and drink, though. There was an evening keynote by Mark Pesce of MooresCloud that talked about their work integrating Raspberry Pi units to control LED displays. An example of this was used in the lightning talks to provide a visual indication of the time remaining for each speaker; as time wound down, the LEDs changed from green to yellow to red to flashing red to off. During the keynote, though, he demonstrated how the Pis could change the light display in response to a stream of Twitter hashtags, creating a virtual tug-of-war. He had half the room send tweets with one hashtag, while the other half tweeted a different hashtag, and the color distribution of the display changed in response. Apologies to anyone who follows me on Twitter for the hashtag spam that night! So while this was a somewhat silly example of what it could do, it certainly demonstrated that you could program the device to respond to any sort of dynamic data.
Sunday morning’s sessions were… ok, you got me. I skipped the sessions at the prompting of Alan Perkins, the director of our Sydney office, who grew up in Hobart and knows the area well. He wanted to take me along with a couple of other Rackers up to the peak of Mount Wellington, which is probably the most ubiquitous visual in the Hobart area. Unfortunately, being the middle of winter there, the roads were closed due to iciness. So instead Alan took us on a tour along the coastline of the Derwent estuary through some of the smaller towns and some truly gorgeous scenery.
We made it back in time for the last morning session, but I needed to prepare for my talk in the afternoon. After lunch Richard Jones gave an excellent talk entitled “Don’t Do This”, which was an exploration of some of the oddities that were possible in the Python language. Fortunately I can say that none of them could be found in my code! Next up was Dylan Lacey from Sauce Labs, with a talk on using Splinter to streamline your testing by providing an API that allows you to automate browser actions. Splinter looks pretty interesting, and if I were doing web acceptance testing, I would definitely check it out.
After the afternoon tea break, I saw Luke Miller’s talk on developing an indie game with Python that featured a gay theme. I had never really thought about it before, but there weren’t any games that featured leading characters who were gay, and which might appeal to gay audiences more than the current selection available. The talk wasn’t just about the game’s theme; it was a very in-depth look into what it takes to develop a game and make it interesting. I must say that I learned more in that talk than in any other I attended, as the material was totally new to me, and it was presented very clearly.
The final session was by Adam Forsyth on Getting the Most Out of StackOverflow. Adam is seriously active on that site, and knows quite a bit about how things work, and how it can be overwhelming at times to a newcomer. He had many tips for those who wanted to both ask better questions that would have a good chance of receiving a great answer, as well as for those who wanted to be able to find questions to answer before others did.
Last but not least were the second day of lightning talks. Once again they were excellent, and again, Thomas Sutton took much better notes than I did.
That was the end of the conference, and Chris Neugebauer, as the conference chair, gave his closing remarks, thanking everyone from the sponsors to the attendees for making PyCon AU 2013 a truly remarkable conference. He received a standing ovation of thanks from the audience for all his hard work, and he deserved every clap.
Many people left after that, but for those of us who remained, things were not over. We had the upper level of Jack Greene’s reserved, and it was packed with Pythonistas enjoying some final food, drink, and conversation.
I’ve been to 10 PyCon US conferences, and while PyCon AU 2013 was much smaller in overall size, it matched its US counterpart in terms of spirit, content, and sense of community. Next year it moves to Brisbane, and I would certainly love to be able to return.
I’ve been involved in discussions with technical people about the disconnect between companies that want/need to hire and retain more technical people, but whose policies seem to favor those in the managerial roles. About a year and a half ago, I had a long talk with a fellow Rackspace employee who was charged with developing our “Technical Career Track”, which was going to help us succeed in attracting and retaining the best technical talent we could. Being a technical person myself, this was of course of great interest to me. But though the plans were extremely well-intentioned, they fell short of their goals.
Since then I’ve had the opportunity to speak at a lot of tech conferences, and I’ve used them to talk with people I’ve met there about how they saw the situation where they work. Wow, talk about opening the floodgates! Almost without exception, every technical person in a company of more than, say, 100 people, felt the same way: that they would be better off switching to a manager role, at least as far as career prospects and salary are concerned.
Based on those discussions, I believe that I’ve identified a fundamental flaw in the way that most organizations are structured that prevents even the best-intentioned companies from being able to treat their technical talent equitably.
I should point out that although I work for Rackspace, and that we are struggling with the same issues, that this is not based solely on this one company. Rather, the discussions I’ve had inside of Rackspace have led me to talk with others when I’ve traveled to conferences, and who are in no way connected to Rackspace, in order to get a variety of insights and observations from employees of many different companies. I felt that the added viewpoints would help me to separate issues that were intrinsic to Rackspace’s culture with those that were common to most tech companies. The points I am making apply to nearly all companies of people I spoke with, including Rackspace, but this is in no way to be taken to be directed at Rackspace in particular.
After much discussion, I can say that the problem is inherent in the way that the Human Resources department organizes the employment hierarchy, and not the result of a company consciously discounting the value of technical jobs. Let me explain: to HR, every employee needs to report to someone, and the person they report to needs to report to someone above them, on and on until eventually everyone reports to the CEO. This is an absolute requirement, and is modeled unquestioningly in every company with which I am familiar.
So what? Well, what that means is that if a new development project is started, a new team is formed, usually consisting of developers, business analysts, etc., and all of these people have to report to a manager. If no one of that group is already designated a manager, someone is placed in that role; a team simply cannot exist without a manager, at least in the view of almost all HR systems. This means that someone has to be promoted to that role, and usually it is someone who has displayed competency, initiative, and other positive qualities in their previous work. On the developer side, though, there is no such automatic role that must be filled. Usually technical levels are designated numerically, with the most junior devs being level 1, and the more experienced devs as level 3, 4, or even 5. In order to be equivalent to the managerial track, the technical track would have to require that every team have a Tech Lead as the leader of the developer team, but that isn’t the case in any company that I interviewed. And like the person selected to become a manager, the newly promoted developer would be the one on the team who has most consistently demonstrated solid work.
Typically technical promotions are always strictly review-based, and only happen on a pre-determined schedule (such as annual reviews). More importantly, they are completely at the discretion of the manager. There may be objective criteria for each level written down somewhere, but simply meeting those criteria is not a guarantee that you will be considered for a promotion to that level. And to make matters worse, in a dynamic, fast-moving company, developers rarely have continuity of management, so while a growing company needs more and more managers, each successive manager evaluates their devs based on their improvement from the day that the manager started, not since the dev started. This was far and away the biggest complaint that I heard, and it was the one constant across the various companies. Every time there is a re-org, or a new team created due to growth, or a re-assignment to another team, all the work you’ve done in the past is discounted, and you’re pushed back to the starting line.
What is the alternative? How can we change this? Imagine a company in which every development team had to have not only a manager, but a technical lead. That Tech Lead would be in most companies a Developer 4, 5 or higher level title, and if no one on the team was already at that level, someone would have to be promoted to it, just as someone would have to be promoted to a manager. It would be just as rigid an HR requirement. Then later, if a team is successful and grows to the point where it needs to be split, on the management side the former manager now becomes a Director or similar role, and a new manager is promoted for each of the new teams. To be equitable, on the technical side, the former Tech Lead should be promoted to whatever the next higher level is (the names they give these roles are laughably grandiose!), and two new Tech Lead promotions need to be made. This would ensure that the number of people at the equivalent level on both the managerial and technical tracks would remain equal, at least for technical teams. And it would remove the subjective, discretionary promotion practices that end up leaving developers feeling like they have no future with a company. The result is that on the technical side of the business, there is a requirement for parity in number between the managerial track roles and the technical track roles.
The objections I’ve heard when I’ve shared these insights is that not every developer is qualified to be a Tech Lead. That’s true, but let me ask you this: why is everyone promoted to a manager role any more qualified? Aren’t there just as many people who get into management only to find themselves either ill-fitted to the role, or in over their heads? The same would be true for developers: some would falter, while others would thrive. Many would not want the tech lead role; they would be more comfortable just writing code instead. That’s no different than someone who doesn’t want to be a manager, and have to deal with personality conflicts, and all the meetings and paperwork associated with the role.
I think that every company that is serious about attracting and retaining technical talent should look at the numbers: do they have just as many people in high technical roles as they do in high managerial roles? Are the numbers of people at the Director or VP level who are technical equal to the numbers of mangers? Are they even close? If not, then you need to question your HR structures. Change the rules so that technical people are required to be promoted at the same rate as managers, and you’ll probably be amazed at the results.
Friday was the pre-conference day, with two miniconfs: one for Django, and the other for OpenStack. While I’d love to spend some time digging deeper into Django, I figured that given my background as an OpenStack developer, the OpenStack miniconf was for me.
There were probably 40 people or so in attendance, and it was a good mix of those who were completely new to OpenStack, those who have looked into it a bit and wanted to learn more, and those who either were core developers or (in my case) a former core dev. Tim Serong from OpenSUSE opened up the day with the talk “WTF is OpenStack?“, which was an excellent introduction for those who had heard a lot about this “cloud” stuff. The presentation included the classic spoof by The Onion about “that Cloud thing” (with apologies to Robert Collins of HP, who really does totally know what that is). He covered all the projects within OpenStack, and how they work together.
Robert Collins then followed with a talk on “Deploying OpenStack using OpenStack“, which tackled the issue that although OpenStack allows you to automate the provisioning of cloud resources, installing OpenStack itself is a terribly manual process. His solution is “TripleO“, which stands for “Openstack On Openstack”. It sounds similar to the iNova project from Rackspace, but with several differences. From the ReadMe:
TripleO is the use of a self hosted OpenStack infrastructure – that is OpenStack bare metal (nova and cinder) + Heat + diskimage-builder + in-image orchestration such as Chef or Puppet – to install, maintain and upgrade itself.
Christopher Yeoh of IBM was next, and gave an excellent overview of the changes coming in v3 of the Nova API. The current v2 API was smartly designed by making only a precious few critical parts part of the core API, and making everything else an extension to this core. The problem is that some of the implementation details that seemed wise at the time are starting to show some cracks, both with consistency in naming and in the connection to the core Nova code. Nova v3 API addresses these problems by removing the requirement that extension name matches the class name, allowing for cleaner (and thus more consistent) extension naming. Extensions now must be derived from a common base class; this was optional in v2. Overall, it was apparent that the people working on the v3 API had learned the lessons that the v2 experience offered, both good and bad, and that as a result the v3 API will be much more consistent and robust.
After lunch Tim Serong gave a talk on the state of OpenStack development on OpenSUSE. They are doing some interesting stuff with Crowbar for deployment, and have spent a lot of time on their internal CI processes. I didn’t take very extensive notes, as this is an area that I have little experience and/or interest in, but it was obvious that Tim is passionate about this, and that they are doing some excellent work at OpenSUSE.
Next up was Robert Collins again, this time talking about testing, covering both improvements in the tools themselves (e.g., the testtools module), as well as his work in creating a test runner runner. No, that wasn’t a stutter – this is a script to run a test runner, such as Jenkins in this case. With the growth of projects in OpenStack, it can take a very long time to run the tests, which is done when a change is first proposed for review, and then again when it has been approved to ensure that no conflicts have creeped in while the change was in the review process. In order to speed this up, his test runner runner breaks the tests into several parallel processes, and then aggregates the results. This means that the time required to run all the tests can be reduced greatly, depending on the number of parallel processes. It also provides for test randomization, which can help reveal hidden test inter-dependencies.
Next up was a talk on how to get involved in OpenStack development, which covered the basics of where you can find bugs to work on, or blueprints to contribute to, as well as the review process and Gerrit. Michael Still from Rackspace was supposed to give this talk, but the birth of his child was a bit more important, so Robert Collins stepped up and did the talk for him. This was more of a review session for me, but it had a lot of useful information for those who were new to OpenStack development.
That was the last talk; after that was the hackfest, which had the goal of getting participants to find a fairly simple bug, fix it, and submit it for review. In practice, though, most of the time was spent helping people get Devstack up and running on their machines. Those of us who were OpenStack ‘veterans’ helped where we could, and in the process I had some great discussions with people about OpenStack, so even though we never got to fix any bugs, I believe that the people there who were new to OpenStack got a lot out of that time.
Finally was the bar track! Many of the miniconf attendees, myself included, retired to one of the many bars here at Wrest Point, and enjoyed a cold beer after a long day of learning about OpenStack.