Day 47: Python Virtualenvs

If you’re not a Python programmer, you probably won’t find much in this post. Sorry.

If you are a Python programmer, then you probably know about virtualenvs, or virtual environments. They allow you to create several different Python environments to work in: each can have it’s own version of Python, as well as its own installed packages. This means I can work on a project that has particular requirements, and then switch to a project with completely different requirements, and the two won’t affect each other.

A while ago I used to use vitualenvwrapper, which made working with virtualenvs a lot easier. But as I switched to Python 3 over the past few years, I started to have some issues where it didn’t work correctly (sorry, I no longer recall precisely what those issues were). I’m sure a lot had to do with the addition of the venv module into Python 3, which allowed you to create a virtualenv by running python -m venv /path/to/virtualenv.

For a while I ran all the commands manually, but I have the quality that makes for good programmers: I’m lazy. A lazy person doesn’t want to do any more work than they have to, so if I can automate something to save time over the long run, I will. And here’s what I came up with:

export VENV_HOME=$HOME/venvs function workon { if [ -z $1 ] then ls -1 $VENV_HOME else source $VENV_HOME/$1/bin/activate fi } function mkvenv { python3 -m venv $VENV_HOME/$1 workon $1 pip install -U pip setuptools wheel pip install ipython pytest-pudb requests } function rmvenv { command -v deactivate rm -rf $VENV_HOME/$1 } _venvdirs() { local cur="$2" COMPREPLY=( $(cd $HOME/venvs && compgen -d -- "${cur}" ) ); } complete -F _venvdirs workon rmvenv

These lines should be added to your .bashrc in Linux, or your .bash_profile for Macs. I haven’t tried them with zsh yet, so no guarantees there. Let’s go over what these lines do.

Line #1 defines the directory where the virtualenvs will be stored. You can store them anywhere; it doesn’t make any difference.

Lines #3–10 define the workon function, which activates the specified virtualenv, or lists all virtualenvs if none is specified. Lines 12–17 define the mkvenv function, which creates a new venv, and lines 19–22 define rmvenv, which deletes virtualenvs when you no longer need them.

I’d like to point out 2 lines in mkvenv that you can customize. Line #15 updates the installed versions of pip, setuptools, and wheel. If for some reason you don’t want the latest versions of these, edit or remove that line.

Line #16 is more interesting: nearly every virtualenv I create needs these packages installed. Rather than install them one-by-one, I add them when I create the virtualenv. If you have different packages you always want available, edit this line.

Finally, lines #24–29 are of utmost importance to someone lazy like myself: they provide auto-completion for the other commands. I had to learn about bash completion to get that working, but it turned out to be much easier than I had imagined.

Here is a gif showing it in action:

Try it out! Let me know if you find this useful, or if you have suggestions for improvement.

Day 45: Composing Music and Code

I am not a musician. Yeah, I was a vocalist in a couple of bands in high school and college, but all I really did is copy the original song. The notion of coming up with a new approach to a song was way beyond my imagination.

I am amazed by people who can write their own original songs. How can you create a melody from… nothing? It really is beyond my ability to comprehend. Lyrics seem straightforward enough, but the music?

A few years ago at a conference I was hanging around in the evening with some friends, several of whom were musicians that had brought their guitars. One of them played a song, and I asked him where the song was from. He replied that it was one of his own. “Wait: you wrote that yourself?” That surprised the hell out of me!

Now that my curiosity was piqued, I couldn’t help but ask him how he does it. He replied that sometimes he has some little bits of music in his head, just a couple of notes. When he’s practicing his guitar, he’d just sit around playing some small riffs; just a few notes here and there. He’ll play one, then change it up a little bit, and if it sounds better, he’ll start working with that. Over time he’ll have a few more such bits floating around in his head, and every now and then he’ll play a few that just start to flow together. At that moment he can hear the overall melody, and he has to write it down, or record it – anything to capture it before it fades. It doesn’t matter what time of day it is, or what else is happening; he can’t do anything else until he has it down on paper/tape/whatever.

Now I understood: his process for composing music was very similar to what I experience when coming up with a coding design to a given problem. I’ll play around in my mind with a few approaches I’ve used in the past, changing them around as needed to see if that would work. At some point, though, the pieces seem to align themselves into the solution I need, and I can see it clearly in my mind. At that point, I have to write down what I’m seeing before I lose it. I can remember times when I missed appointments because I was heads-down writing such a solution.

The act of creation is remarkably similar for both cases: playing around with ideas/riffs, and once the solution “appears”, a furious effort to record it. And it’s that “playing around” part that is really hard to explain to others: to them, it just looks like you’re screwing around instead of working.

I do my best thinking when I’m walking, not when I’m planted in front of a keyboard. If you’re ever stuck trying to come up with a solution, try changing what you’re doing. Go for a walk, work on your garden, vacuum the floor – anything to move away from the computer and get your body moving. That act alone helps free the brain to make connections that it might otherwise never make.

Day 35: Chopped Candidate

Have you ever watched the TV show Chopped? If you haven’t, it’s a competition among 4 chefs. There are 3 rounds, and after each round, one of them is chopped (eliminated), until one remains. The winner gets a cash prize. This would seem like a good way to determine who is the best of the group, right?

The problem is how the competition is run: each round the chefs are given a basket of “mystery” ingredients that they can’t see until the round begins. And more often than not, the basket contains, shall we say, “odd” combinations. One such basket contained blood orange syrup, the African spice blend ras el hanout, hot cross buns, and lamb testicles. The chefs can add other staple ingredients, but those four flavors have to be featured prominently in the result.

And if that isn’t difficult enough, there is a time limit that is always ridiculously short. The chefs had 20 minutes to create an appetizer from the basket I described above: 20 minutes to create a recipe, determine what other ingredients to add, prepare and cook the food, and then plate it for a beautiful presentation.

I must confess that I find the show very entertaining, and have watched countless episodes. And I’m not alone: the show has been running for 44 seasons over the past 11 years. But let me ask you: if you were opening a restaurant, would this be the way you would select your head chef? I would hope not! Any restaurant that would spring surprises on their chefs and expect them to deliver first-rate food in impossibly short time limits wouldn’t last very long.

Which brings me to the point of all this: if you are interviewing for a programmer, do your interviews actually determine how well they would be able to work in your team? How positive their contribution will be?

Making a candidate live code a solution to a problem they’ve never seen before in a short period of time with people watching their every keystroke is the software development equivalent to being on Chopped. I certainly hope that your work environment isn’t anything like that. So why would you think that a live coding session in an interview tells you anything about their potential?

What artificial scenarios like Chopped or live coding interviews do is test a candidate’s ability to handle stress. Personally, I’ve never had a problem with live coding, but then again I’ve never had test anxiety in school, either. I’ve seen many talented developers choke under those circumstances, but that doesn’t mean that I wouldn’t want to have them on my team.

What does it say about your company as a place to work if the bar they have to clear is how well they can handle high levels of stress?

When I first started interviewing candidates when I was at Rackspace, the standard was to have one interviewer do a live coding challenge, and another ask one of those bizarre, abstract brainteasers (“Walk us through your thoughts…”). Once again, these practices just show how nervous someone is in what is already an inherently stressful situation. That link includes a juicy quote:

These types of questions are likely to frustrate some interviewees so watch out for those who aren’t willing to play the game. It’s an interview after all and you make the rules.

Mark Wilkinson, head of recruitment, Coburg Banks

It’s all a game to him, and if asking questions with no right answers eliminates potentially good candidates, tough. It sounds like he is more interested in seeing who can tolerate being bullied than finding the best people for his company.

After sitting through some of these types of interviews at Rackspace, I campaigned internally to change these practices, because I saw some intelligent and capable candidates get flustered and end up looking dumb. I found that there are better ways to determine if someone is a good addition to your team. Perhaps I’ll elaborate more about these in a future post…

Day 31: Using etcd As a Mediator

etcd is a database originally developed by CoreOS, and is most famously used as the database at the heart of Kubernetes. It is a distributed key-value store, which in itself is not all that remarkable. The thing about etcd that makes it so attractive is the ability to watch a key for changes.

Other key/value stores, such as Redis, have implemented a similar feature, and may work just as well for you. I’ve been using etcd for years, and it’s worked well for me, so I’ve never had a reason to try these other tools.

For most data stores, the only way to find if a particular value has changed is to poll. You issue a query for that value on a regular basis, and compare it to the last value returned to see if it has changed. This is terribly inefficient, especially with values that don’t change often. It’s also inexact with respect time, because your system’s reaction to a changed value depends on the interval between polls. Longer intervals, while less chatty, mean that more time will elapse between when the value changes, and when your application responds to that change.

Enter etcd. Instead of polling an etcd server for changes, you can watch for changes. This is essentially a pubsub system that requires almost no configuration to work. When a key is written to etcd, if there are any watchers for that key, a message is sent to them with the new value.

This is kind of dry in theory, so let’s look at a real-world application using this system: my photoviewer and photoserver applications. These applications allow me to display photographs on monitors that can be anywhere with an internet connection, and control each of those displays from a central server. They represent the ultimate convergence of my work as an artist and my love of programming.

Each display consists of a monitor (actually a TV, but all I want is an HDMI input) and a Raspberry Pi that runs the photoviewer application. Each display has a unique ID to identify it, and when a display starts up, it registers itself with the server. The server contains the settings for that display, such as the list of photos to display, and how often to change the displayed photo.

Photoviewer running in my kitchen

I have one such display in the kitchen of my home, and like to change the photos displayed on it from time to time. To do that, I go into my photoserver app and change the album for that display. Almost instantly the image on the display changes. How did that happen? The server is a virtual machine running in the Digital Ocean cloud, not local to the kitchen display.

The reason this works is that I’m also running an etcd server on another cloud instance. When I change any setting for a display, the photoserver app writes a new value for that display’s key. The key consists of the unique ID of the display plus the type of value being changed. For example, if I change the photos I want displayed for a display with the ID of 65febdde-3e8a-4c76-ab8f-d8a653e466c7, the server would write a value of a list of the names of those images to the key /551a441f-8aba-44b5-b70b-349af0be5b67:images.

That application uses the etcd3 library to watch my etcd server for changes to any key beginning with /<unique ID>. The watch() method is called with a callback method, and when a new key is written beginning with that display’s prefix, the value is sent to the callback.

The callback method sees that the full key ends with :images, so it passes the value (the list of image names) to the photo display method, which then retrieves the image and displays it. This happens in real time, without any polling of the server needed.

The original version of these apps used the traditional polling method, which seemed wasteful, considering that it was typically weeks between any changes being made. Switching to an etcd watch makes much more sense from a design perspective, and it greatly simplified the code.

Look for cases in your applications where a response is needed to a change in data. Using etcd as a mediator might be a good approach.

Day 29: Death to Superman

Have you ever worked with a large team on a complex project? Usually there is a mix of experience levels, and those with more experience create the application design as well as the workflow that everyone will use. They also serve as the disseminators of information, especially when a new member joins the team. They are the resources that help everyone become more productive

At least that’s how it’s supposed to work. In a company with a good development culture, knowledge is freely shared, and the goal of the senior developers is to help create new senior developers.

In other situations, there is a different dynamic: the overall knowledge of the project is in the brains of a select few developers, and they consider the intricacies of the application their domain. Often it isn’t a group, but rather a single individual in that role. They begin to act like Superman: swooping in to save the day in a manner that only they can do.

This is common in companies without a healthy developer culture. Typically there is a sole developer assigned to create an application, so of course they are the only one who knows how it works. Or the project could have started with a small team, and eventually everyone leaves the project except for one. Other teams that need that functionality need to go through this person, who is now the bottleneck, the gatekeeper. As new people are added to the team, this one developer keeps them dependent on him (yeah, it’s usually a man) by only sharing bits of knowledge only as needed, and not educating the new members. He tends to treat the other developers as inferior, and as a result, no one else feels competent to handle the work that Superman can do.

Back in the ’90s when I was a junior developer I was placed on a team that had exactly that dynamic. Someone would have a great idea, but nobody would act on it until that one lead dev signed off on it. People were even a little afraid to say that they thought it was a good idea, because if this Superman figure didn’t like it, he wouldn’t simply explain why. Instead he’d make you feel dumb for not understanding every implication your change would have.

Of course, when he wanted to change something, he just did it without involving anyone else. It wasn’t unusual to come in one day to find the part of the code you’d been working on had been changed, or sometimes even deleted. Needless to say, there was a general unhappiness on the team.

After a few months, Superman started throwing his weight around with our boss, taking the attitude “you can’t afford to lose me”. It worked for a while, but after one particularly obnoxious outburst, our boss called his bluff, and Superman quit on the spot and stormed out. Everyone on the team was both relieved that the source of tension was gone, and also afraid of how much more work this would mean for us. We were all afraid that the project would founder, and we would have to re-hire Superman, who would then be even more insufferable.

To our surprise, it wasn’t all that bad at all. Everyone started exploring the code base a bit more, now that we didn’t have Superman to supply that knowledge and make those changes. We started talking among ourselves about things we thought needed to be changed, and team members who were always quiet began to speak up more. The entire dynamic of the team changed for the better. And instead of the project falling apart without Superman to lead the way, it got better. Maybe no single person knew the entire code base like he did, but we all learned a lot more, and with people working together, got more done. We divvied up the code so that each person was responsible for learning that part well enough to be a resource to the others. Knowledge was once again being shared.

So while it’s good to have some knowledgeable people on a team to serve as guides for the newer members, it can become toxic with the wrong people and the wrong environment. If you’re on a team with such a toxic member (or members), don’t worry about what would happen if they left the project. Inevitably, the team will be better off without them. Speak with your manager if they aren’t already aware of the situation, and try to come up with a plan to spread the knowledge around better. And if you are told that they think things are fine the way they are, that’s a very strong signal that it’s time to update your resume. It’s not worth the mental toll to remain in a toxic environment.