Technical – Page 2 – Walking Contradiction

Day 31: Using etcd As a Mediator

etcd is a database originally developed by CoreOS, and is most famously used as the database at the heart of Kubernetes. It is a distributed key-value store, which in itself is not all that remarkable. The thing about etcd that makes it so attractive is the ability to watch a key for changes.

Other key/value stores, such as Redis, have implemented a similar feature, and may work just as well for you. I’ve been using etcd for years, and it’s worked well for me, so I’ve never had a reason to try these other tools.

For most data stores, the only way to find if a particular value has changed is to poll. You issue a query for that value on a regular basis, and compare it to the last value returned to see if it has changed. This is terribly inefficient, especially with values that don’t change often. It’s also inexact with respect time, because your system’s reaction to a changed value depends on the interval between polls. Longer intervals, while less chatty, mean that more time will elapse between when the value changes, and when your application responds to that change.

Enter etcd. Instead of polling an etcd server for changes, you can watch for changes. This is essentially a pubsub system that requires almost no configuration to work. When a key is written to etcd, if there are any watchers for that key, a message is sent to them with the new value.

This is kind of dry in theory, so let’s look at a real-world application using this system: my photoviewer and photoserver applications. These applications allow me to display photographs on monitors that can be anywhere with an internet connection, and control each of those displays from a central server. They represent the ultimate convergence of my work as an artist and my love of programming.

Each display consists of a monitor (actually a TV, but all I want is an HDMI input) and a Raspberry Pi that runs the photoviewer application. Each display has a unique ID to identify it, and when a display starts up, it registers itself with the server. The server contains the settings for that display, such as the list of photos to display, and how often to change the displayed photo.

I have one such display in the kitchen of my home, and like to change the photos displayed on it from time to time. To do that, I go into my photoserver app and change the album for that display. Almost instantly the image on the display changes. How did that happen? The server is a virtual machine running in the Digital Ocean cloud, not local to the kitchen display.

The reason this works is that I’m also running an etcd server on another cloud instance. When I change any setting for a display, the photoserver app writes a new value for that display’s key. The key consists of the unique ID of the display plus the type of value being changed. For example, if I change the photos I want displayed for a display with the ID of 65febdde-3e8a-4c76-ab8f-d8a653e466c7, the server would write a value of a list of the names of those images to the key /551a441f-8aba-44b5-b70b-349af0be5b67:images.

That application uses the etcd3 library to watch my etcd server for changes to any key beginning with /<unique ID>. The watch() method is called with a callback method, and when a new key is written beginning with that display’s prefix, the value is sent to the callback.

The callback method sees that the full key ends with :images, so it passes the value (the list of image names) to the photo display method, which then retrieves the image and displays it. This happens in real time, without any polling of the server needed.

The original version of these apps used the traditional polling method, which seemed wasteful, considering that it was typically weeks between any changes being made. Switching to an etcd watch makes much more sense from a design perspective, and it greatly simplified the code.

Look for cases in your applications where a response is needed to a change in data. Using etcd as a mediator might be a good approach.

Day 29: Death to Superman

Have you ever worked with a large team on a complex project? Usually there is a mix of experience levels, and those with more experience create the application design as well as the workflow that everyone will use. They also serve as the disseminators of information, especially when a new member joins the team. They are the resources that help everyone become more productive

At least that’s how it’s supposed to work. In a company with a good development culture, knowledge is freely shared, and the goal of the senior developers is to help create new senior developers.

In other situations, there is a different dynamic: the overall knowledge of the project is in the brains of a select few developers, and they consider the intricacies of the application their domain. Often it isn’t a group, but rather a single individual in that role. They begin to act like Superman: swooping in to save the day in a manner that only they can do.

This is common in companies without a healthy developer culture. Typically there is a sole developer assigned to create an application, so of course they are the only one who knows how it works. Or the project could have started with a small team, and eventually everyone leaves the project except for one. Other teams that need that functionality need to go through this person, who is now the bottleneck, the gatekeeper. As new people are added to the team, this one developer keeps them dependent on him (yeah, it’s usually a man) by only sharing bits of knowledge only as needed, and not educating the new members. He tends to treat the other developers as inferior, and as a result, no one else feels competent to handle the work that Superman can do.

Back in the ’90s when I was a junior developer I was placed on a team that had exactly that dynamic. Someone would have a great idea, but nobody would act on it until that one lead dev signed off on it. People were even a little afraid to say that they thought it was a good idea, because if this Superman figure didn’t like it, he wouldn’t simply explain why. Instead he’d make you feel dumb for not understanding every implication your change would have.

Of course, when he wanted to change something, he just did it without involving anyone else. It wasn’t unusual to come in one day to find the part of the code you’d been working on had been changed, or sometimes even deleted. Needless to say, there was a general unhappiness on the team.

After a few months, Superman started throwing his weight around with our boss, taking the attitude “you can’t afford to lose me”. It worked for a while, but after one particularly obnoxious outburst, our boss called his bluff, and Superman quit on the spot and stormed out. Everyone on the team was both relieved that the source of tension was gone, and also afraid of how much more work this would mean for us. We were all afraid that the project would founder, and we would have to re-hire Superman, who would then be even more insufferable.

To our surprise, it wasn’t all that bad at all. Everyone started exploring the code base a bit more, now that we didn’t have Superman to supply that knowledge and make those changes. We started talking among ourselves about things we thought needed to be changed, and team members who were always quiet began to speak up more. The entire dynamic of the team changed for the better. And instead of the project falling apart without Superman to lead the way, it got better. Maybe no single person knew the entire code base like he did, but we all learned a lot more, and with people working together, got more done. We divvied up the code so that each person was responsible for learning that part well enough to be a resource to the others. Knowledge was once again being shared.

So while it’s good to have some knowledgeable people on a team to serve as guides for the newer members, it can become toxic with the wrong people and the wrong environment. If you’re on a team with such a toxic member (or members), don’t worry about what would happen if they left the project. Inevitably, the team will be better off without them. Speak with your manager if they aren’t already aware of the situation, and try to come up with a plan to spread the knowledge around better. And if you are told that they think things are fine the way they are, that’s a very strong signal that it’s time to update your resume. It’s not worth the mental toll to remain in a toxic environment.

Day 13: Loss

I’ve been raising caterpillars for about 6 years now. I say “caterpillars” instead of “butterflies”, because once they emerge as an adult butterfly, I let them go, and my involvement ends.

The species I get to raise is the Eastern Black Swallowtail. We grow parsley in our garden, and parsley is one of the favorite host plants where swallowtails like to lay their eggs (along with rue, dill, carrots, fennel, and a few others). I found these cute caterpillars munching away on the parsley one day.

We counted an even dozen of them. As the days passed, though, their numbers steadily decreased. Only 2 of them made it to the pupa stage, and one of those pupas got eaten by a wasp. Finally, though, one of the most gorgeous creatures emerged! She was content to rest on my hand while her wings dried out, and after a while flew off.

After that I decided to try to bring the caterpillars indoors to keep them from predators. I kept them supplied with food, and they grew well. A much higher percentage made it to adulthood, but there were some losses along the way. I had such a loss this morning.

I feed them with sprigs of whatever plant their eggs were placed on. To keep the sprigs fresh, I put them in a vase-like bottle. The problem is that sometimes the caterpillars aren’t too careful, and fall into the water and drown. To prevent that, I covered the opening with foil, and placed the stems through a small hole I poked in it.

What I didn’t take into account was that as the caterpillars move around and jostle the sprigs, the foil would tear a little, and in time the little hole I poked in the foil got bigger. The now-stripped stems then fell down into the water, and one of the caterpillars must have been holding onto one of those, and was carried into the water, too. When I got up this morning, I found him submersed.

By now I’ve raised around 100 caterpillars, and have had my share of them die. Some only get so big, and then suddenly die. Some have crawled out of the container I had them in, and if I didn’t find them in time, they would starve to death.

So I’m not sure why this one bothered me so much. I do feel responsible, in that it was my bad design that led to it drowning.

Sorry, little guy – I’ll make sure that I learn from this, and make things better for future caterpillars that come my way.

Day 11: Spreading Racism

Mass protests in a time of the COVID-19 pandemic: are they foolish gatherings that will only result in spreading the disease?

In an editorial in today’s NY Daily News, the authors make the case that even though protests in response to the execution of George Floyd, Ahmaud Arbery, Breonna Taylor, and far too many others, are important because “racism is a deadly virus, too”.

I agree with the sentiment: racism is deadly, and we cannot just sit back and ignore it. But it is the analogy to a virus that really bothers me.

Viruses spread without intent. They do not choose their hosts. They are opportunistic: if a virus lands on a cell that it can infect, it does so. If it doesn’t, it dies. It does not have the capacity to choose.

Racism, however, is not randomly spread. Instead, it is actively taught to each generation. Children listen to their parents, and largely follow the values that they are taught.

When my kids were young and in day care, one of the other kids there called another kid a derogatory racial name. The woman who ran the day care handled it calmly, explaining to the boy why he shouldn’t call people those names. I talked with her about it afterwards, and she said that her grandmother had a phrase she used when she heard kids say ignorant things: “Well, he didn’t just lick that up off the floor!”

In other words, it wasn’t an accident; it wasn’t randomly spread. Instead, someone spoon-fed that to the child.

The only way to combat racism is education. For years most white people thought that the reports of police misconduct and brutality towards blacks was probably just “a few bad apples”, but that most were fair and respectful. Since the advent of ubiquitous video recording, though, those same white people are getting educated about the reality that POC have known for far too long.

And I hope that children growing up today see these protests, with people of all colors coming together to demand that things improve, and take away from this at the very least the concept that Black Lives Matter. They will be better equipped to deal with a racist relative, and refuse to be spoon-fed that bigotry.

Day 10: Missed Opportunity

It’s been a little over 2 months since I became one of the 40 million people in the US who lost their job as a result of the COVID-19 pandemic. In that time I’ve been reaching out to people in my network, tracking LinkedIn regularly, and keeping an eye out for a new opportunity.

I’ve applied to over a dozen companies, and out of all of those, only one even bothered to write back that they didn’t think I was a good fit for them. For the rest, it was as if my application had been sucked into a black hole.

Until yesterday. I got an email from a well-know tech company saying that they were impressed by my experience, and so we arranged for a video interview with the hiring manager. Finally, it seemed, the world of employment wasn’t looking so bleak!

For the record, it is not my intention to embarrass this company, so I shall not name them. They seemed genuinely interested, and the people I dealt with were both professional and pleasant.

The interview yesterday went well. I was impressed with the hiring manager, who seemed very sharp. I got the impression he was likewise impressed with me, since he told me he would refer me forward to the next phase of the interview process.

A couple of hours later, though, I got a call from someone in HR at this company. The hiring manager had mentioned to him that I was looking for remote work, which I always state clearly up front. It turns out that even though their entire company is working remotely now due to the pandemic, once that’s over they expect everyone to work from one of their offices. In other words, though working remotely has kept the company running, they will not hire anyone who isn’t located where their offices are. As I am not in a position to relocate now (and besides, I love San Antonio!), I politely declined to continue the hiring process. The HR person mentioned that there is talk of opening up the company to hiring remote workers, so I told them that if that ever happens and I’m still available, I would be glad to help them transition to a remote-friendly culture, as I do have a bit of experience with it.

Before the pandemic, it had been very difficult for me to understand why so many tech companies resisted remote work. I suppose its the old “if I can’t see you, how do I know that you’re working?” attitude. But now that they have been forced to do it by circumstances, you’d think that they’d realize that there is no reason not to embrace it, and many reasons to do so:

You now have access to a much wider pool of talent
Relocation expenses are eliminated for most hires
The amount of office space you need to run your business is kept low
Processes are documented better
Workers are generally happier

One of the supposed advantages of working in an office is the spontaneous conversations that happen – the proverbial “water cooler” discussions. Sure, these can be helpful, but all too often the fruit of those discussions is never recorded. When you work with a distributed team, it forces you to document these things, usually in email or a Google doc. Such documentation is helpful for preventing misunderstandings down the road.

Unfortunately, it seems that many companies are cutting edge of tech, but very slow learners when it comes to hiring remote workers. So I’m back to looking for openings and filling out applications. And that company is back to looking for the talent they need to grow.