Month: April 2019

More fun with etcd-compute

Last time I ended my work getting etcd-compute running at the point where I needed to configure the virtual networking. I’ve been busy the past few days with meetings and other work-related stuff, so it’s taken me a while to continue on this experiment. But I have some time now; let’s jump back in!

The reason I thought that I needed to set up virtual networking was that when I ran ip a on my controller node, all I had was the loopback and main ethernet interfaces. The directions for etcd-compute talked about setting up the metadata server by adding the IP address it uses to a virtual bridge: sudo ip addr add 169.254.169.254 dev virbr0. As I didn’t have such a bridge on my VM, I figured I had to add it. I tried sever guides on adding a bridge to an Ubuntu server, but each one ended up messing up the networking, making the VM unreachable. I ended up re-creating my etcd1 so many times that I gave up and figured I try without the metadata server. I started the placement and etcd servers by running docker.sh, and then just on a lark I re-ran ip a. This time it showed:

ed@etcd1:~$ ip a

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: ens3:  mtu 1500 qdisc fq_codel state UP group default qlen 1000

    link/ether fa:16:3e:90:6d:d0 brd ff:ff:ff:ff:ff:ff

    inet 9.114.111.201/24 brd 9.114.111.255 scope global ens3

       valid_lft forever preferred_lft forever

    inet6 fe80::f816:3eff:fe90:6dd0/64 scope link

       valid_lft forever preferred_lft forever

3: virbr0:  mtu 1500 qdisc noqueue state DOWN group default qlen 1000

    link/ether 52:54:00:35:c1:0d brd ff:ff:ff:ff:ff:ff

    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

       valid_lft forever preferred_lft forever

4: virbr0-nic:  mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000

    link/ether 52:54:00:35:c1:0d brd ff:ff:ff:ff:ff:ff

I’m not sure how those entries for ‘virbr0’ and ‘virbr0-nic’ got added (maybe docker added them?), but I wasn’t going to worry about that! So I ran the following commands, and they worked without a problem:

sudo ip addr add 169.254.169.254 dev virbr
sudo python md_server/mdserver/server.py mdserver.conf &

So now that the metadata server is running, time to try running ecompute on all the nodes. I use iTerm2, which has some sweet tools for splitting the terminal screen and running the same command in the different panes. I recorded a script of what happened:

I ran the command ecompute & on all the nodes to start the compute service in the background.

ed@etcd1:~/projects/etcd-compute(master)$ ecompute &
 [1] 4661
 ed@etcd1:~/projects/etcd-compute(master)$ 1556230694.3633301: PID: 4661 [None] {'uuid': '19a89e30-4bdd-49e7-b1a0-d4172bf7b289', 'placement': {'endpoint': 'http://etcd1:8080'}, 'etcd': {'host': 'etcd1'}, 'resize': False, 'bridge': 'br0'}
 1556230694.364856: PID: 4661 [19a89e30-4bdd-49e7-b1a0-d4172bf7b289] {'VCPU': 4, 'DISK_GB': 77, 'MEMORY_MB': 7976}
 1556230694.5012665: PID: 4661 [19a89e30-4bdd-49e7-b1a0-d4172bf7b289] Existing resource provider with gen 7 found with usages: VCPU: 0, MEMORY_MB: 0, DISK_GB: 0.

It’s interesting to see that because I had run this a few times earlier, etcd-compute recognized the UUID of the node, and noted that there was already an entry for that resource provider, with a generation of 7. If I were to stop that ecompute service and then re-start it, I would see the same as above, except this time the generation would be 8. That’s because when the service is killed, it changes the ‘reserved’ amount of its VCPU inventrory to the total amount, effectively preventing that node from being provisioned. That change increments the resource provider’s generation.

At about the 30-second mark, I tried to create a VM by running the command eschedule 'resources=VCPU:1,DISK_GB:1,MEMORY_MB:256' on the etcd3 node. That worked, and almost immediately you can see that it was scheduled to the etcd1 node, and the build process starts. However, there were many errors output, with the main one being error: failed to get domain ‘ff77fe58-e96a-498b-a3f5-a59030987238’. This is repeated several times, along with a bunch of network errors. So at this point I stopped the experiment.

There’s a lot I learned by going through all this, and I see many places where the etcd-compute project could be improved, starting with the documentation. I’d also like to get some less ethereal debugging output, so that when there are problems like I had spinning up a VM, they are recorded for later analysis. I’d also like to learn a lot more about the details of the networking required so that I can make sense of some of the networking errors.

The author of etcd-compute, Chris Dent, and I are hoping to have a mini-sprint on this project next week at the Open Infrastructure Summit in Denver, Colorado. If you will be there and want to join in the fun, drop me an email and I’ll let you know when we settle on a time and place.

Playing with etcd-compute

I’ve been interested in the etcd-compute project by Chris Dent. It’s sort of a lightweight virtual machine manager like OpenStack Nova, but without the complexity and cruft Nova has accumulated over the past 9 years. It takes advantage of technologies that simply didn’t exist in 2010 when Nova was created, using etcd‘s built-in notifications instead of passing large, complex objects over a message bus to make Remote Procedure Calls (RPC).

Keep in mind that Nova does a lot of things that etcd-compute can’t, so this isn’t a potential 1:1 replacement for Nova. But it does have potential as a much lighter replacement for those applications where the full power of Nova isn’t needed.

This post is designed to be obsolete within a week or so. What I’m aiming for is to record what worked for me following Chris’s instructions. Where I run into problems shows one of three things: our systems start out differently, or Chris assumed something that wasn’t in the README.md file, or my brain is not firing on all cylinders. It is my hope that this may help improve the installation instructions, and guide others who may wish to explore etcd-compute.

I don’t have a lot of hardware—ok, any hardware—at my disposal to experiment with, so I started by creating 3 Ubuntu 18.04 VMs in the internal OpenStack cloud for my team here at IBM. Yes, you can run virtualization on top of virtualization, and it’s turtles all the way down. But it does work! I named the instances etcd1, etcd2, and etcd3, with etcd1 being the controller and the others used as standard compute nodes.

There are some requirements—docker.io, virtinst, libvirt-daemon, libvirt-clients, and libguestfs-tools—that need to be installed on all the nodes, so I updated the distro packages and installed the requirements. Unfortunately, libvirtd wouldn’t start, and well, that’s kind of an important piece. So I cleaned house and tried again:

ed@etcd1:~$sudo aptitude purge libvirt-daemon
ed@etcd1:~$sudo apt install -y qemu qemu-kvm libvirt-bin  bridge-utils  virt-manager
ed@etcd1:~$ sudo systemctl enable libvirtd.service
 Synchronizing state of libvirtd.service with SysV service script with /lib/systemd/systemd-sysv-install.
 Executing: /lib/systemd/systemd-sysv-install enable libvirtd
 Created symlink /etc/systemd/system/libvirt-bin.service → /lib/systemd/system/libvirtd.service.
 Created symlink /etc/systemd/system/sockets.target.wants/virtlockd.socket → /lib/systemd/system/virtlockd.socket.
 Created symlink /etc/systemd/system/sockets.target.wants/virtlogd.socket → /lib/systemd/system/virtlogd.socket.
 ed@etcd1:~$ sudo systemctl start libvirtd.service
 ed@etcd1:~$ sudo systemctl status libvirtd.service
 ● libvirtd.service - Virtualization daemon
    Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
    Active: active (running) since Tue 2019-04-23 23:29:30 UTC; 5s ago
      Docs: man:libvirtd(8)
            https://libvirt.org
  Main PID: 5289 (libvirtd)
     Tasks: 19 (limit: 32768)
    CGroup: /system.slice/libvirtd.service
            ├─4486 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_
            ├─4487 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_
            └─5289 /usr/sbin/libvirtd
 Apr 23 23:29:30 egleafe-etcdcompute-1 systemd[1]: Starting Virtualization daemon…
 Apr 23 23:29:30 egleafe-etcdcompute-1 systemd[1]: Started Virtualization daemon.
 Apr 23 23:29:30 egleafe-etcdcompute-1 dnsmasq[4486]: read /etc/hosts - 10 addresses
 Apr 23 23:29:30 egleafe-etcdcompute-1 dnsmasq[4486]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
 Apr 23 23:29:30 egleafe-etcdcompute-1 dnsmasq-dhcp[4486]: read /var/lib/libvirt/dnsmasq/default.hostsfile
 lines 1-17/17 (END)

So I guess it’s working now!

It’s also a pain to always have to use sudo to run docker commands, so add your user to the docker group. The command for this is sudo usermod -a -G docker ed, which adds user ‘ed’ to the group ‘docker’. You have to log out and log back in for it to take effect, but once you do, you can run commands like docker ps -a without sudo.

Also, in my experience I’ve run into various odd problems using the distro version of Python, so I prefer to install from source to get the latest Python (3.7.3 right now).

Being a creature of habit, I like having the project code I’m working with to be under a ~/projects directory. So for each of these instances, I ran the following:

ed@etcd1:~$ mkdir projects

ed@etcd1:~$ cd projects/

ed@etcd1:~/projects$ git clone https://github.com/cdent/etcd-compute.git

Cloning into 'etcd-compute'…

remote: Enumerating objects: 169, done.

remote: Counting objects: 100% (169/169), done.

remote: Compressing objects: 100% (107/107), done.

remote: Total 205 (delta 97), reused 124 (delta 61), pack-reused 36

Receiving objects: 100% (205/205), 54.58 KiB | 119.00 KiB/s, done.

Resolving deltas: 100% (107/107), done.

ed@etcd1:~/projects$ cd etcd-compute/

ed@etcd1:~/projects/etcd-compute(master)$

As the etcd-compute code has its own dependencies, those need to be installed by running sudo python setup.py develop. When I ran that the first time, I got an error when it was trying to install libvirt-python. I tried installing some other libvirt-related libraries and binaries, but I kept getting the same error. After a while I was trying anything I could think of, even running under Python 2! (didn’t work). Maybe it was something about Python 3.7 that was problematic, so I created a venv for Python 3.6, and ran pip install libvirt-python. It installed without a problem. Hmmm. So I fired up a Python 3.7 venv, and it also installed into that. It seems that the installation using setup.py was doing something different than a straight pip install. To test that, I got rid of the venvs, and ran sudo pip install libvirt-python, and it worked just fine. I was then able to install the rest of the dependencies by running sudo python setup.py develop.

Now that the dependencies are installed, we need to create the database for placement, and then modify the dockerenv file so that the OS_PLACEMENT_DATABASE__CONNECTION setting points to that. My database is on a MariaDB server, so I needed to change the value to:

mysql+pymysql://user:secret@my_host/placement

That means, of course, that I need to install pymysql using sudo pip install pymysql before I can make a connection. Once that’s done, I started the docker containers by running ./docker.sh from the primary VM. In my case, that’s etcd1.

That brings up another edit that’s needed on all your “machines”: changing the location of the host in compute.yaml and schedule.yaml. These assume that the host is named ‘ds1’, which isn’t true in this case. I changed ‘ds1’ to ‘etcd1’, and then added an entry in each node’s /etc/hosts file with the IP address of the etcd1 VM.

We also want to create a value for the uuid in compute.yaml. One simple way is to run python -c "import uuid; print(uuid.uuid4())", and copy the output to paste into compute.yaml. Do that on every compute node you are running.

That’s enough for one day. Tomorrow we start with configuring networking!

Withdrawal, Part 2

A few weeks ago I wrote about my first experience attempting to stop taking the painkiller Tramadol that I had been using in my recovery from a total knee replacement. I had been assured by my health professionals that it was not habit forming, but I found that, at least for me and many others, that was not the case.

Before that attempt I was taking 4–50mg tablets a day. I called my surgeon’s office to ask for some advice, and spoke to one of the nurses there. When she heard my question, she replied dismissively along the lines of “oh, don’t worry – you can just stop taking it whenever you don’t need it anymore”. I told her that that was certainly not true, and recounted what I had gone through when I tried to do just that. She was shocked, and said she had never heard of anyone having withdrawal symptoms like that. I mentioned that a quick Google search would show that I was not alone in this regard. But we did discuss a plan to slowly wean myself off of it, and that’s what I began doing. Six-hour intervals between doses became 12 for a few days, then 16 for the next few. After that I started cutting the tablets in half, and taking half a pill every 12 hours, and after a few days increased that to 16 hours. Note that all this time I was continuing to do my physical therapy exercises and walk 30+ minutes a day, without any pain at all. Last Sunday I was out of the house when my next dose was due, and didn’t get back until 4 hours after that. I figured I was at such a low dose that I could now stop taking it without going through as horrible an experience as I had earlier.

Sunday evening I did have some difficulty falling asleep, as my muscles weren’t able to easily relax, but eventually I did get some sleep. Not much, but enough to function on Monday. When I got up I felt fine: no pain, and none of the runny nose, coughing, or sneezing that I had had last time. But by the time night came, I could definitely feel my muscles getting tense, and I knew I would have trouble laying still. It was now about 40 hours since my last dose. I knew that this night would be tough, so I made a plan. I was going to stay up as late as I could, so that I would collapse from exhaustion and fall right asleep! I even took a couple of sleep aid pills to ensure that I would go out like a light!

Well, it was a good plan, but it didn’t work. The sleep aids did their thing, and I got really drowsy. I knew if I tried laying down in our bed that I would end up keeping Linda awake, too, so I laid down in the guest room and tried to rest there. Every time I tried to get in a comfortable position, though, my muscles would get tense and my joints would hurt, and I’d have to move. It felt as though I was twitching uncontrollably. I thought about walking it off, but I felt too drowsy to even sit up! It continued to get worse, tossing and turning repeatedly, never staying in one position for more than 5 seconds or so. At a few points in the night it got so bad that I had to sit up and flail my arms around wildly to get them to feel OK. I would start to doubt the wisdom of continuing – maybe I should take another pill and wean myself even further before stopping. But I figured that I had come this far, and I really wanted to see it through. Besides, I didn’t have any critical meetings or anything at work the next day, so I could take it off to recover.

I would look at the clock every now and then, and remember the time passing midnight, then 1:45am, then 3, then 4:30am, then…

The alarm went off! I had set it for 6am because my wife had to get up early for work. But that meant that I had finally fallen asleep! I got up, went to our bedroom, woke her, gave her a quick summary of the night’s events, and then crawled back into bed. I slept until almost 11am, and though my body felt a whole lot better than it had before, my brain was still pretty fuzzy. I wasn’t able to focus on any task for very long. I’m not sure if it was from sleep deprivation or another symptom of withdrawal. I tried reading Twitter but had to put the phone down after a minute or so. Same thing when I tried to play some games. I started writing this post that afternoon, but after about 2 sentences I gave up. So I watched some TV, laid in bed, walked around our garden, watched some more TV, and generally was a blob all day. When Linda came home that evening, she saw the state I was in, and knew what I needed: a big bowl of homemade ramen, with lots of veggies and hot chilis! It actually seemed to wake my brain up, and I was once again able to focus on things.

Later that evening I was walking to the kitchen, and I noticed that my surgical knee felt… different. It was sore! When I bent it, I could feel the muscles and tendons and everything pressing together painfully in the still-swollen interior of the knee. It had now been 8 weeks since the surgery, and I was told that although the swelling would diminish in the first few weeks, that it wouldn’t be gone for about 6 months! So it appears that the Tramadol was helping to control that low-level pain, even if it wasn’t much help when, say, the physical therapists would force my knee to bend further than it wanted to – that was still extremely painful! But now that this low-level pain is present, I kind of like having it, as it reminds me that I need to keep working on getting it stronger and more flexible.

I’m writing this two days after that difficult night, and the only odd symptoms I’ve had are some strange visual effects. They started later in the evening both days, and it looks like there is a curved band of kaleidoscopic distortion in my right eye. It goes away when I close my eyes, so I’m not sure what it could be. If it keeps happening I’ll get it checked out, but for now, it’s a minor annoyance.

I now have a much greater appreciation of what addicts go through when they try to get clean. My addiction was tiny compared to most, and yet it left me feeling horribly uncomfortable and unable to lay still. It’s a small price to pay for the privilege of having a new knee, but it’s one that I didn’t expect. It’s great to finally be done with the painkillers! I really needed them to make the progress I’ve made, but I didn’t want to be addicted to them any longer than absolutely necessary.

Running Pi-hole

A few days ago I read a tweet from someone who recounted their experience with Pi-hole, which brands itself as “A black hole for Internet advertisements”. My curiosity being piqued, I read up some more, and I liked what I read.

I’ve run ad blocking extensions in my browsers for years, and it’s made using the web bearable. But if you think about it, what those blockers are doing is simply preventing those ads from displaying; they are still fetched from the internet, using up bandwidth, which slows down your browsing.

Pi-hole takes a different approach, acting as the DNS server for your network. When a request is received, it compares it to a curated list of domains known to serve advertisements and track users, and if it is in that list, instead of forwarding the request to that domain, it forwards it to its own built-in web server, and returns a blank page. In other words, no internet traffic at all is generated!

I had an extra Raspberry Pi on hand, so I figured that I’d play around with it this past weekend. I installed the latest Raspbian operating system on it, and followed the simple instructions to git clone the Pi-hole source code and install it. There are several other pages I found detailing the steps, but as it’s pretty straightforward I didn’t feel that I had anything to add by creating yet another step-by-step guide.

Note that you don’t have to use a Raspberry Pi to run Pi-hole; you could run it on just about any Linux system. There is even a Docker image you can pull and run, so you have plenty of options if you don’t have any Raspberry Pis on hand. If you do want to pick up a Pi or three, I recommend PiShop.us. They have the raw components as well as full kits for the Pi, and also lots of other maker-oriented products.

I use the Google Wifi mesh system in my house, so I was a little concerned that getting it to work with the Pi-hole DNS might be tricky, as the Google Wifi system is rather limited in your ability to customize it. I configured the Raspberry Pi to only use the wired ethernet connection, and plugged that into the output jack of the Google Wifi unit. I set the Google Wifi app to give that Raspberry Pi a static IP address. So far, so good. I first tested it by changing my laptop’s DNS to point to the Pi-hole’s address, and loaded a few web pages. The Pi-hole comes with an admin web server that allows you to not only configure things, but also see real-time stats of the traffic that it’s handling.

Pi-hole Admin Dashboard
Pi-hole Admin Dashboard

Once I saw that it was indeed working and working well, I opened the Google Wifi app to the Settings tab, and then opened the Network and General button. That gives you a page with several options; the one you need is Advanced Networking. The top option on that screen is DNS; open that and change it from the default of Automatic to Custom. Set the Primary server to the IP address of the Raspberry Pi running Pi-hole, and the secondary address to some other server (I used Google’s default of 8.8.8.8). Save that, and you’re in business! (see photo below) Now every device on your local network will experience faster, cleaner internet browsing free of ads!

Google Wifi settings
Configuring the Google Wifi app to use the Pi-hole for DNS. Replace the blacked-out address with your device’s IP address.

If you don’t have a central device acting as a router that you can configure, you’ll have to change each device’s settings to use the Pi-hole for its DNS. A little more work, but once it’s done, it’s done.

As an added bonus, once you’re using Pi-hole, you can disable your browser’s ad blocking software, as it is pretty much redundant. Since I’ve disabled those extensions, I haven’t gotten any of those annoying “Please turn off your ad blocker” popups when I go to some sites. I’m sure that there are some techniques that might flag Pi-hole, but so far I haven’t hit any.

I’ve only been using it for a few days now, but the results were instantly noticeable. The percentage of blocked requests varies slightly, but has typically been around 12%. The Pi-hole project is entirely user-supported, and I was more than happy to donate even after using it for a short time.

Withdrawal

So this week was the big post-op appointment with my surgeon following my total knee replacement. Six weeks: that was the first big milestone, where my progress would be assessed to make sure that I was recovering well. It was also the point where some of the more annoying things that I had to do would end (assuming I had progressed well): no more compression stocking; no more sleeping only on my back with my foot in a foam block to stretch the knee, and no more walker. Full disclosure: I hadn’t used the walker at all since about 2 weeks after the surgery, but the doctors insisted on it as a safety measure, so… 😉

I wasn’t too concerned about the exam, as I had been doing the physical therapy religiously all this time, and had a gotten a good deal of flexibility and strength back. And that’s exactly how the exam went: I was given a proverbial gold star, moving me ahead to the next phase. Joy!

One thing that has always concerned me was my use of pain medication. After surgery they start you on Hydrocodone, a strong opiod painkiller, and boy, did I need it! But as I healed the severe pain diminished, and I really only needed it when doing some of the more painful PT exercises. So two weeks later they switched me to Tramadol, a non-opiod pain medication. Since it’s not an opiod, it is supposedly not as addictive, nor as difficult to stop using once you no longer need it.

So, given my progress on the healing front, and my missing being able to share a glass of wine with my wife at dinner, I decided to stop taking the medication. Sure, it would be uncomfortable for a day or two, but after that I’ll be clean!

So my last dose was this past Tuesday at 11am. Wednesday morning I awoke with what appeared to be a cold: runny nose, congestion, watery eyes, sneezing, and a dry, hacking cough. Oh, great – now I have to deal with a cold on top of everything else!

Towards the end of the afternoon, I started to feel lethargic. OK, I thought, this is the drug withdrawal, and I have to tough it out. I was also feeling chilled, but I didn’t know if that was the cold or the withdrawal causing that. My wife took good care of me, and made a delicious ramen to warm me up, with lots of chili to clear my sinuses, and lots of kale and other veggies to keep me strong.

I went to bed, trying to relax enough to fall asleep, but I just couldn’t get comfortable. I had taken some Tylenol to help ease any knee pain, but my knee wasn’t really bothering me that much. No matter how I tried to position myself, I could feel the muscles in my legs twitching, making it impossible to lay still. I didn’t want my constant movement to keep my wife awake, so I got up. Maybe walking around a bit would calm my legs down, so I walked around the house for a while. That helped a little, but I still didn’t feel like I could rest, so I sat down at the computer, and figured I’d see if there were any tips I could find on dealing with Tramadol withdrawal.

Well, it wasn’t hard to find that information! And man was there a ton of it! The first page I read was a message board where different people described their experiences, and it was like a slap in my face! They described everything I had been feeling; even my “cold” wasn’t a cold, but just some of the symptoms of withdrawal!

I continued reading a variety of sites, and it reinforced all of the anecdotal reports of the first site I read. What was worse, though, is the timeline they described for these symptoms: they would peak after 72 hours without the medication, and continue for 10 days or so! I was barely 36 hours into my withdrawal, and I wasn’t going to be able to handle this without locking myself in a padded room for the duration. Every single site recommended the only way to stop is to gradually wean yourself by slowly reducing your dosage, so that’s what I decided to do. I felt defeated: I had been kicking ass on this whole knee-replacement thing since the beginning, and now I finally ran into something that kicked mine. So I reluctantly took a Tramadol pill at 11:30pm, and an hour or so later I was feeling better enough to attempt sleeping. I did manage to get a few hours, so this morning I’m a little sore and grouchy. My (revised) plan is to go with taking it at 12 hour intervals for a few days, then increase it to 16 hour intervals for another few days, and adjust that as needed until I’m off it completely. No heroic cold turkey for me anymore!

Let me close with a current photo of how my knee is progressing. The swelling continues to diminish, but it’s still there. The scar is looking fine, thanks to the Vitamin E oil I rub in it daily.

6 week knee photo
6 weeks after surgery