Category: Cloud

APIs are the lifeblood of any technical system, and a stable, dependable API is absolutely essential for anyone using that system.

Last week there was a discussion in the OpenStack Technical Committee weekly meeting about adding the Monasca project, a new approach to telemetry and monitoring, to the “Big Tent”. There were several factors discussed, both positive and negative, but one stood out: the concern about the differences between the API used by Monasca, and that of the existing telemetry project, Ceilometer. For a little background, Ceilometer has been around for several years, and while it has enjoyed some success, there is a good deal of unhappiness with its current state, and there doesn’t seem to be a focused effort to address that (please, no hate mail from Ceilometer devs – just reporting what I hear!). Hence the appeal of a new project like Monasca.

The concern of several people was that Monasca doesn’t adhere exactly to the same API as Ceilometer, and that this would cause pain for existing Ceilometer users. Some saw this as a major flaw, and one that they thought would prevent Monasca from being part of OpenStack. Others, though, thought that the API is driven by the implementation, and it necessarily would differ in a different project, and that this sort of differentiation is one of the things to be expected by the Big Tent approach.

The reason for this disagreement comes from one point: that the Ceilometer API, having been created first, is now considered by some to be the OpenStack Telemetry API by default. However, the TC has consistently said that they are not and do not want to be a “standards body” for APIs, and I agree with that. But it does pose an issue: does that mean that we are “stuck” with the existing APIs, simply because they already exist? Are we going to reject all new projects that solve a problem in better and efficient ways because those new ways don’t fit into an old project’s paradigm? Note: I am not claiming that Monasca (or any other project) is better or more efficient, as I have no practical experience with it. I’m speaking in more general terms.

There is something to be said for the effects of inertia: if you have already adopted an API of a product, and you are unhappy with that product, you might still resist switching to something better if it requires you to make a lot of changes to the code that interacts with that product. You would give some serious thought to the pain of switching, balancing that against the anticipated benefits once the switch is made. To Monasca’s credit, they handled this with a Ceilometer compatibility layer to make switching easier, acknowledging the dragging effect of inertia on adoption. In my opinion, this is exactly how competition is supposed to work.

So will having a new project that is incompatible with an existing project cause pain for OpenStack users? Of course – no one wants to have to deal with incompatibility. But so will insisting that every new project exactly follow the design of its predecessors in that space.

It would be wonderful if we could all agree ahead of time on what the API for a particular service should be, and then send teams of developers off to create competing implementations of that service, each adhering to the One True API. But that simply isn’t reality. It was stated in the discussion that this would mean that there would now be two OpenStack Telemetry APIs, but I see it differently: there are exactly zero OpenStack telemetry APIs. There is a Ceilometer API, and there is a Monasca API, and there might be some other solution in the future that has yet another API. But none of those are the OpenStack telemetry API, since such a beast doesn’t exist.

The notion of having a body, whether the TC or any other, take on the role of defining an API and enforcing strict adherence to that API definition, will undoubtedly lead to much worse problems than we have now, both technical and political. It is much more preferable to allow new solutions to come up with their own approaches, and adding compatibility shims as needed. In the long run this will allow for a much healthier ecosystem where competition can thrive.

After several days of intense discussions at the Vancouver OpenStack Summit, it’s clear to me that we have a giant pile of technical debt in the scheduler, based on the way we think about resources in a cloud environment. This needs to change.

In the beginning there were numerous compute resources that were managed by Nova. Theoretically, they could be divided up in any way you wanted, but some combinations really didn’t make sense. For example, a single server with 4 CPUs, 32GB of RAM, and 1TB of disk could be sold as several virtual servers, but if the first one requested asked for 1CPU, 32GB RAM and 10GB disk, the rest of the CPUs and disk would be useless. So for that reason, the concept of flavors was born: particular combinations of RAM, CPU and disk that would be the only allowable way to size your VM; this would allow resources to be allocated in ways that would minimize waste. It was also convenient for billing usages, as public cloud providers could charge a set amount per flavor, rather than creating a confusing matrix of prices. In fact, the flavor concept was brought over from Rackspace’s initial public cloud, based on the Slicehost codebase, which used flavors this way. Things were simple, and flavors worked.

Well, at least for a while, but then the notion of “cloud” continue to grow, and the resources to be allocated become more complex than the original notion of “partial slices of a whole thing”, with new things to specify, such as SSD disks, NUMA topologies and PCI devices. These really had nothing to do with the original concept of flavors, but since they were the closest thing to saying “I want a VM that looks like this”, these extra items were grafted onto flavors, as ‘flavor’ became a synonym for “all the stuff I want in my VM”. These additional things didn’t fit into the original idea of a flavor, and instead of recognizing that they are fundamentally different, the data model was updated to add things called ‘extra_specs’. This is wrong on so many levels: they aren’t “extra”; they are as basic to the request as anything else. These extra specs were originally freeform key-value pairs, and you could stuff pretty much anything in there. Now we have begun the process of cleaning this up, and it hasn’t been very pretty.

With the advent of Ironic, though, it’s clear that we need to take a step back and think this through. You can’t allocate parts of a resource in Ironic, because each resource is a single non-virtualized machine. We’ve already broken the original design of one host == one compute node by treating Ironic resources as individual compute nodes, each with a flavor that represents the resources of that machine. Calling the Ironic machine sizes “flavors” just adds to the error.

We need to re-think just what it means to say we have a resource. We have to stop trying to treat all resources as if they can be made to follow the original notion of a divisible pool of stuff, and start to recognize that only some resources follow that pattern, while others are discreet. Discreet resources cannot be divided, and for them, the “flavor” notion simply does not apply. We need to stop trying to cram everything into flavor, and instead treat the request as what we need to persist, with ‘flavor’ being just one possible component of the request. The spec to create a request object is a step in the right direction, but doesn’t do enough to shed this notion of requests only being for divisible compute resources.

Making these changes now would make it a lot easier in the long run to turn the current nova scheduler into a service that can allocate all sorts of resources, and not just divide up compute nodes. I would like to see the notion of resources, requests, and claims all completely revamped during the Liberty cycle, with the changes being completed in M. This will go a long way to making the scheduler cleaner by reducing the technical debt by assumption that we’ve built up in the last 5 years.