Obtaining External Connectivity in OpenStack

So you’ve pulled the trigger and obtained some computing resource on your cloud platform of choice. It’s super easy to get your frontend web servers to talk to your database servers: they’re all in the same cloud, so you just point them at each other’s IPs and they happily start chatting away. Now that your infrastructure is in place, you want to reveal your new project to the world. How do you get external traffic to your servers? This post will discuss doing just that, from the perspective of OpenStack. If you’re using another platform (something based on Linux Containers, for example), the general approach is the same, though some of the specifics may differ.

The OpenStack answer has traditionally been to use ‘floating IPs’. However, floating IPs require a fair bit of configuration inside OpenStack (in the form of network nodes to perform NAT) and by the user, and it’s not always easy to get that right.

However, floating IPs have a deeper problem: they conflate two separate notions. The first is ‘accessibility’: should this machine be reachable from outside the data center? The second doesn’t have a short-and-sweet name, but we call it ‘IP address mobility’: the ability to move an IP address between VMs. This second property is used to make life easier for people using the cloud: you can set up your DNS to point to a single IP address that will never change, and simply move around the machine to which that address points.

In Calico, we take advantage of our pure-L3 model to fundamentally question the floating IP approach by splitting out accessibility and address mobility. For inbound accessibility, we can throw NAT and network nodes out the window. If you want a VM to be reachable from outside the data center, you simply give it an IP address that is reachable from outside the data center. Because we have a completely routed infrastructure this doesn’t prevent the machine communicating directly with other VMs in the data center, but it enables machines outside the data center to contact it as well. Because there’s no NAT all traffic takes the most efficient path to the VM, without passing through other boxes on the way.

For all the other VMs in your deployment, the ones with the private IPs, there is a slightly different requirement. For them, it’s necessary to perform some kind of NAT: their addresses are not going to be routable from the wider network or the internet at large. In this case, the NAT gets performed at the gateway as the traffic leaves the data center. This model is exactly the same as the one used in all large private networks for the last decade, and it’s far and away the simplest. The gateway maintains the stateful NAT and runs BGP so as to know how to route inbound to datacenter VMs: every other box in the network ignores its existence. Simple.

So, how do we handle mobility? We have a few ideas, none of which are set in stone yet, but the most attractive one so far is to use IP Anycast. This is a technique that can only really be applied in pure L3 fabrics. I’m not going to go into too much detail in this post, but the basic principle is that multiple machines are assigned the same IP, and the routing fabric balances load across them. Think of it as something like Amazon’s Elastic Load Balancer, but handled entirely transparently by the Calico fabric. No need for bump-in-the-wire middleboxes, just standard IP routing.

So, if you’re deploying OpenStack with Calico providing its networking, here are the things to remember:

1. Don’t configure floating IPs or routers! They don’t do anything.
2. Instead, create a ‘public’ network with a subnet that your gateway is advertising routes for.
3. When a VM wants a public address, just give it an interface on the ‘public’ network.
4. Make sure your gateway is BGP peered with your Calico nodes.
5. Consult our ‘External Connectivity in OpenStack’ document for more detail.