Advertising Kubernetes Service IPs with Calico and BGP

Two of the most common ways of accessing services hosted within a Kubernetes cluster are through an Ingress resource or Service type Load Balancer. For users of public clouds, these are simple and effective ways to give access to services. Cloud-hosted controllers can do the heavy lifting of assigning public IPs, setting up load balancing, and managing SSL termination.

Operators running privately-hosted Kubernetes clusters on-premises will quickly realize that making services publicly available is more complicated than on public clouds. Will services be on the public internet, or only public to local users? How should ingress, load balancing, IP-assignment, and SSL termination be set up? What’s the most effective way to make services publicly available while maintaining security?

Project Calico offers answers to some of these questions with its Service IP Advertisement feature, which integrates with existing Top of Rack (ToR) infrastructure to provide routing to either private Kubernetes Service IPs, or public Kubernetes External Service IPs. Service IP Advertisement is a good solution if:

  • Your ToR solution is capable of running the Border Gateway Protocol (BGP)
  • You want to share services from your cluster to the rest of your network infrastructure.
  • You want to take advantage of network load balancing.

To enable Service IP Advertisement Calico needs to peer with a BGP router, one that is external from Calico’s internal routers but local to your network. BGP is one of the most fundamental routing protocols used in networking. At a high level, BGP works by sharing routes between trusted peers. When peered with your ToR, Calico shares routes to Kubernetes services, which makes them available to your entire network.

To help with the discussion of this feature, assume we have a network and cluster configured in the following way:

  • A rack of servers, hosting the Kubernetes nodes, are connected to the physical network through a Top of Rack router. 
  • Calico is running as the CNI and Network Policy plugin.
  • Calico and the Top of Rack router are configured to peer together using BGP.

Within this setup, we have the following network configuration:

  • The Top of Rack router’s IP address is 192.168.1.1 and the servers have IP addresses allocated from 192.168.1.0/24.
  • The Kubernetes pod network is configured with CIDR 10.48.0.0/16.
  • The Kubernetes services cluster IP range is configured as 10.49.0.0/16.
  • An external service IP range is configured as 192.168.3.0/24.

The exact configuration of your ToR BGP router is beyond the scope of this post, and will vary depending on the vendor or software package you’re using. If you don’t have a rack of servers available to you but still want to try this feature out and see how it works, the Bird Internet Routing Daemon (BIRD) running on a separate server is a good choice for trying things out.

The first step is to enable peering between the ToR and the Calico network. How this works will vary depending on your ToR implementation, but a few key things to keep in mind are:

  • The ToR router will need to be configured to peer with Calico that’s running on each node.
  • The ToR needs to accept routes and traffic from the external network, external service network, and pod service network.
  • If it’s an option, the ToR should have graceful restarts enabled to prevent network service interruption.

With the ToR configured to accept routes, the next step is to enable peering on the Calico side. Begin by telling Calico about the external BGP router with the following manifest:

calicoctl apply -f - << EOF
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-global-64512
spec:
  peerIP: 192.168.1.1
  asNumber: 64512
EOF

With peering enabled, Calico has made pods first-class citizens on your network, without needing an overlay network, and making them directly accessible outside of the cluster. 

Although peering is enabled, Calico still needs further configuration to expose the Kubernetes service IP range. This can be done by creating a new Calico BGPConfiguration resource:

calicoctl create -f - <<EOF
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  serviceClusterIPs:
  - cidr: 10.49.0.0/16
EOF

With advertisement of the service cluster IP range enabled, your routing table on your ToR will look something like this:

$ ip r
...
10.49.0.0/16
nexthop via 192.168.1.10 dev eth2 weight 1 
nexthop via 192.168.1.11 dev eth2 weight 1 
nexthop via 192.168.1.12 dev eth2 weight 1 
...

Note how the routes to the 10.49.0.0/16 network are ECMP load balanced across the nodes. Any services you expose will be load-balanced by the ToR across all of the nodes. To illustrate this, we can create a basic Nginx service.

control:~$ kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
EOF
control:~$ kubectl expose nginx
control:~$ kubectl get services
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.49.0.1      <none>        443/TCP   19m
nginx        ClusterIP   10.49.62.131   <none>        80/TCP    4m43s

From the external network, you can now access the Kubernetes hosted service at its internal cluster IP address.

external:~$ curl 10.49.222.180
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

Under the covers there are two stages of load-balancing happening here.  First the ToR load-balanced the connection to a cluster IP by routing it to a node.  Then kube-proxy running on that node load-balanced to a specific pod using NAT (Network Address Translation) to change the destination IP from the cluster IP to the IP address of one of the backing pods. The backing pod might be on the local node, or it might be on one of the other nodes, resulting in another network hop.

If we want to avoid the extra potential network hop, we can do so by setting the external traffic policy on the service to local.

control:~$ kubectl patch service nginx \
  -p '{"spec":{"type": "NodePort", "externalTrafficPolicy":"Local"}}'

On your ToR you will now see the addition to the routing tables, including ECMP load balancing of the Nginx service to the specific nodes on which Nginx is running:

$ ip r
...
10.49.0.0/16 
nexthop via 192.168.2.10 dev eth2 weight 1 
nexthop via 192.168.2.11 dev eth2 weight 1 
nexthop via 192.168.2.12 dev eth2 weight 1 
...
10.49.62.131
nexthop via 192.168.2.11 dev eth2 weight 1 
nexthop via 192.168.2.12 dev eth2 weight 1 

This is an easy way to expose Kubernetes network services, but operationally it has a drawback of exposing every service on that IP range in your Kubernetes cluster to the rest of your network. If you want more fine-grained control over which services are made available, or want to assign truly public-facing IP addresses, Calico can address this by advertising external service IPs. The method is similar, with a major difference being that external IPs aren’t managed by the Kubernetes cluster and must be assigned manually to the services. The next example will illustrate that. Begin by reconfiguring the BGPConfiguration to advertise the external IPs rather than the internal IPs (it’s worth noting that you can expose both sets at once, but for this example we want to turn off public access to the internal network while still providing access to the application). 

control :~$calicoctl create -f - <<EOF
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  serviceClusterIPs:
  serviceExternalIPs:
  - cidr: 192.168.3.0/24
EOF

Checking the routes, note that the routes to the cluster IP range are removed and routes to the external service network have been added. 

$ ip r
...
192.168.3.0/24
nexthop via 192.168.1.10 dev eth2 weight 1 
nexthop via 192.168.1.11 dev eth2 weight 1 
nexthop via 192.168.1.12 dev eth2 weight 1 
...

The service’s cluster IP is no longer publicly visible.

external :~$ curl -m 10 10.49.62.131
curl: (28) Connection timed out after 10001 milliseconds

We are now advertising the external IP range, but we also need to assign an external IP to the service:

control:~$ kubectl patch svc nginx \
-p  '{"spec": {"externalIPs": ["192.168.3.180"]}}'
control:~$ kubectl get services
NAME       TYPE       CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
kubernetes ClusterIP  10.49.0.1      <none>          443/TCP        152m
nginx      NodePort   10.49.62.131   192.168.3.180   80:31890/TCP   109m

Check the connectivity:

external:~$ curl 192.168.3.180
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

That completes our quick tour of how Calico can advertise Service and External Service IPs outside of a Kuberentes cluster. For on-premises clouds with BGP enabled routing, it’s a simple solution for giving access to Kubernetes services without the additional work of installing and maintaining custom Kubernetes load-balancer or ingress controllers. If you want to read more about this feature, check out the official Project Calico “Advertise Kubernetes Service IPs” Documentation

Are you using this feature? Do you have any comments or feedback on it or other Calico features? The Project Calico team would love to hear from you. Head on over to our Slack channel, or start a discussion over on the Calico Discourse.


Kubernetes icons CC-BY-4.0, copyright 2019 Linux Foundation

References:
Kubernetes Service IPs
Kubernetes External IPs

Chris Hoge

Chris Hoge

Chris is a Developer Advocate for Project Calico. Prior to joining Tigera he was a Technical Program Manager at the OpenStack Foundation, where he helped launch an interoperability program and coordinated cross-community efforts between the OpenStack and Kubernetes communities. He holds an MS in Applied Mathematics from the University of Colorado.

You Might Also Like

What’s new in Calico v3.14

We’re very excited to announce Calico v3.14.0, which includes a bunch of great features including automatic host endpoint management and a tech preview of Wireguard encryption. Thank you to all

Read More

Join our mailing list​

Get updates on blog posts, new releases and more!

Thanks for signing up!