Dual Stack Operation with Calico on Kubernetes

Dual Stack Operation with Calico on Kubernetes

Introduction

One of the new features Calico v3.11 introduced is full Kubernetes dual stack support – which allows each Kubernetes pod to get an IPv6 address as well as an IPv4 address, and can communicate over both IPv6 and IPv4. This blog explains more about how that works, and how you can make use of it.

 

Calico, IPv4 and IPv6

Calico has actually supported both IPv4 and IPv6 for a long time already. For example, dual stack operation with OpenStack, and IPv6-only operation with Kubernetes, were both possible before v3.11. What really changed in v3.11 was in the integration between Calico and Kubernetes, specifically with Kubernetes v1.16 allowing multiple IP addresses for each pod, and Calico handling that multiplicity correctly.

 

Kubernetes Dual Stack support

Prior to Kubernetes v1.16, the Kubernetes API only supported a single IP address per pod. Typically that would be an IPv4 address. It was also possible to set up an IPv6-only cluster, and in that case it would be an IPv6 address. But the point is that a pod could not have both IPv4 and IPv6. If you’re a guru of the Kubernetes API, you’ll realize that we’re talking here about the PodStatus.PodIP field.

During the development for Kubernetes versions 1.16 and 1.17, the Kubernetes team added the support needed for each pod to have both an IPv4 and an IPv6 address. The full detail of that work is available, but the crucial detail for us here is the introduction of a plural Kubernetes API field, PodStatus.PodIPs. PodStatus.PodIPs can hold multiple IP addresses – thus allowing for both IPv4 and IPv6. The legacy PodStatus.PodIP field is now required to always be the same as PodStatus.PodIPs[0].

 

Fundamentals for IPv6 networking

Calico routes IPv6 traffic from pods over the nodes’ own IPv6 connectivity, so there must also be IPv6 connectivity between the cluster nodes. You can check this by provisioning a global-scope IPv6 address on each node, if there isn’t one already, and using ping6 from one node to the IPv6 address of another node. Similarly, for pods to be able to connect to the outside world over IPv6, there must be IPv6 connectivity from the cluster nodes to the outside world. As far as Calico is concerned, there is no complicated IPv6 over IPv4 magic going on; IPv6 should be native IPv6, just as IPv4 is native IPv4. (Of course, there might still be overlay magic somewhere in the underlying network fabric, but Calico and Kubernetes don’t see or get involved with it.)

 

Enabling Dual Stack with Kubernetes and Calico

So how can you enable dual stack operation when installing a new Kubernetes cluster with Calico? With a high-level installer, it should be as simple as toggling a single knob somewhere, but the installers haven’t had a lot of time yet to provide that, so in the meantime and for the sake of understanding, we have to do it The Hard Way ™…

    1. Before you start it’s good to understand the prerequisites that Kubernetes requires for dual stack with Calico:
      • You must be using Kubernetes 1.16 or later.
      • Your cloud provider or underlying network must be able to provide Kubernetes nodes with routable IPv4 and IPv6 network interfaces.
      • You must run kube-proxy in IPVS mode.
    2. To install a Kubernetes cluster with dual stack this means you need to set the following options during cluster creation (or review and ensure that your installer is setting these):
      • kube-controller-manager options:
        • -feature-gates=”IPv6DualStack=true”
        • -cluster-cidr=<IPv4 CIDR>,<IPv6 CIDR> e.g. -cluster-cidr=10.244.0.0/16,fc00::/24
        • -service-cluster-ip-range=<IPv4 CIDR>,<IPv6 CIDR>
        • -node-cidr-mask-size-ipv4|–node-cidr-mask-size-ipv6 defaults to /24 for IPv4 and /64 for IPv6
      • kubelet options:
        • -feature-gates=”IPv6DualStack=true”
      • kube-proxy options:
        • -proxy-mode=ipvs
        • -cluster-cidrs=<IPv4 CIDR>,<IPv6 CIDR>
        • -feature-gates=”IPv6DualStack=true”
    3. When you or your installer are about to apply the Calico manifest, there are a few things that you need to change first, for which the full details are in our documentation:
      • The CNI config, so that Calico’s IPAM will allocate both IPv6 and IPv4 addresses for each new pod.
      • Setting the following environment variables for the calico-node container:
        • IP6=autodetect, so that Calico will detect the node’s IPv6 address and use this in its BGP IPv6 config.
        • CALICO_IPV6POOL_CIDR=<the same as the IPv6 range you configured as the cluster CIDR to kube-controller-manager and kube-proxy>, so that Calico knows the range from which to allocate IPv6 addresses.
        • FELIX_IPV6SUPPORT=true, so that Felix knows to program routing and iptables for IPv6 as well as for IPv4.

 

That’s it!

If you followed the steps above successfully then you should see that each new pod gets an IPv6 address as well as IPv4, and can communicate with each other and the outside world over IPv6.

Then you can go and join in the debate about IPv6 extension headers 🙂 Thank you for reading and Happy IPv6ing!

Avatar
Neil is a core developer at Project Calico and Tigera. Outside tech he mostly spends his time on choral singing.