Understanding the policy enforcement options with Calico

Calico provides a rich set of network policies with a unified syntax to protect bare metal, VMs, and pods. These policies are scoped as namespaced or global network policy, and can apply to host endpoint (to protect applications running directly on the host – can be bare metal or VM) or workload endpoint (to protect applications running in containers or VMs hosted on the host). Calico policies enable you to enforce security controls at various points in the packet path, via options like preDNAT, untracked, and applyOnForward. When understood properly, these capabilities can help achieve better security and performance. This article explains these Calico policy options (preDNAT, untracked, and applyOnForward) applicable to host endpoints with a focus on packet processing path.

This post assumes that you have a basic understanding of Kubernetes and Calico network policies. Otherwise, we recommend you try the basic network policy tutorial and host protection tutorial using Calico first, before reading through this. We also expect that you have a basic understanding of iptables on Linux.

Calico global network policy enables you to apply a set of access rules to a selector (group of hosts and workloads/pods). This is very useful when implementing a heterogeneous mix of bare metal, VMs and kubernetes infrastructure. Additionally, you can harden your cluster (nodes) using a set of declarative policies, and apply network policy to the incoming traffic (e.g. via service NodePorts or External IPs).

At a fundamental level, when Calico networks a pod (refer to the diagram below), it connects it to the host using a virtual ethernet interface (a veth). When a pod sends traffic, it arrives at the host on this virtual interface and is processed in the same way as if it came in over a physical network interface. By default Calico names those interfaces caliXXX. Since traffic arrives over the virtual interface, it passes through iptables as if the pod was one hop away. So when traffic goes in/out of pod, it gets forwarded from a host standpoint.

On a Kubernetes node running Calico, you can map from a veth to a workload as follows. In the example below, you can see that the veth #10 (calic1cbf1ca0f8) is connected to cnx-manager-* in calico-monitoring namespace.

[centos@ip-172-31-31-46 K8S]$ sudo ip a
...
10: calic1cbf1ca0f8@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
…

[centos@ip-172-31-31-46 K8S]$ calicoctl get wep --all-namespaces
…
calico-monitoring cnx-manager-8f778bd66-lz45m                            ip-172-31-31-46.ec2.internal 192.168.103.134/32
calic1cbf1ca0f8
…

Given that Calico creates the veth interface for every workload, how does it enforce policy? For that, Calico creates hooks into various chains in the network processing path using iptables.

The diagram below shows the chains involved in packet processing in iptables (or, netfilter subsystem). As the packet comes in via network interface, it goes through PREROUTING chain first. Then a routing decision is made and based on that, the packet goes through either INPUT (directed to host processes) or FORWARD (directed to a pod or to another node in the network). From the local process, the packet goes via OUTPUT chain, and then POSTROUTING before being sent on the wire.

Note that a pod is also an external entity (connected to a veth) from iptables processing standpoint. To summarize:

• Forwarded traffic (nat, routed, or to/from a pod) goes through PREROUTING – FORWARD – POSTROUTING chains.
• Traffic to a local host process goes through PREROUTING – INPUT chain.
• Traffic from a local host process goes through OUTPUT – POSTROUTING chain.

Calico provides policy options that allows enforcement across all the chains. With this in mind, let’s navigate various policy options available in Calico. The numbers below map with the numbers in the diagram above.

1. 1. Workload endpoint (pod) policy
   2. Host endpoint policy
   3. ApplyOnForward option
   4. PreDNAT policy
   5. Untracked policy

Let us start with looking at how policy is applied to workload endpoints (Kubernetes pods or OpenStack VMs) first, and then review the policy options for host endpoints.

Workload Endpoints

Workload Endpoint Policy (1)

This is the option to protect your kubernetes pods. Calico implemented Kubernetes Network Policy, plus for a richer set of capabilities, Calico NetworkPolicy and GlobalNetworkPolicy. Calico creates a chain for every pod (workload) and hooks the INPUT and OUTPUT chain of the workload into the filter table of FORWARD chain.

Host Endpoints

Host Endpoint Policy (2)

Besides CNI (container network interface), Calico policies also provide host protection. In Calico, you can define a host endpoint to represent a combination of host interface and optionally, port numbers. These get enforced by filter table in INPUT and OUTPUT chains. As you can see in the diagram, (2) is enforced on local processes on the node/host. So if you created a policy that applies to a host endpoint, it will not impact the traffic going to/from your pods. But this provides a single interface/syntax for you to lock down both your host and pods using Calico policies. That significantly reduces the overhead in managing policies for a heterogeneous network. Cluster hardening is another important use case for host endpoint policy.

ApplyOnForward Policy (3)

ApplyOnForward option is available in Calico global network policy to enable the policy enforcement on all traffic travelling through the host endpoint, including traffic that will be forwarded by the host.  This includes forwarding to a local pod, or forwarding to back out to somewhere else in the network. Calico requires this option to be enabled for PreDNAT and untracked policies as discussed in the following sections. Additionally, ApplyOnForward can be used to police traffic that is hair-pinned back out of the host for use cases such as virtual router or NAT gateway software appliances.

Note that if all you need the same network policy on both host process and pods, then you do not need ApplyOnForward option. You simply need to label the host endpoint and workload endpoint (pod) appropriately. Calico is smart enough to apply policy based on labels, irrespective of endpoint type (hostendpoint or workload).

PreDNAT Policy (4)

In Kubernetes, services are exposed to outside by NodePorts, or optionally (when using Calico) as advertised service Cluster IPs or service External IPs. Kube-proxy load balances the incoming service-bound traffic to the pods backing the service using DNAT. Given this, how do you enforce policies on traffic coming in via NodePorts? Given that those policies need to be enforced before the traffic has been DNAT’ed (mapping from host:port to appropriate service), Calico provide a spec option in globalNetworkPolicy called “preDNAT: true”.

When pre-DNAT is enabled, those policies are implemented at (4) in the diagram – mangle table of PREROUTING chain – just before DNAT. Normal policy order is not honored here, as this happens much earlier in the processing chain. However, multiple preDNAT policies honor policy ordering among themselves.

While designing your pre-DNAT policies, it is important to be precise with the packets you want to handle here and let most fall through. An ‘allow’ action in a pre-DNAT policy will bypass further host endpoint policy, whereas fall-through the pre-DNAT policy will go through the remaining chains.

Calico makes it mandatory to specify applyOnForward option when using preDNAT because, by definition, the destination has not yet been selected. The traffic may be destined to the host process, or it may be forwarded to a pod or to another node.

Untracked Policy (5)

Networks and applications can have wide variations in behavior. In some extreme uses cases  applications may generate lots of connections in rapid bursts. This might cause the conntrack (core component of Linux network stack) to run out of memory.  Traditionally to run this kind of application on Linux you need to manually tune or disable conntrack or write iptables rules to bypass conntrack. Untracked policy in Calico is a simpler, powerful option if you want to process extreme connection rates.  For example massive memcache, or DDOS protection use cases.

Refer to this blog post for further insights, including performance statistics on untracked policy.

When you set the spec option “doNotTrack: true” in a Calico globalNetworkPolicy, it becomes an untracked policy and is enforced at the earliest point in the Linux packet processing pipeline. Looking at the above diagram, the untracked policies are enforced in the PREROUTING and OUTPUT chains of the raw table, before connection tracking (conntrack) is triggered. When a packet is allowed by an untracked policy, it is marked to disable connection tracking for that packet. This means:

• The processing for an untracked policy is per-packet. There is no concept of a connection (or flow). The lack of a connection has some significant implications:
  • If you want to allow both request and response traffic you need both an inbound and an outbound rule (since connection tracking is the way that Calico normally whitelists response traffic).
  • Untracked policy does not work for Kubernetes workloads (pods), because there’s no way to untrack the outbound connection from the pod.
  • NAT does not work correctly with untracked packets (since the kernel stores the NAT mapping in the connection tracking table).
• An allow-all rule in an untracked policy will mark all packets as untracked. This is almost never what is wanted so it’s important to be very selective in which packets are allowed by untracked policies (and allow most traffic to fall through to normal, tracked policy).
• Untracked policies are executed at the very beginning of the kernel’s packet processing pipeline. It is very important to understand this while creating Calico policies. You can have a pod-based policy at order 1 vs. an untracked policy at order 1000. It does not matter. Untracked policy will be executed before the pod based policy. Untracked policies honor ordering within themselves.

As one of the goals of doNotTrack policy is enforce the policy at the earliest point in the Linux packet processing pipeline, Calico makes it mandatory to specify applyOnForward option when using doNotTrack. Referring to the packet processing diagram, notice that untracked policy (5) is enforced before any forwarding decisions. The traffic may be destined to the host process, or it may be forwarded to a pod or to another node.

Summary

We reviewed different policy options (Host endpoint, ApplyOnForward, preDNAT, and Untracked) in Calico and how these are enforced in the packet processing path. Understanding different policy options in Calico helps with efficient and secure policy design. With Calico, you can use a global network policy that applies to a selector (mix of nodes and pods) and enforces policies with various options. This makes it very convenient for security and network admins to secure “all the things” (types of endpoints) using a single policy language with Calico policies.

Acknowledgement: I would like to acknowledge and thank Shaun Crampton and Alex Pollitt for their review and providing valuable insights.

If you enjoyed this blog post then you might also like:

• Free online training at projectcalico.org/events or subscribe to Calico Essentials for personalized training & workshops
• Learn about Calico Enterprise