Introducing XDP-Optimized Denial-of-Service Mitigation

In this article, we’ll explore Calico’s denial-of-service (DoS) mitigation features, including XDP-optimisation support introduced in Calico v3.7.

Using Calico Policy for DoS Mitigation

During a DoS attack, a cluster can receive massive numbers of connection requests from attackers. The faster these connection requests are dropped, the less flooding and overloading to your hosts. You can fend off these attacks with DoS mitigation rules in normal Calico network policy applied directly to your host network interfaces.

But there’s a less well known Calico network policy feature, doNotTrack, that will apply such policies as early as possible in the Linux packet processing pipeline. Prior to Calico 3.7, this meant in the iptables “raw” table, before Linux conntrack and other network processing. As discussed in one of our previous blogs, you can use doNotTrack policy to significantly increase scale for niche very high connection workloads. It’s also a useful feature in your arsenal for defending against DoS attacks as efficiently as possible.

Calico 3.7 enhances this approach further with the introduction of automatic XDP-optimization of DoS mitigation rules.

What is XDP?

XDP (eXpress Data Path) is a Linux kernel capability that allows BPF programs to be inserted very early in the Linux’s packet processing pipeline. Depending on specific network hardware and kernel version, there are three places in the pipeline where an XDP program can be attached:

• offloaded all the way to the NIC (blazing fast, processed in hardware)
• in the NIC driver (very fast, before it reaches the main Linux kernel network stack)
• after the NIC driver, in the kernel’s “generic” processing (early in the main Linux kernel network stack, but not significantly faster than iptables “raw” table)

Calico will use the fastest available hook that is enabled in its configuration.

XDP is a relatively new kernel feature. Calico will only enable it on Linux kernel v4.16 and above due to bugs and missing features in earlier versions.  On earlier kernels, Calico will continue to use the iptables “raw” table to implement all “doNotTrack” rules.

About the new feature

The denial-of-service protection feature allows Calico to automatically optimise certain kinds of host protection policy rules by converting them to XDP programs.  In this first iteration, we’ve optimised a certain sort of policy rule that is useful for very early dropping of traffic from a global network set.  This makes it suitable for dropping traffic quickly from a configurable blacklist of CIDRs, exactly what you might need to block a DoS attack.  (See the how-to guide for all the details.)

OK, so how much difference does it make?

Most people don’t have access to NICs that can offload XDP to hardware, so in this test we use NICs that supported offload to the NIC driver as a more realistic scenario for most readers.

We set up an NGINX pod on a fairly modest two core test host and generated 300 requests/s from another host acting as normal “legitimate” service load. We then simulated a DoS attack using a third host to generate attack packets targeting the NGINX server. We increased the rate of the attack packets until it had an impact on the legitimate traffic.

The first thing to note is that even on this modest host (only 2 cores), using Calico doNotTrack policy enforced with iptables, the DoS attack has very little impact at 3.0 million attack packets/s. The real impact comes beyond that when the host started to become CPU constrained – at which point the legitimate service traffic really starts to suffer with around 80% of legitimate connection attempts failing.

In contrast, with XDP driver enforcement, we were able to push up to 4.4 million packets/s with this test set-up and it only showed early signs of a reduction in legitimate service throughput.

Conclusion

In most normal circumstances, the performance of the Linux networking stack represents an insignificant proportion of your overall application load.  Under DoS attack it is a different story. Calico’s doNotTrack policy can help mitigate a DoS attack by dropping traffic early in the Linux packet processing pipeline.  With the introduction of XDP-optimization in Calico 3.7, if your kernel and NIC support it, then the attack traffic will be dropped by the NIC driver (or in the NIC itself). By policing traffic earlier in the pipeline it reduces the work done per packet and gives a substantial increase in performance for the DoS attack mitigation use-case.

————————————————-

Free Online Training
Access Live and On-Demand Kubernetes Tutorials

Calico Enterprise – Free Trial
Solve Common Kubernetes Roadblocks and Advance Your Enterprise Adoption