Learn

Calico has redefined the way to build out data center networks using a pure Layer 3 approach that is simpler, higher scaling, better performing and more efficient than the standard approach of using overlay networks.

Through environment specific plug-ins, calico integrates seamlessly with cloud orchestration systems such as OpenStack and Docker to provide networking between local and geographically distributed workloads.

Removing the packet encapsulation associated with the standard Layer 2 solution simplifies diagnostics, reduces transport overhead and improves performance.

The Calico approach of using a pure IP network combined with BGP for route distribution allows internet scaling for virtual networks.

Difficulties with traditional overlay networks

Traditional virtual infrastructures have offered a LAN-like (Layer 2) experience to users configuring multiple workloads.  This may require several layers of virtual LANs, bridging and tunnelling to make Layer 2 networking work across multiple physical hosts.  This presents a range of problems.

traditional-no-txt
Scale challenges above few hundred servers / thousands of workloads.
Difficult to troubleshoot due to packet encapsulation.
On/off-ramp device (or virtual router hop) required to access non-virtualized devices.
Every node in the network is state-heavy (e.g. VLANs, tunnels).
Virtual NAT device required to connect a workload to a public “floating IP”.
High availability / load balancing across links requires additional LB function and/or app-specific logic.
Geographically distributed data centers require inter-DC tunnels.
CCNA or equivalent required to understand end-to-end networking

The Calico solution

Calico provides a pure L3 fabric solution for interconnecting Virtual Machines or Linux Containers (“workloads”). Instead of a vSwitch, Calico employs a vRouter function in each compute node. The vRouter leverages the existing L3 forwarding capabilities of the Linux kernel, which are configured by a local agent (“Felix”) that programs the L3 Forwarding Information Base with details of IP addresses assigned to the workloads hosted in that compute node. Felix also programs Access Control Lists in each compute node to enforce whatever security policy may be needed, for example to provide isolation between tenants. The vRouter function makes use of Border Gateway Protocol to advertise (typically) /32 or /128 routes to workloads hosted in each compute node. The BGP stack in each compute node is connected to a virtualised BGP Route Reflector to provide scalability of the control plane.

Calico connects each workload via the vRouter directly to the infrastructure network. There is no overlay, no tunnelling, no VRF tables. There is also no requirement for NAT in a Calico network, since any workload can be assigned a public IP address which is directly reachable from the Internet – provided you configure a security policy to allow this.

Scale to millions of workloads with minimal CPU and network overhead
What is happening is “obvious” – traceroute, ping, etc., work as expected; routing and ACL rules tell you everything you need to know
Path from workload to non-virtualized device is just a route
Physical fabric is state-light (standard IP forwarding only)
External connectivity is achieved by assigning a public IP
Equal Cost Multi-Path (ECMP) any Anycast just work, enabling scalable resilience and full utilization of physical links
Traffic between data centers is natively L3 routed
Basic IP networking knowledge only required

Connectivity through IP routing

In Calico, IP packets to or from a workload are routed and firewalled by the Linux routing table and iptables infrastructure on the workload’s host.   For a workload that is sending packets, Calico ensures that the host is always returned as the next hop MAC address regardless of whatever routing the workload itself might configure. For packets addressed to a workload, the last IP hop is that from the destination workload’s host to the workload itself.

ip-hops

An environment specific plugin (e.g. a Neutron plugin for OpenStack) notifies a Calico agent when it is asked to provision connectivity for a particular workload.  The Calico agent adds to the host routing table direct routes to each workload via a TAP (or veth, etc.).   A BGP client running on the host notices those and distributes them – perhaps via a route reflector – to BGP clients running on other hosts, and hence the indirect routes appear also.

Scalability, performance and simplicity

Calico simplifies the network topology, removing multiple encapsulation and de-encapsulation.  This is key to understanding how a Calico can perform and scale so well.  In particular:

  • Smaller packet sizes mean that there is reduction in possible packet fragmentation.
  • There is a reduction in CPU cycles handling the encap and de-encap.
  • Easier to interpret packets, and therefore easier to troubleshoot.
no-encap
why-bgp-networks

When a workload is created, moved or destroyed, the speed of the propagation of the routing information also plays a factor into the scalability.

Calico offers outstanding scalability because it’s architected on the exact same principles as the Internet, using Border Gateway Protocol as the control plane.  With well-known implementations of BGP able comfortably to handle millions of distinct routes, Calico should scale to meet the network needs of the largest imaginable data centers.  And because Calico connects virtual machines directly via IP, it scales beyond the data center and natively supports cloud connectivity across any geographic distribution.

Security

In principle, routing in a Calico network allows any workload in a data center to communicate with any other – but in general an operator will want to restrict that; for example,to isolate customer A’s workloads from those of customer B.  Therefore Calico also programs iptables on each host, to specify the IP addresses (and optionally ports etc.) that each workload is allowed to send to or receive from.  This programming is ‘bookended’ in that the traffic between workloads X and Y will be firewalled by both X’s host and Y’s host – this helps to keep unwanted traffic off the data center’s core network, and as a secondary defence in case it is possible for a rogue workload to compromise its local host.

Field-hardened traffic separation

Applications running within a given tenant’s virtual network are completely separated from other tenants’ traffic.
Traffic separation is enforced by the Linux networking subsystem’s access control list (ACL) processing — code that has been field-hardened and proven in massive scale production environments for many years.

Fine-grained policy rules

While all virtual networking solutions implement per-tenant traffic separation, most do not support finer-grained security policy (without the addition of a separate firewall service). Through its use of Linux’s native ACL processing, Calico can be extended to allow a wide array of security rules – anything that can be defined in terms of a networking ACL can be implemented in Calico.

This rich policy engine would allow an administrator to, for example, give a tenant a VM which is only allowed external access to a specific set of IP addresses and over a specific TCP port – in effect providing basic firewall functionality within the core implementation of the virtual networking layer.

Security through Simplicity

Calico’s use of flat IP networking principles, without multiple levels of address translation and payload encapsulation, results in a simple “WYSIWYG” networking model where every packet on the network clearly identifies who it is from, and where it is going. Network administrators can therefore easily identify any sources of rogue traffic – whereas complex (and often proprietary) tunneling protocols create layers of obfuscation that can be hard to analyze and debug.

Open standards and Open Source

GitHub

Calico itself is open source and open standards.  We utilize other open source communities and actively push enhancements and fixes to those communities:

  • Linux kernel – layer 3 forwarding and ACL enforcement
  • BIRD – BGP stack
  • Platform plug-in APIs (e.g. Neutron for OpenStack).