Using Ethernet as the interconnect fabric for a Calico installation

I’ve published the first in a series of technical notes talking about how you design your infrastructure to make the most optimal use of Project Calico and the benefits that it provides.  The first installment of that series is a discussion about using an Ethernet interconnect fabric to interconnect Calico nodes.

“But wait,” you might be saying, “all the cool kids are doing IP fabrics for their cloud — it’s the only thing that scales.”  This statement is, of course, correct.  The path that everyone is taking for interconnect design is IP–based, and it is due to concerns about scale.  However, this model is really all about IP at the edge of the network.  In most cloud networking infrastructures, the edge of the network is the top of rack (ToR) switch.  The compute servers below the ToR are simply encapsulating traffic up and sending it up to the ToR.  There is no visible (to the interconnect fabric) aggregation happening at the compute servers.  Calico also pushes IP to the edge, but in a Calico network, that edge is the compute servers themselves.  We’ve just taken the original concept, and driven it to it’s logical conclusion, IP at the first point of aggregation, which is, in reality, on the compute server.

In this technical note, Calico over an Ethernet interconnect fabricI describe an Ethernet–based, truly scalable, interconnect fabric for Calico that substantially simplifies OA&M of the physical network, and removes IP scaling concerns from the switching layer without resorting to overlay networks, and the issues that accompany them.

As always, I welcome your constructive feedback, praise and adoration, and/or flames and invective musings.

Note: Originally, this technical note was actually in this blog post, but with the advent of our spiffy new documentation site docs.projectcalico.org, I’ve moved the actual technical note there, and replaced it with this pointer post.

Christopher Liljenstolpe

Christopher Liljenstolpe

Christopher is the original architect of Project Calico and one of the project's evangelists. In his day job, he's the director of solutions architecture at Metaswitch Networks. Prior to Calico/Metaswitch, he's designed and run some bio-informatics OpenStack clusters, done some SDN architecture work at Big Switch Networks, Run architecture at two large carriers (Telstra - AS1221, and Cable & Wireless/iMCI - AS3561) and been the IP CTO for Alcatel in Asia. He's also run networks in Antarctica (hint, bend radius becomes REALLY important at -50C), and been foolish enough to do a stint as a wg co-chair in the IETF. Occasionally you can have the (mis-)fortune of hearing him speak at conferences and the like.

You Might Also Like

EKS, Bottlerocket, and Calico eBPF

Introduction Bottlerocket is an open source Linux distribution built to run containers securely in scale by Amazon, it is uniquely tailored to improve stability and performance with a great focus

Read More

What’s new in Calico 3.16

We’re very excited to announce Calico v3.16.0! This release includes the eBPF dataplane going GA, the addition of Windows support for Kubernetes, and BGP communities support. In addition, this release

Read More

Join our mailing list​

Get updates on blog posts, new releases and more!

Thanks for signing up!