A while (too long) ago, I promised that I would follow up on my discussion about Ethernet fabrics with a similar discussion on using IP fabrics to interconnect Calico nodes. That work is (finally) done, and can be found in our technical documentation entitled, IP Interconnect Fabrics in Calico.
For those of you who will eventually get around to reading it, but want the spoiler right now, we conclude that IP interconnect fabrics (i.e. running a top of rack switch as a router) is feasible in Project Calico. There are more considerations than in the Ethernet case, discussed in the Calico Over an Ethernet Inteconnect Fabric note, some of which include:
- Size of the routing table in the Calico network and compared to the forwarding table size in the switches being used
- Insuring proper autonomous system boundaries and insuring continuity of those autonomous systems
- Insuring correct next–hop behavior in the routing announcements
We propose two viable BGP models for a Calico IP interconnect fabric:
- An AS per rack model
- An AS per server model
Each of these models can have two variants, depending on whether the spine switches are also BGP routers, or separate Ethernet planes as in the Ethernet fabric option.
Please read the note, and comment, make suggestions, rant & rave, or congratulate our intelligence.