Nov 26, 2021
Increasing Data Centers Interworking Resiliency — Loop Protection via Domain Path
Data Centers Interworking Layer-3 Loops
Today in modern data center network virtualization is supported by creating logical overlay networks with tunnelling protocols (like VXLAN, MPLS and GENEVE) and control plane protocols (like BGP EVPN, MPBGP VPN), which can support sophisticated layer-2 and layer-3 network services over simple IP underlay networks in a more reliable and scalable way. Typically BGP is used as a unified control plane protocol for tenants to perform both intra-DC forwarding and external networks interworking. Modern DC overlays usually can prevent most looping scenarios natively, but when combining them with older network topologies or interworking with different routing domains can still introduce the risk of layer-3 loops which could disrupt network stability and cause service outage.
Below is a typical distributed data centers interworking scenario where a tenant needs to span multiple routing domains with different address families across multiple locations:
· A SR-MPLS VPN based WAN core is used to interconnect a Cisco ACI DC with a remote VXLAN BGP EVPN based DC.
· Cisco ACI is SDN controller based where VXLAN is used for overlay data plane tunnelling and COOP/MPBGP for overlay control plane.
On border leaf’s of both DCs, BGP based IP VRF lite hand-off or VXLAN to SR/LDP MPLS handoff is used for interworking.
Obviously in above scenario, a tenant network needs to span not only EVPN domains but also domains where BGP VPN-IPv4/VPN-IPv6 and IPv4/IPv6 BGP families provide inter-subnet forwarding. BGP Routing loops become highly possible in this sort of interworking scenarios when considering other conditions such as:
• Intentional disabling of existing BGP loop prevention mechanisms like AS Path checks. For example, enable BGP AS override, AS allow-in features.
• Route leaks across different VRFs/VPNs to support inter tenant connectivity.
Below is a simple example of a possible scenario with a layer-3 routing loop:
In this scenario, IP L3out is referring to VRF lite BGP IP VPN based interworking. Infra L3out stands for VXLAN to SR/LDP MPLS handoff based interworking. A prefix P1 received on a Cisco ACI border leaf BL1 from an IP L3out peer is advertised in the fabric using MPBGP VPN [step 1 and 2]. As a transit case, prefix P1 can be advertised externally out through an SR MPLS Infra L3out [step 3]. This prefix can potentially be imported into the fabric from the MPLS core either in same VRF or in a different VRF [step 4 and 5]. A loop will occur when this imported prefix is, in turn, advertised back to the originating TOR either from the same VRF or through a leak from a different VRF [step 6].
For loop protection in DC interworking scenarios like above, we need a solution to specify the interworking aspects between those different families and routing domains.
BGP Domain Path Attribute for Loop Protection
In recent years, a new BGP Path attribute called D-PATH (Domain Path) was introduced in internet-draft EVPN Interworking with IPVPN, which provides loop protection and visibility on the domains a particular route has traversed. The deployment of D-Path can help address potential DC interworking loop issue mentioned in previous scenarios. BGP Domain Path has following attribute per internet-draft:
· The BGP Domain Path Attribute is an optional and transitive BGP path attribute.
· D-Path is composed of a sequence of Domain Segments with each segment being of type <domain segment length, domain segment value>
· Domain segment value is a sequence of one or more Domains with each domain represented by <DOMAIN-ID:SAFI>.
· The domain segment length field is a 1-octet field, containing the number of domains in the segment.
· DOMAIN-ID is a 6-octet field that represents a domain. It is composed of a 4-octet Global Administrator sub-field and a 2-octet Administrator sub-field.
· SAFI is a 1-octet field that indicates the SAFI type in which a route was advertised in the domain. The following SAFI types are supported: IP, VPN and EVPN.
Cisco ACI BGP Domain Path Solution
Beginning with Cisco ACI Release 5.1(3), the new BGP Domain-Path feature is available, which helps with BGP routing loops in the following ways:
· Keeps track of the distinct routing domains traversed by a route within the same VPN or extended VRFs, as well as across different VPNs or VRFs.
· Detects when a route loops back to a VRF in a domain where it has already traversed (typically at a border leaf switch that is the stitching point between domains, but also at an internal switch, in some cases).
· Prevents the route from getting imported or accepted when it would lead to a loop.
The following components are used with the BGP Domain-Path feature for loop prevention:
· Routing domain ID: Every tenant VRF in an ACI site is associated with one internal fabric domain, one domain for each VRF in each SR-MPLS infra L3Out, and one domain for each IP L3Out.
· Domain path: The domain segments traversed by a route are tracked using a BGP domain path attribute:
1)The domain ID of the VRF for the source domain where the route is received is prepended to the domain path
2)The source domain ID is prepended to the domain path while re-originating a route across domains on the border leaf switches
3)An external route is not accepted if any of the local domain IDs for the VRFs is in the domain path
4)The domain path is carried as an optional and transitive BGP path attribute with each domain segment, represented as <Domain-ID:SAFI>
5)The ACI border leaf switches prepend the VRF internal domain ID for both locally originated and external routes to track leaks within the domain
6)A route from the internal domain can be imported and installed in a VRF on a node with a conflicting external domain ID to provide an internal backup or transit path
7)For infra L3Out peers, the advertisement of a route to a peer is skipped if the domain ID of the peer domain is present in the domain path of the route (outbound check is not applicable for IP L3Out peers)
8)The border leaf switches and non-border leaf switches will both process the domain path attribute
Domain Path Loop Protection Example
For a better understanding of Domain Path loop protection related concepts and mechanism, let us take the following scenarios as an example to explain.
When the BGP Domain-Path feature is enabled, each of these domains is assigned a unique routing domain ID:
VRF V1 is associated with one internal fabric domain {V1-Internal}
VRF V2 is associated with one internal fabric domain {V2-Internal}
BGP IP L3out1 for VRF V1 is associated with domain {V1-IPL1}
Infra L3out1 for VRF V1 is associated with domain {V1-IL1}
Infra L3out2 for VRF V1 is associated with domain {V1-IL2}
Infra L3out3 for VRF V2 is associated with domain {V2-IL3}
Each domain segment is represented by <DOMAIN-ID:SAFI>
SAFI Value SAFI type
0 Local Route
1 BGP IP
70 EVPN
128 IPVPN
1. Prefix P1 is received from a tenant IP L3out1 for VRF V1{Domain id: V1-IPL1} on BL1.
2. P1 is advertised from the BL1 into the fabric via MPBGP VPN while prepending first, the source domain-id {V1-IPL1} and then the VRF internal domain id {V1-internal}. On BL2, {V1-Internal:128, V1-IPL1:1} is in P1’s Domain Path attribute.
3. P1 is advertised via Infra L3out1 to peer PE1 with {V1-Internal:128, V1-IPL1:1} in the Domain Path attribute. No infra L3out1 VRF V1 domain id prepend here.
4. P1 is advertised between PEs preserving the Domain Path attribute due to its optional and transitive BGP path attribute even if it is unknow to PEs.
5. On BL3, the looped route P1 is blocked when it is received from PE2 via Infra L3out2 VRF V1 due to the conflicting local domain id {V1-Internal} in Domain Path. On BL4, P1 received from PE3 via Infra L3out3 is imported into fabric in VRF V2 as there is no local domain id conflict in Domain Path.
6. P1 is advertised from BL4 into spine via MPBGP VPN while prepending Infra L3out3 VRF V2 domain-id {V2-IL3} and then the VRF internal domain id {V2-internal}. Finally on spine, { V2-Internal:128, V2-IL3:70, V1-Internal:128, V1-IPL1:1 } is in P1’s Domain Path attribute.
7. The looped route P1 via external interworking MPLS Core is not able imported into V1 on any TOR due to the conflicting Internal Domain Id {V1-Internal}. P1 announced from BL1 via spine MPBGP VPN to other border or non-border leaf is still working as normal, only external looped route P1 is blocked.
Within an ACI fabric, the VRF scope is global and is extended to all switches where it is configured. Therefore, a route that is exported out of a domain in a VRF is blocked from being received back into the VRF on any other switch. This is the expected loop protection behaviour of BGP Domain Path feature in Cisco ACI fabric interworking solution.
Summary
As one of the lead innovators in modern data center fabric and interworking solutions, Cisco, continually streamed new features and functionalities like BGP Domain-Path to further increase the stability of Cisco Data Center fabrics, which is extremely useful to help build and maintain a more stable and resilient loop-free large scale distributed data center network architecture across geographic locations.
I hope you found this post useful to understand how with this latest feature that can fit your needs!
For more information, read the Cisco APIC Layer 3 Networking Configuration Guide, Release 5.1(x)
