MPLS and BGP Multipath, Prefix Independent Convergence (PIC) and Best External

The aim of this post is to explain in detail some key Border Gateway Protocol (BGP) features in a Multiprotocol Label Switching (MPLS) network topology.

Contents

• Network Diagrams
  • Physical Topology
  • IP Addressing and Customer IGP/BGP Topology
  • MPLS, BGP VPNv4, VRF and Redistribution Topology
• BGP Multipath – BGP VPNv4 Full Mesh
  • BGP Multipath – iBGP
  • BGP Multipath – eiBGP
• BGP Multipath – BGP VPNv4 Route Reflector (RR)
  • BGP Multipath – iBGP
• BGP Prefix Independent Convergence (PIC)
• BGP Best Path External
• APPENDIX

Introduction

The configuration options covered in this document are mainly related to two BGP features:

BGP Multipath Load Sharing for EBGP and iBGP which allows the configuration of multipath load balancing with both external BGP (EBGP) and internal BGP (iBGP) paths in BGP networks that are configured to use MPLS Virtual Private Networks (MPLS VPNs). This feature provides improved load balancing deployment and service offering capabilities and it is useful for multi-homed autonomous systems and Provider Edge (PE) routers that import both EBGP and iBGP paths from multihomed and stub networks.

BGP PIC Edge for IP and MPLS VPN feature improves BGP convergence after a network failure. This convergence is applicable to both core and edge failures and can be used in both IP and MPLS networks. This feature creates and stores a backup/alternative path in the Routing Information Base (RIB), Forwarding Information Base (FIB), and Cisco Express Forwarding (CEF) so that when a failure is detected, the backup/alternative path can immediately take over, thus enabling fast failover.

Notice that this post assumes you understand basic MPLS, Label Distribution Protocol (LDP), Virtual Routing and Forwarding (VRF) and PE to Customer Edge (CE) routing configuration.

Network Diagrams

Physical Topology

Figure 1

IP Addressing and Customer IGP/BGP Topology

Figure 2

MPLS, BGP VPNv4, VRF and Redistribution Topology

Figure 3

BGP Multipath – BGP VPNv4 Full Mesh

In this section we are going to analyse step by step the advertisement of BGP VPNv4 prefixes between PEs and the requirements needed in order to achieve load sharing or multipathing when there are multiple paths available for the same final destination.

Figure 4 shows the first network design we are going to use for our study; a Full Mesh iBGP VPNv4 between PE routers (R1 to R6).

Figure 4

R1 – R6:

Customer A and Customer B have two sites: Site A which is dual-homed to a MPLS Service Provider (SP) backbone and Site B that is single-homed to the same MPLS SP. The purpose of this network design is to guarantee connectivity between Customer Sites across the MPLS core network.

Now we are going to focus exactly on how R15, which is located at Customer A Site B, routes traffic to R11’s Loopback0 (172.16.1.11/32), which is located at Customer A Site A.

R1 (PE) installs the OSPF route in the Routing Table for VRF CA that is advertised from R11 (CE).

R2 (PE) has also the OSPF route in the Routing Table for VRF CA to the destination 172.16.1.11/32 from R12 (CE).

R1 and R2, the PEs, perform mutual redistribution between the Customer IGP (OSPF) and BGP VPNv4.

R1:

R2:

Therefore the OSPF routes to 172.16.1.11/32 also appear as BGP VPNv4 routes and can be advertised to the rest of BGP VPNv4 neighbours. In our current topology there is a full mesh between iBGP VPNv4 peers in the MPLS core, so each peer will receive the advertisement from R1 and R2.

A BGP VPNv4 route is composed of the Router Distinguisher (RD) plus the prefix and the length. So the BGP VPNv4 prefix for the IP Prefix 172.16.1.11/32 is 1:2:172.16.1.11/32. The RD is configured under the VRF of R1 and R2 and in this specific scenario they share the same RD for VRF CA (1:2).

R1 and R2:

Because this is a dual-homed design, R1 has two BGP VPNv4 routes for 1:2:172.16.1.11/32, one locally originated and with the next hop value of 10.1.111.11 (R11’s Fa0/0) and one from R2, the other PE serving the same Customer Site, with next hop 2.2.2.2 (R2’s Loopback).

R1 best route for 1:2:172.16.1.11/32 is the one that R1 locally originates (0.0.0.0) and with next-hop 10.1.111.11 (R11’s Fa0/0). Because R1 does the redistribution from OSPF into BGP VPNv4 is going to automatically assign a BGP weight value of 32768 and this wins the BGP Path Selection algorithm. 32768 > 0 (default weight for the BGP VPNv4 route advertised by R2).

The same applies to R2 for 1:2:172.16.1.11/32 BGP VPNv4 prefix. The route locally originated from R2 is preferred over the route advertised by R1 because of the high weight value.

From R5’s point of view (PE for Customer A Site B), we can see that receives both (R1 and R2) BGP VPNv4 routes for 1:2:172.16.1.11/32.

The BGP VPNv4 route preferred for the final destination is through R1. The BGP Best Selection Path is now based on lower Metric (MED). R1’s MED is 11 and is lower than R2’s MED which is 21.

As we can see in the output above, the best route is also imported as BGP VPNv4 route with RD 5:5 (5:5:172.16.1.11/32), the RD configured under VRF CA on R5. R5 is doing mutual redistribution between BGP VPNv4 and the Customer IGP (OSPF).

R5:

Notice that BGP default behaviour is choosing only one best path to the destination based on the BGP Best Path Selection Algorithm. If we do a traceroute to the destination from R5 we can verify that is taking the path through R1 and ignoring the alternative path through R2.

R5 advertises its best route through R1 to the customer. Finally we can verify that the route gets installed in R15’s routing table as an OSPF Inter Area (IA) route. A traceroute command shows end to end reachability.

BGP Multipath – iBGP

The option of having multiple paths to the destination and use only the best path might not be the desirable choice. Instead using some or all of the alternative paths for load sharing might be more efficient.

Figure 5 shows the idea of installing both routes, through R1 and R2, for forwarding.

Figure 5

In order to configure BGP Multipath, the first 9 points of the BGP Best Selection Path algorithm must be identical for all the routes available.

1. Prefer the path with the highest WEIGHT.
2. Prefer the path with the highest LOCAL_PREF.
3. Prefer the path that was locally originated via a network or aggregate BGP subcommand or through redistribution from an IGP.
4. Prefer the path with the shortest AS_PATH.
5. Prefer the path with the lowest origin type.
6. Prefer the path with the lowest multi-exit discriminator (MED).
7. Prefer eBGP over iBGP paths.
   1. If bestpath is selected, go to Step 9 (multipath).
8. Prefer the path with the lowest IGP metric to the BGP next hop.
   1. Continue, even if bestpath is already selected.
9. Determine if multiple paths require installation in the routing table for BGP Multipath.

As we saw and following with our example, R5 has two iBGP VPNv4 routes to 1:2:172.16.1.11/32. However, only the best route (advertised by R1) is imported into the VRF CA as iBGP route. The alternative path gets discarded and it is not used for sending traffic. In order to use both routes for forwarding traffic and we enable BGP Multipath feature.

Then we configure BGP Multipath on R5. Because R5 is who has two possible iBGP VPNv4 routes to the destination.

NOTE: In this specific version the import option is required to be able to install redundant paths that can be configured as backup multipaths for a VRF instance. This keyword can be configured only under a VRF in address family configuration mode.

However, nothing has changed so far. R5 still prefers the route through R1 and it is not using the alternative path advertised by R2.

Now it is time to compare the BGP attributes in order to comply with the BGP requirements for Multipath.

1. Prefer the path with the highest WEIGHT: 0 for both routes.

2. Prefer the path with the highest LOCAL_PREF: 100 for both routes.

3. Prefer the path that was locally originated via a network or aggregate BGP subcommand or through redistribution from an IGP: neither path is locally originated (sourced attribute does not appear).

4. Prefer the path with the shortest AS_PATH: iBGP paths originated in the same AS (Local).

5. Prefer the path with the lowest origin type: incomplete because the routes were redistributed.

6. Prefer the path with the lowest multi-exit discriminator (MED): MEDs are different. R1 reports a MED value of 11 to R5 and R2 reports a MED value of 21. In order to achieve BGP Multipath these values must be equal.

7. Prefer eBGP over iBGP paths: both paths are iBGP (internal).

8. Prefer the path with the lowest IGP metric to the BGP next hop: both routes have the same OSPF cost to the next-hop (32).

9. Determine if multiple paths require installation in the routing table for BGP Multipath: There are two iBGP VPNv4 paths for the final destination 172.16.1.11/32.

Once all the BGP attributes required for Multipath are equal, we can check if multipath has been enabled.

Notice that BGP Multipath has been configured under the BGP address-family IPV4 unicast vrf CA on R5; this is why we see the multipath attributes under the BGP VPNv4 routes with RD 5:5  (R5’s RD). Finally the routes are installed in the Routing Table for R5’s VRF CA.

If we do a traceroute to the final destination we can see that load sharing has been enabled.

R15 sees its normal IA OSPF route to the destination with next-hop R5’s Fa0/1 IP address.

BGP Multipath – eiBGP

Figure 6 illustrates the routing decision to R14’s loopback, 172.16.1.14/32.

Figure 6

Customer B Site A is running EBGP between PEs (AS 200) and CEs (AS 346). R13 and R14 are iBGP neighbours (AS 346) and are also EBGP peers with R3 and R14 (AS 200).

We study now how R4 routes traffic to 172.16.1.14/32 (R14’s loopback). As you can see in the output below, R4 prefers its EBGP route towards the destination (external) although it also receives an alternative iBGP route from R3.

R4#show bgp vpnv4 unicast all 172.16.1.14
BGP routing table entry for 3:4:172.16.1.14/32, version 20
Paths: (2 available, best #1, table CB)
  Advertised to update-groups:
     1
  346
    10.1.144.14 from 10.1.144.14 (14.14.14.14)
      Origin IGP, metric 0, localpref 100, valid, external, best
      Extended Community: RT:346:346
      mpls labels in/out 419/nolabel
  346
    3.3.3.3 (metric 21) from 3.3.3.3 (3.3.3.3)
      Origin IGP, metric 0, localpref 100, valid, internal
      Extended Community: RT:346:346
      mpls labels in/out 419/318

Therefore having R4 able to see both routes we proceed to configure Multipath. Notice this time one route is EBGP (best) and the other route is iBGP so we need to use eibgp option in order to enable load sharing.

R4(config)#router bgp 200
R4(config-router)#address-family ipv4 vrf CB
R4(config-router-af)#maximum-paths eibgp 2 import 2

Next step is verifying the Multipath requirements specified by BGP algorithm.

1. Prefer the path with the highest WEIGHT: both routes have the default value of 0.

2. Prefer the path with the highest LOCAL_PREF: both routes have the default value of 100.

3. Prefer the path that was locally originated via a network or aggregate BGP subcommand or through redistribution from an IGP: none of them is locally originated.

4. Prefer the path with the shortest AS_PATH: both routes have the same value, 346.

5. Prefer the path with the lowest origin type: both routes have IGP origin type.

6. Prefer the path with the lowest multi-exit discriminator (MED): both routes have a metric of 0.

7. Prefer eBGP over iBGP paths: the route through R14 is preferred as it is an EBGP route. Route through R3 is an iBGP route. However, the algorithm says: If bestpath is selected, go to Step 9 (multipath).

8. Prefer the path with the lowest IGP metric to the BGP next hop: this step is bypassed by the algorithm, so having different IGP metrics is not going to affect eibgp Multipath.

R4#show ip route vrf CB 10.1.144.14
Routing entry for 10.1.144.0/24
  Known via “connected”, distance 0, metric 0 (connected, via interface)
  Routing Descriptor Blocks:
  * directly connected, via FastEthernet0/1
      Route metric is 0, traffic share count is 1

R4#show ip route 3.3.3.3
Routing entry for 3.3.3.3/32
  Known via “ospf 1”, distance 110, metric 21, type intra area
  Last update from 10.1.48.8 on FastEthernet0/0, 00:18:33 ago
  Routing Descriptor Blocks:
  * 10.1.48.8, from 3.3.3.3, 00:18:33 ago, via FastEthernet0/0
      Route metric is 21, traffic share count is 1

9. Determine if multiple paths require installation in the routing table for BGP Multipath: we have two possible paths for eibgp Multipath through R14 and through R3.

R4#show bgp vpnv4 unicast all 172.16.1.14
BGP routing table entry for 3:4:172.16.1.14/32, version 34
Paths: (2 available, best #1, table CB)
Multipath: eiBGP
Flag: 0x820
  Advertised to update-groups:
     1
  346
    10.1.144.14 from 10.1.144.14 (14.14.14.14)
      Origin IGP, metric 0, localpref 100, valid, external, multipath, best
      Extended Community: RT:346:346
      mpls labels in/out 420/nolabel
  346
    3.3.3.3 (metric 21) from 3.3.3.3 (3.3.3.3)
      Origin IGP, metric 0, localpref 100, valid, internal, multipath
      Extended Community: RT:346:346
      mpls labels in/out 420/318

R4#show ip route vrf CB
Gateway of last resort is not set
     172.16.0.0/32 is subnetted, 3 subnets
B       172.16.1.16 [200/0] via 6.6.6.6, 00:15:31
B       172.16.1.13 [20/0] via 10.1.144.14, 00:11:22
                    [20/0] via 3.3.3.3, 00:11:22
B       172.16.1.14 [20/0] via 10.1.144.14, 00:11:22
                    [20/0] via 3.3.3.3, 00:11:22
     10.0.0.0/24 is subnetted, 1 subnets
C       10.1.144.0 is directly connected, FastEthernet0/1

BGP Multipath – BGP VPNv4 Route Reflector (RR)

Figure 7

In this topology R10 is configured as BGP VPNv4 Route Reflector (RR). BGP RRs avoid the need of a full mesh between BGP peers but hide topology information in return. In this scenario the PEs peer with R10 and send their BGP VPNv4 routes, then R10 runs BGP Best Path Selection algorithm and advertises by default only its best BGP VPNv4 route for each destination to each PEs.

R1 – R6:

R10:

R10, the Route Reflector (RR), receives all the BGP VPNv4 Updates from the PE routers with all the information about the customer routes. If we focus again on R11’s Loopback, 172.16.1.11/32, we can see that R10 receives the two available paths through R1 and R2 and R1, as we saw, wins the BGP Best Path Selection algorithm due to a lower metric (11<21).

Figure 8

If we check now what routes R5 receives from the RR, we can verify in the output below that only has one route to get to the final destination and this route is through R1 which is the best route of R10 for 1:2:172.16.1.11/32. This route is then imported to VRF CA thanks to redistribution and advertised to R15.

Why R10, the RR, is not advertising both routes to R5? Because by default BGP VPNv4 RR only chooses one best path for the same VPNv4 prefix: RD:Prefix/Length. In our topology, this means that R10 is receiving 1:2:172.16.1.11/32 from R1 and R2 and R10 is not able to differentiate this VPNv4 prefix (it is the same prefix coming from two different PE neighbors). Therefore, it is only able to choose one as the best path.

There are two solutions to this possible scenario:

• Having a BGP VPNv4 Full Mesh between all PEs as I explained in previous sections.
• Making the BGP VPNv4 routes look different changing the RD.
  • R1: RD: 1:2
  • R2: RD: 2:2

BGP Multipath – iBGP

Figure 9

In order to be able to do multipathing on R5, first of all we need to be sure that R5 is receiving both routes. So I decided to implement the change of RD: change R2’s RD to 2:2.

Now R10 sees the routes as different BGP VPNv4 prefixes, 1:2:172.16.1.11/32 and 2:2:172.16.1.11/32, and can run BGP Best Selection Path algorithm independently resulting in both routes being winners of the BGP Best Selection Path algorithm.

Then we check on R5 whether or not it sees both routes reflected by R10. R5 is able to install both BGP VPNv4 routes in its BGP table but only one is selected as best route (through R1 because of the lower metric).

Up to this point R5 can see both routes, so we can enable Multipath on R5. Remember that, as we saw before, in this scenario we need to have the same MED for this to actually work.

Finally, R5 has BGP Multipath enabled as we can check on the output below.

We can actually confirm the traffic path based on the CEF table information and running a traceroute command.

BGP Prefix Independent Convergence (PIC)

BGP PIC is similar to Fast Reroute for EIGRP or OSPF. This feature creates and stores a backup path in the RIB, FIB and CEF so when a failure is detected, the backup path can immediately take over, thus enabling fast failover.

Figure 10

In order to enable PIC, first of all, we need to have the primary and the backup route in the BGP table. In this example the goal is to configure R1 to enable BGP PIC for the prefix 172.16.1.11/32 (R11’s loopback).

Because R1 and R2 have two different RDs configured under the VRF, R10 is able to reflect both BGP VPNv4 routes to the PEs.

We enable BGP PIC on R1.

The best route and the backup/repair route go into the RIB and the FIB and the best route is used, as normal, for forwarding.

However, having only a repair route is not enough to improve BGP Convergence Time. We still need to run Bidirectional Forwarding Detection (BFD) for faster failure detection in order to switch to the repair/backup path. This way, we do not have to wait for the withdraw/update messages before electing the next backup path.

BGP Best Path External

In this section, we try to achieve the same result that we saw in the previous section configuring BGP PIC but using the same RD. In addition, we are going to modify BGP Local Preference (LP) attribute to show the Best Path External feature.

Figure 11 shows the new scenario.

Figure 11

R10 receives the BGP VPNv4 prefix about R14’s loopback (3:4:172.16.1.14/32) from R3 and R4.

The route advertised by R3 is preferred because of the lowest Router ID value (3.3.3.3 < 4.4.4.4) and it is advertised to R4.

However, R4 still prefers its EBGP route to the final destination through R14.

Now we change the BGP VPNv4 LP to a high value on R3 for 3:4:172.16.1.14/32.

Figure 12

If we check R10’s BGP VPNv4 table, the RR only sees the path through R3. The reason is R10 chooses R3 as the best next-hop for the destination and advertises it to R4. R4 runs BGP Best Path Selection algorithm and also prefers the route through R3 because of the higher LP (200>100) and R4 stops advertising its path to the RR.

Now R4 chooses to go through the MPLS network instead of using its directly connected link to R14. R4 losing its alternative EBGP path via R14 might not be desirable to speed convergence. For this specific scenario we use BGP best external feature.

If we check the BGP VPNv4 table on R4 we can see the status “x” which shows that best external is enabled.

Now the alternate path can be installed as a backup/repair path on R4 and can also be advertised to the RR.

R10 knows about both prefixes now. However, R10 is only advertising the primary path or best path to R3. So at this point nothing has changed for R3 yet as the best route is through R3.

At this point remember that the final objective is trying to achieve what we did previously configuring PIC with different RDs: being able to install both routes into RIB and FIB for fast failover. These are the steps we need to follow:

• Run bgp additional-paths select backup command to trigger the calculation of the backup path at R10, which is the path via R4.
• Run bgp additional-paths install command to install the backup path in the forwarding plane. (Do not configure this command if RR is in the control plane).
• Run neighbor advertise diverse-path backup command to trigger the advertisement of the backup path to R3. R3 will learn the best path, (which is the path via R13 because of the higher LP), and it will learn the backup path from R10 (RR).

We can verify that R10 is advertising the additional path through R4 to R3.

We check R3’S BGP VPNv4 table and we can see the backup route has been advertised and installed as a repair path.

References

https://www.cisco.com/c/en/us/td/docs/ios/12_2sx/feature/guide/fsxeibmp.html
https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/xe-3s/irg-xe-3s-book/irg-bgp-mp-pic.html
https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/15-s/irg-15-s-book/irg_diverse_path.html
https://www.cisco.com/c/en/us/td/docs/ios/iproute_bgp/command/reference/irg_book/irg_bgp1.html

Appendix