SR BGP Data Center (iBGP)

Load sr.data.center.ibgp.init.cfg

#R1-R6, CE101, CE102 only
configure
load bootflash:sr.data.center.ibgp.init.cfg
commit replace
y

A VRF is setup on R1 and R2. Configure an underlay data center fabric using only iBGP with SR MPLS. Each router should belong to ASN 65000. Enable multipathing so R1 and R2 have ECMP paths between each other.

Answer

#All routers
segment-routing
 global-block 16000 23999
!
route-policy SET_PREFIX_SID($SID)
 set label-index $SID
end-policy

#R1
router static add ipv4 unicast
 10.1.3.3/32 gi0/0/0/3
!
router bgp 65000
 mpls activate
  int gi0/0/0/3
 add ipv4 uni
  maximum-paths ibgp 4
  allocate-label all
  network 1.1.1.1/32 route-policy SET_PREFIX_SID(1)
 !
 neighbor 10.1.3.3
  remote-as 65000
  add ipv4 labeled-unicast

#R2
router static add ipv4 unicast
 10.2.4.4/32 gi0/0/0/4
!
router bgp 65000
 mpls activate
  int gi0/0/0/4
 add ipv4 uni
  maximum-paths ibgp 4
  allocate-label all
  network 2.2.2.1/32 route-policy SET_PREFIX_SID(2)
 !
 neighbor 10.2.4.4
  remote-as 65000
  add ipv4 labeled-unicast

#R3
router static add ipv4 unicast
 10.1.3.1/32 gi0/0/0/1
 10.3.5.5/32 gi0/0/0/5
 10.3.6.6/32 gi0/0/0/6
!
router bgp 65000
 ibgp policy out enforce-modifications
 mpls activate
  int gi0/0/0/1
  int gi0/0/0/5
  int gi0/0/0/6
 add ipv4 uni
  maximum-paths ibgp 4
  allocate-label all
  network 3.3.3.1/32 route-policy SET_PREFIX_SID(3)
 !
 neighbor 10.1.3.1
  remote-as 65000
  add ipv4 labeled-unicast
   route-reflector-client
   next-hop-self
 neighbor 10.3.5.5
  remote-as 65000
  add ipv4 labeled-unicast
   next-hop-self
 neighbor 10.3.6.6
  remote-as 65000
  add ipv4 labeled-unicast
   next-hop-self

#R4
router static add ipv4 unicast
 10.2.4.2/32 gi0/0/0/2
 10.4.5.5/32 gi0/0/0/5
 10.4.6.6/32 gi0/0/0/6
!
router bgp 65000
 mpls activate
  int gi0/0/0/2
  int gi0/0/0/5
  int gi0/0/0/6
 ibgp policy out enforce-modifications
 add ipv4 uni
  maximum-paths ibgp 4
  allocate-label all
  network 4.4.4.1/32 route-policy SET_PREFIX_SID(4)
 !
 neighbor 10.2.4.2
  remote-as 65000
  add ipv4 labeled-unicast
   route-reflector-client
   next-hop-self
 neighbor 10.4.5.5
  remote-as 65000
  add ipv4 labeled-unicast
   next-hop-self
 neighbor 10.4.6.6
  remote-as 65000
  add ipv4 labeled-unicast
   next-hop-self

#R5
router static add ipv4 unicast
 10.3.5.3/32 gi0/0/0/3
 10.4.5.4/32 gi0/0/0/4
!
router bgp 65000
 mpls activate
  int gi0/0/0/3
  int gi0/0/0/4
 ibgp policy out enforce-modifications
 add ipv4 uni
  maximum-paths ibgp 4
  allocate-label all
  network 5.5.5.1/32 route-policy SET_PREFIX_SID(5)
 !
 neighbor 10.4.5.4
  remote-as 65000
  add ipv4 labeled-unicast
   route-reflector-client
   next-hop-self
 neighbor 10.3.5.3
  remote-as 65000
  add ipv4 labeled-unicast
   route-reflector-client
   next-hop-self

#R6
router static add ipv4 unicast
 10.3.6.3/32 gi0/0/0/3
 10.4.6.4/32 gi0/0/0/4
!
router bgp 65000
 ibgp policy out enforce-modifications
 mpls activate
  int gi0/0/0/3
  int gi0/0/0/4
 add ipv4 uni
  maximum-paths ibgp 4
  allocate-label all
  network 6.6.6.1/32 route-policy SET_PREFIX_SID(6)
 !
 neighbor 10.4.6.4
  remote-as 65000
  add ipv4 labeled-unicast
   route-reflector-client
   next-hop-self
 neighbor 10.3.6.3
  remote-as 65000
  add ipv4 labeled-unicast
   route-reflector-client
   next-hop-self

Explanation

Using iBGP for the data center fabric requires a little more work to achieve eBGP hop-by-hop behavior.

First, we need to activate MPLS on the interfaces explicitly under BGP. When using eBGP with IPv4/LU, this happens automatically. But when using iBGP, we must explicitly define the interfaces. Note that this command is only used in this very specific instance - when using iBGP-LU without an IGP.

router bgp 65000
 mpls activate
  int gi0/0/0/3

Next, each spine layer must act as RR in order to reflect iBGP learned routes to other iBGP neighbors. Additionally, the spine layer must set next-hop-self on these routes. To do so, we must use the special command ibgp policy out enforce-modificiations.

#R6
router bgp 65000
 ibgp policy out enforce-modifications
 !
 neighbor 10.4.6.4
  remote-as 65000
  add ipv4 labeled-unicast
   route-reflector-client
   next-hop-self

Verification

On each router, each prefix should have a nexthop of the directly connected neighbor:

All routers should allocate a label of 16000 + index value, giving the feel of a global label space:

Additionally, R3 and R4 should have ECMP routes to the PEs (R1 and R2).

As before, an end-to-end LSP is established between R1 and R2 which uses ECMP:

Summary

Using iBGP for the data center is similar to eBGP, but some modifications must be made to give the feel of eBGP:

• All routes must be advertised hop-by-hop with the nexthop updated at each hop.

  • This requires next-hop-self plus ibgp policy out enforce-modifications

  • This requires route reflection so that iBGP learned routes are reflected to other iBGP neighbors

• The interfaces must be activated for MPLS under BGP

Loop prevention is now achieved using the cluster list instead of ASN.