Tactical TE Issues
Last updated
Last updated
Load tactical.te.issues.cfg
All links have an IGP cost of 10, except the link CSR6-XR12 which has a cost of 1. The CSR9-XR12, CSR10-XR11, and CSR6-CSR10 links are shut down.
Imagine that CSR8, CSR1, CSR9 and XR11 are all PEs. The network operator uses “tactical” TE, in which the network primarily runs on the IGP, but when needed the operator can deploy TE tunnels to steer traffic around problematic links.
Currently, the CSR6-XR12 link is overloaded. The operator wants to shift some of this traffic along the CSR6-CSR9-CSR10 path. He deploys a bidirectional TE tunnel taking this explicit path. But now this “bypass” path is overloaded too. All traffic between CSR1/CSR8/CSR9 and XR11 is taking this path.
Additionally, there is suboptimal routing. For example, CSR9 takes the path: CSR9-CSR6-CSR9-CSR10-XR12-XR11 to get to XR11, which doesn’t really make sense from a path latency standpoint. Explain these issues and find a way to solve them.
Employing ad-hoc TE tunnels can be cumbersome and introduce problems. First, the TE tunnel will take all traffic. The router only load balances between TE paths and IGP paths if the TE tailend is not present in the IGP path. Second, P-P TE tunnels can introduce hairpining and add latency.
The first problem is that the operator wants to load share across the two paths. The primary link (CSR6-XR12) is operational, but under high load. When the operator creates a TE tunnel using the “bypass route,” all traffic is shifted over this path, and no traffic uses the primary link at all. This is because a TE tunnel cannot load share with an IGP path if the tailend is in the IGP path.
For example, let’s look at CSR6’s two paths to XR11:
via TE tunnel to XR12, then XR12-XR11
via IGP, using the direct link to XR12, then XR12-XR11
The IGP path goes through the TE tunnel tailend (XR12), so it cannot be a candidate for load sharing.
One solution is to create multiple TE tunnels. For example, we can create a second TE tunnel direct to XR12. Now CSR6 will load share among the two paths. This balances traffic between the two paths, which was the operator’s goal. We’ll make sure to do this bidirectionally.
Now both TE tunnels are used to get to PEs such as XR11 and CSR9:
If we traceroute between PEs, we see that traffic is shared among the two TE tunnels:
Strangely, CSR6 does not appear to be load sharing correctly. Every traceroute always goes through Tun0. This seems to be a bug?
While we’ve solved the problem of using both paths, we still can run into non-optimal paths. For example, CSR9 still sees traffic hairpin back through itself. There is a 50% chance this happens since we have two ECMP TE tunnel paths.
To have better control over this, we need to add more TE tunnels. The more PE-PE TE tunnels we have, the more exact control we have over paths. This is a direct tradeoff: adding more TE tunnels results in optimization, but at the cost of additional complexity and additional state.
We can still balance the traffic and avoid non-optimal paths by configuring two bidirectional TE tunnels: CSR9 to XR11 which avoids the CSR6-XR12 link, and CSR8 to XR11 which uses the CSR6-XR12 link.
Now traffic between CSR9 and XR11 is optimal and avoiding the link:
For load sharing purposes, traffic between CSR8 and XR11 uses the primary link:
Note that this lab is slightly unrealistic, because the best option would probably be to just tune the IGP metric of the CSR6-XR12 link so that ECMP paths happen within the core. If you raise the cost to a value such as 15, then CSR8 traffic will go direct to XR11, and CSR9 traffic will bypass the primary link. By playing around with the metric you should be able to shift some traffic off the link and never worry about the hairpining that TE tunnels can cause.
However, the point of this lab was to demonstrate the potential issues with using ad-hoc P-to-P TE tunnels to try to engineer temporary paths. It can quickly get more complicated than you might think. The biggest issue is hairpining which can cause additional latency and uses additional bandwidth needlessly. The solution to this is to deploy more PE-PE TE tunnels to have exact control over the end-to-end path, but this comes at the cost of additional complexity.
Note that using “strategic” TE, in which a full mesh of TE tunnels is used, does not have this problem. If you are using a full mesh of TE tunnels, then you have exact control over every end-to-end path.
Also notice how autoroute announce plays a role in this. You might at first glance think that, because you only deployed a CSR6-XR12 tunnel, it will only affect how CSR6 and XR12 route towards the PEs. Since CSR6 and XR12 are not themselves PEs, then maybe you’d think the TE tunnels are never actually used. The TE tunnel is not announced into the IGP (forwarding adjacency) after all, right?
However, this is wrong because other PEs upstream of the P routers can still use the P routers as their IGP shortest path. The P routers, CSR6 and CSR12, will install label swap operations for the incoming LDP label for destinations which use the TE tunnel, such as XR11, and swap to the TE tunnel. Essentially, even though you are not using forwarding adjacency, if the upstream routers have an IGP bestpath through the TE headend, then the TE tunnel will end up being used. Once traffic gets to the TE headend, it is free to make its own forwarding decision. Because the TE tunnel is in the RIB, this is what gets installed into the LFIB, not the IGP path.
I believe forwarding adjacency only makes a difference if the TE tunnel has a lower IGP cost than the IGP path between the two P routers. (Either way, PE routers see an IGP cost of 1 between CSR6 and XR12 in our lab).
This lab idea came from Ch.9 of the book Traffic Engineering with MPLS by Eric Osborne and Ajay Simha.
