Working through OSPF and MPLS recently I came across an interesting concept. I built out a simple MPLS cloud and had one customer joining two sites across it. The goal was to connect it over MPLS and leverage OSPF for the PE to CE connection. This was quite easy to do and very simple to manage. Proceeding to add a private link between the branch and HQ sites did I realise that OSPF no longer leveraged the MPLS cloud but used the private link.
This is the topology currently. We have two sites connecting to each other via an MPLS cloud.
Here is the routing table as it stands. I have connection from the loopbacks on C1 to the loopbacks on C2.
C1(config-if)#do ping 5.5.5.5 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 5.5.5.5, timeout is 2 seconds: !!!!!
A trace route verifies the path taken.
C1#traceroute 5.5.5.5 Type escape sequence to abort. Tracing the route to 5.5.5.5 1 192.168.14.1 12 msec 20 msec 24 msec 2 192.168.12.2 [MPLS: Labels 18/20 Exp 0] 72 msec 88 msec 72 msec 3 192.168.35.3 [MPLS: Label 20 Exp 0] 84 msec 56 msec 76 msec 4 192.168.35.5 68 msec 84 msec 80 msec
The routing table as it currently stands shows OSPF advertising loopbacks from the Customer sites via the MPLS cloud.
C2(config)#do sh ip route Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route Gateway of last resort is not set O IA 192.168.14.0/24 [110/11] via 192.168.35.3, 01:18:57, Ethernet0/0 C 192.168.45.0/24 is directly connected, Ethernet1/0 4.0.0.0/32 is subnetted, 1 subnets O IA 4.4.4.4 [110/21] via 192.168.35.3, 01:18:57, Ethernet0/0 5.0.0.0/24 is subnetted, 1 subnets C 5.5.5.0 is directly connected, Loopback0 C 192.168.35.0/24 is directly connected, Ethernet0/0
Note that customer routers receive information from Ethernet0/0 – the upward link to the ISP for the Customer device.
C1(config-if)#do sh ip ospf data
OSPF Router with ID (192.168.14.4) (Process ID 2)
Router Link States (Area 0)
Link ID ADV Router Age Seq# Checksum Link count
192.168.14.1 192.168.14.1 811 0x80000005 0x00EA53 1
192.168.14.4 192.168.14.4 882 0x80000008 0x00DF34 2
Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
192.168.14.4 192.168.14.4 882 0x80000003 0x00C686
Summary Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
5.5.5.5 192.168.14.1 811 0x80000003 0x001112
192.168.35.0 192.168.14.1 811 0x80000003 0x005F55
The OSPF database shows that the other customer site are inter-area router. The type 3 inter-area LSAs show network 5.5.5.5 and the uplink to the ISP, 192.168.35.0. These are being advertised through the MPLS cloud and redistributed from MP-BGP SKY address family into OSPF vrf SKY.
Now what happens when we enable the backdoor connection between customer sites? The traffic takes the backdoor link and not the MPLS cloud.
We can confirm that the backdoor link is routing all traffic by checking the OSPF route table.
C1(config-if)#do sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
C 192.168.14.0/24 is directly connected, Ethernet0/0
C 192.168.45.0/24 is directly connected, Ethernet1/0
4.0.0.0/24 is subnetted, 1 subnets
C 4.4.4.0 is directly connected, Loopback0
5.0.0.0/32 is subnetted, 1 subnets
O 5.5.5.5 [110/11] via 192.168.45.5, 00:00:01, Ethernet1/0
O 192.168.35.0/24 [110/20] via 192.168.45.5, 00:00:01, Ethernet1/0
We can definitely see now that 5.5.5.5 and 192.168.35.0 which were advertised previously by the MPLS cloud are now being preferred by the backdoor link. If an administrator is to adjust the interface level OSPF cost this would not affect the route.
C1(config-if)#do sh ip ospf data
OSPF Router with ID (192.168.14.4) (Process ID 2)
Router Link States (Area 0)
Link ID ADV Router Age Seq# Checksum Link count
5.5.5.5 5.5.5.5 19 0x80000009 0x00CB95 3
192.168.14.1 192.168.14.1 978 0x80000005 0x00EA53 1
192.168.14.4 192.168.14.4 18 0x8000000A 0x000AB8 3
192.168.35.3 192.168.35.3 1040 0x80000005 0x00459D 1
Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
192.168.14.4 192.168.14.4 1049 0x80000003 0x00C686
192.168.35.5 5.5.5.5 1048 0x80000003 0x006C82
192.168.45.5 5.5.5.5 19 0x80000001 0x00FDFC
Notice there are no longer any inter-area routes / type 3 summary LSAs? The 5.5.5.5 network is now a type 1 LSA along with 192.168.35.3. As expected a trace route results in:
C1#traceroute 5.5.5.5 Type escape sequence to abort. Tracing the route to 5.5.5.5 1 192.168.45.5 28 msec 24 msec 20 msec
An MPLS link is not preferred in OSPF when there is a back door because intra-area routes are preferred over external routes. Routes that are advertised across a MPLS/VPN that are imported and exported into BGP pass the route information with it. This means upon redistribution out of BGP into OSPF, routes retain their external route marking. Therefore they are marked as external routes and no longer preferred by OSPF. They are a type 5 external LSA. The backdoor link becomes favoured and subsequently used.
An OSPF sham-link can solve this problem. The OSPF sham link provides a logical link between two VRFs. It creates a link that makes the MPLS PE’s participating in the sham link appear as a point to point link within OSPF. These links are able to fool or trick routers in the OSPF domain that this is a better path thus preserving the LSAs as type 1 or type 3.
By using two loopbacks on the respective devices advertised into the BGP address family that corresponds with the customer VRF, OSPF can create a link that is more appealing. By using the command area <area-id> sham-link <source-address> <destination-addres> cost <cost> it is possible to build this link.
area 0 sham-link 33.33.33.33 11.11.11.11 cost 10 area 0 sham-link 11.11.11.11 33.33.33.33 cost 10
When an OSPF sham-link is set it builds a bridge between two VRF’s. By advertising a type 1 LSA (Router) across this link, the OSPF database sees this route and the routes advertised across this link as acceptable. This is due to no longer being an external route and becoming an intra-area route.
Lets verify what we are seeing.
The routing table indicates that we are learning the other sites routes via the MPLS cloud.
C1#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
C 192.168.14.0/24 is directly connected, Ethernet0/0
33.0.0.0/32 is subnetted, 1 subnets
O E2 33.33.33.33 [110/1] via 192.168.14.1, 00:01:08, Ethernet0/0
C 192.168.45.0/24 is directly connected, Ethernet1/0
4.0.0.0/24 is subnetted, 1 subnets
C 4.4.4.0 is directly connected, Loopback0
5.0.0.0/32 is subnetted, 1 subnets
O 5.5.5.5 [110/31] via 192.168.14.1, 00:01:08, Ethernet0/0
11.0.0.0/32 is subnetted, 1 subnets
O E2 11.11.11.11 [110/1] via 192.168.14.1, 00:01:09, Ethernet0/0
O 192.168.35.0/24 [110/30] via 192.168.14.1, 00:01:09, Ethernet0/0
The trace route shows the path we are expecting to see and no hairpin routing is occurring.
C1#traceroute 5.5.5.5 Type escape sequence to abort. Tracing the route to 5.5.5.5 1 192.168.14.1 24 msec 20 msec 24 msec 2 192.168.12.2 [MPLS: Labels 18/20 Exp 0] 84 msec 84 msec 72 msec 3 192.168.35.3 [MPLS: Label 20 Exp 0] 76 msec 68 msec 64 msec 4 192.168.35.5 76 msec 88 msec 72 msec
So lets now take a look at the Sham link adjusted OSPF database.
C1#sh ip ospf data
OSPF Router with ID (192.168.14.4) (Process ID 2)
Router Link States (Area 0)
Link ID ADV Router Age Seq# Checksum Link count
5.5.5.5 5.5.5.5 76 0x8000000C 0x00986B 3
192.168.14.1 192.168.14.1 89 0x80000006 0x006622 2
192.168.14.4 192.168.14.4 79 0x8000000E 0x00D48F 3
192.168.35.3 192.168.35.3 90 0x80000006 0x001B28 2
Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
192.168.14.4 192.168.14.4 1543 0x80000003 0x00C686
192.168.35.5 5.5.5.5 1543 0x80000003 0x006C82
192.168.45.5 5.5.5.5 368 0x80000003 0x00F9FE
Type-5 AS External Link States
Link ID ADV Router Age Seq# Checksum Tag
11.11.11.11 192.168.14.1 233 0x80000001 0x000736 3489660929
11.11.11.11 192.168.35.3 227 0x80000001 0x0067BE 3489660929
33.33.33.33 192.168.14.1 152 0x80000001 0x000FD5 3489660929
33.33.33.33 192.168.35.3 170 0x80000001 0x006F5E 3489660929
Notice that the Sham-links have been advertised through as a Type 5 external LSA link type. The networks from the other customer site are passed over this OSPF sham link as Type 1 router LSA.
When setting a sham link up it is important to set an lower cost than the backdoor link. This will allow traffic to pass through the MPLS cloud as the preferred link and upon failure the backdoor link can be used to maintain connectivity. It is important to remember if you have a backdoor link that you should enable
So a sham link does more than create a link between loopbacks. It alters and adjusts the behaviour of OSPF in the internal OSPF database. It is amazing how much there is to learn.
Additional reading : RFC 4577
Interesting post Anthony. thanks! Ps. don’t know if you’re aware – but in Chrome, the screen captures just show white bars. In IE11, they show the text with scroll bars under each line of text? kind of weird. Thanks!
Glad I’m not the only one.
hi, thanks for this article. Just a humble clarification: Routes traversing the mpls superbackbone won’t be injected as external (E1/2) but as inter-area (O IA) routes.
This doesn’t change a bit in the discussion, anyway. Great work.
mk