Ticket #08

Ticket #08

Network diagram

IGP diagram

BGP diagram


The problem:

R4 can't ping R5's Lo0 IP address. Configure R2 to fix the problem.

The solution:

First, lets try to ping to R5's Lo0 from R4.


On R4:

R4#ping 1.45.5.5

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.45.5.5, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Indeed, R4 can't ping R5's Lo0. Lets check if R4 has a route to R5's Lo0 IP address.

On R4:

R4#sh ip route 1.45.5.5
Routing entry for 1.45.5.5/32
  Known via "eigrp 4", distance 170, metric 2560002816, type external
  Redistributing via eigrp 4
  Last update from 1.45.24.2 on FastEthernet0/1, 00:09:02 ago
  Routing Descriptor Blocks:
  * 1.45.24.2, from 1.45.24.2, 00:09:02 ago, via FastEthernet0/1
      Route metric is 2560002816, traffic share count is 1
      Total delay is 110 microseconds, minimum bandwidth is 1 Kbit
      Reliability 1/255, minimum MTU 1 bytes
      Loading 1/255, Hops 1

R4 has a route to R5's Lo0. Its an external EIGRP route, coming from R2.


Lets check if R5 have a route back to R4's FastEthernet0/1 IP address.

On R5:

R5#sh ip route 1.45.24.4
Routing entry for 1.45.24.0/24
  Known via "rip", distance 120, metric 1
  Redistributing via rip
  Last update from 1.45.15.1 on FastEthernet0/0, 00:00:13 ago
  Routing Descriptor Blocks:
  * 1.45.15.1, from 1.45.15.1, 00:00:13 ago, via FastEthernet0/0
      Route metric is 1, traffic share count is 1

R5 do have a route back to R4 via R1. Its learned from R2 via RIP.


So R4 and R5 have a route to each other. Lets check and see if all the routers in the path between R4 and R5 have a route to R4 and to R5. Lets start from R1.

On R1:

R1#sh ip route 1.45.5.5
% Subnet not in table

How can R1 have no route to R5's Lo0 but R4 do have a route to R5's Lo0. Lets also check a route to R4.

On R1:

R1#sh ip route 1.45.24.4
% Subnet not in table

Now this is really strange, we know the R5 learned the route to R4 via RIP from R1. How can R2 have no route on its own?


Lets check if we happen to have VRFs on R1.

On R1:

R1#sh ip vrf 
  Name                             Default RD          Interfaces
  45                               1:45                Fa0/1

R1 do has a VRF setup, and the interface facing R5 is in a VRF named '45'. Lets check the routing table of VRF '45'.

On R1:

R1#sh ip route vrf 45 

Routing Table: 45
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     1.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
C       1.45.15.0/24 is directly connected, FastEthernet0/1
R       1.45.5.5/32 [120/1] via 1.45.15.5, 00:00:27, FastEthernet0/1
B       1.45.4.4/32 [200/156160] via 1.3.26.2, 00:11:48
B       1.45.24.0/24 [200/0] via 1.3.26.2, 00:11:48

We can now see the routes to both R5's Lo0 via RIP and to R4's F0/1 interface via BGP.


Since we see BGP, and we know from the diagrams that there is a BGP session between R1 and R2, we might have a IP MPLS VPN setup. Lets check it out.

On R1:

R1#sh ip bgp vpnv4 all
BGP table version is 75, local router ID is 1.3.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:45 (default for vrf 45)
*>i1.45.4.4/32      1.3.26.2            156160    100      0 ?
*> 1.45.5.5/32      1.45.15.5                1         32768 ?
*> 1.45.15.0/24     0.0.0.0                  0         32768 ?
*>i1.45.24.0/24     1.3.26.2                 0    100      0 ?
Route Distinguisher: 2:45
*>i1.45.4.4/32      1.3.26.2            156160    100      0 ?
*>i1.45.24.0/24     1.3.26.2                 0    100      0 ?

We can see both routes to R5's Lo0 and R4's F0/1 interface. Let check and see if R2 got a similar setup and routes.

On R2:

R2#sh ip vrf 
  Name                             Default RD          Interfaces
  45                               2:45                Fa0/0
R2#sh ip route vrf 45

Routing Table: 45
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     1.0.0.0/8 is variably subnetted, 4 subnets, 2 masks
B       1.45.15.0/24 [200/0] via 1.3.1.1, 00:02:07
B       1.45.5.5/32 [200/1] via 1.3.1.1, 00:02:07
D       1.45.4.4/32 [90/156160] via 1.45.24.4, 01:11:50, FastEthernet0/0
C       1.45.24.0/24 is directly connected, FastEthernet0/0
R2#sh ip bgp vpnv4 all 
BGP table version is 95, local router ID is 1.3.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:45
*>i1.45.5.5/32      1.3.1.1                  1    100      0 ?
*>i1.45.15.0/24     1.3.1.1                  0    100      0 ?
Route Distinguisher: 2:45 (default for vrf 45)
*> 1.45.4.4/32      1.45.24.4           156160         32768 ?
*>i1.45.5.5/32      1.3.1.1                  1    100      0 ?
*>i1.45.15.0/24     1.3.1.1                  0    100      0 ?
*> 1.45.24.0/24     0.0.0.0                  0         32768 ?

On R2 we see the same VRF, with the same routes and the same VPNv4 BGP paths.


So what can go wrong? Lets move on to check the MPLS path between R2 and R1.


On R2 we can see that the next-hop for the BPG VPNv4 paths is 1.3.1.1, lets make sure we have a LSP (an MPLS path) to 1.3.1.1 .

On R2:

R2#sh mpls forwarding-table 1.3.1.1
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
17     16            1.3.1.1/32        0             Fa0/1      1.3.26.6

The outgoing label is 16. lets follow it on R6.

On R6:

R6#sh mpls forwarding-table labels 16
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop    
Label  Label or VC   or Tunnel Id      Switched      interface              
16     Pop Label     1.3.1.1/32        83592         Fa0/1      1.3.16.1   

When R6 receives a packet labeled 16, it pops the lable from the label stack and send the packet to R1.

Now that we have reached R1 and verified that we have LSP from R2 to R1, lets check the other direction. First we lets refresh our memory and check what is the next hop for the VPNv4 paths on R1.

On R1:

R1#sh ip bgp vpnv4 all
BGP table version is 347, local router ID is 1.3.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:45 (default for vrf 45)
*>i1.45.4.4/32      1.3.26.2            156160    100      0 ?
*> 1.45.5.5/32      1.45.15.5                1         32768 ?
*> 1.45.15.0/24     0.0.0.0                  0         32768 ?
*>i1.45.24.0/24     1.3.26.2                 0    100      0 ?
Route Distinguisher: 2:45
*>i1.45.4.4/32      1.3.26.2            156160    100      0 ?
*>i1.45.24.0/24     1.3.26.2                 0    100      0 ?

The next hop is 1.3.26.2, lets follow the LSP path from R1 to 1.3.26.2

On R1:

R1#sh mpl forwarding-table 1.3.26.2
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop    
Label  Label or VC   or Tunnel Id      Switched      interface              
16     Pop Label     1.3.26.0/24       0             Fa0/0      1.3.16.6

The operation is pop. Is that a problem? To answer that, lets examine the CEF entries on R2 towards R5's Lo0 IP address and on R1 towards R4's FastEthernet0/1 IP address.

On R2:

R2#sh ip cef vrf 45 1.45.5.5
1.45.5.5/32
  nexthop 1.3.26.6 FastEthernet0/1 label 16 19

We can see two labels. 16 is IPv4 label, pointing to R1, and 19 is the VPNv4 label, which should arrive to R1 to indicate that the packet should be sent to R5.

On R1:

R1#sh ip cef vrf 45 1.45.24.4
1.45.24.0/24
  nexthop 1.3.16.6 FastEthernet0/0 label 20

Only one label, and it must be the VPNv4 label. Now we can see the problem. A reply ICMP packet from R5 is sent towards R6 with the label 20, but does R6 knows what label 20 means?

On R6:

R6#sh mpls forwarding-table labels 20
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop    
Label  Label or VC   or Tunnel Id      Switched      interface              
R6#

R6 knows nothing about label #20. And why should it? R6 do not participates in a VPNv4 BGP sessions.

Before we go on, we need to understand why R6 sends implicit NULL label, which is seen on R1 as "pop label" for R2's FastEthernet0/1 IP address? We should notice that R1's BGP received next hop for VPNv4 routes coming from R2 is 1.3.26.2. 1.3.26.2 is directly connected to R6, and that is why R6 sends implicit NULL label, as it is doing PHP.

We must also notice the following console error on R2's console:

On R2:

*Jan 23 17:02:23.515: %BGP-4-VPN_NH_IF: Nexthop 1.3.26.2 may not be reachable from neigbor 1.3.1.1 - not a loopback

R2 was not configured to neighbor with R1 with a source interface of Lo0.

How can we fix the problem without touching R1's configuration? We can ask R2 to manually set the "next-hop" BGP attribute to its Lo0 IP address.

On R2:

R2(config)#route-map rmVPNv4
R2(config-route-map)#set ip next-hop 1.3.2.2
% Warning: Next hop address is our address
R2(config-route-map)#router bgp 12
R2(config-router)#address-family vpnv4
R2(config-router-af)#neighbor 1.3.1.1 route-map rmVPNv4 out
R2(config-router-af)#^Z
R2#clear ip bgp *
R2#
*Jan 23 17:10:10.027: %BGP-5-ADJCHANGE: neighbor 1.3.1.1 Down User reset
R2#
*Jan 23 17:10:11.123: %BGP-5-ADJCHANGE: neighbor 1.3.1.1 Up

Now, lets try again to ping from R4 to R5's Lo0 IP address.

On R4:

R4#ping 1.45.5.5

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.45.5.5, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/20/28 ms

Success!
                                          __ 
.-----.--.--.----.----.-----.-----.-----.|  |
|__ --|  |  |  __|  __|  -__|__ --|__ --||__|
|_____|_____|____|____|_____|_____|_____||__|