Ticket #16

Ticket #17

Network diagram


T17

IGP diagram


T17_igp

The problem:


R3 is the DHCP server on V100. 


R1 was configured to be a backup for the DHCP server on R3. R1 is using object tracking and EEM to do so.


However, the solution is not working and even if it was its too slow and unreliable. Fix the problems. 


Do not use BFD, its not working on dynamips :)


The solution:


First, lets review R1's configuration for the DHCP redundancy.

On R1:

R1#sh run | sec event|track

track 1 ip route 7.7.99.3 255.255.255.255 reachability

event manager applet downDHCP 

 event track 1 state down

 action 1.0 syslog msg "R3's DHCP is down"

 action 2.0 cli command "configure terminal"

 action 2.1 cli command "ip dhcp excluded-address 7.7.100.1 7.7.100.10"

 action 2.2 cli command "ip dhcp pool plT17"

 action 2.3 cli command "network 7.7.100.0 255.255.255.0"

 action 2.4 cli command "default-router 7.7.100.2"

event manager applet upDHCP 

 event track 1 state up

 action 1.0 syslog msg "R3's DHCP is up"

 action 2.0 cli command "configure terminal"

 action 2.1 cli command "no ip dhcp excluded-address 7.7.100.1 7.7.100.10"

 action 2.2 cli command "no ip dhcp pool plT17"



We can see that R1 is tracking the route to R3's Lo0, which should be reachable via R3's F0/0 interface, where DHCP requests are served by R3.


Once R1 losses R3's Lo0 the downDHCP EEM applet is configuring a DHCP server on R1. Once R1's sees R3's Lo0 up again, the upDHCP EEM applet is deleting the DHCP configuration on R1.


Lets try to shut down R3's F0/0 interface and see what happens on R1.


On R1:

R1#

*Jan 17 19:10:37.012: %SYS-5-CONFIG_I: Configured from console by console

R1#

*Jan 17 19:11:16.104: %TRACKING-5-STATE: 1 ip route 7.7.99.3/32 reachability Up->Down

R1#

*Jan 17 19:11:16.132: %HA_EM-6-LOG: downDHCP: R3's DHCP is down

R1#sh run | sec dhcp

 action 2.1 cli command "ip dhcp excluded-address 7.7.100.1 7.7.100.10"

 action 2.2 cli command "ip dhcp pool plT17"

 action 2.1 cli command "no ip dhcp excluded-address 7.7.100.1 7.7.100.10"

 action 2.2 cli command "no ip dhcp pool plT17"



We can see that R1 detects R3's F0/0 failure and also runs the downDHCP applet but nothing happens.


Lets bring R3's F0/0 back, then enable EEM debugging and then shutdown again R3's F0/0 interface.


On R1:

R1#      
*Jan 17 19:14:01.128: %HA_EM-6-LOG: upDHCP: R3's DHCP is up
R1#debug event manager action cli 
Debug EEM action cli debugging is on
*Jan 17 19:15:16.120: %TRACKING-5-STATE: 1 ip route 7.7.99.3/32 reachability Up->Down
*Jan 17 19:15:16.152: %HA_EM-6-LOG: downDHCP: R3's DHCP is down
*Jan 17 19:15:16.156: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : CTL : cli_open called.
*Jan 17 19:15:16.160: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : R1>
*Jan 17 19:15:16.160: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : IN  : R1>ip dhcp excluded-address 7.7.100.1 7.7.100.10
*Jan 17 19:15:16.208: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT :    ^
*Jan 17 19:15:16.208: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : % Invalid input detected at '^' marker.
*Jan 17 19:15:16.208: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : 
*Jan 17 19:15:16.208: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : R1>
*Jan 17 19:15:16.208: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : IN  : R1>ip dhcp pool plT17
*Jan 17 19:15:16.224: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT :    ^
*Jan 17 19:15:16.224: %HA_EM-6-LOG: d
R1#ownDHCP : DEBUG(cli_lib) : : OUT : % Invalid input detected at '^' marker.
*Jan 17 19:15:16.224: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : 
*Jan 17 19:15:16.224: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : R1>
*Jan 17 19:15:16.224: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : IN  : R1>network 7.7.100.0 255.255.255.0
*Jan 17 19:15:16.268: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT :     ^
*Jan 17 19:15:16.268: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : % Invalid input detected at '^' marker.
*Jan 17 19:15:16.268: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : 
*Jan 17 19:15:16.268: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : R1>
*Jan 17 19:15:16.272: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : IN  : R1>default-router 7.7.100.2
*Jan 17 19:15:16.316: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT :     ^
*Jan 17 19:15:16.316: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : % Invalid input detected at '^' marker.
*Jan 17 19:15:16.316: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : 
*Jan 17 19:15:16.316: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : R1>



We can see that some commands are missing: enable and configure terminal. Lets add them to R1's applets and try again, with bouncing R3's F0/0 interface again.


On R1:

R1#un all
All possible debugging has been turned off
R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#event manager applet downDHCP 
R1(config-applet)# action 1.5 cli command "enable"
R1(config-applet)# action 1.7 cli command "configure terminal"
R1(config-applet)#event manager applet upDHCP 
R1(config-applet)# action 1.5 cli command "enable"
R1(config-applet)# action 1.7 cli command "configure terminal"
R1(config-applet)#^Z
R1#
*Jan 17 19:20:01.136: %TRACKING-5-STATE: 1 ip route 7.7.99.3/32 reachability Down->Up
*Jan 17 19:20:01.172: %HA_EM-6-LOG: upDHCP: R3's DHCP is up
R1#
*Jan 17 19:20:01.340: %SYS-5-CONFIG_I: Configured from console by vty0
R1#debug event manager action cli
Debug EEM action cli debugging is on
R1#
*Jan 17 19:20:46.152: %TRACKING-5-STATE: 1 ip route 7.7.99.3/32 reachability Up->Down
*Jan 17 19:20:46.164: %HA_EM-6-LOG: downDHCP: R3's DHCP is down
*Jan 17 19:20:46.164: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : CTL : cli_open called.
*Jan 17 19:20:46.168: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : R1>
*Jan 17 19:20:46.168: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : IN  : R1>enable
*Jan 17 19:20:46.200: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : R1#
*Jan 17 19:20:46.200: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : IN  : R1#configure terminal
*Jan 17 19:20:46.240: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line.  End with CNTL/Z.
*Jan 17 19:20:46.240: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : R1(config)#
*Jan 17 19:20:46.240: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : IN  : R1(config)#ip dhcp excluded-address 7.7.100.1 7.7.100.10
*Jan 17 19:20:46.276: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : R1(config)#
*Jan 17 19:20:46.276: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : IN  : R1(config)#ip dhcp pool plT17
*Jan 17 19:20:46.288: %HA_EM-6-LOG: downDHCP : 
R1#DEBUG(cli_lib) : : OUT : R1(dhcp-config)#
*Jan 17 19:20:46.288: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : IN  : R1(dhcp-config)#network 7.7.100.0 255.255.255.0
*Jan 17 19:20:46.324: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : R1(dhcp-config)#
*Jan 17 19:20:46.328: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : IN  : R1(dhcp-config)#default-router 7.7.100.2
*Jan 17 19:20:46.364: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : OUT : R1(dhcp-config)#
*Jan 17 19:20:46.364: %HA_EM-6-LOG: downDHCP : DEBUG(cli_lib) : : CTL : cli_close called.
*Jan 17 19:20:46.364: %SYS-5-CONFIG_I: Configured from console by vty0
R1# sh run | sec dhcp  
ip dhcp excluded-address 7.7.100.1 7.7.100.10
ip dhcp pool plT17
   network 7.7.100.0 255.255.255.0
   default-router 7.7.100.2 
 action 2.1 cli command "ip dhcp excluded-address 7.7.100.1 7.7.100.10"
 action 2.2 cli command "ip dhcp pool plT17"
 action 2.1 cli command "no ip dhcp excluded-address 7.7.100.1 7.7.100.10"
 action 2.2 cli command "no ip dhcp pool plT17"


Success! R3's F0/0 went down and then R1 configured a DHCP server on the V100.


Now lets see what happens when R3's F0/0 is brought up.


On R1:

R1#un all
All possible debugging has been turned off
R1#
*Jan 17 19:24:16.160: %TRACKING-5-STATE: 1 ip route 7.7.99.3/32 reachability Down->Up
R1#
*Jan 17 19:24:16.176: %HA_EM-6-LOG: upDHCP: R3's DHCP is up
R1#
*Jan 17 19:24:17.352: %SYS-5-CONFIG_I: Configured from console by vty0
R1# sh run | sec dhcp
 action 2.1 cli command "ip dhcp excluded-address 7.7.100.1 7.7.100.10"
 action 2.2 cli command "ip dhcp pool plT17"
 action 2.1 cli command "no ip dhcp excluded-address 7.7.100.1 7.7.100.10"
 action 2.2 cli command "no ip dhcp pool plT17"


Success! Once R3's F0/0 is up R1 deletes the DHCP configuration.


Now lets tackle  the speed of operation issue. On the network diagram we can see that R3 is advertising it Lo0 using EIGRP and R1 is talking RIP. We can assume that R2, which is the only router running both EIGRP and RIP on V100, is redistributing the route. So first lets configure R2 and R3 with minimal EIGRP hello and dead interval timers.


On R2/R3:

R3(config)#int f0/0

R3(config-if)#ip hello-interval eigrp 1 1

R3(config-if)#ip hold-time eigrp 1 1

R3(config-if)#
*Jan 17 19:29:05.204: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 7.7.100.2 (FastEthernet0/0) is down: Interface Goodbye received
*Jan 17 19:29:05.216: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 7.7.100.2 (FastEthernet0/0) is up: new adjacency
R3(config-if)#
*Jan 17 19:29:35.696: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 7.7.100.2 (FastEthernet0/0) is down: Interface Goodbye received
*Jan 17 19:29:35.732: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 7.7.100.2 (FastEthernet0/0) is up: new adjacency
R3(config-if)#
*Jan 17 19:30:07.272: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 7.7.100.2 (FastEthernet0/0) is down: Interface Goodbye received
*Jan 17 19:30:07.276: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 7.7.100.2 (FastEthernet0/0) is up: new adjacency


We can see that EIGRP is not really stable. Lets increase the hold time to 2.


On R2/R3:

R3(config-if)#ip hold-time eigrp 1 2


Now that EIGRP is stable, lets tune RIP.


On R1/R2/R4:

R2(config)#router rip
R2(config-router)#timer basic 1 2 2 2


Success!? Not so fast. There is a hidden issue. Watch what happens when R4's F0/0 interface is shut.


On R1:

R1#
*Jan 17 19:37:16.180: %TRACKING-5-STATE: 1 ip route 7.7.99.3/32 reachability Up->Down
*Jan 17 19:37:16.208: %HA_EM-6-LOG: downDHCP: R3's DHCP is down
R1#
*Jan 17 19:37:16.416: %SYS-5-CONFIG_I: Configured from console by vty0


Remember, R4's F0/0 was shut, not R3's F0/0. What is going on? Now we have two DHCP servers on V100.


Lets bring R4's F0/0 up and traceroute from R1 to R3's Lo0


On R1:

*Jan 17 19:38:46.196: %TRACKING-5-STATE: 1 ip route 7.7.99.3/32 reachability Down->Up
R1#
*Jan 17 19:38:46.216: %HA_EM-6-LOG: upDHCP: R3's DHCP is up
R1#
*Jan 17 19:38:48.376: %SYS-5-CONFIG_I: Configured from console by vty0
R1#traceroute 7.7.99.3

Type escape sequence to abort.
Tracing the route to 7.7.99.3

  1 7.7.100.4 0 msec 0 msec 4 msec
  2 7.7.24.2 4 msec 8 msec 4 msec
  3  * 
    7.7.100.3 12 msec * 


We can see that the path to R3 is going via R4. Why?

Lets have a look at R4's routing table.

On R4:

R4#sh ip route 7.7.99.3
Routing entry for 7.7.99.3/32
  Known via "rip", distance 120, metric 1
  Redistributing via rip
  Last update from 7.7.24.2 on FastEthernet0/1, 00:00:00 ago
  Routing Descriptor Blocks:
  * 7.7.24.2, from 7.7.24.2, 00:00:00 ago, via FastEthernet0/1
      Route metric is 1, traffic share count is 1


Now we know two facts:

  1. R1 do not see the RIP route for R3's Lo0 coming from R2. Which means that R2 is not sending it on V100.
  2. R4 do receive the route on its F0/1 interface.

Why is it happening? Due to split horizon R2 will not send RIP route pointing to F0/0 back to F0/0, even if it was redistributed. 

Notice: If you will check R3's routing table, you would notice that this does not happen for EIGRP, and R3 sees R1's Lo0.

How can we solve that? Easy, lets configure R2 to disable split horizon on R2's F0/0.

On R2:

R2(config)#int f0/0
R2(config-if)#no ip split-horizon 


Now lets try to traceroute again from R1 to R3's Lo0.

On R1:

R1#traceroute 7.7.99.3

Type escape sequence to abort.
Tracing the route to 7.7.99.3

  1 7.7.100.3 16 msec *  32 msec


Now we are going directly from R1 to R3's Lo0. Lets try to bring R4's F0/0 interface down and see what happens on R1.


On R1:


Success!? Nothing happens. Wrong! 

Lets enable R4's F0/0 back, then shut down R3's Lo0 while debugging RIP on R1.

On R1:

....
*Jan 17 20:05:49.812:   7.7.99.3/32 via 0.0.0.0, metric 9, tag 0
...
*Jan 17 20:05:50.560:      7.7.99.3/32 via 0.0.0.0 in 10 hops
...
*Jan 17 20:05:50.720:   7.7.99.3/32 via 0.0.0.0, metric 11, tag 0
...
*Jan 17 20:05:53.436:   7.7.99.3/32 via 0.0.0.0, metric 16, tag 0


We can see that the route to R3's Lo0 is counting to infinity (16) which takes a valuable time, which means that there is no DHCP service for longer periods of time.

Why is it happening? Its because we have disabled split horizon on R2's F0/0 interface, which means that once R3's Lo0 is down, R2 is feeding R4's RIP database with what R4 have sent to R2 (via V24) plus one for the metric.

How can we fix that while keeping R4's redundant path to V100? Since in normal operation, R4 receives the RIP route from R2 with metric 1, lets add to that metric 14, so R4 will never feed R2 the R3's Lo0 route with value less then 16.

On R4:

R4(config)#router rip
R4(config-router)#offset-list 0 in 14
R4(config-router)#do sh ip route 7.7.99.3
Routing entry for 7.7.99.3/32
  Known via "rip", distance 120, metric 15
  Redistributing via rip
  Last update from 7.7.100.3 on FastEthernet0/0, 00:00:00 ago
  Routing Descriptor Blocks:
  * 7.7.100.3, from 7.7.100.2, 00:00:00 ago, via FastEthernet0/0
      Route metric is 15, traffic share count is 1
    7.7.24.2, from 7.7.24.2, 00:00:00 ago, via FastEthernet0/1
      Route metric is 15, traffic share count is 1


Now lets bring down R3's F0/0 and see what happens on R1.

On R1:

R1#debug ip rip
RIP protocol debugging is on
...
*Jan 17 20:14:40.516:      7.7.99.3/32 via 0.0.0.0 in 16 hops  (inaccessible)


Success! You will not see the route to R3's Lo0 counting to infinity.

  _____                             _ 
 / ____|                           | |
| (___  _   _  ___ ___ ___  ___ ___| |
 \___ \| | | |/ __/ __/ _ \/ __/ __| |
 ____) | |_| | (_| (_|  __/\__ \__ \_|
|_____/ \__,_|\___\___\___||___/___(_)