VRRP - track remote ipsec tunnel gateway - kernel route always there

[lhubschmid]

Hello everybody,

I try to use VRRP tracking (vrdst command) to automatically make a VRRP failover when the IPSec VPN (interface mode) is down:

    edit "port2"
        set vdom "root"
        set ip 192.168.0.11 255.255.255.240
        set allowaccess ping
        set bfd enable
        set type physical
        set vrrp-virtual-mac enable
        config vrrp
            edit 5
                set vrip 192.168.0.10
                set priority 255
                set vrdst 192.168.255.2
            next
        end
        set snmp-index 2
    next

VRRP failover works successfully if I configure vrdst with an IP for which the FG does not have a route, e.g. 8.8.8.8.

But when I specify the IP of the tunnel interface remote-gw (local IP: 192.168.255.1, remote IP: 192.168.255.2) VRRP failover does not happen. It seems the local and remote tunnel IP is kept always in the kernel routing table.

Local (192.168.255.1) and remote (192.168.255.2) tunnel IP in "normal" routing table when IPSec VPN is down:

P-dcA-fw1 # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

C 1.1.1.0/30 is directly connected, port1
O E2 10.0.0.0/24 [110/10] via 192.168.0.12, port2, 00:03:53
O E2 10.0.1.0/24 [110/10] via 192.168.0.12, port2, 00:03:53
O 172.16.0.0/25 [110/202] via 192.168.0.12, port2, 00:03:54
S 172.16.20.0/24 [10/0] via 192.168.0.1, port2
S 172.16.21.0/24 [10/0] via 192.168.0.1, port2
C 192.168.0.0/28 is directly connected, port2
C 192.168.0.10/32 is directly connected, port2
C 192.168.99.0/24 is directly connected, port10
S 192.168.100.0/24 [10/0] via 192.168.0.1, port2
C 192.168.255.1/32 is directly connected, To-C-dcA-fw1
C 192.168.255.2/32 is directly connected, To-C-dcA-fw1
O 192.168.255.5/32 [110/1] via 192.168.0.12, port2, 00:03:54
O 192.168.255.6/32 [110/201] via 192.168.0.12, port2, 00:03:54

Now with IPSec VPN down:

P-dcA-fw1 # get vpn ipsec tunnel summary
'To-C-dcA-fw1' 1.1.1.2:0 selectors(total,up): 1/0 rx(pkt,err): 0/0 tx(pkt,err): 0/0

Tunnel interface is up although IPSec VPN is down:

P-dcA-fw1 # get sys int
...
== [ To-C-dcA-fw1 ]
name: To-C-dcA-fw1 ip: 192.168.255.1 255.255.255.255 status: up netbios-forward: disable type: tunnel netflow-sampler: disable sflow-sampler: disable scan-botnet-connections: disable explicit-web-proxy: disable explicit-ftp-proxy: disable wccp: disable

No local or remote tunnel IP in "normal" routing table when IPSec VPN is down:

P-dcA-fw1 # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

C 1.1.1.0/30 is directly connected, port1
O E2 10.0.0.0/24 [110/10] via 192.168.0.12, port2, 00:01:06
O E2 10.0.1.0/24 [110/10] via 192.168.0.12, port2, 00:01:06
O 172.16.0.0/25 [110/202] via 192.168.0.12, port2, 00:01:07
O 172.16.0.80/32 [110/202] via 192.168.0.12, port2, 00:01:07
S 172.16.20.0/24 [10/0] via 192.168.0.1, port2
S 172.16.21.0/24 [10/0] via 192.168.0.1, port2
C 192.168.0.0/28 is directly connected, port2
C 192.168.0.10/32 is directly connected, port2
C 192.168.99.0/24 is directly connected, port10
S 192.168.100.0/24 [10/0] via 192.168.0.1, port2
O 192.168.255.5/32 [110/1] via 192.168.0.12, port2, 00:01:07
O 192.168.255.6/32 [110/201] via 192.168.0.12, port2, 00:01:07

But 192.168.255.1 and 192.168.255.2 are still in the kernel routing table:

P-dcA-fw1 # get router info kernel
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->1.1.1.0/32 pref=1.1.1.1 gwy=0.0.0.0 dev=3(port1)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->1.1.1.1/32 pref=1.1.1.1 gwy=0.0.0.0 dev=3(port1)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->1.1.1.3/32 pref=1.1.1.1 gwy=0.0.0.0 dev=3(port1)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->127.0.0.0/32 pref=127.0.0.1 gwy=0.0.0.0 dev=13(root)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->127.0.0.0/8 pref=127.0.0.1 gwy=0.0.0.0 dev=13(root)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->127.0.0.1/32 pref=127.0.0.1 gwy=0.0.0.0 dev=13(root)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->127.255.255.255/32 pref=127.0.0.1 gwy=0.0.0.0 dev=13(root)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.168.0.0/32 pref=192.168.0.11 gwy=0.0.0.0 dev=4(port2)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.168.0.10/32 pref=192.168.0.10 gwy=0.0.0.0 dev=4(port2)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.168.0.11/32 pref=192.168.0.11 gwy=0.0.0.0 dev=4(port2)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.168.0.15/32 pref=192.168.0.11 gwy=0.0.0.0 dev=4(port2)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.168.99.0/32 pref=192.168.99.11 gwy=0.0.0.0 dev=12(port10)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.168.99.11/32 pref=192.168.99.11 gwy=0.0.0.0 dev=12(port10)
tab=255 vf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.168.99.255/32 pref=192.168.99.11 gwy=0.0.0.0 dev=12(port10)
tab=255 vf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.168.255.1/32 pref=192.168.255.1 gwy=0.0.0.0 dev=15(To-C-dcA-fw1)
tab=254 vf=0 scope=253 type=1 proto=2 prio=0 0.0.0.0/0.0.0.0/0->1.1.1.0/30 pref=1.1.1.1 gwy=0.0.0.0 dev=3(port1)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->10.0.0.0/24 pref=0.0.0.0 gwy=192.168.0.12 dev=4(port2)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->10.0.1.0/24 pref=0.0.0.0 gwy=192.168.0.12 dev=4(port2)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->172.16.0.0/25 pref=0.0.0.0 gwy=192.168.0.12 dev=4(port2)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->172.16.0.80/32 pref=0.0.0.0 gwy=192.168.0.12 dev=4(port2)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->172.16.20.0/24 pref=0.0.0.0 gwy=192.168.0.1 dev=4(port2)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->172.16.21.0/24 pref=0.0.0.0 gwy=192.168.0.1 dev=4(port2)
tab=254 vf=0 scope=253 type=1 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.168.0.0/28 pref=192.168.0.11 gwy=0.0.0.0 dev=4(port2)
tab=254 vf=0 scope=253 type=1 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.168.99.0/24 pref=192.168.99.11 gwy=0.0.0.0 dev=12(port10)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->192.168.100.0/24 pref=0.0.0.0 gwy=192.168.0.1 dev=4(port2)
tab=254 vf=0 scope=253 type=1 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.168.255.2/32 pref=192.168.255.1 gwy=0.0.0.0 dev=15(To-C-dcA-fw1)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->192.168.255.5/32 pref=0.0.0.0 gwy=192.168.0.12 dev=4(port2)
tab=254 vf=0 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0->192.168.255.6/32 pref=0.0.0.0 gwy=192.168.0.12 dev=4(port2)

And here the difference between a traceroute for a "fake" IP with no route vs a traceroute for the remote tunnel IP:

P-dcA-fw1 # execute traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 32 hops max, 3 probe packets per hop, 84 byte packets
 1 traceroute: sendto: Network is unreachable
traceroute: wrote 8.8.8.8 84 chars, ret=-1
 *traceroute: sendto: Network is unreachable
traceroute: wrote 8.8.8.8 84 chars, ret=-1
 *^Ctraceroute: sendto: Network is unreachable
traceroute: wrote 8.8.8.8 84 chars, ret=-1
 *

P-dcA-fw1 # execute traceroute 192.168.255.2
traceroute to 192.168.255.2 (192.168.255.2), 32 hops max, 3 probe packets per hop, 84 byte packets
 1 * * *
 2 * * *^C
 3 *
P-dcA-fw1 #

Note: add-gw-route (phase1-interfae) is by default "disable" - so no effect there.

Is there any possibility to either

- remove the route for the remote tunnel IP from the kernel routing table or

- bring down the tunnel interface when the IPSec VPN is down?

Any help is greatly appreciated!

Kind regards,

Lukas

[Toshi_Esumi]

For your specific VRRP behavior question, you probably need to open a case at TAC to get the right answer. But we won't use VRRP or HSRP (if Cisco to Cisco) to fail-over the route from via one IPSec tunnel to via another path when the IPSec goes down. I see you're using OSPF between devices. You can set up a routing protocol like OSPF over IPSec for the routes to fail over to another path. As long as the FG or Eth interface of the FG is not dead, all packets would hit the FG first (because it's the master among the VRRP members) then just forward to the alternative routes in the "normal" routing table to get to the destinations.

[lhubschmid]

Thanks for your answer! We are already using OSPF to re-route traffic over another IPSec tunnel, but to optimize traffic flow (avoid two unnecessary crossing of DCI) it would be nice to be able to failover the VRRP master.

Edit: it seems we have to open a case for VRRP start-time as well - preempt happens immediately after boot (without waiting the configured 60 seconds), which results in black hole routing (OSPF routes not yet learned).

[Toshi_Esumi]

We used to do that way but encountered some issues if the deice provides DHCP server so now we have iBGP between them while eBGP toward the remote(hub) side, which takes care of WAN side (circuit and IPSec) redundancy. VRRP and HSRP was originally designed to provide LAN side, or hardware, redundancy. Those additional features coming with the standard should be used carefully and wisely, in my opinion.

[oheigl]

Hey Lukas,

did you ever get this working, or what was the response from the support? I need the same setup, and I'm trying with the vrdst through the VPN. The VPN goes down, the route is gone too, but there is no failover happening of the VRRP gateway.

Kind regards

[oheigl]

Sorry guys, was missing a route on the second device, added it and now it works :)