Technical Tip: SD-WAN primary and backup ipsec tunnel Scenario

[achowdhury]

Description

This article describes how to active primary and backup ipsec tunnel same time but traffic will be pass according to the routing table.

 FG(siteA)(10.109.16.153) ====IPSEC==== (10.109.16.152)FG(siteB)

          (10.109.48.153)

    local 10.120.0.0/20< ------------------------------------------------------------------------>local 10.124.0.0/20

Both primary and backup tunnel are up but only traffic will pass to the backup tunnel when primary link fail and while it comeback again traffic switch over to the primary route.

From site A this is the routing:

S    *> 0.0.0.0/0 [1/0] via 10.109.31.254, port1, [0/73]

     *>           [1/0] via 10.109.63.254, port2, [0/5]

C    *> 10.109.16.0/20 is directly connected, port1

C    *> 10.109.48.0/20 is directly connected, port2

C    *> 10.120.0.0/20 is directly connected, port5

S       10.124.0.0/20 [20/0] is directly connected, backupto3hd4--------backup tunnel

S    *> 10.124.0.0/20 [10/0] is directly connected, to3hd4------------primary tunnel

S       10.124.0.0/20 [254/0] is a summary, Null

In the above scenario traffic will pass only the primary tunnel though both tunnels active when primary will fail it goes to the secondary but this will not happened immediately as it depends the DPD setting on the phase-1.

After the interface is disabled, for example, port1.

run the ike debug the dpd message will show for example:

send IKEv1 DPD probe, seqno 29
2019-08-06 09:55:42.312949 ike 0:to3hd4:70: enc 2D566CC3723820F1DAE5A66BE604628F081005017AA79A8D000000600B0000244DA00272A7D3756BBB5BC4FC6E713D82EEEFF05909D83293B10CDDD6E4BEB0C2000000200000000101108D282D566CC3723820F1DAE5A66BE604628F0000001D
2019-08-06 09:55:42.312963 ike 0:to3hd4:70: out 2D566CC3723820F1DAE5A66BE604628F081005017AA79A8D0000006C908688EAE858C6A55FF58B45BB9DC6BDFC27D6F6D71AEA9EEEF4CD32651DB268524DE67BE8D8E506975803B9BE687850DE6405156B132A0D19BA5DEB63DBE128A21D4992C33A1214CE1C8AC58A012F2D
2019-08-06 09:55:42.312974 ike 0:to3hd4:70: could not send IKE Packet(R-U-THERE):10.109.16.153:500->10.109.16.152:500, len=108: error 101:Network is unreachable
2019-08-06 09:55:55.512985 ike 0:backupto3hd4: link is idle 10 10.109.48.153->10.109.16.152:0 dpd=1 seqno=2c

After I manually down the interface:

S > 0.0.0.0/0 [1/0] via 10.109.31.254, port1 inactive, [0/83]               <----- Notice this interface down route inactive
*> [1/0] via 10.109.63.254, port2, [0/13]
C *> 10.109.48.0/20 is directly connected, port2
C *> 10.120.0.0/20 is directly connected, port5
S 10.124.0.0/20 [20/0] is directly connected, backupto3hd4
S *> 10.124.0.0/20 [10/0] is directly connected, to3hd4
S 10.124.0.0/20 [254/0] is a summary, Null

But the primary route for tunnel will not removed immediately.

For my case 60 sec:

FGT3HD-3 # get router info routing-table database

Routing table for VRF=0
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
> - selected route, * - FIB route, p - stale info

S > 0.0.0.0/0 [1/0] via 10.109.31.254, port1 inactive, [0/83]
*> [1/0] via 10.109.63.254, port2, [0/13]
C *> 10.109.48.0/20 is directly connected, port2
C *> 10.120.0.0/20 is directly connected, port5
S *> 10.124.0.0/20 [20/0] is directly connected, backupto3hd4               <----- This is secondary now which is active.
S 10.124.0.0/20 [10/0] is directly connected, to3hd4 inactive
S 10.124.0.0/20 [254/0] is a summary, Null

After manually active port1:

S *> 0.0.0.0/0 [1/0] via 10.109.31.254, port1, [0/83]
*> [1/0] via 10.109.63.254, port2, [0/12]
C *> 10.109.16.0/20 is directly connected, port1
C *> 10.109.48.0/20 is directly connected, port2
C *> 10.120.0.0/20 is directly connected, port5
S 10.124.0.0/20 [20/0] is directly connected, backupto3hd4
S *> 10.124.0.0/20 [10/0] is directly connected, to3hd4
S 10.124.0.0/20 [254/0] is a summary, Null

Again primary is now active.

Solution
In the above scenario Site A has SD-WAN configuration as follows:

#config system virtual-wan-link
    set status enable
    set load-balance-mode measured-volume-based
    config members
        edit 1
            set interface "port1"
            set gateway 10.109.31.254
            set volume-ratio 50
        next
        edit 2
            set interface "port2"
            set gateway 10.109.63.254
            set volume-ratio 10
        next
    end
    config health-check
        edit "isp"
            set server "8.8.8.8"
            set members 1 2
            config sla
                edit 1
                next
            end
        next
    end
end

Site A tunnel configuration:

#FGT3HD-3 (phase1-interface) # sh
config vpn ipsec phase1-interface
    edit "to3hd4"
        set interface "port1"
        set peertype any
        set proposal aes128-sha256 aes256-sha256 aes128-sha1 aes256-sha1
        set dpd on-idle----this needs to be idle
        set comments "VPN: to3hd4 (Created by VPN wizard)"
        set wizard-type static-fortigate
        set remote-gw 10.109.16.152
        set psksecret ENC
    next
    edit "backupto3hd4"
        set interface "port2"
        set peertype any
        set proposal aes128-sha256 aes256-sha256 aes128-sha1 aes256-sha1
        set dpd on-idle
        set comments "VPN: backupto3hd4 (Created by VPN wizard)"
        set wizard-type static-fortigate
        set remote-gw 10.109.16.152
        set psksecret ENC
    next
end

Static route:

#config router static
    edit 1
        set distance 1
        set virtual-wan-link enable
    next
    edit 2
        set device "to3hd4"
        set comment "VPN: to3hd4 (Created by VPN wizard)"
        set dstaddr "to3hd4_remote"
    next
    edit 3
        set distance 254
        set comment "VPN: to3hd4 (Created by VPN wizard)"
        set blackhole enable
        set dstaddr "to3hd4_remote"
    next
    edit 4
        set distance 20
        set device "backupto3hd4"
        set comment "VPN: backupto3hd4 (Created by VPN wizard)"
        set dstaddr "backupto3hd4_remote"
    next
    edit 5
        set distance 254
        set comment "VPN: backupto3hd4 (Created by VPN wizard)"
        set blackhole enable
        set dstaddr "backupto3hd4_remote"
    next
end

Firewall policy:

#config firewall policy
    edit 1
        set name "vpn_to3hd4_local"
        set uuid bca75dd6-b819-51e9-dca2-d305e1d2ee36
        set srcintf "port5"
        set dstintf "to3hd4"
        set srcaddr "to3hd4_local"
        set dstaddr "to3hd4_remote"
        set action accept
        set schedule "always"
        set service "ALL"
        set comments "VPN: to3hd4 (Created by VPN wizard)"
    next
    edit 2
        set name "vpn_to3hd4_remote"
        set uuid bcabc68c-b819-51e9-a50c-4ff0e1e7b271
        set srcintf "to3hd4"
        set dstintf "port5"
        set srcaddr "to3hd4_remote"
        set dstaddr "to3hd4_local"
        set action accept
        set schedule "always"
        set service "ALL"
        set comments "VPN: to3hd4 (Created by VPN wizard)"
    next
    edit 3
        set name "vpn_backupto3hd4_local"
        set uuid db553d66-b819-51e9-3f9b-f1ac3473103a
        set srcintf "port5"
        set dstintf "backupto3hd4"
        set srcaddr "backupto3hd4_local"
        set dstaddr "backupto3hd4_remote"
        set action accept
        set schedule "always"
        set service "ALL"
        set comments "VPN: backupto3hd4 (Created by VPN wizard)"
    next
    edit 4
        set name "vpn_backupto3hd4_remote"
        set uuid db59f6ee-b819-51e9-f483-dcd5ef7c7937
        set srcintf "backupto3hd4"
        set dstintf "port5"
        set srcaddr "backupto3hd4_remote"
        set dstaddr "backupto3hd4_local"
        set action accept
        set schedule "always"
        set service "ALL"
        set comments "VPN: backupto3hd4 (Created by VPN wizard)"
    next
end

Site B:

#FGT3HD-4 (phase1-interface) # sh
config vpn ipsec phase1-interface
    edit "to3hd"
        set interface "port5"
        set peertype any
        set proposal aes128-sha256 aes256-sha256 aes128-sha1 aes256-sha1
        set comments "VPN: to3hd (Created by VPN wizard)"
        set wizard-type static-fortigate
        set remote-gw 10.109.16.153
        set psksecret ENC
    next
    edit "tobackup-tunnel"
        set interface "port5"
        set peertype any
        set proposal aes128-sha256 aes256-sha256 aes128-sha1 aes256-sha1
        set comments "VPN: tobackup-tunnel (Created by VPN wizard)"
        set wizard-type static-fortigate
        set remote-gw 10.109.48.153
        set psksecret ENC
FGT3HD-4 # config vpn ipsec phase2-interface
FGT3HD-4 (phase2-interface) # sh
config vpn ipsec phase2-interface
    edit "to3hd"
        set phase1name "to3hd"
        set proposal aes128-sha1 aes256-sha1 aes128-sha256 aes256-sha256 aes128gcm aes256gcm chacha20poly1305
        set auto-negotiate enable
        set comments "VPN: to3hd (Created by VPN wizard)"
        set src-addr-type name
        set dst-addr-type name
        set src-name "to3hd_local"
        set dst-name "to3hd_remote"
    next
    edit "tobackup-tunnel"
        set phase1name "tobackup-tunnel"
        set proposal aes128-sha1 aes256-sha1 aes128-sha256 aes256-sha256 aes128gcm aes256gcm chacha20poly1305
        set auto-negotiate enable
        set comments "VPN: tobackup-tunnel (Created by VPN wizard)"
        set src-addr-type name
        set dst-addr-type name
        set src-name "tobackup-tunnel_local"
        set dst-name "tobackup-tunnel_remote"
    next
end
FGT3HD-3 (phase2-interface) # sh
config vpn ipsec phase2-interface
    edit "to3hd4"
        set phase1name "to3hd4"
        set proposal aes128-sha1 aes256-sha1 aes128-sha256 aes256-sha256 aes128gcm aes256gcm chacha20poly1305
        set auto-negotiate enable
        set comments "VPN: to3hd4 (Created by VPN wizard)"
        set src-addr-type name
        set dst-addr-type name
        set src-name "to3hd4_local"
        set dst-name "to3hd4_remote"
    next
    edit "backupto3hd4"
        set phase1name "backupto3hd4"
        set proposal aes128-sha1 aes256-sha1 aes128-sha256 aes256-sha256 aes128gcm aes256gcm chacha20poly1305
        set auto-negotiate enable
        set comments "VPN: backupto3hd4 (Created by VPN wizard)"
        set src-addr-type name
        set dst-addr-type name
        set src-name "backupto3hd4_local"
        set dst-name "backupto3hd4_remote"
    next
end

Static route:

   #edit 2
        set device "to3hd"
        set comment "VPN: to3hd (Created by VPN wizard)"
        set dstaddr "to3hd_remote"
    next
    edit 3
        set distance 254
        set comment "VPN: to3hd (Created by VPN wizard)"
        set blackhole enable
        set dstaddr "to3hd_remote"
    next
    edit 4
        set distance 20
        set device "tobackup-tunnel"
        set comment "VPN: tobackup-tunnel (Created by VPN wizard)"
        set dstaddr "tobackup-tunnel_remote"
    next
    edit 5
        set distance 254
        set comment "VPN: tobackup-tunnel (Created by VPN wizard)"
        set blackhole enable
        set dstaddr "tobackup-tunnel_remote"
    next
end

Policy:

#config firewall policy
    edit 1
        set name "vpn_to3hd_local"
        set uuid 013de924-b81a-51e9-aa7f-449389b75c86
        set srcintf "port2"
        set dstintf "to3hd"
        set srcaddr "to3hd_local"
        set dstaddr "to3hd_remote"
        set action accept
        set schedule "always"
        set service "ALL"
        set comments "VPN: to3hd (Created by VPN wizard)"
    next
    edit 2
        set name "vpn_to3hd_remote"
        set uuid 01439e0a-b81a-51e9-ca3a-32e7256898d6
        set srcintf "to3hd"
        set dstintf "port2"
        set srcaddr "to3hd_remote"
        set dstaddr "to3hd_local"
        set action accept
        set schedule "always"
        set service "ALL"
        set comments "VPN: to3hd (Created by VPN wizard)"
    next
    edit 3
        set name "vpn_tobackup-tunnel_local"
        set uuid 763aaac6-b81c-51e9-34da-7b0a390308d1
        set srcintf "port2"
        set dstintf "tobackup-tunnel"
        set srcaddr "tobackup-tunnel_local"
        set dstaddr "tobackup-tunnel_remote"
        set action accept
        set schedule "always"
        set service "ALL"
        set comments "VPN: tobackup-tunnel (Created by VPN wizard)"
    next
    edit 4
        set name "vpn_tobackup-tunnel_remote"
        set uuid 763fc1f0-b81c-51e9-5c0e-a788a5cb9d6a
        set srcintf "tobackup-tunnel"
        set dstintf "port2"
        set srcaddr "tobackup-tunnel_remote"
        set dstaddr "tobackup-tunnel_local"
        set action accept
        set schedule "always"
        set service "ALL"
        set comments "VPN: tobackup-tunnel (Created by VPN wizard)"
    next
end