SDWAN SLA issue over IPSEC

[JeffML]

Hi!

I have a strange issue with the SDWAN and BGP.

We have a Hub and spokes setup, the VPN at Hub is dynamic. There is 2 WANs on the HUB and 2 on the Spokes, so we have 4 IPSEC tunnels between the Hub and a Spoke. Both use a Loopback for the BGP connexion. In fact, on both we see 4 path to each other. The SLA, that checks the path between the Loopback of the spoke to the loopback of the Hub. It seems like only 2 checks works at a time.  If I shut 2 tunnels, the 2 other SLA checks, works again.  How can we fix this.  

config system sdwan
set status enable
config zone
edit "virtual-wan-link"
next
end
config members
edit 5
set interface "SpokeW1-2-HubW1"
set source 10.128.0.8
next
edit 6
set interface "SpokeW1-2-HubW2"
set source 10.128.0.8
next
edit 9
set interface "SpokeW2-2-HubW1"
set source 10.128.0.8
next
edit 10
set interface "SpokeW2-2-HubW2"
set source 10.128.0.8
next
end

config health-check

edit "SLAHub"
set server "10.128.0.1"
set update-static-route disable
set members 5 6 9 10
next

end

end

[aguerriero]

Without a better understanding of your configs I would assume it is an asymmetric routing issue so state is breaking and RPF checks are failing. 

you could temporarily disable RPF and enable asymmetric routing but you would need to understand that this will break advanced inspection features included with UTM.

config system settings
set asymroute enable

I think the way fortinet supports asymmetric support for fortigate is enabling auxiliary sessions 

https://community.fortinet.com/t5/FortiGate/Technical-Tip-SD-WAN-Auxiliary-Sessions/ta-p/229467

[JeffML]

Hi! I tried both solution and that did not work.  But that would make sense in some way.

When I shut some of the VPN, the SLA starts working again

I can post more details about the setup

[aguerriero]

the sdwan needs routable interfaces. 

when you are setting up your ipsecs vpns in interface mode are you are assigning ip addresses to each side /31 or /30 or something?

your zone member config should include the next hop for each member which would be the distant end tunnel ip address for each ipsec tunnel. 

SpokeW1-2-HubW1
100.64.0.0/31
100.64.0.1/31

SpokeW1-2-HubW2
100.64.0.2/31
100.64.0.3/31

SpokeW2-2-HubW1
100.64.0.4/31
100.64.0.5/31

SpokeW2-2-HubW2
100.64.0.6/31
100.64.0.7/31

config system sdwan
set status enable
config zone
edit "virtual-wan-link"
next
end
config members
edit 5
set interface "SpokeW1-2-HubW1"
set gateway 100.64.0.1
next
edit 6
set interface "SpokeW1-2-HubW2"
set gateway 100.64.0.3
next
edit 9
set interface "SpokeW2-2-HubW1"
set gateway 100.64.0.5
next
edit 10
set interface "SpokeW2-2-HubW2"
set gateway 100.64.0.7
next
end

[JeffML]

Not in that case, they are setup within the VPN IPSEC configs themselfs:

Which are my Loopback interfaces.

On Hub:

config vpn ipsec phase1-interface
edit "SpokesWAN2"

set exchange-interface-ip enable
set exchange-ip-addr4 10.128.0.1

set network-overlay enable
next
edit "SpokesWAN1"

set exchange-interface-ip enable
set exchange-ip-addr4 10.128.0.1

set network-overlay enable
next
end

On Spoke:

Branch01 # show vpn ipsec phase1-interface
config vpn ipsec phase1-interface
edit "SpokeW1-2-HubW1"
set exchange-interface-ip enable
set exchange-ip-addr4 10.128.0.8
set network-overlay enable
next
edit "SpokeW1-2-HubW2"

set exchange-interface-ip enable
set exchange-ip-addr4 10.128.0.8

set network-overlay enable
next
edit "SpokeW2-2-HubW1"

set exchange-interface-ip enable
set exchange-ip-addr4 10.128.0.8
set network-overlay enable
next
edit "SpokeW2-2-HubW2"
set exchange-interface-ip enable
set exchange-ip-addr4 10.128.0.8
set network-overlay enable
next
end

Those then add stativc entries in the routing tables of both hub and spokes to each other:

OnHub

S 10.128.0.8/32 [15/0] via SpokesWAN1 tunnel 10.0.0.28, [1/0]
[15/0] via SpokesWAN1 tunnel 10.0.0.29, [1/0]
[15/0] via SpokesWAN2 tunnel 10.0.0.34, [1/0]
[15/0] via SpokesWAN2 tunnel 10.128.0.8, [1/0]

On Spoke:

S 10.128.0.0/24 [1/0] via SpokeW2-2-HubW1 tunnel 10.0.0.1, [1/0]
[1/0] via SpokeW2-2-HubW2 tunnel 10.0.0.2, [1/0]
[1/0] via SpokeW1-2-HubW1 tunnel 11.1.1.2, [1/0]
[1/0] via 11.1.3.1, port1, [1/0]
[1/0] via SpokeW1-2-HubW2 tunnel 12.1.1.2, [1/0]
[1/0] via 12.1.3.1, port2, [1/0]
S 10.128.0.1/32 [15/0] via SpokeW2-2-HubW1 tunnel 10.0.0.1, [1/0]
[15/0] via SpokeW2-2-HubW2 tunnel 10.0.0.2, [1/0]
[15/0] via SpokeW1-2-HubW1 tunnel 11.1.1.2, [1/0]
[15/0] via SpokeW1-2-HubW2 tunnel 12.1.1.2, [1/0]