Performance SLA breaks ipsec tunnels

[Tutek]

Hi,
we have problem with performance sla on one of the branch fortigates, performance sla monitor one of the main office fortigate interface using pings, after added two ipsec tunnels as members of this performance sla it is working fine couple of hours, then performance sla breaks the tunnels (they are both down) and never restore it up, meantime all connectivity between main and branch office wan interfaces are fine so lack connectivity between two fortigate WANs are not problem here. Once we remove ipsec tunnels from the performance sla they immediately started to working. Performance sla is configured on both sides.

[Tutek]

Today second branch office had the same situation, two monitored ipsec tunnels (ping the remote internal server) goes down, I had to remove it from performance sla they immediately changed to green, and added again to performace SLA.

What I have changed, to all phase2-interfaces I added:

set auto-negotiate enable

 Maybe here is the problem that SA expires, then link monitor have no connectivity to ping remote internal server and because of that it remove it from routing?

[pminarik]

The link-monitor with ping means you have constant traffic flowing through the SA, so it shouldn't expire. (it should be renewed when it's nearing expiration)

Perhaps there's something failing when the new SA is about to be negotiated? Or the negotiation itself fails?

[gfleming]

Can you show your SLA config and your SD-WAN rules? 

Also if you are trying to monitor the branch sites from main site it's probably best to monitor a loopback interface.

Second consideration is do you need to monitor SD-WAN from main site to branches? Does most of your traffic initiate from branches? If so you probably don't need SD-WAN on hub, just let traffic flow from branches using branch SD-WAN rules.

[Tutek]

Fortigate hub:

FGT (sdwan) # show
config system sdwan
    set status enable
    config zone
        edit "virtual-wan-link"
        next
        edit "branch1"
        next
        edit "branch2"
        next
    end
    config members
        edit 1
            set interface "port24"
            set gateway x.x.x.x
        next
        edit 2
            set interface "port23"
            set gateway y.y.y.y
        next
        edit 3
            set interface "w1-branch1-w1"
            set zone "branch1"
            set source 10.10.10.1
        next
        edit 4
            set interface "w2-branch1-w1"
            set zone "branch1"
            set source 10.10.10.1
        next
        edit 5
            set interface "w1-branch2-w1"
            set zone "branch2"
            set source 10.10.10.1
        next
        edit 6
            set interface "w2-branch2-w1"
            set zone "branch2"
            set source 10.10.10.1
        next
    end
    config health-check
        edit "Default_DNS"
            set system-dns enable
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Default_Office_365"
            set server "www.office.com"
            set protocol http
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Default_Gmail"
            set server "gmail.com"
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 2
                next
            end
        next
        edit "Default_AWS"
            set server "aws.amazon.com"
            set protocol http
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Default_Google Search"
            set server "www.google.com"
            set protocol http
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Default_FortiGuard"
            set server "fortiguard.com"
            set protocol http
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "SD_WAN_SLA"
            set server "208.91.112.52" "173.243.140.53"
            set interval 1000
            set failtime 10
            set recoverytime 10
            set members 2 1
        next
        edit "branch1_interface"
            set server "10.17.0.1"
            set interval 2000
            set failtime 15
            set recoverytime 15
            set members 3 4
        next
        edit "branch2_interface"
            set server "10.18.0.1"
            set interval 1000
            set failtime 15
            set recoverytime 15
            set members 5 6
        next
    end
    config service
        edit 5
            set name "Fortiguard_Out"
            set mode priority
            set src "all"
            set internet-service enable
            set internet-service-name "Fortinet-FortiGuard" "Fortinet-FortiCloud" "Fortinet-DNS" "Fortinet-FortiGuard.Secure.DNS" "Fortinet-Other"
            set health-check "SD_WAN_SLA"
            set priority-members 1 2
        next
        edit 4
            set name "xxxxxxx_Out_WAN1"
            set dst "xxxxxx" "xxxxxxx"
            set src "all"
            set priority-members 1
        next
        edit 2
            set name "Fortimail_Out_WAN1"
            set dst "all"
            set src "Fortimail"
            set priority-members 1
        next
        edit 7
            set name "PC_Out_Wan1"
            set mode priority
            set dst "all"
            set src "PC"
            set health-check "SD_WAN_SLA"
            set priority-members 1
            set status disable
        next
        edit 8
            set name "hub_to_branch1"
            set dst "branch1_Subnet"
            set src "hub_lan_subnet" "hub_subnet"
            set priority-members 3 4
        next
        edit 9
            set name "hub_to_branch2"
            set dst "branch2_subnet"
            set src "hub_lan_subnet" "hub_subnet"
            set priority-members 6 5
        next
        edit 6
            set name "Internet_Out_Wan2"
            set dst "all"
            set src "all"
            set priority-members 2 1
        next
    end
end

FGT (sdwan) #    

Branch1 (second is configured in the same way:

FGT-branch1 (sdwan) # show
config system sdwan
    set status enable
    config zone
        edit "virtual-wan-link"
        next
        edit "centrala"
        next
    end
    config members
        edit 1
            set interface "wan2"
        next
        edit 2
            set interface "wan1"
            set gateway x.x.x.x
        next
        edit 3
            set interface "w1-centrala-w1"
            set zone "centrala"
            set source 10.17.2.1
        next
        edit 4
            set interface "w1-centrala-w2"
            set zone "centrala"
            set source 10.17.2.1
        next
    end
    config health-check
        edit "Default_DNS"
            set system-dns enable
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Default_Office_365"
            set server "www.office.com"
            set protocol http
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Default_Gmail"
            set server "gmail.com"
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 2
                next
            end
        next
        edit "Default_AWS"
            set server "aws.amazon.com"
            set protocol http
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Default_Google Search"
            set server "www.google.com"
            set protocol http
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Default_FortiGuard"
            set server "fortiguard.com"
            set protocol http
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Centrala"
            set server "10.10.0.1"
            set interval 2000
            set failtime 15
            set recoverytime 15
            set members 3 4
        next
    end
    config service
        edit 1
            set name "Lan_to_Centrala"
            set dst "Centrala_Lan" "Centrala_Lan_Old"
            set src "lan address"
            set priority-members 3 4
        next
        edit 3
            set name "DNS"
            set internet-service enable
            set internet-service-app-ctrl 16195
            set priority-members 3 4
        next
        edit 4
            set name "Fortiguard"
            set internet-service enable
            set internet-service-name "Fortinet-FortiGuard" "Fortinet-FortiCloud"
            set priority-members 4 3
            set status disable
        next
        edit 2
            set name "Lan_to_Internet"
            set dst "all"
            set priority-members 4 3
        next
    end
end

FGT-branch1 (sdwan) #  

Now I added destination networks with option "Blackhole and higher distance on both sides. 

So you advice me to create loopback on hub, and do monitoring on one side only (from branch to hub)? As you can see I use to ping interface gateway on hub side (not any server located in lan), so if fortigate is powered up and running I don't see much difference between the local interface gateway and the logical interface.