SDWAN - BGP behavior

[Morus]

Hi!

So i have an HUB&Spoke topology (OS 7.4.6), the 2 HUBs have only 1 link each meanwhile Spoke has 2. Everything seems working fine, i receive the BPG adv from spoke and viceversa until i try to shutdown one of the WAN link on the spoke side.

So i have 4 tunnel configured on spoke side:
WAN 1

• HUB1-VPN1

• HUB2-VPN1

WAN 2:
- HUB1-VPN1-2
-HUB2-VPN1-2

So when i shut the WAN 2 i still see that BGP is still sending its hello packets via VPN1-2 even if the Fortigate marked bot as "down". In order to make BGP use the other active tunnels, i had to disable VPN1-2 tunnels manually.

Not sure where is the issue, but i attach the configuration of the sdwan section, maybe someone can help me figure out the issue.

config system sdwan
    set status enable
    set fail-detect enable
    config zone
        edit "virtual-wan-link"
        next
        edit "WAN1"
        next
        edit "WAN2"
        next
        edit "HUB1"
            set advpn-select enable
            set advpn-health-check "HUB1_HC"
        next
        edit "HUB2"
            set advpn-select enable
            set advpn-health-check "HUB2_HC"
        next
    end
    config members
        edit 1
            set interface "x4"
            set zone "WAN1"
        next
        edit 2
            set interface "port16"
            set zone "WAN2"
        next
        edit 3
            set interface "HUB1-VPN1"
            set zone "HUB1"
            set source 172.16.5.1
            set cost 10
        next
        edit 4
            set interface "HUB1-VPN1-2"
            set zone "HUB1"
            set source 172.16.5.1
            set cost 15
        next
        edit 5
            set interface "HUB2-VPN1"
            set zone "HUB2"
            set source 172.16.5.1
            set cost 20
        next
        edit 6
            set interface "HUB2-VPN1-2"
            set zone "HUB2"
            set source 172.16.5.1
            set cost 25
        next
    end
    config health-check
        edit "Default_DNS"
            set system-dns enable
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Default_Office_365"
            set server "www.office.com"
            set protocol http
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Default_Gmail"
            set server "gmail.com"
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 2
                next
            end
        next
        edit "Default_Google Search"
            set server "www.google.com"
            set protocol http
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Default_FortiGuard"
            set server "fortiguard.com"
            set protocol http
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "Default_AWS"
            set server "aws.amazon.com"
            set protocol http
            set interval 1000
            set probe-timeout 1000
            set recoverytime 10
            config sla
                edit 1
                    set latency-threshold 250
                    set jitter-threshold 50
                    set packetloss-threshold 5
                next
            end
        next
        edit "HUB1_HC"
            set server "172.16.5.252"
            set failtime 2
            set recoverytime 2
            set sla-fail-log-period 10
            set sla-pass-log-period 10
            set members 3 4
            config sla
                edit 1
                    set latency-threshold 255
                    set jitter-threshold 55
                    set packetloss-threshold 1
                    set priority-in-sla 1
                    set priority-out-sla 3
                next
            end
        next
        edit "HUB2_HC"
            set server "172.16.5.250"
            set sla-fail-log-period 10
            set sla-pass-log-period 10
            set members 5 6
            config sla
                edit 1
                    set latency-threshold 255
                    set jitter-threshold 55
                    set packetloss-threshold 1
                    set priority-in-sla 2
                    set priority-out-sla 4
                next
            end
        next
        edit "WAN_HC"
            set server "8.8.8.8"
            set members 1 2
            config sla
                edit 1
                    set latency-threshold 255
                    set jitter-threshold 55
                    set packetloss-threshold 1
                next
            end
        next
    end
    config service
        edit 1
            set name "ADVPN"
            set mode sla
            set dst "all"
            set src "all"
            config sla
                edit "HUB1_HC"
                    set id 1
                next
                edit "HUB2_HC"
                    set id 1
                next
            end
            set priority-members 3 4
        next
    end
end

DPD on all 4 VPNs are set "On-idle" with retry counters and interval both set as 2

EDIT: i'm using ADVPN 2.0 and BGP on loopback

[xshkurti]

According to this doc you shouldnt have multiple sdwan zones (one per interface), but the design should be to have 1 zone per underlay and 1 zone per overlay. 
Example SD-WAN configurations using ADVPN 2.0 | FortiGate / FortiOS 7.6.2 | Fortinet Document Librar...

Also, BGP configuration should be set to enable link-down detection explained in the link below:

SD-WAN support for ADVPN - Fortinet Community.

[Morus]

even if i enable the link down detection, BGP is still trying to use the dead VPNs.

Those other zones (WAN 1 and WAN 2) were created by the overlay template (thanks to FMG), but i see that only zones HUB1 and HUB2 are used in the BGP and SD-WAN health checks

[Morus · Accepted solution]

I found the issues, basically the healthchecks were configured toward the same loopback used by the hub to close the ADVPN. i think that was a misconfiguration made by the overlay template that i used on the FMG. so i replaced the healthcheck IP with the other loopback configured on the hub.

As soon i did the BGP neighborships were formed again

[maxmike1]

The issue may be with SD-WAN rules or BGP routing. Ensure your SD-WAN rules prioritize active tunnels and enable link monitoring for VPN1-2. Also, check BGP timers to speed up failover and verify routing settings for automatic backup route selection. This should help with smoother, automatic failover.