Troubleshooting Tip: SD‑WAN Overlay Instability  causing BGP Flapping and Loss of Connectivity

Description

This article describes how BGP instability and route flapping can occur in an SD-WAN overlay (ADVPN) environment.

When an SD-WAN overlay (ADVPN) is used, each site establishes a BGP session over the encrypted tunnel. If the overlay experiences instability, for example, member interfaces repeatedly transitioning up/down, health-check SLA failures, or packet loss across the tunnel, the BGP peer relationship may begin flapping.

These repeated BGP reset and re-establishment cycles can generate route flaps, trigger frequent BGP table updates, and potentially disrupt traffic forwarding between sites.

Scope

FortiGate.

Solution

Root causes are usually one (or a combination) of:

• Physical/underlay link issues (packet loss, high latency, jitter).

• Incorrect SD‑WAN health‑check configuration (SLAs too tight).

• BGP timers too aggressive (hold‑time/keep‑alive) → excessive updates.

• Missing or mismatched BFD on the overlay tunnel.

• ADVPN‑specific problems (loopback‑based BGP peering, missing route‑maps).

Section 1: Verify SD‑WAN member health.

Confirms whether the member is Up, Down or Unknown by running the below command:

diagnose sys sdwan member list

Or GUI → Network → SD‑WAN → Members.

Section 2: Check health‑check SLA thresholds.

Over‑aggressive thresholds cause the member to flip to Down → BGP flaps.

show system sdwan health-check

Section 3: Inspect the underlying tunnel.

Packet loss, high jitter, or frequent re‑keying on the IPsec tunnel will trigger BGP resets.

diagnose vpn tunnel list
diagnose vpn tunnel statistics

Section 4: Enable BFD on the tunnel (optional but recommended).

BFD detects a dead tunnel in milliseconds and can keep BGP stable by preventing unnecessary session resets.

config system bfd
edit <tunnel‑name>
set enable enable
set interval 300
set multiplier 3
next
end

Section 5: Review BGP session status & timers.

Verify hold‑time, keep‑alive, and graceful‑restart settings. If the hold‑time is too low, a momentary lapse will cause a flap.

get router info bgp neighbor
show router bgp

Section 6: Adjust BGP timers if needed.

Larger timers absorb brief outages without generating a flap.

config router bgp
config neighbor
edit <peer‑IP>
set holdtime-timer 180
set keepalive-timer 60
next
end
end

Section 7: Enable route‑dampening & Enable graceful‑restart (In case of HA or expect short maintenance).

Suppresses propagation of flapping routes, reduces CPU on peers.

The neighbour keeps using existing routes while the local BGP process restarts, avoiding a full flap.

config router bgp
set dampening enable
set dampening-max-suppress-time 30
end
!
config router bgp
set graceful-restart enable
set graceful-restart-time 120
set graceful-stalepath-time 180
end

Section 8: Monitor the overlay after changes.

Ensure the flapping stops and traffic flows normally.

diagnose sys top (watch CPU, interface‑missed)
diagnose debug flow filter add <src‑IP> (if needed)
diagnose debug enable

Section 9: If flapping persists, collect a diagnostic dump for Fortinet Support for deeper analysis.

execute backup config (save config)
diagnose debug crashlog read
diagnose debug application bgpd 255
diagnose debug enable – reproduce issue – diagnose debug disable

Additional Tip:

• Use BGP per‑overlay. In large deployments, configure a separate BGP AS or VRF for each overlay to isolate flaps to that overlay only.

• Avoid overlapping IP ranges. Overlapping subnets can cause route leakage that appears as flaps.

• Check MTU consistency. Mismatched MTUs on underlay links often lead to packet loss over IPsec, which then triggers BGP resets.

• Synchronise time NTP drift can cause health‑check timestamps to be misinterpreted, leading to premature down events.

• Log collection: Enable BGP logs (set log-bgp enable) and forward them to FortiAnalyzer – the logs contain the exact cause of each session reset.