BGP Flapping / Carrier issue

[RJ1]

I am running a SDWAN infrastructure with one Hub and several spoke (15 spokes). All sites are equipped with FGT  clusters running 7.2.x. we have ADVPN configured with iBGP running over it for route advertisement .On 4-5 spoke sites  I have a strange behavior that is every 2-3 days   have this error in router events logs as below:

BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.168.88.254 Down Hold Timer Expired

BGP: %BGP-3-NOTIFICATION: sending to 192.168.88.254 4/0 (Hold Timer Expired/Unspecified Error Subcode) 0 data-bytes []

BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.168.88.254 Down BGP Notification FSM-ERR

BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor192.168.88.254 Up

The problem is that during this event all connections from spoke to hub are shut during roughly 60 or 120 sec

[Toshi_Esumi]

I simply googled with one of messages and found this. Nothing to do with ADVPN though. But it was inconclusive.
https://community.fortinet.com/t5/Support-Forum/BGP-instability-on-IPSEC-tunnel/m-p/65094

Toshi

[RJ1]

Yes I have gone thru this but as you said it was inconclusive

[Toshi_Esumi]

Because the OP didn't update after my comment.
I also found a KB below in another google search.
https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-BGP-Hold-Timer-Expired-Unspecified-E...

Toshi

[Toshi_Esumi]

But ultimately you need to run a packet capture on TCP 179 on both remote and local sides to capture the moment when that happens. That would tell exactly what was sent but what was not received on the other side in the BGP communication.

Toshi

[RJ1]

It's difficult to perform live packet capture as the incident occurs anytime of the day and gets resolved within 2-3 minutes.

[DUSTINREEVES]

Hey, I'm see'ing the same behavior in my logs, did you ever find a resolution to this issue ?

[FredJamesPTY]

I have the same problem, how did you solve it?

[DUSTINREEVES]

a good chunk of our issues boiled down to having redistributed connected/redistributed static turned on, and we had our HUB sites broadcasting their own routes across the ADVPN link tunnel, so it would try and send traffic to the public ip address, over the IPSEC interface, instead of straight out the WAN interface to the internet.

[Toshi_Esumi]

Then you just need to filter those unwanted ones out and filter in only wanted ones with a proper route-map.
I would filter them at the redistribution not going into BGP table, instead of filtering at the neighbor over IPsecs.

Toshi