OSPF 2-Way/Exstart Stuck

[usednet]

We have deployment with star-topology:

HQ - cluster of FGT-300D (FortiOS 6.4.11)

Spokes - Fortigate 50E (FortiOS 6.2.x), Fortigate 60D (FortiOS 6.0.x), Fortigate 92D (6.2.x), Fortigate 300C (5.2.x)

We have experienced problem with periodically disrupt of dynamic routing (OSPF) with 20-30 spokes. From HQ side status changed to 2-Way, form spokes - ExStart

set-mtu 1300 and set mtu-ignore enable was applyed from all sides 

After 3-4 hours problem was fixed by themself. And this repeat periodically

[srajeswaran]

OSPF state moves from Established to Down, when the keep-alives are missing. In your case, the state is 2 Way and not Down, which means the Hello packets are not missing but the next packets (dbd) packets are lost in transit or the MTU configuration you have applied is not getting into efffect.

Do you use GRE or some other type of interface? If so, we can try applying the MTU configuration on the specific interface and check.

Please take a pcap for OSPF (use filter protocol 89 to capture the OSPF packets only ) from both Hub and Spoke device during problem state.

[srajeswaran]

I understand that the issue gets fixed by itself, which means it may not be a configuration issue, but the actual packets with higher size is getting dropped in transit.

For example, lets say one of the network flaps on your HubSide, when it is sending an LSupdate packet, it may be with a higher size than what is supported by the path .

You mentioned MTU of 1300, are you able to ping with 1300bytes across the link?

Execute below from Hub and then repeat from Spoke side (change the IPs).

execute ping-options data-size 1300

execute ping-options df-bit yes

execute ping <Spoke IP>

[usednet]

Thank you for so fast reply

Yes, FGT is able to ping 1300 bytes across the link

[abarushka]

Hello,

I would like to ask whether OSPF is configured over IPsec?

[usednet]

Yes, OSPF over IPsec

We have 2 ISP in HQ and 2 ISP in Spokes. So we have 4 IPSec tunnels between (with different costs in OSPF)

[usednet]

We do not have GRE, only IPSec between Fortigates

[abarushka]

Hello,

As far as I understand OSPF traffic is sent over IPsec. If it is the case, then there is higher probability that the issue is caused by IPsec than OSPF itself.

I would check whether IPsec tunnels are flapping and sniff OSPF traffic on both OSPF peers when the issue is triggered.

[usednet]

In most cases this situation occurs when Fortigate (spokes) was rebooted. After 3-4 hours after reboot OSPF starts work (without any admin interactions).

But IPSec tunnels starts working with Fortigate

[usednet]

And one more moment - when HQ has FortiOS 6.0.14 - there are no problem at all