Explorer

Question

OSPF 2-Way/Exstart Stuck

Forum|Forum|3 years ago
February 13, 2023
17 replies
7151 views

We have deployment with star-topology:

HQ - cluster of FGT-300D (FortiOS 6.4.11)

Spokes - Fortigate 50E (FortiOS 6.2.x), Fortigate 60D (FortiOS 6.0.x), Fortigate 92D (6.2.x), Fortigate 300C (5.2.x)

We have experienced problem with periodically disrupt of dynamic routing (OSPF) with 20-30 spokes. From HQ side status changed to 2-Way, form spokes - ExStart

set-mtu 1300 and set mtu-ignore enable was applyed from all sides

After 3-4 hours problem was fixed by themself. And this repeat periodically

FortiGate

srajeswaran

Staff

OSPF state moves from Established to Down, when the keep-alives are missing. In your case, the state is 2 Way and not Down, which means the Hello packets are not missing but the next packets (dbd) packets are lost in transit or the MTU configuration you have applied is not getting into efffect.

Do you use GRE or some other type of interface? If so, we can try applying the MTU configuration on the specific interface and check.

Please take a pcap for OSPF (use filter protocol 89 to capture the OSPF packets only ) from both Hub and Spoke device during problem state.

srajeswaran

Staff

I understand that the issue gets fixed by itself, which means it may not be a configuration issue, but the actual packets with higher size is getting dropped in transit.

For example, lets say one of the network flaps on your HubSide, when it is sending an LSupdate packet, it may be with a higher size than what is supported by the path .

You mentioned MTU of 1300, are you able to ping with 1300bytes across the link?

Execute below from Hub and then repeat from Spoke side (change the IPs).

execute ping-options data-size 1300

execute ping-options df-bit yes

execute ping <Spoke IP>

usednetAuthor

Explorer

Thank you for so fast reply

Yes, FGT is able to ping 1300 bytes across the link

srajeswaran

Staff

Do you see any logs under "Router Events" ?

usednetAuthor

Explorer

Here is some debug flow in moment of disruption (from Spoke)

2023-02-13 08:03:46 id=20085 trace_id=55 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.249.247:0->224.0.0.5:0) from local. "
2023-02-13 08:03:46 id=20085 trace_id=55 func=resolve_ip_tuple_fast line=4552 msg="Find an existing session, id-000000b9, original direction"
2023-02-13 08:03:46 id=20085 trace_id=55 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-wan2-wan2"
2023-02-13 08:03:46 id=20085 trace_id=55 func=esp_output4 line=899 msg="encrypting, and send to 94.153.136.246 with source 31.128.74.49"
2023-02-13 08:03:46 id=20085 trace_id=55 func=ipsec_output_finish line=232 msg="send to 31.128.74.1 via intf-port2"
2023-02-13 08:03:47 id=20085 trace_id=56 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.129.247:0->224.0.0.5:0) from wan2-wan1. "
2023-02-13 08:03:47 id=20085 trace_id=56 func=resolve_ip_tuple_fast line=4552 msg="Find an existing session, id-000002dd, original direction"
2023-02-13 08:03:47 id=20085 trace_id=57 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.239.247:0->224.0.0.5:0) from local. "
2023-02-13 08:03:47 id=20085 trace_id=57 func=resolve_ip_tuple_fast line=4552 msg="Find an existing session, id-000000b8, original direction"
2023-02-13 08:03:47 id=20085 trace_id=57 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-wan1-wan2"
2023-02-13 08:03:47 id=20085 trace_id=57 func=esp_output4 line=899 msg="encrypting, and send to 94.153.136.246 with source 94.153.160.230"
2023-02-13 08:03:47 id=20085 trace_id=57 func=ipsec_output_finish line=232 msg="send to 94.153.160.229 via intf-port1"
2023-02-13 08:03:47 id=20085 trace_id=58 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.219.247:0->224.0.0.5:0) from local. "
2023-02-13 08:03:47 id=20085 trace_id=58 func=resolve_ip_tuple_fast line=4552 msg="Find an existing session, id-000000b6, original direction"
2023-02-13 08:03:47 id=20085 trace_id=58 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-wan1-wan1"
2023-02-13 08:03:47 id=20085 trace_id=58 func=esp_output4 line=899 msg="encrypting, and send to 195.24.148.14 with source 94.153.160.230"
2023-02-13 08:03:47 id=20085 trace_id=58 func=ipsec_output_finish line=232 msg="send to 94.153.160.229 via intf-port1"
2023-02-13 08:03:47 id=20085 trace_id=59 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.119.247:0->224.0.0.5:0) from wan1-wan1. "
2023-02-13 08:03:47 id=20085 trace_id=59 func=resolve_ip_tuple_fast line=4552 msg="Find an existing session, id-000002ba, original direction"
2023-02-13 08:03:48 id=20085 trace_id=60 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.229.247:0->224.0.0.5:0) from local. "
2023-02-13 08:03:48 id=20085 trace_id=60 func=resolve_ip_tuple_fast line=4552 msg="Find an existing session, id-000000b7, original direction"
2023-02-13 08:03:48 id=20085 trace_id=60 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-wan2-wan1"
2023-02-13 08:03:48 id=20085 trace_id=60 func=esp_output4 line=899 msg="encrypting, and send to 195.24.148.14 with source 31.128.74.49"
2023-02-13 08:03:48 id=20085 trace_id=60 func=ipsec_output_finish line=232 msg="send to 31.128.74.1 via intf-port2"
2023-02-13 08:03:48 id=20085 trace_id=61 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.219.247:0->224.0.0.5:0) from local. "
2023-02-13 08:03:48 id=20085 trace_id=61 func=resolve_ip_tuple_fast line=4552 msg="Find an existing session, id-000000b6, original direction"
2023-02-13 08:03:48 id=20085 trace_id=61 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-wan1-wan1"
2023-02-13 08:03:48 id=20085 trace_id=61 func=esp_output4 line=899 msg="encrypting, and send to 195.24.148.14 with source 94.153.160.230"
2023-02-13 08:03:48 id=20085 trace_id=61 func=ipsec_output_finish line=232 msg="send to 94.153.160.229 via intf-port1"
2023-02-13 08:03:49 id=20085 trace_id=62 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.229.247:0->224.0.0.5:0) from local. "
2023-02-13 08:03:49 id=20085 trace_id=62 func=resolve_ip_tuple_fast line=4552 msg="Find an existing session, id-000000b7, original direction"
2023-02-13 08:03:49 id=20085 trace_id=62 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-wan2-wan1"
2023-02-13 08:03:49 id=20085 trace_id=62 func=esp_output4 line=899 msg="encrypting, and send to 195.24.148.14 with source 31.128.74.49"
2023-02-13 08:03:49 id=20085 trace_id=62 func=ipsec_output_finish line=232 msg="send to 31.128.74.1 via intf-port2"
2023-02-13 08:03:50 id=20085 trace_id=63 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.239.247:0->224.0.0.5:0) from local. "
2023-02-13 08:03:50 id=20085 trace_id=63 func=resolve_ip_tuple_fast line=4552 msg="Find an existing session, id-000000b8, original direction"
2023-02-13 08:03:50 id=20085 trace_id=63 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-wan1-wan2"
2023-02-13 08:03:50 id=20085 trace_id=63 func=esp_output4 line=899 msg="encrypting, and send to 94.153.136.246 with source 94.153.160.230"
2023-02-13 08:03:50 id=20085 trace_id=63 func=ipsec_output_finish line=232 msg="send to 94.153.160.229 via intf-port1"
2023-02-13 08:03:50 id=20085 trace_id=64 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.249.247:0->224.0.0.5:0) from local. "
2023-02-13 08:03:50 id=20085 trace_id=64 func=resolve_ip_tuple_fast line=4552 msg="Find an existing session, id-000000b9, original direction"
2023-02-13 08:03:50 id=20085 trace_id=64 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-wan2-wan2"
2023-02-13 08:03:50 id=20085 trace_id=64 func=esp_output4 line=899 msg="encrypting, and send to 94.153.136.246 with source 31.128.74.49"
2023-02-13 08:03:50 id=20085 trace_id=64 func=ipsec_output_finish line=232 msg="send to 31.128.74.1 via intf-port2"
2023-02-13 08:03:52 id=20085 trace_id=65 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.239.247:0->224.0.0.5:0) from local. "
2023-02-13 08:03:52 id=20085 trace_id=65 func=resolve_ip_tuple_fast line=4552 msg="Find an existing session, id-000000b8, original direction"
2023-02-13 08:03:52 id=20085 trace_id=65 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-wan1-wan2"
2023-02-13 08:03:52 id=20085 trace_id=65 func=esp_output4 line=899 msg="encrypting, and send to 94.153.136.246 with source 94.153.160.230"
2023-02-13 08:03:52 id=20085 trace_id=65 func=ipsec_output_finish line=232 msg="send to 94.153.160.229 via intf-port1"
2023-02-13 08:03:52 id=20085 trace_id=66 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.219.247:0->224.0.0.5:0) from local. "
2023-02-13 08:03:52 id=20085 trace_id=66 func=resolve_ip_tuple_fast line=4552 msg="Find an existing session, id-000000b6, original direction"
2023-02-13 08:03:52 id=20085 trace_id=66 func=ipsecdev_hard_start_xmit line=121 msg="enter IPsec interface-wan1-wan1"
2023-02-13 08:03:52 id=20085 trace_id=66 func=esp_output4 line=899 msg="encrypting, and send to 195.24.148.14 with source 94.153.160.230"
2023-02-13 08:03:52 id=20085 trace_id=66 func=ipsec_output_finish line=232 msg="send to 94.153.160.229 via intf-port1"
2023-02-13 08:03:54 id=20085 trace_id=67 func=print_pkt_detail line=4489 msg="vd-root received a packet(proto=89, 172.30.139.247:0->224.0.0.5:0) from wan1-wan2. "

abarushka

Staff

Hello,

I can see retransmits in the OSPF logs:

2023-02-13 08:05:22 id=20301 logdesc="Routing log" msg="OSPF: NFSM[wan1-wan1:172.30.219.247-0.0.0.1]: DD Retransmit timer expire

I suspect that the issue is caused by packets drops on IPsec tunnels. In order to isolate the issue you may consider to perform steps below:

- check whether tunnels were going down when the issue was triggered

- in case tunnel didn't go down you may consider to sniff traffic (OSPF) on hub and spoke and check whether OSPF hello packets were sent/received by hub and spoke

- in case OSPF Hello packets were sent but not received you may consider to run the commands below on hub and spoke and check for the errors:

diagnose netlink interface list <Phase 1 name>
get vpn ipsec tunnel details
diag npu <NP type> dce <index>

usednetAuthor

Explorer

Today's disruption affected all spokes... All OSPF-interfaces entered in 2-Way (HQ) and ExStart (Spokes) status and was until admin initiate reboot of HQ cluster.

After that problem has been fixed

How can I add log files to this post?

srajeswaran

Staff

You can use the attachment option while replying.