External interface drops every 10 minutes

[Jim_from_NY]

Good day all, having a weird issue, was hoping for suggestions as to where to look for the cause.

Have a FortiGate-61E v7.2.8,build1639,240313.

The WAN interface connected to the ISP switch drops every 10 minutes, almost as though scheduled, on the hour, 10, 20, 30, 40, and 50 minutes after the hour. It drops for about 20-30 seconds, then comes back up. I lose pings to that interface from the outside. 

The ISP switch also has a Verizon router (different external IP). This connection does not drop.

The Fortigate also has a connection to a Velocloud (which also connects to the ISP switch) for an SD-WAN connection. This connection does not drop. 

The "get system interface physical [interface name]" command shows the interface never going down physically (and I never lose the remote webui connection, but that's using the SD-WAN route).

We've rebooted the router, tried different DNS servers for the dns probe health check, tried to use pings for the health check, disabled SLAs, nothing seems to make a difference. Any suggestions as to what else I can check that could be scheduled to cause such specifically timed drops for that interface?

[kumarh]

Could you share the default SLA configuration, this issue could be related to the Default_Office_365 performance SLA rule because Microsoft have stopped listening on port 80. If you have set to HTTP then change it to ping or other SLA.

Can you collect ICMP packet capture on the machine and on Fortigate interface and reproduce the issue. Also, check what is the memory and CPU utilization when traffic is being dropped.

#fnsysctl ifconfig <wan1> --------------> This command will show you interface drops and error on Fortigate interface

[Jim_from_NY]

Thank you for your replies.

SLA rules are same for all of our sites and includes the default office. But all are set to check at 1000ms. We tried even disabling all of the SLAs.

Name	Detect Server	Protocol
Check_DNS	8.8.8.8, 8.8.4.4	DNS
Default_AWS	http://aws.amazon.com/	HTTP
Default_DNS	96.45.45.45, 96.45.46.46	DNS
Default_FortiGuard	http://fortiguard.com/	HTTP
Default_Gmail	gmail.com	Ping
Default_Google Search	http://www.google.com/	HTTP
Default_Office_365	http://www.office.com/	HTTP

Description :FortiASIC NP6LITE Adapter
Driver Name :FortiASIC NP6LITE Driver
Board :61E
lif id :9
lif oid :73
netdev oid :73
tx group :0
========== Link Status ==========
Admin :up
netdev status :up
autonego_setting:1
link_setting :1
speed_setting :10
duplex_setting :0
Speed :1000
Duplex :Full
link_status :Up
============ Counters ===========
Rx Pkts :9359915
Rx Bytes :6174089288
Tx Pkts :7516431
Tx Bytes :1740950199
Host Rx Pkts :3857876
Host Rx Bytes :1769420568
Host Tx Pkts :3661276
Host Tx Bytes :666167627
Host Tx dropped :0
FragTxCreate :0
FragTxOk :0
FragTxDrop :0

And the drops happen every 10 minutes on a schedule:

 

 

Nothing unusual with CPU/Memory at the times

I took a capture when no one is in the office: During the drop we get a bunch of the retransmissions of the health check packets, but eventually we start getting responses again after about 10-20 seconds (hiding our source external IP on the capture):

 

 

 

 

[JameKenn]

I've a Fortigate 60F at v7.6.0build3401 exhibiting this same behavior, but would like to validate with a graph display like the latency gaps you have here.  How did you produce the graph display?  Is the graph a function on Fortigate unit?  Afterwards, I'll escalate this into a ticket for my unit.

[Jim_from_NY]

Yes, for me it's through the Forti's web ui SD-WAN Performance SLAs

 

[JameKenn]

Yup, I see it now.  Thank you.  Btw, have you found a resolution to this issue?

[Jim_from_NY]

For me, it was a duplicate external IP on that WAN interface (the IP that was configured on the interface that was dropping was also defined elsewhere as a NAT IP for an internal address that was no longer being used). Changing it to the next available IP resolved the issue.

[JameKenn]

I just submitted a ticket to help me resolve my issue.  Your original post is exactly what I am experiencing.  My ISP provider has been to our place and has done extensive testing and replacement of their modem and all premise cabling to try to fix this issue.  To no avail, issue is persistent.  I've also done my fair share of troubleshooting on my side.  Have a decent piece of hardware (Fortigate), but I'm inclined to having some higher grade assistance.  This issue is definitely above my pay grade to tackle alone.  Jim_from_NY, this is Jim_from_WA saying a great big Thank You for your replies.  I'll post a comment here when I resolve this issue.

[arahman]

Hi, also here is the article to check the packet loss, please try this 

https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Diagnose-Packet-Loss/ta-p/192459