Technical Tip: Intermittent traffic drops on FortiGate 200G and 201G SFP ports after firmware upgrade

Description

This article describes an issue affecting FortiGate 200G and 201G devices, where intermittent internet connectivity outages may occur after upgrading from FortiOS v7.4.9 to v7.4.10, v7.6.5, or above.

Symptoms are characterized by a total loss of both inbound and outbound traffic connectivity on the SFP interfaces, specifically, despite the physical link status remaining 'up'.

Scope

 FortiGate 200G and 201G.

Solution

This issue was investigated as part of Bug #1264495, and it was found that the issue only impacts the SFP interfaces (specifically port17 through port20). This issue does not impact the SFP+ interfaces (such as x1 through x8), nor does it impact the RJ45 interfaces.

The root cause was identified as an implementation issue with setting interface speeds. When the SFP interfaces are configured to auto-negotiate interface speed, an intermittent issue can randomly occur that results in traffic no longer flowing through these interfaces for long periods of time (at least several minutes) before resuming without any admin intervention. This issue has been observed to occur on FortiOS v7.4.10 and later, as well as in FortiOS v7.6.5 and later.

Notably, packet captures on the FortiGate may indicate that ARP traffic is being transmitted on these interfaces, but these packets do not end up being sent to connected devices, resulting in communication failures. This issue is also unrelated to hardware acceleration, and so disabling hardware offloading (e.g., set auto-asic-offload disable) will not have a beneficial impact in this case.

Verifying the Issue:

Perform a packet capture on both the FortiGate and the connected device (router, switch with SPAN/mirror ports, client, etc.) to verify if the connected device is actually seeing ARP replies or any other reply traffic sent back from the FortiGate. Additionally, the FortiGate SFP interface state will likely show as 'up' in this case without any interface errors being observed.

Resolution:

As per Bug ID 1264495, this issue has been resolved as of FortiOS v7.4.12, v7.6.7, and v8.0.0, so upgrade to the listed versions (or any later revision) to fully resolve the issue.

Workarounds:

The following are workaround options that can be tried if it is not immediately possible to upgrade the firmware to a resolved version:

Set the interface speed manually on both the FortiGate SFP interface(s) and the connected device interface, rather than using auto-negotiation (e.g., change 1000auto to 1000full). This should prevent the issue from occurring and will grant time to prepare a firmware upgrade:

config system interface
    edit <name>
        set speed 1000full
    next
end

If an outage is actively occurring, then a temporary resolution is to administratively flap the SFP interface on the FortiGate (aka, bringing the interface down and then back up). Take care that admin access is not being handled through this interface during this process; otherwise, it will be disrupted. Alternatively, rebooting the FortiGate will also temporarily resolve the issue:

FortiGate # config system interface
FortiGate (interface) # edit port17
FortiGate (port17) # set status down
FortiGate (port17) # next
FortiGate (interface) # edit port17
FortiGate (port17) # set status up
FortiGate (port17) # end