Technical Tip: FortiGate-70G/71G experiencing unexpected reboots after upgrading to FortiOS v7.6.5 or later (known issue)

Description

This article describes a known issue where the FortiGate-70G/71G models are experiencing unexpected and recurring reboots after upgrading to FortiOS v7.6.5 or later. The FortiGate-50G/51G also appears to experience a similar issue, though the issue does not impact any FortiGate that is running FortiOS v7.4, v7.6.4, or any earlier versions.

Scope

FortiGate.

Solution

After upgrading to FortiOS v7.6.5 or later, the FortiGate-70G may experience kernel panics when performing NTurbo-accelerated flow-based security inspection (for more info regarding NTurbo in general, refer to the following: NTurbo offloads flow-based processing).

This issue is actively being investigated via Issues #1259458 and #1260236. To confirm if this issue is occurring in an environment, it is necessary to set up a serial console connection to the FortiGate and log the console output when a reboot occurs. If kernel panic output is indeed produced at the same time that a reboot occurs, then submit the logged output to Fortinet TAC for further analysis and confirmation. The following KB article has more information on how to perform this serial console-based data-gathering procedure: Troubleshooting Tip: How to deal with a Kernel panic.

When submitting the output of the Kernel panic to Fortinet TAC, submit also the output of FortiGate CLI commands 'execute tac report' and 'diagnose alertconsole list'.

Workarounds:

As noted in the description, this issue impacts the FortiGate-70G/71G and the FortiGate-50G/51G models at this time, and only when running FortiOS v7.6.5 or later (other models are not currently known to be impacted by this issue at this time).

As a workaround, consider one of the following options:

Option 1: Disable NTurbo hardware-acceleration for flow-based inspection to avoid triggering the kernel panics while the issue is investigated. To disable NTurbo, run the commands described below:

• It is strongly recommended to disable this at a global level rather than a per-policy level; the issue may still occur. Refer to the NTurbo documentation linked above to learn more about NTurbo and what the impacts are for disabling it.

• This option works well for clients who are already on the v7.6 firmware branch (e.g., upgrading from v7.6.4 to v7.6.6) since it preserves vulnerability fixes while also reducing/eliminating the NTurbo-related kernel panics discussed in this article.

config ips global
    set np-accel-mode none
end

Option 2: If the FortiGate was recently upgraded to v7.6.5 or later, another option is to roll back to the previous firmware version.

• Before pursuing this option, consider any vulnerabilities that existed in the previously used firmware version. For clients running FortiOS v7.6 on the pre-upgrade firmware, it may be more beneficial to stay on v7.6.5 or later and utilize Option 1 above.

• Otherwise, for FortiGate devices that were previously running v7.4 or earlier, rolling back firmware may be a good option to retain the performance benefits of NTurbo acceleration. However, after performing the rollback, it is recommended to consider upgrading to at least the latest patch release for the firmware branch being used, if that was not already done (for example, if rolling back from v7.6.6 to v7.4.9, consider upgrading from v7.4.9 to v7.4.11 afterwards to address any existing vulnerabilities).

• This article describes options for firmware rollback/downgrades, though take note of the initial recommendations in the Solutions section (i.e., use the rollback or TFTP fresh-install options if available, otherwise downgrade only if no other options remain): Technical Tip: FortiGate Firmware Downgrade for Minor Releases.

Resolution.

The fix for both issues (1259458 and 1260236) is included in FortiOS v7.4.13, v7.6.7, and v8.0.1.