Technical Tip: How to verify a frequent route change that may be causing high CPU usage

[fwilliams]

Description	This article explains how to verify a frequent route change that may be causing high load on the CPU’s softirq.
Scope	FortiOS.
Solution	If a FortiGate unit with a high number of sessions (an environment that has active sessions in the millions) and multiple WAN links is found to be experiencing high CPU (softirq) intermittently, it is worth verifying if the routing is changing frequently. The FortiGate session table keeps valuable information regarding a connection it established between two devices (e.g. source IP, destination IP, source port, destination port, FW policy ID, incoming interface, outgoing interface, session ID, gateway IP address, SNAT information, DNAT information, NPU offload status, protocol etc.) and as packets are flowing through it from these devices, FortiGate continues to check its session table against the packets to see if conditions are still the same as they were when the contract was first signed (i.e. when FortiGate first evaluated the traffic for admission into the session table). If a change is detected by FortiGate, re-evaluation is required and if the number of sessions that qualify for re-evaluation is large (in millions). This can cause a high CPU load. To verify if the routing information is changing frequently, check the device FIB version (forwarding information based) with 'diagnose sys vd list'. The cmd will return the FIB version for every VDOM if the unit is configured to operate in multi-VDOM mode, or return an FIB version for only the root VDOM if it is operating in standalone mode. The version number will increase with every route change. See the following screenshots:   Note: it is possible to track the fib version and policy route version as of FortiOS 7.x. In earlier versions, only the fib version is available. All of the sessions in the FortiGate’s session table are not the same in characteristics. As such, there are conditions FortiGate checks before making the decision to 'mark' a session as 'dirty' from 'may_dirty' during or after a routing change. In newer FortiOS versions (7.x and above), only sessions related to the routing change are changed to 'dirty'. On older FortiOS versions (6.x and below) with the possible exception of 6.4.10 and above (which are more recent), all sessions were changed to 'dirty' on non-NPU platforms, and sessions were offloaded onto the NPU platform.