Technical Tip: How to verify a frequent route change that may be causing high CPU usage

[fwilliams]

Description	This article explains how to verify a frequent route change that may be causing high load on the CPU’s softirq.
Scope	FortiOS.
Solution	If a FortiGate unit with a high number of sessions (an environment that has active sessions in the millions) and multiple WAN links is found to be experiencing high CPU (softirq) intermittently, it is worth verifying if the routing is changing frequently. The FortiGate session table keeps valuable information regarding a connection it established between two devices (e.g. source IP, destination IP, source port, destination port, FW policy ID, incoming interface, outgoing interface, session ID, gateway IP address, SNAT information, DNAT information, NPU offload status, protocol etc.) and as packets are flowing through it from these devices, FortiGate continues to check its session table against the packets to see if conditions are still the same as they were when the contract was first signed (i.e. when FortiGate first evaluated the traffic for admission into the session table). If a change is detected by FortiGate, re-evaluation is required and if the number of sessions that qualify for re-evaluation is large (in millions). This can cause a high CPU load. To verify if the routing information is changing frequently, check the device FIB version (forwarding information based) with 'diagnose sys vd list'. The cmd will return the FIB version for every VDOM if the unit is configured to operate in multi-VDOM mode, or return an FIB version for only the root VDOM if it is operating in standalone mode. The version number will increase with every route change. See the following screenshots:   Notes: it is possible to track the fib version and policy route version as of FortiOS 7.x. In earlier versions, only the fib version is available. There are corner cases with routes changes with no fib_ver value increasing. This issue is tracked under the bug ID 897308, and is resolved in FortiOS versions 7.4.8, 7.6.3 and above. It does not happen with all older FortiOS releases: it also depends on the FortiGate model. To confirm that this issue is happening, print the routing table at different times and verify that there is at least one route change (a new route, a route missing, a route always with the age not increasing and so on) without a fib_ver update. For more information regarding the FortiGate routing table, see Viewing the FortiGate routing table. The system fib version should increase after each fib_ver version increase and the fib_ver should increase when the system fib increases. If the system fib increases without the fib_ver increasing, this is incorrect behavior due to the bug mentioned above. The system fib version may show a negative number like system fib version=-1486343407. This is only a cosmetic bug with no impact, logged under bug ID 1233725. It will be fixed starting from FortiOS 8.0.0.  All of the sessions in the FortiGate's session table are not the same in characteristics. As such, there are conditions FortiGate checks before making the decision to 'mark' a session as 'dirty' from 'may_dirty' during or after a routing change. In newer FortiOS versions 7.0.16, 7.2.8 and 7.4.4 and above, only sessions related to the routing change are changed to 'dirty' (for reference, this change tracked under bug ID 970179 is explained here: FortiOS 7.2.8 resolved issues). On older FortiOS versions (6.x and below) with the possible exception of 6.4.10 and above (which are more recent), all sessions were changed to 'dirty' on non-NPU platforms, and sessions were offloaded onto the NPU platform. For an explanation of how to manage high CPU issues, see Troubleshooting Tip: How high CPU usage should be investigated.