|
For each FortiGate/FortiProxy model there are limits for the maximum numbers of objects, for example the number of firewall policies.
Making the example of a FortiGate 4200F, the maximum number of configurable firewall policies is 400 000, but already when a couple of tens of thousands of firewall policies, for example more than 20 000, are configured, can be observed CPU's cores spikes, after a configuration change.
Here an example of CPU’s cores spikes in a FortiGate, during the first seconds after a firewall policy enable:
FGT-4200F-01 (global)# diagnose sys top 5 10 3
Run Time: 0 days, 1 hours and 2 minutes
3U, 0N, 2S, 95I, 0WA, 0HI, 0SI, 0ST; 387725T, 360904F
wad 17633 R 99.0 0.1 8
cmdbsvr_iprope 18245 R 99.0 0.0 0
cmdbsvr_cfgsave 18246 R 98.5 0.0 6
iked 14695 R 75.1 0.1 79
voipd 14693 R 24.3 0.1 72
bcm.user 3605 S < 6.4 0.0 36
ipsengine 17913 S < 0.4 0.0 74
hatalk 14702 S < 0.4 0.0 30
newc 18247 R < 0.4 0.0 39
wad 14694 S 0.0 0.1 32
With spikes on cores: 0, 6, 8 and 79.
This behavior can be observed in High level FortiGates, with different tens of thousands of firewall policies configured.
cmdbsvr_iprope and cmdbsvr_cfgsave are the daemons managing the unit configuration and for each change they need also to check the consistency of the rest of the configuration, so for them spikes are expected.
The reason of spikes of wad (manage traffic proxing) and iked (manage all VPN IPSec tunnels), is that they need to read the new configuration to check if a change require to negate or permit a new or a existing flow of packets and to perform the 2 activities they need need to read the new configuration and update the sessions table.
Spikes can be observed also in other daemons and of course wad peaks can be observed also in FortiProxy.
Duration of spikes depends on multiple factors:
- The number of firewall policies configured.
- The quantity of traffic the unit is inspecting.
- FortiOS firmware version running.
- Number of VPN IPsec tunnels configured.
and others, should last between 15 seconds to a couple minutes.
Duration depend also from the firmware release running, being available improvements in newest:
- A New Feature with ID 0898200 for the cmdb daemon, contained in the FortiOS GA releass 7.4.2 GA and all newer releases.
- The fixes for the bugs 1096537 and 1173177, contained in the FortiOS GA releases 7.4.9GA and 7.6.4 GA all newer releases.
The wad and the iked peaks can cause different issues like:
- Packet loss for traffic proxate.
- VPN IPSec frequent rekey causing flaps on eventually routing protocols like BGP implemented over it and in case of SD-WAN deployments causing also SD-WAN behavior changes.
In case of FortiGates in High Availability (FGCP Cluster), peaks are observed in all cluster members, but for a longer period and involving additional daemons beyond cmdb and HA daemons, only into the primary unit.
Here is a list of precautions to mitigate this issue:
- Periodically review FortiGate configurations, deleting unused firewall policies and objects.
- In very big network environments, like for example intercontinental companies, avoid to use a FortiGate for 2 or more of these purposes: Data Center Firewall, Perimeter Firewall, SD-WAN Hub, Controller of thousands of FortiSwitches and FortiAPs, root of a Fortinet Fabric with thousands of units and so on.
- In case FortiGate HA Cluster used for different scopes, manage them with different VDOMs, using the High Availability Virtual Clusters feature to use different members of the Cluster as primary unit for different VDOMs.
- Make all FortiGates configuration changes into the FortiManager and only in case of urgent changes push them to the FortiGates during business hours.
Correlated docs and articles:
Troubleshooting Tip: WAD CPU spikes due to configuration changes.
Maximum number of objects configurable for each FortiGate model: Fortinet Max Value Table and Technical Tip: FortiGate maximum values table.
For more information on how the command 'diagnose sys top' works, see Technical Tip: Using the diagnose sys top CLI command.
Activities performed by the most important FortiGate’s daemons: Technical Tip: Short list of processes on the FortiGate.
High CPU Troubleshooting guide: Troubleshooting Tip: How high CPU usage should be investigated.
|
