|
After upgrading to v7.2.9 the IPS Engine 7.00342 triggers a High CPU usage on the FortiGate.
Known issues v7.2.10
The issue is tracked in the internal engineering ticket 1069190.
The problem is resolved in the IPS engine versions 7.00345, 7.00551, 7.01021, and above.
IPS Engine 7.00345 is released with FortiOS 7.2.11, IPS Engine 7.00551 is released with FortiOS v7.4.6, and IPS Engine 7.01021 is released with FortiOS v7.6.1.
For more details related to the IPS Engine upgrade process follow this article: Technical Tip: How to manually upgrade the IPS Engine
To upgrade the IPS Engine version to 7.00345 create a new ticket with TAC Support to request the correct IPS Engine file.
To confirm the current FortiOS version through the CLI run:
get system status
Version: FortiGate-80F v7.2.9,build1688,240813 (GA.M)
To confirm the IPS engine version in use run:
get system auto-update versions
IPS Attack Engine
---------
Version: 7.00342 signed
To confirm the problem run the below commands. The output of the 'diag sys top' command will show IPS engines having a higher CPU load:
Run Time: 0 days, 19 hours and 19 minutes
22U, 0N, 44S, 11I, 0WA, 0HI, 23SI, 0ST; 3614T, 849F
ipsengine 5868 R < 99.0 3.2 7
ipsengine 5876 R < 97.5 3.2 6
ipsengine 5857 R < 97.0 3.3 4
ipsengine 5862 R < 96.5 3.2 0
ipsengine 5872 R < 96.0 3.2 2
ipsengine 5848 R < 95.5 3.2 3
ipsengine 5852 R < 82.6 3.2 5
The CPU load as seen with the command 'get sys perf stat' is mostly in system space:
CPU states: 25% user 44% system 0% nice 9% idle 0% iowait 0% irq 22% softirq
CPU0 states: 29% user 56% system 0% nice 1% idle 0% iowait 0% irq 14% softirq
CPU1 states: 30% user 3% system 0% nice 58% idle 0% iowait 0% irq 9% softirq
CPU2 states: 27% user 53% system 0% nice 1% idle 0% iowait 0% irq 19% softirq
CPU3 states: 17% user 44% system 0% nice 2% idle 0% iowait 0% irq 37% softirq
CPU4 states: 24% user 55% system 0% nice 1% idle 0% iowait 0% irq 20% softirq
CPU5 states: 22% user 22% system 0% nice 6% idle 0% iowait 0% irq 50% softirq
CPU6 states: 21% user 51% system 0% nice 2% idle 0% iowait 0% irq 26% softirq
CPU7 states: 32% user 66% system 0% nice 2% idle 0% iowait 0% irq 0% softirq
At this point, run the CPU profiler to collect more information on process usage, commands below:
diag sys profile cpumask <ID> <----- If all CPUs are busy, then do not need to run this. Otherwise specifying busying CPU ID.
diag sys profile start
<wait 5-10 seconds>
diag sys profile stop
diag sys profile show order
diagnose sys profile show detail
diagnose sys profile sysmap
Collect the above debugs which can be provided to TAC if the case is opened.
The issue is triggered when Application Control is enabled in a proxy-based inspection mode firewall policy and the 'Inspect-all' feature is enabled in either a deep inspection ssl-ssh-profile or in the profile-protocol-options.
Requirements to trigger the problem:
- The firewall policy is in Proxy Mode:
config firewall policy
edit 1
set inspection-mode proxy <---
- Application Control profile enabled in the Firewall Policy:
config firewall policy
set application-list "app_control" <---
config firewall policy
config firewall ssl-ssh-profile
edit "deep_inspect_and_inspect_all"
config SSL
set inspect-all deep-inspection <---
- Or the option 'inspect-all' is enabled in the protocol options:
config firewall profile-protocol-options
edit "custom-default"
set comment "all default services"
set oversize-log enable
config HTTP
set inspect-all enable <---
A workaround is to remove one of these conditions above.
Another workaround is to downgrade the IPS engine from 7.00342 to IPS Engine 7.00341 or below:
- Before the downgrade, enable the auto-update downgrade with the below command:
diagnose autoupdate downgrade enable
|
