|
The System event generally records the CPU usage every 5 minutes by default, as seen in the example below:
date=2024-01-08 time=09:41:11 eventtime=1704703271275848977 tz="+0100" logid="0100040704" type="event" subtype="system" level="notice" vd="root" logdesc="System performance statistics" action="perf-stats" cpu=1 mem=43 totalsession=33 disk=1 bandwidth="10/208" setuprate=0 disklograte=0 fazlograte=0 freediskstorage=15172 sysuptime=523228 waninfo="N/A" msg="Performance statistics: average CPU: 1, memory: 43, concurrent sessions: 33, setup-rate: 0"
However, the CPU usage recorded above is the average CPU usage seen across all the available CPU cores.
Hence, a single CPU core spike may get overlooked on a FortiGate with multiple CPU cores.
Since v7.2.4, a new feature 'log-single-cpu-high' was implemented to allow the logging of a single core once it reaches a configurable threshold.
To log any CPU usage spike seen against a particular core, use the following command:
config system global
set log-single-cpu-high enable
end
log-single-cpu-high: Enable/disable logging in the event of a single CPU core reaching the CPU usage threshold.
The CPU usage threshold is by default 90. It would then generate logs as shown in the sample below if the usage exceeds 90%:
date=2024-01-08 time=09:45:44 eventtime=1666143944430584293 tz="+0100" logid="0100040707" type="event" subtype="system" level="notice" vd="root" logdesc="CPU single core usage statistics" action="cpu-single-core-usage" core=8 cpu=99 msg="CPU core 8 usage reaches: 99"
To change the CPU use threshold, the following command can be used:
config system global
set cpu-use-threshold "Enter an integer value from <50> to <99>"
end
When enabled, CPU single-core usage will be polled every 3 seconds, and any single CPU core usage above the CPU usage threshold will report an event log. If a core is reported, that core will not be checked again for 30 seconds.
Note:
In the System events log, the core-index is different than that of the actual index in CPU data, so that it will show core-1 and core-2 here instea of CPU0, CPU1 in other FortiGate CLI commands. The event logs start counting at 1 while the CLI commands start counting at 0.
date=2025-03-14 time=14:01:16 eventtime=1741935675876953903 tz="+0700" logid="0100040707" type="event" subtype="system" level="notice" vd="root" logdesc="CPU single core usage statistics" action="cpu-single-core-usage" core=2 cpu=92 msg="CPU core 2 usage reaches: 92"
...
date=2025-03-14 time=08:30:49 eventtime=1741915849527939081 tz="+0700" logid="0100040707" type="event" subtype="system" level="notice" vd="root" logdesc="CPU single core usage statistics" action="cpu-single-core-usage" core=1 cpu=95 msg="CPU core 1 usage reaches: 95"
Even when the Firewall has CPU core IDs = 0 and 1, as shown in these outputs:
get system performance status
CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
diagnose sys mpstat 2
Gathering data, wait 2 sec, press any key to quit.
..0..1
TIME CPU %usr %nice %sys %iowait %irq %soft %steal %idle
12:50:45 PM all 0.25 0.00 0.25 0.00 0.00 1.50 0.00 98.00
0 0.50 0.00 0.50 0.00 0.00 2.50 0.00 96.50
1 0.00 0.00 0.00 0.00 0.00 0.50 0.00 99.50
TIME CPU %usr %nice %sys %iowait %irq %soft %steal %idle
12:50:47 PM all 0.75 0.00 0.00 0.00 0.25 1.25 0.00 97.75
0 1.00 0.00 0.00 0.00 0.50 2.00 0.00 96.50
1 0.50 0.00 0.00 0.00 0.00 0.50 0.00 99.00
|
