Technical Tip: High CPU issue due to log_se process

There are scenarios in which the user is using a disk for logging, and if disk rollover is not happening as per the settings.

Log_se process may go high as below.

diagnose sys top 1 10
Run Time: 89 days, 15 hours and 11 minutes
0U, 46N, 5S, 4I, 44WA, 1HI, 0SI, 0ST; 24140T, 11438F
log_se 11066 R N 38.0 0.1 8
log_se 11061 R N 37.5 0.1 10
log_se 10998 D N 37.5 0.1 3
log_se 11003 D N 37.5 0.1 11
log_se 11072 D N 37.5 0.1 14
log_se 11075 D N 37.5 0.1 3
log_se 11000 D N 37.0 0.1 1
log_se 11097 R N 37.0 0.1 15
log_se 11025 D N 36.5 0.1 2
log_se 11016 R N 36.0 0.1 4

Run the command 'get sys perf status', 'dia sys mpstat', and 'dia sys top 1 10' to see in which area the load is present consistently. The output will be something like below (depending on the number of cores the unit has):

The first line, 'CPU states:' shows the average load across all CPU cores. 

get sys perf stat
CPU states: 20% user 60% system 0% nice 20% idle 0% iowait 0% irq 0% softirq
CPU0 states: 21% user 62% system 0% nice 17% idle 0% iowait 0% irq 0% softirq
CPU1 states: 44% user 56% system 0% nice 0% idle 0% iowait 0% irq 0% softirq
CPU2 states: 32% user 58% system 0% nice 10% idle 0% iowait 0% irq 0% softirq
...

In some cases ,the CPU goes high do to iowait and can relate to the log_se issue based on the below logs:

CPU states: 0% user 0% system 7% nice 2% idle 91% iowait 0% irq 0% softirq
CPU0 states: 0% user 0% system 5% nice 3% idle 91% iowait 1% irq 0% softirq
CPU1 states: 0% user 0% system 8% nice 1% idle 91% iowait 0% irq 0% softirq
CPU2 states: 0% user 0% system 8% nice 2% idle 89% iowait 1% irq 0% softirq

• Process log_SE is used by the log disk. It is the process of handling the log search in the GUI.
• All 'log_se' processes are running with low priority (process state N) so it shouldn't affect other running processes.

diagnose sys top 1 10
Run Time: 89 days, 15 hours and 11 minutes
0U, 46N, 5S, 4I, 44WA, 1HI, 0SI, 0ST; 24140T, 11438F
log_se 11066 R N 38.0 0.1 8
log_se 11061 R N 37.5 0.1 10

Related article:
Technical Tip: Using the 'diagnose sys top' CLI command

There is a chance that the firewall disk is not functioning correctly if log_se is high. Try disabling the log_se as a temporary workaround.

config log disk setting
    set status disable
end

If below is the case, then try a workaround by reducing the log-age:

config log disk setting
    set status enable
    set maximum-log-age 3650
end

Then reduce the log age to 7.

config log disk setting
    set status enable
    set maximum-log-age 7
end

 execute formatlogdisk

It is possible to perform a failover and see if another firewall disk is healthy, or to format the disk for the fix if suspected issue with the disk malfunctioning.

Log to be collected:

CLI Session (1)

get sys perf status

diag sys top 2 50

diag sys mpstat 2

For disk logging-related debug:

diagnose debug reset
diagnose debug enable
diagnose debug console timestamp enable
diag debug app miglogd 0x1000
diag deb app httpsd -1

Monitor CPU utilization of 'log_se' processes from FortiGate CLI Session (1) and then stop debugging once the processes are gone and the CPU is back in a normal state.

Stop the debug:

diagnose debug disable
diagnose debug console timestamp disable
diagnose debug app miglogd 0
diagnose debug app httpsd 0
diagnose debug reset

Collect the logs below to verify the disk log storage.

fnsysctl ls -l /var/log/log/root
diagnose hardware deviceinfo disk
fnsysctl df
fnsysctl ls /var/log -l