Technical Tip: High CPU issue due to log_se process

[pkumari]

Description	This article describes how to troubleshoot high CPU issues when log_se is high.
Scope	FortiGate, FortiVM.
Solution	There are scenarios in which if user is using disk for logging & if disk rollover is not happening as per the settings. Log_se process may go high as below. diag sys top 1 10 Run Time: 89 days, 15 hours and 11 minutes 0U, 46N, 5S, 4I, 44WA, 1HI, 0SI, 0ST; 24140T, 11438F log_se 11066 R N 38.0 0.1 8 log_se 11061 R N 37.5 0.1 10 log_se 10998 D N 37.5 0.1 3 log_se 11003 D N 37.5 0.1 11 log_se 11072 D N 37.5 0.1 14 log_se 11075 D N 37.5 0.1 3 log_se 11000 D N 37.0 0.1 1 log_se 11097 R N 37.0 0.1 15 log_se 11025 D N 36.5 0.1 2 log_se 11016 R N 36.0 0.1 4 Run the command 'get sys perf status', 'dia sys mpstat', and 'dia sys top 1 10' to see in which area the load is present consistently. The output will be something as below (depending on the number of cores the unit has): The first line 'CPU states:' shows the average load across all CPU cores.  get sys perf stat CPU states: 20% user 60% system 0% nice 20% idle 0% iowait 0% irq 0% softirq CPU0 states: 21% user 62% system 0% nice 17% idle 0% iowait 0% irq 0% softirq CPU1 states: 44% user 56% system 0% nice 0% idle 0% iowait 0% irq 0% softirq CPU2 states: 32% user 58% system 0% nice 10% idle 0% iowait 0% irq 0% softirq ... Process log_SE is used by the log disk. It is the process of handling the log search in the GUI. All 'log_se' processes are running with low priority (process state N) so it shouldn't affect other running processes. diag sys top 1 10 Run Time: 89 days, 15 hours and 11 minutes 0U, 46N, 5S, 4I, 44WA, 1HI, 0SI, 0ST; 24140T, 11438F log_se 11066 R N 38.0 0.1 8 log_se 11061 R N 37.5 0.1 10 Technical Tip: Using the 'diagnose sys top' CLI command There are chances that the firewall disk is not functioning correctly if log_se is high. Try disabling the log_se as a temporary workaround. It is possible to perform a failover and see if another firewall disk is healthy or to format the disk for the fix if suspect issue with the disk malfunctioning. Log to be collected: CLI Session (1) get sys perf status diag sys top 2 50 diag sys mpstat 2 For disk logging related debug: diagnose debug reset diagnose debug enable diagnose debug console timestamp enable diag debug app miglogd 0x1000 diag deb app httpsd -1 Monitor CPU utilization of 'log_se' processes from FortiGate CLI Session (1) and then stop debugging once the processes are gone and the CPU is back in a normal state. Stop the debug: diagnose debug disable diagnose debug console timestamp disable diag debug app miglogd 0 diag debug app httpsd 0 diagnose debug reset Collect the below logs to verify the disk log storage. fnsysctl ls -l /var/log/log/root diagnose hardware deviceinfo disk fnsysctl df fnsysctl ls /var/log -l