High CPU

[FGFan]

Hi all, My fortigate 110C usually has high CPU problem. I checked the enviroment (temperature, fan...) all is ok. On fortigate, I configured many policy route, I think it is reason for this problem. so how many policy route entry Fortigare recommend to device can run well? anyone can advise me ? Thanks all!

[FGFan]

:( Nobody can help?

[ede_pfau · Accepted solution]

There's not much information you offer. Firmware?

In the CLI, type 'diag deb enable', 'diag sys top' and hit the 'p' key. This will list the running processes, sorted by CPU usage. Hit 'Ctrl-C' to stop and post the output here.

I don't think PBR is the main cause. Routes are only looked up when a new session is started, PBR the same. The FG-100D can handle a lot of new sessions per second.

[Luiz_Alberto_Camilo]

What's high CPU for you ? 

Normally FortiOS would always keep CPU values low like, oscilating bellow 10%. If your fortigate oscillates more than this, you should probably check your firewall rules order. You may have one rule inspecting all traffic for nothing, maybe .. APPControl rules are the ones that consumes more CPU. 

Your device may have reached it's limit also. How's the output of the command "get system performance status" ? 

Several other factors can cause this behavior, but keep in mind that VPN, appcontrol, ips, DoS, would increase your CPU usage, especially if the rules are not optimal ordered. 

Check for interfaces errors like "get hardware nic internal1".

[przemo]

From 2 months I have a similar issue - very high CPU usage (99%..100%) continues for 10 minutes and sometimes for several hours. This issue affects the work of the whole company: - no access to the Internet, or very slow connection, - connection between local LAN very slow, - trouble connecting to the FGT admin panel or unable to connect. - etc

Since the beginning of the year I had to reboot UTM several times. Very often it happens that the restart does not help and the issue still persists (unable to connect to the GUI, no Internet).

I tried to find out the cause of the problem by typing: 'diag sys top', 'top diag sys-summary'. I saw that the IPS service causes high CPU load. I had had enabled this service only in 3 IPv4 policies with the default settings. I turned off the IPS service. It helped for a short time, now FGT still working at high CPU usage - this time the 'scanunitd' and other services causes increased load.

Now in FGT I have implemented about 30 uncomplicated policies, the average total session at 1,200 and about 30 workstations in LAN.

UTM: FGT60D, v 5.2.5.

Example of diag sys top :

Run Time: 10 days, 0 hours and 43 minutes 27U, 0N, 72S, 1I; 1839T, 1263F, 147KF        ipsengine 1286 R < 72.5 5.1      proxyworker 87 S 11.6 1.8        scanunitd 1930 S < 5.0 2.3           newcli 1937 R 2.8 0.7           httpsd 124 S 1.5 1.3          miglogd 58 S 1.5 1.1           fcnacd 74 S 0.9 0.7   merged_daemons 65 S 0.6 0.5        scanunitd 85 S < 0.3 2.2          src-vis 92 S 0.3 0.7 Run Time: 4 days, 17 hours and 43 minutes 71U, 0N, 28S, 1I; 1839T, 1198F, 199KF        scanunitd 892 R < 48.1 2.6        ipsengine 143 S < 22.0 5.4      proxyworker 114 R 11.4 2.3        scanunitd 894 S < 9.6 2.4           newcli 905 R 2.7 0.7          miglogd 88 R 1.8 1.1           httpsd 72 S 1.3 1.4        urlfilter 120 S 1.3 1.1           fcnacd 101 S 0.4 0.7        scanunitd 109 S < 0.0 2.3 Run Time: 4 days, 17 hours and 43 minutes 81U, 0N, 18S, 1I; 1839T, 1198F, 199KF        scanunitd 892 R < 63.6 2.6        ipsengine 143 S < 13.3 5.4        scanunitd 894 S < 9.9 2.4      proxyworker 114 S 7.1 2.3        urlfilter 120 S 2.3 1.1          miglogd 88 S 1.1 1.1           newcli 905 R 0.9 0.7           httpsd 73 S 0.5 1.4           httpsd 72 S 0.1 1.4          updated 116 S 0.1 0.6 Run Time: 4 days, 17 hours and 43 minutes 86U, 0N, 13S, 1I; 1839T, 1199F, 198KF        scanunitd 892 R < 72.6 2.6        ipsengine 143 R < 12.1 5.4        scanunitd 894 S < 7.0 2.4      proxyworker 114 S 4.5 2.3          miglogd 88 S 0.7 1.1           newcli 905 R 0.7 0.7           httpsd 73 S 0.5 1.4          updated 116 S 0.5 0.6         dnsproxy 129 S 0.3 0.6           httpsd 72 S 0.1 1.4

16U, 0N, 79S, 5I; 1839T, 1223F, 139KF          src-vis     1676      R      59.7     0.7           newcli     1735      R       5.8     0.8        ipsengine      143      S <     2.2     5.4          miglogd       88      S       2.2     1.1        scanunitd     1629      S <     0.0     2.3        scanunitd     1728      S <     0.0     2.2        scanunitd      109      S <     0.0     2.2          pyfcgid     1570      S       0.0     1.7        urlfilter      120      S       0.0     1.7          pyfcgid     1571      S       0.0     1.7

Exaample of diag sys top-summary

   CPU [||||||||||||||||||||||||||||||||||||||||] 100.0%    Mem [|||||||||||| ] 31.0% 577M/1839M    Processes: 20 (running=4 sleeping=85)       PID RSS ^CPU% MEM% FDS TIME+ NAME  * 64 103M 34.6 5.6 41 44:23.20 ipsmonitor [x3]    75 23M 25.4 1.3 11 00:05.68 httpclid [x3]    85 44M 5.4 2.4 31 08:39.35 scanunitd [x3]    83 32M 4.7 1.7 1199 03:37.94 proxyd [x3]    84 12M 2.9 0.7 34 02:02.46 imd    60 33M 2.7 1.8 25 03:38.86 httpsd [x5]    58 21M 2.4 1.1 25 18:57.62 miglogd    88 13M 2.3 0.7 30 01:37.27 iked    104 12M 0.4 0.7 29 01:56.00 dnsproxy    90 12M 0.3 0.7 13 00:17.47 snmpd    71 11M 0.1 0.6 15 01:06.22 forticldd    89 12M 0.1 0.7 14 05:18.70 updated    92 14M 0.1 0.8 16 13:45.61 src-vis    1903 32M 0.0 1.8 12 00:17.17 pyfcgid [x4]    38 23M 0.0 1.3 13 02:24.40 cmdbsvr    46 11M 0.0 0.6 87 00:08.41 zebos_launcher [x12]    57 10M 0.0 0.6 12 00:00.17 uploadd    59 9M 0.0 0.5 8 00:00.15 kmiglogd    62 10M 0.0 0.5 8 00:00.00 getty    65 9M 0.0 0.5 10 00:10.57 merged_daemons # diag sys top-summary    CPU [||||||||||||||||||||||||||||||||||||||||] 100.0%    Mem [|||||||||||| ] 32.0% 606M/1839M    Processes: 20 (running=1 sleeping=89)    PID RSS ^CPU% MEM% FDS TIME+ NAME  * 38 23M 0.0 1.3 13 02:57.24 cmdbsvr    46 11M 0.0 0.6 87 00:11.96 zebos_launcher [x12]    57 10M 0.0 0.6 12 00:00.30 uploadd    58 21M 0.0 1.2 25 27:20.30 miglogd    59 9M 0.0 0.5 8 00:00.22 kmiglogd    60 41M 0.0 2.3 27 06:33.20 httpsd [x6]    62 10M 0.0 0.5 8 00:00.00 getty    64 102M 0.0 5.6 41 43:29.73 ipsmonitor [x3]    65 9M 0.0 0.5 10 00:20.64 merged_daemons    66 10M 0.0 0.6 12 00:00.15 fnbamd    68 9M 0.0 0.5 12 00:00.28 fclicense    70 11M 0.0 0.6 21 01:00.59 forticron    71 11M 0.0 0.6 15 01:34.67 forticldd    72 12M 0.0 0.7 39 00:16.64 authd    74 14M 0.0 0.8 18 05:34.38 fcnacd    75 22M 0.0 1.2 11 00:01.81 httpclid [x3]    76 10M 0.0 0.5 10 00:00.14 sqldb    77 20M 0.0 1.1 29 00:01.52 sslvpnd    78 9M 0.0 0.5 11 00:00.24 info_sslvpnd    82 824K 0.0 0.0 4 00:00.20 smbcd      CPU [|||||||||||||||||||||||| ] 61.6%    Mem [|||||||||||| ] 32.0% 606M/1839M    Processes: 20 (running=1 sleeping=89)    PID RSS ^CPU% MEM% FDS TIME+ NAME  * 75 23M 35.8 1.3 11 00:02.44 httpclid [x3]    64 102M 22.1 5.6 41 43:30.10 ipsmonitor [x3]    83 39M 2.4 2.2 1538 45:26.39 proxyd [x3]    58 21M 0.6 1.2 25 27:20.31 miglogd    60 41M 0.6 2.3 27 06:33.21 httpsd [x6]    38 23M 0.0 1.3 13 02:57.24 cmdbsvr    46 11M 0.0 0.6 87 00:11.96 zebos_launcher [x12]    57 10M 0.0 0.6 12 00:00.30 uploadd    59 9M 0.0 0.5 8 00:00.22 kmiglogd    62 10M 0.0 0.5 8 00:00.00 getty    65 9M 0.0 0.5 10 00:20.64 merged_daemons    66 10M 0.0 0.6 12 00:00.15 fnbamd    68 9M 0.0 0.5 12 00:00.28 fclicense    70 11M 0.0 0.6 21 01:00.59 forticron    71 11M 0.0 0.6 15 01:34.67 forticldd    72 12M 0.0 0.7 39 00:16.64 authd    74 14M 0.0 0.8 18 05:34.38 fcnacd    76 10M 0.0 0.5 10 00:00.14 sqldb    77 20M 0.0 1.1 29 00:01.52 sslvpnd    78 9M 0.0 0.5 11 00:00.24 info_sslvpnd # diag sys top-summary   CPU [||||||||||||||||||||||||||||||||||||||||] 100.0%    Mem [|||||||||||||| ] 37.0% 689M/1839M    Processes: 20 (running=4 sleeping=88)    PID RSS ^CPU% MEM% FDS TIME+ NAME  * 3801 31M 0.0 1.7 12 00:12.40 pyfcgid [x4]    38 23M 0.0 1.3 13 04:47.97 cmdbsvr    46 11M 0.0 0.6 87 00:23.50 zebos_launcher [x12]    57 10M 0.0 0.6 12 00:00.47 uploadd    58 21M 0.0 1.2 25 49:21.13 miglogd    59 9M 0.0 0.5 8 00:00.35 kmiglogd    60 43M 0.0 2.4 27 07:32.88 httpsd [x6]    62 10M 0.0 0.5 8 00:00.00 getty    64 115M 0.0 6.3 41 43:42.93 ipsmonitor [x3]    65 9M 0.0 0.5 10 00:27.63 merged_daemons    66 10M 0.0 0.6 12 00:00.20 fnbamd    68 9M 0.0 0.5 12 00:00.41 fclicense    70 11M 0.0 0.6 21 01:24.67 forticron    71 11M 0.0 0.6 15 02:36.86 forticldd    72 12M 0.0 0.7 40 05:55.89 authd    74 14M 0.0 0.8 18 09:58.57 fcnacd    75 31M 0.0 1.7 12 00:06.62 httpclid [x5]    76 10M 0.0 0.5 10 00:00.18 sqldb    77 20M 0.0 1.1 29 00:01.80 sslvpnd    78 9M 0.0 0.5 11 00:00.33 info_sslvpnd    CPU [||||||||||||||||||||||||||||||||||||||||] 100.0%    Mem [||||||||||||| ] 34.0% 638M/1839M    Processes: 20 (running=2 sleeping=89)    PID RSS ^CPU% MEM% FDS TIME+ NAME  * 109 49M 45.0 2.7 33 04:43.86 scanunitd [x3]    91 109M 22.1 5.9 41 49:02.40 ipsmonitor [x3]    102 31M 19.5 1.7 12 00:05.40 httpclid [x5]    107 43M 10.1 2.4 1290 39:55.57 proxyd [x3]    120 20M 1.5 1.1 20 03:32.84 urlfilter    50 33M 0.8 1.8 22 01:00.53 httpsd [x5]    88 20M 0.8 1.1 25 06:49.72 miglogd    116 12M 0.5 0.7 13 02:17.96 updated    45 11M 0.3 0.6 87 00:06.22 zebos_launcher [x12]    101 14M 0.3 0.8 18 01:26.41 fcnacd    129 12M 0.2 0.7 29 01:17.54 dnsproxy    119 14M 0.2 0.8 16 01:34.79 src-vis    128 10M 0.0 0.6 10 00:00.30 alertmail    137 872K 0.0 0.0 13 00:00.17 usbmuxd    138 9M 0.0 0.5 11 00:00.11 fsd    139 13M 0.0 0.7 14 00:00.99 dsd    131 15M 0.0 0.9 18 00:09.88 fgfmd    132 14M 0.0 0.8 22 00:00.79 cw_acd    796 32M 0.0 1.8 12 00:05.25 pyfcgid [x4]    133 9M 0.0 0.5 11 00:00.50 wpad_ac

For a one maybe two days my UTM worked on OS v 5.4 and then was downgraded to 5.2.5

Can you help me?

[Luiz_Alberto_Camilo]

Set a widget on the main screen to observe the interface and internet link usage. 

Check if when your firewall spikes CPU, you're having a high load either on LAN or WAN interfaces. 

to me it looks like on that time, there's extra traffic passing through your firewall. Maybe an WAN attack (DoS sensor will help ) or maybe an heavy user or server doing an Cloud backup or stuff like this (Application control or IPS monitor). 

check your "diag debug crashlog read" also .. 

[FGFan]

Thank all for your help, My Fortigate is FG 110C, version 4.0.

Here are some info I got when did diag sys top command

Run Time:  1 days, 19 hours and 17 minutes 6U, 44S, 50I; 1008T, 732F, 93KF  initXXXXXXXXXXX        1      S       0.0     1.1          cmdbsvr       28      S       0.0     2.0   zebos_launcher       32      S       0.0     1.1          uploadd       33      S       0.0     1.1          miglogd       34      R       0.0     1.6              nsm       35      S       0.0     0.2             ripd       36      S       0.0     0.2           ripngd       37      S       0.0     0.2            ospfd       38      S       0.0     0.2          miglogd       39      S       0.0     1.1           httpsd       40      S       0.0     1.7           proxyd       41      S       0.0     1.1           ospf6d       42      S       0.0     0.2             bgpd       43      S       0.0     0.2            isisd       44      S       0.0     0.2        wad_diskd       45      S       0.0     1.1        scanunitd       46      S <     0.0     1.2    proxyacceptor       47      S       0.0     0.1      proxyworker       48      R       0.0     6.5            getty       49      S <     0.0     1.1