High CPU and Memory Usage

[Miata]

Hi guys

So my FG-60D running 5.2.3 has been at 100% CPU and about 90% memory recently so I thought I would run the diag sys top command as shown below. 

From this command I can see that the scanunitd and IPS engine it taking most of my CPU usage. I don't have vulnerability scanner but I have AV enabled on 17 different policies. I think the box is being overworked, but can I restart any processes or do you guys have any other advice?

Run Time: 42 days, 19 hours and 54 minutes 62U, 0N, 37S, 1I; 439T, 40F, 189KF scanunitd 7079 R < 68.4 3.7 ipsengine 602 S < 19.2 13.1 httpsd 7717 S 2.3 4.2 httpsd 7718 S 1.9 4.2 httpsd 7737 S 1.7 4.2

I also ran get sys performance - Output below

CPU states: 75% user 25% system 0% nice 0% idle CPU0 states: 75% user 25% system 0% nice 0% idle Memory states: 93% used Average network usage: 6282 kbps in 1 minute, 2754 kbps in 10 minutes, 2200 kbps in 30 minutes Average sessions: 1995 sessions in 1 minute, 2178 sessions in 10 minutes, 1824 sessions in 30 minutes

If you have any form of advice in terms of how to manage this more successfully or anything to restart/kill then please let me know, would be greatly appreciated.

Kind regards

Miata

[BarryM]

Update:

The latest firmware, 5.2.10,  fixed the CPU utilization problem. My 300C is running optimally now, as it was before some of the last firmware updates. It was bugs in the firmware that were the cause of the problems. My unit is nowhere near maxed out in features and settings. The CPU barely spikes now.

[MikePruett]

Probably had memory leaks and misc other things causing that. Good that Fortinet fixed it for you!

[jpastor]

Newest versions need more CPU and more memory. D family souldn´t be affected in performance by an upgrade to 5.4 versions. Open a ticket if it´s your case.

[virtualj]

Hello,

I'm also interested in this function to test the traffic between two fortigates. I see two interesting thigs:

the command "diag traffictest run" is like a script. It launchs two times the iperf, as you can see with this:

FW1_xxx_xx # diagnose traffictest show server-intf:    wan1 client-intf:    wan2 port:   9999 proto:  TCP

FW1_xxx_xx # diagnose traffictest run (running in another CLI)

FW1_xxx_xx # fnsysctl ps 8931      0       0       R       /bin/iperf -s -B xx.xx.xx.xx -m root -p 9999 8932      0       0       R       /bin/iperf -c xx.xx.xx.xx -B yy.yy.yy.yy -m root -p 9999

You can see also that there is a listening connection on port 9999:

FW1_xxx_xx # diagnose sys tcpsock | grep 9999 xx.xx.xx.xx:9999->0.0.0.0:0->state=listen err=0 sockflag=0x1 rma=0 wma=0 fma=0 tma=0 xx.xx.xx.xx:9999->yy.yy.yy.yy:5241->state=estabilshed err=0 sockflag=0x1 rma=0 wma=0 fma=0 tma=0 xx.xx.xx.xx:9999->yy.yy.yy.yy:19686->state=estabilshed err=0 sockflag=0x1 rma=0 wma=0 fma=774144 tma=0 yy.yy.yy.yy:19686->xx.xx.xx.xx:9999->state=estabilshed err=0 sockflag=0x1 rma=0 wma=133120 fma=309248 tma=0 yy.yy.yy.yy:5241->xx.xx.xx.xx:9999->state=estabilshed err=0 sockflag=0x1 rma=0 wma=0 fma=0 tma=0

I tried to do the same command on another fortigate with -c xx.xx.xx.xx so it should connect to the other fortigate iperf. Diag sniffer confirm that traffic arrives to the server fortigate, but the syns are dropped. Debug flow shows "iprope_check_failed", like when you are trying to manage the firewall, but don't have trusted hosts or management service enabled.

I tried also a firewall local-in policy to accept anything with no result.

Some other ideas? :)

[Paul_S]

virtualj wrote:

Hello,

I'm also interested in this function to test the traffic between two fortigates. I see two interesting thigs:

the command "diag traffictest run" is like a script. It launchs two times the iperf, as you can see with this:

FW1_xxx_xx # diagnose traffictest show server-intf:    wan1 client-intf:    wan2 port:   9999 proto:  TCP

FW1_xxx_xx # diagnose traffictest run (running in another CLI)

FW1_xxx_xx # fnsysctl ps 8931      0       0       R       /bin/iperf -s -B xx.xx.xx.xx -m root -p 9999 8932      0       0       R       /bin/iperf -c xx.xx.xx.xx -B yy.yy.yy.yy -m root -p 9999

You can see also that there is a listening connection on port 9999:

FW1_xxx_xx # diagnose sys tcpsock | grep 9999 xx.xx.xx.xx:9999->0.0.0.0:0->state=listen err=0 sockflag=0x1 rma=0 wma=0 fma=0 tma=0 xx.xx.xx.xx:9999->yy.yy.yy.yy:5241->state=estabilshed err=0 sockflag=0x1 rma=0 wma=0 fma=0 tma=0 xx.xx.xx.xx:9999->yy.yy.yy.yy:19686->state=estabilshed err=0 sockflag=0x1 rma=0 wma=0 fma=774144 tma=0 yy.yy.yy.yy:19686->xx.xx.xx.xx:9999->state=estabilshed err=0 sockflag=0x1 rma=0 wma=133120 fma=309248 tma=0 yy.yy.yy.yy:5241->xx.xx.xx.xx:9999->state=estabilshed err=0 sockflag=0x1 rma=0 wma=0 fma=0 tma=0

I tried to do the same command on another fortigate with -c xx.xx.xx.xx so it should connect to the other fortigate iperf. Diag sniffer confirm that traffic arrives to the server fortigate, but the syns are dropped. Debug flow shows "iprope_check_failed", like when you are trying to manage the firewall, but don't have trusted hosts or management service enabled.

I tried also a firewall local-in policy to accept anything with no result.

Some other ideas? :)

according to the forum thread here: https://forum.fortinet.com/tm.aspx?m=146386

you cannot run traffictest in server mode on a fortigate. connect to a computer and try the -r flag if you want to have the traffic flow to the fortigate instead of from it.

[virtualj]

Sorry Paul, I want to answer to that thread!

[Qhamani_Zendile]

Good Day All

Client of mine experienced a spike of 99% in CPU Usage on a Fortigate 200E Model. The spike was due to High CPU Usage on the ipsengine process. I restarted the process via CLI and it seemed to resolve the issue. What could possibly be causing the spike on the ipsengine process and how can be prevented from happening again?

[boneyard]

next time just start a new question, lots of answer about specific models or versions.

if the IPS process is high on CPU then probably it is not handling certain traffic well. it might be something very specific that doesn't happen often or just this once. if it happens again raise a ticket with Fortinet support and provide the information they ask for, version, some debugs, ... they might be able to trace where it went wrong.

[Qhamani_Zendile]

Thank you Boneyard

Noted. Much appreciated for the feedback.

[SCSIraidGURU]

I have worked with Cisco, Fortinet, Checkpoint and other firewalls over 35 years.  17 policies on a 60D?   Seems like overkill.   Can you consolidate the policies and processes down?     I would not try that on my 800C in my data center.   I limited policies to default that covers 85% of users.  A custom policy for 10% of the users.  A third policy for 5% of users.   With most firewalls.  You want to create policies to cover most of your users.    Can you give more details on why you have so many policies on a 60D?