- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
High CPU and Memory Usage
Hi guys
So my FG-60D running 5.2.3 has been at 100% CPU and about 90% memory recently so I thought I would run the diag sys top command as shown below.
From this command I can see that the scanunitd and IPS engine it taking most of my CPU usage. I don't have vulnerability scanner but I have AV enabled on 17 different policies. I think the box is being overworked, but can I restart any processes or do you guys have any other advice?
Run Time: 42 days, 19 hours and 54 minutes 62U, 0N, 37S, 1I; 439T, 40F, 189KF scanunitd 7079 R < 68.4 3.7 ipsengine 602 S < 19.2 13.1 httpsd 7717 S 2.3 4.2 httpsd 7718 S 1.9 4.2 httpsd 7737 S 1.7 4.2
I also ran get sys performance - Output below
CPU states: 75% user 25% system 0% nice 0% idle CPU0 states: 75% user 25% system 0% nice 0% idle Memory states: 93% used Average network usage: 6282 kbps in 1 minute, 2754 kbps in 10 minutes, 2200 kbps in 30 minutes Average sessions: 1995 sessions in 1 minute, 2178 sessions in 10 minutes, 1824 sessions in 30 minutes
If you have any form of advice in terms of how to manage this more successfully or anything to restart/kill then please let me know, would be greatly appreciated.
Kind regards
Miata
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Finally, we realized that some interfaces of Fortigate unit that were configured as trunk interfaces (multiple vlans), were receiving more traffic than they have to (have to receive only 1 vlan traffic, and was receiving 10 vlan traffic), so interface got oversubscribed and CPU of Fortigate raised almos al 100%. Allowing only the 1 vlan on the switch, solved the issue.
Check for overloaded / oversubscribed interfaces traffic.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Miata wrote:
scanunitd 7079 R < 68.4 3.7
diag sys kill 11 7079
It shouldn't get that high
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi
Thanks for the comment.
My mistake, this is just an example of the diag sys top command, there are many others that show it fluctuates between the given value and 90%+.
Miata
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I've noticed the same issue on 60D, 90D and others since upgrading from 5.0.10 to 5.2.3
CPU spikes from IPSengine primarily and scanunitd put average cpu about double what it was before upgrading.
I can kill/restart ipsengine but problem comes back. I disabled SSL cert inspection in case that was doing it but no go. Still occurs even on boxes that dont use ANY IPS policies (although app control is enabled on surfing).
Bug in 5.2.3??
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Well I think that generally I'm over working the box, as it is only a 60D. These boxes can't really take a lot of tasks, especially if one of them is to scan every bit of traffic that comes in and out of the box, as well as a bunch of other tasks which I couldn't mention within the size of this text box!
I noticed the vulnerability task was enabled, and so assumed this would be scanning loads of traffic both incoming and outgoing, so I set this only to late at night to run security checks etc. I also killed/restarted the IPS engine which has also helped bring down the processing usage.
Miata
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Yes, you can see high CPU/Memory if you have many task beyond the device capability, the box will exhaust.
However, best thing to do is to optimize the settings.
Like, reducing the session-ttl ( which is 3600 seconds may not be needed in most of the networks) and when can have increased session-ttl for specific protocols and ports if needed.
Also, tweaking the below values (these are not default, they are recommended values):
config system global set tcp-halfclose-timer 30 set tcp-halfopen-timer 30 set tcp-timewait-timer 0 set udp-idle-timer 60 end
Above techniques will help to optimize the performance of a device.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Just to add, Even for the IPS profiles, instead of using the default sensor list, fine tune it by having specific signatures like, with Server based / OS based and so on.
Also, instead of killing a process, I would recommend restarting the application as shown below:
# diagnose test application ipsmonitor IPS Engine Test Usage: 97: Start all IPS engines 98: Stop all IPS engines 99: Restart all IPS engines and monitor
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We see the same symptons on our FGT60D since we upgraded from 5.0.9 to 5.2.4 now CPU spike at 100% and of course response times are very very slow.
There was no change in the amount of sessions nor of the traffic which is going through the FGT60D and with 5.0.9 there was no problem cpu was idled most of the time. So 5.2.4 is doing something different causes high cpu usage.
Is it a bug? Or did someone find out what causes the the high cpu usage for the ipsengine/monitor since the upgrade to 5.2.4
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
having same problem with 5.2.3 scanunitd is 100% and it won't kill the process when i try diag sys kill 11
Anybody know what to do if diag sys kill 11 doesn't work?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm running into the same thing too. I have a 90D, and after upgrading from 5.0.13 to 5.2.7 the CPU is running much higher, and about once a day (usually after hours luckily) the box goes into conserve mode during an IPS scan. I even disabled IPS in the "Features" tab in the GUI. Tried kill 11 also which didn't help. Has anyone figured this out?
BTW - just to throw it out there, really not happy that my WAN load share/backup config got blown out thanks to the "New and improved" Virtual WAN IP. Now I have to come in over the weekend and tear down every single policy in the box just to rebuild them after I configure the new failover.