scanunitd taking 99% cpu

[billp]

Scanunitd is spiking my CPU to 95% fairly consistently. I believe this is the AV scanning component. Could anyone confirm? I' ve been tinkering with the URL and Content filtering recently, so this is also probably straining things a bit more, but it' s clearly Scanunitd at the top of the DIAG SYS TOP command. I am using firmware 4.1.1 and looking at using the option to exempt certain traffic with specific MIME headers. Hoping that might ease things. Any advice re: making AV scanning more efficient would be appreciated. Thanks. Bill

[FortiRack_Eric]

Hi Bill, with diag deb appl scanunit 4 (enter 0 for all options) you can see the load on the scanunit deamon. In normal cases you won' t see such high cpu on a FG. I' m wondering if you have misconfigured something and/or using a box too small for the traffic. Which model are you using? and what' s the typical bandwidth? Regards, Eric

[billp]

I think I had something misconfigured. I have a pair of 111c' s in HA A-A mode. Typical bandwidth is 7-15mbps. Total sessions is about 2k-3k or less. I used the MIME filtering capabilities of 4.x to exclude Audio, Image, and Video files from AV scanning. I also cut down on the full IPS scans I was doing. That seems to have helped, but I' ll keep an eye on with the diag debug app scanunit. Thanks.

[billp]

OK. Dumb questions here trying to understand this command. When using the diag debug appl command, do I need to also enter diag debug enable in order to see output? And then use disable to turn off the output? Or do I use diag debug reset I' ve tried searching the Fortinet sites for definitive docs on this command, could not locate anything. Would also like to understand the various debug levels. I' ve seen " -1" used in some Fortinet docs. Just trying to understand the general concept here so I can use diag debug xxx commands properly. Just seems like a lot of it is not documented (or at least well documented). Thanks for any tips. Bill

[abelio]

When using the diag debug appl command, do I need to also enter diag debug enable in order to see output?

yes

And then use disable to turn off the output? Or do I use diag debug reset

yes; if you have not changed defaults, in your case is the same thing

[ramunas]

Hello,

I have the same situation. Scanuitd takes 100% CPU. We use:

Model name: FortiGate-40C, Number of CPUs: 1

 # diag sys top 1 Run Time: 81 days, 5 hours and 18 minutes 52U, 0N, 47S, 1I; 440T, 53F, 159KF scanunitd 4550 R < 46.6 3.7 scanunitd 4631 R < 46.6 3.6 newcli 6817 R 2.4 3.2 ipsengine 5750 S < 1.4 13.1 httpsd 109 S 0.4 5.5 sslvpnd 68 S 0.4 4.5 proxyworker 77 S 0.4 3.1 scanunitd 6755 S < 0.4 2.9 (I cut part)

There are 3 scanunitd prcesses (why 3?? We have 1 CPU and use only the proxy mode for AV. Can anybody check on FG-40C how many scanunitd are there? (Thanks))

Next - there is ipsengine, which is sleeping (strange, very strange.. we use ips in all main policies.. but we are talking about scanunitd now)

Next - worse

# diag test app scanunit 3 scanunit parent pid = 6753 scanunit child pid = 6755 scanunit child pid = 6756

Only one id matches (!) and this one is id of sleeping process...

Next - worse

# diag debug application scanunit 3

FGT40C3912039776 # sp read ack 10490 from child 6755 6755 _scanunitNiceScan scan 119 bytes => START 6755 _scanunitNiceScan scan 119 bytes => FINISH rc=0 sp read ack 10491 from child 6755 6755 _scanunitNiceScan scan 188 bytes => START 6755 _scanunitNiceScan scan 188 bytes => FINISH rc=0

repeating...

Only process 6755 (which is sleeping) doing something. (While other 2 take 100% CPU)...

I don't understand anything.

Any help would be appreciated.

Best regards, Ramunas

[Robin_Svanberg]

ramunas wrote:

Hello,

I have the same situation. Scanuitd takes 100% CPU. We use:

Model name: FortiGate-40C, Number of CPUs: 1

 # diag sys top 1 Run Time: 81 days, 5 hours and 18 minutes 52U, 0N, 47S, 1I; 440T, 53F, 159KF scanunitd 4550 R < 46.6 3.7 scanunitd 4631 R < 46.6 3.6 newcli 6817 R 2.4 3.2 ipsengine 5750 S < 1.4 13.1 httpsd 109 S 0.4 5.5 sslvpnd 68 S 0.4 4.5 proxyworker 77 S 0.4 3.1 scanunitd 6755 S < 0.4 2.9 (I cut part)

There are 3 scanunitd prcesses (why 3?? We have 1 CPU and use only the proxy mode for AV. Can anybody check on FG-40C how many scanunitd are there? (Thanks))

Next - there is ipsengine, which is sleeping (strange, very strange.. we use ips in all main policies.. but we are talking about scanunitd now)

Next - worse

# diag test app scanunit 3 scanunit parent pid = 6753 scanunit child pid = 6755 scanunit child pid = 6756

Only one id matches (!) and this one is id of sleeping process...

Next - worse

# diag debug application scanunit 3

FGT40C3912039776 # sp read ack 10490 from child 6755 6755 _scanunitNiceScan scan 119 bytes => START 6755 _scanunitNiceScan scan 119 bytes => FINISH rc=0 sp read ack 10491 from child 6755 6755 _scanunitNiceScan scan 188 bytes => START 6755 _scanunitNiceScan scan 188 bytes => FINISH rc=0

repeating...

Only process 6755 (which is sleeping) doing something. (While other 2 take 100% CPU)...

I don't understand anything.

Any help would be appreciated.

Best regards, Ramunas

Hi Ramunas,

this is an old thread, from 2009, so you might experience other issues.

What firmware are you running?

We have a few customers running 30D, 40C and 60D that experiences high CPU usage by scanutid.

Fortinet Support informed us that our issues probably was cauased by a bug in the AV engine. (Atleast in 5.2.X).

A bugfix will be released in 5.2.3 which should be available any day now..

Take a crashlog (diag debug crashlog read) and and send it to Fortinet Support, or upload it to this thread.

Our crashlog had entries like these.

16266: 2015-01-16 09:55:04 <29784> *** signal 14 (Alarm clock) received ***

16267: 2015-01-16 09:55:05 <29784> AVDB 05002000AVDB00201-00023.00626-1501151341 16268: 2015-01-16 09:55:05 <29784> ETDB 05002000AVDB00701-00023.00626-1501151340 16269: 2015-01-16 09:55:05 <29784> AVSO 04000000AVEN00601051591410221129 16270: 2015-01-16 09:55:05 <29784> Register dump: 16271: 2015-01-16 09:55:05 <29784> R0: ffffffff R1: 0000001c R2: 000000ff R3: 00000003 16272: 2015-01-16 09:55:05 <29784> R4: 01f150bf R5: 01f15df0 R6: 01f150bf R7: ffffffff 16273: 2015-01-16 09:55:05 <29784> R8: 00000000 R9: 00000000 R10: 01f15df0 FP: 4220b5a9 16274: 2015-01-16 09:55:05 <29784> IP: 4220b6a2 SP: beff8240 LR: 40d0c13c PC: 40d0c044 16275: 2015-01-16 09:55:05 <29784> CPSR: 20000010 Addr: 00000000 16276: 2015-01-16 09:55:05 <29784> Trap: 00000006 Error: 00000000 OldMask: 00000000 16277: 2015-01-16 09:55:05 <29784> Backtrace: 16278: 2015-01-16 09:55:05 <29784> [0x011122f4] => /bin/scanunitd  16279: 2015-01-16 09:55:05 <29784> [0x40cf9598] => /data/lib/libav.so liboffset 00175598 16280: 2015-01-16 09:55:05 <29784> [0x40b9a2f4] => /data/lib/libav.so (scanvirFile+0x00000600) liboffset

BR Robin

[ramunas]

Hello,

can anybody check how many FG-40C has scanunitd processes?

Thank You in advance,

BR Ramunas

[ramunas]

Thank you a lot for your answer!

In our case it is 5.2.1 firmware, so it could be an issue...

The crashlog says many times "the killed daemon is /bin/scanunitd: status=0xf00 or 0x0, then:

302: 2014-11-07 13:30:09 <00278> firmware FortiGate-40C v5.2.1,build0618b618,140915 (GA) (Release) 303: 2014-11-07 13:30:09 <00278> application scanunit 304: 2014-11-07 13:30:09 <00278> *** signal 14 (Alarm clock) received *** 305: 2014-11-07 13:30:09 <00278> AVDB 05002000AVDB00201-00023.00134-1411070018 306: 2014-11-07 13:30:09 <00278> ETDB 05002000AVDB00701-00001.00000-1210171546 307: 2014-11-07 13:30:09 <00278> AVSO 04000000AVEN00601051561408181504 308: 2014-11-07 13:30:09 <00278> Register dump: 309: 2014-11-07 13:30:09 <00278> R0: 40023afc R1: 40d9af88 R2: 0000005b R3: 00000003 310: 2014-11-07 13:30:09 <00278> R4: 431885a4 R5: 00000034 R6: 431897cb R7: 000000c4 311: 2014-11-07 13:30:09 <00278> R8: beff94f4 R9: 02050740 R10: 40dd510c FP: 00000000 312: 2014-11-07 13:30:09 <00278> IP: 0000d808 SP: beff93c8 LR: 400de254 PC: 40c04054 313: 2014-11-07 13:30:09 <00278> CPSR: 00000010 Addr: 00000000 314: 2014-11-07 13:30:09 <00278> Trap: 00000006 Error: 00000000 OldMask: 00000000 315: 2014-11-07 13:30:09 <00278> Backtrace: 316: 2014-11-07 13:30:09 <00278> [0x0110dce8] => /bin/scanunitd 317: 2014-11-07 13:30:09 <00278> [0x40ce7430] => /data/lib/libav.so liboffset 00175430 318: 2014-11-07 13:30:09 <00278> [0x40b882fc] => /data/lib/libav.so (scanvirFile+0x00000600) liboffset

...

BR Ramunas

[Adrian_Buckley_FTNT]

Scanunit signal 14 means the FortiGate ran into a file that it could not finish the file scan for.

The FortiGate is a firewall, packets need to get places quickly.  The unit (FGT) will only allow 30 seconds to finish an AV scan before the Kernel (Watchdog) will interrupt the scan process and stop it.

A PC doesn't have this limitation, it can take however long it needs to finish a virus scan (users can delay it if they need/want too, or stop it and run it later).

IF there is a file transfer session open at the time, the file information will be included with the crash,  30 seconds is a very long time in terms of packets though.  So tracking down the file that is causing this may be difficult.

Most of the time when you run into a Scanunit Signal 14 the associated HTTP/SMTP/(whatever protocol) session will have closed down by the time 30 seconds happens. So in that case all you get is the signal 14 which only tells you that a file took too long to AV scan.  What you're looking for is a method to reproduce the behavior or a scanunit signal 14 crash combined with a protocol crash, in order to get the location details of the file that's causing it.

If the session is open at the same time that scanunit is closed then you will get 2 back to back crashlog entries.  One for Scanunit and one for the protocol the file was being transferred on (HTTP, SMTP, etc).  Also i'm fairly sure both crashes will have the same associated number in the <> brackets.