Bad engine update???

[seadave]

At 4:23 PST today we started seeing 403 errors when trying to visit sites.  Only way to allow access is simple unfiltered NAT rule.  Searching form shows this happened in the past with a bad AV engine update.  I notice that support.fortinet.com is down with a 500 error so perhaps they self-inflicted the same.  TIME FOR AN ARCHITECTURE MODIFICATION!

Anyone else seeing this?

I'm seeing these in my debugs:

16330: 2017-09-28 17:03:53 <01449> firmware FortiGate-500D v5.4.5,build1138b1138,170531 (GA) (Release) 16331: 2017-09-28 17:03:53 <01449> application scanunit 16332: 2017-09-28 17:03:53 <01449> *** signal 11 (Segmentation fault) received *** 16333: 2017-09-28 17:03:53 <01449> AVDB 05004000AVDB00201-00052.00000-1709281424 16334: 2017-09-28 17:03:53 <01449> ETDB 05004000AVDB00701-00052.00000-1709281423 16335: 2017-09-28 17:03:53 <01449> EXDB 05004000AVDB00401-00001.00000-1210171547 16336: 2017-09-28 17:03:53 <01449> AVSO 04000000AVEN00701052471705041426 16337: 2017-09-28 17:03:53 <01449> Register dump: 16338: 2017-09-28 17:03:53 <01449> RAX: 0000000000000000 RBX: 00007fff067f82a0 16339: 2017-09-28 17:03:53 <01449> RCX: 00000000000000f8 RDX: 000000001258db56 16340: 2017-09-28 17:03:53 <01449> R8: 00000000000000ff R9: 0000000000000000 16341: 2017-09-28 17:03:53 <01449> R10: 0000000000000002 R11: 00007fb5625a0df0 16342: 2017-09-28 17:03:53 <01449> R12: 0000000000000046 R13: 00000000ffffffff 16343: 2017-09-28 17:03:53 <01449> R14: 00007fff067f82a0 R15: 00007fff067f81f0 16344: 2017-09-28 17:03:53 <01449> RSI: 0000000000000000 RDI: 0000000000000002 16345: 2017-09-28 17:03:53 <01449> RBP: 00000000ffffffff RSP: 00007fff067f80e0 16346: 2017-09-28 17:03:53 <01449> RIP: 00007fb5654af827 EFLAGS: 0000000000010212 16347: 2017-09-28 17:03:53 <01449> CS: 0033 FS: 0000 GS: 0000 16348: 2017-09-28 17:03:53 <01449> Trap: 000000000000000e Error: 0000000000000004 16349: 2017-09-28 17:03:53 <01449> OldMask: 0000000000000000 16350: 2017-09-28 17:03:53 <01449> CR2: 0000000000000014 16351: 2017-09-28 17:03:53 <01449> Backtrace: 16352: 2017-09-28 17:03:53 <01449> [0x7fb5654af827] => /data/lib/libav.so 16353: 2017-09-28 17:03:53 <01449> [0x7fb5654b7f45] => /data/lib/libav.so 16354: 2017-09-28 17:03:53 <01449> [0x7fb5654b86f3] => /data/lib/libav.so 16355: 2017-09-28 17:03:53 <01449> [0x7fb5654ab912] => /data/lib/libav.so 16356: 2017-09-28 17:03:53 <01449> [0x7fb5654b44c3] => /data/lib/libav.so 16357: 2017-09-28 17:03:53 <01449> [0x7fb5654baa39] => /data/lib/libav.so 16358: 2017-09-28 17:03:53 <01449> [0x7fb5654da895] => /data/lib/libav.so 16359: 2017-09-28 17:03:53 <01449> [0x7fb5654d87e7] => /data/lib/libav.so 16360: 2017-09-28 17:03:53 <01449> [0x7fb565494ad7] => /data/lib/libav.so (scanvirFile+0x00000187) 16361: 2017-09-28 17:03:53 <01449> [0x01a07ddf] => /bin/scanunitd 16362: 2017-09-28 17:03:53 <01449> [0x01a455ec] => /bin/scanunitd 16363: 2017-09-28 17:03:53 <01449> [0x01a466db] => /bin/scanunitd 16364: 2017-09-28 17:03:53 <01449> [0x010e54f0] => /bin/scanunitd 16365: 2017-09-28 17:03:53 <01449> [0x010e6599] => /bin/scanunitd 16366: 2017-09-28 17:03:53 <01449> [0x019b1c7c] => /bin/scanunitd 16367: 2017-09-28 17:03:53 <01449> [0x010e734d] => /bin/scanunitd 16368: 2017-09-28 17:03:53 <01449> [0x010e0616] => /bin/scanunitd 16369: 2017-09-28 17:03:53 <01449> [0x010e3fde] => /bin/scanunitd 16370: 2017-09-28 17:03:53 <01449> [0x00427c10] => /bin/scanunitd 16371: 2017-09-28 17:03:53 <01449> [0x0042e5c7] => /bin/scanunitd 16372: 2017-09-28 17:03:53 <01449> [0x0042bcf1] => /bin/scanunitd 16373: 2017-09-28 17:03:53 <01449> [0x0042d881] => /bin/scanunitd 16374: 2017-09-28 17:03:53 <01449> [0x0042deff] => /bin/scanunitd 16375: 2017-09-28 17:03:53 <01449> [0x7fb5690e4475] => /fortidev4-x86_64/lib/libc.so.6 16376: 2017-09-28 17:03:53 (__libc_start_main+0x000000f5) liboffset 00021475 16377: 2017-09-28 17:03:53 <01449> [0x00425065] => /bin/scanunitd 16378: 2017-09-28 17:03:53 [AV Engine <1449>] Last file info: 16379: 2017-09-28 17:03:53 [AV Engine <1449>] filename: bag, filesize: 7151 16380: 2017-09-28 17:03:53 [AV Engine <1449>] Native script imagebase: 0x12546000 16381: 2017-09-28 17:03:53 [AV Engine <1449>] cprl sigid: 489591, bintype: 00000400 16382: 2017-09-28 17:03:53 scanunit=worker pid=1449 exittype=signal code=11 total=7996 free=5679 16383: 2017-09-28 17:03:53 scanunit crash: signal=11, src-ip=172.21.11.126, dst-ip=104.80.89.9, 16384: 2017-09-28 17:03:53 request-uri=http://init-p01st.push.apple.com/bag

diag autoupdate ver output:

AV Engine --------- Version: 5.00247 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu May 4 14:26:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

Virus Definitions --------- Version: 52.00001 Contract Expiry Date: Mon Jul 16 2018 Last Updated using push update on Thu Sep 28 17:00:11 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: Updates Installed

Extended set --------- Version: 52.00000 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu Sep 28 14:23:00 2017 Last Update Attempt: n/a Result: Updates Installed

Extreme set --------- Version: 1.00000 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Wed Oct 17 15:47:00 2012 Last Update Attempt: n/a Result: Updates Installed

Mobile Malware Definitions --------- Version: 52.00000 Contract Expiry Date: Sat Jun 2 2018 Last Updated using push update on Thu Sep 28 17:00:11 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: Updates Installed

Attack Definitions --------- Version: 6.00741 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Tue Dec 1 02:30:00 2015 Last Update Attempt: n/a Result: Updates Installed

Attack Extended Definitions --------- Version: 12.00234 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu Sep 28 01:27:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

IPS Malicious URL Database --------- Version: 1.00775 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu Sep 28 07:29:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

Flow-based Virus Definitions --------- Version: 52.00000 Contract Expiry Date: Mon Jul 16 2018 Last Updated using push update on Thu Sep 28 17:00:11 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: Updates Installed

Botnet Definitions --------- Version: 4.00058 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu Sep 28 10:00:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

IPS Attack Engine --------- Version: 3.00430 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Tue Aug 22 20:13:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

Internet-service Database Apps --------- Version: 2.00702 Contract Expiry Date: n/a Last Updated using manual update on Wed Sep 27 11:15:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

Internet-service Database Maps --------- Version: 2.00702 Contract Expiry Date: n/a Last Updated using manual update on Wed Sep 27 11:15:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

Botnet Domain Database --------- Version: 1.00505 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu Aug 11 12:09:00 2016 Last Update Attempt: n/a Result: Updates Installed

Modem List --------- Version: 0.000

Device and OS Identification --------- Version: 1.00061 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Fri Sep 8 17:49:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

IP Geography DB --------- Version: 1.067 Contract Expiry Date: n/a Last Update Date: Fri Aug 4 15:07:26 2017

Certificate Bundle --------- Version: 1.00005 Last Update Date: Thu May 5 10:58:00 2016

FDS Address --------- 208.91.112.78-443

URL White list --------- Version: 1.00810 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu Sep 28 08:05:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

[tanr · Accepted solution]

@seadave,

Did TAC say anything about the newer virus definitions vs. the AV engine?  I thought we didn't get a new AV engine, just new virus definitions.

On our 300D and 100D (5.4.5) once virus definitions were updated from 52.00001 to 52.00003 and flow-based virus definitions were updated from 52.00001 to 52.00002 I stopped seeing the crashes.

[Yamada_Takahiro3]

Dose anyone have official announce or report from fortinet?

[seadave · Accepted solution]

Not yet, I'll update when my ticket is updated.  If they do so at all.  It seems fairly obvious now that the cause was a bad AV Defs update.  I'm now on 52.00005 with no issues.  I started considering a large purchase of FortiSwitches today.  I guess this is my reward ;)

You can check via the console with the "diag autoupdate ver" command:

AV Engine --------- Version: 5.00247 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu May 4 14:26:00 2017 Last Update Attempt: Thu Sep 28 20:36:17 2017 Result: No Updates

Virus Definitions --------- Version: 52.00005 Contract Expiry Date: Mon Jul 16 2018 Last Updated using scheduled update on Thu Sep 28 20:36:17 2017 Last Update Attempt: Thu Sep 28 20:36:17 2017 Result: Updates Installed

If you have a FAZ 5.6, we realized today you can configure an event handler to alert you when the "app crash" event fires.

Should give you a small heads up when this is happening instead of the line of people knocking on your door.

[tanr]

Thanks for finding the FAZ application crash event handler! 

I see the same one with FAZ 5.4.5 and will add an alert for myself.

[seadave]

tanr wrote:

Thanks for finding the FAZ application crash event handler! 

I see the same one with FAZ 5.4.5 and will add an alert for myself.

We just went to a local Fortinet Tech refresh and they showed us a lot with the FAZ.  I think you should consider 5.6 it has a ton of nice new features.  You can download the VM and test with that.  I don't know how anyone with a network of more than 20 people can operate without the FAZ.  Critical tool for resolving things like this.  We first saw the App Crash via the Events/Event Handler view.  Ironically Fortinet defines it as a "Medium" event.  We changed to "Critical" and configured the alerts so we'll know right away next time.

Of course going in via the console and checking "diag debug crashlog read" gives you the same indicators in a slightly less refined way.  I wish the output of that was a widget on the Dashboard.

[tanr]

The FAZ 5.6 feature list looks nice.  I'm leery about changing our FAZ to a .0 release, though.  We had a number of problems with FAZ 5.4.0.  We didn't really consider it reliable till 5.4.3.

How has your experience with the FAZ 5.6.0 been?

[Cyrielr]

Thanks guys for this tips. It's now configured on our Faz ;)

[hmtay_FTNT]

We are very sorry for the inconveniences caused with this crash. It should not have happened and we regretted it. I can confirm that there was a faulty signature that got through as a corner case. The latest signature database should have fixed the problem and preventive measures have been taken to prevent issues like this from happening again. Sorry once more.