IPSSensor + SSL Inspection + Chrome = IPS Engine crash and network connectivity interrupt

[x_member]

So this is giving me some serious grief on a FGT 60D with 5.2.7 installed, using Chrome to access a variety of sites. The issue was not present until a few days ago (long past when we upgraded from 5.2.3). No changes to policies / sensors in the meantime.

We've not been able to pin down a sample site, but I can generate the issue with some regularity loading a set of tabs (mixed https and http sites). Other times a single HTTPS site is sufficient to cause the issue.

Any guidance on how to move forward with this is much appreciated - I'm about to raise a ticket with Fortinet but have had appallingly slow response times in the past. For the moment we've banned Chrome use on the internal network - I'm able to use a (near) identical backup unit to investigate further on a virtualised test network and have been able to reproduce the issue on that setup.

Overview:

Using the latest version of Chrome 64 bit on Windows 10, although I've also been able to reproduce the issue on a Windows 7 64bit machine.

Internal - WAN traffic is permitted through with deep packet inspection, IPS Sensor, AV sensor and very limited WebFilter (excludes adult, potentially liable and security risk categories).

Using Chrome an intermittent Signal 11 crash occurs in the ips engine (see logs below). This interrupts all traffic flowing through the FGT including our customer facing sites.

Removing the SSL DPI from the policy seems to remove the issue (still confirming that). I'm investigating various possible avenues (DPI FQDN exemptions, disabling QUIC, etc.) but have not yet been able to nail down the root cause.

Event Log:

30 09:27:31 failed to send urlfilter packet [size="2"]31 09:27:31 Pid: 04282, application: ipsengine 03.164, Firmware: FortiGate-60D v5.2.7,build0718b718,160328 (GA) (Release), Signal 11 received, Backtrace: [0x30d83dd8] [0x30d96eec] [0x30d8d3c0] [0x30d8d5c4] [0x30d925a8] [0x30d92b80] [0x30d8664c] [0x30d869bc] [0x30d87508] [0x30d81ab8] [0x30c88a64] [0x30c9a59c] [0x30c65cdc] [0x008b0468] [0x008b22a0] [0x008b39c8] [0x00039908] [0x008b3e68] [0x008b4e2c] [0x00039908] [0x0003937c] [0x00037420] [0x00038f5c] [0x0003688c] [0x300e1bc4][/size]

diagnose test application ipsmonitor 3:

pid = 4282(master), duration = 107 (s) at Fri May 13 09:27:31 2016

code = 10, reason: seg fault

diagnose debug crashlog read:

5888: 2016-05-13 09:27:31 <04282> firmware FortiGate-60D v5.2.7,build0718b718,160328 (GA) (Release)

5889: 2016-05-13 09:27:31 <04282> application ipsengine 03.164 5890: 2016-05-13 09:27:31 <04282> *** signal 11 (Segmentation fault) received *** 5891: 2016-05-13 09:27:31 <04282> Register dump: 5892: 2016-05-13 09:27:31 <04282> R0: 0000001d R1: 7efc8f2c R2: 7efc902c R3: eb01ab00 5893: 2016-05-13 09:27:31 <04282> R4: 7efc8f04 R5: 00000984 R6: 0000001d R7: 3106a700 5894: 2016-05-13 09:27:31 <04282> R8: 30f04f90 R9: 31069d7c R10: 0000001d FP: 7efc8e9c 5895: 2016-05-13 09:27:31 <04282> IP: 00000004 SP: 7efc8c08 LR: 30d96eec PC: 30d83dd8 5896: 2016-05-13 09:27:31 <04282> CPSR: 20000010 Addr: eb01b764 5897: 2016-05-13 09:27:31 <04282> Trap: 0000000e Error: 00000000 OldMask: 00000800 5898: 2016-05-13 09:27:31 <04282> Backtrace: [size="2"]5899: 2016-05-13 09:27:31 <04282> [0x30d83dd8] => /data/lib/libips.so liboffset 00135dd8[/size] [size="2"]5900: 2016-05-13 09:27:31 <04282> [0x30d96eec] => /data/lib/libips.so liboffset 00148eec[/size] [size="2"]5901: 2016-05-13 09:27:31 <04282> [0x30d8d3c0] => /data/lib/libips.so liboffset 0013f3c0[/size] [size="2"]5902: 2016-05-13 09:27:31 <04282> [0x30d8d5c4] => /data/lib/libips.so liboffset 0013f5c4[/size] [size="2"]5903: 2016-05-13 09:27:31 <04282> [0x30d925a8] => /data/lib/libips.so liboffset 001445a8[/size] [size="2"]5904: 2016-05-13 09:27:32 <04282> [0x30d92b80] => /data/lib/libips.so liboffset 00144b80[/size] [size="2"]5905: 2016-05-13 09:27:32 <04282> [0x30d8664c] => /data/lib/libips.so liboffset 0013864c[/size] [size="2"]5906: 2016-05-13 09:27:32 <04282> [0x30d869bc] => /data/lib/libips.so liboffset 001389bc[/size] [size="2"]5907: 2016-05-13 09:27:32 <04282> [0x30d87508] => /data/lib/libips.so liboffset 00139508[/size] [size="2"]5908: 2016-05-13 09:27:32 <04282> [0x30d81ab8] => /data/lib/libips.so liboffset 00133ab8[/size] [size="2"]5909: 2016-05-13 09:27:32 <04282> [0x30c88a64] => /data/lib/libips.so liboffset 0003aa64[/size] [size="2"]5910: 2016-05-13 09:27:32 <04282> [0x30c9a59c] => /data/lib/libips.so liboffset 0004c59c[/size] [size="2"]5911: 2016-05-13 09:27:32 <04282> [0x30c65cdc] => /data/lib/libips.so liboffset 00017cdc[/size] [size="2"]5912: 2016-05-13 09:27:32 <04282> [0x008b0468] => /bin/ipsengine [/size] [size="2"]5913: 2016-05-13 09:27:32 <04282> [0x008b22a0] => /bin/ipsengine [/size] [size="2"]5914: 2016-05-13 09:27:32 <04282> [0x008b39c8] => /bin/ipsengine [/size] [size="2"]5915: 2016-05-13 09:27:32 <04282> [0x00039908] => /bin/ipsengine [/size] [size="2"]5916: 2016-05-13 09:27:32 <04282> [0x008b3e68] => /bin/ipsengine [/size] [size="2"]5917: 2016-05-13 09:27:32 <04282> [0x008b4e2c] => /bin/ipsengine [/size] [size="2"]5918: 2016-05-13 09:27:32 <04282> [0x00039908] => /bin/ipsengine [/size] [size="2"]5919: 2016-05-13 09:27:32 <04282> [0x0003937c] => /bin/ipsengine [/size] [size="2"]5920: 2016-05-13 09:27:32 <04282> [0x00037420] => /bin/ipsengine [/size] [size="2"]5921: 2016-05-13 09:27:32 <04282> [0x00038f5c] => /bin/ipsengine [/size] [size="2"]5922: 2016-05-13 09:27:32 <04282> [0x0003688c] => /bin/ipsengine [/size] [size="2"]5923: 2016-05-13 09:27:32 <04282> [0x300e1bc4] => ../lib/libc.so.6 (__libc_start_main+0x00000110) [/size] 5924: 2016-05-13 09:27:32 liboffset 00017bc4

[x_member]

Just a quick update as testing is still ongoing.

Looks like a patch to ips engine will fix this. I'm currently testing on our backup fortigate with a support-supplied 03.166 engine.

[x_member]

Support have confirmed that this is a known issue with 03.164 ipsengine shipped with 5.2.7.

I'm currently trying to establish whether I should wait for 5.2.8 (release date unknown) or manually update to ipsengine 03.166 (testing and other changes unknown).

Final update: After spending some time discussing with TAC, I've pushed the 3.166 engine to our live firewall to resolve this. There are no plans at this time to automatically push this updated engine via FortiCloud, so it may need requesting via a ticket if required.

[Bunce]

Thanks for posting this info CodeMonkey