Crashing of Application WAD

[Jeroen]

Hello to you all,

I have a problem with crashing application within the Fortigate 1500D. So far I can see it is the WAD application (Cache & Wan optimization deamon). The strange thing that some traffic seems to have problems with it crashing. The next strange thing is that the feature “WAN Link Load Balancing” isn’t activated. What can be the cause of this application to be crashing all the time.

Firmware: V5.2.1

Type Fortigate: 1500D

Has somebody have a idea?

date=2014-12-31 time=13:24:02 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28447, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]" date=2014-12-31 time=13:20:50 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28442, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]" date=2014-12-31 time=13:19:52 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28441, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]" date=2014-12-31 time=13:19:02 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28435, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]" date=2014-12-31 time=13:18:08 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28431, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x01216efa] [0x012252d4] [0x0121378c] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]"

[dieter]

In our case the crashes seem to be related to specific ssl decrypted traffic.

TA suggested upgrading to 5.6.3, but without much explanation (bug report or other related issue)...

[rickyrickuk]

dieter wrote:

In our case the crashes seem to be related to specific ssl decrypted traffic.

TA suggested upgrading to 5.6.3, but without much explanation (bug report or other related issue)...

We are getting these multiple times a minute (FG500E) during peak times and every couple of minutes during off peak, we are on 5.6.3 already so not sure that will fix.

[zack]

Anyone find a solution to this? We have a Fortigate 200D cluster that is doing the same thing. The WAD daemon keeps crashing when we have UTM policies in place. It was happening on 5.6.8 and support recommended we upgrade. We're on 5.6.12 now and it's still happening. So far support is stumped.... If anyone came across a solution, sharing it here would be much appreciated. 

We're not using any caching..

[kphed]

zack wrote:
Anyone find a solution to this? We have a Fortigate 200D cluster that is doing the same thing. The WAD daemon keeps crashing when we have UTM policies in place. It was happening on 5.6.8 and support recommended we upgrade. We're on 5.6.12 now and it's still happening. So far support is stumped.... If anyone came across a solution, sharing it here would be much appreciated.    We're not using any caching..

Same issue on 5.6.11!! Any help?!

[zack]

I opened a ticket with Fortinet and they acknowledged it was a bug. Furthermore, since our 200Ds were end of life and not receiving firmware updates anymore - there would be no "official" fix. Since I was under a support agreement and I escalated the issue (since they could offer no solution), Fortinet ended up creating a custom firmware image off of the 6.0.9 branch for us which contained the fix. 

The ticket number was:

3909060

We had to run for an extended time with our proxy based inspection down which meant no UTM policies. I was NOT happy about that. 

But there's good news. Fortinet sold us some new firewalls at a discount because of all the issues. We got 3 100Fs. Bad news - the WAD service still crashes on them too running 6.2.4. I have a ticket in on that one too. Ticket # 4088435. Still no fix!

My next firewall will NOT be a Fortinet. I buy these firewalls to run UTM policies. If I can't run them because it causes the firewall to crash, then WTF am I paying for? I can download a basic port filtering firewall/router for free! (Untangle of PFSense).

I know it sucks to hear, but my only recommendation can be to buy another brand at this point. 

[zack]

You have to disable proxy based inspection (SSL inspection). Set it to certificate only as opposed to full SSL inspection. By the way - this will disable the effectiveness of your UTM policies since the firewall can't look inside SSL sessions to inspect the traffic. But at least they'll stop crashing. 

You need to engage Fortinet support and ask for a custom firmware image for your firewalls that contain the fix.