Crashing of Application WAD

[Jeroen]

Hello to you all,

I have a problem with crashing application within the Fortigate 1500D. So far I can see it is the WAD application (Cache & Wan optimization deamon). The strange thing that some traffic seems to have problems with it crashing. The next strange thing is that the feature “WAN Link Load Balancing” isn’t activated. What can be the cause of this application to be crashing all the time.

Firmware: V5.2.1

Type Fortigate: 1500D

Has somebody have a idea?

date=2014-12-31 time=13:24:02 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28447, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]" date=2014-12-31 time=13:20:50 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28442, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]" date=2014-12-31 time=13:19:52 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28441, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]" date=2014-12-31 time=13:19:02 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28435, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]" date=2014-12-31 time=13:18:08 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28431, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x01216efa] [0x012252d4] [0x0121378c] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]"

[Dave_Hall]

Someone correct me on this, but I thought that daemon was also responsible for web caching.

If you are not actually using web caching on the Fortigate then I suggest disabling that feature. (e.g. zeroing the logdisk storage allocated to it, and also disabling any web caching features on your web firewall traffic policy.)

If this was a smaller unit (.e.g. <200B) I would say the logdisk is likely corrupted and needs to be reformatted.  Caution though if you choose to do this you will loose your logs, DLP files, backup configs, etc.  But I would wait to see what the other vets on the forums say about this.

[Jeroen]

I found a BUG ID 0259651 with a memory leak in the WAD Worker Process. It's been fixed in release 5.2.2. Fortinet support engineer could not verify that it's is the same bug as reported above. So I am going to find a maintenance window to update the Fortigate 1500D unit. To see if that issue has been fixed.

[ISOffice]

Hi all,

We have had an on-going issue with web caching with our 100D Cluster (v5.2.3, build 670) and I'm starting to see the same error messages as Jeroen. Due to caching not working on v5.2.2, we upgraded to v5.2.3. I have a call logged with Fortinet in relation to this issue and will post any further developments.

John P

[Jeroen · Accepted solution]

We have updated the 1500D unit to version 5.2.2 and this solved the issue at our side. We have planned a update to version 5.2.3 as I see your message this is maybe not a good idea as the problems seems to return at version 5.2.3. I am curious at what Fortinet will return in the ticket.

ISOffice wrote:

Hi all,

We have had an on-going issue with web caching with our 100D Cluster (v5.2.3, build 670) and I'm starting to see the same error messages as Jeroen. Due to caching not working on v5.2.2, we upgraded to v5.2.3. I have a call logged with Fortinet in relation to this issue and will post any further developments.

John P

[Upendra]

Hello

I having FortiGate-100D with FortiOS 5.4.0 and since from last 1 week. I am getting so many same error logs. Please help in this regards.

Troubleshooting done

1) exe formatlogdisk

2) Disable Local Reports.

3) Disable Threat weight.

------------------------------------------------------------------------------------------------------------

#3ActioncrashDate/Time15:50:26Level Log DescriptionApplication crashedLog ID32546MessagePid: 00548, application: wad, Firmware: FortiGate-100D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 11 received, Backtrace: [0x7ffa36a73c42]Sub TypesystemTimestamp11/7/2016, 3:50:26 PMVirtual Domainroot

----------------------------

Regards

Upendra Makwana

[denkerzone]

Hello

I am having a device FGT100D. The firmware is 5.4.0. In this firmware I faced some Application Crashed issues. The firmware is upgraded by me to 5.4.1, now. Still I am facing the same issue.

[align=center]Few crash logs during 5.4.0:[/align]

date=2016-11-07 time=17:17:30 logid=0100032546 type=event subtype=system level=warning vd=root logdesc="Application crashed" action=crash msg="Pid: 00232, application: wad, Firmware: FortiGate-100D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 7 received, Backtrace: [0x7f04e7251be3]"

date=2016-11-07 time=17:11:13 logid=0100032546 type=event subtype=system level=warning vd=root logdesc="Application crashed" action=crash msg="Pid: 04195, application: fort4195 �&#, Firmware: FortiGate-100D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 11 received, Backtrace: [0x0167491e] [0x01667988] [0x016659ea] [0x01625d6a] [0x01626734] [0x016677e1] [0x01665996] [0x0166b74e] [0x0166bbc2] [0x0076b1a6] [0x0076544b] [0x012139fc] [0x01213b62] [0x01213b89] [0x01213b89] [0x01213b89] [0x01213bf9] [0x00760c55] [0x0043d5e0] [0x004436e7] [0x004416b0] [0x00443318] [0x0043ab87] [0x7f1918cfd475] [0x0043ac61]"

[align=center]Few crash logs during 5.4.1:[/align]

date=2016-11-08 time=18:31:28 logid=0100032546 type=event subtype=system level=warning vd=root logdesc="Application crashed" action=crash msg="Pid: 00220, application: ipsengine 03.279, Firmware: FortiGate-100D v5.4.1,build1064b1064,160608 (GA) (Release), Signal 11 received, Backtrace: [0x7f6b89bd48a1] [0x7f6b89bd634d] [0x7f6b899ac71c] [0x7f6b899ac759] [0x7f6b899b8983] [0x7f6b899b981a] [0x7f6b8995eefd] [0x7f6b8993c809] [0x00cfb8b4] [0x00cfbddc] [0x00cfc0e4] [0x00cfc177] [0x00cfe15a] [0x00427550] [0x00cfe94a] [0x00cff040] [0x00427550] [0x0042dd77] [0x0042b4d0] [0x0042d138] [0x0042d8d3] [0x7f6b8dab9475] [0x00424c49]"

date=2016-11-08 time=18:04:47 logid=0100032546 type=event subtype=system level=warning vd=root logdesc="Application crashed" action=crash msg="Pid: 00283, application: wad, Firmware: FortiGate-100D v5.4.1,build1064b1064,160608 (GA) (Release), Signal 11 received, Backtrace: [0x7fdd2da47e96] [0x0138a091] [0x014973e6] [0x014800e8] [0x0142e36a] [0x0142e596] [0x0142dacd] [0x0146c61e] [0x0042737c] [0x0042d51f] [0x7fdd2d91e475] [0x00424c49]"

Thank you.

Regards,

Harshil Gupta

[ronnyg]

With 5.6 and 5.6.1, same problem on a 101E :(

I was wondering some traffic was dropped,

disabling antivir in the rule, let it work. My first idea was the IDP is hanging, but it is the antivir rule.

Bad bug again ;(. It was there in FortiOS from 5.2 and comes back again and again ;-(

Event Message Pid: 03367, application: wad, Firmware: FortiGate-101E v5.6.1,build1484b1484,170727 (GA) (Release), Signal 11 received, Backtrace: [0x0091bc9e] [0x0091d2a8] [0x0091eb50] [0x00928414] [0x00928a00] [0x009291f6] [0x0092a21e] [0x0092f226] [0x00930ba8] [0x00996146] [0x009963b0] [0x00975a14] [0x009a56a6] [0x00024044] [0x00027678] [0x36cbff86]

[a_ymeri]

similar bug for sslvpn daemon application, FortiGate 51E, FortiOS 5.6.2

 application: sslvpnd,  Signal 11 received, Backtrace: [0x005502c8] [0x00551674] [0x005576c4] [0x007befbe] [0x007c6cf6] [0x007efc0c] [0x007efc2e] [0x007f1b34] [0x007f5a96] [0x007fab60] [0x0080f22e] [0x0080fed8] [0x008105de] [0x00810aa2] [0x000283ac] [0x0002c010] [0x0002a594] [0x0002b74a] [0x0002bde6] [0x56d46f86]

[dieter]

We're seeing similar wad crashes on a 500D cluster (happens only on the master).

Pid: 01795, application: wad, Firmware: FortiGate-500D v5.6.2,build1486b1486,170816 (GA) (Release), Signal 11 received, Backtrace: [0x7f4aedd789f9] [0x0159fe08] [0x01561065] [0x01bd39fb] [0x01bc7ee1] [0x015643d5] [0x01564fb6] [0x015791e0] [0x0153ba5a] [0x0158e3db] [0x0042a38c] [0x00430866] [0x7f4aedc69475] [0x004279e9]

Multiple crashes daily