FortiGate 400E FortiOS 7.0.13 WAD extreme CPU usage

[etkonwgno]

Hi, sharing my expirience from several days ago, not sure if anyone else had similar experience.

Was upgrading all my FortiGates several days ago following the release of 7.0.13, various models with different configurations and from different versions, all went really well without issues until the last FW on the list, our 400E cluster, used for Explicit proxy and ssl vpn, considering its cpu at peak hours is 10% with 10K sessions I would say it doesn't do that much. I begin the upgrade from 7.0.12 - so a minor upgrade, I gave it several minutes, it switched back over to the primary unit - so upgrade done.

After the upgrade, I connected back to org net, ssl vpn works fine - good. but then I see "ERR_EMPTY_RESPONE" when browsing via the proxy - something is wrong, check the FGT dashboard - CPU is at 99% (All core - not single core) its stays at 99% for a minute, WAD is crashing, after a minute again 100% and WAD is crashing.. tried a reboot and immediately the same behavior - on secondary machine, After the reboot, primary up - same thing.

Thankfully downgrade back to 7.0.12 worked fine, I've tried upgrading again - exactly the same thing happened, upgrade was done at night so load was minimal, sadly even at that time I didn't really have time to debug so I just downgraded again to 7.0.12 , both times the issue disappeared when going back to 7.0.12 so its 100% the versions fault .. opened a TAC, obviously they sent a mile long list of debug commands and now I need to schedule another maintenance window just to temporarily make the firewall unusable and hope that my "luck" with downgrading will not suddenly end.

[abarushka]

Hello,

Frequent wad crashes can potentially cause high CPU usage. You may consider to download debug.log file and check frequency of wad crashes by searching for string "wad crashed <> times. The latest crash was at <>."

[Kangming]

Hi 

Could you share your ticket id? I'd like to see the relevant configuration files and captured debug information. Thanks.