FortiWifi 60D firewall 100% CPU randomly

[Scryden]

Hello,

I hope this is posted in the right forum.

FortiWifi 60D

Firmware 6.0

Please bear with me as this is quite a story.

I am struggling with an ongoing issue with a client's FortiWifi 60D firewall since September 2018. I've had multiple remote sessions with Fortinet Tech Support, for multiple hours on multiple days and Fortinet cannot figure out a solution to my problem which worries me a lot. I am trying this forum as a last line of defense before I switch them to a different brand firewall.

The firewall chokes itself at completely random times and runs on 100% (system) CPU usage. When this happens, it drops all in and outbound internet traffic. In other words, the entire office cannot access the internet. Some times this happens 2 days in a row and some times they run an entire month without any issues before this happens again. It does not only happen at peak moments. It also occurs on weekends when the office is empty and nobody is at work. There is no loop or broadcast storm in the network, as servers, computers and printers can talk to each other without a single issue when this problem occurs. So the problem is solely with the Fortinet's CPU. Restarting the firewall does not resolve the problem.

The only way to make the Fortinet calm down is to physically disconnect desktops from the network. Apparently there is something in the network triggering the Fortinet to kill itself. Which computer in the network it is, is always completely random. But once I have found the problematic machine, the fortinet calms down. When I then reconnect the desktop's cable to the network switch, everything remains normal. Fortinet Tech support witnessed this behavior and they told me there is a bug in the firmware I ran back then (v5.2.5) that causes the firewall to kill itself when desktop computers run the FortiClient software and download their definitions/updates from the firewall. The tech recommended me to upgrade to the latest version (6.0.x). So I did and it did not resolve the problem. They ran about a month without an issue but then the same thing happened again. So I uninstalled the FortiClient software on literally all the desktops, but that still did not resolve the problem. So I called Fortinet support again, they ran a "diag sys top" to look at the CPU consumption of the processes and that list of processes did not even add up to 100% CPU. But a "get system performance status" does show 100% System CPU. We then tried to mirror the traffic that goes to and from the inside interface of the firewall and redirected it to a laptop with Wireshark. We could not find anything that points to the problem. So, you won't believe this, but we ended the call with "I don't know". Now all my hope was gone. So I demonstrated the same trick to the tech and disconnected the computers from the network switches one by one until the Fortinet calmed down. The tech took a note of that and ended the call. 

Now about 2 weeks later without issues, the problem is back again and twice in a row on one day. My client's business is impacted heavily by this so I hope somebody on this forum can shine some light on this issue. If not, then after dealing with this problem for half a year I will boot the Fortinet out. One more thing I want to add: my client has multiple FortiWifi 60D sitting as spares on the shelf. I even tried to replace the firewall with a spare unit and it runs into the exact same issue. The fact that the intervals are completely random makes it hard to pinpoint.

Thanks.

[Dibbers]

I ended up finding the cause for my issues. Viewing the system event log, I noticed that the fgt was doing a scheduled fabric update right before the CPU spike notifications. 

I changed the fortiguard update settings from Scheduled to Push, and so far the notifications for high CPU usage have stopped. 

[rstefano]

I found a completely different reason for the high CPU utilization. It was the act of device scanning on the internal lan interface. If you navigate to network then interface, edit the lan interface and under the section marked networked devices disable Active scanning.  The firewall is now back to running at its typical 5 to 9% and spiking to 30 and 40 periodically but that's normal based on traffic patterns. Why this happened after the upgrade is beyond me. I'm wondering if it is something corrupted during the upgrade and if doing a clean install would simply clear the issue but leave the active scanning in place

[rstefano]

This just happened to me as well on a 60D that I upgraded from 6.0.12 to 6.0.14. I'm going to downgrade and let you know what happens

[rstefano]

Forgot to mention that running 'diag sys top' reveals that the netscan service is consuming about 93% of the CPU consistently. I haven't downgraded yet and the firewall is no longer blocking all traffic anymore but it's inconsistent. I'll update you as soon as I can