FortiWifi 60D firewall 100% CPU randomly

[Scryden]

Hello,

I hope this is posted in the right forum.

FortiWifi 60D

Firmware 6.0

Please bear with me as this is quite a story.

I am struggling with an ongoing issue with a client's FortiWifi 60D firewall since September 2018. I've had multiple remote sessions with Fortinet Tech Support, for multiple hours on multiple days and Fortinet cannot figure out a solution to my problem which worries me a lot. I am trying this forum as a last line of defense before I switch them to a different brand firewall.

The firewall chokes itself at completely random times and runs on 100% (system) CPU usage. When this happens, it drops all in and outbound internet traffic. In other words, the entire office cannot access the internet. Some times this happens 2 days in a row and some times they run an entire month without any issues before this happens again. It does not only happen at peak moments. It also occurs on weekends when the office is empty and nobody is at work. There is no loop or broadcast storm in the network, as servers, computers and printers can talk to each other without a single issue when this problem occurs. So the problem is solely with the Fortinet's CPU. Restarting the firewall does not resolve the problem.

The only way to make the Fortinet calm down is to physically disconnect desktops from the network. Apparently there is something in the network triggering the Fortinet to kill itself. Which computer in the network it is, is always completely random. But once I have found the problematic machine, the fortinet calms down. When I then reconnect the desktop's cable to the network switch, everything remains normal. Fortinet Tech support witnessed this behavior and they told me there is a bug in the firmware I ran back then (v5.2.5) that causes the firewall to kill itself when desktop computers run the FortiClient software and download their definitions/updates from the firewall. The tech recommended me to upgrade to the latest version (6.0.x). So I did and it did not resolve the problem. They ran about a month without an issue but then the same thing happened again. So I uninstalled the FortiClient software on literally all the desktops, but that still did not resolve the problem. So I called Fortinet support again, they ran a "diag sys top" to look at the CPU consumption of the processes and that list of processes did not even add up to 100% CPU. But a "get system performance status" does show 100% System CPU. We then tried to mirror the traffic that goes to and from the inside interface of the firewall and redirected it to a laptop with Wireshark. We could not find anything that points to the problem. So, you won't believe this, but we ended the call with "I don't know". Now all my hope was gone. So I demonstrated the same trick to the tech and disconnected the computers from the network switches one by one until the Fortinet calmed down. The tech took a note of that and ended the call. 

Now about 2 weeks later without issues, the problem is back again and twice in a row on one day. My client's business is impacted heavily by this so I hope somebody on this forum can shine some light on this issue. If not, then after dealing with this problem for half a year I will boot the Fortinet out. One more thing I want to add: my client has multiple FortiWifi 60D sitting as spares on the shelf. I even tried to replace the firewall with a spare unit and it runs into the exact same issue. The fact that the intervals are completely random makes it hard to pinpoint.

Thanks.

[Alexis_G]

Hi

How many Desktops ? Is the sizing of the FGT adequate for your needs ?

Do you have application , antivirus control ?

Did you sniffered traffic to see if any Workstation network card gone crazy ... ?

Did you ensure that there is no infected workstation ?

If i were you I would install latest 5.4.x (as most stable)

FGT60D is not for large scale intergations, if you have a lot of features enabled you might encounter sizing issues.

[Scryden]

jklapas wrote:

Hi

How many Desktops ? Is the sizing of the FGT adequate for your needs ?

Do you have application , antivirus control ?

Did you sniffered traffic to see if any Workstation network card gone crazy ... ?

Did you ensure that there is no infected workstation ?

If i were you I would install latest 5.4.x (as most stable)

FGT60D is not for large scale intergations, if you have a lot of features enabled you might encounter sizing issues.

Hi jklapas,

See my answers below:

- We have 42 desktops in the office and 2 servers that are only for internal use (AD and file server). The issue started in September 2018. Before that they never had a problem with it.

- They do not have any applications on-premise. Everything runs in the cloud except the local fileserver. The only antivirus they had was FortiClient and I uninstalled it from all the computers upon the Fortinet Tech's recommendation.

- We port mirrored the switchport to which the firewall is connected and redirected it to a laptop with Wireshark. We did not see anything that points to the problem.

- I do not know which virus attacks solely the CPU in a firewall, but I am pretty sure it is not a virus. It is also always another connection in the network that seem to cause it. Like I said, I disconnect the workstations one by one from the network switch until the Fortinet calms down. It is always a different workstation, never the same one.

- Version 5.4 seems a bit of a jump since I am running on 6.0 now and again, upon the Fortinet tech's recommendation. Also, I had the identical problem on version 5.2.5.

- The Fortinet tech disabled literally all the features in the firewall. There is nothing enabled anymore beside some ipv4 access rule policies. Everything else like virusscan, ssl inspection, you name it is all disabled.

[Alexis_G]

So I suppose you need to troubleshoot this strange connection which you indicated that its the one causing this problem.

[Scryden]

jklapas wrote:

So I suppose you need to troubleshoot this strange connection which you indicated that its the one causing this problem.

The problem is, it's always a different connection that seems to cause it. I have had occassions where the specific cable was connected to a computer in sleep mode... It's never the same computer causing it. Always random. That is why I always have to disconnect all the PCs one by one from the network switch until the Fortinet chills out.

[lobstercreed]

Do you have any off-box logging (FortiAnalyzer or FortiCloud) configured?  I would turn logging on for all rules and see if there is any traffic pattern discernible in those logs that corresponds to the CPU spike.  It does seem to be traffic-related based on the fact that unplugging a station resolves it.  Of course, logging may not work quite right if the CPU is maxed, but it *could* help.  I would have thought Fortinet support might have suggested this too though?

I would also 2nd jklapas suggestion of running a different code train.  As you said, 6.0.x did not resolve the bug, so it seems like it couldn't hurt anything (though I understand the hesitancy to downgrade in general).  We have been running 5.6.x for ages now and have had basically no problems (albeit on totally different hardware), and I would recommend it over 5.4.x because it has much closer to the 6.0.x features on a more stable train.

[rwpatterson]

If you have shelved spares, have you tried running HA mode? Maybe offloading some 'stuff' may get you over the hump with a smaller impact while you have a chance to debug.

[Scryden]

So we have decided to replace the Fortinet with a Cisco ASA 5508-x firewall as this issue occured again 2 days in a row. I was logging all traffic from inside to outside and at the moment it happened, the only thing I could see in the logs were computers talking to Google DNS, Gmail, Office365 and FortiGuard. I dont know about the latter one.

Now I have a question. Since all of our remote users are already used to the FortiClient for remote VPN access and I want to save the client some money on an AnyConnect VPN license with Cisco, is it possible to put the Fortinet firewall behind the Cisco ASA as an internal VPN server? In other words, the Fortinet will not be a gateway anymore but will just be an inside host. I want to port forward all required FortiClient VPN ports to the Fortinet firewall so that people are still able to VPN in remotely. That way we dont need to change the VPN clients and license to AnyConnect. So basically, this is the same idea as an OpenVPN server you can run within your LAN. Can I do such a setup? Will that work? If so, which ports do I need to forward in the ASA to the Fortinet? We are using IPSec VPN for remote access, not SSL. Is that UDP/500 only? Or more?

I should mention that we have multiple Site2Site IPSec tunnels between different branches. They probably use UDP/500 as well. Those Site2Site tunnels will be made in the ASA. So the only thing I want to forward is FortiClient remote VPN access.

Thank you.

EDIT: I figured it might be easier to use SSL VPN as I can configure it to use port 8443 for example. If I port forward 8443 in the ASA and forward it to the Fortinet, will that work for remote SSL VPN?

[Scryden]

So the Fortinet just spiked up to 100% CPU again. It started at 11:30PM and the office is entirely empty and most computers are switched off at the end of the day. I was able to make a screenshot of the dashboard and the list of processes from the CLI:

If you take a look at the process list on the right, the numbers by far don't add up to 99% CPU, yet 96% goes to System.

I checked the traffic log from the time the CPU spike started and all I see is a handful of DNS queries:

This is driving me insane it makes no sense whatsoever. We are having this problem almost every day now. Times are completely random. Office is empty. Its midnight. Computers are powered down. What else can I do to troubleshoot this?

[Dibbers]

Scryden wrote:

So the Fortinet just spiked up to 100% CPU again. It started at 11:30PM and the office is entirely empty and most computers are switched off at the end of the day. I was able to make a screenshot of the dashboard and the list of processes from the CLI:

If you take a look at the process list on the right, the numbers by far don't add up to 99% CPU, yet 96% goes to System.

I checked the traffic log from the time the CPU spike started and all I see is a handful of DNS queries:

This is driving me insane it makes no sense whatsoever. We are having this problem almost every day now. Times are completely random. Office is empty. Its midnight. Computers are powered down. What else can I do to troubleshoot this?

Did you ever find a solution for this issue? I have a pair of 60Ds in HA that just randomly started having CPU spikes to 98-100%. They are running v6.0.9.