Troubleshooting Tip: FortiAP go offline when the FortiOS cw_acd process becomes stuck at 99% CPU usage

A FortiGate running 7.4.5-7.4.7 or 7.6.2 can suddenly lose connection with all managed FortiAP's leading to an outage on wireless services.

The following symptoms are observed:

1. On WiFi & Switch Controller -> Managed FortiAP's, all managed APs show as offline:

2. When trying to run wireless diagnostics commands, no output is returned or an error is returned.

FG01-D# diag wireless-controller wlac -c wtp
-------------------------------Total 0 WTPs----------------------------

FG01-D# diag wireless-controller wlac -c stats

Polling data ...
Polling data ...
Polling data ...
Polling data ...
Polling data ...
cw_daemon_info_print_general_param: Could not open /tmp/cwb_result.

3. Verifying system performance with the command below shows one of the CPU cores with no idle cycles with all going to system/user space.

FG01-D # get system performance status

4. When running 'diagnose system top'  shows the daemon ('cw_acd') responsible for handling communication with the fortiAP's consuming large amounts of CPU cycles:

If the above four symptoms are seen, it is possible the FortiGate is experiencing the symptoms of a problem that can occur on systems running 7.4.5-7.4.7 and 7.6.2 that was recently rectified on 7.4.8,7.6.3 and above. This is visible in the resolved issues of 7.4.8 and 7.6.3 (1101583) under the WiFi Controller section:

Resolved issues | FortiGate / FortiOS 7.4.8 | Fortinet Document Library
Resolved issues | FortiGate / FortiOS 7.6.3 | Fortinet Document Library

There is no workaround to prevent re-occurrence on impacted versions. While it is possible to recover the system back to a working state by manually restarting the cw_acd daemon , it can become stuck again and cause another outage later on. Rectifying the problem requires updating the firmware to the version listed above or later.   

To restart the cw_acd daemon, the following commands can be used:

FG01-D # fnsysctl killall cw_acd

Another method is to kill the process using process-id:

FG01-D # diagnose sys process pidof cw_acd <----- This command is used to find process-id.
FG01-D # diagnose sys kill 11 <pid>

It is important to verify that the process was restarted by running 'diagnose sys process pidof cw_acd' before and after restarting the process with the commands listed above.