Troubleshooting Tip: FortiAP go offline when the FortiOS cw_acd process becomes stuck at 99% CPU usage

[arahman]

Description	This article describes steps to take for a number of observed instances when all managed FortiAP's go offline due to the cw_acd process being stuck at 99% CPU utilization on FortiOS 7.4.5-7.4.7,7.6.2.
Scope	FortiGate FortiOS v7.4.5 to 7.4.7 , v7.6.2.
Solution	A FortiGate running 7.4.5-7.4.7 or 7.6.2 can suddenly lose connection with all managed FortiAP's leading to an outage on wireless services. The following symptoms are observed: On WiFi & Switch Controller -> Managed FortiAP's, all managed APs show as offline: When trying to run wireless diagnostics commands, no output is returned or an error is returned. FG01-D# diag wireless-controller wlac -c wtp -------------------------------Total 0 WTPs---------------------------- FG01-D# diag wireless-controller wlac -c stats Polling data ... Polling data ... Polling data ... Polling data ... Polling data ... cw_daemon_info_print_general_param: Could not open /tmp/cwb_result. Verifying system performance with the command below shows one of the CPU cores with no idle cycles with all going to system/user space. FG01-D # get system performance status When running 'diagnose system top'  shows the daemon ('cw_acd') responsible for handling communication with the fortiAP's consuming large amounts of CPU cycles: If the above four symptoms are seen, it is possible the FortiGate is experiencing the symptoms of a problem that can occur on systems running 7.4.5-7.4.7 and 7.6.2 that was recently rectified on 7.4.8,7.6.3 and above. This is visible in the resolved issues of 7.4.8 and 7.6.3 (1101583) under the WiFi Controller section: Resolved issues | FortiGate / FortiOS 7.4.8 | Fortinet Document Library Resolved issues | FortiGate / FortiOS 7.6.3 | Fortinet Document Library There is no workaround to prevent re-occurrence on impacted versions. While it is possible to recover the system back to a working state by manually restarting the cw_acd daemon , it can become stuck again and cause another outage later on. Rectifying the problem requires updating the firmware to the version listed above or later.    To restart the cw_acd daemon, the following commands can be used: FG01-D # fnsysctl killall cw_acd FG01-D # diagnose system kill 11 <process id of cw_acd> It is important to verify that the process was restarted by running 'diagnose sys process pidof cw_acd' before and after restarting the process with the commands listed above.