|
Root cause:
The root cause has been identified as memory exhaustion in the wireless controller module. The memory exhaustion takes some time to occur; it may happen after several weeks of continuous operation on affected firewall firmware versions.
How to identify the issue using diagnostic commands:
Packet sniffer for port 5246 traffic shows incoming multicast and broadcast for FortiAPs, but no responses.
diagnose sniffer packet any "port 5246" 4 0 a
2025-04-22 11:29:13.261571 lan in 192.168.11.56.5246 -> 224.0.1.140.5246: udp 384
2025-04-22 11:29:14.041659 lan in 192.168.11.56.5246 -> 224.0.1.140.5246: udp 384
2025-04-22 11:29:15.981491 lan in 192.168.11.56.5246 -> 255.255.255.255.5246: udp 384
2025-04-22 11:29:17.261489 lan in 192.168.11.56.5246 -> 255.255.255.255.5246: udp 384
The cw_acd daemon consumes high CPU utilization as shown below.
diagnose sys top 2 20
Run Time: 101 days, 10 hours and 28 minutes
5U, 0N, 8S, 87I, 0WA, 0HI, 0SI, 0ST; 1917T, 387F
cw_acd 232 R 99.9 2.8 4
cw_acd daemon debug reports 'NO MEM' errors:
diagnose debug application cw_acd -1
diagnose debug console timestamp enable
diagnose debug enable
2025-05-07 11:27:54 95140.516 581 --- ==========================cwAcProcInputLocalMsg 18 1=========================
2025-05-07 11:27:54 95140.516 581 ERR: NO MEM for USER_LOCAL_MSG
2025-05-07 11:27:54 95140.516 581 --- ERR: NO MEM for USER_CTLMSG_UDP_RECV
2025-05-07 11:27:54 95140.516 581 --- ==========================cwAcProcInputLocalMsg 18 1=========================
2025-05-07 11:27:54 95140.516 581 ERR: NO MEM for USER_LOCAL_MSG
2025-05-07 11:27:54 95140.516 581 --- ERR: NO MEM for USER_CTLMSG_UDP_RECV
These messages indicate there is no memory available to address the incoming UDP control traffic.
As a result, all managed FortiAP devices show as offline, and wireless users are unable to access the network. This issue can also cause FortiSwitches to show as offline, but it does not affect environments that have no managed FortiAP devices.
Fix for this issue:
The issue is tracked by internal ID# 1151713 and has been resolved in:
v7.4.8 (available to download from the Fortinet support portal).
v7.6.4 (available to download from the Fortinet support portal).
v8.0.0 (scheduled to be released in February 2026).
These timelines for firmware release are estimates and may be subject to change.
Workaround for this issue until the GA build is available:
Restart the cw_acd process to bring the access points online.
diagnose sys process pidof cw_acd <----- Find the process ID for the cw_acd process.
diagnose sys kill 11 <process id>
Note:
Restarting the cw_acd process cleanly using 'diagnose test application cw_acd 99' is not possible while the process memory is exhausted.
|
