Description | This article describes high memory usage due to the cw_acd process and potential causes. The cw_acd process is the capwap daemon. |
Scope | FortiGate v7.2.x, 7.4.x, 7.6.x. |
Solution |
In this case, during initial troubleshooting high active memory is visible due to the cw_acd process and it is observed that the memory is constantly rising.
get hardware memory MemTotal: 8170452 kB MemFree: 1824136 kB Cached: 1355868 kB Active: 3343060 kB Active(anon): 3025392 kB Slab: 601072 kB
diag sys top-mem cw_acd (303): 1314123kB ipsengine (27218): 130072kB ipsengine (27459): 127907kB wad (334): 98792kB node (5089): 92812kB Top-5 memory used: 1763706kB
While this can point towards a memory leak this may not actually be the case. By default FortiGate stores information without ever deleting it on its own. This can be seen with the command diagnose wireless-controller wlac -c stats. Down below is an example of high usage in the cw_wids_mac_oui_tree. The table is used to store MAC addresses that could not be found in the known OUI database.
diagnose wireless-controller wlac -c stats cw_ap_rogue_wtp_tree : cnt=39307 mem=( 408B, 16MB) cw_wids_mac_oui_tree : cnt=2303275 mem=( 88B, 202MB) cw_wids_wep_iv_tree : cnt=779362 mem=( 128B, 99MB)
For perspective, if printing out this information with diagnose wireless-controller wlac show FortiGate will produce hundreds of thousands of lines of output.
diagnose wireless-controller wlac show all … Total 6387794 mac oui attacks detected (tree size 1875071)
Since this data is not purged over time it continues to increase, which may look like a memory leak, but in reality is not. It is possible to turn off the specific WIDS features if they are not used. In this case, set invalid-mac-oui to disable. By default, these features are enabled.
config wireless-controller wids-profile set weak-wep-iv <enable/disable> set null-ssid-probe-resp <enable/disable> set long-duration-attack <enable/disable>
Another thing that FortiGate keeps on storing and never deleting by default is rogue APs. This can be investigated with diagnose wireless-controller wlac -c ap-rogue listing all the rogue Ap information collected over time.
The corresponding entry can be found in the list displayed with diagnose wireless-controller wlac -c stats:
diagnose wireless-controller wlac -c stats cw_ap_rogue_wtp_tree : cnt=39307 mem=( 408B, 16MB)
diagnose wireless-controller wlac -c ap-rogue … C - Configured (G:accept, B:rogue, S:suppress, U:unconfigured) M - AC managed (V:vdom, C:AC, N:unmanaged) W - On wire (Y:yes, N:no) P - Phishing (F:fake, O:offending, N:no) Total Rogue-AP:1531 Rogue-AP-WTP(displayed):7718 Rogue-AP-WTP(total):7791 Total Entries: 1531
Unlike with the wids profiles, it is possible to purge this information by setting a timer when this information is getting deleted. Down below is the setting which will delete these entries every 24 hours.
config wireless-controller timers
As a workaround the cw_acd process can be killed. It cannot be gracefully restarted. It is possible to kill this process with the fnsysctl killall cw_acd. This can further be automated, if necessary. The script down below kills all the cw_acd processes every 24 hours.
config system auto-script
Down below are useful commands for investigating memory-related issues in general and with the cw_acd process in particular.
config global
get wireless-controller status diagnose wireless-controller wlac -c mpmt |