FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
Ehanssen
Staff
Staff
Article Id 355837
Description This article describes high memory usage due to the cw_acd process and potential causes. The cw_acd process is the capwap daemon.
Scope FortiGate v7.2.x, 7.4.x, 7.6.x.
Solution
  • Symptoms:

In this case, during initial troubleshooting high active memory is visible due to the cw_acd process and it is observed that the memory is constantly rising.

 

get hardware memory

MemTotal:        8170452 kB

MemFree:         1824136 kB

Cached:          1355868 kB

Active:          3343060 kB 

Active(anon):    3025392 kB

Slab:             601072 kB

 

diag sys top-mem

cw_acd (303): 1314123kB

ipsengine (27218): 130072kB

ipsengine (27459): 127907kB

wad (334): 98792kB

node (5089): 92812kB

Top-5 memory used: 1763706kB

  

  • Wids profiles:

While this can point towards a memory leak this may not actually be the case. By default FortiGate stores information without ever deleting it on its own. This can be seen with the command diagnose wireless-controller wlac -c stats.

Down below is an example of high usage in the cw_wids_mac_oui_tree. The table is used to store MAC addresses that could not be found in the known OUI database.

 

diagnose wireless-controller wlac -c stats

cw_ap_rogue_wtp_tree                    : cnt=39307      mem=(     408B, 16MB)

cw_wids_mac_oui_tree                    : cnt=2303275    mem=(      88B, 202MB)    

cw_wids_wep_iv_tree                     : cnt=779362     mem=(     128B, 99MB)  

 

For perspective, if printing out this information with diagnose wireless-controller wlac show FortiGate will produce hundreds of thousands of lines of output.

 

diagnose wireless-controller wlac show all

…

Total 6387794 mac oui attacks detected (tree size 1875071)

 

Since this data is not purged over time it continues to increase, which may look like a memory leak, but in reality is not. It is possible to turn off the specific WIDS features if they are not used. In this case, set invalid-mac-oui to disable. By default, these features are enabled.

 

config wireless-controller wids-profile
    set wireless-bridge <enable/disable>

    set weak-wep-iv <enable/disable>

    set null-ssid-probe-resp <enable/disable>
    set invalid-mac-oui <enable/disable>

    set long-duration-attack <enable/disable>

 

  • Rogue APs:

Another thing that FortiGate keeps on storing and never deleting by default is rogue APs. This can be investigated with diagnose wireless-controller wlac -c ap-rogue listing all the rogue Ap information collected over time.

 

The corresponding entry can be found in the list displayed with diagnose wireless-controller wlac -c stats:

 

diagnose wireless-controller wlac -c stats

cw_ap_rogue_wtp_tree                    : cnt=39307      mem=(     408B, 16MB)

 

diagnose wireless-controller wlac -c ap-rogue

   …

   C - Configured  (G:accept, B:rogue, S:suppress, U:unconfigured)

   M - AC managed  (V:vdom, C:AC, N:unmanaged)

   W - On wire     (Y:yes, N:no)

   P - Phishing    (F:fake, O:offending, N:no)

   Total Rogue-AP:1531 Rogue-AP-WTP(displayed):7718 Rogue-AP-WTP(total):7791

   Total Entries: 1531

 

Unlike with the wids profiles, it is possible to purge this information by setting a timer when this information is getting deleted. Down below is the setting which will delete these entries every 24 hours.

 

config wireless-controller timers
    set rogue-ap-cleanup 1440
end


Per default the timer is configured as set rogue-ap-cleanup 0, meaning the list is never deleted.
Over time the memory usage will therefore increase. It therefore might cause a memory leak-looking behavior where memory increases and never gets released.

 

  • Potential Workaround:

As a workaround the cw_acd process can be killed. It cannot be gracefully restarted. It is possible to kill this process with the fnsysctl killall cw_acd.

This can further be automated, if necessary. The script down below kills all the cw_acd processes every 24 hours.

 

config system auto-script
    edit "killall_cw_acd"
        set interval 86400
        set repeat 0
        set start auto

        set script "fnsysctl killall cw_acd"

    next
end

 

Down below are useful commands for investigating memory-related issues in general and with the cw_acd process in particular.

 

  • Performance:

 

config global
get sys status
get sys performance status
diagnose hardware sysinfo memory
diagnose sys vd stats
diagnose sys top-mem 99
diagnose sys top-fd 50
diagnose sys top 1 99 5
diagnose sys mpstat 2 5
fnsysctl ps aux
fnsysctl cat /proc/net/sockstat
fnsysctl cat /proc/softirqs
fnsysctl ifconfig
sudo root diagnose netlink interface list
diagnose sys vd list
diagnose sys cmdb info
diag debug report

 

  • Wireless:

 

get wireless-controller status

diagnose wireless-controller wlac -c mpmt
diagnose wireless-controller wlac -c ws
diagnose wireless-controller wlac -c vap
diagnose wireless-controller wlac -c stats
diagnose wireless-controller wlac -d usage
diagnose wireless-controller wlac show mp
diagnose wireless-controller wlac -p data -c stats
diagnose wireless-controller wlac -p 1 -c stats
diagnose wireless-controller wlac -p 1 show mp
diagnose wireless-controller wlac show all
diagnose wireless-controller wlac -c ap-rogue
diagnose wireless-controller wlac -c sta-cap