|
Initial troubleshooting:
Verify the device has high active memory and higher than expected memory use by the cw_acd process.
get hardware memory
MemTotal: 8170452 kB
MemFree: 1824136 kB
Cached: 1355868 kB
Active: 3343060 kB
Active(anon): 3025392 kB
Slab: 601072 kB
diag sys top-mem
cw_acd (303): 1314123kB
ipsengine (27218): 130072kB
ipsengine (27459): 127907kB
wad (334): 98792kB
node (5089): 92812kB
Top-5 memory used: 1763706kB
If high memory use is not seen in any cw_acd process, it is likely a different issue. In that case, further isolate the location of the memory use following this KB article: Troubleshooting Tip: How to do initial troubleshooting of high memory utilization issues
If cw_acd is identified as having high memory use, there are different optimizations available in v7.4.4 and later and v7.6.1 and later. Check the remainder of this article for which optimizations may apply.
Diagnosing high memory use by WIDS features:
While high memory use can point towards a memory leak, this may not actually be the case. FortiGate stores Wireless Intrusion Detection System (WIDS) information without any deletion timer.
Granular cw_acd memory use can be seen with the command 'diagnose wireless-controller wlac -c stats'. Below is an example of high usage in the cw_wids_mac_oui_tree and some other locations. The mac_oui_tree table stores MAC addresses that could not be found in the known OUI database.
diagnose wireless-controller wlac -c stats
<selected lines shown>
cw_wids_wl_bridge_tree : cnt=163663 mem=( 128B, 20MB)
cw_wids_nl_pbresp_tree : cnt=24304 mem=( 136B, 3MB)
cw_wids_long_dur_tree : cnt=165816 mem=( 144B, 23MB)
cw_wids_mac_oui_tree : cnt=2303275 mem=( 88B, 202MB)
cw_wids_wep_iv_tree : cnt=779362 mem=( 128B, 99MB)
For perspective, if printing out this information with 'diagnose wireless-controller wlac show' FortiGate will produce hundreds of thousands of output lines.
diagnose wireless-controller wlac show all
# <hundreds of thousands of lines of output>
# Total 6387794 mac oui attacks detected (tree size 1875071)
Resolving high memory use by WIDS features (v7.6.1 and later):
In FortiOS v7.6.1 and later, it is possible to configure a timer and maximum value for WIDS database entries:
config wireless-controller timers
set wids-entry-cleanup (<0> to <4294967295> minutes to keep wids entry after it is gone, 0 to never expire)
end
config wireless-controller global
set max-wids-entry (<0> to <4294967295> maximum entries per table, 0 for no maximum)
end
See reference ID 1013290 in FortiOS v7.6.1 Release Notes: Changes in CLI. This change is also under review for a future v7.4 release.
Workaround for high memory use by WIDS features (v7.4.7 and earlier):
To prevent unwanted memory use by the WIDS database, disable unused WIDS features.
By default, the following features are enabled on a WIDS profile and can consume high memory in some wireless environments.
config wireless-controller wids-profile
edit <profile_name>
set wireless-bridge <enable/disable>
set weak-wep-iv <enable/disable>
set null-ssid-probe-resp <enable/disable>
set invalid-mac-oui <enable/disable>
set long-duration-attack <enable/disable>
next
end
Diagnosing high memory use by Rogue APs list:
Another thing that FortiGate does not delete by default is a list of detected rogue APs and other wireless devices. 'diagnose wireless-controller wlac -c ap-rogue' shows the rogue AP list collected over time.
The corresponding entry can be found in the list displayed with 'diagnose wireless-controller wlac -c stats':
diagnose wireless-controller wlac -c stats
cw_rbtts_sta_cap_tree : cnt=524416 mem=( 248B, 130MB)
cw_sta_cap_wtp_tree : cnt=645861 mem=( 288B, 186MB)
cw_rbtts_ap_rogue_tree : cnt=234567 mem=( 560B, 131MB)
cw_ap_rogue_wtp_tree : cnt=456789 mem=( 408B, 186MB)
diagnose wireless-controller wlac -c ap-rogue
…
C - Configured (G:accept, B:rogue, S:suppress, U:unconfigured)
M - AC managed (V:vdom, C:AC, N:unmanaged)
W - On wire (Y:yes, N:no)
P - Phishing (F:fake, O:offending, N:no)
Total Rogue-AP:234567 Rogue-AP-WTP(displayed):450000 Rogue-AP-WTP(total):456789
Total Entries: 234567
Resolving high memory use by Rogue APs list (v7.4.4 and later):
In v7.4.4 and above, several enhancements are available for wireless station record memory optimization. See reference ID 983561 in FortiOS v7.4.4 Release Notes: New features or enhancements.
It is possible to set a timer to purge these records if the device has not been seen in some time. By default the Rogue AP timer is configured as 'set rogue-ap-cleanup 0', meaning a detected Rogue AP is never forgotten.
The following setting removes an entry 24 hours after it was last seen:
config wireless-controller timers
set sta-cap-cleanup 1440
set rogue-ap-cleanup 1440
set rogue-sta-cleanup 1440
set ble-device-cleanup 1440
end
A maximum number of entries can be configured to limit the maximum possible memory use:
config wireless-controller global
set max-sta-cap (<0> to <4294967295>, default is 0 for no maximum)
set max-rogue-ap (<0> to <4294967295>, default is 0 for no maximum)
set max-rogue-sta (<0> to <4294967295>, default is 0 for no maximum)
set max-ble-device (<0> to <4294967295>, default is 0 for no maximum)
end
If FortiGate is managing several FortiAPs that are each checking for rogue APs, the same Station Capability or rogue AP entry may get detected multiple times by several nearby FortiAPs. A limit on the number of FortiAPs associated with a detection entry can further limit memory use:
config wireless-controller global
set max-sta-cap-wtp (<1> to <8>, default is 0.)
set max-rogue-ap-wtp (<1> to <8>, default is 0.)
end
Workaround for high memory use by cw_acd (required for FortiOS v7.4.3 and earlier):
As a workaround, the cw_acd process can be killed to clear all WIDS and Rogue AP tables. The cw_acd process cannot be gracefully restarted, and some managed FortiAPs may reboot when it is killed. The command to kill cw_acd is 'fnsysctl killall cw_acd'.
This can further be automated, if necessary. The script down below kills all the cw_acd processes every 24 hours.
config system auto-script
edit "killall_cw_acd"
set interval 86400
set repeat 0
set start auto
set script "fnsysctl killall cw_acd"
next
end
Diagnostic commands:
The following commands can be useful for investigating memory-related issues in general and with the cw_acd process in particular.
Performance:
config global
get sys status
get sys performance status
diagnose hardware sysinfo memory
diagnose sys vd stats
diagnose sys top-mem 99
diagnose sys top-fd 50
diagnose sys top 1 99 5
diagnose sys mpstat 2 5
fnsysctl ps aux
fnsysctl cat /proc/net/sockstat
fnsysctl cat /proc/softirqs
fnsysctl ifconfig
sudo root diagnose netlink interface list
diagnose sys vd list
diagnose sys cmdb info
diag debug report
Wireless:
get wireless-controller status
diagnose wireless-controller wlac -c mpmt
diagnose wireless-controller wlac -c ws
diagnose wireless-controller wlac -c vap
diagnose wireless-controller wlac -c stats
diagnose wireless-controller wlac -d usage
diagnose wireless-controller wlac show mp
diagnose wireless-controller wlac -p data -c stats
diagnose wireless-controller wlac -p 1 -c stats
diagnose wireless-controller wlac -p 1 show mp
diagnose wireless-controller wlac show all
diagnose wireless-controller wlac -c ap-rogue
diagnose wireless-controller wlac -c sta-cap
|
