|
Symptoms:
-
High memory usage: WAD processes use a lot of memory.
-
Persistent client sessions: Client sessions remain active, accumulating in memory.
-
Conserve mode activation: The device enters conserve mode to manage low memory availability.
Diagnostic commands to address this software issue:
- System Performance Status (Memory Availability): The output highlights the low availability of free and freeable memory, indicating a high memory utilization rate, which could affect system performance if not monitored closely.
Command:
get system performance status
Snippet output:
CPU states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 1% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU6 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU7 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
Memory: 8186884k total, 7002956k used (85.5%), 755896k free (9.2%), 428032k freeable (5.3%)
- Diagnose System Top-Memory: The output shows high memory usage by multiple 'wad' processes, each consuming significant memory resources, highlighting their impact on overall system memory.
Command:
diagnose sys top-mem 99
Output:
wad (13857): 480113kB
wad (13853): 479525kB
wad (13851): 478800kB
wad (13856): 478227kB
wad (13855): 478115kB
wad (13858): 477965kB
wad (13854): 477797kB
wad (13852): 477550kB
ipsengine (559): 97110kB
ipsengine (560): 96120kB
node (260): 72884kB
appDemo (159): 66164kB
ipshelper (302): 43478kB
cid (332): 39401kB
cmdbsvr (194): 36082kB
miglogd (385): 30612kB
- Diagnostic Debug Crashlog Read: The snippet indicates the system entered a memory conserve mode due to high memory usage, which is a critical state that can affect system operations and performance.
Command:
diag debug crashlog read
Snippet output:
16341: 2025-02-26 12:27:20 service=kernel conserve=on total='7995 MB' used='7035 MB' red='7035 MB'
16342: 2025-02-26 12:27:20 green='6555 MB' msg='Kernel enters memory conserve mode'
- Diagnose WAD Memory Track: The output displays high memory consumption details for 'wad_m_fstr_factory_dispatcher' and 'wad_m_virtual_server'. During monitoring, it will be possible to see these processes exhibit high 'active' counts increasing and significant memory allocations, indicating their substantial consumption of system memory.
Command:
diagnose wad memory track
Snippet output:
============ Sum general memory stats for 12 wad processes =============
id allocs frees reallocs avg_sz in_str active bytes max cmem object name
47 119048365 119047234 0 474 535748 1131 536879 2333251431 wad_m_fstr_factory_dispatcher
147 27958043 27956525 0 1111 1209530 1518 1687679 2372829409 wad_m_virtual_server
Solution:
The issue has been resolved in the v7.2.11:1733, v7.4.8, and v7.6.3.
These updates will ensure that virtual server sessions are properly terminated and memory is managed effectively, thereby preventing the activation of conserve mode.
Immediate Workaround:
Utilizing an automated script to restart the WAD process can temporarily mitigate memory build-up until the firmware update is applied. These measures are designed to maintain device performance and reduce the frequency of entering conserve mode.
Related articles:
|
