|
If WAD processes hang or WAD takes up lots of memory, it is possible to restart the WAD process to resolve it.
Refer to below steps for FortiGate or FortiProxy devices :
Method 1.
Step 1: Run the CLI command 'get system performance status', the output will look similar to the sample below:
get system performance status
CPU states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU0 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
Memory: 2004540k total, 586528k used (29%), 1418012k free (71%)
Average network usage: 1 / 0 kbps in 1 minute, 0 / 0 kbps in 10 minutes, 0 / 0 kbps in 30 minutes
Average sessions: 25 sessions in 1 minute, 25 sessions in 10 minutes, 25 sessions in 30 minutes
Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 0 days, 23 hours, 41 minutes
Run the command above a few times and compare patterns of memory usage, throughput, and number of sessions.
Step 2: To check the WAD parent process with the command diag sys top-summary -------->The top-summary command was deprecated on FOS 6.4x and onwards
FPX # diag sys top-summary
CPU [||||||||||||||||||||||||||||||||||||||||] 100.0%
Mem [||||||||||||||||||||||||||||| ] 73.7% total (3.4% reclaimable)
Processes: 20 (running=1 sleeping=123)
PID RSS ^CPU% MEM% FDS TIME+ NAME
* 1089 36M 2.9 1.8 12 00:01.22 sshd [x4]
1115 56M 1.0 2.8 136 32:41.10 wad [x7]
1287 58M 0.0 2.9 13 00:08.15 pyfcgid [x4]
1046 51M 0.0 2.6 10 06:30.74 cmdbsvr
Use the options available to expand the list of processes, for example:
FPX # diagnose sys top 1 25 3
Where:
- '1' is the interval in seconds between each refresh. A low number for the refresh interval is useful for isolating the processes causing CPU spikes, but may not allow a large list of applications to be displayed.
- '25' is the number of lines being displayed during an iteration.
- '3' is the number of iterations this command should display.
Step 3: Restart the process with command # 'diag sys kill 11 <pid>' or using 'fnsysctl killall wad'
FPX # diag sys kill 11 1115
or
FPX # fnsysctl killall wad
Step 4: To verify and find the FPX created new pid value for the WAD parent process.
FPX # diag sys top-summary
CPU [||||||||||||||||||||||||||||||||||||||||] 100.0%
Mem [|||||||||||||||||||||||||||| ] 71.8% total (3.4% reclaimable)
Processes: 20 (running=4 sleeping=118)
PID RSS ^CPU% MEM% FDS TIME+ NAME
* 23682 49M 0.0 2.5 12 00:00.42 pyfcgid [x4]
1046 51M 0.0 2.6 10 06:30.77 cmdbsvr
1182 143M 0.0 7.2 32 06:28.71 scanunitd [x3]
23843 35M 0.0 1.8 65 00:03.25 wad [x7]
1087 55M 0.0 2.8 18 03:42.72 httpsd [x5]
FPX crashlog generates a wad signal 11 log
FPX # diag debug crashlog read
1876: 2022-05-23 01:15:28 <01115> *** signal 11 (Segmentation fault) received ***
1877: 2022-05-23 01:15:28 <01115> Register dump:
1878: 2022-05-23 01:15:28 <01115> RAX: fffffffffffffffc RBX: 0000000000000004
1879: 2022-05-23 01:15:28 <01115> RCX: 00007ff8874eadc0 RDX: 0000000000000006
1880: 2022-05-23 01:15:28 <01115> R8: 0000000000000000 R9: 0000000000000008
1881: 2022-05-23 01:15:28 <01115> R10: 0000000000001388 R11: 0000000000000246
1882: 2022-05-23 01:15:28 <01115> R12: 0000000000000018 R13: 0000000000000000
1883: 2022-05-23 01:15:28 <01115> R14: 0000000000000000 R15: 0000000000000000
1884: 2022-05-23 01:15:28 <01115> RSI: 0000000003d66be0 RDI: 0000000000000005
1885: 2022-05-23 01:15:28 <01115> RBP: 00007ffd8fd815e0 RSP: 00007ffd8fd815b8
1886: 2022-05-23 01:15:28 <01115> RIP: 00007ff8874eadc0 EFLAGS: 0000000000000246
1887: 2022-05-23 01:15:28 <01115> CS: 0033 FS: 0000 GS: 0000
1888: 2022-05-23 01:15:28 <01115> Trap: 0000000000000000 Error: 0000000000000000
1889: 2022-05-23 01:15:28 <01115> OldMask: 0000000000000000
1890: 2022-05-23 01:15:28 <01115> CR2: 0000000000000000
1891: 2022-05-23 01:15:28 <01115> stack: 0x7ffd8fd815b8 - 0x7ffd8fd822d0
1892: 2022-05-23 01:15:28 <01115> Backtrace:
1893: 2022-05-23 01:15:28 <01115> [0x7ff8874eadc0] => /fortidev/lib/x86_64-linux-gnu/libc.so.6
1894: 2022-05-23 01:15:28 (epoll_pwait+0x00000020) liboffset 000f4dc0
1895: 2022-05-23 01:15:28 <01115> [0x00ec0202] => /bin/wad
1896: 2022-05-23 01:15:28 <01115> [0x00f1e204] => /bin/wad
1897: 2022-05-23 01:15:28 <01115> [0x0042ec84] => /bin/wad
1898: 2022-05-23 01:15:28 <01115> [0x00434ebf] => /bin/wad
1899: 2022-05-23 01:15:28 <01115> [0x00432128] => /bin/wad
1900: 2022-05-23 01:15:28 <01115> [0x00432518] => /bin/wad
1901: 2022-05-23 01:15:28 <01115> [0x004342d4] => /bin/wad
1902: 2022-05-23 01:15:28 <01115> [0x00434ad5] => /bin/wad
1903: 2022-05-23 01:15:28 <01115> [0x7ff887416eaa] => /fortidev/lib/x86_64-linux-gnu/libc.so.6
1904: 2022-05-23 01:15:28 (__libc_start_main+0x000000ea) liboffset 00020eaa
1905: 2022-05-23 01:15:28 <01115> [0x0042b5ca] => /bin/wad
1906: 2022-05-23 01:15:29 <01115> process=wad type=0 idx=-1 av-scanning=no total=2006 free=626 mmu=1176
1907: 2022-05-23 01:15:29 mu=616 m=28 f=20 r=0
1908: 2022-05-23 01:15:29 <01115> cur_bank=(nil) curl_tl=0x28b2020 curl_tm=(nil)
1909: 2022-05-23 01:15:29 <01115> (session info)
1910: 2022-05-23 01:15:29 the killed daemon is /bin/wad: status=0xb00
Crash log interval is 3600 seconds
Method 2.
FPX # diag debug enable
# diagnose test application wad 2000 <----- Go to the WAD manager.
Set diagnosis process to default: WAD manager process pid=23843.
FPX # diagnose test application wad 99 <----- To restart the WAD process.
Always gracefully stopping wad manager......
FPX # diagnose test application wad 2000
Set diagnosis process to default: WAD manager process pid=23948 <----- New WAD manager generated.
To view all the existing wad process,
FPX # # diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=23948 diagnosis=yes.
Process [1]: type=worker(2) index=0 pid=23955 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [2]: type=algo(3) index=0 pid=23953 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [3]: type=informer(4) index=0 pid=23951 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [4]: type=user-info(5) index=0 pid=23954 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=debug(8) index=0 pid=23950 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [6]: type=config-notify(9) index=0 pid=23952 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
If the issue persists after restarting the processes, contact technical support for further assistance.
Related article:
Technical Tip: WAD Daemon Signal 11 Crash due to WebFilter Logging
|
