|
If WAD processes hang or WAD takes up lots of memory, it is possible to restart the WAD process to resolve it.
Refer to below steps for FortiGate or FortiProxy devices :
Method 1.
Step 1: Run the CLI command 'get system performance status', the output will look similar to the sample below:
get system performance status
CPU states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU0 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
Memory: 2004540k total, 586528k used (29%), 1418012k free (71%)
Average network usage: 1 / 0 kbps in 1 minute, 0 / 0 kbps in 10 minutes, 0 / 0 kbps in 30 minutes
Average sessions: 25 sessions in 1 minute, 25 sessions in 10 minutes, 25 sessions in 30 minutes
Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 0 days, 23 hours, 41 minutes
Run the command above a few times and compare patterns of memory usage, throughput, and number of sessions.
Step 2: To check the WAD parent process with the command diag sys top-summary -------->The top-summary command was deprecated on FOS 6.4x and onwards
FPX # diag sys top-summary
CPU [||||||||||||||||||||||||||||||||||||||||] 100.0%
Mem [||||||||||||||||||||||||||||| ] 73.7% total (3.4% reclaimable)
Processes: 20 (running=1 sleeping=123)
PID RSS ^CPU% MEM% FDS TIME+ NAME
* 1089 36M 2.9 1.8 12 00:01.22 sshd [x4]
1115 56M 1.0 2.8 136 32:41.10 wad [x7]
1287 58M 0.0 2.9 13 00:08.15 pyfcgid [x4]
1046 51M 0.0 2.6 10 06:30.74 cmdbsvr
Use the options available to expand the list of processes, for example:
FPX # diagnose sys top 1 25 3
Where:
- '1' is the interval in seconds between each refresh. A low number for the refresh interval is useful for isolating the processes causing CPU spikes, but may not allow a large list of applications to be displayed.
- '25' is the number of lines being displayed during an iteration.
- '3' is the number of iterations this command should display.
Step 3: Restart the process with command # 'diag sys kill 11 <pid>' or using 'fnsysctl killall wad'
FPX # diag sys kill 11 1115
or
FPX # fnsysctl killall wad
Step 4: To verify and find the FPX created new pid value for the WAD parent process.
FPX # diag sys top-summary
CPU [||||||||||||||||||||||||||||||||||||||||] 100.0%
Mem [|||||||||||||||||||||||||||| ] 71.8% total (3.4% reclaimable)
Processes: 20 (running=4 sleeping=118)
PID RSS ^CPU% MEM% FDS TIME+ NAME
* 23682 49M 0.0 2.5 12 00:00.42 pyfcgid [x4]
1046 51M 0.0 2.6 10 06:30.77 cmdbsvr
1182 143M 0.0 7.2 32 06:28.71 scanunitd [x3]
23843 35M 0.0 1.8 65 00:03.25 wad [x7]
1087 55M 0.0 2.8 18 03:42.72 httpsd [x5]
FPX crashlog generates a wad signal 11 log
FPX # diag debug crashlog read
1876: 2022-05-23 01:15:28 <01115> *** signal 11 (Segmentation fault) received ***
1877: 2022-05-23 01:15:28 <01115> Register dump:
1878: 2022-05-23 01:15:28 <01115> RAX: fffffffffffffffc RBX: 0000000000000004
1879: 2022-05-23 01:15:28 <01115> RCX: 00007ff8874eadc0 RDX: 0000000000000006
1880: 2022-05-23 01:15:28 <01115> R8: 0000000000000000 R9: 0000000000000008
1881: 2022-05-23 01:15:28 <01115> R10: 0000000000001388 R11: 0000000000000246
1882: 2022-05-23 01:15:28 <01115> R12: 0000000000000018 R13: 0000000000000000
1883: 2022-05-23 01:15:28 <01115> R14: 0000000000000000 R15: 0000000000000000
1884: 2022-05-23 01:15:28 <01115> RSI: 0000000003d66be0 RDI: 0000000000000005
1885: 2022-05-23 01:15:28 <01115> RBP: 00007ffd8fd815e0 RSP: 00007ffd8fd815b8
1886: 2022-05-23 01:15:28 <01115> RIP: 00007ff8874eadc0 EFLAGS: 0000000000000246
1887: 2022-05-23 01:15:28 <01115> CS: 0033 FS: 0000 GS: 0000
1888: 2022-05-23 01:15:28 <01115> Trap: 0000000000000000 Error: 0000000000000000
1889: 2022-05-23 01:15:28 <01115> OldMask: 0000000000000000
1890: 2022-05-23 01:15:28 <01115> CR2: 0000000000000000
1891: 2022-05-23 01:15:28 <01115> stack: 0x7ffd8fd815b8 - 0x7ffd8fd822d0
1892: 2022-05-23 01:15:28 <01115> Backtrace:
1893: 2022-05-23 01:15:28 <01115> [0x7ff8874eadc0] => /fortidev/lib/x86_64-linux-gnu/libc.so.6
1894: 2022-05-23 01:15:28 (epoll_pwait+0x00000020) liboffset 000f4dc0
1895: 2022-05-23 01:15:28 <01115> [0x00ec0202] => /bin/wad
1896: 2022-05-23 01:15:28 <01115> [0x00f1e204] => /bin/wad
1897: 2022-05-23 01:15:28 <01115> [0x0042ec84] => /bin/wad
1898: 2022-05-23 01:15:28 <01115> [0x00434ebf] => /bin/wad
1899: 2022-05-23 01:15:28 <01115> [0x00432128] => /bin/wad
1900: 2022-05-23 01:15:28 <01115> [0x00432518] => /bin/wad
1901: 2022-05-23 01:15:28 <01115> [0x004342d4] => /bin/wad
1902: 2022-05-23 01:15:28 <01115> [0x00434ad5] => /bin/wad
1903: 2022-05-23 01:15:28 <01115> [0x7ff887416eaa] => /fortidev/lib/x86_64-linux-gnu/libc.so.6
1904: 2022-05-23 01:15:28 (__libc_start_main+0x000000ea) liboffset 00020eaa
1905: 2022-05-23 01:15:28 <01115> [0x0042b5ca] => /bin/wad
1906: 2022-05-23 01:15:29 <01115> process=wad type=0 idx=-1 av-scanning=no total=2006 free=626 mmu=1176
1907: 2022-05-23 01:15:29 mu=616 m=28 f=20 r=0
1908: 2022-05-23 01:15:29 <01115> cur_bank=(nil) curl_tl=0x28b2020 curl_tm=(nil)
1909: 2022-05-23 01:15:29 <01115> (session info)
1910: 2022-05-23 01:15:29 the killed daemon is /bin/wad: status=0xb00
Crash log interval is 3600 seconds
Method 2.
FPX # diag debug enable
# diagnose test application wad 2000 <----- Go to the WAD manager.
Set diagnosis process to default: WAD manager process pid=23843.
FPX # diagnose test application wad 99 <----- To restart the WAD process.
Always gracefully stopping wad manager......
FPX # diagnose test application wad 2000
Set diagnosis process to default: WAD manager process pid=23948 <----- New WAD manager generated.
To view all the existing wad process,
FPX # # diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=23948 diagnosis=yes.
Process [1]: type=worker(2) index=0 pid=23955 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [2]: type=algo(3) index=0 pid=23953 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [3]: type=informer(4) index=0 pid=23951 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [4]: type=user-info(5) index=0 pid=23954 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=debug(8) index=0 pid=23950 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [6]: type=config-notify(9) index=0 pid=23952 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Method 3:
This is usually the quickest way to restart all WAD processes.
Step 1: Verify the process IDs of the current WAD processes:
diagnose sys process pidof wad
Step 2: Restart all WAD processes.
diagnose test application wad 99
Step 3: Verify the process IDs of the current WAD processes again and confirm that they changed.
diagnose sys process pidof wad
If the issue persists after restarting the processes, contact technical support for further assistance.
Note:
The WAD process cannot be restarted under a specific VDOM, and restarting the WAD process is a temporary workaround.
Here are some impacts of restarting the WAD process:
- Temporary Disruption: Restarting the WAD process will temporarily disrupt WAF (Web Application Firewall) functionality while the process is being restarted. During this time, the WAF may not be actively inspecting and filtering web traffic.
- Memory Release: Restarting the WAD process can help release any excessive memory usage or memory leaks that may be causing performance issues.
- Configuration Retention: Restarting the WAD process typically does not affect the configuration settings of the WAF. The system should retain the configured policies and rules after the restart. A prior current backup is always recommended.
- Service Continuity: It is advisable to restart the WAD process during a maintenance window or during periods of low traffic to minimize the impact on service continuity.
Related article:
Technical Tip: WAD Daemon Signal 11 Crash due to WebFilter Logging
|
