Technical Tip: How to restart the WAD process

Rosalyn · ‎05-23-2022

Description

This article describes how to restart the WAD process.

Scope

In the event that wad processes hang or WAD taking up lots of memory, it is possible to restart WAD process to resolve it.

If the issue persists after restarting the processes, contact the technical support for further assistance.

Solution

Method 1:

To check the WAD parent process with command diag sys top-summary:

FPX # diag sys top-summary
CPU [||||||||||||||||||||||||||||||||||||||||] 100.0%
Mem [||||||||||||||||||||||||||||| ] 73.7% total (3.4% reclaimable)

Processes: 20 (running=1 sleeping=123)

PID RSS ^CPU% MEM% FDS TIME+ NAME
* 1089 36M 2.9 1.8 12 00:01.22 sshd [x4]
1115 56M 1.0 2.8 136 32:41.10 wad [x7]
1287 58M 0.0 2.9 13 00:08.15 pyfcgid [x4]
1046 51M 0.0 2.6 10 06:30.74 cmdbsvr

Restart the process with command # 'diag sys kill 11 <pid>' or using 'fnsysctl killall wad'

FPX # diag sys kill 11 1115

or

FPX # fnsysctl killall wad

To verify and find the FPX created new pid value for WAD parent process.

FPX # diag sys top-summary
CPU [||||||||||||||||||||||||||||||||||||||||] 100.0%
Mem [|||||||||||||||||||||||||||| ] 71.8% total (3.4% reclaimable)

Processes: 20 (running=4 sleeping=118)

PID RSS ^CPU% MEM% FDS TIME+ NAME
* 23682 49M 0.0 2.5 12 00:00.42 pyfcgid [x4]
1046 51M 0.0 2.6 10 06:30.77 cmdbsvr
1182 143M 0.0 7.2 32 06:28.71 scanunitd [x3]
23843 35M 0.0 1.8 65 00:03.25 wad [x7]
1087 55M 0.0 2.8 18 03:42.72 httpsd [x5]

FPX crashlog generates a wad signal 11 log
FPX # diag debug crashlog read
1876: 2022-05-23 01:15:28 <01115> *** signal 11 (Segmentation fault) received ***
1877: 2022-05-23 01:15:28 <01115> Register dump:
1878: 2022-05-23 01:15:28 <01115> RAX: fffffffffffffffc RBX: 0000000000000004
1879: 2022-05-23 01:15:28 <01115> RCX: 00007ff8874eadc0 RDX: 0000000000000006
1880: 2022-05-23 01:15:28 <01115> R8: 0000000000000000 R9: 0000000000000008
1881: 2022-05-23 01:15:28 <01115> R10: 0000000000001388 R11: 0000000000000246
1882: 2022-05-23 01:15:28 <01115> R12: 0000000000000018 R13: 0000000000000000
1883: 2022-05-23 01:15:28 <01115> R14: 0000000000000000 R15: 0000000000000000
1884: 2022-05-23 01:15:28 <01115> RSI: 0000000003d66be0 RDI: 0000000000000005
1885: 2022-05-23 01:15:28 <01115> RBP: 00007ffd8fd815e0 RSP: 00007ffd8fd815b8
1886: 2022-05-23 01:15:28 <01115> RIP: 00007ff8874eadc0 EFLAGS: 0000000000000246
1887: 2022-05-23 01:15:28 <01115> CS: 0033 FS: 0000 GS: 0000
1888: 2022-05-23 01:15:28 <01115> Trap: 0000000000000000 Error: 0000000000000000
1889: 2022-05-23 01:15:28 <01115> OldMask: 0000000000000000
1890: 2022-05-23 01:15:28 <01115> CR2: 0000000000000000
1891: 2022-05-23 01:15:28 <01115> stack: 0x7ffd8fd815b8 - 0x7ffd8fd822d0
1892: 2022-05-23 01:15:28 <01115> Backtrace:
1893: 2022-05-23 01:15:28 <01115> [0x7ff8874eadc0] => /fortidev/lib/x86_64-linux-gnu/libc.so.6
1894: 2022-05-23 01:15:28 (epoll_pwait+0x00000020) liboffset 000f4dc0
1895: 2022-05-23 01:15:28 <01115> [0x00ec0202] => /bin/wad
1896: 2022-05-23 01:15:28 <01115> [0x00f1e204] => /bin/wad
1897: 2022-05-23 01:15:28 <01115> [0x0042ec84] => /bin/wad
1898: 2022-05-23 01:15:28 <01115> [0x00434ebf] => /bin/wad
1899: 2022-05-23 01:15:28 <01115> [0x00432128] => /bin/wad
1900: 2022-05-23 01:15:28 <01115> [0x00432518] => /bin/wad
1901: 2022-05-23 01:15:28 <01115> [0x004342d4] => /bin/wad
1902: 2022-05-23 01:15:28 <01115> [0x00434ad5] => /bin/wad
1903: 2022-05-23 01:15:28 <01115> [0x7ff887416eaa] => /fortidev/lib/x86_64-linux-gnu/libc.so.6
1904: 2022-05-23 01:15:28 (__libc_start_main+0x000000ea) liboffset 00020eaa
1905: 2022-05-23 01:15:28 <01115> [0x0042b5ca] => /bin/wad
1906: 2022-05-23 01:15:29 <01115> process=wad type=0 idx=-1 av-scanning=no total=2006 free=626 mmu=1176
1907: 2022-05-23 01:15:29 mu=616 m=28 f=20 r=0
1908: 2022-05-23 01:15:29 <01115> cur_bank=(nil) curl_tl=0x28b2020 curl_tm=(nil)
1909: 2022-05-23 01:15:29 <01115> (session info)
1910: 2022-05-23 01:15:29 the killed daemon is /bin/wad: status=0xb00
Crash log interval is 3600 seconds

Method 2.

FPX # diag debug enable
# diagnose test application wad 2000 <----- Go to the WAD manager.
Set diagnosis process to default: WAD manager process pid=23843.

FPX # diagnose test application wad 99 <----- To restart the WAD process.
Always gracefully stopping wad manager......

FPX # diagnose test application wad 2000
Set diagnosis process to default: WAD manager process pid=23948 <----- New WAD manager generated.

To view all the existing wad process,
FPX # # diagnose test application wad 1000
Process [0]: WAD manager type=manager(0) pid=23948 diagnosis=yes.
Process [1]: type=worker(2) index=0 pid=23955 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [2]: type=algo(3) index=0 pid=23953 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [3]: type=informer(4) index=0 pid=23951 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [4]: type=user-info(5) index=0 pid=23954 state=running
diagnosis=no debug=enable valgrind=supported/disabled
Process [5]: type=debug(8) index=0 pid=23950 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled
Process [6]: type=config-notify(9) index=0 pid=23952 state=running
diagnosis=no debug=enable valgrind=unsupported/disabled

Related articles:

Technical Tip: WAD Daemon Signal 11 Crash due to WebFilter Logging