Troubleshooting Tip: FortiGate WAD crash (Signal 11) at wad_http_sstrm_alloc_msg

Description

This article describes an issue where the WAD process crashes on a FortiGate device with signal 11 (Segmentation fault) at wad_http_sstrm_alloc_msg

Scope

FortiGate FortiOS v7.4.9, v7.4.10(interim), v7.4.11 and v7.6.1 B3426.

Solution

The fix introduces a validation check to ensure that the corresponding request exists before associating it with the response. This prevents the WAD process from encountering a null reference and crashing when a second response is received without a matching request.

Crashlog example:

27: 2024-09-10 15:08:44 <00376> firmware FortiGate-101F v7.6.1,build3426b3426,240909 (interim)
28: 2024-09-10 15:08:44 (Release)
29: 2024-09-10 15:08:44 <00376> application wad
30: 2024-09-10 15:08:44 <00376> *** signal 11 (Segmentation fault) received ***
31: 2024-09-10 15:08:44 <00376> AVDB 92.16925(08/29/0024 14:28)
32: 2024-09-10 15:08:44 <00376> ETDB 92.16925(08/29/0024 14:27)
33: 2024-09-10 15:08:44 <00376> AVSO 04000000AVEN00601-00007.00031-2407120401
34: 2024-09-10 15:08:44 <00376> Register dump:
35: 2024-09-10 15:08:44 <00376> R0: 0000000000000000 R1: 000000555d15e5f8 R2: 0000007fb18798d0
36: 2024-09-10 15:08:44 <00376> R3: 0000007fb1879850 R4: 0000000000000001 R5: 0000000000000004
37: 2024-09-10 15:08:44 <00376> R6: 0000000000000000 R7: 0000007fb15dca28 XR: 0000005557696bf0
38: 2024-09-10 15:08:44 <00376> R9: 0000000000000000 R10: ec43f01a1ea0920b R11: 8b3f2677a01d3d5e
39: 2024-09-10 15:08:44 <00376> R12: 0000000000000000 R13: 8b3f2677a01d3d5e R14: 0000000000000000
40: 2024-09-10 15:08:44 <00376> R15: 0000000000000000 IP0: 000000555a542db0 IP1: 0000007fb6d12c00
41: 2024-09-10 15:08:44 <00376> PR: 0000000000000047 R19: 0000007fb1879780 R20: 0000007fa9d81650
42: 2024-09-10 15:08:44 <00376> R21: 0000007fb1be1010 R22: 0000007fb1be0d68 R23: 0000007fb1be0da0
43: 2024-09-10 15:08:44 <00376> R24: 0000007fb1be0d20 R25: 0000000000000000 R26: 0000002200000003
44: 2024-09-10 15:08:44 <00376> R27: 0000007f987141b8 R28: 0000000000000000 FP: 0000007fc60293c0
45: 2024-09-10 15:08:44 <00376> fault_address: 0000000000000098 sp: 0000007fc60293c0
46: 2024-09-10 15:08:44 <00376> pc: 0000005557432b78 lr: 0000005557432b6c
47: 2024-09-10 15:08:44 <00376> pstate: 80000000 (Nzcv daif -PAN -UAO)

48: 2024-09-10 15:08:44 <00376> Backtrace:
49: 2024-09-10 15:08:44 <00376> [0x01eddb78] => /bin/wad
50: 2024-09-10 15:08:44 <00376> [0x01e6c344] => /bin/wad
51: 2024-09-10 15:08:44 <00376> [0x01e6cc80] => /bin/wad
52: 2024-09-10 15:08:44 <00376> [0x02145e8c] => /bin/wad
53: 2024-09-10 15:08:44 <00376> [0x02e66844] => /bin/wad
54: 2024-09-10 15:08:44 <00376> [0x02e682f8] => /bin/wad
55: 2024-09-10 15:08:44 <00376> [0x02e498fc] => /bin/wad
56: 2024-09-10 15:08:44 <00376> [0x02143574] => /bin/wad
57: 2024-09-10 15:08:44 <00376> [0x0214426c] => /bin/wad
58: 2024-09-10 15:08:44 <00376> [0x0218d094] => /bin/wad
59: 2024-09-10 15:08:44 <00376> [0x020dd694] => /bin/wad
60: 2024-09-10 15:08:44 <00376> [0x020dda4c] => /bin/wad
61: 2024-09-10 15:08:44 <00376> [0x021b4f5c] => /bin/wad
62: 2024-09-10 15:08:44 <00376> [0x003dfdf8] => /bin/wad
63: 2024-09-10 15:08:44 <00376> [0x00020f24] => /usr/lib/aarch64-linux-gnu.6
64: 2024-09-10 15:08:44 (__libc_start_main)
65: 2024-09-10 15:08:44 <00376> [0x003db7a8] => /bin/wad
66: 2024-09-10 15:08:44 <00376> [0x003db7a8] => /bin/wad
67: 2024-09-10 15:08:44 <00376> fortidev 6.0.2.0008
68: 2024-09-10 15:08:44 <00376> process=wad type=2 idx=0 av-scanning=no total=3614 free=675
69: 2024-09-10 15:08:44 mmu=196906518 mu=105243995 m=35037737 f=34844399 r=0
70: 2024-09-10 15:08:44 <00376> cur_bank=(nil) curl_tl=0x555d1e2430 curl_tm=0x7fb2f69508
71: 2024-09-10 15:08:44 <00376> (session info) vf=1 session-id=323496708 app_type=1 dyn_type=0
72: 2024-09-10 15:08:44 non_tp=1, pol_id=1, h2=0, src/port=192.12.13.103:44716, dst/port=23.83.76.38:44
73: 2024-09-10 15:08:44 3, usr/grp=(/)
74: 2024-09-10 15:08:44 [AV Engine <376>] AV Engine version: 7.0.31
75: 2024-09-10 15:08:44 [AV Engine <376>] Last file info:
76: 2024-09-10 15:08:44 [AV Engine <376>] filename: file-sample_1MB.doc, filesize: 16384, filebuffer:
77: 2024-09-10 15:08:44 0x555d0ff510
78: 2024-09-10 15:08:44 [AV Engine <376>] Native script imagebase: 0x7fb33d0000
79: 2024-09-10 15:08:44 [AV Engine <376>] Native script imagesize: 0xa000
80: 2024-09-10 15:08:44 [AV Engine <376>] AV Engine imagebase: 0x7fb2643000

In the example above, the WAD process crashed due to signal 11 (Segmentation fault).

This issue is triggered under traffic load conditions where multiple policies are applied, and the upstream server sends a second response without a corresponding request.

There is no available workaround for this issue. It is recommended to roll back to a previous stable FortiGate version until a fix is applied.

The issue is resolved in FortiOS v7.4.12:2890 and FortiOS v7.6.1:3428.