|
After upgrading FortiGate to v7.4.x or v7.6.x, the Security Fabric connections to downstream FortiGates fail.
On the upstream FortiGate, downstream FortiGate is not detected.
FGT-Root# diagnose sys csf downstream
FGT-Root#
[Output is empty]
On the downstream FortiGate, status is stuck at ‘Connecting.’
FGT-Azure-VM # diagnose sys csf upstream
Upstream Information:
Serial Number:
IP: 10.40.50.68
Connecting interface: port1
Connection status: Connecting
The crashlog shows the following:
diagnose debug crashlog read
1: <05104> firmware FortiGate-201G v7.4.8,build2776b2776,250422 (interim)
2: (Release)
3: <05104> application csfd-unpriv
4: <05104> *** signal 31 (Bad system call) received ***
5: <05104> Register dump:
6: <05104> RAX: 0000000000000007 RBX: 000000000a9118e0
7: <05104> RCX: 00007f7bbcac7a53 RDX: 00000000ffffffff
8: <05104> R08: 0000000000000002 R09: 00007f7bb7b168a0
9: <05104> R10: 00007f7bb7b16131 R11: 0000000000000246
10: <05104> R12: 000000000a9118e0 R13: 000000000a912fd0
11: <05104> R14: 0000000000000000 R15: 00000000ffffffff
12: <05104> RSI: 0000000000000001 RDI: 00007ffe450e2274
13: <05104> RBP: 00007ffe450e22d0 RSP: 00007ffe450e2248
14: <05104> RIP: 00007f7bbcac7a53 EFLAGS: 0000000000000246
15: <05104> CS: 0033 FS: 0000 GS: 0000
16: <05104> Trap: 0000000000000000 Error: 0000000000000000
17: <05104> OldMask: 0000000000000000
18: <05104> CR2: 0000000000000000
19: <05104> Backtrace:
20: <05104> [0x7f7bbcaa0003] => /lib/libc.so.6 (nanosleep+0013) liboffset 000c7003 => ?? ??:0
21: <05104> [0x7f7bbca9ff3a] => /lib/libc.so.6 (sleep+003a) liboffset 000c6f3a => ?? ??:0
22: <05104> [0x009025fe] => /bin/csfd => nstd_daemon_unpriv_crash_sigaction at /code/daemon/nstd/nstd_daemon_unpriv.c:76
23: <05104> [0x7f7bbca11ec0] => /lib/libc.so.6 (killpg+0040) liboffset 00038ec0 => ?? ??:0
24: <05104> [0x7f7bbcac7a53] => /lib/libc.so.6 (__poll+0013) liboffset 000eea53 => ?? ??:0
25: <05104> [0x7f7bb7b129b2] => /lib/libtss2-tcti-device.so.0 liboffset 000029b2 => ?? ??:0
26: <05104> [0x7f7bb9a7c87a] => /lib/libtss2-sys.so.1 (Tss2_Sys_ExecuteFinish+00ca) => ?? ??:0
27: liboffset 0000f87a
28: <05104> [0x7f7bba0f30af] => /lib/libtss2-esys.so.0 (Esys_Hash_Finish+00bf) => ?? ??:0
29: liboffset 0002f0af
30: <05104> [0x7f7bba0f35f3] => /lib/libtss2-esys.so.0 (Esys_Hash+0053) liboffset => ?? ??:0
31: 0002f5f3
32: <05104> [0x7f7bb7b49c91] => /lib/ossl-modules/tpm2.so (OSSL_provider_init+6e21) => ?? ??:0
33: liboffset 0000ec91
34: <05104> [0x7f7bb7b4a99f] => /lib/ossl-modules/tpm2.so (OSSL_provider_init+7b2f) => ?? ??:0
35: liboffset 0000f99f
36: <05104> [0x7f7bbc394091] => /lib/libssl.so.3 (SSL_in_before+b501) liboffset => ?? ??:0
37: 000ba091
38: <05104> [0x7f7bbc3891b9] => /lib/libssl.so.3 (SSL_in_before+0629) liboffset => ?? ??:0
39: 000af1b9
40: <05104> [0x00932cec] => /bin/csfd => nstd_downstream_ssl_accept at /code/daemon/nstd/plugin/nstd_conn_unpriv.c:1065 (discriminator 17)
41: <05104> [0x02f34896] => /bin/csfd => fos_epoll_work_with_prepare at /code/migbase/osapi/fos_epoll.c:107
42: <05104> [0x00910b53] => /bin/csfd => nstd_unpriv_main at /code/daemon/nstd/nstd_main_unpriv.c:161
43: <05104> [0x00451b97] => /bin/csfd => fortiexec_call_main at /code/sysinit/fortiexec.c:796
44: <05104> [0x7f7bbc9fce1b] => /lib/libc.so.6 (__libc_start_main+00eb) liboffset => ?? ??:0
45: 00023e1b
46: <05104> [0x0044d66a] => /bin/csfd => _start at /build/glibc/glibc-2.30/csu/../sysdeps/x86_64/start.S:122
47: <05104> fortidev 6.0.2.0008
Sniffer shows root and/or downstream FortiGate are sending RST packets.
diag sniffer packet any 'tcp port 8013' 4 0 l
Using Original Sniffing Mode
interfaces=[any]
filters=[tcp port 8013 or udp port 8014]
2025-07-21 12:08:06.823310 port1 out 10.20.30.1.5195 -> 10.40.50.68.8013: syn 1505897159
2025-07-21 12:08:06.839180 port1 in 10.40.50.68.8013 -> 10.20.30.1.5195: syn 2355787558 ack 1505897160
2025-07-21 12:08:06.839231 port1 out 10.20.30.1.5195 -> 10.40.50.68.8013: ack 2355787559
2025-07-21 12:08:08.856954 port1 in 10.40.50.68.8013 -> 10.20.30.1.5195: fin 2355787559 ack 1505897160
2025-07-21 12:08:08.860030 port1 out 10.20.30.1.5195 -> 10.40.50.68.8013: ack 2355787560
2025-07-21 12:08:14.888067 port1 out 10.20.30.1.5195 -> 10.40.50.68.8013: psh 1505897160 ack 2355787560
2025-07-21 12:08:14.903740 port1 in 10.40.50.68.8013 -> 10.20.30.1.5195: rst 2355787560
The following logs are seen in the csfd debugs:
diagnose debug application csfd -1
diagnose debug console timestamp enable
diagnose debug enable
<5075-M> 10 daemon_chan_data_cb()-73: Received internal msg NSTD_INTERNAL_MSG_UNPRIV_DYING data_len=16
<5075-M> 04 daemon_recv_internal_msg_unpriv_dying()-111:
<5075-M> 02 daemon_recv_internal_msg_unpriv_dying()-121: Unpriv dying (sig=31), attached to it to collect backtrace
<5075-M> 04 reap_killed_children()-316:
<5075-M> 04 daemon_sigchld_stopped_hd()-275:
<5075-M> 02 daemon_sigchld_stopped_hd()-277: Unpriv stopped, collecting backtrace
<5075-M> 2000000 nstd_task_runner_handle_sigchld()-732:
<5075-M> 04 generic_event_logging_plugin()-846:
<5075-M> 04 generic_event_ha_sync_plugin()-223:
<5075-M> 800 generic_event_auth_check()-1822:
<5075-M> 40000 nstd_sync_generic_event()-1069:
<5075-M> 40000 handle_generic_event_global_obj()-1108:
<5075-M> 100 nstd_tree_updater_generic_event_handler()-927:
<5075-M> 100 nstd_tree_generic_poll_data_updater_hd()-554:
<5075-M> 04 nstd_chan_data_ep_hd()-194:
<5075-M> 02 nstd_chan_data_ep_hd()-197: chan epoll error events=17
<5075-M> 02 daemon_chan_err_cb()-150: Restart unpriv due to chan error.
<5075-M> 04 nstd_daemon_stop_unpriv()-60:
<5075-M> 02 nstd_daemon_stop_unpriv()-65: Sending SIGTERM to unpriv
<5075-M> 04 reap_killed_children()-316:
<5075-M> 04 daemon_reap_unpriv()-83:
<5075-M> 02 daemon_reap_unpriv()-85: unpriv dead, schedule to recreate unpriv in 10 seconds
<5075-M> 04 conn_destruct_all()-708:
<5075-M> 02 daemon_send_internal_msg()-365: Failed to send internal msg NSTD_INTERNAL_MSG_DISCONNECT_FGT, unpriv inactive
<5075-M> 10000000 __fgt_destruct()-258: closing downstream 10.20.30.1.5195
This issue has been resolved in the upcoming v7.4.8, v7.6.4, v8.0.0.
If the connecting device is FortiGate Azure VM, refer to this article for the fix release: Technical Tip: Security Fabric Connection Failure on FortiGate Azure VM and FortiGate 120G/121G Afte...
The following information is required by FortiGate TAC for investigation:
Debugs:
diagnose debug console timestamp enable
diagnose debug application csfd -1
diagnose debug enable
<reproduce the issue>
diagnose debug reset
Packet capture:
diagnose sniffer packet any "port 8013" 4 0 l
Configuration file of FortiGate.
|
