|
After upgrading FortiGate to v7.6.1, v7.6.2, or v7.4.8, 'csfd-unpriv' daemon crashes are observed on FortiGate Azure VM and 120G/121G Series, leading to Security Fabric Connectivity issues.
diagnose debug crashlog read
10724: 2025-06-16 10:55:10 <12392> firmware FortiGate-VM64-AZURE v7.4.8,build2795b2795,250523 (GA.M)
10725: 2025-06-16 10:55:10 (Release)
10726: 2025-06-16 10:55:10 <12392> application csfd-unpriv
10727: 2025-06-16 10:55:10 <12392> *** signal 31 (Bad system call) received ***
10728: 2025-06-16 10:55:10 <12392> Register dump:
10729: 2025-06-16 10:55:10 <12392> RAX: 0000000000000060 RBX: 00007ffec330c7c0
10730: 2025-06-16 10:55:10 <12392> RCX: 00007ffec3374d79 RDX: 000000001a4a349b
10731: 2025-06-16 10:55:10 <12392> R08: 00005621cb75c330 R09: 0000000000000000
10732: 2025-06-16 10:55:10 <12392> R10: fffffffffffff4ed R11: 0000000000000246
10733: 2025-06-16 10:55:10 <12392> R12: 0000000000000000 R13: 00007ffec3371080
10734: 2025-06-16 10:55:10 <12392> R14: 00007ffec330c784 R15: 001a4a349b531285
10735: 2025-06-16 10:55:10 <12392> RSI: 0000000000000000 RDI: 00007ffec330c7c0
10736: 2025-06-16 10:55:10 <12392> RBP: 00007ffec330c7b0 RSP: 00007ffec330c780
10737: 2025-06-16 10:55:10 <12392> RIP: 00007ffec3374d79 EFLAGS: 0000000000000246
10738: 2025-06-16 10:55:10 <12392> CS: 0033 FS: 0000 GS: 0000
10739: 2025-06-16 10:55:10 <12392> Trap: 0000000000000000 Error: 0000000000000000
10740: 2025-06-16 10:55:10 <12392> OldMask: 0000000000000000
10741: 2025-06-16 10:55:10 <12392> CR2: 0000000000000000
10742: 2025-06-16 10:55:10 <12392> Backtrace:
10743: 2025-06-16 10:55:10 <12392> [0x7f0fb083ff23] => /lib/libc.so.6 {0x7f0fb0744000}
10744: 2025-06-16 10:55:10 <12392> [0x5621bbdb15e9] => /bin/csfd {0x5621bb5d3000}
10745: 2025-06-16 10:55:10 <12392> [0x5621bbdb42ca] => /bin/csfd {0x5621bb5d3000}
10746: 2025-06-16 10:55:10 <12392> [0x5621bbdb4a34] => /bin/csfd {0x5621bb5d3000}
10747: 2025-06-16 10:55:10 <12392> [0x7f0fb077cec0] => /lib/libc.so.6 {0x7f0fb0744000}
10748: 2025-06-16 10:55:10 <12392> [0x7ffec3374d79] => [vdso] {0x7ffec3374000}
10749: 2025-06-16 10:55:10 <12392> [0x7f0fb00b29ce] => /lib/libssl.so.3 {0x7f0fb0089000}
10750: 2025-06-16 10:55:10 <12392> [0x7f0fb00d5f01] => /lib/libssl.so.3 {0x7f0fb0089000}
10751: 2025-06-16 10:55:10 <12392> [0x7f0fb00d6292] => /lib/libssl.so.3 {0x7f0fb0089000}
10752: 2025-06-16 10:55:10 <12392> [0x7f0fb01390fa] => /lib/libssl.so.3 {0x7f0fb0089000}
10753: 2025-06-16 10:55:10 <12392> [0x7f0fb01381b9] => /lib/libssl.so.3 {0x7f0fb0089000}
10754: 2025-06-16 10:55:10 <12392> [0x5621bbde4e76] => /bin/csfd {0x5621bb5d3000}
10755: 2025-06-16 10:55:10 <12392> [0x5621bbde5730] => /bin/csfd {0x5621bb5d3000}
10756: 2025-06-16 10:55:10 <12392> [0x5621bbde5980] => /bin/csfd {0x5621bb5d3000}
10757: 2025-06-16 10:55:10 <12392> [0x5621be448246] => /bin/csfd {0x5621bb5d3000}
10758: 2025-06-16 10:55:10 <12392> [0x5621bbdc2a8b] => /bin/csfd {0x5621bb5d3000}
10759: 2025-06-16 10:55:10 <12392> [0x5621bb8c7413] => /bin/csfd {0x5621bb5d3000}
10760: 2025-06-16 10:55:10 <12392> [0x7f0fb0767e1b] => /lib/libc.so.6 {0x7f0fb0744000}
10761: 2025-06-16 10:55:10 <12392> [0x5621bb8c229a] => /bin/csfd {0x5621bb5d3000}
10762: 2025-06-16 10:55:10 <12392> fortidev 6.0.2.0008
10763: 2025-06-16 10:55:29 csfd-unpriv previously crashed 6 times. The last crash was at 2025-06-16
10764: 2025-06-16 10:55:29 10:55:10.
Sniffer shows root and/or downstream FortiGate are sending RST packets:
diagnose sniffer packet any "port 8013" 4 0 l
Using Original Sniffing Mode
interfaces=[any]
filters=[port 8013]
10.100.1.4.8013: syn 3087971223
2025-01-06 15:44:08.865667 port2 out 10.200.1.4.9129 -> 10.100.1.4.8013: syn 3087971223
2025-01-06 15:44:08.865672 sriovslv1 out 10.200.1.4.9129 -> 10.100.1.4.8013: syn 3087971223
2025-01-06 15:44:08.865672 sriovslv1 out 10.200.1.4.9129 -> 10.100.1.4.8013: syn 3087971223
2025-01-06 15:44:09.071266 port2 in 10.100.1.4.8013 -> 10.200.1.4.9129: syn 3045993988 ack 3087971224
2025-01-06 15:44:09.071266 port2 in 10.100.1.4.8013 -> 10.200.1.4.9129: syn 3045993988 ack 3087971224
2025-01-06 15:44:09.071320 port2 out 10.200.1.4.9129 -> 10.100.1.4.8013: rst 3087971224
2025-01-06 15:44:09.071320 port2 out 10.200.1.4.9129 -> 10.100.1.4.8013: rst 3087971224
2025-01-06 15:44:09.071324 sriovslv1 out 10.200.1.4.9129 -> 10.100.1.4.8013: rst 3087971224
2025-01-06 15:44:09.071324 sriovslv1 out 10.200.1.4.9129 -> 10.100.1.4.8013: rst 3087971224
The following logs are seen in the csfd debugs:
diagnose debug application csfd -1
diagnose debug enable
2025-06-19 12:04:05 <2266-M> 10 daemon_chan_data_cb()-73: Received internal msg NSTD_INTERNAL_MSG_UNPRIV_DYING data_len=16
2025-06-19 12:04:05 <2266-M> 04 daemon_recv_internal_msg_unpriv_dying()-111:
2025-06-19 12:04:05 <2266-M> 02 daemon_recv_internal_msg_unpriv_dying()-121: Unpriv dying (sig=31), attached to it to collect backtrace
2025-06-19 12:04:05 <2266-M> 04 reap_killed_children()-316:
2025-06-19 12:04:05 <2266-M> 04 daemon_sigchld_stopped_hd()-275:
2025-06-19 12:04:05 <2266-M> 02 daemon_sigchld_stopped_hd()-277: Unpriv stopped, collecting backtrace
2025-06-19 12:04:05 <2266-M> 2000000 nstd_task_runner_handle_sigchld()-732:
2025-06-19 12:04:05 <2266-M> 04 generic_event_logging_plugin()-846:
2025-06-19 12:04:05 <2266-M> 04 generic_event_ha_sync_plugin()-223:
2025-06-19 12:04:05 <2266-M> 800 generic_event_auth_check()-1822:
2025-06-19 12:04:05 <2266-M> 40000 nstd_sync_generic_event()-1069:
2025-06-19 12:04:05 <2266-M> 40000 handle_generic_event_global_obj()-1108:
2025-06-19 12:04:05 <2266-M> 100 nstd_tree_updater_generic_event_handler()-927:
2025-06-19 12:04:09 <2266-M> 100 nstd_tree_generic_poll_data_updater_hd()-554:
2025-06-19 12:04:10 <2266-M> 04 nstd_chan_data_ep_hd()-194:
2025-06-19 12:04:10 <2266-M> 02 nstd_chan_data_ep_hd()-197: chan epoll error events=25
2025-06-19 12:04:10 <2266-M> 02 daemon_chan_err_cb()-150: Restart unpriv due to chan error.
2025-06-19 12:04:10 <2266-M> 04 nstd_daemon_stop_unpriv()-60:
2025-06-19 12:04:10 <2266-M> 02 nstd_daemon_stop_unpriv()-65: Sending SIGTERM to unpriv
2025-06-19 12:04:10 <2266-M> 04 reap_killed_children()-316:
2025-06-19 12:04:10 <2266-M> 04 daemon_reap_unpriv()-83:
2025-06-19 12:04:10 <2266-M> 02 daemon_reap_unpriv()-85: unpriv dead, schedule to recreate unpriv in 10 seconds
2025-06-19 12:04:10 <2266-M> 04 conn_destruct_all()-714:
2025-06-19 12:04:10 <2266-M> 02 daemon_send_internal_msg()-365: Failed to send internal msg NSTD_INTERNAL_MSG_DISCONNECT_FGT, unpriv inactive
This issue has been resolved in:
- v7.4.9 (scheduled to be released in August 2025).
- v7.6.3 (available to download from the Fortinet support portal). These timelines for firmware release are estimates and may be subject to change.
General debug information required by FortiGate TAC for investigation:
Debugs:
diagnose debug console timestamp enable
diagnose debug application csfd -1
diagnose debug enable
<reproduce the issue>
diagnose debug disable
diagnose debug reset
diagnose sniffer packet any "port 8013" 4 0 l
TAC Report:
execute tac report
Configuration file of the FortiGate.
Related documents:
7.4.8-fortios-release-notes
Troubleshooting Tip: Security Fabric could not established after upgrading FortiGate VM-Azure to v7....
|
