Description
This article describes possible troubleshooting if issues arise when adding a FortiGate to an existing Security Fabric. Useful information about the Security Fabric can be found there Fortinet Security Fabric v6.0.6 and Fortinet Security Fabric v6.2.0
Scope
FortiGate.
Solution
The Fortinet Security Fabric is a feature that provides visibility on connected Fortinet devices, especially FortiGates, in a single root FortiGate. Sometimes issues can arise when a FortiGate is added to an existing Security Fabric, impeding visibility and communication between the Fabric nodes.
Errors will mainly be displayed in the Security Fabric section in the FortiGate GUI. Error messages regarding FortiView and/or FortiAnalyzer usually indicate an issue on that FortiGate communicating with the Fabric FortiAnalyzer or some issues with logs, but not a connectivity issue between two FortiGates.
If an issue arises, the following troubleshooting can be done:
In CLI, collect this output:
diagnose debug reset
diagnose debug app csfd -1
diagnose debug enable
Once the output is collected, to disable the debugging:
diagnose debug disable
Observe what error messages show up in the CLI. A common error can be that CAs (Certificate Authorities) are missing. This can lead to errors like the following:
<2761> 02 __ssl_recv()-596: ssl error: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
This indicates that one FortiGate does not trust the certificates used by the other FortiGate; to fix this, download the CAs on each FortiGate and import them to the other.
Also, restart the CSF daemon after this has occurred.
To restart the CSF daemon:
- Find the daemon process ID (PID):
diagnose sys process pidof csfd <----- In v6.0 and higher.
fnsysctl cat /var/run/csf.pid <----- In v5.6.
- Kill the daemon (it will restart automatically):
diagnose sys kill 11 <process ID>
Restarting the csf daemon in general can resolve some issues as well.
General troubleshooting should also be done:
- Verify the affected FortiGates can reach each other (ping, https, ssh).
- Check the crashlog on each FortiGate for crashes with these processes: csf, miglogd.
diagnose debug crashlog read
Check the release notes for the firmware versions of the devices for possible known issues regarding Security Fabric.
Note:
Make sure there is no compatibility issue by verifying that the FortiGates are in a similar firmware version if possible, and that any FortiAnalyzer (and FortiManager, if exist) are in a compatible firmware version. FortiAnalyzer/FortiManager has to be at least the same branch as the highest FortiGate.
Compatibility matrices can be found here: Fortinet Document Library in the FortiManager or FortiAnalyzer section.
- It is also possible to check the following debug commands in the root FortiGate if it is not possible to authorize any device:
diagnose debug reset
diagnose debug cli 8
diagnose debug enable
diagnose sys csf authorization accept <serial number of device>
- If it gives a duplicate error <duplicated action=find-dup>, check whether any existing fabric device has an index value of 0 or not under the trusted list. If any device has an index value of 0, then we can change it to a non-zero value to resolve the issue.
- Following is the reference output:
conf sys csf
config trusted-list
show
edit "FG101E4Q17-----9" <----- Index value is 3.
set serial "FG101E4Q170----9"
set ha-members "FG101E4Q170----8"
set index 3
edit "FG6H1ETB219----0" <----- Index value is 0.
set serial "FG6H1ETB219----0"
set ha-members "FG6H1ETB219----0"
next
Note:
TCP MSS issues may affect Fabric communications, especially if a FortiGate has an MTU-limiting device in the path. The sniffer shows differing MSS in SYN/SYN-ACK. This can be fixed by adding 'set tcp-mss 1300' on the involved interface to re-establish the connectivity.
Another cause of the Security Fabric issue is that the root FortiGate is dropping the connection from the downstream FortiGate. The following debug flow on the root FortiGate shows the connection getting dropped. No trusthost or local-in-policy is configured.
# id=65308 trace_id=1 func=print_pkt_detail line=6005 msg="vd-root:0 received a packet(proto=6, 10.201.3.49:13688->10.201.4.152:8014) tun_id=0.0.0.0 from port10. flag [S], seq 4200947104, ack 0, win 29200"
id=65308 trace_id=1 func=init_ip_session_common line=6206 msg="allocate a new session-00000333"
id=65308 trace_id=1 func=__vf_ip_route_input_rcu line=1989 msg="find a route: flag=80000000 gw-0.0.0.0 via root"
id=65308 trace_id=1 func=fw_local_in_handler line=620 msg="iprope_in_check() check failed on policy 0, drop"
The issue is due to the port for the Security Fabric not matching.
Below is the downstream FortiGate configuration with port 8014. The default port is 8013.
downstream # show system csf
config system csf
set status enable
set uid "e2896ff361ecc38ec3ec6fd7489af4ec"
set upstream "10.201.4.152"
set upstream-port 8014
end
To fix this, change the upstream-port to default on the downstream FortiGate or change the fortiservice-port on the root FortiGate to match the port.
root # show sys global
config system global
set fortiservice-port 8014
end
