FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
msingh_FTNT
Staff & Editor
Staff & Editor
Article Id 230103
Description This article describes helpful debug commands to use for troubleshooting security fabric issues.
Scope FortiGate v6.4, v7.0, and higher.
Solution

The Security Fabric establishes an intelligent architecture for seamless communication between security devices. This integrated framework facilitates the detection, monitoring, and remediation of threats across the entire attack surface. Support for hardware, virtual, and cloud-based environments ensures comprehensive protection and visibility throughout the network. The Security Fabric solution includes two or more FortiGate firewalls in NAT mode and a logging system. The logging system must be a FortiAnalyzer, FortiAnalyzer Cloud, or FortiGate Cloud. All FortiGates must operate in NAT mode to participate in Security Fabric.

 

Sniffer commands to troubleshoot communication issues between upstream and downstream FortiGate in the Security Fabric:

 

   diagnose sniffer packet any "tcp port 8013 or udp port 8014" 6 0 a

 

The default ports are TCP/8013 and UDP/8014. The upstream port TCP/8013 can be customized by

 

config system csf

    set upstream-port <port number>

end

 

Debug commands to find any issues related to Security Fabric /Security Fabric performance issues.

 

diagnose sys csf downstream

 

Sample output:

 

diagnose sys csf downstream

 

1: FGVM0DOWNSTREAM (10.10.10.3) Management-IP: (null) Management-port:0 parent: (null)
path:(null)
data received: N downstream intf: upstream intf:tunn1 upstream vdom:root
admin-port:0 authorizer:FGVM0UPSTREAM

 

diagnose test app csf 1

 

Another option for troubleshooting the connection is to verify the authorization list on the fabric root appliance (accept it if one is pending):

 

diagnose sys csf authorization pending-list

 

Sample output:

 

diagnose sys csf authorization pending-list

 

Serial IP Address HA-Members Appliance Path
---------------------------------------------------------------------
FGVM0DOWNSTREAM 10.5.135.32 fortigate FGVM0UPSTREAM:FGVM0DOWNSTREAM

 

diagnose sys csf authorization accept [SN_of_pending_FGT]

 

In specific cases can also be useful:

 

diagnose sys csf upstream

 

Sample output:

 

diagnose sys csf upstream

 

Upstream Information:
Serial Number:FGVM0UPSTREAM
IP:10.10.10.1
Connecting interface:tunn1
Connection status:Authorized

 

diagnose sys csf global

 

Run these before opening the GUI, then examine the parts that are lagging in the GUI.

 

diagnose debug disable

diagnose debug reset

diagnose debug console timestamp enable

diagnose debug enable

diagnose debug application csfd -1

 

Open the GUI and replicate the issue, and then stop the capture with the command below:

 

diagnose debug disable

 

Commands to identify a high CPU issue caused by the csfd daemon:

get sys performance status
diagnose sys top 2 50
<----- Run it for 15 seconds and press q to quit.
diagnose sys mpstat
diagnose hard sysinfo interrupt
<----- Run multiple times, every 10 minutes.

diagnose debug console timestamp enable

diagnose debug application csfd -1

diagnose debug enable

 

Run the following commands five times each when csfd is busy or 'csfd debug output' stops working.

 

Find the CSF process ID by issuing: 

 

diagnose sys process pidof csfd

 

Then use the process ID for the following commands: 

 

diagnose sys process dump <csfd pid>

diagnose sys process pstack <csfd pid>

diagnose sys process trace <csfd pid> 5


Commands to identify a High Memory issue caused by the daemon csfd:

diagnose test app csfd 1 -> Show stats.

diagnose test app csfd 4 -> Start diagnostic stat collection.

diagnose test app csfd 7 -> Print collected diagnostics stats. 

diagnose test app csfd 10 -> Dump daemon info.

diagnose test app csfd 15 -> Show query cache status.

diagnose test app csfd 45 -> Show worker process information.

diagnose test app csfd 51

diagnose test app csfd 52

diagnose test app csfd 60 -> Show MAC cache status.

diagnose test app csfd 110

diagnose test app csfd 122

diagnose test app csfd 123

diagnose test app csfd 124

diagnose test app csfd 255 -> Dump Table Counts.

 

Note:
Make sure that the upstream device IP is not part of the configuration of the downstream device (IP address conflict).


Example:

Using the upstream device IP 172.16.24.1 in any interface of the downstream device. In such a case, the Fabric Status will be 'Not Connected'.