Troubleshooting Tip: How to troubleshoot connectivity issues between FortiGate and FortiManager

rameshk_FTNT · ‎03-16-2020

Description

This article describes how the 'FGFM' protocol is used to communicate between FortiGate and FortiManager devices and guides troubleshooting protocol-related issues.

The FGFM protocol operates over SSL (Secure Sockets Layer) using TCP port 541 on IPv4.

Scope

FortiGate and FortiManager.

Solution

Overview of 'FGFM' protocol:
The 'FGFM' protocol implements a secure communication protocol with the following functions:

FortiGate reachability status (from FortiManager).
FortiManager reachability status (from FortiGate).
Configuration installation and retrieval.
Script push.
JSON monitoring via RTM.

Exceptions:
The following communications between FortiGate and FortiManager units are handled outside of the 'FGFM' protocol and are managed by the FortiGuard protocol:

FortiGuard package downloads (Antivirus, IPS, Virus Scan, etc.).
FortiGuard query (WebFilter, AS).
Firmware Downloads.The 'FGFM' protocol runs over SSL (Secure Sockets Layer) using TCP port 541 under IPv4. FortiManager 6.2 supports the use of IPv6.
Both FortiGate and FortiManager units have an 'FGFM' daemon running exclusively for FortiGate to FortiManager communication.
The FortiManager unit listens on TCP port 541 for an incoming session request. The FortiGate unit establishes an SSL session with the FortiManager.
Both units use TCP port 541 for sending and receiving messages.
The 'FGFM' daemon handles all FortiGate to FortiManager (and vice versa) authentication, keep-alive messages, and actions resulting from them (such as instructing another daemon on a FortiGate device to update its configuration or various database files).

Troubleshooting:

In order to troubleshoot the connectivity between FortiGate and FortiManager check the steps below:

Check if 'FMG-Access' is enabled under the FortiGate management port. Go to Network -> Interfaces -> Double-click the management port -> Administrative access and check 'FMG-Access' is enabled.

Check the SSL compatibility.

TLS/SSL version: The TLS/SSL versions should match on both FortiManager and FortiGate.

FortiManager:

config system global
set fgfm-ssl-protocol {follow-global-ssl-protocol | sslv3 | tlsv1.0 | tlsv1.1 | tlsv1.2 | tlsv1.3}

FortiGate:

config system global
set ssl-min-proto-version {SSLv3 | TLSv1 | TLSv1-1 | TLSv1-2 | TLSv1-3}

Check Encryption algorithm:

The encryption algorithm can be the same on both devices or higher on FortiGate.

FortiManager:

config system global
set enc-algorithm {custom | high | medium | low}

Notes:

default: High-strength algorithms and the medium-strength 128-bit key length algorithms below.
RC4-SHA, RC4-MD5, RC4-MD.
high: 128-bit and larger key length algorithms. DHE-RSA-AES256-SHA, AES256-SHA, EDH-RSA-DES-CBC3-SHA, DES-CBC3-SHA, DES-CBC3-MD5, DHE-RSA-AES128-SHA, AES128-SHA.
low: 64-bit or 56-bit key length algorithms without export restrictions. EDH-RSA-DES-CDBC-SHA, DES-CBC-SHA, DES-CBC-MD5.

FortiGate:

config system central-management
set enc-algorithm {default | high | low}

DNS configuration:

Additionally check if the DNS servers are reachable from FortiGate under Network -> DNS.

Run the command 'diagnose fdsm central-mgmt-status' in FortiGate.

This command provides the connectivity and registration status of the FortiGate with the FortiManager.

diagnose fdsm central-mgmt-status

Telnet to the FortiManager IP on port 541 to ensure reachability.

execute telnet <FMG-IP> 541

Ensure proper MTU size end-to-end from FortiGate to FortiManager by executing the following commands on the FortiGate CLI:

execute ping-options df-bit yes

execute ping-options data-size <1472> -> ICMP will add 8 bytes for the ICMP header.

execute ping-options source <FortiGate IP used in fmg-source-ip in central mgmt>

execute ping <FortiManager IP>

If there was packet loss, change the data size to 1470/1400/1350/1320/1312 and verify on which data size value had no packet loss. Then, if necessary, adjust the FortiManager side with the following CLI commands:

config system interface

edit <interface_name>

set MTU <new_value>

The port connected to FortiGate can be found under System Settings -> Network.

Note: The MTU is checked using the FortiGate ping-options, as explained above, because FortiManager does not provide this functionality. However, if an adjustment is required, it must be made on the FortiManager side, not in the FortiGate configuration.

This is because the FortiManager certificate is sent in DF-flagged packets, which may be dropped if the path MTU is below 1500 bytes, or fragmented if intermediate network equipment does not properly respect the DF flag. Lowering the MTU in the FortiManager configuration instructs it to split the FGFM certificate into smaller packets, allowing it to traverse MPLS, IPsec, or other tunneling solutions that reduce the effective payload size.

The source IP of the FortiGate can also be configured to use the respective IP of the FortiGate, which is reachable with the FortiManager. This is useful for cases such as VPN access.

config system central-management
set fmg-source-ip <FortiGate-IP>
end

The interface from which the FortiGate initiates the connection can also be specified.

config system central-management
set interface-select-method specify

set interface <interface_name>
end

If FortiGate loses the FGFM tunnel, it may be linked to FortiManager being behind a NAT and sending the wrong IP to FortiGate. To solve this, configure the management address (Configuring the management address) on a FortiManager that is behind a NAT device so the FortiGate can initiate a connection to the FortiManager. Configuring the management address setting in the CLI ensures that FortiManager knows the public IP and can configure it on the FortiGate.

config systems admin setting

set mgmt-fqdn <IP> <FQDN>

end

Note:

Use mgmt-fqdn even if only IP addresses are used and no FQDNs.

Collect debug log outputs in FortiGate:

diagnose debug reset
diagnose debug application fgfmd -1
diagnose debug console timestamp enable
diagnose debug enable

To disable the debugs:
diagnose debug disable

Collect debug log outputs in FortiManager:

diagnose debug reset
diagnose debug application fgfmsd -1 <Device-Name> <----- Filter (DeviceName) related messages. Use <DeviceName>="" to reset. Multiple device names should be separated by commas, e.g., Host1, Host2. Can be filtered by IP (x.x.x.x) as well.
diagnose debug timestamp enable
diagnose debug enable

Note: To use this filter, the device has to be registered in the device database. When troubleshooting registration issues, leave the filter field empty and, once collected, filter the log by device serial-number.

To generate the output in the debugs, re-initiate the connection from the FortiGate (or) from the FortiManager, by using either of the methods below:

Re-initiate the connection from the FortiGate CLI by restarting the 'FGFM' daemon in FortiGate

fnsysctl killall fgfmd

Refresh connectivity between FortiGate and FortiManager in Security Fabric -> Central Management in FortiGate.

Claim the tunnel from FortiManager CLI using the syntax below.

execute fgfm reclaim-dev-tunnel <device_name>

Run Sniffers on FortiGate.

diagnose sniffer packet any "port 541 and host x.x.x.x" 6 0 a <----- Where x.x.x.x is the FortiManager IP address.

At the same time, run a sniffer on FortiManager with the following syntax:

diagnose sniffer packet any "port 541 and host y.y.y.y" 3 0 a <----- Where y.y.y.y is the FortiGate IP address.

If the traffic from the remote FortiGate is received, but there is no reply from FortiManager, and the sniffer looks like the following:

FMG-VM64 # diagnose sniffer packet any " host 10.4.110.118 " 4
interfaces=[any]
filters=[ host 10.4.110.118 ]
3.407606 10.4.110.118.42754 -> 192.168.4.250.541: syn 3738660390
4.426946 10.4.110.118.42754 -> 192.168.4.250.541: syn 3738660390
6.511686 10.4.110.118.42754 -> 192.168.4.250.541: syn 3738660390

Then check for the local-in policy on FortiManager using the command below:

FMG-VM64 # show system local-in-policy
config system local-in-policy
edit 1
set action accept
set dport 541
set intf "port1"
set protocol TCP
set src 192.168.1.99 255.255.255.255
next
edit 2
set dport 541
set intf "port1"
set protocol TCP
set dport 541
set src 10.4.110.118 255.255.255.255
next
end

Scenarios are described below where connectivity between FortiGate and FortiManager is failing.

Scenario 1:

The FortiGate tries to register with the FortiManager Cloud, but if the 'fgfm-deny-unknown' setting is enabled, the FortiManager Cloud cannot accept the request from the FortiGate.

Disable the 'fgfm-deny-unknown' on the FortiManager Cloud and restart the FGFM tunnel on the FortiGate. Then, register for FortiManager again. After this, it will be possible to successfully authorize the FortiGate on the FortiManager Cloud.

On FortiManager:

config system global
set fgfm-deny-unknown disable

Related articles:

Scenario 2:

In some scenarios with VDOMs, the admin may see the 'Network is Unreachable' message in the fgfmd debug even when connectivity tests like ping or telnet pass. This results in the FortiGate failing to connect to the FortiManager.

diagnose debug application fgfmd -1
FGFMs: Create session 0xfcee7d0.
FGFMs: setting session 0xfcee7d0 exclusive=0
FGFMs: connect error: Network is unreachable

Verify that the VDOM configured on the interface (e.g., mgmt2) matches the VDOM for central management. If there is a mismatch, the debug output may display a 'Network is Unreachable' message.

config system interface

edit "mgmt2" <-----
set vdom "FW_EJEMPLO" <-----
set ip 10.80.15.1 255.255.255.0

set allowaccess ping https ssh snmp http fgfm

set type physical

set role lan

set snmp-index 2

config system central-management
set mode normal

set type fortimanager

set fmg "10.80.15.60"

set fmg-source-ip 0.0.0.0

set fmg-source-ip6 ::

set local-cert ''

unset ca-cert
set vdom "root" <----- Mismatch: should be FW_EJEMPLO.
  set fmg-update-port 8890
  set include-default-servers enable
  set enc-algorithm low
  set interface-select-method specify
set interface "mgmt2" <-----
end

Scenario 3:

Starting with FortiManager version 7.4.7 (Special Notices) and v7.6.3 (Special Notices), connections from VM-based devices to FortiManager are restricted by default for security reasons. As part of this change, FortiManager no longer permits VM platform connections over FGFM unless explicitly allowed. To allow VM platform connection in FGFM, enter the following command in the FortiManager CLI:

config system global

set fgfm-allow-vm enable

end

Scenario 4:

Ensure the custom certification is correct when connecting with FortiGate. FortiManager v7.2.5 introduces a new verification of the CN or SAN of a custom certificate uploaded by the FortiGate, where it should include the FortiGate Serial Number either in the CN or SAN.

The tunnel connection may fail, or clusters may go down if a matching serial number is not found.

config system global
set fgfm-peercert-withoutsn enable <----- Removed on v7.2.10/v7.4.6/v7.6.1 onward.

end

For more information about this behavior, check this KB article: Technical Tip: Setup custom certificate for FGFM protocol.

Other scenarios are mentioned in Common issues while adding FortiGate to FortiManager, and their solutions.

Related articles:

Troubleshooting Tip: How to troubleshoot connectivity issues between FortiGate and FortiManager

You are leaving our website