Troubleshooting Tip: FortiGate to FortiAnalyzer connectivity

wogasawara · ‎03-23-2018

Description

This article describes how to troubleshoot connectivity issues between FortiGate and FortiAnalyzer.
This article additionally describes how the OFTPD protocol is used to create two communication streams between FortiGate and FortiAnalyzer devices.

Scope

OFTP uses TCP/514 for connectivity, health check, file transfer and log display from FortiGate.

Log communication happens over either TCP OR UDP 514:

TCP/514 is used for log transmission with the reliable option enabled.
UDP/514 is used for log transmission with the reliable option disabled.

Solution

The following sections describe how to verify and correct FortiAnalyzer connectivity issues.

Section 1: FortiGate and FortiAnalyzer firmware compatibility.

As a general rule, FortiAnalyzer should always have the same firmware release or be higher than that running on the FortiGate.

Check the compatibility matrix.

For example:

FortiAnalyzer on v5.6 and FortiGate on v5.4 or v5.6 will work.
FortiAnalyzer on v5.4 and FortiGate on v5.6 will not work.

Section 2: Verify FortiAnalyzer configuration on the FortiGate.
Note:

If a VPN is used for the communication between FortiAnalyzer and FortiGate, the source IP must be set.

The following FortiGate Log settings are used to send logs to the FortiAnalyzer:

get log fortianalyzer setting
status              : enable
ips-archive         : enable
server              : 10.34.199.143
enc-algorithm       : high
conn-timeout        : 10
monitor-keepalive-period: 5
monitor-failure-retry-period: 5
certificate         :
source-ip           : X.X.X.X -----> If the VPN is used to send logs
upload-option       : 5-minute -----> Upload logs every 5 minutes.
reliable            : disable -----> Logs are sent over UDP.

Note:
Log transmission uses TCP or UDP channels depending on reliable settings. It should be enabled to be encrypted.

The following FortiGate Log filter settings affect the number of logs sent:

get log fortianalyzer filter
severity            : information <- The number of logs sent depends on the severity level e.g. information, warning, or critical. Different settings may give the impression that no logs are forwarded.
forward-traffic     : enable
local-traffic       : enable
multicast-traffic   : enable
sniffer-traffic     : enable
anomaly             : enable
voip                : enable
dlp-archive         : enable
dns                 : enable
filter              :         <- Configuring filters can result in fewer logs being sent. Verify the filter settings to check if logs are being filtered.
filter-type         : include <- Will only forward logs matching filter criteria.

To verify the FortiGate event log settings and filters use the following commands:

get log eventfilter
get log setting
get sys setting

Note:

Some log settings are set in different parts of the FortiGate configuration.

Log settings like usernames in uppercase, policy-name and policy-comment are under 'config log setting'.
VPN tunnel stats information is under 'config system setting'.
For FortiGate Clusters, configuring a HA-Group name under HA settings is mandatory.

Section 3: Once the settings are verified, check connectivity from the GUI and the CLI of the FortiGate.

CLI:

exec log fortianalyzer test-connectivity

Look for the following error message:

Failed to get FAZ's status. Authentication Failed. (-19) <- Side effect of FortiGate not being registered in the FortiAnlalyzer. In the FortiAnalyzer GUI under Device Manager add the FortiGate.

GUI:

The following prompt will appear 'FortiGate not authorized. Log in to the logging device and confirm registration of this device.'

Verify connectivity when a FortiGate is registered on a FortiAnalyzer.

Use the following commands will verify connectivity:

Successful sending of logs:

exec log fortianalyzer test-connectivity
FortiAnalyzer Host Name: FAZVM64
FortiGate Device ID: FGT1234567890
Registration: registered
Connection: allow
Disk Space (Used/Allocated): 0/Unlimited MB
Total Free Space: 831949 MB
Log: Tx & Rx (28 logs received since 02:00:18 02/20/18)
IPS Packet Log: Tx & Rx
Content Archive: Tx & Rx
Quarantine: Tx & Rx

The issue with sending the Logs:

exec log fortianalyzer test-connectivity
FortiAnalyzer Host Name: FAZVM64
FortiGate Device ID: FGT1KD3915802143
Registration: registered
Connection: allow
Disk Space (Used/Allocated): 0/Unlimited MB
Total Free Space: 819502 MB
Log: Tx & Rx (log not received) <- Check if UDP is used (reliable is disabled under log setting).
IPS Packet Log: Tx & Rx
Content Archive: Tx & Rx
Quarantine: Tx & Rx

Basic network connectivity tests using ping, traceroute, and telnet tests.

Run the tests from the FortiGate and FortiAnalyzer CLI.

Note:

10.34.199.143 is the FortiAnalyzer IP, use the management IP of the FortiGate when testing from the FortiAnalyzer CLI.

exec ping 10.34.199.143
PING 10.34.199.143 (10.34.199.143): 56 data bytes
64 bytes from 10.34.199.143: icmp_seq=0 ttl=62 time=0.3 ms
64 bytes from 10.34.199.143: icmp_seq=1 ttl=62 time=0.3 ms
64 bytes from 10.34.199.143: icmp_seq=2 ttl=62 time=0.2 ms
64 bytes from 10.34.199.143: icmp_seq=3 ttl=62 time=0.2 ms
64 bytes from 10.34.199.143: icmp_seq=4 ttl=62 time=0.2 ms
--- 10.34.199.143 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.2/0.2/0.3 ms

exec traceroute 10.34.199.143
traceroute to 10.34.199.143 (10.34.199.143), 32 hops max, 3 probe packets per hop, 84 byte packets
1 10.107.3.108 0.070 ms 0.060 ms 0.053 ms
2 10.40.31.254 0.083 ms 0.122 ms 0.075 ms
3 10.34.199.143 0.217 ms 0.233 ms 0.120 ms

exec telnet 10.34.199.143 514
Trying 10.34.199.143...
Connected to 10.34.199.143.

Note:

Although ping and traceroute tests are successful, the connectivity may still fail. If this is the case, verify if TCP/UDP 514 ports are open on the intermediate devices (for example: firewalls) between FortiGate and FortiAnalyzer.

If one sees that the FortiGate can connect using the exec telnet command but not using the exec log fortianalyzer test-connectivity command, this might be linked to the MTU size issue.

To confirm the MTU size for FortiGate traffic forwarded to FortiAnalyzer by executing the following commands on the FortiGate CLI:

exe ping-options df-bit yes -> do not fragment ICMP packet.
exe ping-options data-size 1500 -> ICMP will add 8 bytes for the ICMP header.
exe ping x.x.x.x -> where x.x.x.x is FortiAnalyzer-IP.

If there was packet loss, change the data size to 1470/1400/ 1350/ 1320/ 1312 and verify on which data size value there was no packet loss.

Then adjust if needed:

Technical Note: How to adjust the Maximum Transmission Unit (MTU) value on a FortiGate interface

If FortiGate has VDOMs enabled, validate the management VDOM.

Select the VDOM that has communication with the FortiAnalyzer:

config global
show full system global | grep management-vdom

config system global

edit management-vdom <VDOM>

end

Test the FortiAnalyzer connectivity.

Test as follows:

Run the following command on the FortiAnalyzer to ensure proper log permissions are enabled on the FortiGate device:

execute log device permissions <device_id> all enable

Replace <device_id> with a specific FortiGate device ID.

Check if there is connectivity and logs are sent to the FortiAnalyzer as expected or not.

Section 4: Advanced commands to check connectivity.

Using the sniffer command on the FortiGate and the FortiAnalyzer.

On the FortiGate CLI:

diag sniffer packet any 'host x.x.x.x and port 514' 6 0 l

diag debug reset

diag debug app fgtlogd 255(since 7.2)

diag debug app miglogd 255

diag debug enable

x.x.x.x is the IP address of the FortiAnalyzer.

On the FortiAnalyzer CLI:

diag sniffer packet any 'host y.y.y.y and port 514' 3 0 l

y.y.y.y is the IP address of the FortiGate.

Then select Test Connectivity under Log Setting of the FortiGate GUI or run the command ‘diag log test’ from the CLI, packets received and sent from both devices should be seen.

Note:

Analyze the SYN and ACK numbers in the communication.

Analyzing OFTPD application debugging on the FortiAnalyzer.

Debugging the 'OFTPD' daemon for connectivity issues:

diag debug app oftpd 8 10.40.19.108 <- Alternatively, a device name can be used. IP is preferable.

diag debug timestamp enable
diag debug enable

diag test app oftp 99 (to restart oftp)

Then select Test Connectivity under Log Setting of the FortiGate GUI or run the command ‘diag log test’ from the CLI, packets received and sent from both devices should be seen.

A successful attempt will display 'Login Request' messages:

2018-02-20 15:50:51 oftpd_handle_session:3303: sock[29] ip[10.40.19.108] - Handle 'LOGIN_REQUEST' request type=2.
2018-02-20 15:50:51 handle_login:1961: sock[29] ip[10.40.19.108] - host = 'FGT1234567890'
2018-02-20 15:50:51 handle_login:1989: sock[29] ip[10.40.19.108] - Version: FortiGate-1000D v5.6.3,build1547,171204 (GA)
Virus-DB: 1.00123(2015-12-11 13:18)
IPS-DB: 6.00741(2015-12-01 02:30)
APP-DB: 6.00741(2015-12-01 02:30)
Industrial-DB: 6.00741(2015-12-01 02:30)
Serial-Number: FGT1234567890
Botnet DB: 1.00000(2012-05-28 22:51)
Virtual domain configuration: disable
Current HA mode: standalone
Current HA group:

2018-02-20 15:50:51 handle_login:1966: sock[29] ip[10.40.19.108] - vdom = 1
2018-02-20 15:50:51 oftpd_handle_session:3286: sock[29] ip[10.40.19.108] - [oftpd_handle_session] the peer close the connection.
2018-02-20 15:50:51 oftpd_close_session:2600: sock[29] ip[10.40.19.108] - Client connection closed. Reason 8(the peer close the connection)

Disable the debug using the below set of commands:

diag debug disable
diag debug timestamp disable
diag debug app oftpd 0

Section 5: If the connectivity issue is still not resolved or isolated, collect the following information for Fortinet TAC to use for further investigation.

On the FortiGate:

Was there any recent firmware upgrade done on the FortiGate after which connectivity issues occurred? If yes, indicate the upgrade path followed.
Attach the latest unencrypted configuration backup of the FortiGate.
Open an SSH session with FortiGate using PUTTY and log all the output to a file (Session -> Logging -> All session output -> Log File name -> Save the file as *.log).

Run the commands and attach the log file to the ticket.

get sys status
get sys performance status(run it 4-5 times with an interval of 3 sec)
diag sys top 1 25(run it for 8-10 seconds and then press ‘q’ to quit)
get log fortianalyzer setting
get log fortianalyzer filter
get log setting
get log eventfilter
exec traceroute <FortiAnalyzer IP address>
exec ping <FortiAnalyzer IP address>

exec telnet <FortiAnalyzer IP address> 514
exec log fortianalyzer test-connectivity
diag sys flash list
diag test app miglogd 6

diag test app fgtlogd 4 (since 7.4.0 to replace diag test app miglogd 6)

diag test app fgtlogd 5 (run 3 times)

diag log kernel-stats
diag debug crashlog read

On the FortiAnalyzer:

If there was any recent firmware upgrade done on the FortiAnalyzer after which connectivity issues occurred, indicate the upgrade path followed.
Attach the latest unencrypted configuration backup of the FortiGate.
Open an SSH session with FortiGate using PUTTY and log all the output to a file Session -> Logging -> All session output -> Log File name -> Save the file as *.log.

Run the commands and attach the log file to the ticket.

get sys status
get sys performance (run it 4-5 times with an interval of 10 sec)
exec top (run it for 8-10 seconds and then press ‘q’ to quit)
diag fortilogd lograte (run it 4-5 times with an interval of 10 sec)
diag fortilogd msgrate (run it 4-5 times with an interval of 10 sec)
diag fortilogd msgrate-device (run it 4-5 times with an interval of 10 sec)
diag fortilogd msgrate-type (run it 4-5 times with an interval of 10 sec)
diag fortilogd msgrate-total (run it 4-5 times with an interval of 10 sec)
diagnose test application oftp 5
diagnose test application oftp 6
diagnose test application oftp 7
diagnose test application oftp 10
diagnose test application fortilogd 1
diagnose test application fortilogd 2
diagnose test application fortilogd 3
diagnose test application fortilogd 4
diagnose test application fortilogd 7
diagnose test application fortilogd 10
diagnose test application sqllogd 9

Related articles: