Description
This article describes how to troubleshoot connectivity issues between FortiGate and FortiAnalyzer. This article additionally describes how the OFTPD protocol is used to create two communication streams between FortiGate and FortiAnalyzers.
Scope
OFTP uses TCP/514 for connectivity, health check, file transfer and log display from FortiGate.
Log communication happens over either TCP or UDP 514:
- TCP/514 is used for log transmission with the reliable option enabled.
- UDP/514 is used for log transmission with the reliable option disabled.
Solution
The following sections describe how to verify and correct FortiAnalyzer connectivity issues.
Section 1: FortiGate and FortiAnalyzer firmware compatibility.
As a general rule, FortiAnalyzer should always have the same firmware release or be higher than that running on the FortiGate.
Check the FortiOS Compatibility Tool.
For example:
- FortiAnalyzer on v5.6 and FortiGate on v5.4 or v5.6 will work.
- FortiAnalyzer on v5.4 and FortiGate on v5.6 will not work.
Section 2: Verify the FortiAnalyzer configuration on the FortiGate.
Note:
If a VPN is used for the communication between FortiAnalyzer and FortiGate, the source IP must be set.
The following FortiGate Log settings are used to send logs to the FortiAnalyzer:
get log fortianalyzer setting
status : enable
ips-archive : enable
server : 10.34.199.143
enc-algorithm : high
conn-timeout : 10
monitor-keepalive-period: 5
monitor-failure-retry-period: 5
certificate :
source-ip : X.X.X.X -----> If the VPN is used to send logs
upload-option : 5-minute -----> Upload logs every 5 minutes.
reliable : disable -----> Logs are sent over UDP.
Note:
Log transmission uses TCP or UDP channels depending on reliable settings. It should be enabled to be encrypted.
The following FortiGate Log filter settings affect the number of logs sent:
get log fortianalyzer filter
severity : information <- The number of logs sent depends on the severity level, for example. information, warning, or critical. Different settings may give the impression that no logs are forwarded.
forward-traffic : enable
local-traffic : enable
multicast-traffic : enable
sniffer-traffic : enable
anomaly : enable
voip : enable
dlp-archive : enable
dns : enable
filter : <- Configuring filters can result in fewer logs being sent. Verify the filter settings to check if logs are being filtered.
filter-type : include <- Will only forward logs matching filter criteria.
To verify the FortiGate event log settings and filters, use the following commands:
get log eventfilter
get log setting
get sys setting
Note:
Some log settings are set in different parts of the FortiGate configuration.
- Log settings like usernames in uppercase, policy-name, and policy-comment are under 'config log setting'.
- VPN tunnel stats information is under the 'config system setting'.
- For FortiGate Clusters, configuring an HA-Group name under HA settings is mandatory.
Section 3: Once the settings are verified, check connectivity from the GUI and the CLI of the FortiGate.
CLI:
execute log fortianalyzer test-connectivity
Situation 1:
execute log fortianalyzer test-connectivity
Failed to get FAZ's status. Authentication Failed. (-19)
This is a side effect of FortiGate not being registered in the FortiAnalyzer. In the FortiAnalyzer GUI under Device Manager, add the FortiGate.
The following prompt will appear: 'FortiGate not authorized. Log in to the logging device and confirm registration of this device.'
Situation 2:
execute log fortianalyzer test-connectivity
Failed to get FAZ's status. Authentication Failed. (-19)
The following is observed under debug commands:
diagnose debug reset
diagnose debug app fgtlogd 255
diagnose debug app miglogd 255
diagnose debug enable
fgtlog_faz_stop_oftp_ex()-951: faz:10.34.199.143 connection close. reason:connection timeout // 10.34.199.143 is the IP of
FortiAnalyzer:
Solution:
Add the source IP in the configuration of 'config log fortianalyzer setting'.
Situation 3:
execute log fortianalyzer test-connectivity
Failed to get FAZ's status. No response from server. (-20)
Solution:
This error indicates a routing issue in the network. Check the routing.
Section 4: Verify connectivity when a FortiGate is registered on a FortiAnalyzer.
Successful sending of logs:
execute log fortianalyzer test-connectivity
FortiAnalyzer Host Name: FAZVM64
FortiGate Device ID: FGT1234567890
Registration: registered
Connection: allow
Disk Space (Used/Allocated): 0/Unlimited MB
Total Free Space: 831949 MB
Log: Tx & Rx (28 logs received since 02:00:18 02/20/18)
IPS Packet Log: Tx & Rx
Content Archive: Tx & Rx
Quarantine: Tx & Rx
The issue with sending the Logs:
execute log fortianalyzer test-connectivity
FortiAnalyzer Host Name: FAZVM64
FortiGate Device ID: FGT1KD3915802143
Registration: registered
Connection: allow
Disk Space (Used/Allocated): 0/Unlimited MB
Total Free Space: 819502 MB
Log: Tx & Rx (log not received) <- Check if UDP is used (reliable is disabled under log setting).
IPS Packet Log: Tx & Rx
Content Archive: Tx & Rx
Quarantine: Tx & Rx
Section 5: Basic network connectivity tests using ping, traceroute, and telnet tests.
Run the tests from the FortiGate and FortiAnalyzer CLI.
Note:
10.34.199.143 is the FortiAnalyzer IP, use the management IP of the FortiGate when testing from the FortiAnalyzer CLI.
execute ping 10.34.199.143
PING 10.34.199.143 (10.34.199.143): 56 data bytes
64 bytes from 10.34.199.143: icmp_seq=0 ttl=62 time=0.3 ms
64 bytes from 10.34.199.143: icmp_seq=1 ttl=62 time=0.3 ms
64 bytes from 10.34.199.143: icmp_seq=2 ttl=62 time=0.2 ms
64 bytes from 10.34.199.143: icmp_seq=3 ttl=62 time=0.2 ms
64 bytes from 10.34.199.143: icmp_seq=4 ttl=62 time=0.2 ms
--- 10.34.199.143 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.2/0.2/0.3 ms
execute traceroute 10.34.199.143
traceroute to 10.34.199.143 (10.34.199.143), 32 hops max, 3 probe packets per hop, 84 byte packets
1 10.107.3.108 0.070 ms 0.060 ms 0.053 ms
2 10.40.31.254 0.083 ms 0.122 ms 0.075 ms
3 10.34.199.143 0.217 ms 0.233 ms 0.120 ms
execute telnet 10.34.199.143 514
Trying 10.34.199.143...
Connected to 10.34.199.143.
Note:
Although ping and traceroute tests are successful, the connectivity may still fail. If this is the case, verify if TCP/UDP 514 ports are open on the intermediate devices (for example, firewalls) between FortiGate and FortiAnalyzer.
If FortiGate can connect using the exec telnet command but not using the exec log fortianalyzer test-connectivity command, this might be linked to the MTU size issue.
To confirm the MTU size for FortiGate traffic forwarded to FortiAnalyzer by executing the following commands on the FortiGate CLI:
execute ping-options df-bit yes -> do not fragment ICMP packet.
execute ping-options data-size 1500 -> ICMP will add 8 bytes for the ICMP header.
execute ping x.x.x.x -> where x.x.x.x is FortiAnalyzer-IP.
If there was packet loss, change the data size to 1470/1400/ 1350/ 1320/ 1312 and verify on which data size value there was no packet loss.
Then adjust if needed:
Technical Note: How to adjust the Maximum Transmission Unit (MTU) value on a FortiGate interface
Section 6: If FortiGate has VDOMs enabled, validate the management VDOM.
Select the VDOM that has communication with the FortiAnalyzer:
config global
show full system global | grep management-vdom
config system global
edit management-vdom <VDOM>
end
Test the FortiAnalyzer connectivity.
Section 7: Run the following command on the FortiAnalyzer to ensure proper log permissions are enabled on the FortiGate device:
execute log device permissions <device_id> all enable
Replace <device_id> with a specific FortiGate device ID.
Check if there is connectivity and if logs are sent to the FortiAnalyzer as expected or not.
Section 8: Advanced commands to check connectivity.
Using the sniffer command:
On the FortiGate CLI:
diagnose sniffer packet any 'host x.x.x.x and port 514' 6 0 l <- x.x.x.x is the IP address of the FortiAnalyzer.
On the FortiAnalyzer CLI:
diagnose sniffer packet any 'host y.y.y.y and port 514' 3 0 l <- y.y.y.y is the IP address of the FortiGate.
After, select Test Connectivity under the Log Settings of the FortiGate GUI or run the command 'diag log test' from the CLI. Packets received and sent from both devices should be seen.
Note: Analyze the SYN and ACK numbers in the communication.
Issue the following debug commands in FortiGate:
diagnose debug reset
diagnose debug app fgtlogd 255(since 7.2)
diagnose debug app miglogd 255
diagnose debug enable
Analyzing OFTPD application debugging on the FortiAnalyzer.
diagnose debug app oftpd 8 10.40.19.108 <- Alternatively, a device name can be used. IP is preferable.
diagnose debug timestamp enable
diagnose debug enable
diagnose test app oftpd 99 (to restart oftp)
Then select Test Connectivity under Log Setting of the FortiGate GUI or run the command 'diag log test' from the CLI, packets . received and sent from both devices should be seen.
A successful attempt will display 'Login Request' messages:
2018-02-20 15:50:51 oftpd_handle_session:3303: sock[29] ip[10.40.19.108] - Handle 'LOGIN_REQUEST' request type=2.
2018-02-20 15:50:51 handle_login:1961: sock[29] ip[10.40.19.108] - host = 'FGT1234567890'
2018-02-20 15:50:51 handle_login:1989: sock[29] ip[10.40.19.108] - Version: FortiGate-1000D v5.6.3,build1547,171204 (GA)
Virus-DB: 1.00123(2015-12-11 13:18)
IPS-DB: 6.00741(2015-12-01 02:30)
APP-DB: 6.00741(2015-12-01 02:30)
Industrial-DB: 6.00741(2015-12-01 02:30)
Serial-Number: FGT1234567890
Botnet DB: 1.00000(2012-05-28 22:51)
Virtual domain configuration: disable
Current HA mode: standalone
Current HA group:
2018-02-20 15:50:51 handle_login:1966: sock[29] ip[10.40.19.108] - vdom = 1
2018-02-20 15:50:51 oftpd_handle_session:3286: sock[29] ip[10.40.19.108] - [oftpd_handle_session] the peer close the connection.
2018-02-20 15:50:51 oftpd_close_session:2600: sock[29] ip[10.40.19.108] - Client connection closed. Reason 8(the peer close the connection)
Disable the debug commands using the following set of commands:
diagnose debug disable
diagnose debug timestamp disable
diagnose debug app oftpd 0
Section 9: If the above steps do not help, verify the time between FortiGate and FortiAnalyzer. If there is difference in time between the devices, synchronize the time between them.
Section 10: Ensure the FIPS mode is not enabled on the FortiGate by running the CLI command "get system status". If enabled, follow the below KB Article: Technical Tip: FortiGate FIPS-CC enabled to send log to FortiAnalyzer
Section 11: If the connectivity issue is still not resolved or isolated, collect the following information for Fortinet TAC to use for further investigation.
On the FortiGate:
- Consider whether there was any recent firmware upgrade done on the FortiGate after which connectivity issues occurred. If yes, indicate the upgrade path followed.
- Attach the latest unencrypted configuration backup of the FortiGate.
- Open an SSH session with FortiGate using PUTTY and log all the output to a file (Session -> Logging -> All session output -> Log File name -> Save the file as *.log).
Run the commands and attach the log file to the ticket.
get sys status
get sys performance status(run it 4-5 times with an interval of 3 sec)
diagnose sys top 1 25(run it for 8-10 seconds and then press ‘q’ to quit)
get log fortianalyzer setting
get log fortianalyzer filter
get log setting
get log eventfilter
execute traceroute <FortiAnalyzer IP address>
execute ping <FortiAnalyzer IP address>
execute telnet <FortiAnalyzer IP address> 514
execute log fortianalyzer test-connectivity
diagnose sys flash list
diagnose test app miglogd 6
diagnose test app fgtlogd 4 <-- Since 7.4.0 to replace diag test app miglogd 6.
diagnose test app fgtlogd 5 <-- Run 3 times.
diagnose log kernel-stats
diagnose debug crashlog read
- If the phenomenon below where the statistic of 'sent' does not increase as time flies by is discovered, killing fgtlogd process could be helped (for killing processes, refer to this article: Technical Tip: Find and restart/kill a process on a FortiGate by the process ID (PID) via pidof),
diagnose test application fgtlogd 4
Queues in all miglogds: cur:31373 total-so-far:383012484
global log dev statistics:
faz=5651550035, faz_cloud=0, fds_log=5651550035
faz 0: sent=5420899784, failed=0, cached=0, dropped=0
Num of REST URLs: 0
fds: sent=5418468459, failed=0, cached=0, dropped=2490221
Num of REST URLs: 0
diagnose test application fgtlogd 4
Queues in all miglogds: cur:31373 total-so-far:383012484
global log dev statistics:
faz=5651661887, faz_cloud=0, fds_log=5651661887
faz 0: sent=5420899784, failed=0, cached=0, dropped=0
Num of REST URLs: 0
fds: sent=5418468459, failed=0, cached=0, dropped=2490221
Num of REST URLs: 0
On the FortiAnalyzer:
- If there was any recent firmware upgrade done on the FortiAnalyzer after which connectivity issues occurred, indicate the upgrade path followed.
- Attach the latest unencrypted configuration backup of the FortiGate.
- Open an SSH session with FortiGate using PUTTY and log all the output to a file Session -> Logging -> All session output -> Log File name -> Save the file as *.log.
Run the commands and attach the log file to the ticket.
get sys status
get sys performance <- Run it 4-5 times with an interval of 10 sec.
execute top <- Run it for 8-10 seconds and then press ‘q’ to quit.
diagnose fortilogd lograte <- Run it 4-5 times with an interval of 10 sec.
diagnose fortilogd msgrate <- Run it 4-5 times with an interval of 10 sec.
diagnose fortilogd lograte-device <- Run it 4-5 times with an interval of 10 sec.
diagnose fortilogd lograte-type <- Run it 4-5 times with an interval of 10 sec.
diagnose fortilogd lograte-total <- Run it 4-5 times with an interval of 10 sec.
diagnose fortilogd logvol-adom all
diagnose test application oftp 5
diagnose test application oftp 6
diagnose test application oftp 7
diagnose test application oftp 10
diagnose test application fortilogd 1
diagnose test application fortilogd 2
diagnose test application fortilogd 3
diagnose test application fortilogd 4
diagnose test application fortilogd 7
diagnose test application fortilogd 10
diagnose test application sqllogd 9
Section 12: An Automation stitch can also be created to notify of FortiAnalyzer connectivity failure.
Navigate to Security Fabric -> Automation -> Create new -> Add Trigger -> FortiOS Event log -> Event 'FortiAnalyzer connection failed'.
Related articles:
Technical Tip: FortiGate is not able to send logs to FortiAnalyzer
Technical Note: How to create a log file of a session using PuTTY
Technical Tip: Ticket Creation via the Support Portal
Technical Note: Traffic Types and TCP/UDP Ports used by Fortinet Products
Troubleshooting Tips: No logs received on FortiAnalyzer
Technical Tip: How to setup a custom certificate regarding OFTP protocol
Technical Tip: Getting error: 'failed to get faz's status. invalid error number (0).(0)'
