FortiAnalyzer
FortiAnalyzer can receive logs and Windows host events directly from endpoints connected to EMS, and you can use FortiAnalyzer to analyze the logs and run reports.
wogasawara
Staff
Staff

Description


This article describes how to troubleshoot connectivity issues between FortiGate and FortiAnalyzer

Scope


This article describes how the OFTPD protocol is used to create two communication streams between FortiGate and FortiAnalyzer devices.

OFTP uses TCP/514 for connectivity, health check, file transfer and log display from FortiGate

Log communication happens over either TCP OR UDP 514 ,

-TCP/514 used for log transmission with the reliable option enabled
-UDP/514 used for log transmission with the reliable option disabled


Solution
The following sections describe how to verify and correct FortiAnalyzer connectivity issues.

Section 1: FortiGate and FortiAnalyzer firmware compatibility

As a general rule, FortiAnalyzer should always be the same firmware release equal to or higher than that running on the FortiGate.
Note: this may not be true at the patch level -- for more detail, see "Compatibility with FortiOS" document for FortiAnalyzer on https://docs.fortinet.com/product/fortianalyzer

For example:
- FortiAnalyzer on v5.6 and FortiGate on v5.4 or v5.6 will work
- FortiAnalyzer on v5.4 and FortiGate on v5.6 will not work
Section 2: Verify FortiAnalyzer configuration on the FortiGate

The following FortiGate Log settings are used to send logs to the FortiAnalyzer:
# get log fortianalyzer setting
status              : enable
ips-archive         : enable
server              : 10.34.199.143
enc-algorithm       : high    
conn-timeout        : 10
monitor-keepalive-period: 5
monitor-failure-retry-period: 5
certificate         :
source-ip           :
upload-option       : 5-minute -----> Upload logs every 5 minutes.
reliable            : disable  -----> Logs are sent over UDP.
Note.
Log transmission uses TCP or UDP channel depending on  reliable settings. It should be enabled to be encrypted.

The following FortiGate Log filter settings affect the amount of logs sent:

(global) # get log fortianalyzer filter
severity            : information ---> The amount of logs sent depend on the severity level e.g. information, warning or critical.
                                       Different settings may give the impression that no logs are forwarded.
forward-traffic     : enable
local-traffic       : enable
multicast-traffic   : enable
sniffer-traffic     : enable
anomaly             : enable
voip                : enable
dlp-archive         : enable
dns                 : enable 
filter              :         ---> Configuring filters can result in less logs being sent                                      
                                   verify the filter settings to check if logs are being filtered
filter-type         : include ---> Will only forward logs matching filter criteria
To verify the FortiGate event log settings and filters use the folloing commands
(vdom-name) # get log eventfilter
(vdom-name) # get log setting
(vdom-name) # get sys setting
Note: Some log settings are set in different parts of the FortiGate configuration
- Log settings like usernames in uppercase, policy-name and policy-comment are under “config log setting
- VPN tunnel stats information is under “config system setting
- For FortiGate Clusters configuring a HA-Group name under HA settings is mandatory

Section 3: Once the settings are verified,  check connectivity from the GUI and the CLI of the FortiGate

CLI:

# exec log fortianalyzer test-connectivity

Look for the following error message:
Failed to get FAZ's status. Authentication Failed. (-19) > Side effect of FortiGate not being registered in the FortiAnlalyzer. In the FortiAnalyzer GUI under Device manager add the FortiGate
GUI: The following will prompt will appear "FortiGate not authorized. Log in to logging device and confirm registration of this device."

 
Verify connectivity when a FortiGate is registered on a FortiAnalyzer

Use the following commands will verify connectivity:

Successful sending of logs

#exec log fortianalyzer test-connectivity
FortiAnalyzer Host Name: FAZVM64
FortiGate Device ID: FGT1234567890
Registration: registered
Connection: allow
Disk Space (Used/Allocated): 0/Unlimited MB
Total Free Space: 831949 MB
Log: Tx & Rx (28 logs received since 02:00:18 02/20/18)
IPS Packet Log: Tx & Rx
Content Archive: Tx & Rx
Quarantine: Tx & Rx
Issue with sending the Logs:
# exec log fortianalyzer test-connectivity
FortiAnalyzer Host Name: FAZVM64
FortiGate Device ID: FGT1KD3915802143
Registration: registered
Connection: allow
Disk Space (Used/Allocated): 0/Unlimited MB
Total Free Space: 819502 MB
Log: Tx & Rx (log not received)  > Check if UDP is used (reliable is disabled under log setting)
IPS Packet Log: Tx & Rx
Content Archive: Tx & Rx
Quarantine: Tx & Rx
Basic network connectivity tests using ping, traceroute and telnet tests

Run the tests from the FortiGate and FortiAnalyzer CLI:

Note: 10.34.199.143 is the FortiAnalyzer IP, use the management IP of the FortiGate when testing from the FortiAnalyzer CLI

# exec ping 10.34.199.143
PING 10.34.199.143 (10.34.199.143): 56 data bytes
64 bytes from 10.34.199.143: icmp_seq=0 ttl=62 time=0.3 ms
64 bytes from 10.34.199.143: icmp_seq=1 ttl=62 time=0.3 ms
64 bytes from 10.34.199.143: icmp_seq=2 ttl=62 time=0.2 ms
64 bytes from 10.34.199.143: icmp_seq=3 ttl=62 time=0.2 ms
64 bytes from 10.34.199.143: icmp_seq=4 ttl=62 time=0.2 ms
--- 10.34.199.143 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.2/0.2/0.3 ms
# exec traceroute 10.34.199.143 
traceroute to 10.34.199.143 (10.34.199.143), 32 hops max, 3 probe packets per hop, 84 byte packets
 1  10.107.3.108  0.070 ms  0.060 ms  0.053 ms
 2  10.40.31.254  0.083 ms  0.122 ms  0.075 ms
 3  10.34.199.143  0.217 ms  0.233 ms  0.120 ms
# exec telnet 10.34.199.143 514
Trying 10.34.199.143...
Connected to 10.34.199.143.

Note: Although ping and traceroute tests are successful, the connectivity may still fail. If this is the case, verify if TCP/UDP 514 ports are open on the intermediate devices (e.g. firewalls) between FortiGate and FortiAnalyzer.

Section 4: Advanced commands to check connectivity

Using the sniffer command on the FortiGate and the FortiAnalyzer

On the FortiGate CLI:

# diag sniffer packet any 'host x.x.x.x and port 514' 4 0 l 
x.x.x.x is the IP address of the FortiAnalyzer

On the FortiAnalyzer CLI:
# diag sniffer packet any 'host y.y.y.y and port 514' 4 0 l 
y.y.y.y is the IP address of the FortiGate

Then click on Test Connectivity under Log Setting of the FortiGate GUI or run the command ‘diag log test’ form the CLI, you should see packets received and sent from both devices.

Note: Analyze the SYN and ACK numbers in the communication

Analyzing OFTPD application debugging on the FortiAnalyzer

Debugging the “OFTPD” deamon for connectivity issues:

# diag debug app oftpd 8 10.40.19.108  > Or device name can be used. IP is preferable
# diag debug timestamp enable
# diag debug enable
Then click on Test Connectivity under Log Setting of the FortiGate GUI or run the command ‘diag log test’ form the CLI, you should see packets received and sent from both devices.

A successful attempt will display "Login Request" messages:

2018-02-20 15:50:51 oftpd_handle_session:3303: sock[29] ip[10.40.19.108] - Handle 'LOGIN_REQUEST' request type=2.
2018-02-20 15:50:51 handle_login:1961: sock[29] ip[10.40.19.108] - host = 'FGT1234567890'
2018-02-20 15:50:51 handle_login:1989: sock[29] ip[10.40.19.108] - Version: FortiGate-1000D v5.6.3,build1547,171204 (GA)
Virus-DB: 1.00123(2015-12-11 13:18)
IPS-DB: 6.00741(2015-12-01 02:30)
APP-DB: 6.00741(2015-12-01 02:30)
Industrial-DB: 6.00741(2015-12-01 02:30)
Serial-Number: FGT1234567890
Botnet DB: 1.00000(2012-05-28 22:51)
Virtual domain configuration: disable
Current HA mode: standalone
Current HA group:

2018-02-20 15:50:51 handle_login:1966: sock[29] ip[10.40.19.108] - vdom = 1
2018-02-20 15:50:51 oftpd_handle_session:3286: sock[29] ip[10.40.19.108] - [oftpd_handle_session] the peer close the connection.
2018-02-20 15:50:51 oftpd_close_session:2600: sock[29] ip[10.40.19.108] - Client connection closed. Reason 8(the peer close the connection)
Disable the debug using below set of commands
# diag debug disable
# diag debug timestamp disable
# diag debug app oftpd 0
Section 5: If the connectivity issue is still not resolved or isolated, collect the following information for Fortinet TAC to use for further investigation

On the FortiGate:
- Was there any recent firmware upgrade done on the FortiGate after which connectivity issues occurred? If yes, please indicate the upgrade path followed
- Attach the latest unencrypted configuration backup of the FortiGate.
- Open a ssh session with FortiGate using PUTTY and log all the output to a file (Session > Logging > All session output > Log File name > Save the file as *.log).
Run the commands and attach the log file to the ticket.
get sys status   
get sys performance status(run it 4-5 times with an interval of 3 sec)
diag sys top 1 25(run it for 8-10 seconds and then press ‘q’ to quit)
get log fortianalyzer setting
get log fortianalyzer filter
get log setting
get log eventfilter
exec traceroute <FortiAnalyzer IP address>
exec ping <FortiAnalyzer IP address>
exec log fortianalyzer test-connectivity
diag sys flash list
diag test app miglogd 6
diag log kernel-stats
diag debug crashlog read
On the FortiAnalyzer:
- Was there any recent firmware upgrade done on the FortiAnalyzer after which connectivity issues occurred? If yes, please indicate the upgrade path followed
- Attach the latest unencrypted configuration backup of the FortiGate.
- Open a ssh session with FortiGate using PUTTY and log all the output to a file (Session > Logging > All session output > Log File name > Save the file as *.log).

Run the commands and attach the log file to the ticket.
get sys status
get sys performance (run it 4-5 times with an interval of 10 sec)
exec top  (run it for 8-10 seconds and then press ‘q’ to quit)
diag fortilogd lograte (run it 4-5 times with an interval of 10 sec)
diag fortilogd msgrate (run it 4-5 times with an interval of 10 sec)
diag fortilogd msgrate-device (run it 4-5 times with an interval of 10 sec)
diag fortilogd msgrate-type (run it 4-5 times with an interval of 10 sec)
diag fortilogd msgrate-total (run it 4-5 times with an interval of 10 sec)
diagnose test application oftp 5
diagnose test application oftp 6
diagnose test application oftp 7
diagnose test application oftp 10
diagnose test application fortilogd 1
diagnose test application fortilogd 2
diagnose test application fortilogd 3
diagnose test application fortilogd 4
diagnose test application fortilogd 7
diagnose test application fortilogd 10
diagnose test application sqllogd 9

Related Articles

Technical Note: How to create a log file of a session using PuTTY

Technical Tip: Ticket Creation via the Support Portal

Technical Note: FortiAnalyzer is not accepting logs, event log reports unable to accept logs from de...

Technical Note: Traffic Types and TCP/UDP Ports used by Fortinet Products

Troubleshooting Tips: No logs received on FortiAnalyzer