Description
This article is intended to guide administrators when troubleshooting connectivity issues between the FortiGate and their FortiAnalyzer and/or Syslog servers.
This will include suggestions for packet captures, debug commands, and other initial troubleshooting tips.
Note that this is not meant to be an exhaustive troubleshooting guide, as it is written for the purposes of investigating connectivity issues from the perspective of the FortiGate.
If more in-depth troubleshooting is required then it is recommended to create a ticket with either FortiGate TAC or FortiAnalyzer TAC, depending on the specific nature of the inquiry/issue.
Scope
FortiGate, FortiAnalyzer (limited).
Table of Contents
1) Frequently-Used Troubleshooting Commands
2) General Troubleshooting Steps
3) Troubleshooting Steps: FortiAnalyzer
4) Troubleshooting Steps: Syslog
Frequently-Used Troubleshooting Commands
# diagnose sniffer packet <interface OR any> '<filters>' <verbosity> <packet count> <local vs. absolute time>
- CLI Packet Capture Tool available in FortiOS. Useful for verifying basics of packet flow such as Source and Destination IPs/Ports, as well as which Ingress/Egress interfaces are being used on the FortiGate.
- A GUI Packet Capture tool is also available under Network -> Diagnostics -> Packet Capture (Network -> Packet Capture in FortiOS 7.0 and earlier).
# diagnose test application miglogd <#>
# diagnose test application miglogd 1 <----- Show global log setting.
# diagnose test application miglogd 2 <----- Show VDOM log setting.
- Instantaneous debugs that read and output information from the miglogd process (the command can be run with no numeric argument to produce a list of allowed options).
- The examples above will show connection states to FortiAnalyzer and Syslog, as well as certain flags that correspond to the underlying configuration.
# diagnose debug application miglogd -1
# diagnose debug enable
- Live debug output from the miglogd process as it occurs.
Different but complementary information to the above instantaneous commands.
# execute log fortianalyzer test-connectivity
- Tests connectivity and outputs information on various aspects of the FortiAnalyzer connection.
General Troubleshooting Steps
This section discusses some suggestions that are common to troubleshooting connections from the FortiGate to both FortiAnalyzer and syslog servers.
These suggestions help to rule out the more-common issues that an administrator might observe either during or after setup.
Reference material related to the following steps will be included in the Related Articles section, check below for further topics/information.
1) Run packet captures to confirm that the FortiGate is sending traffic to the Logging Server.
- FortiAnalyzer receives traffic using both TCP/514 and UDP/514 (if reliable is not enabled), whereas syslog will listen on either TCP/514 or UDP/514 depending on the mode being used.
- Use the packet capture to check what outgoing interface the FortiGate is using, what source and destination IP addresses are being specified, and whether or not there is any response from the remote FortiAnalyzer/syslog server (e.g. the TCP three-way handshake).
- With that in mind, the following is a sample command for the CLI packet sniffer:
# diag sniff pack any 'host <log server IP address> and port 514' 4 0 1
- The above command captures traffic on any interface that includes both the log server's IP address and port 514 (either UDP or TCP). It captures at verbosity level 4, includes an unlimited packet count, and specifies the time format to match the administrator's time zone.
2) Check the source-IP being used if the FortiGate uses an IPsec tunnel to reach the log server.
- The FortiGate will use the IP address of the outgoing interface as the source when initiating traffic to an external resource (i.e. self-originated traffic).
- Many administrators will choose not to set an tunnel IP address (along with the remote tunnel address) on their IPsec tunnels, as it is not strictly required for a functional setup.
- However, this means that the IPsec tunnel will not have an IP address readily available for the FortiGate to use when generating its own traffic.
- When this occurs, The FortiGate will instead select an IP address from the first available interface on its indexed list, usually a 'mgmt' or 'wan' interface whose IP address is either not allowed across the VPN or will be dropped by routing/policy conflicts.
- This can be resolved by either a) setting tunnel and remote IP addresses on the IPsec tunnel interface on the FortiGate(s), or b) using the source-ip option available in the CLI log configuration:
# config log fortianalyzer setting
set source-ip <IP address on the FortiGate>
end
# config log syslogd setting
set source-ip <IP address on the FortiGate>
end
Troubleshooting Steps: FortiAnalyzer
This section includes suggestions specific to FortiAnalyzer connections.
1) Check that the FortiGate is authorized by the FortiAnalyzer.
- The FortiGate must be authorized by the FortiAnalyzer before it can use it as a log facility.
- If the above packet capture test indicates that there is working network connectivity between the FortiGate and FortiAnalyzer, then one could use the commands in the Frequently-Used Troubleshooting Commands section to check if authorization is the issue from the FortiGate.
- The following screenshots show samples of what a disconnected, unauthorized FortiGate may show:
FortiGate CLI showing the enabled but disconnected configuration for FortiAnalyzer.
FortiAnalyzer Web GUI reporting an unauthorized device.
FortiAnalyzer Web GUI demonstrating how to authorize an unauthorized FortiGate
2) FortiGate and FortiAnalyzer-VM have working network connectivity, but the certificate verification is failing due to an incorrect FortiAnalyzer serial number.
- By default, the FortiGate will perform certificate verification against the FortiAnalyzer when it is first configured in the GUI. This process involves checking the FortiAnalyzer's presented certificate, specifically the Serial Number contained within it.
- However, earlier license files on the virtual-machine FortiAnalyzers (2018 and earlier) may not include the actual serial number of the FortiAnalyzer-VM, which can lead to verification issues.
- Refer to the following Community KB article for steps on how this can be resolved: Technical Tip: FortiAnalyzer certificate does not reflect correct serial number
Troubleshooting Steps: Syslog
1) FortiGate has confirmed network connectivity to the Syslog server, but the logs are not in the correct format.
- The FortiGate supports a number of formats with syslog, including default, CSV, CEF, and RFC5424 (added in FortiOS 6.4.6, 7.0.0 and later).
- Consult with the vendor/provider for the target syslog server to see what formats supported, then adjust the format on the FortiGate using the following CLI configuration:
# config log syslogd setting
set format <default | csv | cef | rfc5424>
end
2) FortiGate has confirmed network connectivity to the Syslog server using Reliable (TCP-based) syslog, but the multiple logs received on the syslog server are not being separated correctly into individual entries.
- One explanation for this issue could be that the syslog server does not support octet-counted framing, a function specified in RFC6587 section 3.4.1.
- As a primer, the FortiGate will send multiple logs per packet to the syslog server when using TCP-based syslog. By comparison, UDP-based syslog results in one log message sent per packet.
- In order to distinguish the individual logs within that TCP payload, the FortiGate follows RFC6587 and uses octet-counted framing to specify how long each log entry is so that the receiving syslog server can interpret each log correctly.
- However, some syslog servers (such as rsyslog) may default to the traditional 'Non-Transparent Framing', which results in these log entries being misinterpreted upon receipt.
- The solution is to modify the Syslog server and enable octet-counted framing in order to be compatible with the FortiGate in Reliable mode.
Further Information
Miscellaneous CLI options
A full CLI reference for both config log fortianalyzer setting and config log syslogd setting are linked below in the Related Articles section, but this section will discuss some notable options in the configuration and what they do:
certificate-verification (FortiAnalyzer) - 'Enable/disable identity verification of FortiAnalyzer by use of certificate.'
- FortiAnalyzer will present a certificate bearing its serial number to the FortiGate, which the administrator can choose to trust as a method of authentication.
- When configuring FortiAnalyzer in the GUI, certificate-verification is already enabled by default unless 'Allow access to FortiGate REST API' is disabled.
reliable (FortiAnalyzer) - 'Enable/disable reliable logging to FortiAnalyzer.'
- Disabled by default, enabling this option results in the FortiGate using TCP/514 for log uploads to FortiAnalyzer, rather than UDP/514.
mode (Syslog) - 'Remote syslog logging over UDP/Reliable TCP.'
- Options include udp, legacy-reliable (TCP and based on the older RFC3195), and reliable (TCP and based on the newer RFC6587).
facility (Syslog) - 'Remote syslog facility.'
- Syslog facility is defined within RFC5424 and is used to determine which processes on the client had created the message, and they can be used as a way of filtering which messages will be sent out to the remote syslog server (default is local7)
format (Syslog) - 'Log format.'
- Used to set which Syslog format the FortiGate will use when sending out to the remote syslog server.
- As mentioned above, the options include default, csv, cef, and rfc5424.
source-ip (Both) - 'Source IPv4 or IPv6 address used to communicate with FortiAnalyzer.'
- This setting is present in many of the FortiGate's 'self-originated traffic' configurations, including DNS, FortiGuard, SNMP, RADIUS, etc.
- Highly-useful when a specific source IP is needed for successful communication through the network.
interface-select-method (Both) - 'Specify how to select outgoing interface to reach server.'
- By default, the FortiGate follows the routing table to determine the interface used to reach FortiAnalyzer.
- This option can allow the administrator to use SD-WAN rules to control routing to the log server, or one can also set a static manual interface (assuming multiple valid options are available).
