Troubleshooting Tip: Unable to connect to FortiGuard servers

[patelr]

Description	This article explains the possible reasons why FortiGate is unable to connect FortiGuard servers and offers steps to troubleshoot the problem. The 'Unable to connect to FortiGuard servers' error message can be seen in two places under Dashboard -> Status -> Licenses. Go under System -> FortiGuard -> FortiGuard Updates. FortiOS v7.6.x: Go under Dashboard -> Status -> Licenses. Go under System -> FortiGuard -> Subscriptions -> Licensed.
Scope	FortiGate, FortiWeb.
Solution	The communication between FortiGate and FortiGuard for web filtering and antispam is different from the communication for antivirus and IPS. If public FortiGuard services or servers are suspected to be down in one or more region(s), the following link provides status, outage, maintenance etc for such services:FortiGuard Anycast Query Status. Check connectivity to FortiGuard servers by checking to ensure FortiGate correctly resolves DNS with the following hostnames: execute ping service.fortiguard.net execute ping update.fortiguard.net execute ping guard.fortinet.net If DNS resolves but there is no ping, check the routing table (get router info routing-table all) to determine the best route for FortiGuard services (if ISDB is used in SD-WAN/policy route or static route) or the default route. If all traffic is sent via IPsec tunnel, ensure that: Firewall policies are in place. Phase 2 selectors are configured on both ends of the IPsec tunnel. If SD-WAN is configured as the default route, verify: SD-WAN rules for all traffic or FortiGuard services FortiGuard configuration settings config system fortiguard     set interface-select-method sdwan end See Technical Tip: Functionality of 'set interface-select-method' for local-traffic with SD-WAN for more information. Note: Suppose connectivity to FortiGuard is required through one of the outgoing interfaces. In that case, it is possible to use 'interface-select-method' with the 'specify' action and then select the outgoing interface under the FortiGuard configuration.  config system fortiguard     set interface-select-method {specify}     set interface <WAN Interface> end If the DNS resolves and able to ping, move to Step 2. If it is not resolving, check the DNS configuration. One of the most possible causes is when the 'Use FortiGuard Servers' option is changed to 'Specify' for use with an internal DNS server, without switching the DNS protocols or validating if the new DNS Server supports DoT (default setting of FortiGuard servers) which uses TCP 853 or DoH that uses TCP 443. If that happens, the DNS server will be unreachable, and DNS resolution will not work, as seen in the next picture.  After changing to UDP 53, the DNS will be reachable, and the resolution should start working. The following are the commands to change the protocol via CLI: config system dns     set protocol cleartext <- DoT is for using TCP/853, and cleartext is used for UDP/53. end In some cases, verify the system dns config. Check any source-ip config and make sure the IP configured is public-facing and not an internal IP address. This is because the DNS needs to communicate to the public WAN to be able to reach the FortiGuard servers. If there are any internal 'source-ip' configured, unset to default (0.0.0.0) or configure a public-facing IP.  config system dns     set source-ip 0.0.0.0 <- default Run 'diagnose debug rating' in the CLI: diagnose debug rating The output of the command 'diagnose debug rating' displays flags next to the servers: I: The server initially connected to validate the license and fetch the server list. Typically, only one server will have this flag. The IP address FortiGate received when resolving the name service.fortiguard.net. If the administrator has not overwritten the FortiGuard FQDN or IP address in the FortiGuard configuration, there are usually two or three servers with this flag. S: The IP address FortiGate received from FortiManager. T: The server is not replying to FortiGate queries. F: The server is down. If all servers in the list have F(ailed), this may mean either that all FortiGuard servers on the Fortinet side are down (unlikely) or that this FortiGate has a problem reaching them at the network level. If the PPPoE interface is being used as a WAN interface, review the MTU value and configure TCP-MSS on the PPPoE interface. fnsysctl ifconfig -a The PPPoE interface name will be displayed as 'ppp(x)'. ppp1 Link encap:Point-Point Protocol inet addr:x.x.x.x Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1492 Metric:1 The MSS value should be = MTU - 40. Cannot view some web sites when using PPPoE. For example, if the MTU of the PPPoE interface is 1492: config system interface     edit <interface name>          set tcp-mss 1452  end Even if the FortiGate is not using PPPoE, there could still be MTU issues along the path, either in the directly connected ISP, or a transit ISP. Lowering the MTU is still a valid test. The best way to determine if there are MTU issues along the path is to perform a packet capture for the FortiGuard server IP the FortiGate is connecting to, and if the larger packets after the initial TCP handshake (1500 bytes MTU) do no get ACK'ed, it means they are getting dropped somewhere and MTU could be the reason. Under global settings, make sure that set cloud-communication is set to enable. FGT # config system global FGT (global) # set cloud-communication ? enable     Allow cloud communication. disable    Disable all cloud-related settings. FGT (global) #set cloud-communication enable FGT (global) #end If there are VDOMs enabled, all communication to the FortiGuard network is initiated from the management/root VDOM only. config sys global     set management-vdom "root" end However, from firmware v7.2 onwards in multi-VDOM mode, users can choose from which VDOM FortiGuard services and updates are initiated instead of being locked to the management VDOM. config global     config system fortiguard         set vdom "root"     end end The VDOM specified should be able to reach the internet and should be able to resolve DNS queries. To set up FortiGuard services on a non-management VDOM: Specify the VDOM to be used under 'config system fortiguard'. Ensure that the specific VDOM has connectivity to the internet. FortiGate should be able to resolve the DNS from within the VDOM, so that the FortiGuard services may resolve the server name using the specific VDOM. Verify Antivirus & IPS Definition and Filtering Services: Check Antivirus & IPS Definition under System -> FortiGuard -> AntiVirus & IPS Updates -> Update AV and IPS Definitions. FortiOS v7.2.x: The active subscriptions under System -> FortiGuard are being updated with the 'FortiGuard Updates', the options for 'AntiVirus & IPS Updates' have been removed. FortiOS v7.6.x: The active subscriptions under System -> FortiGuard -> Subscriptions -> Licensed are being updated with the 'FortiGuard Updates' System -> FortiGuard -> FortiGuard settings. Check Filtering Services under System -> FortiGuard -> Filtering. FortiOS v7.6.x: Check Filtering Services under System -> FortiGuard -> FortiGuard settings and then expand the Filtering option. Anycast servers: If FortiGuard is not reachable via the default Anycast method, it is suggested to disable Anycast and switch back to unicast servers (Anycast communication). config system fortiguard     set fortiguard-anycast disable     set protocol udp     set port 8888     set sdns-server-ip 208.91.112.220 173.243.140.53 210.7.96.53    end For FortiGuard's SDNS rating service, there are two modes in FortiOS: Mode 1: Use of unicast network, which uses the DNS over UDP protocol for FortiGate and FortiGuard transactions. Mode 2: Use of Anycast network (default), which uses DNS over TLS protocol for FortiGate and FortiGuard transactions. To use UDP/53:  config system fortiguard     set fortiguard-anycast disable     set sdns-server-ip 208.91.112.220 173.243.140.53 210.7.96.53 end To use DoT (TCP/853). config system fortiguard     set fortiguard-anycast enable end With Anycast, FortiGate is only aware of one single server IP. This is a floating IP address that will connect to the closest server geographically, and if this server is down, it will point to another server instead. With Unicast, the FortiGate must maintain a list of servers that it tries, and if one stops working, it then switches over to another. In many cases, problems related to FortiGuard are caused by ISPs. Some ISPs block traffic on port 53 that is not DNS or that contains large packets. In those cases, the solution is to use port 8888. Other ISPs block traffic to HTTPS port 8888. In those cases, the solution is to use UDP port 53.  Related port information: Encrypted Virus Samples auto-submitted to FortiGuard – 25. DNS lookups – 53 UDP. FortiGuard Server List requests to FortiGuard – 53 UDP. AntiSpam or Web Filtering rating lookup queries to FortiGuard – 53 UDP or 8888 UDP. URL/AS rating lookup queries to FortiGuard – 53 UDP. Real-time Black List (RBL) lookup requests to RBL services – 53 UDP. Fortinet Device Registration to FortiGuard – 80 HTTP. Firmware and Signature Downloads from FortiGuard – 443 HTTPS. FortiGuard Server List requests to FortiGuard – 1027 UDP / 1031 UDP. AntiSpam and Web Filtering rating lookup requests – 1027 UDP / 1031 UDP. AV/IPS Push / FortiGuard to FortiGate – 9443 UDP. To configure FortiGate to use worldwide servers or only servers located in the USA, run the following command in the CLI: config system fortiguard     set update-server-location [usa/any] end This can also be done under System -> FortiGuard -> FortiGuard Update in the GUI. FortiOS v7.6.x: This can also be done under System -> FortiGuard -> FortiGuard settings. Some debug commands for FortiGuard: diagnose debug reset diagnose debug application update -1 diagnose debug console timestamp enable diagnose debug enable Force FortiGuard update after running debug application update -1: execute update-now The following command can also fix various issues with FortiGuard servers. In this example, 212.48.23.12 is used as the IP for the interface to the ISP router. config system fortiguard     set fortiguard-anycast disable     set protocol udp     set port 8888     set sdns-server-ip 208.91.112.220 173.243.140.53 210.7.96.53     set ddns-server-ip 173.243.138.225     set source-ip 212.48.23.12 end config system dns     set primary 8.8.8.8     set source-ip 212.48.23.12 end Source IP should match the IP address on the WAN port of FortiGate (ISP port), If the Source IP is changed, DNS and FortiGuard settings should be changed as well. Note: Despite changing the FortiGuard settings to anycast disable and configuring the UDP protocol to reach the FortiGuard servers, the connection still fails because the DNS servers will still use the 'DNS over TLS' settings, which is a default setting. In this case, public IPs will be reachable, but DNS resolution fails. Changing the DNS settings to use 'DNS over UDP' will help in resolving the domain names properly. Try to take the following debugs : diagnose debug reset diagnose debug application update -1 diagnose debug console timestamp enable diagnose debug enable execute update-now In this case, the output of the debugs looks like: upd_act_setup_with_action[190]-Trying FDS 173.243.138.68:443 tcp_connect_fds[231]-select() timed out upd_comm_connect_fds[423]-Failed TCP connect Try to use 'interface-select-method' with the 'specify' action and then select the outgoing interface under the FortiGuard configuration as mentioned above. This error may also be related to FortiGate not being able to establish a TCP connection to FortiGuard through the proxy if the proxy is configured on FortiGate. Verify on the proxy that it allows FortiGuard traffic and does not block TCP connections. Additionally, make sure that the proxy is not doing SSL inspection for FortiGuard traffic. Starting from v7.2, initiating FortiGuard services and updates is not locked to the management VDOM. See Allow FortiGuard services and updates to initiate from a traffic VDOM for more information. If the issue persists, contact Fortinet support License errors may be found in two places, as shown below: Dashboard -> Status -> Licenses. System -> Config -> FortiGuard. If DNS issues occur, the following output will be seen in the CLI: Verify DNS settings in FortiWeb under Network -> DNS. Verify the reachability of the DNS IP. If the DNS is private, change to a public DNS and verify the connectivity. If the DNS resolves, check reachability. If there is any firewall, allow the ICMP for FortiWeb. If there is any firewall in place, allow traffic to all destinations for testing purposes. If this fixes the license issue, it indicates the issue is on the firewall policy level. If all checks have been performed successfully, collect the following logs and share them in a ticket created with support: diagnose debug application fds 7 diagnose debug application update 7 diagnose debug console timestamp enable diagnose debug enable execute update-now To disable the debug: diagnose debug disable diagnose debug reset FortiGuard licenses and the communication to FortiGuard are viewable via GUI through System -> FortiGuard as well as through the CLI using the following commands: diagnose autoupdate versions The output is quite long, but it is possible to pipe the output to grep: diagnose autoupdate versions | grep -A6 "IPS Attack Engine" If the communication to FortiGuard is OK, but still the Web Filter is shown as 'disabled' in the 'diag debug rating' output: diagnose debug rating Locale : english Service : Web-filter Status : Disable  <-- Service : Antispam Status : Disable Service : Virus Outbreak Prevention Status : Disable The Web Filter should be licensed: Check the firewall policy. If there is no Web Filter Profile configured in the firewall policy, the web filter will be shown as disabled in the 'diag debug rating' output.  Note: If the error persists after applying changes, try accessing the same page in another browser or a private window, as the browser might be displaying a cached version of the page. Check if there are any upstream devices towards the internet, and make sure to allow UDP port 53 or port 8888 services in the upstream devices and not block these services for the FortiGuard Connectivity. To change the FortiGuard anycast status from disabled to enabled: config system fortiguard     set fortiguard-anycast enable     set fortiguard-anycast-source fortinet end Related articles:  Technical Tip: FortiGuard Flags and Meaning Technical Tip: FortiGuard is not reachable via Anycast default method Technical Tip: FortiGuard Overview and Troubleshooting Technical Tip: FortiGuard Communication problem: web filtering/rating is working but the update fail...