Technical Tip: FortiGate Troubleshooting DNS commands

[Jonathan_Body_FTNT]

Description

This article describes the different debug information that can be collected from the CLI of the FortiGate, prior to FortiOS 3.0 MR6 and since MR7.

The FortiGate uses DNS for several of its functions, including communication with FortiGuard, sending email alerts, and URL blocking (using FQDN).

A DNS query is updated every time that a DNS traffic is passing through FortiGate.

Scope

FortiGate.

Solution

Before FortiOS 3.0 MR6, DNS troubleshooting was performed via the haproxy command :

diag debug haproxy dump

DNS proxy cache dump:
Cached [0x8c15c18]: Questions in query:
QR: update.fortiguard.net.
Cached [0x8c156d8]: Questions in query:
QR: www.fortinet.com.
---End of DNS proxy cache dump---

diag debug haproxy fqdndump

FQDN entry dump:

www.fortinet.com: ID(107) REF(1) EXPIRE(1224623673, ttl 3600) VD(0, ref 1)
---End of FQDN entry dump (total 1)--

Since MR7, a dnsproxy debug command is available on the FortiGate and can be queried with the following variants:

diag test application dnsproxy ?

1. Clear dns cache

2. Show stats

3. Dump DNS setting

4. Reload FQDN

5. Requery FQDN

6. Dump FQDN

Below are examples of what the output should show when enabled.

• To clear the DNS cache:

diag test application dnsproxy 1  -  DNS statistics :
diag test application dnsproxy 2
DNS_CACHE: alloc=4
DNS UDP: req=13, res=4, fwd=4, hits=9, alloc=0 cur=4
FQDN: alloc=1
DNS TCP: req=0, alloc=0  -  DNS settings:

diag test application dnsproxy 3
Management: vd=root, id=0, master=1:1
DNS server 0: x.x.x.x:53
DNS server 1: x.x.x.x:53
DNS server 2: x.x.x.x:53
DNS server 3: x.x.x.x:53
DNS_CACHE: hash-size=2048, ttl=1800, min-ttl=60, max-num=5000
DNS FD: mgmt_s=7, mgmt_c=8, mgmt_c2=9, ha_s=5 ha_c=6 unix_s=10, unix_nb_s=11, unix_nc_s=12
relay dmz in root: fd=13
relay internal in root: fd=14
dns_out_sock=8, mgmt_recreate_sock=0 mgmt_switched=0, jiffies=91669
FQDN: hash_size=1024, current_query=1024
DNS FD: tcp_s=15

diag test application dnsproxy 6

vfid=0 name=www.fortinet.com: timer running, min_ttl=43200:43129, slot=-1, num=1 X.X.X.X

This will simply reload and re-query the FQDN:

diag test application dnsproxy 4
diag test application dnsproxy 5

There is also another variant that can be used to test and query a specific URL and follow the DNS lookup request on the FortiGate, this can be done by enabling the following debug and performing an ICMP test, the example uses www.fortinet.com as follows:

diag debug application dnsproxy -1

execute ping www.fortinet.com
unix_receive_request()-521
handle_dns_request()-378: pktlen=32, qr=0
dns_forward_request()-303
dns_forward_request()-316: Send 32B to x.y.z.t:53 via fd=8
mgmt_receive_response()-504
mgmt_receive_response()-510: len=116, addr=x.y.z.t:53, addrlen=16
handle_dns_response()-423
dns_set_min_ttl()-153: QR: www.fortinet.com
dns_set_min_ttl()-161: Offset of 1st RR: 32 Number of RR's: 5
dns_set_min_ttl()-171: RR TTL: 43200
dns_set_min_ttl()-171: RR TTL: 277
dns_set_min_ttl()-171: RR TTL: 277
dns_set_min_ttl()-171: RR TTL: 277
dns_set_min_ttl()-171: RR TTL: 277
dns_cache_response()-229: Min ttl = 277
dns_forward_response()-330
dns_forward_response()-334: Send 32B via fd=10

In version 5.4 of FortiGate, more information was added:

diag test application dnsproxy ?

1. Clear DNS cache                                           

2. Show stats                                                

3. Dump DNS setting                                          

4. Reload FQDN                                               

5. Requery FQDN                                              

6. Dump FQDN                                                 

7. Dump DNS cache                                            

8. Dump DNS DB                                               

9. Reload DNS DB                                             

10. Dump secure DNS policy/profile                           

11. Dump Botnet domain                                       

12. Reload Secure DNS setting                                

13. Show Hostname cache                                      

14. Clear Hostname cache                                     

15. DNS debug bit mask

For Version 6.0 of FortiGate, more information was added:

diag test application dnsproxy ?

1. Clear DNS cache                                           

2. Show stats                                                

3. Dump DNS setting                                          

4. Reload FQDN                                               

5. Requery FQDN                                              

6. Dump FQDN                                                 

7. Dump DNS cache                                            

8. Dump DNS DB                                               

9. Reload DNS DB                                             

10. Dump secure DNS policy/profile                           

11. Dump Botnet domain                                       

12. Reload Secure DNS setting                                

13. Show Hostname cache                                      

14. Clear Hostname cache 

15. Show SDNS rating cache    

16. Clear SDNS rating cache                                

17. DNS debug bit mask

Note: Include any of these debugs in the Support ticket raised when trying to resolve a DNS issue on the FortiGate. Fortinet support will advise further should other debugging be required.

For versions 7.0, 7.2, 7.4, and 7.6 FortiGate, more information was added:

diag test application dnsproxy ?

1. Clear DNS cache
2. Show stats
3. Dump DNS setting
4. Reload FQDN
5. Requery FQDN
6. Dump FQDN
7. Dump DNS cache
8. Dump DNS DB
9. Reload DNS DB
10. Dump secure DNS policy/profile
11. Dump Botnet domain
12. Reload Secure DNS setting
13. Show Hostname cache
14. Clear Hostname cache
15. Show SDNS rating cache
16. Clear SDNS rating cache
17. DNS debug bit mask
18. DNS debug obj mem
99. Restart dnsproxy worker

Scenario:

• Users can load the homepage of a website however after trying to log in or to access a different link on the website, the page keeps loading.
• The firewall policy for this traffic uses the FQDN address object for the destination only.
• Checking the Forward Traffic Logs page under the Log&Report menu shows traffic is allowed to the FQDN however after some time traffic events with the result client-rst, server-rst or timeout can be found.
• There is a good chance this problem is a product of that website sub-directory pages are hosted on ip addresses that belong to other domains or do not resolve that website hostname. An example of such a scenario is hosting the website services over Content Delivery Network - CDN - servers.
• One way to find out is to check other traffic from the same source IP at the same time the access to sub-pages of that website failed or did not load. If there are destinations that are being Denied at the same time that did not have any name resolution showing on the logs or showing they belong to CDN vendors such as AKAMAI, Fastly, or AWS this indicates this traffic is meeting this scenario.
• A solution would be to expand the allowed destination on the firewall policy matching this traffic to include the additional IP addresses or hostnames required to access the failed pages.

FortiOS 7.0.0 and above, 'diag test application dnsproxy 15' will not show SDNS cache results because dnsfilter in flow mode is handled by the IPS engine. If changing to proxy mode, the results will be displayed.

From v7.6.0, the DNS system database config has the option to configure the 'source-ip-interface' to overcome the challenges of dynamic IP address change.

Refer to the below doc for more information:

Implement the interface name as the source IP address in RADIUS, LDAP, and DNS configurations

Refer to the below document:
DNS filter handled by IPS engine in flow mode

Related article:

Technical Tip: How to perform a hostname to IP address resolution on a FortiGate unit