FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
This article describes when Google DNS appears in logs with error Deny: DNS error. By design, FortiGate looks for invalid/failed DNS traffic and will mark it as action=dns or in the GUI as "Deny: DNS error".

This happens if the DNS query is not successful to return any other status than NOERROR.This is an expected behavior in version 5.4/5.6 where the firewall logs any invalid DNS traffic.

The firewall action itself is allow/pass, but the bad reply from the server is not forwarded back to the requesting client thus showing the "Deny: DNS Error" message.
Invalid DNS traffic is i.e. UDP packets on port 53 that are not DNS traffic, packets that are over sized, bad checksum etc. This error message should also be seen for DNS traffic that causes an error, i.e.when the request to or reply from a DNS server is invalid.

Identify cause in logs which appears Google DNS with error Deny: DNS error,  applies for devices that DNS resolution is made for public google DNS
1) Using FortiGate packet capture tool (for mid-high models) or (for low models) capture SNMP port 53, please refer to the related article.

2) Capture port 53 and check at the same time
FortiGate GUI logs, when a log "Google DNS with error Deny: DNS error" appears stop capture.

3) Convert or open .pcap file, then find by hour/port/DNS flags or IP the packet that match with GUI log:

4) The following are examples of cause log "Google DNS with error Deny: DNS error"

Example 1

LAN PC request resolution for domain
FortiGate bypass this request to Google DNS.

Google DNS answer with "no such name", because it can't find that domain in its database.

FortiGate GUI log will appear as "Deny: DNS error"

Example 2:

Same behavior for domain c4locator

Note: This is not a FortiGate failure, some applications contains domains that are not published in public DNS

Related Articles

Technical Note: How to import 'diagnose sniffer packet' data to WireShark - Ethereal application