DNS server on loopback not working after upgrade to 5.6.3

[MrSinners]

Hello,

After upgrading from 5.4.5 to 5.6.3, the DNS service on a loopback interface is not working anymore. The traffic is arriving on the VDOM and the debug flow shows the traffic is accepted and sent to the loopback. However the DNS service itself does not reply. A wireshark shows exactly the same behavior, the packet is incoming on the VDOM, but there's no reply back. DNSproxy debugging shows:

udp_receive_request()-2065: vd=8, intf=39, len=32, alen=16, 10.1.0.47:42857=>172.16.10.13 handle_dns_request()-1395: id:0xabb5 pktlen=32, qr=0 req_type=3 get_intf_policy()-876: ifindex=39

A config compare between 5.4.5 and 5.6.3 shows no modified configuration whatsoever related to DNS or the loopback interface. Running the DNS service on a physical interface does seems to be working.

Has anyone by chance run into this before? It seems like a bug or behavior change.

[emnoc · Accepted solution]

I don't know what else to suggest outside re-apply the cfg and make sure  the status is enable. We have ipv6/ipv4 bound on the  loopback and it works 100%

e.g 

AUN1LSE01:~ root# host -6  -T -t  a www.1plus1eq2.com  '2001:db8:101::1'

Using domain server:

Name: 2001:db8:101::1

Address: 2001:db8:101::1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# host -t a www.1plus1eq2.com 192.168.1.1

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

config system interface

    edit "loop0"

        set vdom "mgmt"

        set ip 128.0.0.1 255.0.0.0

        set allowaccess ping https ssh

        set type loopback

        set snmp-index 9

        config ipv6

            set ip6-address 2001:db8:101::1/128

            set ip6-allowaccess ping ssh

        end

    next

end

config system dns-database

    edit "1plus1eq2.com"

        set status enable

        set domain "1plus1eq2.com"

        set type master

        set view public

        set ttl 300

        set authoritative enable

        unset forwarder

        set source-ip 0.0.0.0

        config dns-entry

            edit 1

                set status enable

                set type A

                set ttl 30

                set hostname "www.1plus1eq2.com"

                set ip 1.1.1.1

            next

            edit 2

                set status enable

                set type MX

                set ttl 7

                set hostname "mail.1plus1eq2.com"

                set preference 1000

This an authoritative  DNS for a single zone btw

 v5.6.3 FGT90D

Ken

[emnoc · Accepted solution]

Hey MrSinne guess what you found a bug in v5.6.3 and DNS-server running a dual-stack  on loopback causes th e ipv4 answers to be dropped

e.g ( 19.168.1.1 == phy 128.0.0.1 == loopback )

AUN1LSE01:~ root# host -t mx  1plus1eq2.com 192.168.1.1 

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

1plus1eq2.com mail is handled by 1000 mail.1plus1eq2.com.

AUN1LSE01:~ root# host -t a  1plus1eq2.com 192.168.1.1

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

Host 1plus1eq2.com not found: 3(NXDOMAIN)

AUN1LSE01:~ root# host -t a  www.1plus1eq2.com 192.168.1.1

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# host -t a  www.1plus1eq2.com '2001:db8:101::1'

Using domain server:

Name: 2001:db8:101::1

Address: 2001:db8:101::1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# host -6  -t a  www.1plus1eq2.com '2001:db8:101::1'

Using domain server:

Name: 2001:db8:101::1

Address: 2001:db8:101::1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# 

AUN1LSE01:~ root# 

AUN1LSE01:~ root# host -6  -t a  www.1plus1eq2.com '2001:db8:101::1'

Using domain server:

Name: 2001:db8:101::1

Address: 2001:db8:101::1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# host -t a  www.1plus1eq2.com 192.168.1.1

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

( now loopback ipv4  udp with dans-helpers "default" )

AUN1LSE01:~ root# host -t mx  1plus1eq2.com 128.0.0.1

;; connection timed out; no servers could be reached

AUN1LSE01:~ root# host -t mx  1plus1eq2.com 128.0.0.1

;; connection timed out; no servers could be reached

AUN1LSE01:~ root# 

AUN1LSE01:~ root# host -t a www.1plus1eq2.com 128.0.0.1

;; connection timed out; no servers could be reached

AUN1LSE01:~ root# 

if you run to a  physical address it works 100% of  the time { ipv4/6 } . My lab is primary ipv6  so it was working always against the ipv6 address and I never assumed ipv4 was broken ;(

Strange but again this is FTNT  and v5.6.3 should be avoided ;)

Ken

[MrSinners · Accepted solution]

Hey Ken,

Its interesting that you could reproduce the issue with a dual stack configuration. The odd thing is that in this environment there is no dual stack config, ipv6 is not used. I tried to reproduce the issue on my own FG with 5.6.3 but in this case DNS is responding, so its not clear how the issue is triggered, but it has to be a bug. Even if it would work again with reapplying the config, it doesn't feel like a safe solution for its purpose as it may just as well break again. I had to improvise with a workaround by shutting the loopback, then moving the DNS zone to a windows DNS server and using a VIP to make it reachable on the previous IP of the loopback int. Ill create a ticket and see if its a known issue. Thanks for your help!

[tinyadmin]

Dear all,

I see same problem on FortiOS 6.2.1 on 60D and same workaround: enable DNS-Server on source interface, too.

Thanks for the hint.

Does anyone opened a bug for this issue?

Regards

Tinyadmin

[AdiMizil]

Hi guys, 

Yes, I can confirm that the problem it's still  present in FortiOS 6.2.2. / FGT80E. Thanks for sharing your solution with us. 

I was about to bang my head on the wall after troubleshooting DNS issues with remote access tests using FortiClient. 

DNS server was configured as an internal LAN IP  interface and later I created a DNS server on a  loopback interface. Traffic was flowing from by Forticlient laptop to the FGT80E v 6.2.2, but nothing was coming back.

I have tried using this command on the FGT  :  " diagnose sniffer packet any 'dst port 53' 4 20 a " and in the same time I was capturing traffic on the virtual interface on my laptop with Wireshark .. traffic hitting the FGT, not coming back. 

in Windows command promot  "nslookup yahoo.com 1.1.1.1" was working only using public DNS servers. FRUSTRATING !!  

https://docs.fortinet.com/document/fortigate/6.2.2/cookbook/126629/dns-troubleshooting  - good to know. 

After creating DNS server also on the SSL VPN interface (ssl.root)  and pointing remote access users to the  loopback IP address - DNS service started to respond to requests.   But it's slow on first request. After caching the results it's acceptable. 

Kind regards, 

Adi

[Rollmops]

Hi @ All,

I know it is an old post but I read this one and after 6 hours troubleshooting I found an answer.

Starting from FortiOS version 5.6 onward, the DNS Server behavior was changed to drop DNS requests on interfaces not found in the dns-server table. If a DNS Server is configured on an internal port, for example port1, then FortiGate will resolve only DNS queries coming over port1. If the DNS-server was configured on a loopback interface, but the DNS queries are reaching the FortiGate over a physical interface, in this example port1, then port1 must be added to the DNS-server table:

#config system dns-server edit "DMZ-1" next edit "DMZ-2" next edit "port1" next end

https://kb.fortinet.com/kb/documentLink.do?externalID=FD46186

regards

Andreas

[Leenchen]

Confirmed this is still behavior in 7.0.2
The solution for me was to enable dns on the incoming interface as well.

[ede_pfau]

As stated in one of the first posts, "it's not a but, it's a feature!".

@FTNT: maybe there should be a hint added to the regular documentation (i.e. the Handbook) in the chapter on DNS servers. This would spare users, and eventually FTNT customer support, a lot of troubleshooting.