DNS server on loopback not working after upgrade to 5.6.3

[MrSinners]

Hello,

After upgrading from 5.4.5 to 5.6.3, the DNS service on a loopback interface is not working anymore. The traffic is arriving on the VDOM and the debug flow shows the traffic is accepted and sent to the loopback. However the DNS service itself does not reply. A wireshark shows exactly the same behavior, the packet is incoming on the VDOM, but there's no reply back. DNSproxy debugging shows:

udp_receive_request()-2065: vd=8, intf=39, len=32, alen=16, 10.1.0.47:42857=>172.16.10.13 handle_dns_request()-1395: id:0xabb5 pktlen=32, qr=0 req_type=3 get_intf_policy()-876: ifindex=39

A config compare between 5.4.5 and 5.6.3 shows no modified configuration whatsoever related to DNS or the loopback interface. Running the DNS service on a physical interface does seems to be working.

Has anyone by chance run into this before? It seems like a bug or behavior change.

[emnoc · Accepted solution]

I don't know what else to suggest outside re-apply the cfg and make sure  the status is enable. We have ipv6/ipv4 bound on the  loopback and it works 100%

e.g 

AUN1LSE01:~ root# host -6  -T -t  a www.1plus1eq2.com  '2001:db8:101::1'

Using domain server:

Name: 2001:db8:101::1

Address: 2001:db8:101::1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# host -t a www.1plus1eq2.com 192.168.1.1

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

config system interface

    edit "loop0"

        set vdom "mgmt"

        set ip 128.0.0.1 255.0.0.0

        set allowaccess ping https ssh

        set type loopback

        set snmp-index 9

        config ipv6

            set ip6-address 2001:db8:101::1/128

            set ip6-allowaccess ping ssh

        end

    next

end

config system dns-database

    edit "1plus1eq2.com"

        set status enable

        set domain "1plus1eq2.com"

        set type master

        set view public

        set ttl 300

        set authoritative enable

        unset forwarder

        set source-ip 0.0.0.0

        config dns-entry

            edit 1

                set status enable

                set type A

                set ttl 30

                set hostname "www.1plus1eq2.com"

                set ip 1.1.1.1

            next

            edit 2

                set status enable

                set type MX

                set ttl 7

                set hostname "mail.1plus1eq2.com"

                set preference 1000

This an authoritative  DNS for a single zone btw

 v5.6.3 FGT90D

Ken

[emnoc · Accepted solution]

Hey MrSinne guess what you found a bug in v5.6.3 and DNS-server running a dual-stack  on loopback causes th e ipv4 answers to be dropped

e.g ( 19.168.1.1 == phy 128.0.0.1 == loopback )

AUN1LSE01:~ root# host -t mx  1plus1eq2.com 192.168.1.1 

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

1plus1eq2.com mail is handled by 1000 mail.1plus1eq2.com.

AUN1LSE01:~ root# host -t a  1plus1eq2.com 192.168.1.1

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

Host 1plus1eq2.com not found: 3(NXDOMAIN)

AUN1LSE01:~ root# host -t a  www.1plus1eq2.com 192.168.1.1

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# host -t a  www.1plus1eq2.com '2001:db8:101::1'

Using domain server:

Name: 2001:db8:101::1

Address: 2001:db8:101::1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# host -6  -t a  www.1plus1eq2.com '2001:db8:101::1'

Using domain server:

Name: 2001:db8:101::1

Address: 2001:db8:101::1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# 

AUN1LSE01:~ root# 

AUN1LSE01:~ root# host -6  -t a  www.1plus1eq2.com '2001:db8:101::1'

Using domain server:

Name: 2001:db8:101::1

Address: 2001:db8:101::1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# host -t a  www.1plus1eq2.com 192.168.1.1

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

( now loopback ipv4  udp with dans-helpers "default" )

AUN1LSE01:~ root# host -t mx  1plus1eq2.com 128.0.0.1

;; connection timed out; no servers could be reached

AUN1LSE01:~ root# host -t mx  1plus1eq2.com 128.0.0.1

;; connection timed out; no servers could be reached

AUN1LSE01:~ root# 

AUN1LSE01:~ root# host -t a www.1plus1eq2.com 128.0.0.1

;; connection timed out; no servers could be reached

AUN1LSE01:~ root# 

if you run to a  physical address it works 100% of  the time { ipv4/6 } . My lab is primary ipv6  so it was working always against the ipv6 address and I never assumed ipv4 was broken ;(

Strange but again this is FTNT  and v5.6.3 should be avoided ;)

Ken

[MrSinners · Accepted solution]

Hey Ken,

Its interesting that you could reproduce the issue with a dual stack configuration. The odd thing is that in this environment there is no dual stack config, ipv6 is not used. I tried to reproduce the issue on my own FG with 5.6.3 but in this case DNS is responding, so its not clear how the issue is triggered, but it has to be a bug. Even if it would work again with reapplying the config, it doesn't feel like a safe solution for its purpose as it may just as well break again. I had to improvise with a workaround by shutting the loopback, then moving the DNS zone to a windows DNS server and using a VIP to make it reachable on the previous IP of the loopback int. Ill create a ticket and see if its a known issue. Thanks for your help!

[emnoc]

Yes we had the same issue. Did you reapply the cfg ?  And  issues the  fwpolicy is allowing the traffic ( I know you said yes above )

Ken

[MrSinners]

Hi Ken,

Thanks for your response!

The config has not been reapplied, at least not that im aware off. The FG has been recently added to FMG and had its first install. Should the config be reapplied? How did you manage to solve this? The policy is allowing traffic, but does not show up in the debug flow trace (Allowed by policy-X) as its a local interface.

[emnoc · Accepted solution]

I don't know what else to suggest outside re-apply the cfg and make sure  the status is enable. We have ipv6/ipv4 bound on the  loopback and it works 100%

e.g 

AUN1LSE01:~ root# host -6  -T -t  a www.1plus1eq2.com  '2001:db8:101::1'

Using domain server:

Name: 2001:db8:101::1

Address: 2001:db8:101::1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# host -t a www.1plus1eq2.com 192.168.1.1

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

config system interface

    edit "loop0"

        set vdom "mgmt"

        set ip 128.0.0.1 255.0.0.0

        set allowaccess ping https ssh

        set type loopback

        set snmp-index 9

        config ipv6

            set ip6-address 2001:db8:101::1/128

            set ip6-allowaccess ping ssh

        end

    next

end

config system dns-database

    edit "1plus1eq2.com"

        set status enable

        set domain "1plus1eq2.com"

        set type master

        set view public

        set ttl 300

        set authoritative enable

        unset forwarder

        set source-ip 0.0.0.0

        config dns-entry

            edit 1

                set status enable

                set type A

                set ttl 30

                set hostname "www.1plus1eq2.com"

                set ip 1.1.1.1

            next

            edit 2

                set status enable

                set type MX

                set ttl 7

                set hostname "mail.1plus1eq2.com"

                set preference 1000

This an authoritative  DNS for a single zone btw

 v5.6.3 FGT90D

Ken

[emnoc · Accepted solution]

Hey MrSinne guess what you found a bug in v5.6.3 and DNS-server running a dual-stack  on loopback causes th e ipv4 answers to be dropped

e.g ( 19.168.1.1 == phy 128.0.0.1 == loopback )

AUN1LSE01:~ root# host -t mx  1plus1eq2.com 192.168.1.1 

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

1plus1eq2.com mail is handled by 1000 mail.1plus1eq2.com.

AUN1LSE01:~ root# host -t a  1plus1eq2.com 192.168.1.1

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

Host 1plus1eq2.com not found: 3(NXDOMAIN)

AUN1LSE01:~ root# host -t a  www.1plus1eq2.com 192.168.1.1

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# host -t a  www.1plus1eq2.com '2001:db8:101::1'

Using domain server:

Name: 2001:db8:101::1

Address: 2001:db8:101::1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# host -6  -t a  www.1plus1eq2.com '2001:db8:101::1'

Using domain server:

Name: 2001:db8:101::1

Address: 2001:db8:101::1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# 

AUN1LSE01:~ root# 

AUN1LSE01:~ root# host -6  -t a  www.1plus1eq2.com '2001:db8:101::1'

Using domain server:

Name: 2001:db8:101::1

Address: 2001:db8:101::1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

AUN1LSE01:~ root# host -t a  www.1plus1eq2.com 192.168.1.1

Using domain server:

Name: 192.168.1.1

Address: 192.168.1.1#53

Aliases: 

www.1plus1eq2.com has address 1.1.1.1

( now loopback ipv4  udp with dans-helpers "default" )

AUN1LSE01:~ root# host -t mx  1plus1eq2.com 128.0.0.1

;; connection timed out; no servers could be reached

AUN1LSE01:~ root# host -t mx  1plus1eq2.com 128.0.0.1

;; connection timed out; no servers could be reached

AUN1LSE01:~ root# 

AUN1LSE01:~ root# host -t a www.1plus1eq2.com 128.0.0.1

;; connection timed out; no servers could be reached

AUN1LSE01:~ root# 

if you run to a  physical address it works 100% of  the time { ipv4/6 } . My lab is primary ipv6  so it was working always against the ipv6 address and I never assumed ipv4 was broken ;(

Strange but again this is FTNT  and v5.6.3 should be avoided ;)

Ken

[MrSinners · Accepted solution]

Hey Ken,

Its interesting that you could reproduce the issue with a dual stack configuration. The odd thing is that in this environment there is no dual stack config, ipv6 is not used. I tried to reproduce the issue on my own FG with 5.6.3 but in this case DNS is responding, so its not clear how the issue is triggered, but it has to be a bug. Even if it would work again with reapplying the config, it doesn't feel like a safe solution for its purpose as it may just as well break again. I had to improvise with a workaround by shutting the loopback, then moving the DNS zone to a windows DNS server and using a VIP to make it reachable on the previous IP of the loopback int. Ill create a ticket and see if its a known issue. Thanks for your help!

[emnoc]

Did you try  with  TCP and UDP and with the dns-udp helper removed? I bet this is a bug, I never  tested  ipv4 until you posted this thread but this problem was found in two other  FGTs running v5.6.3

Ken

[MrSinners · Accepted solution]

Hi Emnoc,

For this issue I made a Fortinet ticket. As it turns out, there was a change of behavior in 5.6. There was a security flaw related to DNS in 5.4 which has been fixed. The consequence of this fix is that the DNS service has to be enabled also on the source interface receiving the DNS traffic. This is also why reproducing the issue at home was not successful for me, as DNS was enabled on my LAN also so I couldn't break it. After removing DNS on my local LAN, the DNS service on the Loopback interface indeed stopped responding. It still doesn't make that much sense to enable DNS as a service on the receiving interface, but it is the solution. Thanks for your help :)

[emnoc]

No thank you , I'm too surprise of this behavior. I should test it in v6.0. Fortinet should update the known issues so others who  read release note can be warn.

[ekenny]

Confirmed this is still the behavior in 6.0.x.  Fix was enabling the DNS server on the source interface as MrSinners mentioned.