Hello,
After upgrading from 5.4.5 to 5.6.3, the DNS service on a loopback interface is not working anymore. The traffic is arriving on the VDOM and the debug flow shows the traffic is accepted and sent to the loopback. However the DNS service itself does not reply. A wireshark shows exactly the same behavior, the packet is incoming on the VDOM, but there's no reply back. DNSproxy debugging shows:
udp_receive_request()-2065: vd=8, intf=39, len=32, alen=16, 10.1.0.47:42857=>172.16.10.13 handle_dns_request()-1395: id:0xabb5 pktlen=32, qr=0 req_type=3 get_intf_policy()-876: ifindex=39
A config compare between 5.4.5 and 5.6.3 shows no modified configuration whatsoever related to DNS or the loopback interface. Running the DNS service on a physical interface does seems to be working.
Has anyone by chance run into this before? It seems like a bug or behavior change.
Solved! Go to Solution.
Nominating a forum post submits a request to create a new Knowledge Article based on the forum post topic. Please ensure your nomination includes a solution within the reply.
I don't know what else to suggest outside re-apply the cfg and make sure the status is enable. We have ipv6/ipv4 bound on the loopback and it works 100%
e.g
AUN1LSE01:~ root# host -6 -T -t a www.1plus1eq2.com '2001:db8:101::1'
Using domain server:
Name: 2001:db8:101::1
Address: 2001:db8:101::1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
AUN1LSE01:~ root# host -t a www.1plus1eq2.com 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
config system interface
edit "loop0"
set vdom "mgmt"
set ip 128.0.0.1 255.0.0.0
set allowaccess ping https ssh
set type loopback
set snmp-index 9
config ipv6
set ip6-address 2001:db8:101::1/128
set ip6-allowaccess ping ssh
end
next
end
config system dns-database
edit "1plus1eq2.com"
set status enable
set domain "1plus1eq2.com"
set type master
set view public
set ttl 300
set authoritative enable
unset forwarder
set source-ip 0.0.0.0
config dns-entry
edit 1
set status enable
set type A
set ttl 30
set hostname "www.1plus1eq2.com"
set ip 1.1.1.1
next
edit 2
set status enable
set type MX
set ttl 7
set hostname "mail.1plus1eq2.com"
set preference 1000
This an authoritative DNS for a single zone btw
v5.6.3 FGT90D
Ken
PCNSE
NSE
StrongSwan
Hey MrSinne guess what you found a bug in v5.6.3 and DNS-server running a dual-stack on loopback causes th e ipv4 answers to be dropped
e.g ( 19.168.1.1 == phy 128.0.0.1 == loopback )
AUN1LSE01:~ root# host -t mx 1plus1eq2.com 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:
1plus1eq2.com mail is handled by 1000 mail.1plus1eq2.com.
AUN1LSE01:~ root# host -t a 1plus1eq2.com 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:
Host 1plus1eq2.com not found: 3(NXDOMAIN)
AUN1LSE01:~ root# host -t a www.1plus1eq2.com 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
AUN1LSE01:~ root# host -t a www.1plus1eq2.com '2001:db8:101::1'
Using domain server:
Name: 2001:db8:101::1
Address: 2001:db8:101::1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
AUN1LSE01:~ root# host -6 -t a www.1plus1eq2.com '2001:db8:101::1'
Using domain server:
Name: 2001:db8:101::1
Address: 2001:db8:101::1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
AUN1LSE01:~ root#
AUN1LSE01:~ root#
AUN1LSE01:~ root# host -6 -t a www.1plus1eq2.com '2001:db8:101::1'
Using domain server:
Name: 2001:db8:101::1
Address: 2001:db8:101::1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
AUN1LSE01:~ root# host -t a www.1plus1eq2.com 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
( now loopback ipv4 udp with dans-helpers "default" )
AUN1LSE01:~ root# host -t mx 1plus1eq2.com 128.0.0.1
;; connection timed out; no servers could be reached
AUN1LSE01:~ root# host -t mx 1plus1eq2.com 128.0.0.1
;; connection timed out; no servers could be reached
AUN1LSE01:~ root#
AUN1LSE01:~ root# host -t a www.1plus1eq2.com 128.0.0.1
;; connection timed out; no servers could be reached
AUN1LSE01:~ root#
if you run to a physical address it works 100% of the time { ipv4/6 } . My lab is primary ipv6 so it was working always against the ipv6 address and I never assumed ipv4 was broken ;(
Strange but again this is FTNT and v5.6.3 should be avoided ;)
Ken
PCNSE
NSE
StrongSwan
Hey Ken,
Its interesting that you could reproduce the issue with a dual stack configuration. The odd thing is that in this environment there is no dual stack config, ipv6 is not used. I tried to reproduce the issue on my own FG with 5.6.3 but in this case DNS is responding, so its not clear how the issue is triggered, but it has to be a bug. Even if it would work again with reapplying the config, it doesn't feel like a safe solution for its purpose as it may just as well break again. I had to improvise with a workaround by shutting the loopback, then moving the DNS zone to a windows DNS server and using a VIP to make it reachable on the previous IP of the loopback int. Ill create a ticket and see if its a known issue. Thanks for your help!
Hi Emnoc,
For this issue I made a Fortinet ticket. As it turns out, there was a change of behavior in 5.6. There was a security flaw related to DNS in 5.4 which has been fixed. The consequence of this fix is that the DNS service has to be enabled also on the source interface receiving the DNS traffic. This is also why reproducing the issue at home was not successful for me, as DNS was enabled on my LAN also so I couldn't break it. After removing DNS on my local LAN, the DNS service on the Loopback interface indeed stopped responding. It still doesn't make that much sense to enable DNS as a service on the receiving interface, but it is the solution. Thanks for your help :)
Yes we had the same issue. Did you reapply the cfg ? And issues the fwpolicy is allowing the traffic ( I know you said yes above )
Ken
PCNSE
NSE
StrongSwan
Hi Ken,
Thanks for your response!
The config has not been reapplied, at least not that im aware off. The FG has been recently added to FMG and had its first install. Should the config be reapplied? How did you manage to solve this? The policy is allowing traffic, but does not show up in the debug flow trace (Allowed by policy-X) as its a local interface.
I don't know what else to suggest outside re-apply the cfg and make sure the status is enable. We have ipv6/ipv4 bound on the loopback and it works 100%
e.g
AUN1LSE01:~ root# host -6 -T -t a www.1plus1eq2.com '2001:db8:101::1'
Using domain server:
Name: 2001:db8:101::1
Address: 2001:db8:101::1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
AUN1LSE01:~ root# host -t a www.1plus1eq2.com 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
config system interface
edit "loop0"
set vdom "mgmt"
set ip 128.0.0.1 255.0.0.0
set allowaccess ping https ssh
set type loopback
set snmp-index 9
config ipv6
set ip6-address 2001:db8:101::1/128
set ip6-allowaccess ping ssh
end
next
end
config system dns-database
edit "1plus1eq2.com"
set status enable
set domain "1plus1eq2.com"
set type master
set view public
set ttl 300
set authoritative enable
unset forwarder
set source-ip 0.0.0.0
config dns-entry
edit 1
set status enable
set type A
set ttl 30
set hostname "www.1plus1eq2.com"
set ip 1.1.1.1
next
edit 2
set status enable
set type MX
set ttl 7
set hostname "mail.1plus1eq2.com"
set preference 1000
This an authoritative DNS for a single zone btw
v5.6.3 FGT90D
Ken
PCNSE
NSE
StrongSwan
Hey MrSinne guess what you found a bug in v5.6.3 and DNS-server running a dual-stack on loopback causes th e ipv4 answers to be dropped
e.g ( 19.168.1.1 == phy 128.0.0.1 == loopback )
AUN1LSE01:~ root# host -t mx 1plus1eq2.com 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:
1plus1eq2.com mail is handled by 1000 mail.1plus1eq2.com.
AUN1LSE01:~ root# host -t a 1plus1eq2.com 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:
Host 1plus1eq2.com not found: 3(NXDOMAIN)
AUN1LSE01:~ root# host -t a www.1plus1eq2.com 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
AUN1LSE01:~ root# host -t a www.1plus1eq2.com '2001:db8:101::1'
Using domain server:
Name: 2001:db8:101::1
Address: 2001:db8:101::1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
AUN1LSE01:~ root# host -6 -t a www.1plus1eq2.com '2001:db8:101::1'
Using domain server:
Name: 2001:db8:101::1
Address: 2001:db8:101::1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
AUN1LSE01:~ root#
AUN1LSE01:~ root#
AUN1LSE01:~ root# host -6 -t a www.1plus1eq2.com '2001:db8:101::1'
Using domain server:
Name: 2001:db8:101::1
Address: 2001:db8:101::1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
AUN1LSE01:~ root# host -t a www.1plus1eq2.com 192.168.1.1
Using domain server:
Name: 192.168.1.1
Address: 192.168.1.1#53
Aliases:
www.1plus1eq2.com has address 1.1.1.1
( now loopback ipv4 udp with dans-helpers "default" )
AUN1LSE01:~ root# host -t mx 1plus1eq2.com 128.0.0.1
;; connection timed out; no servers could be reached
AUN1LSE01:~ root# host -t mx 1plus1eq2.com 128.0.0.1
;; connection timed out; no servers could be reached
AUN1LSE01:~ root#
AUN1LSE01:~ root# host -t a www.1plus1eq2.com 128.0.0.1
;; connection timed out; no servers could be reached
AUN1LSE01:~ root#
if you run to a physical address it works 100% of the time { ipv4/6 } . My lab is primary ipv6 so it was working always against the ipv6 address and I never assumed ipv4 was broken ;(
Strange but again this is FTNT and v5.6.3 should be avoided ;)
Ken
PCNSE
NSE
StrongSwan
Hey Ken,
Its interesting that you could reproduce the issue with a dual stack configuration. The odd thing is that in this environment there is no dual stack config, ipv6 is not used. I tried to reproduce the issue on my own FG with 5.6.3 but in this case DNS is responding, so its not clear how the issue is triggered, but it has to be a bug. Even if it would work again with reapplying the config, it doesn't feel like a safe solution for its purpose as it may just as well break again. I had to improvise with a workaround by shutting the loopback, then moving the DNS zone to a windows DNS server and using a VIP to make it reachable on the previous IP of the loopback int. Ill create a ticket and see if its a known issue. Thanks for your help!
Did you try with TCP and UDP and with the dns-udp helper removed? I bet this is a bug, I never tested ipv4 until you posted this thread but this problem was found in two other FGTs running v5.6.3
Ken
PCNSE
NSE
StrongSwan
Hi Emnoc,
For this issue I made a Fortinet ticket. As it turns out, there was a change of behavior in 5.6. There was a security flaw related to DNS in 5.4 which has been fixed. The consequence of this fix is that the DNS service has to be enabled also on the source interface receiving the DNS traffic. This is also why reproducing the issue at home was not successful for me, as DNS was enabled on my LAN also so I couldn't break it. After removing DNS on my local LAN, the DNS service on the Loopback interface indeed stopped responding. It still doesn't make that much sense to enable DNS as a service on the receiving interface, but it is the solution. Thanks for your help :)
No thank you , I'm too surprise of this behavior. I should test it in v6.0. Fortinet should update the known issues so others who read release note can be warn.
PCNSE
NSE
StrongSwan
Confirmed this is still the behavior in 6.0.x. Fix was enabling the DNS server on the source interface as MrSinners mentioned.
Select Forum Responses to become Knowledge Articles!
Select the “Nominate to Knowledge Base” button to recommend a forum post to become a knowledge article.
User | Count |
---|---|
1698 | |
1092 | |
752 | |
446 | |
228 |
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.
Copyright 2024 Fortinet, Inc. All Rights Reserved.