Why are the DoT/DoH mechanisms used by Fortigates so abominably slow?

Forum|Forum|3 years ago
April 20, 2023
11 replies
7264 views

So, I'd put in a support ticket for this against the 6.4.x firmware because it definitely behaves there's a bug where VIPs are being applied to everything and were likely ignoring the src-filters, and just got back around to poking at it on the newer 7.0.11 firmware today.

The root of this issue is that since there's no way to set the nameservers being used in the parameters being sent to Windows Native VPN tunnels, if one wishes to use a separate view for hosts "inside" the firewall that includes perhaps zones that don't exist, one has to perform a simple bit of DNS abduction using a VIP. That is, you install a pair of VIPs that only apply to the "dialin" VPN users via a src-filter definition, mapping the IP addresses of the Fortinet's two nameservers to two internal nameservers, drop in a policy that facilitates this hitting the VIPs (also carefully specifying the source addresses used by the VPN clients) and bob's your uncle. Now all DNS queries made by the VPN client which has no choice but to send all traffic to the internal network get redirected to your internal nameservers so your internal scopes resolve correctly even for the dialin people, and all is well.

...except now I see that the VIP (at least on 7.0.x, and for 601E as well as 101E) isn't the actual problem. The problem is that the default Fortinet nameservers are fantastically sluggish and since it's been going on for some weeks now I highly doubt it's some momentary routing problem in any of the five ISPs I've got units going through.

Earlier today, I put together a couple of little pastebombs so I could easily rip the three definitions out and put them right back to test further. With them in place and the default nameserver settings, (as accessed the hard way via `execute ping gmail.com`) nameservice barely functions. From the web GUI under Network->DNS there are two angry red boxes telling me that the nameservers are taking anywhere from 3,000ms to (more often) 10,000-15,000ms to respond (or are simply unreachable), and by replacing those default servers with some widely-known ANYNET addresses, i.e., 8.8.8.8 and 1.1.1.1 the problem continues as long as DNS over TLS (853/tcp) is selected. Flip the DNS server setting over to 'Selected' and leave the same nameservers in place, enable plain DNS, and everything goes just fine. Instantly times of less than 1,000ms are reported (and for 8.8.8.8 it's usually under 50ms). Just to look at things from a second angle, you can open up a console to the device and start running `execute ping somedomain.foo` (obviously using a domain that won't be in the cache) and it'll very likely timeout saying "Unable to resolve hostname" while DoT is being used.

Using knot on any given Linux workstation I can make DoT queries to 8.8.8.8 (and other servers not 96.45.45.45/96.45.46.46) and get responses in under 100-200ms just fine, so I am disinclined to believe this is a mistake or false positive limited to the web GUI. Whatever implementation is being used by the FortiGates seems to be just horrendously slow and needs to be fixed. I can't see how this is not preventing devices from getting their Geo-IP and various signature updates in a timely fashion.

FortiGate

gfleming

Staff

The DNS servers configured in the FortiGate's DNS settings have nothing to do with the response times of your internal DNS servers. Unless, of course you are using the FortiGate as a DNS forwarder on your internal DNS servers. Are you?

PaulRobertsAuthor

New Member

This has nothing, absolutely nothing to do with our internal DNS servers.

It has to do with the fact that something is profoundly screwed up with the DoT mechanism used by Fortigates, resulting in some absolutely ridiculous latency, even when and especially when using the default settings for the devices. Although I originally put in a support ticket because I thought something in the internally generated rules caused by the fairly straightforward VIPs I'd created (and I'd never had real cause to think the reported latency was anything other than a UI mistake--not the actual value), I literally disproved the idea that it might have been a VIP causing the problems earlier today because the problem remains when the VIP isn't present, and also happens to the 100+ other devices (all of which were using the default settings for DNS) which never had any such VIP in the first place.

This impacts not only the Fortigate appliances' ability to fetch their own updates (because it can't find the update server if if the DNS times out), but in some cases (since the same issue appears to impact it's querying of the DNS filter rating servers) clients which are simply making their own queries which are being filtered by the Fortigate appliances. Devices which can't even do their basic network functionality DNS lookup will simply decide they have no internet access (because they basically don't when the firewall is eating their DNS queries). The workstation I was doing the checking with knot from is not subject to any sort of DNS filtering by the Fortinet. The only reason this hasn't been excruciatingly obvious from day one is that the devices are caching DNS responses, but you can literally open a console, bang out `execute ping somenewhostnamenoonehaslookedupyet` and watch it fail the majority of the time.

Turning that junk off fixed quite a number of issues. I can literally (because I just did so) turn 53/tcp off and 853/tcp back on and watch the reported latency from 96.45.45.45/96.45.46.46 go from 20ms right back to >10,000ms. This problem exists in the Fortigate's default configuration.

The fix?

config system dns

set protocol cleartext

end

...and just like that, reported latency drops back to a perfectly acceptable <200ms range.

...which has to be done from a console because the web GUI literally won't let you disable DoT in favor of plain 53/udp as long as "Use Fortigate Servers" is lit.

Now... If you can't be bothered to read a post carefully enough to actually understand the problem, stop wasting other people's time.

gfleming

Staff

I read your entire post and I asked you a follow up question. If you're going to accuse me of wasting your time and not reading your post, the best you could do is answer the question I posed in my reply as all I am doing is trying to help you by better understanding how your network is configured.

You are complaining about FortiGate DNS response time being sluggish (local-out traffic) but you are also talking about VPN clients and using VIPs to redirect DNS lookups to your internal DNS servers (forwarded traffic) which has no bearing on local-out traffic from the FortiGate. I'm trying to understand the relationship between both of those things in your network and your configuration. As normally, there should be no relationship. Unless, as I asked already, you are using the FortiGate as your DNS forwarder on your internal DNS servers.

Just have some respect for the people that are here helping you on their free time while we're at it. Thanks.

PaulRobertsAuthor

New Member

That's going to be a hard no. While I understand your bosses have you guys under orders to respond to every post within 24 hours, I'm pretty sure when they came up with that what they had in mind was not low-effort garbage posts. The only "effort" you seem to be making is in bending over backwards to deny that there is ever any kind of problem. You literally zeroed in on the a thing that has no actual bearing on the problem and ignored everything else. As to your question about whether or not we're using the Fortigate as a DNS forwarder, that's not even a type of query I have attempted to test yet and I've already narrowed the problem down to just one thing so I will be ignoring that question.

I would think that by now you should be able to figure out, based on my rather explicit response, that the problem is not the "response time" of the Fortigate, but how long it takes to resolve anything in the first place and that this seems to be causing other problems elsewhere in addition to the problems it causes for the device. I have in the past noticed parts of this mess when I've had to troubleshoot why a device didn't have anything but an ancient copy of (among other things) the GeoIP database, because it couldn't resolve the update host. That did lead me to discover that the logs don't accurately report why hosts were being blocked by the GeoIP filter. The logs show what country an address belongs to at the time you view the logs, not at the time the decision to block was made, and as much as I hate to admit it, Florida is still a part of the US so yeah seeing the Fortigate logging that it blocked Florida is going to make me look rather carefully at things. I would say "efficiency over accuracy" but the route someone took for that code was neither accurate nor efficient. ...but I have no reason to bother even trying to report that as a bug when it'll just be hand-waved away. I'm perfectly happy to let detail-oriented lawyers working for someone else discover that the hard way (even though it's fairly unlikely to come up it does represent what lawyers refer to as "doubt" as to the accuracy of the logs).

I specifically mentioned in the very first post a section of the GUI whose sole function appears to be to report the time it takes the Fortigate itself to get a response to the queries that (in the default configuration!) it is making to Fortigate's DNS servers. I've been seeing that indicating a problem for weeks, and the only reason I was ever willing to assume that it didn't reflect reality was that I've already seen at least one other case where part of a Fortinet's reporting emits random numbers instead of the correct values, and no one seemed to actually care about that. You personally discounted that one as being a real problem, by the way.

I have illustrated how this ridiculous latency negatively impacts the functionality of both the device itself, and may very well be impacting any client systems relying on the appliance for connectivity under a configuration that simply involves using functionality that is central to why people pay money for the things--namely the filtering of "bad stuff" via scrutiny of DNS queries. I've seen the same bizarrely high values showing up next to the DNS filtering servers at well (although it's not something I've seen in the last 48 hours). It makes perfect sense that if it can't go check on a new domain it might very well just not send a response to the client system that made the recursive query. Being that the reported latency is wildly variable, I wouldn't be surprised to find that it's simply slightly less obvious when applied to the DNS filter rating servers (and as I write this I have the UI open with the problematic feature enabled just to watch it in realtime and one of the DNS filter rating servers just jumped to 14,540ms).

I have explained that this absurdly high latency remains no matter whose nameservers a Fortinet is pointed at. I have tested the same type of queries using an ordinary linux machine from an internal IP address that isn't subject to DNS filtering to exclude the possibility that our network path is somehow broken. Using knot I was able to perform 100 different queries to 8.8.8.8/1.1.1.1 that all got responses in under 200ms, even going through an impacted Fortinet 601E (but not asking it to do any of the work beyond just toss the packets). Personally, I think getting to the point where everything involved is Fortinet's purview is as far as I really should have needed to go with this, but I like to be thorough because it often leads to other discoveries.

I have now explicitly indicated the single configuration setting necessary to turn the bad behavior on and off at will. With DoT or DoH enabled, both 601E and 101F models exhibit the same problem. Re-enable plaintext and disable DoT/DoH and the problem goes away. I could presume that it might be something screwy with 96.45.45.45/96.45.46.46 because I can't seem to get knot to give me anything but a TLS handshake failure (even when specifying the TLS hostname shown in the returned cert), but the Fortigate exhibits this symptom when pointed at other people's nameservers so all I can show there is that maybe both the Fortigate and the 95.x nameservers are wonky. However, none of that is my responsibility to fix. Your DoT implementation appears to be hot garbage. How is taking more than ten seconds for a simple query considered acceptable performance?

It's not our ISP (because more than ten different ISPs and even more ASNs are involved) causing the problem with some weird routing. It's highly unlikely that it's a regional routing problem because we've got it showing up in locations scattered throughout most of the eastern US (which would make it a really big region and DownDetector would be getting major traffic). It's not just one model because this is happening on at least two different models. It's not our configuration decisions because again, these are the default settings that came in with the 7.0.11 firmware and the only thing I'm touching via a local-in policy is restricting access to the management services ports (in part because in 2023 some vendors have still not gotten the message about unused features being enabled by default). It's not the VIP configuration because this is happening after the VIP and policy were removed and is also happening on over 100 devices which never had the VIP in the first place, ergo there is no possible "relationship" between these things, and you are ignoring that I've said so both implicitly and explicitly.

Let me just simplify all this for you...

It's not DNS.
It was never DNS.
It's the Fortigate.

Someone probably needs to look into why this issue has remained through multiple revisions of the firmware because the web UI has literally been reporting this straight into people's eyes for who knows how long, but some folks might now be drawing their own conclusions about the why of that.

RachelGomez123

New Member

It's difficult to determine the exact reason why the DoT/DoH mechanisms used by Fortigates may be slow without more information about the specific setup and configurations being used. However, there are several potential factors that could be contributing to slow performance:

Network latency: DoT/DoH traffic is encrypted and requires additional processing compared to unencrypted traffic, which can increase network latency. Depending on the network topology and the distance between the Fortigate and the DNS server, latency could be a contributing factor to slow performance.

DNS server performance: The speed of DoT/DoH resolution can also depend on the performance of the DNS server being used. If the DNS server is overloaded or experiencing issues, this could slow down resolution times.

Fortigate configuration: The Fortigate device itself may also be configured in a way that is contributing to slow DoT/DoH performance. For example, if the Fortigate is configured to use an inefficient encryption algorithm, this could impact performance.

Regards,

Rachel Gomez

Sign up

Login with SSO

Login to the community

Login with SSO