DoS Sensors Being Triggered by Google

[MontanaMike]

So we're a pretty Google Apps heavy company.  We use a lot of their services and recently it's all been running slow and I think I might have stumbled on what might be causing the problem but I'm unsure as why or how to correct it.

I have DoS policy (below) that, according to the logs, is getting triggered by Google and i'm unsure why.  Port1 is our internet interface and I've got the policy applied to just DNS for some reason that eludes me right now.  I don't want to remove it but I have no problems increasing the thresholds to see what happens.  Also note, that Google services were extremely slow but mostly just for Chrome browser users which was really odd.  If a user used IE or Firefox then everything seemed somewhat normal.  I think Google is pushing something back to Chrome but why it's triggering a DoS sensor (udp_flood) that is running on udp/53 is beyond me.  Attached is also a screenshot of the log message.

Has anybody seen this behavior?  Suggestions?

edit 7 set status enable set interface "port1" set srcaddr "all" set dstaddr "all" set service "DNS"

[TheJaeene]

Hi Mike,

your UDP flood threshold of 300 seems a bit low.

Also your DOS Policy looks kinda odd.

Try a higher threshold for example:

config firewall DoS-policy

edit 7 set interface "port1" set srcaddr "all" set dstaddr "all" set service "DNS"

config anomaly

edit "udp_flood" set status enable set log enable set action block set threshold 1000

Regards,

Jan

[MontanaMike]

Jan,

What makes it odd?  Anyway, I've sort of resolved the issues we were seeing by creating a new object (and group) for all of Googles domain names by FQDN and a new DoS Policy above the old one and increased the udp_flood thresholds until it worked.  Currently it's at 2000 and it seems to be working normally.  I didn't want to increase my general DoS policy udp_flood any higher than 300 because 300 meets our needs.

[emnoc]

Will the source going by that log looks to be your own machine. Since the DST/SRC are wildcards any is this trigger really because of google or do you have a udp_flood event going else where?

What I would do is narrow my DST address for DNS services for the fw-policy & remonitor. I would inspect the source address host.

[TheJaeene]

Hi Mike,

since I also encountered similar DoS Policy oddities I did dive a bit deeper into it.

The following setup was used:

Firewall Policy allowing only Ports which are needed to Browse the Web (53,80,443)

DoS Policy on the internal Interface with all thresholds set to 500 and Block.

Application Control with Monitor all.

Strangely some Android devices and PCs using Chrome are triggering IP and TCP Src Session alarms just after connecting to Google Hosts/Services on Ports that are not allowed by the Firewall policy.

As far as I researched until now, there where no 500+ Sessions established by that hosts.

Odd Odd.....

I have do do some more packet capturing I guess....

Regards,

Jan

[simonpt]

Mike

We've been seeing the same thing.  Looking at your log, it's not DNS (udp/53) -- it's udp/443, which is part of Google's experimental transport layer network protocol called QUIC (Quick UDP Internet Connections).  It's an interesting concept -- using UDP instead of TCP -- but unfortunately the return UDP traffic from Google to the client appears as a UDP flood to the FGT, if there's a lot of it, which there is if the user is watching a YouTube video.

I logged a ticket with Fortinet.  They referred me to this article: http://kb.fortinet.com/kb/documentLink.do?externalID=FD36680.

HTH

Simon

[IanSwett]

Hi, I'm a developer at Google working on QUIC.  QUIC is enabled very widely now and soon other servers are intending to support it, such as Akamai.  

I've heard a few customer complaints directly of users having issues with Fortinet and QUIC, it seems likely it's related to this DoS detection.  Is there something I can do to help, so DoS mode doesn't get triggered by QUIC?

Or is this a case of users who haven't updated to the most recent firmware?

Thanks, Ian

[simonpt]

Hi Ian

I don't believe upgrading to the latest firmware will help.  I've done a quick scan of the latest release notes and the only QUIC reference I could find is a bug fix for 'QUIC support in passive device detection' (bug ID 300577).  That doesn't address the main issues Fortinet users encounter with QUIC which are false DoS detection and failed SSL inspection.  (BTW, that fix is in v5.4.0, which is too new and unproven for anyone to safely run on production FortiGates.)

Probably the best thing Google could do is to remove the 'experimental' label from QUIC.  Fortinet cites this as the reason they don't support QUIC and why we should disable/block it to force Chrome to use traditional protocols.  Here's the latest article on the subject: http://kb.fortinet.com/kb/documentLink.do?externalID=FD36529.

Rgds, Simon

[IanSwett]

Thanks Simon.

As a result of your feedback, we've changed the QUIC page at Chromium to remove the experimental label and note that it does improve users latency in our experiments.

[link]https://www.chromium.org/quic[/link]

[netmin]

Similar to what Jan reported, DoS policies on external _and_ internal interfaces may demonstrate strange behavior at times. While testing different AV software variants (some time ago) a colleague started the 'home network scan' feature (I believe it was Avast) and it was 'freaking out' as it didn't reach any public DNS servers directly (created 3K or so new sessions), because local DNS service is provided to all LANs on a loopback interface in the root vdom.

So that would have been a good test for a DoS policy on the internal interface...set up as follows:

WAN<---root vdom [DNS service on loopback intf]<--->client vdom--->LAN

DoS policies on the WAN interface, source: all, destination: external address ranges already existed.

New DoS policies on the LAN interface ... source: all, destination: all (with higher limits).

Now, internal clients got blocked when reaching the DoS policy limit that was set on the WAN interface (far below the LAN interface policy limit) while accessing the local DNS service that is running on a loopback interface in the root vdom. Not really what we would have expected...

Btw: there also exist other applications using UDP/443, i.e. Cisco AnyConnect clients might use it (DTLS).