Fortigates with PPPoE WAN suddenly need TCP MSS 1452 on INSIDE to access HTTPS

[randomcatperson]

Reports started rolling in on Monday, 22 April 2024, at approximately 07:00 AWST (Sunday, 21 April 2024, 23:00 UTC) that people could not access HTTPS websites but could access HTTP websites.

We found that the common denominator was that only customers with a PPPoE dialler for the WAN/Internet service were impacted.

We've seen this in the past and resolved it by adding "set tcp-mss 1452" to the LAN/INSIDE interface/s configuration of our Fortigates.

We provide Internet to many of the customers, but not all - it does not seem to be isolated to a specific network or ISP.
I wasn't involved in the initial information gathering, so I'm unsure if it was limited to specific sites, but I was aware that sites such as office.com (all of Microsoft), www.bom.gov.au, and www.news.com.au were impacted.

The question is, why is it only impacting Fortigates (we have customers with different firewall vendors who didn't have the same issue), & why were they working fine before when the reports came in?

We had done no firmware updates to any of the units, nothing was changed from our side.
Most (if not all) units have automatic Fortiguard updates.

FortiGate

[Toshi_Esumi]

It's probably not because of the destinations but because of the paths to the destinations, which might not pass larger packets than that point of MTU size and just drop them without fragmenting them. 
PPPoE header size: 8, IP header size: 20, and TCP header size:20, total is 48. So your calculation is correct but I would set it at policies toward the internet. Because you wouldn't need to shorten it for LAN to LAN traffic.

If you swap your FGT with a different vendor device by keeping the same ISP/source public IP and the same server destination (means the path is the same), and you still see different behaviors, that might be caused by the difference of default PMTU discovery behavior between the devices.
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Enable-path-MTU-discovery/ta-p/202217

You might want to tweak it at FGT and other vendor devices to see if it changes behaviors.


Toshi

[randomcatperson]

Hi, Toshi,
Thanks a lot for your reply.

We'll look to enable PMTU discovery & also likely look to include MSS adjusting as a default templated configuration to avoid future scenarios.

Our primary concern is that unchanged/static configuration that worked for months (or possibly years in some cases) didn't change, and this issue suddenly manifested across many Fortigate devices across multiple ISPs, but consistently with PPPoE WAN services.

[randomcatperson · Accepted solution]

Looks like this has happened due to a default setting change in Chrome (and Edge is a derivative of Chrome) changing “TLS 1.3 Hybridized Kyber Support” from disabled to enabled which apparently adds extra data in the client hello message:

https://www.reddit.com/r/sysadmin/comments/1carvpd/chrome_124_breaks_tls_handshake/

Chrome 124 was released April 16th (not all machines would have immediately pulled down the update) so this lines up with the timeline of when issues have cropped up at various customer sites sites. Lowering TCP MSS has worked around this issue by allowing packets with extra data in them to go through without packet defragmentation happening. There are many threads recently about this:
https://www.reddit.com/r/chrome/comments/1c8ucus/problems_with_chrome_and_cloudflare/
https://www.reddit.com/r/chrome/comments/1c83js4/firewall_issues_after_latest_update/
https://www.reddit.com/r/sonicwall/comments/1cac4ii/content_filter_blocking_cfs_legitimate_traffic/ (Sonic Wall)
https://www.reddit.com/r/sysadmin/comments/1ca1chq/hello_im_new_to_both_this_subreddit_and_a/

It (TLS 1.3 Hybridized Kyber support) has been known for a while as an issue with Fortigates but only now has the defaults changed in Chrome: https://community.fortinet.com/t5/Support-Forum/SSL-Deep-Inspection-Google-Chrome/td-p/286352/page/2

With regards to Edge: "Also downgrading to 123.X.X versions works" according to: https://www.reddit.com/r/sonicwall/comments/1cac4ii/content_filter_blocking_cfs_legitimate_traffic/.

Edge 124 was released 18-Apr-2024.

[JNDias]

1st workaround is to change the policy Inspection mode from "Flow-based" to "Proxy-based.

Please note that Proxy-based inspection does not allow traffic to be offloaded to NPUs.

How to enable the Proxy-based opinion:

config system settings
 set gui-proxy-inspection enable
end

2nd workaround is to change to MSS on the firewall policy.

[randomcatperson]

As mentioned in the original post—"[we] resolved it by adding "set tcp-mss 1452" to the LAN/INSIDE interface/s configuration of our Fortigates"—the purpose of my post was to try to understand why, seemingly overnight, an entire fleet of FortiGate firewalls started having a problem that hadn't been there before.
We had anecdotal evidence that it wasn't impacting other vendor firewalls - but this has since been shown to be incorrect. The issue impacted other vendors.

Ultimately, per my own "accepted solution", the cause was a Chromium update which impacted anything based on it - including Edge.

[JNDias]

Hi, @randomcatperson Thank you for your insightful reply and thorough investigation; your efforts greatly benefit the community.

Regarding the issue at hand, I've highlighted two potential workarounds that have been successful for other customers. Your confirmation reinforces their efficacy.

I will keep posting on potential further advancements or developments from our end.

[DPadula]

Here is the article wrote based on these finding.

https://community.fortinet.com/t5/tkb/workflowpage/tkb-id/TKB20/article-id/8338

Related articles:

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Web-filter-is-not-blocking-websites-on-Goo...

[randomcatperson]

Thanks for the second "related articles" link - it has some interesting information.
The first link, however, gives "access denied" even though I am signed in.

[DPadula]

Hi Ramdomcatperson
Try this one please: https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Web-pages-not-loading-or-taking-too-...