Performance Issues Routing Internet Traffic over site to site VPN tunnel when session count goes up

[MickeyD1]

We have a unique problem that has me a bit stumped.

Our end user laptops route their untrusted internet traffic over client IPSEC VPN tunnels to our Fortigate VM in Azure for threat scanning and then out Microsoft's internet connection.  This has worked fine for a couple years , but increasingly, some internet sites don't like  internet traffic originating from Microsoft's IP's, and won't allow the traffic or require the user to be authenticated.  (Ticketmaster, Reddit and Youtube are a few)

We have a couple physical locations with Fortigate 60Fs that are connected to our Fortigate VM in Azure via Site-to-Site VPN tunnels, and they have lots of spare internet bandwidth at those sites, so we thought we would try routing the internet traffic from our end user VPN tunnels over the site-to-site tunnel and out the internet connection at one of those sites to get around the Microsoft IP issues.  We setup the site-to-site VPN tunnel on the Azure Fortigate to the remote 60F inside an SD-WAN zone, and configured an SD-WAN rule to route the internet traffic over that tunnel and then out to the internet.  This all works just fine with a single client laptop...latency to google DNS is sub 30ms and googling "What is my IP" on the client laptop reveals the IP of the internet connection at the remote site.  All good so far...

The issues start when we increase the number of clients using the remote internet connection from one test machine to all of them.  As the session count on the "VPN to internet policy" on the 60F increases, the clients start to experience high latency and packet loss to the internet...it seems to get worse over time as the session count climbs until it's basically unusable.  Interestingly, while we experience poor performance to the internet, if we ping the 60F's internal interface IP from one of our client laptops (which traverses the same site-to-site tunnel but a different firewall policy) and latency to it is just fine....it appears to be the egress to the internet where the problem starts.  CPU and memory usage on both Fortigate's look fine, and it does not appear to be a bandwidth issue....the bandwidth on the 60F's internet interface is far below it's capacity....it appears to be caused by the number of sessions increasing.    We have also tried the exact same config with a seperate 60F at a different physical site and ran into the same problem. 

Anyone have any ideas of what might be causing this issue?  We disabled anything not needed on the firewall policies involved (AV, webfiltering, IPS, etc) but that didn't improve the situation.  It feels like some kind of session limitation related to NAT'ing more than 1000sessions from the VPN tunnel out to the internet, but not sure what to do about it.

[MickeyD1]

This morning we tried routing our VPN users internet traffic over a different site to site VPN tunnel to physical site that has a Fortigate 300E and out the internet there. Despite the internet bandwidth being considerably smaller at that location, we are not experiencing and of the issues we saw trying to route the traffic out the Fortigate 60F's internet connection. At this point, it feels like the issue was caused by some kind of bug/limitation in the 60F's.