Low throughput over IPsec running on top of VXLAN over IPsec (FortiGate 7.6.6)

Hi all,

I’m looking for advice or similar experience with a layered VPN design on FortiGate.

Topology

1. The core transport network is built on FortiGate devices using VXLAN over IPsec across the Internet (site-to-site).
2. Inside the VXLAN, there is a single L2 broadcast domain, and over this network I establish a second (service) IPsec tunnel between two FortiGate devices (for a specific customer).
3. Behind the service (customer) FortiGates:
   • On one side: a server
   • On the other side: a client PC 

Problem:

Throughput between the client PC and the server is very low (~9 Mbps).

Testing is done using iperf.

What I verified

• If I connect the customer FortiGates directly (without VXLAN) or over the Internet with a single IPsec tunnel, I get around 500 Mbps.
• This seems to rule out:
  • FortiGate performance limits
  • PC/server performance
  • Internet bandwidth issues

Hypothesis

I suspect the issue is related to MTU / MSS / fragmentation due to multiple encapsulation layers:

• inner IPsec
• VXLAN
• outer IPsec

However, I’m not sure:

• where exactly MTU/MSS should be adjusted (core vs edge, physical vs tunnel interfaces)
• what values are typically used in such designs

Questions

• Has anyone implemented a similar IPsec over VXLAN over IPsec design on FortiGate?
• Where is the correct place to tune MTU / TCP MSS in this kind of topology?
• Are there recommended values or best practices for FortiGate in this scenario?
• Any specific debugging steps to confirm MTU/fragmentation issues?

Any insights or real-world experience would be highly appreciated.

Thanks!

FortiGate 

#IPSec

#VXLan