Sharing WAN between VDOMs & IPSec routing problem

Dear Fortinet pros,

I’ve been thinking for a while about the best way to share a single ISP across multiple customers on a FortiGate cluster (100F), with maximum performance as the main goal. My first approach—using EMAC VLAN interfaces—ran into a problem: IPsec tunnels aren’t hardware-accelerated in that setup and only achieve about 5–10% of the possible bandwidth. Support advised using standard VLAN interfaces to enable hardware acceleration.

So I’ve switched to the following design:

Basics: The FortiGate cluster is connected to the data center network via an LACP trunk (x1 + x2 interfaces). There are no FortiSwitches. All network interfaces are placed under an 802.3ad aggregate named “DC-Network.” The goal is to use only VLAN interfaces there.

The IP addresses below are masked: all 172.x.x.x/x addresses represent public subnets, and all 10.x.x.x/x addresses are private ranges. There’s a public IP block 172.19.1.0/27 that I want to split into smaller /29 subnets for individual customers. This public block is connected to the internet through an ISP router over a public transfer network 172.19.19.0/29.

Concrete setup: Via the root VDOM, I connect the ISP’s public transfer network (ROOT-PUBT01). Using the npu0_vlink interfaces and private transfer networks (ROOT-CustX <-> CustX-WAN), each customer VDOM is then connected to its assigned public /29 (CustX-INET). Within each customer VDOM, there are additional private VLANs (CustX-APP) for the servers. The full layout is shown in the network diagram.

General: Does this setup make sense as-is, or do you have suggestions to improve it?

Specific: Right now I have the feeling that routing from CustB-APP <-> CustB-INET <-> CustB-WAN doesn’t actually follow that path; instead it seems to always take the shortcut CustB-APP <-> CustB-WAN.

For example, I need to create the policy for inbound traffic via a Virtual IP on the CustB-WAN interface, not on CustB-INET. Also, an IPsec tunnel that terminates on the CustB-INET interface doesn’t really work. Interestingly, the tunnel comes up and shows as active, but packets originating from the CustB-APP network never find their way into the tunnel.

A packet trace (debug flow) shows the issue as: no route to <IP of remote-gw>, drop (see screenshot):

So I have to change the interface of the phase1-interface to CustB-WAN and configured the local-gw to the public IP address:

config vpn ipsec phase1-interface     edit "S2S-Tunnel"         set interface "CustB-WAN"         set local-gw 172.19.1.9         set authmethod signature         set net-device disable         set proposal aes128-sha256 aes256-sha256 aes128-sha1 aes256-sha1         set remote-gw 172.16.1.1         set certificate "XXXXXX"         set peer "XXXXXX"     next end

Additionally I had to create following policy with an IP pool "S2S-Tunnel_local-ip":

config firewall ippool     edit "S2S-Tunnel_local-ip"         set startip 172.19.1.9         set endip 172.19.1.9         set arp-reply disable     next end config firewall policy     edit 18         set srcintf "S2S-Tunnel"         set dstintf "CustB-WAN"         set action accept         set srcaddr "all"         set dstaddr "all"         set schedule "always"         set service "ALL"         set nat enable         set ippool enable         set poolname "S2S-Tunnel_local-ip"     next end

These workarounds seem strange to me. Am I missing something? Do I need to configure anything else?

Apart from that, I think my current setup is very clear and structured and the IPsec tunnel gets created with npu_flag=03. This means it is fully hardware accelerated.

I'm looking forward to hearing your ideas.