Sharing WAN between VDOMs & IPSec routing problem

[Marc4228]

Dear Fortinet pros,

I’ve been thinking for a while about the best way to share a single ISP across multiple customers on a FortiGate cluster (100F), with maximum performance as the main goal. My first approach—using EMAC VLAN interfaces—ran into a problem: IPsec tunnels aren’t hardware-accelerated in that setup and only achieve about 5–10% of the possible bandwidth. Support advised using standard VLAN interfaces to enable hardware acceleration.

So I’ve switched to the following design:

Basics: The FortiGate cluster is connected to the data center network via an LACP trunk (x1 + x2 interfaces). There are no FortiSwitches. All network interfaces are placed under an 802.3ad aggregate named “DC-Network.” The goal is to use only VLAN interfaces there.

The IP addresses below are masked: all 172.x.x.x/x addresses represent public subnets, and all 10.x.x.x/x addresses are private ranges. There’s a public IP block 172.19.1.0/27 that I want to split into smaller /29 subnets for individual customers. This public block is connected to the internet through an ISP router over a public transfer network 172.19.19.0/29.

Concrete setup: Via the root VDOM, I connect the ISP’s public transfer network (ROOT-PUBT01). Using the npu0_vlink interfaces and private transfer networks (ROOT-CustX <-> CustX-WAN), each customer VDOM is then connected to its assigned public /29 (CustX-INET). Within each customer VDOM, there are additional private VLANs (CustX-APP) for the servers. The full layout is shown in the network diagram.

General: Does this setup make sense as-is, or do you have suggestions to improve it?

Specific: Right now I have the feeling that routing from CustB-APP <-> CustB-INET <-> CustB-WAN doesn’t actually follow that path; instead it seems to always take the shortcut CustB-APP <-> CustB-WAN.

For example, I need to create the policy for inbound traffic via a Virtual IP on the CustB-WAN interface, not on CustB-INET. Also, an IPsec tunnel that terminates on the CustB-INET interface doesn’t really work. Interestingly, the tunnel comes up and shows as active, but packets originating from the CustB-APP network never find their way into the tunnel.

A packet trace (debug flow) shows the issue as: no route to <IP of remote-gw>, drop (see screenshot):

So I have to change the interface of the phase1-interface to CustB-WAN and configured the local-gw to the public IP address:

config vpn ipsec phase1-interface
    edit "S2S-Tunnel"
        set interface "CustB-WAN"
        set local-gw 172.19.1.9
        set authmethod signature
        set net-device disable
        set proposal aes128-sha256 aes256-sha256 aes128-sha1 aes256-sha1
        set remote-gw 172.16.1.1
        set certificate "XXXXXX"
        set peer "XXXXXX"
    next
end

Additionally I had to create following policy with an IP pool "S2S-Tunnel_local-ip":

config firewall ippool
    edit "S2S-Tunnel_local-ip"
        set startip 172.19.1.9
        set endip 172.19.1.9
        set arp-reply disable
    next
end
config firewall policy
    edit 18
        set srcintf "S2S-Tunnel"
        set dstintf "CustB-WAN"
        set action accept
        set srcaddr "all"
        set dstaddr "all"
        set schedule "always"
        set service "ALL"
        set nat enable
        set ippool enable
        set poolname "S2S-Tunnel_local-ip"
    next
end

These workarounds seem strange to me. Am I missing something? Do I need to configure anything else?

Apart from that, I think my current setup is very clear and structured and the IPsec tunnel gets created with npu_flag=03. This means it is fully hardware accelerated.

I'm looking forward to hearing your ideas.

[zokalmo1]

Yeah this VDOM for us is new, the pair of Fortigates we have on the east coast is only used by us. This new one however is used on the west coast and shared by ourselves and our subsidiary. Isolated networks for now so this seems like the perfect solution. Thanks so much!

[Marc4228]

After writing this article, we implemented a slightly modified approach that solved the problems mentioned. We now dispense with the CustX-INET interfaces and simply configure the public IP addresses as secondary IPs on the CustX-WAN interfaces in addition to the routing address.

For example, the configuration of the CustA-WAN interface is as follows:

config system interface
  edit "CustA-WAN"
    set vdom "CustA"
    set ip 172.19.1.1 255.255.255.0
    set allowaccess ping
    set monitor-bandwidth enable
    set role wan
    set snmp-index 74
    set secondary-IP enable
    set interface "npu0_vlink1"
    set vlanid 1001
    config secondaryip
       edit 1
          set ip 172.20.20.2 255.255.255.252
       next
    end
  next
end

I have configured the routing address as a secondary address so that outgoing NAT works as usual without any special configuration.