Description
This article shows the behavior of a VXLAN over an IPSEC setup when the tunnel traverses VDOMs.
Solution
This is the topology:
On the first setup, a Software VDOM link on the second FortiGate is used at the right to route traffic between the Root VDOM and the VPN-Vdom:
Due to a limitation on the Software VDOM Link, the VXLAN traffic will not work properly.
If a ping is initiated from host 172.16.200.1 to host 172.16.200.2, and then the traffic is sniffed on both FortiGates, everything will look fine on the first FortiGate, but on the second, the ARP request will not be forwarded properly.
The ARP request is forwarded from the first FortiGate over the VXLAN interface in the outbound direction:
fortilab # diag sniffer packet any 'host 172.16.200.1' 4 0
Using Original Sniffing Mode
interfaces=[any]
filters=[host 172.16.200.1]
50.070325 port7 in arp who-has 172.16.200.2 tell 172.16.200.1
50.070366 vxlan out arp who-has 172.16.200.2 tell 172.16.200.1
50.070425 vxlan-sw in arp who-has 172.16.200.2 tell 172.16.200.1
51.016478 port7 in arp who-has 172.16.200.2 tell 172.16.200.1
51.016494 vxlan out arp who-has 172.16.200.2 tell 172.16.200.1
51.016546 vxlan-sw in arp who-has 172.16.200.2 tell 172.16.200.1
52.016127 port7 in arp who-has 172.16.200.2 tell 172.16.200.1
52.016141 vxlan out arp who-has 172.16.200.2 tell 172.16.200.1
52.016185 vxlan-sw in arp who-has 172.16.200.2 tell 172.16.200.1
53.024351 port7 in arp who-has 172.16.200.2 tell 172.16.200.1
53.024377 vxlan out arp who-has 172.16.200.2 tell 172.16.200.1
53.024419 vxlan-sw in arp who-has 172.16.200.2 tell 172.16.200.1
54.016371 port7 in arp who-has 172.16.200.2 tell 172.16.200.1
54.016384 vxlan out arp who-has 172.16.200.2 tell 172.16.200.1
54.016425 vxlan-sw in arp who-has 172.16.200.2 tell 172.16.200.1
55.016345 port7 in arp who-has 172.16.200.2 tell 172.16.200.1
However, on the second FortiGate, the ARP request is received but is not forwarded to the host:
fortilab2 (VPN-Vdom) # diag sniffer packet any 'host 172.16.200.1' 4 0
interfaces=[any]
filters=[host 172.16.200.1]
13.049544 vxlan in arp who-has 172.16.200.2 tell 172.16.200.1
13.049558 vxlan-sw in arp who-has 172.16.200.2 tell 172.16.200.1
13.995666 vxlan in arp who-has 172.16.200.2 tell 172.16.200.1
13.995681 vxlan-sw in arp who-has 172.16.200.2 tell 172.16.200.1
14.995316 vxlan in arp who-has 172.16.200.2 tell 172.16.200.1
14.995332 vxlan-sw in arp who-has 172.16.200.2 tell 172.16.200.1
16.003499 vxlan in arp who-has 172.16.200.2 tell 172.16.200.1
16.003510 vxlan-sw in arp who-has 172.16.200.2 tell 172.16.200.1
16.995494 vxlan in arp who-has 172.16.200.2 tell 172.16.200.1
16.995505 vxlan-sw in arp who-has 172.16.200.2 tell 172.16.200.1
17.995473 vxlan in arp who-has 172.16.200.2 tell 172.16.200.1
This is due to a limitation of the Software VDOM link.
If the configuration is switched on the second FortiGate to an NPU VDOM link, end to end communication will be established:
18.193556 vxlan in arp who-has 172.16.200.2 tell 172.16.200.1
18.193560 port16 out arp who-has 172.16.200.2 tell 172.16.200.1
18.193561 vxlan-sw in arp who-has 172.16.200.2 tell 172.16.200.1
18.194615 port16 in arp reply 172.16.200.2 is-at 00:74:61:63:29:01
18.194618 vxlan out arp reply 172.16.200.2 is-at 00:74:61:63:29:01
47.245781 vxlan in 172.16.200.1 -> 172.16.200.2: icmp: echo request
47.245784 port16 out 172.16.200.1 -> 172.16.200.2: icmp: echo request
47.250965 port16 in 172.16.200.2 -> 172.16.200.1: icmp: echo reply
47.250968 vxlan out 172.16.200.2 -> 172.16.200.1: icmp: echo reply
48.239543 vxlan in 172.16.200.1 -> 172.16.200.2: icmp: echo request
48.239546 port16 out 172.16.200.1 -> 172.16.200.2: icmp: echo request
48.240281 port16 in 172.16.200.2 -> 172.16.200.1: icmp: echo reply
48.240283 vxlan out 172.16.200.2 -> 172.16.200.1: icmp: echo reply
49.239837 vxlan in 172.16.200.1 -> 172.16.200.2: icmp: echo request
49.239840 port16 out 172.16.200.1 -> 172.16.200.2: icmp: echo request
49.240456 port16 in 172.16.200.2 -> 172.16.200.1: icmp: echo reply
49.240459 vxlan out 172.16.200.2 -> 172.16.200.1: icmp: echo reply
50.239503 vxlan in 172.16.200.1 -> 172.16.200.2: icmp: echo request
50.239506 port16 out 172.16.200.1 -> 172.16.200.2: icmp: echo request
50.240343 port16 in 172.16.200.2 -> 172.16.200.1: icmp: echo reply
50.240346 vxlan out 172.16.200.2 -> 172.16.200.1: icmp: echo reply
If the NPU VDOM link is unavailable, the alternative is to create inter-vdom links as type 'ethernet':
config global
config system vdom-link
edit "<vdom-link-name>"
set type ethernet
next
end
Related articles:
Technical Tip: VXLAN over IPsec for multiple VLANs using software switch
