FortiGate VXLAN with virtual wire pair only works one way?

[InnovatumIT]

I have been trying to establish a VXLAN link to evaluate connecting two buildings with the same L2 VLANs over the internet.

The physical setup:

Site 1

 Laptop 1 to untagged VLAN8 port on HPE Aruba 2530
 Port on HPE Aruba 2530 with VLAN5 and VLAN8 tagged (nothing untagged) to "port15" on primary of Fortigate 200F (7.2.4) HA cluster

Site 2

Laptop 2 to untagged VLAN8 port on HPE Aruba 2530
Port on HPE Aruba 2530 with VLAN5 and VLAN8 tagged (nothing untagged) to "internal5" on Fortigate 60F (7.2.5)

Site 1 and 2 connected via Internet.

The problem I'm having is that only some of the traffic seems to get through the tunnel.
On site2, running a sniffer on 60F internal5 with no filter, I see a lot of UDP broadcast and ARP originating from
Site 1 as well as DHCP discovery from Laptop2. The laptop gets no DHCP offer but running wireshark I can see the same UDP broadcasts from site 1. Using a fixed IP and pinging resources on site1 also fails.
Running diagnose sys vxlan fdb list <vxlan> i get a list of mac addresses from site 1, on both VLANs, with remote_ip set to site1 of the tunnel.

Doing the same on Site 1 returns only a 00:00:00:00:00:00 mac pointing to site2 tunnel ip.
There I can't detect the traffic from site2. It seems that the tunnel only works one way? Are there any additional requirements on the switches besides handeling VLAN tagging?

I have been following "VLAN over IPsec tunnel with virtual wire pair" from the FortiOS 7.2.4 admin guide
https://docs.fortinet.com/document/fortigate/7.2.4/administration-guide/821119/vxlan-over-ipsec-tunn...

Any help I can get is much appreciated!

Configurations from both sites:

Site1:

config system vxlan
    edit "vxlan-thn-wi"
        set interface "Thn-Wi"
        set vni 10
        set remote-ip "10.199.10.2"
    next
end

config system virtual-wire-pair
    edit "vwp-thn-wi"
        set member "port15" "vxlan-thn-wi"
        set wildcard-vlan enable
    next
end

config system interface
    edit "Thn-Wi"
        set vdom "root"
        set ip 10.199.10.1 255.255.255.255
        set type tunnel
        set remote-ip 10.199.10.2 255.255.255.252
        set snmp-index 18
        set interface "EXTNet"
    next
end

config system interface
    edit "vxlan-thn-wi"
        set vdom "root"
        set type vxlan
        set snmp-index 46
        set interface "Thn-Wi"
    next
end

config vpn ipsec phase1-interface
    edit "Thn-Wi"
        set interface "EXTNet"
        set peertype any
        set net-device disable
        set proposal aes256-sha1
        set remote-gw 11.22.33.44
        set psksecret ENC xxxxxxxxxxxxxxxxxxxxxxxx
    next
end

config vpn ipsec phase2-interface
    edit "Thn-Wi"
        set phase1name "Thn-Wi"
        set proposal aes256-sha1
        set auto-negotiate enable
    next
end

config firewall policy
    edit 68
        set name "VWP-Thn-Wi"
        set uuid 1f7b01d8-41a5-51ee-8fbd-d4220e557bdf
        set srcintf "port15" "vxlan-thn-wi"
        set dstintf "port15" "vxlan-thn-wi"
        set action accept
        set srcaddr "all"
        set dstaddr "all"
        set schedule "always"
        set service "ALL"
        set logtraffic all
    next
end

Site 2

config system vxlan
    edit "vxlan-thn-wi"
        set interface "Thn-Wi"
        set vni 10
        set remote-ip "10.199.10.1"
    next
end

config system virtual-wire-pair
    edit "vwp-thn-wi"
        set member "internal5" "vxlan-thn-wi"
        set wildcard-vlan enable
    next
end

config system interface
    edit "Thn-Wi"
        set vdom "root"
        set ip 10.199.10.2 255.255.255.255
        set type tunnel
        set remote-ip 10.199.10.1 255.255.255.252
        set snmp-index 17
        set interface "wan1"
    next
end

config system interface
    edit "vxlan-thn-wi"
        set vdom "root"
        set type vxlan
        set snmp-index 18
        set interface "Thn-Wi"
    next
end

config vpn ipsec phase1-interface
    edit "Thn-Wi"
        set interface "wan1"
        set peertype any
        set net-device disable
        set proposal aes256-sha1
        set remote-gw 55.66.77.88
        set psksecret ENC xxxxxxxxxxxxxxxxxxxxxxxxx
    next
end

config vpn ipsec phase2-interface
    edit "Thn-Wi"
        set phase1name "Thn-Wi"
        set proposal aes256-sha1
        set auto-negotiate enable
    next
end

config firewall policy
    edit 2
        set name "VWP-Thn-Wi"
        set uuid 6e274f4c-41ac-51ee-477a-d68e4bac6cee
        set srcintf "vxlan-thn-wi" "internal5"
        set dstintf "vxlan-thn-wi" "internal5"
        set action accept
        set srcaddr "all"
        set dstaddr "all"
        set schedule "always"
        set service "ALL"
    next
end

[hbac]

Hi @InnovatumIT, 

Your configuration looks correct. However, we don't see configuration of "port15" on primary of Fortigate and "internal5" on Fortigate 60F. Do those interfaces have VLAN subinterfaces configured? Is the traffic from laptop 1 and laptop 2 being tagged or not? You can try to ping from laptop 1 to laptop2 and run sniffer on both FortiGates to see how the traffic is flowing. (replace x.x.x.x with the destination IP address) 

di sniffer packet any 'host x.x.x.x and icmp' 4 0 l 

Regards,

[InnovatumIT]

Hi Hbac,

I haven't configured any VLANs on the Fortigate ports connecting to the switches. The 60F is "out of the box" and I only removed internal5 from the default internal network. Port15 on the main firewall is also removed from any other assignments.

-------------
config system interface
edit "port15"
set vdom "root"
set type physical
set snmp-index 37
next
end
------------

config system interface
edit "internal5"
set vdom "root"
set type physical
set snmp-index 37
next
end

I've tried connecting the two switches directly to verify that the VLAN trunk ports works ok.

Unfortunatly Laptop2 doesn't get an ARP reply to resolve the IP address and send ICMP ping. The ARP request can be seen on the 60F vxlan-interface:

diagnose sniffer packet any 'arp' 4 0 l
2023-08-28 12:02:05.019056 vxlan-thn-wi out arp who-has 192.168.102.11 tell 192.168.102.6

I can't find the same packet exiting on port15/vxlan-thn-wi on the other fortigate. On 60F there is a lot of other ARP-traffic originating from the main site so traffic seems to flow from Site1 to Site2 but not the other way.

IPSec GUI traffic stats on Site1 says 100B incoming and ~2 GB outgoing and Site2 says also ~2 GB Incoming but 5,33MB outgoing. Both Fortigates can ping eachother using the public IP on Wan.

When I bring the tunnel down and it reconnects, the "Incoming data" on Site1 once again stops at 100B.

I can't find see any errors when using the "Show matching logs" on IPSec status page on both Fortigates. Status also indicates that the tunnel is Up.

Regards,

[InnovatumIT]

Update: I was able to get the vxlan link working by using set npu-offload disable on the ipsec phase1-interface on both Fortigates. Now I'm able to ping devices on both sides. I only get 250-300Mbit throghput so it would be great if npu offload could still be used somehow.

One issue that remains is that devices on the 60F side can only access devices on the same subnet on the 200F. The 200F is the gateway for the combined network but pings originating from accross the vxlan link going out to the internet seems to get no reply.   I can see the first icmp request  going out on port15, then out on the correct VLAN interface on x1 and finaly out on x1 itself. Subesquent requests stops at port15 and no icmp replies seems to come back from internet.

[hbac]

Hi @InnovatumIT, 

It is better to run debug flow on both firewalls to see where the traffic is dropping. Below are an example of debug flow commands:

di deb disable
di deb res
diagnose debug flow filter clear
di deb flow filter addr 8.8.8.8
di deb flow filter proto 1
diagnose debug flow show function-name enable
di deb flow show iprope en
diagnose debug console timestamp enable
diagnose debug flow trace start 500
diagnose debug enable

Regards,

[pminarik]

virtual-wire-pair (VWP) is basically a tiny transparent-mode setup. No routing is expected, what enters port5 leaves out of the VXLAN virtual interface, and vice versa. No other path is possible for traffic entering/leaving these two ports. You cannot route traffic out of this setup.

To be able to do routing, you will need to change the setup and use a software switch instead of a VWP, and then you will be able to do <soft-switch> --> <destination interface> policies.

[InnovatumIT]

Hi!

I expected the VXLAN/Virtual Wire link to be transparent to most functions in the Fortigate as it was described as just a virtual cable between two switches. If I understand correcty, the traffic that exits the VW port is not routable even it it enters the fortigate again on a different port.

Our main F200 is using the 10G x1 port for all networks. They enter as a VLAN trunk and the F200 acts as FW/GW for VLAN interfaces on x1, including VLAN 5 and 8 that are passed through the VXLAN link.

 If the traffic entering x1 on vlan 8 is broken from first having traversed the Fortigate VXLANlink, is there any way that a similar setup could be achieved?  If I use a virtual switch instead of virtual wire, and that switch as source, can I differentiate between the the VLXAN VLANs in the firewall rules? Sorry for my confuesed questions. :)

[pminarik]

That's a misunderstanding then.

The idea of virtual wire pair is that it is a "virtual wire". If a packet enters a wire on one side, it should leave on the other. That's all there is to it. A passing packet cannot be steered away from this pair of interfaces.

Switching, nor routing, is not a feature of a wire, whether it's physical, or virtual.

You could switch/route such packet further if you looped it back into a proper routed interface (or into a HW/SW switch interface; for example, if your internal5 was wired back into a different soft-switch interface on the FGT), but these setups have been known to cause issues, especially when UTM is involved. (a single packet passing twice through a single VDOM is "danger zone")

But back to your scenario: If the 200F is supposed to be the gateway for the subnet that is on both sides and VXLANed together, then my suggestion would be to "convert" the VWP on the 200F side into a soft-switch.

The 60F-side can stay as a VWP, unless you expect to occasionally want to route out of the subnet on the 60F already (e.g. what do you do when the 200F is down/unreachable? Is it OK for the subnet on the 60F side to stay cut off, or do you want to give it some backup internet access directly through that 60F?).

[InnovatumIT]

Thanks for the fast reply! I'm going to try soft switch instead of virtual wire.  You mention that the same packet appearing multiple times in the same VDOM is a problem. Could I create a new VDOM, used only for the VXLAN link config, to prevent the problem with packets not beeing routable after passing VXLAN-VW? Packets from 60F exits the VDOM2 VXLAN link through port 15, is then physically sent to root VDOM via x1?  The VDOM2 would then need its own internet facing interface and ip i guess.

[pminarik]

Yes, I've seen VDOMs used a couple times to deal with weird designs where a packet had to pass through the FortiGate twice. This would need to be done on the FortiGate which has both the VWP (in VDOM-X), and the interface which serves as the gateway for the subnet (VDOM-Y) to route traffic further.