Tested - VXLAN over IPsec feature on 5.4.1

Forum|Forum|9 years ago
June 29, 2016
3 replies
41138 views

I tested this feature with a collegue yesterday. We enabled the vxlan encapsulation on the phase1-interface, and created a bridge interface/switch containing a physical port and the vxlan-if. It worked as a charm, and when we enabled vlanforward on the physical interface and the ipsec interface, we could also send tagged vlans over the tunnel. This is a feature we have been waiting for, since most other solutions for handling L2 traffic between remote sites/datacenters usually comes with a hefty price tag.

In our customer case, we will have two IP-sec/vxlan tunnels (one active, one redundant) on different 1Gbit ISP/WAN connections between two HA clusters located in different datacenters, effectively bridging selected vlans in the two sites together.

The lab equipment consists of two 400D running 5.4.1 and two Alcatel-Lucent 6850E switches (one on each side, obviously).

We will continue the testing tomorrow. Next step is to do some serious load testing on this setup over some time and to check for problems with stability, latency etc. It need to scale properly, and to be rock solid to be of any interest of course.

Will post our findings here.

Laters.

pcraponi

New Member

Good to know... Congrats... :)

Paulo R., NSE8

kallbrandtAuthor

New Member

We did setup a 2nd redundant IPsec/vxlan-tunnel (monitor pointed at 1st tunnel), configured as the first tunnel, and then tested wan link failure. An important parameter here is the DPD-settings in the primary tunnel. We run the tests with failover after 3 failed polls 5 seconds apart, so 15 seconds before the 2nd tunnel comes up. We can tune it down of course, but you don't want to risk a link-flapping situation either. The 2nd tunnel will go down as soon as the 1st comes up again. It works, but it takes some time probably due to the mac-adress move. The switches has mac-spoofing protection that probably matters too, but haven't had time to check the logs yet. Failover time is approx. 30 seconds.

We will try to add the 2nd tunnel interface to the bridge interface /switch of the first tunnel tomorrow and see if that works. If so, no mac-address move between ports :)

Also some performance testing tomorrow.

kallbrandtAuthor

New Member

The redundancy setup worked like a charm, and putting both IPsec-tunnels in the same virtual switch as the vxlan-interface also worked and lowered the failover time.

However....

The performance is sub-par, to say the least. We did some testing using iperf. With 2 computers on each side of the vxlan tunnel, we got some 83Mbit/s. Tried file transfer also, same speed. That's something like 1/170th of the 400D's max IPsec throughput...

When sending tagged packets through the tunnel (each computer connected to access port with default VLAN in switch, and the switch connected to firewall with the VLAN tagged on port), the speed dropped to approx. 32Mbit/s. That's down to 1/450th of the max IPsec throughput... Fragmentation or packet size shouldn't be an issue since the 802.1Q information is located in the inner ethernet header, inside the vxlan encapsulation. BUT, there are no docs on how the Fortigate handles this, or if you are supposed to tweak the tunnel interface mtu size or on the physical interface.

The virtual switch may very well be the culprit here. It may be possible to use the hardware switch instead, will have to check that one out before dropping this altogether.

Without some miracle with the hardware switch that solves everything, this very promising setup is a total no-go as it is right now. It will work for bridging two remote offices witch a bunch of computers together, but hardly for datacenter traffic, needing at least 1Gbit throughput. Too bad.

Max throughput over a plain IPsec tunnel is 14 Gbit/s according to the 400D docs, and in any way faster than we could measure - We got wire speed in every test over IPsec.