Tested - VXLAN over IPsec feature on 5.4.1

[kallbrandt]

I tested this feature with a collegue yesterday. We enabled the vxlan encapsulation on the phase1-interface, and created a bridge interface/switch containing a physical port and the vxlan-if. It worked as a charm, and when we enabled vlanforward on the physical interface and the ipsec interface, we could also send tagged vlans over the tunnel. This is a feature we have been waiting for, since most other solutions for handling L2 traffic between remote sites/datacenters usually comes with a hefty price tag.

In our customer case, we will have two IP-sec/vxlan tunnels (one active, one redundant) on different 1Gbit ISP/WAN connections between two HA clusters located in different datacenters, effectively bridging selected vlans in the two sites together.

The lab equipment consists of two 400D running 5.4.1 and two Alcatel-Lucent 6850E switches (one on each side, obviously).

We will continue the testing tomorrow. Next step is to do some serious load testing on this setup over some time and to check for problems with stability, latency etc. It need to scale properly, and to be rock solid to be of any interest of course.

Will post our findings here.

Laters.

[ede_pfau]

Why don't you put both IPsec interfaces into a zone? This way, there will only be one session in the session table and no delay on the FGT's side. OTOH, it could well be that zones don't take IPsec interfaces, and the MAC addresses will still be different.

[kallbrandt]

@rcareeras: No news yet, vacation time here etc... Firewalls will be registered this week I think. Will update as soon as I have any new information. In this case, we have to hope for the best, because the tac will of course ask for logs, more testing and stuff. And the firewalls are getting prepped for the plain IPsec setup as we speak, customer wants it online doing its IPsec magic now.now.

So, they will have to take my word for it... I just want an answer to wether the performance issue is something that's fixable in newer code or not. Time will tell.

@ede_pfau: I'm not sure I understand what you mean, so please elaborate if I'm lost..

I don't remember if we had the IPsec-interfaces in a zone for the lab, but zones is always the way to go as soon as you have more then one interface doing the same thing. Always. For this setup with a redundant tunnel on another wan-connection, I doubt it would matter since the redundant tunnel is disabled until the 1st tunnel goes down. The switches used has mac-spoofing protection on by default, and in this case, the mac-adresses that are present in the vlans on one switchport suddenly move to another port without any changes in link status wich is suspicious. The VXLAN setup forces you to use one dedicated physical interface bridged to the VXLAN-tunnel wich is run inside an IPsec-tunnel. So, the VLANs and their corresponding mac-addresses flip between two different physical ports in the firewall when you fail-over between the IPsec-tunnels. One would like the ability to send out something like a gratuitous arp here...

[rcarreras]

@kallbrandt : Thank you!! Please follow up this thread with some news!! Good Luck!

[ak_shukla]

Dear Team, 

Request you to please share the VXlan over IPsec configuration , i have build a lab setup which is not working .. just want to cross check with my present configuration .. 

SETUP

Laptop--> Extreme switch ---> Firewall 800C----IPSEC ----> Firewall 800C---> Extreme Switch ----> Laptop

Please find below Switch config at both sites 

NETRED_NOC1.50 #

--------------------------

NETRED_NOC1.50 # sh configuration # # Module devmgr configuration. # configure snmp sysName "NETRED_NOC1" configure snmp sysContact "support@extremenetworks.com, +1 888 257 3000" configure sys-recovery-level switch reset

# # Module vlan configuration. # configure vlan default delete ports all configure vr VR-Default delete ports 1-64 configure vr VR-Default add ports 1-64 configure vlan default delete ports 1-64 enable jumbo-frame ports all create vlan "intercon" configure vlan intercon tag 150 create vlan "loopback" configure vlan loopback tag 140 enable loopback-mode vlan loopback create vlan "vl200" configure vlan vl200 tag 200

configure vlan intercon add ports 39 tagged configure vlan vl200 add ports 5 untagged configure vlan intercon ipaddress 100.0.0.142 255.255.255.0 enable ipforwarding vlan intercon configure vlan loopback ipaddress 2.2.2.2 255.255.255.255 enable ipforwarding vlan loopback

# # Module mcmgr configuration. # disable igmp snooping vlan "vl200"

# # Module otm configuration. # configure virtual-network local-endpoint ipaddress 1.1.1.1 vr "VR-Default" create virtual-network "VNET" flooding standard configure virtual-network "VNET" vxlan vni 200 configure virtual-network "VNET" add vlan vl200 configure virtual-network "VNET" monitor on

# # Module ospf configuration. # configure ospf routerid 10.10.10.142 enable ospf vxlan-extensions enable ospf configure ospf add vlan intercon area 0.0.0.0 configure ospf add vlan loopback area 0.0.0.0

# --------------------------------------------------------------------------------------------------

NETRED_Noc2 

NETRED_NOC2.16 # sh configuration # # Module devmgr configuration. # configure snmp sysName "NETRED_NOC2" configure snmp sysContact "support@extremenetworks.com, +1 888 257 3000" configure sys-recovery-level switch reset

# # Module vlan configuration. # configure vlan default delete ports all configure vr VR-Default delete ports 1-64 configure vr VR-Default add ports 1-64 configure vlan default delete ports 1-64 enable jumbo-frame ports all create vlan "intercon" configure vlan intercon tag 150 create vlan "loopback" configure vlan loopback tag 140 enable loopback-mode vlan loopback create vlan "vl200" configure vlan vl200 tag 200

configure vlan intercon add ports 39 tagged configure vlan vl200 add ports 5 untagged configure vlan intercon ipaddress 99.1.1.1 255.255.255.0 enable ipforwarding vlan intercon configure vlan loopback ipaddress 2.2.2.2 255.255.255.255 enable ipforwarding vlan loopback

# Module mcmgr configuration. # disable igmp snooping vlan "vl200"

# # Module otm configuration. # configure virtual-network local-endpoint ipaddress 2.2.2.2 vr "VR-Default" create virtual-network "VNET" flooding standard configure virtual-network "VNET" vxlan vni 200 configure virtual-network "VNET" add vlan vl200 configure virtual-network "VNET" monitor on

# Module ospf configuration. # configure ospf routerid 1.1.1.1 enable ospf vxlan-extensions enable ospf configure ospf add vlan intercon area 0.0.0.0 configure ospf add vlan loopback area 0.0.0.0

# # NETRED_NOC2.17 #

Request you to please HELP !!!

Thanks in advance 

Regards,

Ashish 

[ak_shukla]

One More Information 

VXlan is showing UP, laptop are not able to ping each other .. i can see on connected laptop mac entry .. not able to see far end laptop mac entry  

Firewall version  is v5.2.3,build670 (GA)

Thanks !!

[kallbrandt]

Hello,

The VXLAN forwarding inside an IPsec-tunnel is a feature in version 5.4.1 of FortiOS.

You'll need to upgrade to 5.4.1 in order to enable that functionality, and the VXLAN feature itself needs to be enabled inside the IPsec-tunnel, otherwise the firewalls will no be able to handle the VXLAN-packets.

Edit: It would be nice to know if your setup is possible to use with the Fortigate approach to VXLAN, since it's the Fortigate itself that acts as a VXLAN gateway in their setup. In your setup it is the Extreme switches. Will that work with Fortigates take on VXLAN? I don't know.

[kallbrandt]

I have now finally created a ticket regarding this matter - The firewalls in the lab had to be used during summer, so we had to tear down the lab setup, didn't register them etc etc. We now registered them, rebuilt the lab, and measured the performance again - 77mbit over the VXLAN encapsulation, 950mbit without (standard laptop nics directly connected).

[kallbrandt]

Ok, we got an answer:

"... you will never reach same bandwidth for VXLAN over IPSec, as IPSec alone because VXLAN feature currently cannot be fully offloaded to hardware." Figures...

Turning off npu and asic offload in the IPsec-tunnel and the rules adds 10Mbit to the throughput.

Disabling all cryptos in the phase2 settings of the tunnel adds another 30mbit to the throughput.

So, it is possible to get somewhere around 130Mbit/s, but with a setup no one would ever use, unfortunately.

Let's just hope this feature will be possible to offload later on then. I also hope that full support for VNID tagging etc comes too. Will post again if we get any useful info in this matter.

[kallbrandt]

Update: Answer from Fortinet TAC:

"The bug ID is 0369137. This has been fixed in interim build 1071 and patch will be added to next GA release i.e 5.4.2. The expected release date is between Dec 2016 - Jan17 . We also have beta release(5.4.2.Beta20) available for testing purpose so if you want to try you download it from our support portal and test."

Woho!

But seeing is believing, always.

:)

[jvanderzee]

Any news on this being offloaded to the hardware?  I see that 5.4.2 and 5.4.3 are out but unsure if this feature was included.