VXLAN over IPSEC

[jfgagnon]

Hi!

I was wondering if any of you could helpo me out making this work,

I"m runnning 2 VM64 Fortigate on a ESXi server, through 2 VyOS router to emulate.

Version 5.6.3

The tunnel is up, but somehow, ARP requests are not getting through:

FortiGate-VM64 # diag netlink brctl name host VXLAN-INTERFACE show bridge control interface VXLAN-INTERFACE host. fdb: size=2048, used=3, num=3, depth=1 Bridge VXLAN-INTERFACE host table port no device devname mac addr ttl attributes 1 6 port4 00:0c:29:d6:62:ab 51 Hit(51) 2 17 VXLAN 5e:9f:e8:0f:21:a6 0 Local Static 1 6 port4 00:0c:29:0f:47:91 0 Local Static

interfaces=[any] filters=[host 10.0.11.100 and arp] 15.470412 port4 in arp who-has 10.0.11.101 tell 10.0.11.100 15.470449 VXLAN out arp who-has 10.0.11.101 tell 10.0.11.100 16.487104 port4 in arp who-has 10.0.11.101 tell 10.0.11.100 16.487121 VXLAN out arp who-has 10.0.11.101 tell 10.0.11.100 17.511047 port4 in arp who-has 10.0.11.101 tell 10.0.11.100 17.511059 VXLAN out arp who-has 10.0.11.101 tell 10.0.11.100 18.535167 port4 in arp who-has 10.0.11.101 tell 10.0.11.100 18.535191 VXLAN out arp who-has 10.0.11.101 tell 10.0.11.100

here's my config:

edit "port2" set vdom "root" set ip 84.84.85.2 255.255.255.0 set allowaccess ping set type physical set alias "WAN1" set role wan set snmp-index 2 next

edit "VXLAN" set vdom "root" set type tunnel set snmp-index 12 set interface "port2" next

config vpn ipsec phase1-interface edit "VXLAN" set interface "port2" set peertype any set proposal des-md5 set encapsulation vxlan set encapsulation-address ipv4 set encap-local-gw4 84.84.85.2 set encap-remote-gw4 84.84.86.2 set remote-gw 84.84.86.2 set psksecret ENC OWif8UtnjVfxFQDRN8ajAv/Ten/+O8xoWmIRA1fylLgeGljO1jb+irdNGhDpwlOJD5SJzW4uycM4fDZ2ISwWZUzCCeGKS2q2Df8PQ+qz4Q3pKS4FRd1/IpIYC1dcnnpsEixK5NuYyThTKHc9AoCZF0FT3akcZjevsHKb9m+CV/6VNE9ZY6mDy9bwcDrc7mSiie+mIg== next end

config vpn ipsec phase2-interface edit "VXLAN_ph2" set phase1name "VXLAN" set proposal des-md5 next end

config system switch-interface edit "VXLAN-INTERFACE" set vdom "root" set member "port4" "VXLAN" set intra-switch-policy explicit next end

config firewall policy edit 1 set name "VXLAN-INCOMING" set uuid 1d96cbcc-3d91-51e8-585d-00de8ce55269 set srcintf "VXLAN" set dstintf "port4" set srcaddr "all" set dstaddr "all" set action accept set schedule "always" set service "ALL" set nat enable next edit 2 set name "VXLAN-OUTGOING" set uuid 2c5fe85a-3d91-51e8-7c00-653d11fab724 set srcintf "port4" set dstintf "VXLAN" set srcaddr "all" set dstaddr "all" set action accept set schedule "always" set service "ALL" set nat enable next end

Thanks for the help!

[jfgagnon]

No one?

[emnoc]

The cli-cmd diag debug flow would be useful here but when you dump oin either end of the ipsec do you see ARPs?

Also what does diag vpn tunnel list shows for any counters ( tx/rcv-enc )

BTW: your configurations looks right

ken

[jfgagnon]

I don't see ARP getting inside the tunnel

FortiGate-VM64 # diag sniffer packet any 'host 10.0.11.101 and arp' 4 interfaces=[any] filters=[host 10.0.11.101 and arp] 0.925592 port4 in arp who-has 10.0.11.100 tell 10.0.11.101 0.925610 VXLAN out arp who-has 10.0.11.100 tell 10.0.11.101 1.949396 port4 in arp who-has 10.0.11.100 tell 10.0.11.101 1.949411 VXLAN out arp who-has 10.0.11.100 tell 10.0.11.101 2.973408 port4 in arp who-has 10.0.11.100 tell 10.0.11.101 2.973430 VXLAN out arp who-has 10.0.11.100 tell 10.0.11.101 3.997561 port4 in arp who-has 10.0.11.100 tell 10.0.11.101 3.997577 VXLAN out arp who-has 10.0.11.100 tell 10.0.11.101

[HA]

Hi,

Maybe something missing on the physical interface to forward broadcast ??

Here's my config...

config system interface edit "wan1" set ip 10.0.0.1 255.255.255.0 next edit "wan2" set vlanforward enable set broadcast-foward enable set l2forward enable set stpforward enable set netbios-forward enable next edit "VxLan-IPsec" set vlanforward enable set broadcast-foward enable set l2forward enable set stpforward enable set netbios-forward enable next end config system switch-interface edit "VxLan-Switch" set member "wan2" "VxLan-IPsec" set intra-switch-policy explicit next end

Hope it can help you...

Regards,

HA

[emnoc]

I don't see ARP getting inside the tunnel

the VXLAN is name is your tunnel,you need to execute  the same on the opposite side but again you need to lok at vpn tunnel statistics and diag debug flow

You should see the messages for the action of vxlan or something similar .

ken

[jfgagnon]

Any commands I could use then?

Cause what I got not very useful so far

[emnoc]

i would  start with cli-cmd

diag vpn tunne list name VXLAN

Execute the  same on both  sides of the ipsec-gw, do you see tunnel counters for ENC-domain  tx/recv pkts ? if not  than you have a problem

e.g

VPN1  sent 3400 pkts  than VPn2 should have  3400

if  VPN1 received  200 pkts than VPN2 should  have sent 2000 pkts

for diag debug flow, run the search here for  examples  but again cli cmd

diag debug en

diag debug reset

diag debug flow filter addr x.x.x.x

diag debug flow show console enable

diag debug flow trace start 100

( now generate some traffic for the ENC_DOMAINS and see what happens )

Ken

[jfgagnon]

FortiGate-VM64 # diag debug flow show console enable

command parse error before 'console' Command fail. Return code -61

[jfgagnon]

It does look like traffic is not getting through, i only see bytes going up in incoming on one side, and outgoing on the other side