VXLAN over IPSEC don't work

[5q46n2te8jPWJY]

Hello everyone,

I’m trying to set up VLAN over IPSEC using VXLAN on my FortiGate, but I’m facing issues where the tunnel doesn’t seem to work as expected.

The tunnel comes up, but traffic from the VLAN doesn’t seem to pass through. I’ve tried troubleshooting using various methods, but I haven’t been able to resolve the issue.

Here’s the configuration I’m using for VXLAN 506:

On site A :

config system interface
    edit "IPSEC_VXLAN"
        set vdom "VDOM1"
        set ip 10.5.5.1 255.255.255.255
        set allowaccess ping
        set type tunnel
        set remote-ip 10.5.5.2 255.255.255.252
        set snmp-index 42
        set interface "vlnk_HYP0"
    next
end
config system interface
    edit "VLAN_506"
        set vdom "VDOM1"
        set role lan
        set snmp-index 506
        set interface "port1"
        set vlanid 506
    next
end
config system vxlan
    edit "VXLAN_506"
        set interface "IPSEC_VXLAN"
        set vni 506
        set remote-ip "10.5.5.2"
    next
end
config system switch-interface
    edit "VXLAN506-SW"
        set vdom "VDOM1"
        set member "VLAN_506" "VXLAN_506"
    next
end
config system interface
    edit "VXLAN506-SW"
        set vdom "VDOM1"
        set ip 10.112.7.254 255.255.255.0
        set allowaccess ping
        set type switch
        set lldp-reception enable
        set lldp-transmission enable
        set role lan
        set snmp-index 23
        set ip-managed-by-fortiipam disable
    next
end

On site B : 

config system interface
    edit "IPSEC_VXLAN"
        set vdom "VDOM1"
        set ip 10.5.5.2 255.255.255.255
        set allowaccess ping
        set type tunnel
        set remote-ip 10.5.5.1 255.255.255.252
        set snmp-index 42
        set interface "vlnk_HYP0"
    next
end
config system interface
    edit "VLAN_506"
        set vdom "VDOM1"
        set role lan
        set snmp-index 506
        set interface "port1"
        set vlanid 506
    next
end
config system vxlan
    edit "VXLAN_506"
        set interface "IPSEC_VXLAN"
        set vni 506
        set remote-ip "10.5.5.1"
    next
end
config system switch-interface
    edit "VXLAN506-SW"
        set vdom "VDOM1"
        set member "VLAN_506" "VXLAN_506"
    next
end
config system interface
    edit "VXLAN506-SW"
        set vdom "VDOM1"
        set ip 10.112.7.254 255.255.255.0
        set allowaccess ping
        set type switch
        set lldp-reception enable
        set lldp-transmission enable
        set role lan
        set snmp-index 23
        set ip-managed-by-fortiipam disable
    next
end

I try to ping a machine in VLAN 506 from site A (a2:43:6b:c4:a9:23) to site B (ca:3a:14:30:9a:3a)

It don't work

When I run diagnose sys vxlan fdb list VXLAN_506 on each side, I view :

On site A : 

diagnose sys vxlan fdb list VXLAN_506
mac=00:00:00:00:00:00 state=0x0082 remote_ip=10.5.5.2 port=4789 vni=506 ifindex=66
mac=00:09:0f:09:00:00 state=0x0002 remote_ip=10.5.5.2 port=4789 vni=506 ifindex=66
mac=ca:3a:14:30:9a:3a state=0x0002 remote_ip=10.5.5.2 port=4789 vni=506 ifindex=66

On site B : 

diagnose sys vxlan fdb list VXLAN_506
mac=00:00:00:00:00:00 state=0x0082 remote_ip=10.5.5.1 port=4789 vni=506 ifindex=112
mac=0e:32:0f:49:db:46 state=0x0002 remote_ip=10.5.5.1 port=4789 vni=506 ifindex=112

Mac addresse of Site A Server is not present in site B table...

I think it's the cause of my problem, but why ?

Can anyone help me to debug ?

Thanks !

[dbhavsar]

Hello @5q46n2te8jPWJY ,

- under the config system switch-interface, can you check for the following command:
set intra-switch-policy implicit

Reference article:
https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-setup-a-VXLAN-over-IPsec-deployment... 

Also check for the firewall policies.

[5q46n2te8jPWJY]

Hello,

Thank you for your reply. Before looking your command, I first checked if the Mac address was still missing from table. It appeared in the meantime.

However, I still can't ping between my 2 sites.

I added your command because it wasn't there, but it didn't change anything.

Do you have another idea ?

[dbhavsar]

Hi @5q46n2te8jPWJY ,

Have you checked the firewall policies? or run the following debugs and see if the traffic is leaving the FGT or not:
get router info routing-table details <source-ip>
get router info routing-table details <destination-ip>
di de reset
diagnose debug flow filter addr xx.xx.xx.xx yy.yy.yy.yy and <--- xx is source-IP and yy is destination-ip
di de flow filter proto 1
diagnose debug flow show function enable
diagnose debug flow show iprope enable
diagnose debug console timestamp enable
diagnose debug flow trace start 999
diagnose debug enable

[5q46n2te8jPWJY]

Here the output from site A

get router info routing-table details 10.112.7.1 (Site A server address)

Routing table for VRF=0
Routing entry for 10.112.7.0/24
  Known via "connected", distance 0, metric 0, best
  * is directly connected, VXLAN506-SW

get router info routing-table details 10.112.7.7 (Site B server address)

Routing table for VRF=0
Routing entry for 10.112.7.0/24
  Known via "connected", distance 0, metric 0, best
  * is directly connected, VXLAN506-SW

I also run from site A

diagnose debug reset
diagnose debug flow filter addr 10.112.7.1
diagnose debug enable

and

diagnose debug reset
diagnose debug flow filter addr 10.112.7.7
diagnose debug enable

I have nothing in console... the same on site B...

[dbhavsar]

Hello @5q46n2te8jPWJY ,

- Have you tried pinging destination while the debugs were running on the device? Or I would suggest to open a TAC case to further dig into this issue.

[5q46n2te8jPWJY]

Yes, of course ;)

I'll open TAC case

Thank you for your help

[5q46n2te8jPWJY · Accepted solution]

Hi,

I solved my problem myself, I have VDOM in my conf, I had to define the Ethernet type on each vdom.

Since it was done, it works perfectly!

[ricky_andre_76]

what do you mean exactly ? what commands did you type in to define Ethernet type on each vdom ?

[5q46n2te8jPWJY]

config global
    config system vdom-link
        edit "<vdom-link-name>"
            set type ethernet
        next
    end

https://community.fortinet.com/t5/FortiGate/Technical-Tip-VXLAN-over-IPSEC-inter-VDOM-behavior/ta-p/...