Default Gateway for vxlan over ipsec

[StephenL]

I have setup two sites with a Fortgate 100F device (7.0.5) at each site.

I have set-up and have operational a vxlan connection between the two sites over an IPSEC tunnel.

At site A there is a monitoring VM with a fixed IP address (192.168.200.2/24 GW 192.168.200.1) and at site B a test VM with a fixed IP address 192.168.200.3/24 GW 192.168.200.1)

The issue I have is the test VM is unable to ping the GW IP address 192.168.200.1 or anything beyond the GW)

The testing at the moment is:

The Monitoring VM (192.168.200.2) is able to ping the test VM (192.168.200.3)

The Monitoring VM (192.168.200.2) is able to ping the GW (192.168.200.1)

The Monitoring VM (192.168.200.2) is able to ping 8.8.8.8

Can RDP from Monitoring VM (192.168.200.2) to Test VM (192.168.200.3)

The Test VM (192.168.200.3) is able to ping the Monitoring VM (192.168.200.2)

The Test VM (192.168.200.3) is able to ping other devices at Site A in the 192.168.200.x/24 range

The Test VM (192.168.200.3) is unable to ping the GW (192.168.200.1) - Request time-out

The diagram below outlines the configuration

It would seem the VXLAN is operational as traffic follows in both directions

External access at Site A via the Software Switch with an IP address of 192.168.200.1 is operational

Ping is allowed for the Software Switch IP 192.168.200.1

Firewall Rules for Zone_200 allow all 192.168.200.0/24 traffic out for ping

VLANing is working via the Fortigate Redundant switch / VLAN switch)

Am I missing something about the configuration of VXLAN gateway addresses.

I have used the technical guide  https://community.fortinet.com/t5/FortiGate/Technical-Tip-VXLAN-over-IPsec-for-multiple-VLANs-using-... as the basis for the VXLAN.  Aside from the IP addresses where the document refers to Internal1 I am using a VLAN Switch (I need high availability using independent switches).

And technical guide https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-route-traffic-from-VXlan-to-other-v... for the routing

Am I missing something about VXLANs and default gate

[akristof]

Hi,

Thank you for your question. Your setup looks correct or at least I don't see any reason why TestVM is not able to ping GW. Let's start with basics. If you ping GW 192.168.200.1 from TestVM, do you see incoming icmp requests on both FortiGates in both zones? This would be my first step, to find where is icmp-request dropped. If it is dropped on FortIGate SiteA, it would be the best scenario because it would be some local problem. If we see icmp-request leaving FortiGateB but not received on FortiGateA, then we can check Ipsec tunnel if tunnel is without any problem.

https://docs.fortinet.com/document/fortigate/7.0.2/administration-guide/680228/performing-a-sniffer-...

Note - use verbose level 4, example
diag sniffer packet any "host 192.168.200.1 and icmp" 4 0 l

[StephenL]

Hi Adrian,

I ran a packet capture for the VXLAN interfaces at both site A and site B

Neither package capture showed any ping packets for the TestVM to the 192.168.200.1 address but did for pings to/from the Monitor VM.

However the packet capture for the Fortigate Software Switch at Site B (has no IP address assign did show pings but no replies.

This would seem to indicate that the pings from the test VM to the gateway address are being directed to the site software switch and not over the vxlan.

Stephen

[akristof]

Hi,

Thank you. Even if sw-switch at SiteB has no IP address, if you have "any" interface in packet capture, you should see icmp-request come from Vlan200 and enter Vxlan interface. The fact, that TestVM is able to ping MonitoringVM is saying that Vxlan over IPsec is ok. Can you compare on both VMs, after you try to ping GW, arp database? To check if the arp entry is the same?

[StephenL]

Site A -  diag sniffer packet any "host 192.168.200.241 and icmp" - Test VM

Site B - diag sniffer packet any "host 192.168.200.1 and icmp" 

Same behavior as shown in the packet captures.

Stephen

[akristof]

Hi,

Ok. I don't see interfaces in that packet capture at SiteB, but based on timestamps, one packet is incoming from Vlan, other is outgoing via Vxlan. In that, I can suggest to do ESP packet capture (or UDP/4500 if NAT-T is active) and decrypt these packets. Then you can see if icmp request are correctly encrypted and send via Ipsec. Same thing you can at SiteA to see if packets are decrypted. But I recommend to disable npu-offload under phase1 to make sure that you will all incoming/outgoing ESP packets.

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Decrypt-ESP-packets/ta-p/198431

[StephenL]

Hi,

ARP Test VM

ARP Monitor VM

MAC Addresses are the same

Wireshark output from Site B Fortigate

No ping traffic from test VM to gateway address - So failing to traverse the tunnel? But has learnt the MAC address.

SA information is just 

src 0.0.0.0/0.0.0.0

dst 0.0.0.0/0.0.0.0

Stephen

[akristof · Accepted solution]

Hey, I have an idea. Can you run this commands on FortiGate at SiteB:

diag sys vxlan fdb list <NameOfVxlanVtep>
fnsysctl ifconfig <SoftwareSwitchName> --- This please take from both devices

I am suspecting that it related to that virtual MAC address. If you have 100F on both sites and both sites are running HA cluster and group-id is 0, then there is possibility that mac address generated for software switch on each device is the same. If you would see that both software switches have the same mac address, then you will need to change group-id under one cluster to other value. But for that, I recommend to have direct access (console or OOB management) and do it outside of business hours as you would play with HA. 

[StephenL]

Hi Adrian,

Thanks.  I checked and both software switches have the same MAC.  Just arranging a time to change the group-id

Stephen

[StephenL]

Thanks Adrian,

I changed HA setting at one end of the vxlan for the group-id.  Changed from 0 to 1. The change did caused a failover of the Fortigate device

Set-up a new software switch (had a different MAC address) and now able to access the gateway device and beyond.

Stephen