Skip to main content
StephenL
StephenL
New Member
August 21, 2022
Solved

Default Gateway for vxlan over ipsec

  • August 21, 2022
  • 5 replies
  • 8369 views

I have setup two sites with a Fortgate 100F device (7.0.5) at each site.

 

I have set-up and have operational a vxlan connection between the two sites over an IPSEC tunnel.

 

At site A there is a monitoring VM with a fixed IP address (192.168.200.2/24 GW 192.168.200.1) and at site B a test VM with a fixed IP address 192.168.200.3/24 GW 192.168.200.1)

 

The issue I have is the test VM is unable to ping the GW IP address 192.168.200.1 or anything beyond the GW)

 

The testing at the moment is:

 

The Monitoring VM (192.168.200.2) is able to ping the test VM (192.168.200.3)

The Monitoring VM (192.168.200.2) is able to ping the GW (192.168.200.1)

The Monitoring VM (192.168.200.2) is able to ping 8.8.8.8

Can RDP from Monitoring VM (192.168.200.2) to Test VM (192.168.200.3)

The Test VM (192.168.200.3) is able to ping the Monitoring VM (192.168.200.2)

The Test VM (192.168.200.3) is able to ping other devices at Site A in the 192.168.200.x/24 range

The Test VM (192.168.200.3) is unable to ping the GW (192.168.200.1) - Request time-out

 

The diagram below outlines the configuration

VXLAN.jpg

 

It would seem the VXLAN is operational as traffic follows in both directions

External access at Site A via the Software Switch with an IP address of 192.168.200.1 is operational

Ping is allowed for the Software Switch IP 192.168.200.1

Firewall Rules for Zone_200 allow all 192.168.200.0/24 traffic out for ping

VLANing is working via the Fortigate Redundant switch / VLAN switch)

 

Am I missing something about the configuration of VXLAN gateway addresses.

 

I have used the technical guide  https://community.fortinet.com/t5/FortiGate/Technical-Tip-VXLAN-over-IPsec-for-multiple-VLANs-using-software/ta-p/195488 as the basis for the VXLAN.  Aside from the IP addresses where the document refers to Internal1 I am using a VLAN Switch (I need high availability using independent switches).

 

And technical guide https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-route-traffic-from-VXlan-to-other-vlans-or/ta-p/191346 for the routing

 

StephenL_0-1661059486386.png

 

Am I missing something about VXLANs and default gate

Best answer by akristof

Hey, I have an idea. Can you run this commands on FortiGate at SiteB:

diag sys vxlan fdb list <NameOfVxlanVtep>
fnsysctl ifconfig <SoftwareSwitchName> --- This please take from both devices

I am suspecting that it related to that virtual MAC address. If you have 100F on both sites and both sites are running HA cluster and group-id is 0, then there is possibility that mac address generated for software switch on each device is the same. If you would see that both software switches have the same mac address, then you will need to change group-id under one cluster to other value. But for that, I recommend to have direct access (console or OOB management) and do it outside of business hours as you would play with HA. 

5 replies

akristof
Staff
akristof
Staff
August 22, 2022

Hi,

Thank you for your question. Your setup looks correct or at least I don't see any reason why TestVM is not able to ping GW. Let's start with basics. If you ping GW 192.168.200.1 from TestVM, do you see incoming icmp requests on both FortiGates in both zones? This would be my first step, to find where is icmp-request dropped. If it is dropped on FortIGate SiteA, it would be the best scenario because it would be some local problem. If we see icmp-request leaving FortiGateB but not received on FortiGateA, then we can check Ipsec tunnel if tunnel is without any problem.

 

https://docs.fortinet.com/document/fortigate/7.0.2/administration-guide/680228/performing-a-sniffer-trace-cli-and-packet-capture

Note - use verbose level 4, example
diag sniffer packet any "host 192.168.200.1 and icmp" 4 0 l

StephenL
StephenLAuthor
New Member
August 23, 2022

Hi Adrian,

 

I ran a packet capture for the VXLAN interfaces at both site A and site B

 

Neither package capture showed any ping packets for the TestVM to the 192.168.200.1 address but did for pings to/from the Monitor VM.

 

However the packet capture for the Fortigate Software Switch at Site B (has no IP address assign did show pings but no replies.

 

This would seem to indicate that the pings from the test VM to the gateway address are being directed to the site software switch and not over the vxlan.

 

Stephen

akristof
Staff
akristof
Staff
August 23, 2022

Hi,

Thank you. Even if sw-switch at SiteB has no IP address, if you have "any" interface in packet capture, you should see icmp-request come from Vlan200 and enter Vxlan interface. The fact, that TestVM is able to ping MonitoringVM is saying that Vxlan over IPsec is ok. Can you compare on both VMs, after you try to ping GW, arp database? To check if the arp entry is the same?

StephenL
StephenLAuthor
New Member
August 23, 2022

Hi,

 

ARP Test VM

StephenL_0-1661258077345.png

 

ARP Monitor VM

StephenL_1-1661258146334.png

 

MAC Addresses are the same

 

Wireshark output from Site B Fortigate

 

StephenL_2-1661258220553.png

 

No ping traffic from test VM to gateway address - So failing to traverse the tunnel? But has learnt the MAC address.

 

SA information is just 

src 0.0.0.0/0.0.0.0

dst 0.0.0.0/0.0.0.0

 

Stephen

 

 

akristof
Staff
akristofAnswer
Staff
August 23, 2022

Hey, I have an idea. Can you run this commands on FortiGate at SiteB:

diag sys vxlan fdb list <NameOfVxlanVtep>
fnsysctl ifconfig <SoftwareSwitchName> --- This please take from both devices

I am suspecting that it related to that virtual MAC address. If you have 100F on both sites and both sites are running HA cluster and group-id is 0, then there is possibility that mac address generated for software switch on each device is the same. If you would see that both software switches have the same mac address, then you will need to change group-id under one cluster to other value. But for that, I recommend to have direct access (console or OOB management) and do it outside of business hours as you would play with HA. 

StephenL
StephenLAuthor
New Member
August 23, 2022

Hi Adrian,

 

Thanks.  I checked and both software switches have the same MAC.  Just arranging a time to change the group-id

 

Stephen

StephenL
StephenLAuthor
New Member
August 24, 2022

Thanks Adrian,

 

I changed HA setting at one end of the vxlan for the group-id.  Changed from 0 to 1. The change did caused a failover of the Fortigate device

Set-up a new software switch (had a different MAC address) and now able to access the gateway device and beyond.

 

Stephen

akristof
Staff
akristof
Staff
August 24, 2022

Great. Thanks for info. I am glad that we were able to find the problem :)

Terms & ConditionsAccessibility statement