Weird issue with Fortinet

[usmansa1]

Hi, 

We are experiencing a very weird issue with Fortinet, we installed two Fortinet devices in two cities, topology is mentioned below :- 

FORTINET_CITY1 >> CISCO-SWITCH_CITY1 >> ISP_R1_CITY1 ==VXLAN== ISP_R2_CITY2 >> CISCO ACI >> CISCO-SWITCH_CITY2 >> FORTINET_CITY2 

The communication between FORTINET_CITY_1 and FORTINET_CITY_2 is disrupted suddenly, we did extensive troubleshooting but we couldn't able to locate the issue. The  most strange thing is that, we configured layer-3 VLAN interface on CISCO-SWITCH_CITY1 and CISCO-SWITCH_CITY2 and both switches can able to ping each other.  FORTINET_CITY1 is able to ping CISCO-SWITCH_CITY2 and CISCO-SWITCH_CITY1 but couldn't able to ping  FORTINET_CITY2, on the other hand FORTINET_CITY2 can ping CISCO-SWITCH_CITY2  but cannot ping beyond that, we checked with ISP and they said no issue found, we can see the ARP request coming from FORTINET_CITY1 on FORTINET_CITY2 but we dont see that ARP request reaching to FORTINET_CITY1, it seems like that arp request is not going out of FORTINET_CITY2. Can someone suggest some idea, thanks 

[AEK]

Hi

• Is FG_City1 HA cluster and FG_City2 HA cluster as well? If so, make sure you used different cluster ID for each pair to avoid MAC conflict
• Do you see the ping requests leaving the FG with "diag sniffer packet"?

[mpeddalla]

Hello  ,

Thank you for contacting the Fortinet Forum portal.

I would recommend to verify first which topology scenario comes under and then start troubleshooting after checking configuration.

https://docs.fortinet.com/document/fortigate/7.4.3/administration-guide/548660/general-vxlan-configu...

-You can review the below article for vxlan configuration with ISP in between two FortiGate:

https://community.fortinet.com/t5/FortiGate/Technical-Note-Virtual-Extensible-LAN-VXLAN-configuratio...

-As suggested by @AEK  run sniffer on each hop try to start ping from one of the ends to other and verify were the connection is dropping based on that you can verify the flow of traffic.

diagnose sniffer packet any "host x.x.x.x and host y.y.y.y" 4 0 l  

ctrl+c to stop sniffer [x.x.x.x is IP address from source and y.y.y.y destination]

Best regards,

Manasa.

If you feel the above steps helped resolve the issue, mark the reply as solved so that other customers can get it easily while searching for similar scenarios.

[usmansa1]

Hi, we tried sniffer on faulty side but it didnt help, we can see the ARP request coming from other city and it also responding back but dont understand why it is not going out of the ISP link

[ebilcari]

You can check the ARP tables of the FGTs and the MAC address table in the two edge switches. Are the two FGTs in the same L2 broadcast network? If yes both switches should have the MAC addresses of the FGTs.

[usmansa1]

When we changed ISP link to second ISP it started working again 

[funkylicious]

Hi,

Which device does the VxLAN encap/decap ? I would asume that the ISP routers are doing that and the rest of the devices on the network only 'speak' VLAN ?

If so, assuming that on the sw the port where FG2 connects to a port in access mode in the same VLAN ID on which you tested/create the SVI's from FGT1 which worked ?