Skip to main content
5q46n2te8jPWJY
5q46n2te8jPWJY
Explorer II
October 22, 2024
Solved

Issue with Cross-VLAN Communication over VXLAN/IPSEC between Two Sites

  • October 22, 2024
  • 6 replies
  • 7030 views

Hello,

 

I am encountering an issue with my configuration that I am unable to resolve.

 

I have two sites connected with VXLAN over IPSEC. On each of my sites, I have two VLANs (VLAN 10 and VLAN 20).

 

VXLAN Fortigate.drawio.png

 

Here are my tests:

 

  • SITE A / VLAN 10 successfully pings SITE B / VLAN 10
  • SITE A / VLAN 20 successfully pings SITE B / VLAN 20
  • SITE A / VLAN 10 successfully pings SITE A / VLAN 20
  • SITE A / VLAN 20 successfully pings SITE A / VLAN 10
  • SITE B / VLAN 10 successfully pings SITE A / VLAN 10
  • SITE B / VLAN 20 successfully pings SITE A / VLAN 20
  • SITE B / VLAN 10 successfully pings SITE B / VLAN 20
  • SITE B / VLAN 20 successfully pings SITE B / VLAN 10

However,

 

  • SITE A / VLAN 10 cannot ping SITE B / VLAN 20
  • SITE A / VLAN 20 cannot ping SITE B / VLAN 10
  • SITE B / VLAN 10 cannot ping SITE A / VLAN 20
  • SITE B / VLAN 20 cannot ping SITE A / VLAN 10

Do you have any idea why?

 

Thanks for your help!

Best answer by 5q46n2te8jPWJY

Thank you to the TAC support engineers who, after quite a bit of investigation, detected a duplicate MAC address between the two FortiGate devices.

 

A big thank you as well to everyone who helped me with the search!

6 replies

HarshChavda
Staff
HarshChavda
Staff
October 22, 2024

Hello @5q46n2te8jPWJY ,

 

Are you using 0.0.0.0/0 in phase2 selector of IPSEC tunnel, if not can you verify if these VLANs are included. We can also check with running a sniffer on both side.

Reference: https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-Routing-Issue/ta-p/195727

5q46n2te8jPWJY
5q46n2te8jPWJYAuthor
Explorer II
October 25, 2024

I confirm that I am indeed using the correct phase2 selector, as recommended, with 0.0.0.0/0.

hazim
Staff
hazim
Staff
October 23, 2024

Hello @5q46n2te8jPWJY 

 

You can run debug flow to see the traffic outgoing/incoming to verify:

 

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Using-filters-to-review-traffic-traversing-the/ta-p/195025

5q46n2te8jPWJY
5q46n2te8jPWJYAuthor
Explorer II
October 25, 2024

.

tachen
tachen
New Member
October 23, 2024

How you gateway design each VLAN, do you use EVPN control plane?

 

FortiOS 7.4.5 not support IRB nor anycast gateway.

5q46n2te8jPWJY
5q46n2te8jPWJYAuthor
Explorer II
October 25, 2024

I believe you might be mistaken. In the video here, it seems to work without needing that specific configuration like IRB or anycast gateway.

5q46n2te8jPWJY
5q46n2te8jPWJYAuthor
Explorer II
October 25, 2024

@hazim 

 

When I run a ping from SITE A / VLAN 10 to SITE B / VLAN 20,

 

On site A Fortigate CLI, I see that :

 

 

diagnose debug flow filter clear diagnose debug flow filter daddr 10.1.20.2 diagnose debug enable diagnose debug flow trace start 5
id=65308 trace_id=137 func=print_pkt_detail line=5920 msg="vd-VDOM1:0 received a packet(proto=1, 10.1.10.1:16213->10.1.20.2:2048) tun_id=0.0.0.0 from VXLAN10-SW. type=8, code=0, id=16213, seq=1." id=65308 trace_id=137 func=init_ip_session_common line=6110 msg="allocate a new session-210f0670" id=65308 trace_id=137 func=__vf_ip_route_input_rcu line=1988 msg="find a route: flag=00000000 gw-0.0.0.0 via VXLAN20-SW" id=65308 trace_id=137 func=__iprope_tree_check line=524 msg="gnum-100004, use int hash, slot=5, len=9" id=65308 trace_id=137 func=fw_forward_handler line=998 msg="Allowed by Policy-247:" id=65308 trace_id=137 func=ip_session_confirm_final line=3128 msg="npu_state=0x100, hook=4" id=65308 trace_id=138 func=print_pkt_detail line=5920 msg="vd-VDOM1:0 received a packet(proto=1, 10.1.10.1:16213->10.1.20.2:2048) tun_id=0.0.0.0 from VXLAN10-SW. type=8, code=0, id=16213, seq=2." id=65308 trace_id=138 func=resolve_ip_tuple_fast line=6013 msg="Find an existing session, id-210f0670, original direction" id=65308 trace_id=138 func=npu_handle_session44 line=1342 msg="Trying to offloading session from VXLAN10-SW to VXLAN20-SW, skb.npu_flag=00000400 ses.state=00000200 ses.npu_state=0x00000100" id=65308 trace_id=138 func=fw_forward_dirty_handler line=444 msg="state=00000200, state2=00000000, npu_state=00000100" id=65308 trace_id=139 func=print_pkt_detail line=5920 msg="vd-VDOM1:0 received a packet(proto=1, 10.1.10.1:16213->10.1.20.2:2048) tun_id=0.0.0.0 from VXLAN10-SW. type=8, code=0, id=16213, seq=3." id=65308 trace_id=139 func=resolve_ip_tuple_fast line=6013 msg="Find an existing session, id-210f0670, original direction" id=65308 trace_id=139 func=npu_handle_session44 line=1342 msg="Trying to offloading session from VXLAN10-SW to VXLAN20-SW, skb.npu_flag=00000400 ses.state=00000200 ses.npu_state=0x00000100" id=65308 trace_id=139 func=fw_forward_dirty_handler line=444 msg="state=00000200, state2=00000000, npu_state=00000100" id=65308 trace_id=140 func=print_pkt_detail line=5920 msg="vd-VDOM1:0 received a packet(proto=1, 10.1.10.1:16213->10.1.20.2:2048) tun_id=0.0.0.0 from VXLAN10-SW. type=8, code=0, id=16213, seq=4." id=65308 trace_id=140 func=resolve_ip_tuple_fast line=6013 msg="Find an existing session, id-210f0670, original direction" id=65308 trace_id=140 func=ipv4_fast_cb line=53 msg="enter fast path" id=65308 trace_id=141 func=print_pkt_detail line=5920 msg="vd-VDOM1:0 received a packet(proto=1, 10.1.10.1:16213->10.1.20.2:2048) tun_id=0.0.0.0 from VXLAN10-SW. type=8, code=0, id=16213, seq=5." id=65308 trace_id=141 func=resolve_ip_tuple_fast line=6013 msg="Find an existing session, id-210f0670, original direction" id=65308 trace_id=141 func=ipv4_fast_cb line=53 msg="enter fast path"

 

 

 On site B Fortigate, with the same CLI, I see nothing...

 

Do you have an idea ?

funkylicious
SuperUser
funkylicious
SuperUser
October 25, 2024

I recreated a similar lab like in the video, hit the same issue where i couldnt ping from S1/A to S2/B and did a debug and looked similar like yours, did a diagnose netlink brctl list and diagnose netlink brctl name host <> to confirm that I can see the mac of host in S2/B in FGT-A and then the ping worked

"jack of all trades, master of none"
5q46n2te8jPWJY
5q46n2te8jPWJYAuthor
Explorer II
October 25, 2024

Which version do you use for your lab? 

jdelafuente_FTNT
Staff & Editor
jdelafuente_FTNT
Staff & Editor
October 26, 2024

I had a similar issue few year ago, problem was in my gateways, remember if fortigate see same packet coming for second time in same direction will drop duenetwork loop, then wich and where is your gateway for each vlan?

5q46n2te8jPWJY
5q46n2te8jPWJYAuthor
Explorer II
October 26, 2024

On each fortigate, I have a software switch for each VLAN/VXLAN, wich is the gateway

 

Fortigate A :

VXLAN10-SW 10.1.10.254/24

VXLAN20-SW 10.1.20.254/24

 

Fortigate B :

VXLAN10-SW 10.1.10.254/24

VXLAN20-SW 10.1.20.254/24

 

Each PC can ping internet, so I think gateway are correct, isn't it ?

jdelafuente_FTNT
Staff & Editor
jdelafuente_FTNT
Staff & Editor
October 26, 2024

Here is the problem you have duplicated IP issue, remember it's an extended LAN same broadcast domain, you can't have same IP in FGT-A and FGT-B try this:

-> Remove interface IP in FortiGate-B, keep in FortiGate-A and try again.

Keep in mind.

You don't need IP in FGT-B communication between networks works with gateway only in FGT-A. (all outgoing traffic from FGB-B will go out through FGT-A)

In you want, it is possible to define a different IP for example VXLAN10-SW 10.1.10.253/24 but remember a network with 2 gateways represent a real network challenge to prevent asymmetric route.

Best regards.

5q46n2te8jPWJY
5q46n2te8jPWJYAuthorAnswer
Explorer II
November 21, 2024

Thank you to the TAC support engineers who, after quite a bit of investigation, detected a duplicate MAC address between the two FortiGate devices.

 

A big thank you as well to everyone who helped me with the search!

FortiGab
FortiGab
Explorer
September 4, 2025

deduplicate ip address are related to vip IPs i guess

how did u solve?

Terms & ConditionsAccessibility statement