VXLAN only works for local traffic

[Agent_1994]

Hello,

 I’m simulating a scenario for a customer with GNS3. Each “site” has a FortiGate with two VDOMs: root and internal.

internal has 3 subnets:

[ul]

port1+vx-lan1 is a software switch, the VXLAN VTEP is at the other site. intra switch policy is set to explicit.

port2 and port3 are, let’s say, “normal” networks (different addressing at each site)[/ul]

 The VXLAN VTEP’s are the internal VDOM inter-VDOM link IP addresses at each site (192.168.1.254/30, 192.168.2.254/30).

 The only policy at internal is one that allows traffic from anywhere to anywhere (src: any, dst: any).

 Now for the root VDOM: it has 3 links to reach site 2, each one has a VPN and all VPNS are in a SD-WAN interface.

 I can:

[ul]

ping from one VTEP to the other without any problem.

ping from PC1 (192.168.1.3) to FW_Sitio1 (192.168.1.1)

ping from FW_Sitio1 (192.168.1.1) to FW_Sitio2’s IP at the other end of the VXLAN tunnel (192.168.1.2).[/ul]

 However, I cannot ping from PC1 (192.168.1.3) to FW_Sitio2’s IP (192.168.1.2). ARP works, I can see the 192.168.1.2’s MAC in the ARP table, but the pings never leave FW_Sitio1.

 If you can't see the image, it's at: https://share.getcloudapp.com/BluZyPJ0

 

 I debugged this and I can see that the packets will not leave FW_Sitio1, this is what I got with debug flow:

ARP Packets (working)

 

id=20085 trace_id=1 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4814->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=1 func=init_ip_session_common line=5666 msg="allocate a new session-00001617" id=20085 trace_id=1 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=1 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=1 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=1 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=1 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=1 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=1 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"

id=20085 trace_id=2 func=print_pkt_detail line=5501 msg="vd-root:0 received a packet(proto=17, 192.168.1.254:4814->192.168.2.254:4789) from int-ext0. " id=20085 trace_id=2 func=__iprope_check line=2128 msg="gnum-100009, check-ffffffffa0023ef1" id=20085 trace_id=2 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=20085 trace_id=2 func=init_ip_session_common line=5666 msg="allocate a new session-00001618" id=20085 trace_id=2 func=iprope_dnat_check line=4882 msg="in-[int-ext0], out-[]" id=20085 trace_id=2 func=iprope_dnat_check line=4895 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=2 func=vf_ip_route_input_common line=2596 msg="find a route: flag=04000000 gw-192.168.2.130 via inter_sitio2" id=20085 trace_id=2 func=iprope_fwd_check line=731 msg="in-[int-ext0], out-[inter_sitio2], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0" id=20085 trace_id=2 func=__iprope_tree_check line=554 msg="gnum-100004, use addr/intf hash, len=2" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-matched, act-accept" id=20085 trace_id=2 func=__iprope_user_identity_check line=1697 msg="ret-matched" id=20085 trace_id=2 func=__iprope_check line=2128 msg="gnum-4e20, check-ffffffffa0025b48" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check line=2147 msg="gnum-4e20 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=20085 trace_id=2 func=__iprope_check_one_policy line=2099 msg="policy-1 is matched, act-accept" id=20085 trace_id=2 func=iprope_fwd_auth_check line=786 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-1" id=20085 trace_id=2 func=fw_forward_handler line=771 msg="Allowed by Policy-1:" id=20085 trace_id=2 func=ipsecdev_hard_start_xmit line=777 msg="enter IPsec interface-inter_sitio2" id=20085 trace_id=2 func=esp_output4 line=904 msg="IPsec encrypt/auth" id=20085 trace_id=2 func=ipsec_output_finish line=622 msg="send to 192.168.122.63 via intf-port10"

Ping (not working)

 

id=20085 trace_id=5 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4815->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=5 func=init_ip_session_common line=5666 msg="allocate a new session-0000161b" id=20085 trace_id=5 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=5 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=5 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=5 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=5 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=5 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=5 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"

id=20085 trace_id=6 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4816->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=6 func=init_ip_session_common line=5666 msg="allocate a new session-0000161d" id=20085 trace_id=6 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=6 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=6 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=6 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=6 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=6 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=6 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"

id=20085 trace_id=7 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4817->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=7 func=init_ip_session_common line=5666 msg="allocate a new session-0000161e" id=20085 trace_id=7 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=7 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=7 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=7 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=7 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=7 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=7 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"

 Any help will be appreciated.

Thanks, Max

[emnoc]

Strange but glad it worked out.

Ken Felix

[asif_janjua_88]

We are facing the similar kind of an issue where we are able to reach from PC behind a Site A firewall to PC2 behind Site B Firewall.  The MAC ARP response is properly received from one way.

However, if we ping from PC2 to PC1 then ARP reply what is MAC of PC1 is received by Site A firewall but Site A firewall is not putting it in VXLAN tunnel. Can any one help that in which case why the Firewall is not putting the MAC response in VXLAN tunnel.

[asif_janjua_88]

We have managed to resolve the issue by setting the load balancing on port group to "Route based on physical NIC based". To set the configuration, select the port group then Teaming and Failover ---> here set the load balancing to "Route based on physical NIC based".