Skip to main content
Agent_1994
Agent_1994
New Member
April 15, 2020
Question

VXLAN only works for local traffic

  • April 15, 2020
  • 2 replies
  • 16396 views

Hello,

 

 I’m simulating a scenario for a customer with GNS3. Each “site” has a FortiGate with two VDOMs: root and internal.

internal has 3 subnets:

[ul]
  • port1+vx-lan1 is a software switch, the VXLAN VTEP is at the other site. intra switch policy is set to explicit.
  • port2 and port3 are, let’s say, “normal” networks (different addressing at each site)[/ul]

     The VXLAN VTEP’s are the internal VDOM inter-VDOM link IP addresses at each site (192.168.1.254/30, 192.168.2.254/30).

     

     The only policy at internal is one that allows traffic from anywhere to anywhere (src: any, dst: any).

     

     Now for the root VDOM: it has 3 links to reach site 2, each one has a VPN and all VPNS are in a SD-WAN interface.

     

     I can:

    [ul]
  • ping from one VTEP to the other without any problem.
  • ping from PC1 (192.168.1.3) to FW_Sitio1 (192.168.1.1)
  • ping from FW_Sitio1 (192.168.1.1) to FW_Sitio2’s IP at the other end of the VXLAN tunnel (192.168.1.2).[/ul]

     However, I cannot ping from PC1 (192.168.1.3) to FW_Sitio2’s IP (192.168.1.2). ARP works, I can see the 192.168.1.2’s MAC in the ARP table, but the pings never leave FW_Sitio1.

     

     If you can't see the image, it's at: https://share.getcloudapp.com/BluZyPJ0

     

     

     

     I debugged this and I can see that the packets will not leave FW_Sitio1, this is what I got with debug flow:

    ARP Packets (working)

     

    id=20085 trace_id=1 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4814->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=1 func=init_ip_session_common line=5666 msg="allocate a new session-00001617" id=20085 trace_id=1 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=1 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=1 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=1 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=1 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=1 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=1 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"

    id=20085 trace_id=2 func=print_pkt_detail line=5501 msg="vd-root:0 received a packet(proto=17, 192.168.1.254:4814->192.168.2.254:4789) from int-ext0. " id=20085 trace_id=2 func=__iprope_check line=2128 msg="gnum-100009, check-ffffffffa0023ef1" id=20085 trace_id=2 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=20085 trace_id=2 func=init_ip_session_common line=5666 msg="allocate a new session-00001618" id=20085 trace_id=2 func=iprope_dnat_check line=4882 msg="in-[int-ext0], out-[]" id=20085 trace_id=2 func=iprope_dnat_check line=4895 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=2 func=vf_ip_route_input_common line=2596 msg="find a route: flag=04000000 gw-192.168.2.130 via inter_sitio2" id=20085 trace_id=2 func=iprope_fwd_check line=731 msg="in-[int-ext0], out-[inter_sitio2], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0" id=20085 trace_id=2 func=__iprope_tree_check line=554 msg="gnum-100004, use addr/intf hash, len=2" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-matched, act-accept" id=20085 trace_id=2 func=__iprope_user_identity_check line=1697 msg="ret-matched" id=20085 trace_id=2 func=__iprope_check line=2128 msg="gnum-4e20, check-ffffffffa0025b48" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check line=2147 msg="gnum-4e20 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=20085 trace_id=2 func=__iprope_check_one_policy line=2099 msg="policy-1 is matched, act-accept" id=20085 trace_id=2 func=iprope_fwd_auth_check line=786 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-1" id=20085 trace_id=2 func=fw_forward_handler line=771 msg="Allowed by Policy-1:" id=20085 trace_id=2 func=ipsecdev_hard_start_xmit line=777 msg="enter IPsec interface-inter_sitio2" id=20085 trace_id=2 func=esp_output4 line=904 msg="IPsec encrypt/auth" id=20085 trace_id=2 func=ipsec_output_finish line=622 msg="send to 192.168.122.63 via intf-port10"

     

    Ping (not working)

     

    id=20085 trace_id=5 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4815->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=5 func=init_ip_session_common line=5666 msg="allocate a new session-0000161b" id=20085 trace_id=5 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=5 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=5 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=5 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=5 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=5 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=5 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"

    id=20085 trace_id=6 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4816->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=6 func=init_ip_session_common line=5666 msg="allocate a new session-0000161d" id=20085 trace_id=6 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=6 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=6 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=6 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=6 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=6 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=6 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"

    id=20085 trace_id=7 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4817->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=7 func=init_ip_session_common line=5666 msg="allocate a new session-0000161e" id=20085 trace_id=7 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=7 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=7 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=7 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=7 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=7 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=7 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"

     

     Any help will be appreciated.

     

    Thanks, Max

    • 2 replies

    emnoc
    emnoc
    New Member
    April 15, 2020

    1st nice drawing 

     

    2nd can you explain the SDWAN  and can you eliminate and test with a single link from location A to Z

     

    Lastly, in the route table can you packet dump on either side of the  VXLAN to see if packets are getting to and carried thru

     diag sniffer packet vx-lan1 

     

    And last have you ran "diag vpn tunnel list" and review the tunnel count if any?

     

    e.g

     

                                    ''diag vpn tunnel list | grep bytes"

     

    Ken Felix

     

    Agent_1994
    Agent_1994Author
    New Member
    April 16, 2020

    Ken,

     Thanks for the reply, I investigated further and found this so far:

     

     First some relevant data:

    MAC addresses

    [ul]

  • 192.168.1.1 (FW_Sitio1/internal/port1): 0c:6b:24:a1:87:00

  • 192.168.1.2 (FW_Sitio2/internal/port1): 0c:6b:24:7c:7c:00
  • 192.168.1.3 (PC1): 00:50:79:66:68:00
  • (PC2 never got an IP address): 00:50:79:66:68:01[/ul]

    VXLAN FDB

    FW_Sitio1 after pinging FW_Sitio2

    FW_Sitio1 (internal) # diagnose sys vxlan fdb list vx-lan1 mac=00:00:00:00:00:00 state=0x0082 flags=0x00 remote_ip=192.168.2.254 port=4789 vni=100 ifindex=18 mac=0c:6b:24:7c:7c:00 state=0x0002 flags=0x00 remote_ip=192.168.2.254 port=4789 vni=100 ifindex=18

    FW_Sitio2 after pinging PC1 (failed) and FW_Sitio1

    FW_Sitio2 (internal) # diagnose sys vxlan fdb list vx-lan1 mac=00:00:00:00:00:00 state=0x0082 flags=0x00 remote_ip=192.168.1.254 port=4789 vni=100 ifindex=18 mac=0c:6b:24:a1:87:00 state=0x0002 flags=0x00 remote_ip=192.168.1.254 port=4789 vni=100 ifindex=18

    VPC1 (Sitio1) after pinging FW_Sitio1 and FW_Sitio2:

    VPCS> arp

    0c:6b:24:7c:7c:00 192.168.1.2 expires in 32 seconds 0c:6b:24:a1:87:00 192.168.1.1 expires in 66 seconds

    About your questions...

     I removed the SD-WAN for now, all I use now is mpls-sitioX. Why the SD-WAN? The customer has 4 sites with one or two paths to each other. To increase redundancy we will add an Internet VPN to these paths. So, one site can have from one up to three ways of reaching another site. In order to leave the ISPs outside the routing configuration (believe me, I have enough material for a complete season of "Tales from the Crypt" with this), we will use VPNs in the existing connections. Wrapping this up: the SD-WAN will have 3 VPNS.

     

    1st) That drawing is a GNS3 environment, I'm not that skilled at drawing :)

     

    2nd) What I learned from the packet captures is:

     

     This is how a ping from PC1 to FW_Sitio2 looks like:

     

    2020-04-16 17:57:21.953601 port1 in arp who-has 192.168.1.2 (ff:ff:ff:ff:ff:ff) tell 192.168.1.3 2020-04-16 17:57:21.953621 vx-lan1 out arp who-has 192.168.1.2 (ff:ff:ff:ff:ff:ff) tell 192.168.1.3 2020-04-16 17:57:21.953672 lan1 in arp who-has 192.168.1.2 (ff:ff:ff:ff:ff:ff) tell 192.168.1.3 2020-04-16 17:57:21.955282 vx-lan1 in arp reply 192.168.1.2 is-at c:6b:24:7c:7c:0 2020-04-16 17:57:21.955286 port1 out arp reply 192.168.1.2 is-at c:6b:24:7c:7c:0

    2020-04-16 17:57:21.956365 port1 in 192.168.1.3 -> 192.168.1.2: icmp: echo request 2020-04-16 17:57:21.956370 vx-lan1 out 192.168.1.3 -> 192.168.1.2: icmp: echo request

     

     I captured traffic and converted it to wireshark, captures are at https://druidics.s3-us-we....com/vxlan_capture.zip if you want to see.

    [ul]
  • When I captured pings (vxlan) from FW_Sitio1 to FW_Sitio2, I saw the packets going this path: FW_Sitio1/internal/port1 -> FW_Sitio1/internal/vx-lan1 -> encapsulation -> FW_Sitio1/internal/int-ext1 (vdom link) -> FW_Sitio1/root/int-ext0 (vdom link) -> FW_Sitio1/root/mpls-sitio2 (vpn).
  • When I captured pings from PC1 to FW_Sitio2's IP, I saw the same path, but they stop when arriving at int-ext0. They never go further.[/ul]

     It looks like the root vdom is blocking them :o

     

    Thanks, Max

    • Agent_1994
    Agent_1994Author
    New Member
    April 17, 2020

    The plot thickens...

    I'll reinstall FW_Sitio1...

     

    Agent_1994
    Agent_1994Author
    New Member
    April 24, 2020

    I finally got a reply from the TAC, and they made me configure the inter-vdom link with the "set type ethernet" option, ie:

     

    config system vdom-link     edit "int-ext"         set type ethernet     next end

     

     And just like that it started working. I don't know how this relates to a switch interface with a physical port and a vxlan tunnel that uses a loopback as VTEP, but it does.

     

     I'm posting this just in case someone stumbles upon this problem.

     

     Thanks for everyone who replied this post.

     

    Max

    emnoc
    emnoc
    New Member
    April 24, 2020

    Strange but glad it worked out.

     

    Ken Felix

    Terms & ConditionsAccessibility statement