VXLAN only works for local traffic

[Agent_1994]

Hello,

 I’m simulating a scenario for a customer with GNS3. Each “site” has a FortiGate with two VDOMs: root and internal.

internal has 3 subnets:

[ul]

port1+vx-lan1 is a software switch, the VXLAN VTEP is at the other site. intra switch policy is set to explicit.

port2 and port3 are, let’s say, “normal” networks (different addressing at each site)[/ul]

 The VXLAN VTEP’s are the internal VDOM inter-VDOM link IP addresses at each site (192.168.1.254/30, 192.168.2.254/30).

 The only policy at internal is one that allows traffic from anywhere to anywhere (src: any, dst: any).

 Now for the root VDOM: it has 3 links to reach site 2, each one has a VPN and all VPNS are in a SD-WAN interface.

 I can:

[ul]

ping from one VTEP to the other without any problem.

ping from PC1 (192.168.1.3) to FW_Sitio1 (192.168.1.1)

ping from FW_Sitio1 (192.168.1.1) to FW_Sitio2’s IP at the other end of the VXLAN tunnel (192.168.1.2).[/ul]

 However, I cannot ping from PC1 (192.168.1.3) to FW_Sitio2’s IP (192.168.1.2). ARP works, I can see the 192.168.1.2’s MAC in the ARP table, but the pings never leave FW_Sitio1.

 If you can't see the image, it's at: https://share.getcloudapp.com/BluZyPJ0

 

 I debugged this and I can see that the packets will not leave FW_Sitio1, this is what I got with debug flow:

ARP Packets (working)

 

id=20085 trace_id=1 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4814->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=1 func=init_ip_session_common line=5666 msg="allocate a new session-00001617" id=20085 trace_id=1 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=1 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=1 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=1 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=1 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=1 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=1 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"

id=20085 trace_id=2 func=print_pkt_detail line=5501 msg="vd-root:0 received a packet(proto=17, 192.168.1.254:4814->192.168.2.254:4789) from int-ext0. " id=20085 trace_id=2 func=__iprope_check line=2128 msg="gnum-100009, check-ffffffffa0023ef1" id=20085 trace_id=2 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=20085 trace_id=2 func=init_ip_session_common line=5666 msg="allocate a new session-00001618" id=20085 trace_id=2 func=iprope_dnat_check line=4882 msg="in-[int-ext0], out-[]" id=20085 trace_id=2 func=iprope_dnat_check line=4895 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=2 func=vf_ip_route_input_common line=2596 msg="find a route: flag=04000000 gw-192.168.2.130 via inter_sitio2" id=20085 trace_id=2 func=iprope_fwd_check line=731 msg="in-[int-ext0], out-[inter_sitio2], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0" id=20085 trace_id=2 func=__iprope_tree_check line=554 msg="gnum-100004, use addr/intf hash, len=2" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-matched, act-accept" id=20085 trace_id=2 func=__iprope_user_identity_check line=1697 msg="ret-matched" id=20085 trace_id=2 func=__iprope_check line=2128 msg="gnum-4e20, check-ffffffffa0025b48" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check line=2147 msg="gnum-4e20 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=20085 trace_id=2 func=__iprope_check_one_policy line=2099 msg="policy-1 is matched, act-accept" id=20085 trace_id=2 func=iprope_fwd_auth_check line=786 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-1" id=20085 trace_id=2 func=fw_forward_handler line=771 msg="Allowed by Policy-1:" id=20085 trace_id=2 func=ipsecdev_hard_start_xmit line=777 msg="enter IPsec interface-inter_sitio2" id=20085 trace_id=2 func=esp_output4 line=904 msg="IPsec encrypt/auth" id=20085 trace_id=2 func=ipsec_output_finish line=622 msg="send to 192.168.122.63 via intf-port10"

Ping (not working)

 

id=20085 trace_id=5 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4815->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=5 func=init_ip_session_common line=5666 msg="allocate a new session-0000161b" id=20085 trace_id=5 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=5 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=5 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=5 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=5 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=5 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=5 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"

id=20085 trace_id=6 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4816->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=6 func=init_ip_session_common line=5666 msg="allocate a new session-0000161d" id=20085 trace_id=6 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=6 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=6 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=6 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=6 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=6 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=6 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"

id=20085 trace_id=7 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4817->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=7 func=init_ip_session_common line=5666 msg="allocate a new session-0000161e" id=20085 trace_id=7 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=7 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=7 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=7 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=7 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=7 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=7 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"

 Any help will be appreciated.

Thanks, Max

[emnoc]

1st nice drawing 

2nd can you explain the SDWAN  and can you eliminate and test with a single link from location A to Z

Lastly, in the route table can you packet dump on either side of the  VXLAN to see if packets are getting to and carried thru

 diag sniffer packet vx-lan1 

And last have you ran "diag vpn tunnel list" and review the tunnel count if any?

e.g

                                ''diag vpn tunnel list | grep bytes"

Ken Felix

[Agent_1994]

Ken,

 Thanks for the reply, I investigated further and found this so far:

 First some relevant data:

MAC addresses

[ul]

192.168.1.1 (FW_Sitio1/internal/port1): 0c:6b:24:a1:87:00

192.168.1.2 (FW_Sitio2/internal/port1): 0c:6b:24:7c:7c:00

192.168.1.3 (PC1): 00:50:79:66:68:00

(PC2 never got an IP address): 00:50:79:66:68:01[/ul]

VXLAN FDB

FW_Sitio1 after pinging FW_Sitio2

FW_Sitio1 (internal) # diagnose sys vxlan fdb list vx-lan1 mac=00:00:00:00:00:00 state=0x0082 flags=0x00 remote_ip=192.168.2.254 port=4789 vni=100 ifindex=18 mac=0c:6b:24:7c:7c:00 state=0x0002 flags=0x00 remote_ip=192.168.2.254 port=4789 vni=100 ifindex=18

FW_Sitio2 after pinging PC1 (failed) and FW_Sitio1

FW_Sitio2 (internal) # diagnose sys vxlan fdb list vx-lan1 mac=00:00:00:00:00:00 state=0x0082 flags=0x00 remote_ip=192.168.1.254 port=4789 vni=100 ifindex=18 mac=0c:6b:24:a1:87:00 state=0x0002 flags=0x00 remote_ip=192.168.1.254 port=4789 vni=100 ifindex=18

VPC1 (Sitio1) after pinging FW_Sitio1 and FW_Sitio2:

VPCS> arp

0c:6b:24:7c:7c:00 192.168.1.2 expires in 32 seconds 0c:6b:24:a1:87:00 192.168.1.1 expires in 66 seconds

About your questions...

 I removed the SD-WAN for now, all I use now is mpls-sitioX. Why the SD-WAN? The customer has 4 sites with one or two paths to each other. To increase redundancy we will add an Internet VPN to these paths. So, one site can have from one up to three ways of reaching another site. In order to leave the ISPs outside the routing configuration (believe me, I have enough material for a complete season of "Tales from the Crypt" with this), we will use VPNs in the existing connections. Wrapping this up: the SD-WAN will have 3 VPNS.

1st) That drawing is a GNS3 environment, I'm not that skilled at drawing :)

2nd) What I learned from the packet captures is:

 This is how a ping from PC1 to FW_Sitio2 looks like:

2020-04-16 17:57:21.953601 port1 in arp who-has 192.168.1.2 (ff:ff:ff:ff:ff:ff) tell 192.168.1.3 2020-04-16 17:57:21.953621 vx-lan1 out arp who-has 192.168.1.2 (ff:ff:ff:ff:ff:ff) tell 192.168.1.3 2020-04-16 17:57:21.953672 lan1 in arp who-has 192.168.1.2 (ff:ff:ff:ff:ff:ff) tell 192.168.1.3 2020-04-16 17:57:21.955282 vx-lan1 in arp reply 192.168.1.2 is-at c:6b:24:7c:7c:0 2020-04-16 17:57:21.955286 port1 out arp reply 192.168.1.2 is-at c:6b:24:7c:7c:0

2020-04-16 17:57:21.956365 port1 in 192.168.1.3 -> 192.168.1.2: icmp: echo request 2020-04-16 17:57:21.956370 vx-lan1 out 192.168.1.3 -> 192.168.1.2: icmp: echo request

 I captured traffic and converted it to wireshark, captures are at https://druidics.s3-us-we....com/vxlan_capture.zip if you want to see.

[ul]

When I captured pings (vxlan) from FW_Sitio1 to FW_Sitio2, I saw the packets going this path: FW_Sitio1/internal/port1 -> FW_Sitio1/internal/vx-lan1 -> encapsulation -> FW_Sitio1/internal/int-ext1 (vdom link) -> FW_Sitio1/root/int-ext0 (vdom link) -> FW_Sitio1/root/mpls-sitio2 (vpn).

When I captured pings from PC1 to FW_Sitio2's IP, I saw the same path, but they stop when arriving at int-ext0. They never go further.[/ul]

 It looks like the root vdom is blocking them :o

Thanks, Max

[Agent_1994]

The plot thickens...

I'll reinstall FW_Sitio1...

[emnoc]

Question did you runa diag sniffer packet on the remote firewall to see if that icmp request comes in and if it leaves the firewall

e.g

   diag sniffer packet any "icmp" 5 

Ken Felix

[Agent_1994]

I reinstalled FW_Sitio1 from scratch, and the original problem is still happening.

The sniffer didn't show anything but the ARP at the other end when PC1 was pinging.

[emnoc]

Did you do a "diag vpn tunnel list" and compare  SPI for in/out ----> out/in ?

Also make sure nothing funky with blocking local ESP traffic between gateways. if that echo-request was encapsulated, it should sown a decryption at the fare end.

You might want find the wan interface and do a dump also at that point

e.g

  diag sniffer packet wan1 " proto 50 and host x.x.x.x"  # x.x.x.x would be the other gateway that is sending the ping in the tunnel.

Ken Felix

[emnoc]

duplicated

[Agent_1994]

Ken,

 The packet never got to the remote firewall. Today I did this:

[ul]

rebuilt the lab using vmware esxi, this was done to discard a GNS3 bug and/or KVM VM bug. same results.

reconfigured the lab to use a VPN with VXLAN encapsulation (recipe at https://docs.fortinet.com/document/fortigate/6.2.0/cookbook/821119/vxlan-over-ipsec-tunnel), from the "internal" vdom.

did a small lab without VDOMS, it worked.[/ul]

 The captures are at https://pastebin.com/m31rkuvC, the TL;DR is: for ARP requests I see the packets leaving through the "mpls_sitio2" tunnel at the root VDOM. For ICMP packets they go up to "int-ext0" in the root VDOM, but not further.

 The capture has FW_Sitio2's point of view at the end.

Thanks, Max

[Agent_1994]

I finally got a reply from the TAC, and they made me configure the inter-vdom link with the "set type ethernet" option, ie:

config system vdom-link     edit "int-ext"         set type ethernet     next end

 And just like that it started working. I don't know how this relates to a switch interface with a physical port and a vxlan tunnel that uses a loopback as VTEP, but it does.

 I'm posting this just in case someone stumbles upon this problem.

 Thanks for everyone who replied this post.

Max