Hello,
I’m simulating a scenario for a customer with GNS3. Each “site” has a FortiGate with two VDOMs: root and internal.
internal has 3 subnets:
[ul]The VXLAN VTEP’s are the internal VDOM inter-VDOM link IP addresses at each site (192.168.1.254/30, 192.168.2.254/30).
The only policy at internal is one that allows traffic from anywhere to anywhere (src: any, dst: any).
Now for the root VDOM: it has 3 links to reach site 2, each one has a VPN and all VPNS are in a SD-WAN interface.
I can:
[ul]However, I cannot ping from PC1 (192.168.1.3) to FW_Sitio2’s IP (192.168.1.2). ARP works, I can see the 192.168.1.2’s MAC in the ARP table, but the pings never leave FW_Sitio1.
If you can't see the image, it's at: https://share.getcloudapp.com/BluZyPJ0
I debugged this and I can see that the packets will not leave FW_Sitio1, this is what I got with debug flow:
id=20085 trace_id=1 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4814->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=1 func=init_ip_session_common line=5666 msg="allocate a new session-00001617" id=20085 trace_id=1 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=1 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=1 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=1 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=1 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=1 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=1 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"
id=20085 trace_id=2 func=print_pkt_detail line=5501 msg="vd-root:0 received a packet(proto=17, 192.168.1.254:4814->192.168.2.254:4789) from int-ext0. " id=20085 trace_id=2 func=__iprope_check line=2128 msg="gnum-100009, check-ffffffffa0023ef1" id=20085 trace_id=2 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=20085 trace_id=2 func=init_ip_session_common line=5666 msg="allocate a new session-00001618" id=20085 trace_id=2 func=iprope_dnat_check line=4882 msg="in-[int-ext0], out-[]" id=20085 trace_id=2 func=iprope_dnat_check line=4895 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=2 func=vf_ip_route_input_common line=2596 msg="find a route: flag=04000000 gw-192.168.2.130 via inter_sitio2" id=20085 trace_id=2 func=iprope_fwd_check line=731 msg="in-[int-ext0], out-[inter_sitio2], skb_flags-02000000, vid-0, app_id: 0, url_cat_id: 0" id=20085 trace_id=2 func=__iprope_tree_check line=554 msg="gnum-100004, use addr/intf hash, len=2" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-matched, act-accept" id=20085 trace_id=2 func=__iprope_user_identity_check line=1697 msg="ret-matched" id=20085 trace_id=2 func=__iprope_check line=2128 msg="gnum-4e20, check-ffffffffa0025b48" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check_one_policy line=1889 msg="checked gnum-4e20 policy-6, ret-no-match, act-accept" id=20085 trace_id=2 func=__iprope_check line=2147 msg="gnum-4e20 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000" id=20085 trace_id=2 func=__iprope_check_one_policy line=2099 msg="policy-1 is matched, act-accept" id=20085 trace_id=2 func=iprope_fwd_auth_check line=786 msg="after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-1" id=20085 trace_id=2 func=fw_forward_handler line=771 msg="Allowed by Policy-1:" id=20085 trace_id=2 func=ipsecdev_hard_start_xmit line=777 msg="enter IPsec interface-inter_sitio2" id=20085 trace_id=2 func=esp_output4 line=904 msg="IPsec encrypt/auth" id=20085 trace_id=2 func=ipsec_output_finish line=622 msg="send to 192.168.122.63 via intf-port10"
id=20085 trace_id=5 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4815->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=5 func=init_ip_session_common line=5666 msg="allocate a new session-0000161b" id=20085 trace_id=5 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=5 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=5 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=5 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=5 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=5 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=5 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"
id=20085 trace_id=6 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4816->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=6 func=init_ip_session_common line=5666 msg="allocate a new session-0000161d" id=20085 trace_id=6 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=6 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=6 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=6 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=6 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=6 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=6 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"
id=20085 trace_id=7 func=print_pkt_detail line=5501 msg="vd-internal:0 received a packet(proto=17, 192.168.1.254:4817->192.168.2.254:4789) from vx-lan1. " id=20085 trace_id=7 func=init_ip_session_common line=5666 msg="allocate a new session-0000161e" id=20085 trace_id=7 func=iprope_dnat_check line=4882 msg="in-[], out-[int-ext1]" id=20085 trace_id=7 func=iprope_dnat_check line=4895 msg="result: skb_flags-00000000, vid-0, ret-no-match, act-accept, flag-00000000" id=20085 trace_id=7 func=__iprope_check line=2128 msg="gnum-100004, check-ffffffffa003c260" id=20085 trace_id=7 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-1, ret-no-match, act-drop" id=20085 trace_id=7 func=__iprope_check_one_policy line=1889 msg="checked gnum-100004 policy-0, ret-no-match, act-drop" id=20085 trace_id=7 func=__iprope_check line=2147 msg="gnum-100004 check result: ret-no-match, act-drop, flag-00000000, flag2-00000000" id=20085 trace_id=7 func=iprope_policy_group_check line=4345 msg="after check: ret-no-match, act-drop, flag-00000000, flag2-00000000"
Any help will be appreciated.
Thanks, Max
Nominating a forum post submits a request to create a new Knowledge Article based on the forum post topic. Please ensure your nomination includes a solution within the reply.
1st nice drawing
2nd can you explain the SDWAN and can you eliminate and test with a single link from location A to Z
Lastly, in the route table can you packet dump on either side of the VXLAN to see if packets are getting to and carried thru
diag sniffer packet vx-lan1
And last have you ran "diag vpn tunnel list" and review the tunnel count if any?
e.g
''diag vpn tunnel list | grep bytes"
Ken Felix
PCNSE
NSE
StrongSwan
Ken,
Thanks for the reply, I investigated further and found this so far:
First some relevant data:
FW_Sitio1 (internal) # diagnose sys vxlan fdb list vx-lan1 mac=00:00:00:00:00:00 state=0x0082 flags=0x00 remote_ip=192.168.2.254 port=4789 vni=100 ifindex=18 mac=0c:6b:24:7c:7c:00 state=0x0002 flags=0x00 remote_ip=192.168.2.254 port=4789 vni=100 ifindex=18
FW_Sitio2 (internal) # diagnose sys vxlan fdb list vx-lan1 mac=00:00:00:00:00:00 state=0x0082 flags=0x00 remote_ip=192.168.1.254 port=4789 vni=100 ifindex=18 mac=0c:6b:24:a1:87:00 state=0x0002 flags=0x00 remote_ip=192.168.1.254 port=4789 vni=100 ifindex=18
VPCS> arp
0c:6b:24:7c:7c:00 192.168.1.2 expires in 32 seconds 0c:6b:24:a1:87:00 192.168.1.1 expires in 66 seconds
I removed the SD-WAN for now, all I use now is mpls-sitioX. Why the SD-WAN? The customer has 4 sites with one or two paths to each other. To increase redundancy we will add an Internet VPN to these paths. So, one site can have from one up to three ways of reaching another site. In order to leave the ISPs outside the routing configuration (believe me, I have enough material for a complete season of "Tales from the Crypt" with this), we will use VPNs in the existing connections. Wrapping this up: the SD-WAN will have 3 VPNS.
1st) That drawing is a GNS3 environment, I'm not that skilled at drawing :)
2nd) What I learned from the packet captures is:
This is how a ping from PC1 to FW_Sitio2 looks like:
2020-04-16 17:57:21.953601 port1 in arp who-has 192.168.1.2 (ff:ff:ff:ff:ff:ff) tell 192.168.1.3 2020-04-16 17:57:21.953621 vx-lan1 out arp who-has 192.168.1.2 (ff:ff:ff:ff:ff:ff) tell 192.168.1.3 2020-04-16 17:57:21.953672 lan1 in arp who-has 192.168.1.2 (ff:ff:ff:ff:ff:ff) tell 192.168.1.3 2020-04-16 17:57:21.955282 vx-lan1 in arp reply 192.168.1.2 is-at c:6b:24:7c:7c:0 2020-04-16 17:57:21.955286 port1 out arp reply 192.168.1.2 is-at c:6b:24:7c:7c:0
2020-04-16 17:57:21.956365 port1 in 192.168.1.3 -> 192.168.1.2: icmp: echo request 2020-04-16 17:57:21.956370 vx-lan1 out 192.168.1.3 -> 192.168.1.2: icmp: echo request
I captured traffic and converted it to wireshark, captures are at https://druidics.s3-us-we....com/vxlan_capture.zip if you want to see.
[ul]It looks like the root vdom is blocking them :o
Thanks, Max
The plot thickens...
I'll reinstall FW_Sitio1...
Question did you runa diag sniffer packet on the remote firewall to see if that icmp request comes in and if it leaves the firewall
e.g
diag sniffer packet any "icmp" 5
Ken Felix
PCNSE
NSE
StrongSwan
I reinstalled FW_Sitio1 from scratch, and the original problem is still happening.
The sniffer didn't show anything but the ARP at the other end when PC1 was pinging.
Did you do a "diag vpn tunnel list" and compare SPI for in/out ----> out/in ?
Also make sure nothing funky with blocking local ESP traffic between gateways. if that echo-request was encapsulated, it should sown a decryption at the fare end.
You might want find the wan interface and do a dump also at that point
e.g
diag sniffer packet wan1 " proto 50 and host x.x.x.x" # x.x.x.x would be the other gateway that is sending the ping in the tunnel.
Ken Felix
PCNSE
NSE
StrongSwan
duplicated
PCNSE
NSE
StrongSwan
Ken,
The packet never got to the remote firewall. Today I did this:
[ul]The captures are at https://pastebin.com/m31rkuvC, the TL;DR is: for ARP requests I see the packets leaving through the "mpls_sitio2" tunnel at the root VDOM. For ICMP packets they go up to "int-ext0" in the root VDOM, but not further.
The capture has FW_Sitio2's point of view at the end.
Thanks, Max
I finally got a reply from the TAC, and they made me configure the inter-vdom link with the "set type ethernet" option, ie:
config system vdom-link edit "int-ext" set type ethernet next end
And just like that it started working. I don't know how this relates to a switch interface with a physical port and a vxlan tunnel that uses a loopback as VTEP, but it does.
I'm posting this just in case someone stumbles upon this problem.
Thanks for everyone who replied this post.
Max
Select Forum Responses to become Knowledge Articles!
Select the “Nominate to Knowledge Base” button to recommend a forum post to become a knowledge article.
User | Count |
---|---|
1679 | |
1085 | |
752 | |
446 | |
226 |
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.
Copyright 2024 Fortinet, Inc. All Rights Reserved.