Fortigate SD-WAN Hub cannot connect to Spokes

[BrianHJones0217]

I am trying to build a lab for SDWAN using the Fortigates and ADVPN as it is similar to a client environment that I support.  The tunnels come up fine and BGP comes up find as well. However, the PCs cannot ping each other.  The firewall rule is pretty much wide open.  All three firewalls seem to have the same symptom as I don't believe the traffic is passing from the inside interface to the ADVPN tunnel.  Here are the technical details that I have to share:

Packet Capture from Hub->SpokeA (same results for Hub->SpokeB, SpokeA->Hub, SpokeB->Hub)

Using Original Sniffing Mode
interfaces=[any]
filters=[host 192.168.10.240 and icmp]
1.735240 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2.735788 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
3.735260 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
4.736537 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
5.736059 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
6.736208 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
7.736246 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
8.736187 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
9.736327 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
10.736504 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request

Packet Capture on Dest FW shows not traffic inbound in all cases.

Diag Debug Flow Trace from Hub-SpokeA

id=65308 trace_id=72 func=init_ip_session_common line=6043 msg="allocate a new session-00000876, tun_id=0.0.0.0"
id=65308 trace_id=72 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=72 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=72 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=72 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"
id=65308 trace_id=73 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 192.168.10.240:54538->192.168.20.240:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=54538, seq=2."
id=65308 trace_id=73 func=init_ip_session_common line=6043 msg="allocate a new session-00000877, tun_id=0.0.0.0"
id=65308 trace_id=73 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=73 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=73 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=73 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"
id=65308 trace_id=74 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 192.168.10.240:54538->192.168.20.240:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=54538, seq=3."
id=65308 trace_id=74 func=init_ip_session_common line=6043 msg="allocate a new session-00000878, tun_id=0.0.0.0"
id=65308 trace_id=74 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=74 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=74 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=74 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"

[syordanov]

Hello BrianHJones0217,

So you have ADVPN topology with BGP , one HUB and several spokes.

From provided sniffer and debug flow i can see that there is allocated session :

id=65308 trace_id=73 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 192.168.10.240:54538->192.168.20.240:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=54538, seq=2."
id=65308 trace_id=73 func=init_ip_session_common line=6043 msg="allocate a new session-00000877, tun_id=0.0.0.0"
id=65308 trace_id=73 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=73 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=73 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=73 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"

Can you try the following debug flow :

SSH No1:

diagnose debug reset

diagnose debug disable

diagnose debug flow trace stop

diagnose debug flow filter clear

diagnose debug flow show iprope enable

diagnose  debug  flow show function-name enable

diagnose debug flow filter daddr 192.168.20.240

diagnose  debug  console timestamp enable

diagnose debug flow trace start 10000

diagnose debug enable

SSH No2:

diag sys session filter src XXX.XXX.XXX.XXX <---- source IP

diag sys session filter dst 192.168.20.240 <---- destination IP

diag sys session list

Do you have a valid route for 192.168.20.240, if yes  is it pointing to spokeA?

[BrianHJones0217]

Yes, there is a valid route pointing to SpokeA.  Here is the routing table for both Hub and SpokeA:

Hub Routing Table

S* 0.0.0.0/0 [1/0] via 60.120.1.1, port4, [1/0]
[1/0] via 12.24.1.1, port3, [10/0]
C 12.24.1.0/29 is directly connected, port3
C 60.120.1.0/29 is directly connected, port4
C 100.10.0.0/23 is directly connected, advpn1-hub
C 100.10.0.1/32 is directly connected, advpn1-hub
C 192.168.10.0/24 is directly connected, port1
B 192.168.20.0/24 [200/0] via 100.10.0.10 (recursive is directly connected, advpn1-hub), 00:24:39, [1/0]
B 192.168.30.0/24 [200/0] via 100.10.0.11 (recursive is directly connected, advpn1-hub), 00:35:02, [1/0]
C 200.10.0.0/23 is directly connected, advpn2-hub
C 200.10.0.1/32 is directly connected, advpn2-hub

Spoke A Routing Table

S* 0.0.0.0/0 [1/0] via 60.120.2.1, port4, [1/0]
[1/0] via 12.24.2.1, port3, [10/0]
C 12.24.2.0/29 is directly connected, port3
C 60.120.2.0/29 is directly connected, port4
S 100.10.0.1/32 [5/0] via advpn1-spoke tunnel 12.24.1.2, [1/0]
C 100.10.0.10/32 is directly connected, advpn1-spoke
B 192.168.10.0/24 [200/0] via 100.10.0.1 (recursive via advpn1-spoke tunnel 12.24.1.2), 00:24:46, [1/0]
C 192.168.20.0/24 is directly connected, port1
B 192.168.30.0/24 [200/0] via 100.10.0.11 (recursive via 60.120.2.1, port4), 00:24:46, [1/0]
S 200.10.0.1/32 [5/0] via advpn2-spoke tunnel 60.120.1.2, [1/0]
C 200.10.0.10/32 is directly connected, advpn2-spoke

I will try to get the diag debug flows shortly.  Thanks.

[BrianHJones0217]

Here is the diag debug flows:

SSH no 1:

2023-11-20 05:22:59 id=65308 trace_id=10 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 192.168.10.240:48906->192.168.20.240:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=48906, seq=9."
2023-11-20 05:22:59 id=65308 trace_id=10 func=init_ip_session_common line=6043 msg="allocate a new session-0000006c, tun_id=0.0.0.0"
2023-11-20 05:22:59 id=65308 trace_id=10 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
2023-11-20 05:22:59 id=65308 trace_id=10 func=iprope_dnat_tree_check line=824 msg="len=0"
2023-11-20 05:22:59 id=65308 trace_id=10 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2023-11-20 05:22:59 id=65308 trace_id=10 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"

SSH no 2:

FW-1 # diag sys session filter src 192.168.10.240

FW-1 # diag sys session filter dst 192.168.20.240

FW-1 # diag sys session list
total session 0

[syordanov]

Hello,

You can check the local/remote networks under phase-2 selectors on the HUB and spoke if they are 0.0.0.0/0.

Are you able to ping from the HUB with source IP 100.10.0.1 destination IP 100.10.0.10 ?

Regards,

Fortinet

[BrianHJones0217]

Here is the hub side:

name=advpn1-hub_1 ver=1 serial=6 12.24.1.2:0->12.24.2.2:0 tun_id=100.10.0.10 tun_id6=::10.0.0.6 dst_mtu=1500 dpd-link=on weight=1
bound_if=5 lgwy=static/1 tun=intf mode=dial_inst/3 encap=none/74408 options[122a8]=npu rgwy-chg frag-rfc run_state=0 role=sync-primary accept_traffic=1 overlay_id=0

parent=advpn1-hub index=1
proxyid_num=1 child_num=0 refcnt=5 ilast=0 olast=0 ad=s/1
stat: rxp=396 txp=411 rxb=24518 txb=25932
dpd: mode=on-idle on=1 idle=5000ms retry=3 count=0 seqno=59
natt: mode=none draft=0 interval=0 remote_port=0
fec: egress=0 ingress=0
proxyid=advpn1-hub proto=0 sa=1 ref=3 serial=1 ads
src: 0:0.0.0.0-255.255.255.255:0
dst: 0:0.0.0.0-255.255.255.255:0
SA: ref=3 options=20a02 type=00 soft=0 mtu=1446 expire=41519/0B replaywin=2048
seqno=19c esn=0 replaywin_lastseq=0000018d qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=43187/43200
dec: spi=beaf4a42 esp=des key=8 87bf0913737de236
ah=md5 key=16 b74a6bb29df58441d9952c9e4ad1c79d
enc: spi=3d96d73f esp=des key=8 7df0edf1ff8d2e41
ah=md5 key=16 7b381d0378a5aa40471e8c29a7a20303
dec:pkts/bytes=792/49036, enc:pkts/bytes=822/74212
npu_flag=00 npu_rgwy=12.24.2.2 npu_lgwy=12.24.1.2 npu_selid=3 dec_npuid=0 enc_npuid=0

name=advpn1-hub_0 ver=1 serial=4 12.24.1.2:0->12.24.3.2:0 tun_id=100.10.0.11 tun_id6=::10.0.0.4 dst_mtu=1500 dpd-link=on weight=1
bound_if=5 lgwy=static/1 tun=intf mode=dial_inst/3 encap=none/74408 options[122a8]=npu rgwy-chg frag-rfc run_state=0 role=sync-primary accept_traffic=1 overlay_id=0

parent=advpn1-hub index=0
proxyid_num=1 child_num=0 refcnt=5 ilast=2 olast=2 ad=s/1
stat: rxp=405 txp=404 rxb=25416 txb=25436
dpd: mode=on-idle on=1 idle=5000ms retry=3 count=0 seqno=58
natt: mode=none draft=0 interval=0 remote_port=0
fec: egress=0 ingress=0
proxyid=advpn1-hub proto=0 sa=1 ref=3 serial=1 ads
src: 0:0.0.0.0-255.255.255.255:0
dst: 0:0.0.0.0-255.255.255.255:0
SA: ref=3 options=20a02 type=00 soft=0 mtu=1446 expire=41521/0B replaywin=2048
seqno=195 esn=0 replaywin_lastseq=00000196 qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=43190/43200
dec: spi=beaf4a40 esp=des key=8 dd891a4f05397232
ah=md5 key=16 979d8c887e066c82877b2f31eb49740e
enc: spi=aee69ae7 esp=des key=8 05bb79d311fcb78d
ah=md5 key=16 2f375e0697cf5b0fccb438c639af4d16
dec:pkts/bytes=810/50832, enc:pkts/bytes=808/72828
npu_flag=00 npu_rgwy=12.24.3.2 npu_lgwy=12.24.1.2 npu_selid=1 dec_npuid=0 enc_npuid=0

Here is SpokeA

name=advpn1-spoke ver=1 serial=1 12.24.2.2:0->12.24.1.2:0 tun_id=12.24.1.2 tun_id6=::12.24.1.2 dst_mtu=1500 dpd-link=on weight=1
bound_if=5 lgwy=static/1 tun=intf mode=auto/1 encap=none/568 options[0238]=npu create_dev frag-rfc role=sync-primary accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=4 ilast=0 olast=0 ad=r/2
stat: rxp=427 txp=412 rxb=26916 txb=25502
dpd: mode=on-idle on=1 idle=5000ms retry=3 count=0 seqno=3
natt: mode=none draft=0 interval=0 remote_port=0
fec: egress=0 ingress=0
proxyid=advpn1-spoke proto=0 sa=1 ref=3 serial=1 auto-negotiate adr
src: 0:0.0.0.0-255.255.255.255:0
dst: 0:0.0.0.0-255.255.255.255:0
SA: ref=3 options=3a203 type=00 soft=0 mtu=1446 expire=41162/0B replaywin=2048
seqno=19d esn=0 replaywin_lastseq=000001ac qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=42902/43200
dec: spi=3d96d73f esp=des key=8 7df0edf1ff8d2e41
ah=md5 key=16 7b381d0378a5aa40471e8c29a7a20303
enc: spi=beaf4a42 esp=des key=8 87bf0913737de236
ah=md5 key=16 b74a6bb29df58441d9952c9e4ad1c79d
dec:pkts/bytes=854/53832, enc:pkts/bytes=824/73430
npu_flag=00 npu_rgwy=12.24.1.2 npu_lgwy=12.24.2.2 npu_selid=0 dec_npuid=0 enc_npuid=0

Here is Spoke B

name=advpn1-spoke ver=1 serial=1 12.24.3.2:0->12.24.1.2:0 tun_id=12.24.1.2 tun_id6=::12.24.1.2 dst_mtu=1500 dpd-link=on weight=1
bound_if=5 lgwy=static/1 tun=intf mode=auto/1 encap=none/568 options[0238]=npu create_dev frag-rfc role=primary accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=4 ilast=5 olast=5 ad=r/2
stat: rxp=436 txp=437 rxb=27404 txb=27384
dpd: mode=on-idle on=1 idle=5000ms retry=3 count=0 seqno=1
natt: mode=none draft=0 interval=0 remote_port=0
fec: egress=0 ingress=0
proxyid=advpn1-spoke proto=0 sa=1 ref=3 serial=1 adr
src: 0:0.0.0.0-255.255.255.255:0
dst: 0:0.0.0.0-255.255.255.255:0
SA: ref=3 options=32202 type=00 soft=0 mtu=1446 expire=41088/0B replaywin=2048
seqno=1b6 esn=0 replaywin_lastseq=000001b5 qat=0 rekey=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=42903/43200
dec: spi=aee69ae7 esp=des key=8 05bb79d311fcb78d
ah=md5 key=16 2f375e0697cf5b0fccb438c639af4d16
enc: spi=beaf4a40 esp=des key=8 dd891a4f05397232
ah=md5 key=16 979d8c887e066c82877b2f31eb49740e
dec:pkts/bytes=872/54808, enc:pkts/bytes=874/78520
npu_flag=00 npu_rgwy=12.24.1.2 npu_lgwy=12.24.3.2 npu_selid=0 dec_npuid=0 enc_npuid=0

[BrianHJones0217]

Here are the ping results:

Hub Side

exec ping 100.10.0.10
PING 100.10.0.10 (100.10.0.10): 56 data bytes

--- 100.10.0.10 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

exec ping 100.10.0.11
PING 100.10.0.11 (100.10.0.11): 56 data bytes
64 bytes from 100.10.0.11: icmp_seq=0 ttl=255 time=1.2 ms
64 bytes from 100.10.0.11: icmp_seq=1 ttl=255 time=2.7 ms
64 bytes from 100.10.0.11: icmp_seq=2 ttl=255 time=1.3 ms
64 bytes from 100.10.0.11: icmp_seq=3 ttl=255 time=1.3 ms
64 bytes from 100.10.0.11: icmp_seq=4 ttl=255 time=1.4 ms

--- 100.10.0.11 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 1.2/1.5/2.7 ms

Spoke A Side

exec ping 100.10.0.1
PING 100.10.0.1 (100.10.0.1): 56 data bytes
64 bytes from 100.10.0.1: icmp_seq=0 ttl=255 time=0.9 ms
64 bytes from 100.10.0.1: icmp_seq=1 ttl=255 time=2.3 ms
64 bytes from 100.10.0.1: icmp_seq=2 ttl=255 time=0.6 ms
64 bytes from 100.10.0.1: icmp_seq=3 ttl=255 time=2.2 ms
64 bytes from 100.10.0.1: icmp_seq=4 ttl=255 time=1.3 ms

--- 100.10.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.6/1.4/2.3 ms

exec ping 100.10.0.11
PING 100.10.0.11 (100.10.0.11): 56 data bytes

--- 100.10.0.11 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

SpokeB Side

exec ping 100.10.0.1
PING 100.10.0.1 (100.10.0.1): 56 data bytes
64 bytes from 100.10.0.1: icmp_seq=0 ttl=255 time=1.0 ms
64 bytes from 100.10.0.1: icmp_seq=1 ttl=255 time=1.0 ms
64 bytes from 100.10.0.1: icmp_seq=2 ttl=255 time=1.0 ms
64 bytes from 100.10.0.1: icmp_seq=3 ttl=255 time=2.5 ms
64 bytes from 100.10.0.1: icmp_seq=4 ttl=255 time=0.8 ms

--- 100.10.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.8/1.2/2.5 ms

exec ping 100.10.0.10
PING 100.10.0.10 (100.10.0.10): 56 data bytes

--- 100.10.0.10 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

[syordanov]

Dear BrianHJones0217,

Can you please check for errors on the VPN tunnel interface on the spoke and HUB :

fnsysctl ifconfig name_of_your_VPN

If possible i will recommend to open a ticket to TAC so we can have a closer look of the issue .

Regards,

Fortinet

[BrianHJones0217]

That command is not working.

FW-1# fnsysctl ifconfig advpn1-hub
Unknown action 0

I didn't think I could open a ticket for a lab, but if I can I will.

[hbac]

Hi @BrianHJones0217,

Can you run this capture sniffer on the hub and try to ping again? 

di sniffer packet any 'host 192.168.20.240 and icmp' 4 0 l 

Are you using physical FortiGates or VM for your lab? What is the firmware version? 

Regards,