Fortigate SD-WAN Hub cannot connect to Spokes

[BrianHJones0217]

I am trying to build a lab for SDWAN using the Fortigates and ADVPN as it is similar to a client environment that I support.  The tunnels come up fine and BGP comes up find as well. However, the PCs cannot ping each other.  The firewall rule is pretty much wide open.  All three firewalls seem to have the same symptom as I don't believe the traffic is passing from the inside interface to the ADVPN tunnel.  Here are the technical details that I have to share:

Packet Capture from Hub->SpokeA (same results for Hub->SpokeB, SpokeA->Hub, SpokeB->Hub)

Using Original Sniffing Mode
interfaces=[any]
filters=[host 192.168.10.240 and icmp]
1.735240 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2.735788 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
3.735260 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
4.736537 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
5.736059 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
6.736208 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
7.736246 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
8.736187 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
9.736327 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
10.736504 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request

Packet Capture on Dest FW shows not traffic inbound in all cases.

Diag Debug Flow Trace from Hub-SpokeA

id=65308 trace_id=72 func=init_ip_session_common line=6043 msg="allocate a new session-00000876, tun_id=0.0.0.0"
id=65308 trace_id=72 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=72 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=72 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=72 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"
id=65308 trace_id=73 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 192.168.10.240:54538->192.168.20.240:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=54538, seq=2."
id=65308 trace_id=73 func=init_ip_session_common line=6043 msg="allocate a new session-00000877, tun_id=0.0.0.0"
id=65308 trace_id=73 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=73 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=73 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=73 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"
id=65308 trace_id=74 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 192.168.10.240:54538->192.168.20.240:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=54538, seq=3."
id=65308 trace_id=74 func=init_ip_session_common line=6043 msg="allocate a new session-00000878, tun_id=0.0.0.0"
id=65308 trace_id=74 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=74 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=74 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=74 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"

[BrianHJones0217]

Here is a session capture through the Hub. 

(Note: I never see the packet leave the firewall (this is consistent with the problem before and what I am seeing across all four sites):

2024-03-05 07:28:44 id=65308 trace_id=1 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 10.148.5.10:11578->10.148.128.10:2048) tun_id=0.0.0.0 from port3. type=8, code=0, id=11578, seq=1."
2024-03-05 07:28:45 id=65308 trace_id=1 func=init_ip_session_common line=6043 msg="allocate a new session-0000016f, tun_id=0.0.0.0"
2024-03-05 07:28:45 id=65308 trace_id=1 func=iprope_dnat_check line=5302 msg="in-[port3], out-[]"
2024-03-05 07:28:45 id=65308 trace_id=1 func=iprope_dnat_tree_check line=824 msg="len=0"
2024-03-05 07:28:45 id=65308 trace_id=1 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2024-03-05 07:28:45 id=65308 trace_id=1 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-172.30.0.8 via advpn-hub"
2024-03-05 07:28:46 id=65308 trace_id=2 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 10.148.5.10:12090->10.148.128.10:2048) tun_id=0.0.0.0 from port3. type=8, code=0, id=12090, seq=2."
2024-03-05 07:28:47 id=65308 trace_id=2 func=init_ip_session_common line=6043 msg="allocate a new session-00000171, tun_id=0.0.0.0"
2024-03-05 07:28:47 id=65308 trace_id=2 func=iprope_dnat_check line=5302 msg="in-[port3], out-[]"
2024-03-05 07:28:47 id=65308 trace_id=2 func=iprope_dnat_tree_check line=824 msg="len=0"
2024-03-05 07:28:47 id=65308 trace_id=2 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2024-03-05 07:28:47 id=65308 trace_id=2 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-172.30.0.8 via advpn-hub"
2024-03-05 07:28:48 id=65308 trace_id=3 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 10.148.5.10:12602->10.148.128.10:2048) tun_id=0.0.0.0 from port3. type=8, code=0, id=12602, seq=3."
2024-03-05 07:28:49 id=65308 trace_id=3 func=init_ip_session_common line=6043 msg="allocate a new session-00000172, tun_id=0.0.0.0"
2024-03-05 07:28:49 id=65308 trace_id=3 func=iprope_dnat_check line=5302 msg="in-[port3], out-[]"
2024-03-05 07:28:49 id=65308 trace_id=3 func=iprope_dnat_tree_check line=824 msg="len=0"
2024-03-05 07:28:49 id=65308 trace_id=3 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2024-03-05 07:28:49 id=65308 trace_id=3 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-172.30.0.8 via advpn-hub"
2024-03-05 07:28:50 id=65308 trace_id=4 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 10.148.5.10:13114->10.148.128.10:2048) tun_id=0.0.0.0 from port3. type=8, code=0, id=13114, seq=4."
2024-03-05 07:28:51 id=65308 trace_id=4 func=init_ip_session_common line=6043 msg="allocate a new session-00000175, tun_id=0.0.0.0"
2024-03-05 07:28:51 id=65308 trace_id=4 func=iprope_dnat_check line=5302 msg="in-[port3], out-[]"
2024-03-05 07:28:51 id=65308 trace_id=4 func=iprope_dnat_tree_check line=824 msg="len=0"
2024-03-05 07:28:51 id=65308 trace_id=4 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2024-03-05 07:28:51 id=65308 trace_id=4 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-172.30.0.8 via advpn-hub"
2024-03-05 07:28:52 id=65308 trace_id=5 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 10.148.5.10:13626->10.148.128.10:2048) tun_id=0.0.0.0 from port3. type=8, code=0, id=13626, seq=5."
2024-03-05 07:28:53 id=65308 trace_id=5 func=init_ip_session_common line=6043 msg="allocate a new session-00000176, tun_id=0.0.0.0"
2024-03-05 07:28:53 id=65308 trace_id=5 func=iprope_dnat_check line=5302 msg="in-[port3], out-[]"
2024-03-05 07:28:53 id=65308 trace_id=5 func=iprope_dnat_tree_check line=824 msg="len=0"
2024-03-05 07:28:53 id=65308 trace_id=5 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2024-03-05 07:28:53 id=65308 trace_id=5 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-172.30.0.8 via advpn-hub"