Skip to main content
BrianHJones0217
BrianHJones0217
New Member
November 19, 2023
Question

Fortigate SD-WAN Hub cannot connect to Spokes

  • November 19, 2023
  • 15 replies
  • 8321 views

I am trying to build a lab for SDWAN using the Fortigates and ADVPN as it is similar to a client environment that I support.  The tunnels come up fine and BGP comes up find as well. However, the PCs cannot ping each other.  The firewall rule is pretty much wide open.  All three firewalls seem to have the same symptom as I don't believe the traffic is passing from the inside interface to the ADVPN tunnel.  Here are the technical details that I have to share:

 

Packet Capture from Hub->SpokeA (same results for Hub->SpokeB, SpokeA->Hub, SpokeB->Hub)

Using Original Sniffing Mode
interfaces=[any]
filters=[host 192.168.10.240 and icmp]
1.735240 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2.735788 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
3.735260 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
4.736537 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
5.736059 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
6.736208 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
7.736246 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
8.736187 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
9.736327 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
10.736504 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request

 

Packet Capture on Dest FW shows not traffic inbound in all cases.

 

Diag Debug Flow Trace from Hub-SpokeA

id=65308 trace_id=72 func=init_ip_session_common line=6043 msg="allocate a new session-00000876, tun_id=0.0.0.0"
id=65308 trace_id=72 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=72 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=72 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=72 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"
id=65308 trace_id=73 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 192.168.10.240:54538->192.168.20.240:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=54538, seq=2."
id=65308 trace_id=73 func=init_ip_session_common line=6043 msg="allocate a new session-00000877, tun_id=0.0.0.0"
id=65308 trace_id=73 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=73 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=73 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=73 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"
id=65308 trace_id=74 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 192.168.10.240:54538->192.168.20.240:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=54538, seq=3."
id=65308 trace_id=74 func=init_ip_session_common line=6043 msg="allocate a new session-00000878, tun_id=0.0.0.0"
id=65308 trace_id=74 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=74 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=74 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=74 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"

15 replies

Show previous replies
v_ceban
Staff & Editor
v_ceban
Staff & Editor
March 4, 2024

Please disable add-route on the ipsec phase-1 interface on all involved parts.
config vpn ipsec phase1-interface

    edit <tunnel name>

       set add-route disable
    next

end
Also, make sure the IP addresses and subnet is correctly configured on tunnel interfaces.
On HUB the remote IP should be a dump IP (Not used by any spoke) with the appropriate subnet that will cover all remote locations IP.
The remote IP should be HUB tunnel interface IP with the same subnet on spokes.

BrianHJones0217
BrianHJones0217Author
New Member
March 4, 2024

The set add-route disable is configured on all the tunnels on all four firewalls.

 

IPs are configured correctly on the tunnel interfaces. 

 

on the HUB, the remote-ip is a DUMP not used by any spoke with the correct subnet. 

on the SPOKEs, the remote IP is the hub tunnel interface IP with the correct subnet.

BrianHJones0217
BrianHJones0217Author
New Member
March 5, 2024

All the IPs on the Internet side are reachable from all firewalls, so they can ping the external addresses. 


Here are the phase-1 configs:

 

(Hub)
config vpn ipsec phase1-interface
edit "advpn-hub"
set type dynamic
set interface "port1"
set peertype any
set net-device disable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-sender enable
set auto-discovery-forwarder enable
set psksecret ENC ...
set dpd-retryinterval 5
next
edit "advpn2-hub"
set type dynamic
set interface "port2"
set peertype any
set net-device disable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-sender enable
set auto-discovery-forwarder enable
set psksecret ENC ...
set dpd-retryinterval 5
next
end


(Spoke)
config vpn ipsec phase1-interface
edit "advpn-spoke"
set interface "port1"
set peertype any
set net-device enable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-receiver enable
set remote-gw 100.1.1.2
set psksecret ENC ...
set dpd-retryinterval 5
next
edit "advpn2-spoke"
set interface "port2"
set peertype any
set net-device enable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-receiver enable
set remote-gw 200.1.1.2
set psksecret ENC ...
set dpd-retryinterval 5
next
end

 

BrianHJones0217
BrianHJones0217Author
New Member
March 5, 2024

Here are the system interface configs:

 

(Hub)

edit "advpn-hub"
set vdom "root"
set ip 172.30.0.1 255.255.255.255
set allowaccess ping
set type tunnel
set tcp-mss 1360
set remote-ip 172.30.0.2 255.255.254.0
set snmp-index 9
set interface "port1"
set mtu-override enable
set mtu 1400
next
edit "advpn2-hub"
set vdom "root"
set ip 172.31.0.1 255.255.255.255
set allowaccess ping
set type tunnel
set tcp-mss 1360
set remote-ip 172.31.0.2 255.255.254.0
set snmp-index 10
set interface "port2"
set mtu-override enable
set mtu 1400
next

 

(Spoke)
edit "advpn-spoke"
set vdom "root"
set ip 172.30.0.8 255.255.255.255
set type tunnel
set remote-ip 172.30.0.1 255.255.254.0
set snmp-index 9
set interface "port1"
next
edit "advpn2-spoke"
set vdom "root"
set ip 172.31.0.8 255.255.255.255
set type tunnel
set remote-ip 172.31.0.1 255.255.254.0
set snmp-index 10
set interface "port2"
nextt
end

 

BrianHJones0217
BrianHJones0217Author
New Member
March 5, 2024

Here is a session capture through the Hub. 

(Note: I never see the packet leave the firewall (this is consistent with the problem before and what I am seeing across all four sites):

 

2024-03-05 07:28:44 id=65308 trace_id=1 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 10.148.5.10:11578->10.148.128.10:2048) tun_id=0.0.0.0 from port3. type=8, code=0, id=11578, seq=1."
2024-03-05 07:28:45 id=65308 trace_id=1 func=init_ip_session_common line=6043 msg="allocate a new session-0000016f, tun_id=0.0.0.0"
2024-03-05 07:28:45 id=65308 trace_id=1 func=iprope_dnat_check line=5302 msg="in-[port3], out-[]"
2024-03-05 07:28:45 id=65308 trace_id=1 func=iprope_dnat_tree_check line=824 msg="len=0"
2024-03-05 07:28:45 id=65308 trace_id=1 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2024-03-05 07:28:45 id=65308 trace_id=1 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-172.30.0.8 via advpn-hub"
2024-03-05 07:28:46 id=65308 trace_id=2 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 10.148.5.10:12090->10.148.128.10:2048) tun_id=0.0.0.0 from port3. type=8, code=0, id=12090, seq=2."
2024-03-05 07:28:47 id=65308 trace_id=2 func=init_ip_session_common line=6043 msg="allocate a new session-00000171, tun_id=0.0.0.0"
2024-03-05 07:28:47 id=65308 trace_id=2 func=iprope_dnat_check line=5302 msg="in-[port3], out-[]"
2024-03-05 07:28:47 id=65308 trace_id=2 func=iprope_dnat_tree_check line=824 msg="len=0"
2024-03-05 07:28:47 id=65308 trace_id=2 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2024-03-05 07:28:47 id=65308 trace_id=2 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-172.30.0.8 via advpn-hub"
2024-03-05 07:28:48 id=65308 trace_id=3 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 10.148.5.10:12602->10.148.128.10:2048) tun_id=0.0.0.0 from port3. type=8, code=0, id=12602, seq=3."
2024-03-05 07:28:49 id=65308 trace_id=3 func=init_ip_session_common line=6043 msg="allocate a new session-00000172, tun_id=0.0.0.0"
2024-03-05 07:28:49 id=65308 trace_id=3 func=iprope_dnat_check line=5302 msg="in-[port3], out-[]"
2024-03-05 07:28:49 id=65308 trace_id=3 func=iprope_dnat_tree_check line=824 msg="len=0"
2024-03-05 07:28:49 id=65308 trace_id=3 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2024-03-05 07:28:49 id=65308 trace_id=3 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-172.30.0.8 via advpn-hub"
2024-03-05 07:28:50 id=65308 trace_id=4 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 10.148.5.10:13114->10.148.128.10:2048) tun_id=0.0.0.0 from port3. type=8, code=0, id=13114, seq=4."
2024-03-05 07:28:51 id=65308 trace_id=4 func=init_ip_session_common line=6043 msg="allocate a new session-00000175, tun_id=0.0.0.0"
2024-03-05 07:28:51 id=65308 trace_id=4 func=iprope_dnat_check line=5302 msg="in-[port3], out-[]"
2024-03-05 07:28:51 id=65308 trace_id=4 func=iprope_dnat_tree_check line=824 msg="len=0"
2024-03-05 07:28:51 id=65308 trace_id=4 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2024-03-05 07:28:51 id=65308 trace_id=4 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-172.30.0.8 via advpn-hub"
2024-03-05 07:28:52 id=65308 trace_id=5 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 10.148.5.10:13626->10.148.128.10:2048) tun_id=0.0.0.0 from port3. type=8, code=0, id=13626, seq=5."
2024-03-05 07:28:53 id=65308 trace_id=5 func=init_ip_session_common line=6043 msg="allocate a new session-00000176, tun_id=0.0.0.0"
2024-03-05 07:28:53 id=65308 trace_id=5 func=iprope_dnat_check line=5302 msg="in-[port3], out-[]"
2024-03-05 07:28:53 id=65308 trace_id=5 func=iprope_dnat_tree_check line=824 msg="len=0"
2024-03-05 07:28:53 id=65308 trace_id=5 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
2024-03-05 07:28:53 id=65308 trace_id=5 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-172.30.0.8 via advpn-hub"

Terms & ConditionsAccessibility statement