Help
Support Forum
The Forums are a place to find answers on a range of Fortinet products from peers and product experts.
BrianHJones0217
New Contributor

Created on ‎11-19-2023 09:33 AM Edited on ‎11-19-2023 02:52 PM

Fortigate SD-WAN Hub cannot connect to Spokes

I am trying to build a lab for SDWAN using the Fortigates and ADVPN as it is similar to a client environment that I support.  The tunnels come up fine and BGP comes up find as well. However, the PCs cannot ping each other.  The firewall rule is pretty much wide open.  All three firewalls seem to have the same symptom as I don't believe the traffic is passing from the inside interface to the ADVPN tunnel.  Here are the technical details that I have to share:

 

Packet Capture from Hub->SpokeA (same results for Hub->SpokeB, SpokeA->Hub, SpokeB->Hub)

Using Original Sniffing Mode
interfaces=[any]
filters=[host 192.168.10.240 and icmp]
1.735240 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2.735788 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
3.735260 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
4.736537 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
5.736059 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
6.736208 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
7.736246 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
8.736187 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
9.736327 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
10.736504 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request

 

Packet Capture on Dest FW shows not traffic inbound in all cases.

 

Diag Debug Flow Trace from Hub-SpokeA

id=65308 trace_id=72 func=init_ip_session_common line=6043 msg="allocate a new session-00000876, tun_id=0.0.0.0"
id=65308 trace_id=72 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=72 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=72 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=72 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"
id=65308 trace_id=73 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 192.168.10.240:54538->192.168.20.240:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=54538, seq=2."
id=65308 trace_id=73 func=init_ip_session_common line=6043 msg="allocate a new session-00000877, tun_id=0.0.0.0"
id=65308 trace_id=73 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=73 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=73 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=73 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"
id=65308 trace_id=74 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 192.168.10.240:54538->192.168.20.240:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=54538, seq=3."
id=65308 trace_id=74 func=init_ip_session_common line=6043 msg="allocate a new session-00000878, tun_id=0.0.0.0"
id=65308 trace_id=74 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=74 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=74 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=74 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"

Labels:
3722
0 Kudos
20 REPLIES 20
BrianHJones0217
New Contributor
In response to hbac

Created on ‎12-08-2023 09:53 AM

FW-1 # di sniffer packet any 'host 192.168.20.240 and icmp' 4 0 l
Using Original Sniffing Mode
interfaces=[any]
filters=[host 192.168.20.240 and icmp]
2023-12-08 09:51:52.515476 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:53.514307 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:54.515223 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:55.515620 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:56.516672 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:57.517134 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:58.517453 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:59.518315 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:00.518840 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:01.519108 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:02.519798 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:03.519635 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:04.520208 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:05.520533 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:06.520789 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:07.521770 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:08.522154 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
^C
17 packets received by filter
0 packets dropped by kernel

 

This is a virtual lab in EVE-NG using Fortigate-VM64-KVM v7.2.6

1471
0 Kudos
hbac
Staff
Staff
In response to BrianHJones0217

Created on ‎12-08-2023 11:09 AM

@BrianHJones0217,

 

We don't see any outbound traffic. Have you tried to bounce the tunnel? You can also try the following commands:

 

config vpn ipsec phase1-interface 

edit advpn1-hub

set npu-offload disable 

end 

 

Restart the tunnel and try again. 

 

Regards, 

1464
0 Kudos
BrianHJones0217
New Contributor
In response to hbac

Created on ‎12-08-2023 11:26 AM

FW-1 # show vpn ipsec phase1-interface
config vpn ipsec phase1-interface
edit "advpn1-hub"
set type dynamic
set interface "port3"
set peertype any
set net-device disable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-sender enable
set psksecret ENC dDJbLruAOD4Xvv7iMAHvFA0Aj15ThTm4SBsGTO4kALhRjJSvggXaw35GLdWpBLzu4gpJnJNl4mwp67FlACyduHHKIsXD+DAtwj76mx0BFQuZ0XFBABcXV6As5wzdRMrxxugEzNACKoM2wNIRQKINHx599qOq2/bUCMJ7bRXUeTSCMK6TlegtajfJIXvQkW0ffRGhbw==
set dpd-retryinterval 5
next
edit "advpn2-hub"
set type dynamic
set interface "port4"
set peertype any
set net-device disable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-sender enable
set psksecret ENC b4W13OF0Gq1C0ZGmdeQ2DpAGsixjUmoz6VMG+FpAI8z2kLF0mVv9/3DFmXXp6+Ji9w/VudLTZs0xx5hoM3Mc35HyRhvMgCZo0ZW81ZUlgR26WYuke9ocb8oDTKIYzeMGs9Yd84f/QXsyg33+1Xsi6YZunXtePHbxvNRFSlgdZDOLZ6KkAt5F6oaFIxbxn27s5Kzmvg==
set dpd-retryinterval 5
next
end

 

FW-1 # diag vpn tunnel list | grep tun_id
name=advpn2-hub_0 ver=1 serial=9 60.120.1.2:0->60.120.3.2:0 tun_id=200.10.0.11 tun_id6=::10.0.0.9 dst_mtu=1500 dpd-link=on weight=1
name=advpn2-hub_1 ver=1 serial=a 60.120.1.2:0->60.120.2.2:0 tun_id=200.10.0.10 tun_id6=::10.0.0.10 dst_mtu=1500 dpd-link=on weight=1
name=advpn1-hub_0 ver=1 serial=7 12.24.1.2:0->12.24.3.2:0 tun_id=100.10.0.11 tun_id6=::10.0.0.7 dst_mtu=1500 dpd-link=on weight=1
name=advpn1-hub ver=1 serial=1 12.24.1.2:0->0.0.0.0:0 tun_id=10.0.0.1 tun_id6=::10.0.0.1 dst_mtu=0 dpd-link=on weight=1
name=advpn1-hub_1 ver=1 serial=8 12.24.1.2:0->12.24.2.2:0 tun_id=100.10.0.10 tun_id6=::10.0.0.8 dst_mtu=1500 dpd-link=on weight=1
name=advpn2-hub ver=1 serial=2 60.120.1.2:0->0.0.0.0:0 tun_id=10.0.0.2 tun_id6=::10.0.0.2 dst_mtu=0 dpd-link=on weight=1


FW-1 # diag vpn tunnel flush

 

FW-1 # diag vpn tunnel list | grep tun_id
name=advpn2-hub_0 ver=1 serial=d 60.120.1.2:0->60.120.2.2:0 tun_id=200.10.0.10 tun_id6=::10.0.0.13 dst_mtu=1500 dpd-link=on weight=1
name=advpn2-hub_1 ver=1 serial=e 60.120.1.2:0->60.120.3.2:0 tun_id=200.10.0.11 tun_id6=::10.0.0.14 dst_mtu=0 dpd-link=on weight=1
name=advpn1-hub_0 ver=1 serial=b 12.24.1.2:0->12.24.3.2:0 tun_id=100.10.0.11 tun_id6=::10.0.0.11 dst_mtu=1500 dpd-link=on weight=1
name=advpn1-hub ver=1 serial=1 12.24.1.2:0->0.0.0.0:0 tun_id=10.0.0.1 tun_id6=::10.0.0.1 dst_mtu=0 dpd-link=on weight=1
name=advpn1-hub_1 ver=1 serial=c 12.24.1.2:0->12.24.2.2:0 tun_id=100.10.0.10 tun_id6=::10.0.0.12 dst_mtu=1500 dpd-link=on weight=1
name=advpn2-hub ver=1 serial=2 60.120.1.2:0->0.0.0.0:0 tun_id=10.0.0.2 tun_id6=::10.0.0.2 dst_mtu=0 dpd-link=on weight=1

 

FW-1 # di sniffer packet any 'host 192.168.20.240 and icmp' 4 0 l
Using Original Sniffing Mode
interfaces=[any]
filters=[host 192.168.20.240 and icmp]
2023-12-08 11:24:46.648236 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:47.647886 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:48.648001 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:49.648625 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:50.648888 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:51.649256 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:52.649598 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:53.650510 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
^C
8 packets received by filter
0 packets dropped by kernel

1462
0 Kudos
syordanov
Staff
Staff

Created on ‎12-10-2023 11:58 PM

Hello,

'FW-1' is the spoke or the HUB ?

 

 

.
1438
0 Kudos
BrianHJones0217
New Contributor
In response to syordanov

Created on ‎12-11-2023 07:15 AM

The config is from the hub. 

1431
0 Kudos
BrianHJones0217
New Contributor

Created on ‎03-04-2024 12:23 PM

I want to work on this issue again.  I have rebuilt the lab from the ground up and I am running into the same problem.  Here is the current setup.
SDWAN Lab v2.png

No matter which site choose, the traffic never go past the firewall.  The routing and VPN tunnels seems to be all there.    Here is the routing from the the hub (USDAD):

 

USDAD-FW01 # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
V - BGP VPNv4
* - candidate default

Routing table for VRF=0
S* 0.0.0.0/0 [10/0] via 100.1.1.1, port1, [1/0]
[10/0] via 200.1.1.1, port2, [1/0]
C 10.148.5.0/24 is directly connected, port3
B 10.148.128.0/23 [200/0] via 172.30.0.8 (recursive via advpn-hub tunnel 172.30.0.8), 00:12:36, [
1/0]
B 10.149.96.0/24 [200/0] via 172.30.0.22 (recursive via advpn-hub tunnel 172.30.0.22), 00:12:39,
[1/0]
B 10.151.176.0/23 [200/0] via 172.30.0.59 (recursive via advpn-hub tunnel 172.30.0.59), 00:12:36,
[1/0]
C 100.1.1.0/24 is directly connected, port1
C 172.30.0.1/32 is directly connected, advpn-hub
C 172.30.0.2/31 is directly connected, advpn-hub
S 172.30.0.8/32 [15/0] via advpn-hub tunnel 172.30.0.8, [1/0]
S 172.30.0.22/32 [15/0] via advpn-hub tunnel 172.30.0.22, [1/0]
S 172.30.0.59/32 [15/0] via advpn-hub tunnel 172.30.0.59, [1/0]
C 172.31.0.0/23 is directly connected, advpn2-hub
C 172.31.0.1/32 is directly connected, advpn2-hub
C 200.1.1.0/24 is directly connected, port2

 

Here is the routing for one of the three spokes, USHO1:

 

USHO1-FW01 # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
V - BGP VPNv4
* - candidate default

Routing table for VRF=0
S* 0.0.0.0/0 [10/0] via 100.2.2.1, port1, [1/0]
[10/0] via 200.2.2.1, port2, [1/0]
B 10.148.5.0/24 [200/0] via 172.30.0.1 (recursive via advpn-spoke tunnel 100.1.1.2), 00:13:49,
[1/0]
C 10.148.128.0/23 is directly connected, port3
B 10.149.96.0/24 [200/0] via 172.30.0.22 (recursive via advpn-spoke tunnel 100.1.1.2), 00:13:20
, [1/0]
B 10.151.176.0/23 [200/0] via 172.30.0.59 (recursive via advpn-spoke tunnel 100.1.1.2), 00:13:2
0, [1/0]
C 100.2.2.0/24 is directly connected, port1
S 172.30.0.0/23 [5/0] via advpn-spoke tunnel 100.1.1.2, [1/0]
S 172.30.0.1/32 [15/0] via advpn-spoke tunnel 100.1.1.2, [1/0]
C 172.30.0.8/32 is directly connected, advpn-spoke
S 172.31.0.0/23 [5/0] via advpn2-spoke tunnel 200.1.1.2, [1/0]
S 172.31.0.1/32 [15/0] via advpn2-spoke tunnel 200.1.1.2, [1/0]
C 172.31.0.8/32 is directly connected, advpn2-spoke
C 200.2.2.0/24 is directly connected, port2

 

Here is the sniffer trace:

 

USHO1-FW01 # diag sniffer packet any 'host 10.148.128.10 and icmp' 4
Using Original Sniffing Mode
interfaces=[any]
filters=[host 10.148.128.10 and icmp]
12.186705 port3 in 10.148.128.10 -> 10.148.5.10: icmp: echo request
14.187416 port3 in 10.148.128.10 -> 10.148.5.10: icmp: echo request
16.187657 port3 in 10.148.128.10 -> 10.148.5.10: icmp: echo request
18.187783 port3 in 10.148.128.10 -> 10.148.5.10: icmp: echo request
20.187973 port3 in 10.148.128.10 -> 10.148.5.10: icmp: echo request
^C
5 packets received by filter
0 packets dropped by kernel

 

Here is the firewall rule from the spoke config:

 

config firewall policy
edit 1
set name "outbound-advpn"
set uuid 00f36e2a-da44-51ee-63fd-389fab9f7e79
set srcintf "port3"
set dstintf "advpn-spoke" "advpn2-spoke"
set action accept
set srcaddr "all"
set dstaddr "all"
set schedule "always"
set service "ALL"
next
edit 2
set name "inbound-advpn"
set uuid f52121b8-da44-51ee-fc62-1f2707c4b57e
set srcintf "advpn-spoke" "advpn2-spoke"
set dstintf "port3"
set action accept
set srcaddr "all"
set dstaddr "all"
set schedule "always"
set service "ALL"
next
end

1194
0 Kudos
v_ceban
Staff
Staff

Created on ‎03-04-2024 12:36 PM

Please disable add-route on the ipsec phase-1 interface on all involved parts.
config vpn ipsec phase1-interface

    edit <tunnel name>

       set add-route disable
    next

end
Also, make sure the IP addresses and subnet is correctly configured on tunnel interfaces.
On HUB the remote IP should be a dump IP (Not used by any spoke) with the appropriate subnet that will cover all remote locations IP.
The remote IP should be HUB tunnel interface IP with the same subnet on spokes.

Vladislav Ceban
1188
0 Kudos
BrianHJones0217
New Contributor

Created on ‎03-04-2024 01:17 PM

The set add-route disable is configured on all the tunnels on all four firewalls.

 

IPs are configured correctly on the tunnel interfaces. 

 

on the HUB, the remote-ip is a DUMP not used by any spoke with the correct subnet. 

on the SPOKEs, the remote IP is the hub tunnel interface IP with the correct subnet.

1180
0 Kudos
BrianHJones0217
New Contributor

Created on ‎03-05-2024 07:05 AM

All the IPs on the Internet side are reachable from all firewalls, so they can ping the external addresses. 


Here are the phase-1 configs:

 

(Hub)
config vpn ipsec phase1-interface
edit "advpn-hub"
set type dynamic
set interface "port1"
set peertype any
set net-device disable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-sender enable
set auto-discovery-forwarder enable
set psksecret ENC ...
set dpd-retryinterval 5
next
edit "advpn2-hub"
set type dynamic
set interface "port2"
set peertype any
set net-device disable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-sender enable
set auto-discovery-forwarder enable
set psksecret ENC ...
set dpd-retryinterval 5
next
end


(Spoke)
config vpn ipsec phase1-interface
edit "advpn-spoke"
set interface "port1"
set peertype any
set net-device enable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-receiver enable
set remote-gw 100.1.1.2
set psksecret ENC ...
set dpd-retryinterval 5
next
edit "advpn2-spoke"
set interface "port2"
set peertype any
set net-device enable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-receiver enable
set remote-gw 200.1.1.2
set psksecret ENC ...
set dpd-retryinterval 5
next
end

 

1165
0 Kudos
BrianHJones0217
New Contributor

Created on ‎03-05-2024 07:07 AM

Here are the system interface configs:

 

(Hub)

edit "advpn-hub"
set vdom "root"
set ip 172.30.0.1 255.255.255.255
set allowaccess ping
set type tunnel
set tcp-mss 1360
set remote-ip 172.30.0.2 255.255.254.0
set snmp-index 9
set interface "port1"
set mtu-override enable
set mtu 1400
next
edit "advpn2-hub"
set vdom "root"
set ip 172.31.0.1 255.255.255.255
set allowaccess ping
set type tunnel
set tcp-mss 1360
set remote-ip 172.31.0.2 255.255.254.0
set snmp-index 10
set interface "port2"
set mtu-override enable
set mtu 1400
next

 

(Spoke)
edit "advpn-spoke"
set vdom "root"
set ip 172.30.0.8 255.255.255.255
set type tunnel
set remote-ip 172.30.0.1 255.255.254.0
set snmp-index 9
set interface "port1"
next
edit "advpn2-spoke"
set vdom "root"
set ip 172.31.0.8 255.255.255.255
set type tunnel
set remote-ip 172.31.0.1 255.255.254.0
set snmp-index 10
set interface "port2"
nextt
end

 

1163
0 Kudos
Announcements

Select Forum Responses to become Knowledge Articles!

Select the “Nominate to Knowledge Base” button to recommend a forum post to become a knowledge article.

Labels
Top Kudoed Authors
View all
Broad. Integrated. Automated.

The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.

Social Media
Security Research
Company
News & Articles
Contact Us

Copyright 2024 Fortinet, Inc. All Rights Reserved.