Fortigate SD-WAN Hub cannot connect to Spokes

BrianHJones0217 · ‎11-19-2023

I am trying to build a lab for SDWAN using the Fortigates and ADVPN as it is similar to a client environment that I support. The tunnels come up fine and BGP comes up find as well. However, the PCs cannot ping each other. The firewall rule is pretty much wide open. All three firewalls seem to have the same symptom as I don't believe the traffic is passing from the inside interface to the ADVPN tunnel. Here are the technical details that I have to share:

Packet Capture from Hub->SpokeA (same results for Hub->SpokeB, SpokeA->Hub, SpokeB->Hub)

Using Original Sniffing Mode
interfaces=[any]
filters=[host 192.168.10.240 and icmp]
1.735240 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2.735788 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
3.735260 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
4.736537 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
5.736059 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
6.736208 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
7.736246 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
8.736187 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
9.736327 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
10.736504 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request

Packet Capture on Dest FW shows not traffic inbound in all cases.

Diag Debug Flow Trace from Hub-SpokeA

id=65308 trace_id=72 func=init_ip_session_common line=6043 msg="allocate a new session-00000876, tun_id=0.0.0.0"
id=65308 trace_id=72 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=72 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=72 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=72 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"
id=65308 trace_id=73 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 192.168.10.240:54538->192.168.20.240:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=54538, seq=2."
id=65308 trace_id=73 func=init_ip_session_common line=6043 msg="allocate a new session-00000877, tun_id=0.0.0.0"
id=65308 trace_id=73 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=73 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=73 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=73 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"
id=65308 trace_id=74 func=print_pkt_detail line=5857 msg="vd-root:0 received a packet(proto=1, 192.168.10.240:54538->192.168.20.240:2048) tun_id=0.0.0.0 from port1. type=8, code=0, id=54538, seq=3."
id=65308 trace_id=74 func=init_ip_session_common line=6043 msg="allocate a new session-00000878, tun_id=0.0.0.0"
id=65308 trace_id=74 func=iprope_dnat_check line=5302 msg="in-[port1], out-[]"
id=65308 trace_id=74 func=iprope_dnat_tree_check line=824 msg="len=0"
id=65308 trace_id=74 func=iprope_dnat_check line=5314 msg="result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000"
id=65308 trace_id=74 func=__vf_ip_route_input_rcu line=2001 msg="find a route: flag=00000000 gw-100.10.0.10 via advpn1-hub"

BrianHJones0217 · ‎12-08-2023

FW-1 # di sniffer packet any 'host 192.168.20.240 and icmp' 4 0 l
Using Original Sniffing Mode
interfaces=[any]
filters=[host 192.168.20.240 and icmp]
2023-12-08 09:51:52.515476 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:53.514307 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:54.515223 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:55.515620 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:56.516672 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:57.517134 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:58.517453 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:51:59.518315 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:00.518840 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:01.519108 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:02.519798 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:03.519635 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:04.520208 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:05.520533 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:06.520789 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:07.521770 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 09:52:08.522154 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
^C
17 packets received by filter
0 packets dropped by kernel

This is a virtual lab in EVE-NG using Fortigate-VM64-KVM v7.2.6

hbac · ‎12-08-2023

@BrianHJones0217,

We don't see any outbound traffic. Have you tried to bounce the tunnel? You can also try the following commands:

config vpn ipsec phase1-interface

edit advpn1-hub

set npu-offload disable

end

Restart the tunnel and try again.

Regards,

BrianHJones0217 · ‎12-08-2023

FW-1 # show vpn ipsec phase1-interface
config vpn ipsec phase1-interface
edit "advpn1-hub"
set type dynamic
set interface "port3"
set peertype any
set net-device disable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-sender enable
set psksecret ENC dDJbLruAOD4Xvv7iMAHvFA0Aj15ThTm4SBsGTO4kALhRjJSvggXaw35GLdWpBLzu4gpJnJNl4mwp67FlACyduHHKIsXD+DAtwj76mx0BFQuZ0XFBABcXV6As5wzdRMrxxugEzNACKoM2wNIRQKINHx599qOq2/bUCMJ7bRXUeTSCMK6TlegtajfJIXvQkW0ffRGhbw==
set dpd-retryinterval 5
next
edit "advpn2-hub"
set type dynamic
set interface "port4"
set peertype any
set net-device disable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-sender enable
set psksecret ENC b4W13OF0Gq1C0ZGmdeQ2DpAGsixjUmoz6VMG+FpAI8z2kLF0mVv9/3DFmXXp6+Ji9w/VudLTZs0xx5hoM3Mc35HyRhvMgCZo0ZW81ZUlgR26WYuke9ocb8oDTKIYzeMGs9Yd84f/QXsyg33+1Xsi6YZunXtePHbxvNRFSlgdZDOLZ6KkAt5F6oaFIxbxn27s5Kzmvg==
set dpd-retryinterval 5
next
end

FW-1 # diag vpn tunnel list | grep tun_id
name=advpn2-hub_0 ver=1 serial=9 60.120.1.2:0->60.120.3.2:0 tun_id=200.10.0.11 tun_id6=::10.0.0.9 dst_mtu=1500 dpd-link=on weight=1
name=advpn2-hub_1 ver=1 serial=a 60.120.1.2:0->60.120.2.2:0 tun_id=200.10.0.10 tun_id6=::10.0.0.10 dst_mtu=1500 dpd-link=on weight=1
name=advpn1-hub_0 ver=1 serial=7 12.24.1.2:0->12.24.3.2:0 tun_id=100.10.0.11 tun_id6=::10.0.0.7 dst_mtu=1500 dpd-link=on weight=1
name=advpn1-hub ver=1 serial=1 12.24.1.2:0->0.0.0.0:0 tun_id=10.0.0.1 tun_id6=::10.0.0.1 dst_mtu=0 dpd-link=on weight=1
name=advpn1-hub_1 ver=1 serial=8 12.24.1.2:0->12.24.2.2:0 tun_id=100.10.0.10 tun_id6=::10.0.0.8 dst_mtu=1500 dpd-link=on weight=1
name=advpn2-hub ver=1 serial=2 60.120.1.2:0->0.0.0.0:0 tun_id=10.0.0.2 tun_id6=::10.0.0.2 dst_mtu=0 dpd-link=on weight=1

FW-1 # diag vpn tunnel flush

FW-1 # diag vpn tunnel list | grep tun_id
name=advpn2-hub_0 ver=1 serial=d 60.120.1.2:0->60.120.2.2:0 tun_id=200.10.0.10 tun_id6=::10.0.0.13 dst_mtu=1500 dpd-link=on weight=1
name=advpn2-hub_1 ver=1 serial=e 60.120.1.2:0->60.120.3.2:0 tun_id=200.10.0.11 tun_id6=::10.0.0.14 dst_mtu=0 dpd-link=on weight=1
name=advpn1-hub_0 ver=1 serial=b 12.24.1.2:0->12.24.3.2:0 tun_id=100.10.0.11 tun_id6=::10.0.0.11 dst_mtu=1500 dpd-link=on weight=1
name=advpn1-hub ver=1 serial=1 12.24.1.2:0->0.0.0.0:0 tun_id=10.0.0.1 tun_id6=::10.0.0.1 dst_mtu=0 dpd-link=on weight=1
name=advpn1-hub_1 ver=1 serial=c 12.24.1.2:0->12.24.2.2:0 tun_id=100.10.0.10 tun_id6=::10.0.0.12 dst_mtu=1500 dpd-link=on weight=1
name=advpn2-hub ver=1 serial=2 60.120.1.2:0->0.0.0.0:0 tun_id=10.0.0.2 tun_id6=::10.0.0.2 dst_mtu=0 dpd-link=on weight=1

FW-1 # di sniffer packet any 'host 192.168.20.240 and icmp' 4 0 l
Using Original Sniffing Mode
interfaces=[any]
filters=[host 192.168.20.240 and icmp]
2023-12-08 11:24:46.648236 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:47.647886 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:48.648001 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:49.648625 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:50.648888 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:51.649256 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:52.649598 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
2023-12-08 11:24:53.650510 port1 in 192.168.10.240 -> 192.168.20.240: icmp: echo request
^C
8 packets received by filter
0 packets dropped by kernel

syordanov · ‎12-10-2023

Hello,

'FW-1' is the spoke or the HUB ?

.

BrianHJones0217 · ‎12-11-2023

The config is from the hub.

BrianHJones0217 · ‎03-04-2024

I want to work on this issue again. I have rebuilt the lab from the ground up and I am running into the same problem. Here is the current setup.

No matter which site choose, the traffic never go past the firewall. The routing and VPN tunnels seems to be all there. Here is the routing from the the hub (USDAD):

USDAD-FW01 # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
V - BGP VPNv4
* - candidate default

Routing table for VRF=0
S* 0.0.0.0/0 [10/0] via 100.1.1.1, port1, [1/0]
[10/0] via 200.1.1.1, port2, [1/0]
C 10.148.5.0/24 is directly connected, port3
B 10.148.128.0/23 [200/0] via 172.30.0.8 (recursive via advpn-hub tunnel 172.30.0.8), 00:12:36, [
1/0]
B 10.149.96.0/24 [200/0] via 172.30.0.22 (recursive via advpn-hub tunnel 172.30.0.22), 00:12:39,
[1/0]
B 10.151.176.0/23 [200/0] via 172.30.0.59 (recursive via advpn-hub tunnel 172.30.0.59), 00:12:36,
[1/0]
C 100.1.1.0/24 is directly connected, port1
C 172.30.0.1/32 is directly connected, advpn-hub
C 172.30.0.2/31 is directly connected, advpn-hub
S 172.30.0.8/32 [15/0] via advpn-hub tunnel 172.30.0.8, [1/0]
S 172.30.0.22/32 [15/0] via advpn-hub tunnel 172.30.0.22, [1/0]
S 172.30.0.59/32 [15/0] via advpn-hub tunnel 172.30.0.59, [1/0]
C 172.31.0.0/23 is directly connected, advpn2-hub
C 172.31.0.1/32 is directly connected, advpn2-hub
C 200.1.1.0/24 is directly connected, port2

Here is the routing for one of the three spokes, USHO1:

USHO1-FW01 # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
V - BGP VPNv4
* - candidate default

Routing table for VRF=0
S* 0.0.0.0/0 [10/0] via 100.2.2.1, port1, [1/0]
[10/0] via 200.2.2.1, port2, [1/0]
B 10.148.5.0/24 [200/0] via 172.30.0.1 (recursive via advpn-spoke tunnel 100.1.1.2), 00:13:49,
[1/0]
C 10.148.128.0/23 is directly connected, port3
B 10.149.96.0/24 [200/0] via 172.30.0.22 (recursive via advpn-spoke tunnel 100.1.1.2), 00:13:20
, [1/0]
B 10.151.176.0/23 [200/0] via 172.30.0.59 (recursive via advpn-spoke tunnel 100.1.1.2), 00:13:2
0, [1/0]
C 100.2.2.0/24 is directly connected, port1
S 172.30.0.0/23 [5/0] via advpn-spoke tunnel 100.1.1.2, [1/0]
S 172.30.0.1/32 [15/0] via advpn-spoke tunnel 100.1.1.2, [1/0]
C 172.30.0.8/32 is directly connected, advpn-spoke
S 172.31.0.0/23 [5/0] via advpn2-spoke tunnel 200.1.1.2, [1/0]
S 172.31.0.1/32 [15/0] via advpn2-spoke tunnel 200.1.1.2, [1/0]
C 172.31.0.8/32 is directly connected, advpn2-spoke
C 200.2.2.0/24 is directly connected, port2

Here is the sniffer trace:

USHO1-FW01 # diag sniffer packet any 'host 10.148.128.10 and icmp' 4
Using Original Sniffing Mode
interfaces=[any]
filters=[host 10.148.128.10 and icmp]
12.186705 port3 in 10.148.128.10 -> 10.148.5.10: icmp: echo request
14.187416 port3 in 10.148.128.10 -> 10.148.5.10: icmp: echo request
16.187657 port3 in 10.148.128.10 -> 10.148.5.10: icmp: echo request
18.187783 port3 in 10.148.128.10 -> 10.148.5.10: icmp: echo request
20.187973 port3 in 10.148.128.10 -> 10.148.5.10: icmp: echo request
^C
5 packets received by filter
0 packets dropped by kernel

Here is the firewall rule from the spoke config:

config firewall policy
edit 1
set name "outbound-advpn"
set uuid 00f36e2a-da44-51ee-63fd-389fab9f7e79
set srcintf "port3"
set dstintf "advpn-spoke" "advpn2-spoke"
set action accept
set srcaddr "all"
set dstaddr "all"
set schedule "always"
set service "ALL"
next
edit 2
set name "inbound-advpn"
set uuid f52121b8-da44-51ee-fc62-1f2707c4b57e
set srcintf "advpn-spoke" "advpn2-spoke"
set dstintf "port3"
set action accept
set srcaddr "all"
set dstaddr "all"
set schedule "always"
set service "ALL"
next
end

v_ceban · ‎03-04-2024

Please disable add-route on the ipsec phase-1 interface on all involved parts.
config vpn ipsec phase1-interface

edit <tunnel name>

set add-route disable
next

end
Also, make sure the IP addresses and subnet is correctly configured on tunnel interfaces.
On HUB the remote IP should be a dump IP (Not used by any spoke) with the appropriate subnet that will cover all remote locations IP.
The remote IP should be HUB tunnel interface IP with the same subnet on spokes.

Vladislav Ceban

BrianHJones0217 · ‎03-04-2024

The set add-route disable is configured on all the tunnels on all four firewalls.

IPs are configured correctly on the tunnel interfaces.

on the HUB, the remote-ip is a DUMP not used by any spoke with the correct subnet.

on the SPOKEs, the remote IP is the hub tunnel interface IP with the correct subnet.

BrianHJones0217 · ‎03-05-2024

All the IPs on the Internet side are reachable from all firewalls, so they can ping the external addresses.

Here are the phase-1 configs:

(Hub)
config vpn ipsec phase1-interface
edit "advpn-hub"
set type dynamic
set interface "port1"
set peertype any
set net-device disable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-sender enable
set auto-discovery-forwarder enable
set psksecret ENC ...
set dpd-retryinterval 5
next
edit "advpn2-hub"
set type dynamic
set interface "port2"
set peertype any
set net-device disable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-sender enable
set auto-discovery-forwarder enable
set psksecret ENC ...
set dpd-retryinterval 5
next
end

(Spoke)
config vpn ipsec phase1-interface
edit "advpn-spoke"
set interface "port1"
set peertype any
set net-device enable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-receiver enable
set remote-gw 100.1.1.2
set psksecret ENC ...
set dpd-retryinterval 5
next
edit "advpn2-spoke"
set interface "port2"
set peertype any
set net-device enable
set proposal des-md5 des-sha1 des-sha256
set add-route disable
set dpd on-idle
set npu-offload disable
set auto-discovery-receiver enable
set remote-gw 200.1.1.2
set psksecret ENC ...
set dpd-retryinterval 5
next
end

BrianHJones0217 · ‎03-05-2024

Here are the system interface configs:

(Hub)

edit "advpn-hub"
set vdom "root"
set ip 172.30.0.1 255.255.255.255
set allowaccess ping
set type tunnel
set tcp-mss 1360
set remote-ip 172.30.0.2 255.255.254.0
set snmp-index 9
set interface "port1"
set mtu-override enable
set mtu 1400
next
edit "advpn2-hub"
set vdom "root"
set ip 172.31.0.1 255.255.255.255
set allowaccess ping
set type tunnel
set tcp-mss 1360
set remote-ip 172.31.0.2 255.255.254.0
set snmp-index 10
set interface "port2"
set mtu-override enable
set mtu 1400
next

(Spoke)
edit "advpn-spoke"
set vdom "root"
set ip 172.30.0.8 255.255.255.255
set type tunnel
set remote-ip 172.30.0.1 255.255.254.0
set snmp-index 9
set interface "port1"
next
edit "advpn2-spoke"
set vdom "root"
set ip 172.31.0.8 255.255.255.255
set type tunnel
set remote-ip 172.31.0.1 255.255.254.0
set snmp-index 10
set interface "port2"
nextt
end