Fortigate 60E SDWAN rule not work.

[downlinkvip1]

I have an ADVPN setup between Hub and Spoke. At the Spoke, I get BGP routes like that.

LAN - HUB(WAN1) - SPOKE (WAN1) - LAN

# get router info routing-table bgp
Routing table for VRF=0
B       10.0.10.10/32 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       10.0.10.11/32 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 02:59:13, [1/0]
B       10.0.10.12/32 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 00:28:36, [1/0]
B       10.0.10.13/32 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 01:19:56, [1/0]
B       10.0.10.14/32 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 01:52:28, [1/0]
B       10.0.10.15/32 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 00:04:37, [1/0]
B       10.100.100.1/32 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       10.100.100.2/32 [200/0] via 10.10.2.3 (recursive is directly connected, ADVPN-CMC_0), 03:00:50, [1/0]
B       10.100.100.3/32 [200/0] via 10.10.2.6 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       10.100.100.5/32 [200/0] via 10.10.2.5 (recursive is directly connected, ADVPN-CMC_2), 01:52:50, [1/0]
B       10.100.100.7/32 [200/0] via 10.10.2.7 (recursive is directly connected, ADVPN-CMC_1), 00:14:50, [1/0]
B       172.16.10.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       172.16.80.0/24 [200/0] via 10.10.2.5 (recursive is directly connected, ADVPN-CMC_2), 01:52:50, [1/0]
B       172.16.100.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       172.17.17.0/24 [200/0] via 10.10.2.6 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       192.168.1.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       192.168.10.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       192.168.15.0/24 [200/0] via 10.10.2.3 (recursive is directly connected, ADVPN-CMC_0), 03:00:50, [1/0]
B       192.168.20.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       192.168.25.0/24 [200/0] via 10.10.2.7 (recursive is directly connected, ADVPN-CMC_1), 00:14:50, [1/0]
B       192.168.43.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       192.168.50.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       192.168.60.0/24 [200/0] via 10.10.2.6 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       192.168.65.0/24 [200/0] via 10.10.2.6 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       192.168.70.0/24 [200/0] via 10.10.2.8 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       192.168.80.0/24 [200/0] via 10.10.2.5 (recursive is directly connected, ADVPN-CMC_2), 01:52:50, [1/0]
B       192.168.81.0/24 [200/0] via 10.10.2.5 (recursive is directly connected, ADVPN-CMC_2), 01:52:50, [1/0]
B       192.168.85.0/24 [200/0] via 10.10.2.5 (recursive is directly connected, ADVPN-CMC_2), 01:52:50, [1/0]
B       192.168.100.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]
B       192.168.200.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.213 vrf 0), 03:01:04, [1/0]

For eg, I tracert from my local subnet to 192.168.25.0/24 or 192.168.50.0/24, it should go to 10.10.2.x ..., but tracert result alway show that, it go directly to WAN' gateway and time out IP like this:

C:\Windows\system32>tracert 192.168.50.254

Tracing route to 192.168.50.254 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  192.168.90.254 
  2     3 ms     2 ms     1 ms  [123.29.4.114] 
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.

Can you help give me any keyword or hint, so I can resolve this issue?

[aionescu]

Hello @downlinkvip1 ,

Welcome to the community.

Can you paste the output of "get router info routing-table database".

You also mention that the SDWAN rule is not working.  Can you provide some details about the configuration? Have you configured any health-checks? If yes, what is the state?

[downlinkvip1]

Hi @aionescu,

Indeed, after a few days, I even deleted the SDWAN rule. So, the traffic will go based on the routing table, right?

Routing table for VRF=0
B       0.0.0.0/0 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
S    *> 0.0.0.0/0 [1/0] via 123.29.4.xxx, ppp3, [1/0]
     *>           [1/0] via 123.29.4.xxx, ppp4, [1/0]
     *>           [1/0] via 183.91.0.xxx, ppp2, [1/0]
S       10.0.0.5/32 [5/0] via DCGE110-PC3 tunnel 10.0.0.3 vrf 0 inactive, [1/0]
B    *> 10.0.10.10/32 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 00:08:12, [1/0]
S    *> 10.10.1.0/24 [5/0] via ADVPN-VNPT tunnel 113.160.108.168 vrf 0, [1/0]
C    *> 10.10.1.4/32 is directly connected, ADVPN-VNPT
S    *> 10.10.2.0/24 [5/0] via ADVPN-CMC tunnel 183.91.15.xxx vrf 0, [1/0]
S    *> 10.10.2.1/32 [15/0] via ADVPN-CMC tunnel 183.91.15.xxx vrf 0, [1/0]
C    *> 10.10.2.3/32 is directly connected, ADVPN-CMC_0
C    *> 10.10.2.4/32 is directly connected, ADVPN-CMC
     *>              is directly connected, ADVPN-CMC_1
     *>              is directly connected, ADVPN-CMC_0
     *>              is directly connected, ADVPN-CMC_2
C    *> 10.10.2.5/32 is directly connected, ADVPN-CMC_2
C    *> 10.10.2.7/32 is directly connected, ADVPN-CMC_1
B    *> 10.100.100.1/32 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 10.100.100.2/32 [200/0] via 10.10.2.3 (recursive is directly connected, ADVPN-CMC_0), 06:26:33, [1/0]
B    *> 10.100.100.3/32 [200/0] via 10.10.2.6 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 10.100.100.5/32 [200/0] via 10.10.2.5 (recursive is directly connected, ADVPN-CMC_2), 02:42:33, [1/0]
B    *> 10.100.100.7/32 [200/0] via 10.10.2.7 (recursive is directly connected, ADVPN-CMC_1), 06:51:33, [1/0]
C    *> 10.100.100.90/32 is directly connected, loopback
C    *> 45.122.233.3/32 is directly connected, ppp2
C    *> 113.160.96.171/32 is directly connected, ppp4
C    *> 113.160.206.239/32 is directly connected, ppp3
C    *> 123.29.4.114/32 is directly connected, ppp3
     *>                 is directly connected, ppp4
B    *> 172.16.10.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 172.16.80.0/24 [200/0] via 10.10.2.5 (recursive is directly connected, ADVPN-CMC_2), 02:42:33, [1/0]
O       172.16.90.0/24 [110/1] is directly connected, VLAN99, 2d10h32m, [1/0]
C    *> 172.16.90.0/24 is directly connected, VLAN99
B    *> 172.16.100.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 172.17.17.0/24 [200/0] via 10.10.2.6 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
C    *> 183.91.0.138/32 is directly connected, ppp2
B    *> 192.168.1.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 192.168.10.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 192.168.15.0/24 [200/0] via 10.10.2.3 (recursive is directly connected, ADVPN-CMC_0), 06:26:33, [1/0]
B    *> 192.168.20.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 192.168.25.0/24 [200/0] via 10.10.2.7 (recursive is directly connected, ADVPN-CMC_1), 06:51:33, [1/0]
B    *> 192.168.43.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 192.168.50.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 192.168.60.0/24 [200/0] via 10.10.2.6 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 192.168.65.0/24 [200/0] via 10.10.2.6 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 192.168.70.0/24 [200/0] via 10.10.2.8 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 192.168.80.0/24 [200/0] via 10.10.2.5 (recursive is directly connected, ADVPN-CMC_2), 02:42:33, [1/0]
B    *> 192.168.81.0/24 [200/0] via 10.10.2.5 (recursive is directly connected, ADVPN-CMC_2), 02:42:33, [1/0]
B    *> 192.168.85.0/24 [200/0] via 10.10.2.5 (recursive is directly connected, ADVPN-CMC_2), 02:42:33, [1/0]
O       192.168.90.0/24 [110/1] is directly connected, VLAN90, 2d10h32m, [1/0]
C    *> 192.168.90.0/24 is directly connected, VLAN90
O       192.168.91.0/24 [110/1] is directly connected, VLAN91, 2d10h32m, [1/0]
C    *> 192.168.91.0/24 is directly connected, VLAN91
O       192.168.95.0/24 [110/1] is directly connected, VLAN95, 2d10h32m, [1/0]
C    *> 192.168.95.0/24 is directly connected, VLAN95
B    *> 192.168.100.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]
B    *> 192.168.200.0/24 [200/0] via 10.10.2.1 (recursive via ADVPN-CMC tunnel 183.91.15.xxx vrf 0), 1d22h26m, [1/0]

I still got the error. For eg, 192.168.25.1 will go through the tunnel but 192.168.25.2 will go directly to WAN gateway.

[aionescu]

Hi @downlinkvip1 can you share also the output of get router policy

Also, make sure there is no session between the communicating hosts and then generate the traffic while running the following commands:

diagnose debug flow filter addr x.x.x.x <------ where x.x.x.x is the source of the traffic
diagnose debug flow trace start 100
diagnose debug enable

..................

then show the session with:

diagnose sys session filter src x.x.x.x where x.x.x.x is the source of the traffic
diagnose sys session filter dst y.y.y.ywhere y.y.y.y is the destination of the traffic
diagnose sys session list

[downlinkvip1]

Hi @aionescu 

"get router policy" shows nothing.

At this weekend, I will run debug command and send to you. Thank you!

[downlinkvip1]

Hi @aionescu ,

I just use your debug command to troubleshoot the connection from HQ' host 192.168.1.10 to Br'host 192.168.90.188 (through ADVPN) and get this log.

id=65308 trace_id=135 func=print_pkt_detail line=5902 msg="vd-root:0 received a packet(proto=1, 192.168.1.10:2->192.168.90.188:2048) tun_id=113.160.108.168 from ADVPN-VNPT. type=8, code=0, id=2, seq=34699."
id=65308 trace_id=135 func=resolve_ip_tuple_fast line=5985 msg="Find an existing session, id-01ee5cfd, original direction"
id=65308 trace_id=135 func=npu_handle_session44 line=1175 msg="Trying to offloading session from ADVPN-VNPT to VLAN90, skb.npu_flag=00000400 ses.state=00010204 ses.npu_state=0x04000000"
id=65308 trace_id=135 func=fw_forward_dirty_handler line=414 msg="state=00010204, state2=00000001, npu_state=04000000"
id=65308 trace_id=136 func=print_pkt_detail line=5902 msg="vd-root:0 received a packet(proto=1, 192.168.1.10:2->192.168.90.188:2048) tun_id=113.160.108.168 from ADVPN-VNPT. type=8, code=0, id=2, seq=34700."
id=65308 trace_id=136 func=resolve_ip_tuple_fast line=5985 msg="Find an existing session, id-01ee5cfd, original direction"
id=65308 trace_id=136 func=npu_handle_session44 line=1175 msg="Trying to offloading session from ADVPN-VNPT to VLAN90, skb.npu_flag=00000400 ses.state=00010204 ses.npu_state=0x04000000"
id=65308 trace_id=136 func=fw_forward_dirty_handler line=414 msg="state=00010204, state2=00000001, npu_state=04000000"

The gateway of branch subnet 192.168.90.0/24 is 192.168.90.254 (is a VLAN interface at physical Fortigate port 7). 

From HQ site, we can ping to 192.168.90.254 (through ADVPN), but can not ping to 192.168.90.188. 

Please kindly help as I don't know why branch Fortigate not forward packet out port 7 or something like that.