Link selection in SDWAN

[JanDevos]

Hi support forum, 

I am already for quite a time trying to influence SDWAN's path selection on my GNS3 simulation environment, without success.

I run FortiOS 7.0.9, for obvious reasons.

The topology is very easy : two FG's connected to each other via 'WAN1', 'WAN2' and 'back to back'.  WAN1 and WAN2 are routed networks (a couple of CIsco's).  Test flow is between loopback interface of both FG's.  The healthchecks use also these loopback IP's as probe addresses.

I have embraced the three phy interfaces (WAN1, 2 and back to back) by one SDWAN software interface.  In my simplest configuration, I don't add any cost and any SLA.  A ping health check monitors their availabilty.  No service is defined at this stage.

The three members just appear with similar attributes under SDWAN.  All 3 are alive, as long as the health-check states them available.

FW-Gos # diag sys sdwan memb
Member(2): interface: VLAN_999, flags=0x0 , gateway: 9.9.3.2, priority: 1 1024, weight: 0
Member(1): interface: port6, flags=0x0 , gateway: 9.9.9.2, priority: 1 1024, weight: 0
Member(3): interface: port2, flags=0x0 , gateway: 8.8.3.2, priority: 1 1024, weight: 0

The actual path followed by the test flow is via member (3, port2). I suppose this is due to the fact that it appears as first in the physical routing table, automatically generated by the SDWAN gateway configuration.

FW-Gos # get router info routing-table stat
Routing table for VRF=0
S 10.10.10.2/32 [1/0] via 8.8.3.2, port2, [1/0]       <<<< first one in routing table 
[1/0] via 9.9.3.2, VLAN_999, [1/0]
[1/0] via 9.9.9.2, port6, [1/0]

Now, i want SDWAN to make an 'intelligent' decision, and overrule the above route selection.  Therefore, I assigned a differential cost (5-10-15) to the three members, favoring VLAN_999.  E.g. for VLAN_999:

config members
edit 2
set interface "VLAN_999"
set gateway 9.9.3.2
set cost 5
next

Verifying the member status, they are again all on-par, and there is no appearance of cost, nor of any other differentiating attribute that could influence interface selection by SDWAN.

FW-Gos # diag sys sdwan memb
Member(2): interface: VLAN_999, flags=0x0 , gateway: 9.9.3.2, priority: 1 1024, weight: 0
Member(1): interface: port6, flags=0x0 , gateway: 9.9.9.2, priority: 1 1024, weight: 0
Member(3): interface: port2, flags=0x0 , gateway: 8.8.3.2, priority: 1 1024, weight: 0

... and the selected path stays member 3 = port 2.

Thereafter I tried by defining a service that takes in all traffic (dest = src='all'), aimed to select the link by 'quality' (= set mode priority), and factoring in the latency (= set link-cost-factor latency -- it does not appear in the below 'show', i suppose because latency is the default cost factor).

FW-Gos (service) # show
config service
edit 1
set name "anyService"
set mode priority
set dst "all"
set src "all"
set health-check "PingSLA"FW-Gos # diag sys sdwan memb
Member(2): interface: VLAN_999, flags=0x0 , gateway: 9.9.3.2, priority: 1 1024, weight: 0
Member(1): interface: port6, flags=0x0 , gateway: 9.9.9.2, priority: 1 1024, weight: 0
Member(3): interface: port2, flags=0x0 , gateway: 8.8.3.2, priority: 1 1024, weight: 0

set priority-members 2 3 1
next
end

 This configuration is still not of any influence on the instataneous member link status

FW-Gos # diag sys sdwan memb
Member(2): interface: VLAN_999, flags=0x0 , gateway: 9.9.3.2, priority: 1 1024, weight: 0
Member(1): interface: port6, flags=0x0 , gateway: 9.9.9.2, priority: 1 1024, weight: 0
Member(3): interface: port2, flags=0x0 , gateway: 8.8.3.2, priority: 1 1024, weight: 0

The service reveals the three member links as qualifying to forward the traffic within this service, with the lowest-latency link as first in order, which  seems logic.

Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
Gen(3), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(priority), link-cost-factor(latency), link-cost-threshold(10), heath-check(PingSLA)
Members(3):
1: Seq_num(1 port6), alive, latency: 3.701, selected
2: Seq_num(2 VLAN_999), alive, latency: 63.098, selected
3: Seq_num(3 port2), alive, latency: 60.418, selected
Src address(1):
0.0.0.0-255.255.255.255

Dst address(1):
0.0.0.0-255.255.255.255

However, SDWAN still forwards via  port2.

Interesting to note is that failover from the forwarding interface to an alternate interface works as a charm (max. 5 pings lost, this is even faster than what I would expect by the default behavior of 30 packets having to time out over 500 millisec before failing over = 15 sec).  (port2 > VLAN_999 > port6).  When a failing interface restarts its life, it takes the flow back immediately.

Any help or sug

[Julien87]

Hi Jan,

you can access in the site https://training.fortinet.com and after login, you can find the nse selfpaced pdf.

[JanDevos]

Found now, and contains interesting things.  

[JanDevos]

Can you please leave this thread open, as i am not yet able to steer SDWAN traffic the way I want;

[Julien87]

If you want work in Lab, i have send a private message. 

Julien

[JanDevos]

I am now in the above topology and trying to do some traffic steering.  All test flows go from EP-GOS (7.7.3.2)  to EP-EuP (7.7.2.2), i.e. 'left to right'.  Of the three paths, I disfavored the 'blue' routed connection (set priority 10 on SDWAN member).  This leaves me with 2 ECMP routes, the third one being less preferred, see below.

FW-Gos # diag sys sdwan route
SD-WAN route(1) 10.10.10.2 255.255.255.255, gw(9.9.3.2), oif(VLAN_999, 18), distance/weight/prio(1/0/10)  >>> becomes backup
SD-WAN route(1) 10.10.10.2 255.255.255.255, gw(9.9.9.2), oif(port6, 8), distance/weight/prio(1/0/1)
SD-WAN route(1) 10.10.10.2 255.255.255.255, gw(8.8.3.2), oif(port2, 4), distance/weight/prio(1/0/1)

However, all traffic goes over one of the two remaining ECMP routes (port6), never selecting the other (port2).  This is e.g. shown by the session table in de FG of the source side (diag sys session list, with filter on  source address 7.7.3.2).  And also when applying a 'diag sniffer packet' on the egress interfaces of this same FG: nothing forwarded on port2, all forwarded on port6.  And of course, nothing forwarded on VLAN_999, as expected, due to its lower pref. 

Health check is "alive" for all three links.

relevant configs are now (on FW-GOS1, FG of source side, which has to steer the traffic from 'left to right':

FW-Gos (sdwan) # config members

FW-Gos (members) # show
config members
edit 2
set interface "VLAN_999"
set gateway 9.9.3.2
set priority 10
next
edit 1
set interface "port6"
set gateway 9.9.9.2
next
edit 3
set interface "port2"
set gateway 8.8.3.2
next
end

and no SLA set in health-check, so no SDWAN rule applies

FW-Gos (PingSLA) # show
config health-check
edit "PingSLA"
set server "10.10.10.2"
set members 0
next
end

As an ennoying side effect, we often have the case that new sessions are not established at all while existing sessions continue to work.  Only after some arbitrary time (minutes) new sessions restart to establish, but still on the same path, not balanced over the other ECMP path.

[JanDevos]

... also, when disabling port6, all sessions transparently (within some sec) fail over to port2, as expected, and when also port2 disappears, same rapid failover to the backup port VLAN_999 occurs.  So, failover seems to work as a charm.  When port2 is back, it takes back the sessions from thel less-preferred VLAN_999 (allthough a route cache entry for VLAN_999 still exists!).  This take back is somewhat slower than the above failover, for some unknown reason.  When finally also port6 is back, all sessions stay on port2, i.e. sessions are not re-balanced over both ECMP interfaces.  Also, there is no cache entry for port6.  

Remark also that in the session list output there are no SDWAN 'lines'.  But this is probably normal, as currently i have no SDWAN rules/service in place, just health-checks.  

But I would expect that in this case, with a FIB having 2 ECMP's, we would see the sessions round-robin balanced.

[JanDevos]

... selection goes to 3 different egress interfaces, depending on the case

1) case-1: ping from FG1 to FG2, ping-option 'use-sdwan no' : goes to port2, the first one of the routing table 

FW-Gos # get router info routing-table static
Routing table for VRF=0
S 10.10.10.2/32 [1/0] via 8.8.3.2, port2, [1/0]
[1/0] via 9.9.3.2, VLAN_999, [1/0]
[1/0] via 9.9.9.2, port6, [1/0]

2) case-2 : same ping, but with ping-option use-sdwan yes 

first in rank member link is used (port6), as this one is lowest latency, and latency is used as SLA-factor

Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
Gen(28), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
Members(3):
1: Seq_num(1 port6), alive, sla(0x1), gid(0), cfg_order(2), cost(15), selected
2: Seq_num(2 VLAN_999), alive, sla(0x0), gid(0), cfg_order(0), cost(5), selected
3: Seq_num(3 port2), alive, sla(0x0), gid(0), cfg_order(1), cost(10), selected

3) Telnet from FG1 to FG2 (same source and dest IP as above cases): port2 is taken, rather than taking like in case-2 the one with the lowest latency (port6), as determined by latency-SLA.  Even when configuring the lowest latency interface also with the lowest cost, telnet session still goes over port2.  

Why does SDWAN select different interfaces for ICMP vs TCP?