Link selection in SDWAN

[JanDevos]

Hi support forum, 

I am already for quite a time trying to influence SDWAN's path selection on my GNS3 simulation environment, without success.

I run FortiOS 7.0.9, for obvious reasons.

The topology is very easy : two FG's connected to each other via 'WAN1', 'WAN2' and 'back to back'.  WAN1 and WAN2 are routed networks (a couple of CIsco's).  Test flow is between loopback interface of both FG's.  The healthchecks use also these loopback IP's as probe addresses.

I have embraced the three phy interfaces (WAN1, 2 and back to back) by one SDWAN software interface.  In my simplest configuration, I don't add any cost and any SLA.  A ping health check monitors their availabilty.  No service is defined at this stage.

The three members just appear with similar attributes under SDWAN.  All 3 are alive, as long as the health-check states them available.

FW-Gos # diag sys sdwan memb
Member(2): interface: VLAN_999, flags=0x0 , gateway: 9.9.3.2, priority: 1 1024, weight: 0
Member(1): interface: port6, flags=0x0 , gateway: 9.9.9.2, priority: 1 1024, weight: 0
Member(3): interface: port2, flags=0x0 , gateway: 8.8.3.2, priority: 1 1024, weight: 0

The actual path followed by the test flow is via member (3, port2). I suppose this is due to the fact that it appears as first in the physical routing table, automatically generated by the SDWAN gateway configuration.

FW-Gos # get router info routing-table stat
Routing table for VRF=0
S 10.10.10.2/32 [1/0] via 8.8.3.2, port2, [1/0]       <<<< first one in routing table 
[1/0] via 9.9.3.2, VLAN_999, [1/0]
[1/0] via 9.9.9.2, port6, [1/0]

Now, i want SDWAN to make an 'intelligent' decision, and overrule the above route selection.  Therefore, I assigned a differential cost (5-10-15) to the three members, favoring VLAN_999.  E.g. for VLAN_999:

config members
edit 2
set interface "VLAN_999"
set gateway 9.9.3.2
set cost 5
next

Verifying the member status, they are again all on-par, and there is no appearance of cost, nor of any other differentiating attribute that could influence interface selection by SDWAN.

FW-Gos # diag sys sdwan memb
Member(2): interface: VLAN_999, flags=0x0 , gateway: 9.9.3.2, priority: 1 1024, weight: 0
Member(1): interface: port6, flags=0x0 , gateway: 9.9.9.2, priority: 1 1024, weight: 0
Member(3): interface: port2, flags=0x0 , gateway: 8.8.3.2, priority: 1 1024, weight: 0

... and the selected path stays member 3 = port 2.

Thereafter I tried by defining a service that takes in all traffic (dest = src='all'), aimed to select the link by 'quality' (= set mode priority), and factoring in the latency (= set link-cost-factor latency -- it does not appear in the below 'show', i suppose because latency is the default cost factor).

FW-Gos (service) # show
config service
edit 1
set name "anyService"
set mode priority
set dst "all"
set src "all"
set health-check "PingSLA"FW-Gos # diag sys sdwan memb
Member(2): interface: VLAN_999, flags=0x0 , gateway: 9.9.3.2, priority: 1 1024, weight: 0
Member(1): interface: port6, flags=0x0 , gateway: 9.9.9.2, priority: 1 1024, weight: 0
Member(3): interface: port2, flags=0x0 , gateway: 8.8.3.2, priority: 1 1024, weight: 0

set priority-members 2 3 1
next
end

 This configuration is still not of any influence on the instataneous member link status

FW-Gos # diag sys sdwan memb
Member(2): interface: VLAN_999, flags=0x0 , gateway: 9.9.3.2, priority: 1 1024, weight: 0
Member(1): interface: port6, flags=0x0 , gateway: 9.9.9.2, priority: 1 1024, weight: 0
Member(3): interface: port2, flags=0x0 , gateway: 8.8.3.2, priority: 1 1024, weight: 0

The service reveals the three member links as qualifying to forward the traffic within this service, with the lowest-latency link as first in order, which  seems logic.

Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
Gen(3), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(priority), link-cost-factor(latency), link-cost-threshold(10), heath-check(PingSLA)
Members(3):
1: Seq_num(1 port6), alive, latency: 3.701, selected
2: Seq_num(2 VLAN_999), alive, latency: 63.098, selected
3: Seq_num(3 port2), alive, latency: 60.418, selected
Src address(1):
0.0.0.0-255.255.255.255

Dst address(1):
0.0.0.0-255.255.255.255

However, SDWAN still forwards via  port2.

Interesting to note is that failover from the forwarding interface to an alternate interface works as a charm (max. 5 pings lost, this is even faster than what I would expect by the default behavior of 30 packets having to time out over 500 millisec before failing over = 15 sec).  (port2 > VLAN_999 > port6).  When a failing interface restarts its life, it takes the flow back immediately.

Any help or sug

[Julien87]

Hi Jan,

What is your load balande mode in implicit rule ?

Have you Policy base route ?

Best regards,

[JanDevos]

Julien,

Loadbalancing is per default (source IP based). Be aware I have only one single flow.  I have no PBR.  3 static routes are implicitly generated by SDWAN’s single statement ‘

config router static

    edit 1

        set dst 10.10.10.2 255.255.255.255

        set distance 1

        set sdwan-zone "virtual-wan-link"

    next

resulting in

FW-Gos # get router info routing-table static

Routing table for VRF=0

S       10.10.10.2/32 [1/0] via 8.8.3.2, port2, [1/0]

                      [1/0] via 9.9.3.2, VLAN_999, [1/0]

                      [1/0] via 9.9.9.2, port6, [1/0]

Here is the rest of the interesting configuration sections.  For the ‘originating’ node (I.e. the one originating the Ping test flow.  10.10.10.3 >10.10.10.2)

FW-Gos # config sys sdwan

FW-Gos (sdwan) # show

config system sdwan

    set status enable

    config zone

        edit "virtual-wan-link"

        next

    end

    config members

        edit 2

            set interface "VLAN_999"

            set gateway 9.9.3.2

            set cost 5

        next

        edit 1

            set interface "port6"

            set gateway 9.9.9.2

            set cost 15

        next

        edit 3

            set interface "port2"

            set gateway 8.8.3.2

            set cost 10

        next

    end

    config health-check

        edit "Default_DNS"

            set system-dns enable

            set interval 1000

        < ... and all the default gratuitous probes go here, not of any use ...  the used probe is here below; >

        edit "PingSLA"

            set server "10.10.10.2"

            set members 0

        next

    end

    config service

        edit 1

            set name "anyService"

            set mode priority

            set dst "all"

            set src "all"

            set health-check "PingSLA"

            set priority-members 2 3 1

        next

    end

end

FW-Gos (sdwan) #

[Julien87]

Hi Jan

Can you try with set mode sla in your config service.

Because, the cost  is using by the lowest cost as stragegy rules.

The mode priority use the latency.

Best regards,

[JanDevos]

Julien, I did this already earlier, but with pleasure, I restore this 'mode sla' configuration.  Here it is.  

Service is now with mode sla 

FW-Gos (service) # show
config service
edit 1
set name "anyService"
set mode sla
set dst "all"
set src "all"
config sla
edit "PingSLA"
set id 1
next
end

set priority-members 2 3 1
next
end

It points to a modified health-check, that now contains latency as cost factor.

FW-Gos (PingSLA) # show
config health-check
edit "PingSLA"
set server "10.10.10.2"
set members 0
config sla
edit 1
set link-cost-factor latency
set latency-threshold 15
next

end

next 

end

The resulting service shows up like this

FW-Gos # diag sys sdwan servic

Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
Gen(4), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
Members(3):
1: Seq_num(1 port6), alive, sla(0x1), gid(0), cfg_order(2), cost(15), selected
2: Seq_num(2 VLAN_999), alive, sla(0x0), gid(0), cfg_order(0), cost(5), selected
3: Seq_num(3 port2), alive, sla(0x0), gid(0), cfg_order(1), cost(10), selected
Src address(1):
0.0.0.0-255.255.255.255

Dst address(1):
0.0.0.0-255.255.255.255

Sequence makes sense : the one with lowest latency is first.  The two others are close to each other in latency (within the bounds of threshold = 15), so there the one with lowest cost has taken precedence.  Note that all three are 'selected', but only the low-latency one is flagged sla '0x1'
This is confirmed by the appearance of the links in the health-check. Note that here as well, only the 'low latency' link is flagged with sla map '0x1'

FW-Gos # diag sys sdwan health
Health Check(PingSLA):
Seq(2 VLAN_999): state(alive), packet-loss(0.000%) latency(56.930), jitter(9.193), bandwidth-up(999998), bandwidth-dw(999998), bandwidth-bi(1999996) sla_map=0x0
Seq(1 port6): state(alive), packet-loss(0.000%) latency(3.161), jitter(0.831), bandwidth-up(999998), bandwidth-dw(999998), bandwidth-bi(1999996) sla_map=0x1
Seq(3 port2): state(alive), packet-loss(0.000%) latency(60.312), jitter(3.275), bandwidth-up(999998), bandwidth-dw(999998), bandwidth-bi(1999996) sla_map=0x0

We would now expect that the low latency link takes in the test flow.  But it is not.  Test flow still goes on port2 (the third in rank).

[JanDevos]

Julien, et al, 

I think I found the source of the problem.

When I explicitly ask 'use-sdwan yes' in the ping-options for the test flow, the test flow is now absorbed by the first-in-rank egress interface, i.e. the one with the lowest latency.  I am curious how SDWAN would behave for a flow other than ping.

[Julien87]

Hi Jan,

haaa ok, indeed if you launch the requests from the fortinet equipment, you have to define with which source you have to do it.

I prefer use a device PC in the LAN behind the Fortinet, for this reason.

You can perhaps use source-interface (LAN) for Telnet. I cannot check this one this day.

For information you have this option for multiple service (DNS, Fortiguard)

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Functionality-of-set-interface-select-meth... 

[JanDevos]

That's what I did, Julien, see my results below (which are not consistent for both types of traffic, telnet and ICMP).  Anyway, I am now extending my test setup with two 'true' endpoints (cisco routers) upstream the two fortigates (diagram, where you also see the three 'WAN's).  I let you know the results.

[Julien87]

Jan

I strongly recommend the self training nse 7 sd wan. All the parameters and how to calculate are well detailed.

I'm working on it , and it's informative.

[JanDevos]

Julien, thank you for this recommendation, but I only find an 'OT' flavor of NSE7, not an SDWAN flavour.  NSE7 OT Security 7.2