Local out traffic selects the wrong SDWAN gateway for outgoing traffic ?

[BensonLEI]

Hi, guys,

I am currently using Fortigate 400E with FortiOS v7.0.3, with the SDWAN configuration of 3 internet lines.

I tried to test the destination IP with traceroute/pingtest as the following test cases:

SDWAN configuration:

1. service rule = Maximized

2. SDWAN Mode(load-balance hash-mode=round-robin)
3. load-balance-mode : source-dest-ip-based

4. duplication-max-num : 3

Firewall policy "local-in-policy" :

Forti400e01 (local-in-policy) # show
config firewall local-in-policy
end

My traceroute test ( pingtest is similar )

Case 1 --- Traceroute test by using dedicated internet line
==========================================
Forti400e01 # exec traceroute-options device port3
Forti400e01 # exec traceroute 91.240.118.105

Forti400e01 # exec traceroute-options device port4
Forti400e01 # exec traceroute 91.240.118.105

Forti400e01 # exec traceroute-options device port5
Forti400e01 # exec traceroute 91.240.118.105

Case 2 --- Traceroute test by using SDWAN rule
=============================================
Forti400e01 # exec traceroute-options source 100.100.100.10
Forti400e01 # exec traceroute-options use-sdwan yes
Forti400e01 # exec traceroute 91.240.118.105

Forti400e01 # exec traceroute-options source 222.222.222.22
Forti400e01 # exec traceroute-options use-sdwan yes
Forti400e01 # exec traceroute 91.240.118.105

Forti400e01 # exec traceroute-options source 111.111.111.11
Forti400e01 # exec traceroute-options use-sdwan yes
Forti400e01 # exec traceroute 91.240.118.105

Case 3 -- Traceroute test by using Source IP
==============================================
Forti400e01 # exec traceroute-options source 100.100.100.10
Forti400e01 # exec traceroute 91.240.118.105

Forti400e01 # exec traceroute-options source 222.222.222.22
Forti400e01 # exec traceroute 91.240.118.105

Forti400e01 # exec traceroute-options source 111.111.111.11
Forti400e01 # exec traceroute 91.240.118.105

Test results:

==========

For case 1,  every 'traceroute' completes perfectly;

For case 3, most of 'traceroute' completes (around 1 or 2 times failures in 10 times);

For case 2, most of 'traceroute' can not be performed ( 70% number of times  failure )

The  cause is found in the Route Cache (Forti400e01 # diag ip rtcache list) :

family=02 tab=254 vrf=0 vf=0 type=01 tos=0 flag=00000200
100.100.100.10@0->91.240.118.105@11(port4) gwy=222.222.222.2 prefsrc=0.0.0.0
ci: ref=0 lastused=41 expire=0 err=00000000 used=12 br=0 pmtu=1500

what is the reason the test always goes to the wrong gateway ( from port3 --> port4 ) ?

Any issue of my test ?  any suggestion/recommendation ?

With many thanks

Benson

[jintrah_FTNT]

Hi,

What would be the purpose of the test or what you are aiming for? Do you expect traffic sourced with source IP of isp1 and send out through isp2 be routed back to isp2 interface by isp2 and other upstream in the path towards 91.240.118.105?

best regards,

Jin

[BensonLEI]

Thanks for your reply, my queries/questions/conerns:

1.  What is reason when I do traceroute with sourced IP = isp1, the traffic goes through the wrong and not-working gateway/isp2 ( but not goes through isp1 ) ? by what condtion/standard/configuration/policy/route the fortigate makes this decision ?

2. can I make change of these condition/standard/configuration/policy/route ?

Thanks a lot

Benson

[jintrah_FTNT]

Hi Benson,

During test2 and test3, was any changes done in routing table or config? or both these tests were done with same configuration? Was there a route in route-table for destination 91.240.118.105 during any of the test 2 or test 3 case?

Best regards,

Jin

[BensonLEI]

Hi, Jin,

Same configuration for all tests; and the default route 0.0.0.0 0.0.0 for all internet lines as below:

Forti400e01 # get router info routing-table static
Routing table for VRF=0
S* 0.0.0.0/0 [1/0] via 100.100.100.1, port3
                      [1/0] via 222.222.222.2, port4
                      [1/0] via 111.111.111.1, port5

Forti400e01 # diag sys sdwan service
.....
.....
Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla
Gen(4), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(load-balance hash-mode=round-robin)
Members(3):
1: Seq_num(1 port3), alive, sla(0x1), gid(2), num of pass(1), selected
2: Seq_num(2 port4), alive, sla(0x1), gid(2), num of pass(1), selected
3: Seq_num(3 port5), alive, sla(0x1), gid(2), num of pass(1), selected
Src address(1):
0.0.0.0-255.255.255.255

Dst address(1):
0.0.0.0-255.255.255.255

....
Forti400e01 #

Thanks

Benson

[BensonLEI]

Hin Jin,

Traceroute and ping test failure is found for the SDWAN configuration of 3 internet lines; 

This problem seems not be found for the SDWAN configuration of 2 internet lines;

I shall verify this problem ( by upgrading the FortiOS to v7.2.2 ).

Thanks so much for your kind input in advance.

[akristof]

Hello,

The difference in your Test1 and Test3 case is the source. On first case, you are specifying source interface. So basically, you are "skipping" whole routing lookup, there is just check if the route to destination via that interface exists to get the gateway. And then the source IP is taken from outgoing interface.

In test case scenario 3, you are specifying source IP address. Without the use of sdwan, FortiOS will do standard routing lookup. Only difference is that it will select outgoing route based on ldb-algorithm and it will IP you already specified for hash (not sure what you mean by SDWAN mode round-robin and load-balance mode source-dest-ip).

And that's it. Do not expect that if you specify source-ip traffic will leave via interface that has this IP address.

I am not sure if it clarifies anything, but if you have any questions, let me know.

[BensonLEI]

Hi, guys, 

Thanks so much for your inputs.

Based on your document: 

 SD-WAN Architecture for Enterprise | FortiGate / FortiOS 7.0.0 | Fortinet Documentation Library

7.0 SD-WAN routing logic

Once configured, SD-WAN takes the responsibility of intelligent traffic steering. But how does it interact with the traditional routing subsystem?

The following main rules apply by default:

1. SD-WAN rules are matched only if the best route to the destination points to SD-WAN.

   The best route to the destination must point to any SD-WAN Member—not necessarily the one selected to forward the traffic. This check allows you to easily fit SD-WAN functionality into your existing network topology without disrupting services that are not supposed to be handled by SD-WAN. For example, you may have an out-of-band management network or a group of sites that have not (yet) migrated to SD-WAN. If the best route to the destination does not point to your SD-WAN bundle, the traffic will be handled by conventional routing.

One of our Fortigate with the SDWAN of 2 internet lines, there is no problem for the traceroute tests ( case 1, case 2, case 3).

But traceroute test fails in both case 2 and case 3, while with the SDWAN  of 3 internet lines ( SDWAN issue for 3 lines ? ).

p.s. the above traceroute test results are applied for all SDWAN service-rules cases ( lowest cost (SLA), Maximize Bandwidth (SLA)..)

[BensonLEI]

Hi, guys,

May I know what this item means " Device: auto" from the following commands ?

Forti400e_1 # exec traceroute-options source 111.111.111.11
Forti400e_1 # exec traceroute-options view-settings
Traceroute Options:
Number of probes per hop: 3
Source Address: 111.111.111.11
Device: auto
Use SD-WAN: no

Thanks a lot